DeepAI AI Chat
Log In Sign Up

Denotationally Correct, Purely Functional, Efficient Reverse-mode Automatic Differentiation

by   Mathieu Huot, et al.
University of Oxford

Reverse-mode differentiation is used for optimization, but it introduces references, which break the purity of the underlying programs, making them notoriously harder to optimize. We present a reverse-mode differentiation on a purely functional language with array operations. It is the first one to deliver a provably efficient, purely functional, and denotationally correct reverse-mode differentiation. We show that our transformation is semantically correct and verifies the cheap gradient principle. Inspired by PROPs and compilation to categories, we introduce a novel intermediate representation that we call 'unary form'. Our reverse-mode transformation is factored as a compilation scheme through this intermediate representation. We obtain provably efficient gradients by performing general partial evaluation optimizations after our reverse-mode transformation, as opposed to manually derived ones. For simple first-order programs, the obtained output programs resemble static-single-assignment (SSA) code. We emphasize the modularity of our approach and show how our language can easily be enriched with more optimized primitives, as required for some speed-ups in practice.


page 1

page 2

page 3

page 4


Differentiating a Tensor Language

How does one compile derivatives of tensor programs, such that the resul...

Efficient CHAD

We show how the basic Combinatory Homomorphic Automatic Differentiation ...

Efficient and Sound Differentiable Programming in a Functional Array-Processing Language

Automatic differentiation (AD) is a technique for computing the derivati...

CHAD for Expressive Total Languages

We show how to apply forward and reverse mode Combinatory Homomorphic Au...

Dynamic Automatic Differentiation of GPU Broadcast Kernels

We show how forward-mode automatic differentiation (AD) can be employed ...

Monoidal Reverse Differential Categories

Cartesian reverse differential categories (CRDCs) are a recently defined...

The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

As computational challenges in optimization and statistical inference gr...