DeepAI AI Chat
Log In Sign Up

Differentiating a Tensor Language

by   Gilbert Bernstein, et al.

How does one compile derivatives of tensor programs, such that the resulting code is purely functional (hence easier to optimize and parallelize) and provably efficient relative to the original program? We show that naively differentiating tensor code—as done in popular systems like Tensorflow and PyTorch—can cause asymptotic slowdowns in pathological cases, violating the Cheap Gradients Principle. However, all existing automatic differentiation methods that guarantee this principle (for variable size data) do so by relying on += mutation through aliases/pointers—which complicates downstream optimization. We provide the first purely functional, provably efficient, adjoint/reverse-mode derivatives of array/tensor code by explicitly accounting for sparsity. We do this by focusing on the indicator function from Iverson's APL. We also introduce a new "Tensor SSA" normal form and a new derivation of reverse-mode automatic differentiation based on the universal property of inner-products.


page 6

page 7

page 8

page 14

page 18

page 22

page 27

page 29


Denotationally Correct, Purely Functional, Efficient Reverse-mode Automatic Differentiation

Reverse-mode differentiation is used for optimization, but it introduces...

Source-to-Source Automatic Differentiation of OpenMP Parallel Loops

This paper presents our work toward correct and efficient automatic diff...

CHAD for Expressive Total Languages

We show how to apply forward and reverse mode Combinatory Homomorphic Au...

A Simple and Efficient Tensor Calculus for Machine Learning

Computing derivatives of tensor expressions, also known as tensor calcul...

Reverse AD at Higher Types: Pure, Principled and Denotationally Correct

We show how to define source-code transformations for forward- and rever...

The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

As computational challenges in optimization and statistical inference gr...

Banded Matrix Operators for Gaussian Markov Models in the Automatic Differentiation Era

Banded matrices can be used as precision matrices in several models incl...