Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

10/04/2020
by   William S. Moses, et al.
0

Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine learning workflows.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/26/2018

Automatic differentiation in ML: Where we are and where we should be going

We review the current state of automatic differentiation (AD) for array ...
research
10/23/2018

Automatic Full Compilation of Julia Programs and ML Models to Cloud TPUs

Google's Cloud TPUs are a promising new hardware architecture for machin...
research
04/13/2023

Fast And Automatic Floating Point Error Analysis With CHEF-FP

As we reach the limit of Moore's Law, researchers are exploring differen...
research
09/05/2022

Forward-Mode Automatic Differentiation of Compiled Programs

Algorithmic differentiation (AD) is a set of techniques that provide par...
research
04/24/2023

HDCC: A Hyperdimensional Computing compiler for classification on embedded systems and high-performance computing

Hyperdimensional Computing (HDC) is a bio-inspired computing framework t...
research
05/29/2019

MCompiler: A Synergistic Compilation Framework

This paper presents a meta-compilation framework, the MCompiler. The mai...
research
09/26/2018

Relay: A New IR for Machine Learning Frameworks

Machine learning powers diverse services in industry including search, t...

Please sign up or login with your details

Forgot password? Click here to reset