DeepAI AI Chat
Log In Sign Up

Categorical Foundations of Gradient-Based Learning

03/02/2021
by   G. S. H. Cruttwell, et al.
0

We propose a categorical foundation of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach also generalises beyond neural networks (modelled in categories of smooth maps), accounting for other structures relevant to gradient-based learning such as boolean circuits. Finally, we also develop a novel implementation of gradient-based learning in Python, informed by the principles introduced by our framework.

READ FULL TEXT

page 1

page 2

page 3

page 4

01/26/2021

Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits

We introduce Reverse Derivative Ascent: a categorical analogue of gradie...
07/07/2016

Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent

We present a unifying framework for adapting the update direction in gra...
02/07/2022

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Countless signal processing applications include the reconstruction of s...
06/13/2021

Category Theory in Machine Learning

Over the past two decades machine learning has permeated almost every re...
03/12/2022

Categories of Differentiable Polynomial Circuits for Machine Learning

Reverse derivative categories (RDCs) have recently been shown to be a su...
08/21/2020

Topological Gradient-based Competitive Learning

Topological learning is a wide research area aiming at uncovering the mu...
06/02/2020

Consistent Estimators for Learning to Defer to an Expert

Learning algorithms are often used in conjunction with expert decision m...