DeepAI AI Chat
Log In Sign Up

Categorical Foundations of Gradient-Based Learning

by   G. S. H. Cruttwell, et al.

We propose a categorical foundation of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach also generalises beyond neural networks (modelled in categories of smooth maps), accounting for other structures relevant to gradient-based learning such as boolean circuits. Finally, we also develop a novel implementation of gradient-based learning in Python, informed by the principles introduced by our framework.


page 1

page 2

page 3

page 4


Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits

We introduce Reverse Derivative Ascent: a categorical analogue of gradie...

Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent

We present a unifying framework for adapting the update direction in gra...

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Countless signal processing applications include the reconstruction of s...

Category Theory in Machine Learning

Over the past two decades machine learning has permeated almost every re...

Categories of Differentiable Polynomial Circuits for Machine Learning

Reverse derivative categories (RDCs) have recently been shown to be a su...

Topological Gradient-based Competitive Learning

Topological learning is a wide research area aiming at uncovering the mu...

Consistent Estimators for Learning to Defer to an Expert

Learning algorithms are often used in conjunction with expert decision m...