DeepAI AI Chat
Log In Sign Up

An induction proof of the backpropagation algorithm in matrix notation

by   Dirk Ostwald, et al.

Backpropagation (BP) is a core component of the contemporary deep learning incarnation of neural networks. Briefly, BP is an algorithm that exploits the computational architecture of neural networks to efficiently evaluate the gradient of a cost function during neural network parameter optimization. The validity of BP rests on the application of a multivariate chain rule to the computational architecture of neural networks and their associated objective functions. Introductions to deep learning theory commonly present the computational architecture of neural networks in matrix form, but eschew a parallel formulation and justification of BP in the framework of matrix differential calculus. This entails several drawbacks for the theory and didactics of deep learning. In this work, we overcome these limitations by providing a full induction proof of the BP algorithm in matrix notation. Specifically, we situate the BP algorithm in the framework of matrix differential calculus, encompass affine-linear potential functions, prove the validity of the BP algorithm in inductive form, and exemplify the implementation of the matrix form BP algorithm in computer code.


page 6

page 8


Training Neural Networks without Backpropagation: A Deeper Dive into the Likelihood Ratio Method

Backpropagation (BP) is the most important gradient estimation method fo...

When Explanations Lie: Why Modified BP Attribution Fails

Modified backpropagation methods are a popular group of attribution meth...

Artificial Neural Networks for Beginners

The scope of this teaching package is to make a brief induction to Artif...

Alternating optimization method based on nonnegative matrix factorizations for deep neural networks

The backpropagation algorithm for calculating gradients has been widely ...

Using Artificial Bee Colony Algorithm for MLP Training on Earthquake Time Series Data Prediction

Nowadays, computer scientists have shown the interest in the study of so...

Understanding Predictive Coding as an Adaptive Trust-Region Method

Predictive coding (PC) is a brain-inspired local learning algorithm that...

The Backpropagation algorithm for a math student

A Deep Neural Network (DNN) is a composite function of vector-valued fun...