DeepAI AI Chat
Log In Sign Up

The differential calculus of causal functions

04/24/2019
by   David Sprunger, et al.
Radboud Universiteit
0

Causal functions of sequences occur throughout computer science, from theory to hardware to machine learning. Mealy machines, synchronous digital circuits, signal flow graphs, and recurrent neural networks all have behaviour that can be described by causal functions. In this work, we examine a differential calculus of causal functions which includes many of the familiar properties of standard multivariable differential calculus. These causal functions operate on infinite sequences, but this work gives a different notion of an infinite-dimensional derivative than either the Fréchet or Gateaux derivative used in functional analysis. In addition to showing many standard properties of differentiation, we show causal differentiation obeys a unique recurrence rule. We use this recurrence rule to compute the derivative of a simple recurrent neural network called an Elman network by hand and describe how the computed derivative can be used to train the network.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/29/2020

Linearizing Combinators

In 2017, Bauer, Johnson, Osborne, Riehl, and Tebbe (BJORT) showed that t...
03/04/2019

Differentiable Causal Computations via Delayed Trace

We investigate causal computations taking sequences of inputs to sequenc...
02/04/2020

Cartesian Difference Categories: Extended Report

Cartesian differential categories are categories equipped with a differe...
09/14/2020

A Finitist's Manifesto: Do we need to Reformulate the Foundations of Mathematics?

There is a problem with the foundations of classical mathematics, and po...
07/02/2022

Combinatory Adjoints and Differentiation

We develop a compositional approach for automatic and symbolic different...
01/08/2021

Infinite-dimensional Folded-in-time Deep Neural Networks

The method recently introduced in arXiv:2011.10115 realizes a deep neura...
09/13/2022

Monadic and Comonadic Aspects of Dependency Analysis

Dependency analysis is vital to several applications in computer science...