Semantics, Representations and Grammars for Deep Learning

by   David Balduzzi, et al.

Deep learning is currently the subject of intensive study. However, fundamental concepts such as representations are not formally defined -- researchers "know them when they see them" -- and there is no common language for describing and analyzing algorithms. This essay proposes an abstract framework that identifies the essential features of current practice and may provide a foundation for future developments. The backbone of almost all deep learning algorithms is backpropagation, which is simply a gradient computation distributed over a neural network. The main ingredients of the framework are thus, unsurprisingly: (i) game theory, to formalize distributed optimization; and (ii) communication protocols, to track the flow of zeroth and first-order information. The framework allows natural definitions of semantics (as the meaning encoded in functions), representations (as functions whose semantics is chosen to optimized a criterion) and grammars (as communication protocols equipped with first-order convergence guarantees). Much of the essay is spent discussing examples taken from the literature. The ultimate aim is to develop a graphical language for describing the structure of deep learning algorithms that backgrounds the details of the optimization procedure and foregrounds how the components interact. Inspiration is taken from probabilistic graphical models and factor graphs, which capture the essential structural features of multivariate distributions.


page 1

page 2

page 3

page 4


A structural operational semantics for interactions with a look at loops

Message Sequence Charts Sequence Diagrams are graphical models that ...

Benchpress: a scalable and platform-independent workflow for benchmarking structure learning algorithms for graphical models

Describing the relationship between the variables in a study domain and ...

Deep Learning of Representations: Looking Forward

Deep learning research aims at discovering learning algorithms that disc...

Operational Semantics with Hierarchical Abstract Syntax Graphs

This is a motivating tutorial introduction to a semantic analysis of pro...

Probabilistic Dependency Graphs

We introduce Probabilistic Dependency Graphs (PDGs), a new class of dire...

Intensionalizing Abstract Meaning Representations: Non-Veridicality and Scope

Abstract Meaning Representation (AMR) is a graphical meaning representat...

Please sign up or login with your details

Forgot password? Click here to reset