Semi-discrete optimization through semi-discrete optimal transport: a framework for neural architecture search

06/26/2020
by   Nicolas Garcia Trillos, et al.
8

In this paper we introduce a theoretical framework for semi-discrete optimization using ideas from optimal transport. Our primary motivation is in the field of deep learning, and specifically in the task of neural architecture search. With this aim in mind, we discuss the geometric and theoretical motivation for new techniques for neural architecture search (in the companion work <cit.>; we show that algorithms inspired by our framework are competitive with contemporaneous methods). We introduce a Riemannian like metric on the space of probability measures over a semi-discrete space ℝ^d ×𝒢 where 𝒢 is a finite weighted graph. With such Riemmanian structure in hand, we derive formal expressions for the gradient flow of a relative entropy functional, as well as second order dynamics for the optimization of said energy. Then, with the aim of providing a rigorous motivation for the gradient flow equations derived formally we also consider an iterative procedure known as minimizing movement scheme (i.e., Implicit Euler scheme, or JKO scheme) and apply it to the relative entropy with respect to a suitable cost function. For some specific choices of metric and cost, we rigorously show that the minimizing movement scheme of the relative entropy functional converges to the gradient flow process provided by the formal Riemannian structure. This flow coincides with a system of reaction-diffusion equations on ℝ^d.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/26/2020

Traditional and accelerated gradient descent for neural architecture search

In this paper, we introduce two algorithms for neural architecture searc...
03/16/2018

Natural gradient via optimal transport I

We study a natural Wasserstein gradient flow on manifolds of probability...
02/07/2020

Wasserstein Proximal Gradient

We consider the task of sampling from a log-concave probability distribu...
02/11/2018

Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Bayesian Optimisation (BO) refers to a class of methods for global optim...
06/13/2020

Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search

Neural architecture search (NAS) automates the design of deep neural net...
08/23/2021

Relative Entropy-Regularized Optimal Transport on a Graph: a new algorithm and an experimental comparison

Following [21, 23], the present work investigates a new relative entropy...
11/17/2020

Asymptotics of Entropy-Regularized Optimal Transport via Chaos Decomposition

Consider the problem of estimating the optimal coupling (i.e., matching)...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.