Strong mixed-integer programming formulations for trained neural networks

by   Ross Anderson, et al.

We present an ideal mixed-integer programming (MIP) formulation for a rectified linear unit (ReLU) appearing in a trained neural network. Our formulation requires a single binary variable and no additional continuous variables beyond the input and output variables of the ReLU. We contrast it with an ideal "extended" formulation with a linear number of additional continuous variables, derived through standard techniques. An apparent drawback of our formulation is that it requires an exponential number of inequality constraints, but we provide a routine to separate the inequalities in linear time. We also prove that these exponentially-many constraints are facet-defining under mild conditions. Finally, we present computational results showing that dynamically separating from the exponential inequalities 1) is much more computationally efficient and scalable than the extended formulation, 2) decreases the solve time of a state-of-the-art MIP solver by a factor of 7 on smaller instances, and 3) nearly matches the dual bounds of a state-of-the-art MIP solver on harder instances, after just a few rounds of separation and in orders of magnitude less time.



There are no comments yet.


page 1

page 2

page 3

page 4


ReLU Networks as Surrogate Models in Mixed-Integer Linear Programs

We consider the embedding of piecewise-linear deep neural networks (ReLU...

Mixed integer formulations using natural variables for single machine scheduling around a common due date

While almost all existing works which optimally solve just-in-time sched...

Binary extended formulations and sequential convexification

A binarization of a bounded variable x is a linear formulation with vari...

Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility Study

Deep Neural Networks (DNNs) are very popular these days, and are the sub...

Enhanced Formulation for Guillotine 2D Cutting Problems

We advance the state of the art in Mixed-Integer Linear Programming (MIL...

Extended formulation and valid inequalities for the multi-item inventory lot-sizing problem with supplier selection

This paper considers the multi-item inventory lot-sizing problem with su...

A Polyhedral Study for the Cubic Formulation of the Unconstrained Traveling Tournament Problem

We consider the unconstrained traveling tournament problem, a sports tim...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.