Sparse Flows: Pruning Continuous-depth Models

06/24/2021
by   Lucas Liebenwein, et al.
28

Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. Moreover, pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. Finally, we show that by applying pruning we can obtain insightful information about the design of better neural ODEs.We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.

READ FULL TEXT
research
06/19/2018

Neural Ordinary Differential Equations

We introduce a new family of deep neural network models. Instead of spec...
research
02/22/2021

Neural Delay Differential Equations

Neural Ordinary Differential Equations (NODEs), a framework of continuou...
research
06/09/2019

The Generalization-Stability Tradeoff in Neural Network Pruning

Pruning neural network parameters to reduce model size is an area of muc...
research
07/19/2020

Hypersolvers: Toward Fast Continuous-Depth Models

The infinite-depth paradigm pioneered by Neural ODEs has launched a rena...
research
09/20/2020

TorchDyn: A Neural Differential Equations Library

Continuous-depth learning has recently emerged as a novel perspective on...
research
02/19/2020

Dissecting Neural ODEs

Continuous deep learning architectures have recently re-emerged as varia...
research
03/19/2022

TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed

Continuous normalizing flows (CNFs) construct invertible mappings betwee...

Please sign up or login with your details

Forgot password? Click here to reset