Engineering flexible machine learning systems by traversing functionally invariant paths in weight space

04/30/2022
by   Guruprasad Raghavan, et al.
11

Deep neural networks achieve human-like performance on a variety of perceptual and decision making tasks. However, deep networks perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Here, we develop a mathematical and algorithmic framework that enables continual training of deep neural networks on a broad range of objectives by defining path connected sets of neural networks that achieve equivalent functional performance on a given machine learning task while modulating network weights to achieve high-performance on a secondary objective. We view the weight space of a neural network as a curved Riemannian manifold and move a neural network along a functionally invariant path in weight space while searching for networks that satisfy a secondary objective. We introduce a path-sampling algorithm that trains networks with millions of weight parameters to learn a series of image classification tasks without performance loss. The algorithm generalizes to accommodate a range of secondary objectives including weight-pruning and weight diversification and exhibits state of the art performance on network compression and adversarial robustness benchmarks. Broadly, we demonstrate how the intrinsic geometry of machine learning problems can be harnessed to construct flexible and robust neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 6

page 8

page 9

page 10

page 13

research
06/05/2021

Solving hybrid machine learning tasks by traversing weight space geodesics

Machine learning problems have an intrinsic geometric structure as centr...
research
05/12/2020

RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

We propose RSO (random search optimization), a gradient free Markov Chai...
research
06/17/2020

Constraint-Based Regularization of Neural Networks

We propose a method for efficiently incorporating constraints into a sto...
research
09/07/2022

The Role Of Biology In Deep Learning

Artificial neural networks took a lot of inspiration from their biologic...
research
12/20/2022

Constructing Organism Networks from Collaborative Self-Replicators

We introduce organism networks, which function like a single neural netw...
research
12/12/2020

Sparsifying networks by traversing Geodesics

The geometry of weight spaces and functional manifolds of neural network...
research
02/13/2020

Classifying the classifier: dissecting the weight space of neural networks

This paper presents an empirical study on the weights of neural networks...

Please sign up or login with your details

Forgot password? Click here to reset