Equivariant Architectures for Learning in Deep Weight Spaces

01/30/2023
by   Aviv Navon, et al.
0

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

READ FULL TEXT

page 4

page 17

page 24

research
02/27/2023

Permutation Equivariant Neural Functionals

This work studies the design of neural networks that can process the wei...
research
03/03/2020

Deep Learning in Memristive Nanowire Networks

Analog crossbar architectures for accelerating neural network training a...
research
05/22/2023

Neural Functional Transformers

The recent success of neural networks as implicit representation of data...
research
11/13/2020

Investigating Learning in Deep Neural Networks using Layer-Wise Weight Change

Understanding the per-layer learning dynamics of deep neural networks is...
research
01/30/2019

Tensorized Embedding Layers for Efficient Model Compression

The embedding layers transforming input words into real vectors are the ...
research
12/14/2016

Permutation-equivariant neural networks applied to dynamics prediction

The introduction of convolutional layers greatly advanced the performanc...
research
04/04/2019

Template-Based Automatic Search of Compact Semantic Segmentation Architectures

Automatic search of neural architectures for various vision and natural ...

Please sign up or login with your details

Forgot password? Click here to reset