Hyper-Representations for Pre-Training and Transfer Learning

07/22/2022
by   Konstantin Schürholt, et al.
7

Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we extend hyper-representations for generative use to sample new model weights as pre-training. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and a sampling method based on the empirical density of hyper-representations. The models generated using our methods are diverse, performant and capable to outperform conventional baselines for transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/29/2022

Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Learning representations of neural network weights given a model zoo is ...
research
10/07/2021

Conceptual Expansion Neural Architecture Search (CENAS)

Architecture search optimizes the structure of a neural network for some...
research
06/07/2015

Knowledge Transfer Pre-training

Pre-training is crucial for learning deep neural networks. Most of exist...
research
11/16/2018

Domain Adaptive Transfer Learning with Specialist Models

Transfer learning is a widely used method to build high performing compu...
research
11/18/2019

Towards Making Deep Transfer Learning Never Hurt

Transfer learning have been frequently used to improve deep neural netwo...
research
12/01/2020

Solvable Model for Inheriting the Regularization through Knowledge Distillation

In recent years the empirical success of transfer learning with neural n...
research
03/02/2019

neuralRank: Searching and ranking ANN-based model repositories

Widespread applications of deep learning have led to a plethora of pre-t...

Please sign up or login with your details

Forgot password? Click here to reset