Comparing the Parameter Complexity of Hypernetworks and the Embedding-Based Alternative

02/23/2020
by   Tomer Galanti, et al.
16

In the context of learning to map an input I to a function h_I:X→R, we compare two alternative methods: (i) an embedding-based method, which learns a fixed function in which I is encoded as a conditioning signal e(I) and the learned function takes the form h_I(x) = q(x,e(I)), and (ii) hypernetworks, in which the weights θ_I of the function h_I(x) = g(x;θ_I) are given by a hypernetwork f as θ_I=f(I). We extend the theory of <cit.> and provide a lower bound on the complexity of neural networks as function approximators, i.e., the number of trainable parameters. This extension, eliminates the requirements for the approximation method to be robust. Our results are then used to compare the complexities of q and g, showing that under certain conditions and when letting the functions e and f be as large as we wish, g can be smaller than q by orders of magnitude. In addition, we show that for typical assumptions on the function to be approximated, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2023

Hubbard-Stratonovich Detector for Simple Trainable MIMO Signal Detection

Massive multiple-input multiple-output (MIMO) is a key technology used i...
research
06/24/2020

Towards Understanding Hierarchical Learning: Benefits of Neural Representations

Deep neural networks can empirically perform efficient hierarchical lear...
research
10/08/2018

Empirical Bounds on Linear Regions of Deep Rectifier Networks

One form of characterizing the expressiveness of a piecewise linear neur...
research
07/28/2021

Neural Network Approximation of Refinable Functions

In the desire to quantify the success of neural networks in deep learnin...
research
11/25/2014

Efficient Algorithms for Bayesian Network Parameter Learning from Incomplete Data

We propose an efficient family of algorithms to learn the parameters of ...
research
11/03/2020

Small footprint Text-Independent Speaker Verification for Embedded Systems

Deep neural network approaches to speaker verification have proven succe...
research
05/18/2022

Trading Positional Complexity vs. Deepness in Coordinate Networks

It is well noted that coordinate-based MLPs benefit – in terms of preser...

Please sign up or login with your details

Forgot password? Click here to reset