Periodic Extrapolative Generalisation in Neural Networks

09/21/2022
by   Peter Belcak, et al.
0

The learning of the simplest possible computational pattern – periodicity – is an open problem in the research of strong generalisation in neural networks. We formalise the problem of extrapolative generalisation for periodic signals and systematically investigate the generalisation abilities of classical, population-based, and recently proposed periodic architectures on a set of benchmarking tasks. We find that periodic and "snake" activation functions consistently fail at periodic extrapolation, regardless of the trainability of their periodicity parameters. Further, our results show that traditional sequential models still outperform the novel architectures designed specifically for extrapolation, and that these are in turn trumped by population-based training. We make our benchmarking and evaluation toolkit, PerKit, available and easily accessible to facilitate future work in the area.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/15/2020

Neural Networks Fail to Learn Periodic Functions and How to Fix It

Previous literature offers limited clues on how to learn a periodic func...
research
10/26/2021

Periodic Activation Functions Induce Stationarity

Neural network models are known to reinforce hidden data biases, making ...
research
06/17/2020

Implicit Neural Representations with Periodic Activation Functions

Implicitly defined, continuous, differentiable signal representations pa...
research
04/18/2023

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Many industrial and real life problems exhibit highly nonlinear periodic...
research
10/16/2019

Quasiperiodic bobbin lace patterns

Bobbin lace is a fibre art form in which threads are braided together to...
research
10/24/2021

Conditional Generation of Periodic Signals with Fourier-Based Decoder

Periodic signals play an important role in daily lives. Although convent...
research
02/27/2019

Formal structure of periodic system of elements

For more than 150 years the structure of the periodic system of the chem...

Please sign up or login with your details

Forgot password? Click here to reset