Encouraging an Appropriate Representation Simplifies Training of Neural Networks

11/17/2019
by   Krisztian Buza, et al.
0

A common assumption about neural networks is that they can learn an appropriate internal representations on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/29/2020

Insights on Training Neural Networks for QUBO Tasks

Current hardware limitations restrict the potential when solving quadrat...
research
10/31/2022

CorrLoss: Integrating Co-Occurrence Domain Knowledge for Affect Recognition

Neural networks are widely adopted, yet the integration of domain knowle...
research
01/24/2017

Training Group Orthogonal Neural Networks with Privileged Information

Learning rich and diverse representations is critical for the performanc...
research
10/07/2020

Abductive Knowledge Induction From Raw Data

For many reasoning-heavy tasks, it is challenging to find an appropriate...
research
01/28/2022

With Greater Distance Comes Worse Performance: On the Perspective of Layer Utilization and Model Generalization

Generalization of deep neural networks remains one of the main open prob...
research
04/10/2021

Regression Networks For Calculating Englacial Layer Thickness

Ice thickness estimation is an important aspect of ice sheet studies. In...
research
03/02/2020

Out-of-Distribution Generalization via Risk Extrapolation (REx)

Generalizing outside of the training distribution is an open challenge f...

Please sign up or login with your details

Forgot password? Click here to reset