Informed Pre-Training on Prior Knowledge

05/23/2022
by   Laura von Rueden, et al.
0

When training data is scarce, the incorporation of additional prior knowledge can assist the learning process. While it is common to initialize neural networks with weights that have been pre-trained on other large data sets, pre-training on more concise forms of knowledge has rather been overlooked. In this paper, we propose a novel informed machine learning approach and suggest to pre-train on prior knowledge. Formal knowledge representations, e.g. graphs or equations, are first transformed into a small and condensed data set of knowledge prototypes. We show that informed pre-training on such knowledge prototypes (i) speeds up the learning processes, (ii) improves generalization capabilities in the regime where not enough training data is available, and (iii) increases model robustness. Analyzing which parts of the model are affected most by the prototypes reveals that improvements come from deeper layers that typically represent high-level features. This confirms that informed pre-training can indeed transfer semantic knowledge. This is a novel effect, which shows that knowledge-based pre-training has additional and complementary strengths to existing approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/31/2022

Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition

Self-supervised learning (SSL) to learn high-level speech representation...
research
11/01/2022

Informed Priors for Knowledge Integration in Trajectory Prediction

Informed machine learning methods allow the integration of prior knowled...
research
05/09/2022

EigenNoise: A Contrastive Prior to Warm-Start Representations

In this work, we present a naive initialization scheme for word vectors ...
research
10/19/2022

A Linguistic Investigation of Machine Learning based Contradiction Detection Models: An Empirical Analysis and Future Perspectives

We analyze two Natural Language Inference data sets with respect to thei...
research
12/14/2022

Towards Linguistically Informed Multi-Objective Pre-Training for Natural Language Inference

We introduce a linguistically enhanced combination of pre-training metho...
research
03/13/2020

Dynamic transformation of prior knowledge intoBayesian models for data streams

We consider how to effectively use prior knowledge when learning a Bayes...
research
01/14/2022

Learning from One and Only One Shot

Humans can generalize from only a few examples and from little pre-train...

Please sign up or login with your details

Forgot password? Click here to reset