On Incorporating Semantic Prior Knowlegde in Deep Learning Through Embedding-Space Constraints

09/30/2019
by   Damien Teney, et al.
52

The knowledge that humans hold about a problem often extends far beyond a set of training data and output labels. While the success of deep learning mostly relies on supervised training, important properties cannot be inferred efficiently from end-to-end annotations alone, for example causal relations or domain-specific invariances. We present a general technique to supplement supervised training with prior knowledge expressed as relations between training instances. We illustrate the method on the task of visual question answering to exploit various auxiliary annotations, including relations of equivalence and of logical entailment between questions. Existing methods to use these annotations, including auxiliary losses and data augmentation, cannot guarantee the strict inclusion of these relations into the model since they require a careful balancing against the end-to-end objective. Our method uses these relations to shape the embedding space of the model, and treats them as strict constraints on its learned representations. In the context of VQA, this approach brings significant improvements in accuracy and robustness, in particular over the common practice of incorporating the constraints as a soft regularizer. We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used. It demonstrates the value of an additional training signal that is otherwise difficult to extract from end-to-end annotations alone.

READ FULL TEXT
research
05/04/2020

Visual Question Answering with Prior Class Semantics

We present a novel mechanism to embed prior knowledge in a model for vis...
research
11/30/2018

From Known to the Unknown: Transferring Knowledge to Answer Questions about Novel Visual and Semantic Concepts

Current Visual Question Answering (VQA) systems can answer intelligent q...
research
02/27/2021

Incorporating Causal Graphical Prior Knowledge into Predictive Modeling via Simple Data Augmentation

Causal graphs (CGs) are compact representations of the knowledge of the ...
research
05/04/2022

Self-Taught Metric Learning without Labels

We present a novel self-taught framework for unsupervised metric learnin...
research
05/07/2018

Improving Knowledge Graph Embedding Using Simple Constraints

Embedding knowledge graphs (KGs) into continuous vector spaces is a focu...
research
09/29/2020

Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation

Neural models that independently project questions and answers into a sh...

Please sign up or login with your details

Forgot password? Click here to reset