PROTOtypical Logic Tensor Networks (PROTO-LTN) for Zero Shot Learning

by   Lia Morra, et al.

Semantic image interpretation can vastly benefit from approaches that combine sub-symbolic distributed representation learning with the capability to reason at a higher level of abstraction. Logic Tensor Networks (LTNs) are a class of neuro-symbolic systems based on a differentiable, first-order logic grounded into a deep neural network. LTNs replace the classical concept of training set with a knowledge base of fuzzy logical axioms. By defining a set of differentiable operators to approximate the role of connectives, predicates, functions and quantifiers, a loss function is automatically specified so that LTNs can learn to satisfy the knowledge base. We focus here on the subsumption or isOfClass predicate, which is fundamental to encode most semantic image interpretation tasks. Unlike conventional LTNs, which rely on a separate predicate for each class (e.g., dog, cat), each with its own set of learnable weights, we propose a common isOfClass predicate, whose level of truth is a function of the distance between an object embedding and the corresponding class prototype. The PROTOtypical Logic Tensor Networks (PROTO-LTN) extend the current formulation by grounding abstract concepts as parametrized class prototypes in a high-dimensional embedding space, while reducing the number of parameters required to ground the knowledge base. We show how this architecture can be effectively trained in the few and zero-shot learning scenarios. Experiments on Generalized Zero Shot Learning benchmarks validate the proposed implementation as a competitive alternative to traditional embedding-based approaches. The proposed formulation opens up new opportunities in zero shot learning settings, as the LTN formalism allows to integrate background knowledge in the form of logical axioms to compensate for the lack of labelled examples.


Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

Semantic Image Interpretation is the task of extracting a structured sem...

Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings

We improve zero-shot learning (ZSL) by incorporating common-sense knowle...

Faster-LTN: a neuro-symbolic, end-to-end object detection architecture

The detection of semantic relationships between objects represented in a...

Wider Vision: Enriching Convolutional Neural Networks via Alignment to External Knowledge Bases

Deep learning models suffer from opaqueness. For Convolutional Neural Ne...

DeepPSL: End-to-end perception and reasoning with applications to zero shot learning

We introduce DeepPSL a variant of Probabilistic Soft Logic (PSL) to prod...

Representing Prior Knowledge Using Randomly, Weighted Feature Networks for Visual Relationship Detection

The single-hidden-layer Randomly Weighted Feature Network (RWFN) introdu...

Augmenting Neural Nets with Symbolic Synthesis: Applications to Few-Shot Learning

We propose symbolic learning as extensions to standard inductive learnin...

Please sign up or login with your details

Forgot password? Click here to reset