Grammar-Based Grounded Lexicon Learning

by   Jiayuan Mao, et al.

We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts. At the core of G2L2 is a collection of lexicon entries, which map each word to a tuple of a syntactic type and a neuro-symbolic semantic program. For example, the word shiny has a syntactic type of adjective; its neuro-symbolic semantic program has the symbolic form λx. filter(x, SHINY), where the concept SHINY is associated with a neural network embedding, which will be used to classify shiny objects. Given an input sentence, G2L2 first looks up the lexicon entries associated with each token. It then derives the meaning of the sentence as an executable neuro-symbolic program by composing lexical meanings based on syntax. The recovered meaning programs can be executed on grounded inputs. To facilitate learning in an exponentially-growing compositional space, we introduce a joint parsing and expected execution algorithm, which does local marginalization over derivations to reduce the training time. We evaluate G2L2 on two domains: visual reasoning and language-driven navigation. Results show that G2L2 can generalize from small amounts of data to novel compositions of words.


page 1

page 2

page 3

page 4


The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that lear...

Compositional Generalization via Neural-Symbolic Stack Machines

Despite achieving tremendous success, existing deep learning models have...

Dynamic Neural Program Embedding for Program Repair

Neural program embeddings have shown much promise recently for a variety...

Density Matrices for Derivational Ambiguity

Recent work on vector-based compositional natural language semantics has...

Towards Concept Formation Grounded on Perception and Action of a Mobile Robot

The recognition of objects and, hence, their descriptions must be ground...

Type-Driven Incremental Semantic Parsing with Polymorphism

Semantic parsing has made significant progress, but most current semanti...

Visually Grounded Neural Syntax Acquisition

We present the Visually Grounded Neural Syntax Learner (VG-NSL), an appr...

Please sign up or login with your details

Forgot password? Click here to reset