A Benchmark for Systematic Generalization in Grounded Language Understanding

03/11/2020
by   Laura Ruis, et al.
8

Human language users easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret compositions unseen in training. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in models of situated language understanding. We take inspiration from standard models of meaning composition in formal linguistics. Going beyond an earlier related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world. This allows us to build novel generalization tasks that probe the acquisition of linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.

READ FULL TEXT
research
08/06/2020

Compositional Networks Enable Systematic Generalization for Grounded Language Understanding

Humans are remarkably flexible when understanding new sentences that inc...
research
01/27/2022

Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understanding

Compositional generalization is a troubling blind spot for neural langua...
research
09/29/2020

Think before you act: A simple baseline for compositional generalization

Contrarily to humans who have the ability to recombine familiar expressi...
research
09/25/2021

Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

We analyze the grounded SCAN (gSCAN) benchmark, which was recently propo...
research
07/19/2018

Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Systematic compositionality is the ability to recombine meaningful units...
research
09/11/2020

Systematic Generalization on gSCAN with Language Conditioned Embedding

Systematic Generalization refers to a learning algorithm's ability to ex...
research
10/01/2019

Emergent Systematic Generalization in a Situated Agent

The question of whether deep neural networks are good at generalising be...

Please sign up or login with your details

Forgot password? Click here to reset