Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

by   Linlu Qiu, et al.

We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding. First, we study which aspects of the original benchmark can be solved by commonly used methods in multi-modal research. We find that a general-purpose Transformer-based model with cross-modal attention achieves strong performance on a majority of the gSCAN splits, surprisingly outperforming more specialized approaches from prior work. Furthermore, our analysis suggests that many of the remaining errors reveal the same fundamental challenge in systematic generalization of linguistic constructs regardless of visual context. Second, inspired by this finding, we propose challenging new tasks for gSCAN by generating data to incorporate relations between objects in the visual environment. Finally, we find that current models are surprisingly data inefficient given the narrow scope of commands in gSCAN, suggesting another challenge for future work.


A Benchmark for Systematic Generalization in Grounded Language Understanding

Human language users easily interpret expressions that describe unfamili...

SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach

The SimpleQuestions dataset is one of the most commonly used benchmarks ...

XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding

Transformer-based models are widely used in natural language understandi...

Systematic Generalization: What Is Required and Can It Be Learned?

Numerous models for grounded language understanding have been recently p...

Compositional Networks Enable Systematic Generalization for Grounded Language Understanding

Humans are remarkably flexible when understanding new sentences that inc...

Think before you act: A simple baseline for compositional generalization

Contrarily to humans who have the ability to recombine familiar expressi...

Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes

Current work on automatic coreference resolution has focused on the Onto...

Please sign up or login with your details

Forgot password? Click here to reset