DeepAI AI Chat
Log In Sign Up

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

by   Peter Shaw, et al.

Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? To better assess this capability, we propose new train and test splits of non-synthetic datasets. We demonstrate that strong existing semantic parsing approaches do not yet perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a high-precision grammar-based approach with a pre-trained sequence-to-sequence model. It outperforms existing approaches across several compositional generalization challenges, while also being competitive with the state-of-the-art on standard evaluations. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing.


page 1

page 2

page 3

page 4


Improving Compositional Generalization with Latent Structure and Data Augmentation

Generic unstructured neural networks have been shown to struggle on out-...

Compositional Generalization for Natural Language Interfaces to Web APIs

This paper presents Okapi, a new dataset for Natural Language to executa...

Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

While mainstream machine learning methods are known to have limited abil...

Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions

Idiomatic expressions (IEs), characterized by their non-compositionality...

Analyzing Compositionality-Sensitivity of NLI Models

Success in natural language inference (NLI) should require a model to un...

Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification

Pre-trained seq2seq models excel at graph semantic parsing with rich ann...

Compositional Generalization via Semantic Tagging

Although neural sequence-to-sequence models have been successfully appli...