DeepAI AI Chat
Log In Sign Up

Structural generalization is hard for sequence-to-sequence models

by   Yuekun Yao, et al.

Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, including ones that require predicting linguistic structure. However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the reasons for these limitations.


page 1

page 2

page 3

page 4


Compositional Generalization Requires Compositional Parsers

A rapidly growing body of research on compositional generalization inves...

Addressing the Data Sparsity Issue in Neural AMR Parsing

Neural attention models have achieved great success in different NLP tas...

Neural AMR: Sequence-to-Sequence Models for Parsing and Generation

Sequence-to-sequence models have shown strong performance across a broad...

The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models

Seq2Seq based neural architectures have become the go-to architecture to...

Compositional Generalization via Semantic Tagging

Although neural sequence-to-sequence models have been successfully appli...

Unobserved Local Structures Make Compositional Generalization Hard

While recent work has convincingly showed that sequence-to-sequence mode...

Improving Sequence-to-Sequence Learning via Optimal Transport

Sequence-to-sequence models are commonly trained via maximum likelihood ...