DeepAI
Log In Sign Up

DALL-E 2 Fails to Reliably Capture Common Syntactic Processes

10/23/2022
by   Evelina Leivada, et al.
1

Machine intelligence is increasingly being linked to claims about sentience, language processing, and an ability to comprehend and transform natural language into a range of stimuli. We systematically analyze the ability of DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference, passives, word order, coordination, comparatives, negation, ellipsis, and structural ambiguity. Whereas young children routinely master these phenomena, learning systematic mappings between syntax and semantics, DALL-E 2 is unable to reliably infer meanings that are consistent with the syntax. These results challenge recent claims concerning the capacity of such systems to understand of human language. We make available the full set of test materials as a benchmark for future testing.

READ FULL TEXT

page 4

page 6

page 8

page 10

page 12

page 19

page 20

page 22

10/02/2020

Syntax Representation in Word Embeddings and Neural Networks – A Survey

Neural networks trained on natural language processing tasks capture syn...
12/02/2019

BLiMP: A Benchmark of Linguistic Minimal Pairs for English

We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLi...
08/26/2019

Does BERT agree? Evaluating knowledge of structure dependence through agreement relations

Learning representations that accurately model semantics is an important...
06/06/2021

A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans

We present a targeted, scaled-up comparison of incremental processing in...
02/20/2020

The Fluidity of Concept Representations in Human Brain Signals

Cognitive theories of human language processing often distinguish betwee...
07/22/2019

Less (Data) Is More: Why Small Data Holds the Key to the Future of Artificial Intelligence

The claims that big data holds the key to enterprise successes and that ...