Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

03/29/2017
by   Will Monroe, et al.
0

We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework. Experiments show that this combined pragmatic model interprets color descriptions more accurately than the classifiers from which it is built, and that much of this improvement results from combining the speaker and listener perspectives. We observe that pragmatic reasoning helps primarily in the hardest cases: when the model must distinguish very similar colors, or when few utterances adequately express the target color. Our findings make use of a newly-collected corpus of human utterances in color reference games, which exhibit a variety of pragmatic behaviors. We also show that the embedded speaker model reproduces many of these pragmatic behaviors.

READ FULL TEXT

page 2

page 11

research
06/13/2016

Learning to Generate Compositional Color Descriptions

The production of color language is essential for grounded language gene...
research
10/09/2020

Pragmatically Informative Color Generation by Grounding Contextual Modifiers

Grounding language in contextual information is crucial for fine-grained...
research
07/05/2021

Accessible Color Cycles for Data Visualization

Color cycles, ordered sets of colors for data visualization, that balanc...
research
04/11/2022

Linguistic communication as (inverse) reward design

Natural language is an intuitive and expressive way to communicate rewar...
research
03/11/2018

Generating Bilingual Pragmatic Color References

Contextual influences on language exhibit substantial language-independe...
research
05/31/2023

Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind

Dialogue participants may have varying levels of knowledge about the top...
research
04/02/2016

Reasoning About Pragmatics with Neural Listeners and Speakers

We present a model for pragmatically describing scenes, in which contras...

Please sign up or login with your details

Forgot password? Click here to reset