Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

09/13/2021
by   Mostafa Abdou, et al.
0

Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases – (Paris, Capital, France). However, simple relations of this type can often be recovered heuristically and the extent to which models implicitly reflect topological structure that is grounded in world, such as perceptual structure, is unknown. To explore this question, we conduct a thorough case study on color. Namely, we employ a dataset of monolexemic color terms and color chips represented in CIELAB, a color space with a perceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text-derived color term representations, we find significant correspondence. Analyzing the differences in alignment across the color spectrum, we find that warmer colors are, on average, better aligned to the perceptual color space than cooler ones, suggesting an intriguing connection to findings from recent work on efficient communication in color naming. Further analysis suggests that differences in alignment are, in part, mediated by collocationality and differences in syntactic usage, posing questions as to the relationship between color perception and usage and context.

READ FULL TEXT

page 14

page 15

page 16

page 17

page 18

page 19

page 20

page 21

11/23/2012

Improving Perceptual Color Difference using Basic Color Terms

We suggest a new color distance based on two observations. First, percep...
05/16/2018

Color naming reflects both perceptual structure and communicative need

Gibson et al. (2017) argued that color naming is shaped by patterns of c...
10/15/2021

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

Recent work has raised concerns about the inherent limitations of text-o...
05/28/2022

Perceptually Optimized Color Selection for Visualization

We propose an approach, called the Equilibrium Distribution Model (EDM),...
09/17/2019

Grounding learning of modifier dynamics: An application to color naming

Grounding is crucial for natural language understanding. An important su...
10/09/2020

Pragmatically Informative Color Generation by Grounding Contextual Modifiers

Grounding language in contextual information is crucial for fine-grained...
09/07/2020

Semantic Discriminability for Visual Communication

To interpret information visualizations, observers must determine how vi...