Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

by   Mostafa Abdou, et al.

Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases – (Paris, Capital, France). However, simple relations of this type can often be recovered heuristically and the extent to which models implicitly reflect topological structure that is grounded in world, such as perceptual structure, is unknown. To explore this question, we conduct a thorough case study on color. Namely, we employ a dataset of monolexemic color terms and color chips represented in CIELAB, a color space with a perceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text-derived color term representations, we find significant correspondence. Analyzing the differences in alignment across the color spectrum, we find that warmer colors are, on average, better aligned to the perceptual color space than cooler ones, suggesting an intriguing connection to findings from recent work on efficient communication in color naming. Further analysis suggests that differences in alignment are, in part, mediated by collocationality and differences in syntactic usage, posing questions as to the relationship between color perception and usage and context.


page 14

page 15

page 16

page 17

page 18

page 19

page 20

page 21


Improving Perceptual Color Difference using Basic Color Terms

We suggest a new color distance based on two observations. First, percep...

Color naming reflects both perceptual structure and communicative need

Gibson et al. (2017) argued that color naming is shaped by patterns of c...

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

Recent work has raised concerns about the inherent limitations of text-o...

Perceptually Optimized Color Selection for Visualization

We propose an approach, called the Equilibrium Distribution Model (EDM),...

Grounding learning of modifier dynamics: An application to color naming

Grounding is crucial for natural language understanding. An important su...

Pragmatically Informative Color Generation by Grounding Contextual Modifiers

Grounding language in contextual information is crucial for fine-grained...

Semantic Discriminability for Visual Communication

To interpret information visualizations, observers must determine how vi...