Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics

by   David Schlangen, et al.

Propelling, and propelled by, the "deep learning revolution", recent years have seen the introduction of ever larger corpora of images annotated with natural language expressions. We survey some of these corpora, taking a perspective that reverses the usual directionality, as it were, by viewing the images as semantic annotation of the natural language expressions. We discuss datasets that can be derived from the corpora, and tasks of potential interest for computational semanticists that can be defined on those. In this, we make use of relations provided by the corpora (namely, the link between expression and image, and that between two expressions linked to the same image) and relations that we can add (similarity relations between expressions, or between images). Specifically, we show that in this way we can create data that can be used to learn and evaluate lexical and compositional grounded semantics, and we show that the "linked to same image" relation tracks a semantic implication relation that is recognisable to annotators even in the absence of the linking image as evidence. Finally, as an example of possible benefits of this approach, we show that an exemplar-model-based approach to implication beats a (simple) distributional space-based one on some derived datasets, while lending itself to explainability.


page 3

page 4

page 5

page 6

page 7


Natural Language Semantics and Computability

This paper is a reflexion on the computability of natural language seman...

Semantic Relation Classification: Task Formalisation and Refinement

The identification of semantic relations between terms within texts is a...

A New Semantic Theory of Natural Language

Formal Semantics and Distributional Semantics are two important semantic...

SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference

We present SherLIiC, a testbed for lexical inference in context (LIiC), ...

Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

Standard evaluations of deep learning models for semantics using natural...

Casting a Wide Net: Robust Extraction of Potentially Idiomatic Expressions

Idiomatic expressions like `out of the woods' and `up the ante' present ...

Grounded Semantic Composition for Visual Scenes

We present a visually-grounded language understanding model based on a s...

Please sign up or login with your details

Forgot password? Click here to reset