Evaluating Scoped Meaning Representations

by   Rik van Noord, et al.
University of Groningen

Semantic parsing offers many opportunities to improve natural language understanding. We present a semantically annotated parallel corpus for English, German, Italian, and Dutch where sentences are aligned with scoped meaning representations in order to capture the semantics of negation, modals, quantification, and presupposition triggers. The semantic formalism is based on Discourse Representation Theory, but concepts are represented by WordNet synsets and thematic roles by VerbNet relations. Translating scoped meaning representations to sets of clauses enables us to compare them for the purpose of semantic parser evaluation and checking translations. This is done by computing precision and recall on matching clauses, in a similar way as is done for Abstract Meaning Representations. We show that our matching tool for evaluating scoped meaning representations is both accurate and efficient. Applying this matching tool to three baseline semantic parsers yields F-scores between 43 in meaning by comparing meaning representations of translations. This comparison turns out to be an additional way of (i) finding annotation mistakes and (ii) finding instances where our semantic analysis needs to be improved.


SMARAGD: Synthesized sMatch for Accurate and Rapid AMR Graph Distance

The semantic similarity of graph-based meaning representations, such as ...

From Spatial Relations to Spatial Configurations

Spatial Reasoning from language is essential for natural language unders...

Visual Semantic Parsing: From Images to Abstract Meaning Representation

The success of scene graphs for visual scene understanding has brought a...

Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semantics

Building robust natural language understanding systems will require a cl...

Widely Interpretable Semantic Representation: Frameless Meaning Representation for Broader Applicability

This paper presents a novel semantic representation, WISeR, that overcom...

A Type-coherent, Expressive Representation as an Initial Step to Language Understanding

A growing interest in tasks involving language understanding by the NLP ...

Please sign up or login with your details

Forgot password? Click here to reset