SEMA: an Extended Semantic Evaluation Metric for AMR

05/28/2019
by   Rafael T. Anchieta, et al.
0

Abstract Meaning Representation (AMR) is a recently designed semantic representation language intended to capture the meaning of a sentence, which may be represented as a single-rooted directed acyclic graph with labeled nodes and edges. The automatic evaluation of this structure plays an important role in the development of better systems, as well as for semantic annotation. Despite there is one available metric, smatch, it has some drawbacks. For instance, smatch creates a self-relation on the root of the graph, has weights for different error types, and does not take into account the dependence of the elements in the AMR structure. With these drawbacks, smatch masks several problems of the AMR parsers and distorts the evaluation of the AMRs. In view of this, in this paper, we introduce an extended metric to evaluate AMR parsers, which deals with the drawbacks of the smatch metric. Finally, we compare both metrics, using four well-known AMR parsers, and we argue that our metric is more refined, robust, fairer, and faster than smatch.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/16/2022

Persian Abstract Meaning Representation

Abstract Meaning Representation (AMR) is an annotation framework represe...
research
05/24/2022

A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating

Evaluating the quality of generated text is difficult, since traditional...
research
09/29/2020

Weight Prediction for Variants of Weighted Directed Networks

A weighted directed network (WDN) is a directed graph in which each edge...
research
07/25/2022

How much of UCCA can be predicted from AMR?

In this paper, we consider two of the currently popular semantic framewo...
research
07/21/2018

Metric Violation Distance: Revisited and Extended

Metric data plays an important role in various settings such as metric-b...
research
01/29/2020

AMR Similarity Metrics from Principles

Different metrics have been proposed to compare Abstract Meaning Represe...
research
05/11/2023

SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

The Smatch metric is a popular method for evaluating graph distances, as...

Please sign up or login with your details

Forgot password? Click here to reset