Anchoring and Agreement in Syntactic Annotations

05/15/2016
by   Yevgeni Berzak, et al.
0

We present a study on two key characteristics of human syntactic annotations: anchoring and agreement. Anchoring is a well known cognitive bias in human decision making, where judgments are drawn towards pre-existing values. We study the influence of anchoring on a standard approach to creation of syntactic resources where syntactic annotations are obtained via human editing of tagger and parser output. Our experiments demonstrate a clear anchoring effect and reveal unwanted consequences, including overestimation of parsing performance and lower quality of annotations in comparison with human-based annotations. Using sentences from the Penn Treebank WSJ, we also report systematically obtained inter-annotator agreement estimates for English dependency parsing. Our agreement results control for parser bias, and are consequential in that they are on par with state of the art parsing performance for English newswire. We discuss the impact of our findings on strategies for future annotation efforts and parser evaluations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/29/2017

Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold

We present a new, efficient frame-semantic parser that labels semantic a...
research
10/12/2022

Better Smatch = Better Parser? AMR evaluation is not so simple anymore

Recently, astonishing advances have been observed in AMR parsing, as mea...
research
07/03/2018

Simpler but More Accurate Semantic Dependency Parsing

While syntactic dependency annotations concentrate on the surface or fun...
research
04/06/2021

SERRANT: a syntactic classifier for English Grammatical Error Types

SERRANT is a system and code for automatic classification of English gra...
research
05/08/2019

Automatic Inference of Minimalist Grammars using an SMT-Solver

We introduce (1) a novel parser for Minimalist Grammars (MG), encoded as...
research
04/21/2019

A Study on Agreement in PICO Span Annotations

In evidence-based medicine, relevance of medical literature is determine...
research
04/06/2019

Speeding Up Natural Language Parsing by Reusing Partial Results

This paper proposes a novel technique that applies case-based reasoning ...

Please sign up or login with your details

Forgot password? Click here to reset