The Fragility of Multi-Treebank Parsing Evaluation

09/14/2022
by   Iago Alonso-Alonso, et al.
0

Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2023

Model averaging approaches to data subset selection

Model averaging is a useful and robust method for dealing with model unc...
research
10/06/2020

On the Role of Supervision in Unsupervised Constituency Parsing

We analyze several recent unsupervised constituency parsing models, whic...
research
05/18/2019

On greedy heuristics for computing D-efficient saturated subsets

Let F be a set consisting of n real vectors of dimension m ≤ n. For any ...
research
05/17/2020

Robust subset selection

The best subset selection (or "best subsets") estimator is a classic too...
research
12/15/2021

Bayesian Mendelian randomization with study heterogeneity and data partitioning for large studies

Background: Mendelian randomization (MR) is a useful approach to causal ...
research
10/09/2019

On Post-Selection Inference in A/B Tests

When a large number of simultaneous statistical inferences are conducted...
research
09/11/2019

Generalized Optimal Two-way Relays Subsets Pairings in Cloud-based Region Cognitive Networks

Communication reliability improving is one of most important research re...

Please sign up or login with your details

Forgot password? Click here to reset