AI Chat AI Image Generator AI Video Text to Speech

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

05/28/2020

∙

by Oskar Wysocki, et al.

∙

∙

SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

Oskar Wysocki
8 publications
Malina Florea
2 publications
André Freitas
80 publications

page 1

page 2

page 3

page 4

research

∙ 07/16/2021

Architectures of Meaning, A Systematic Corpus Analysis of NLP Systems

This paper proposes a novel statistical corpus analysis framework target...

12 Oskar Wysocki, et al. ∙

research

∙ 08/16/2019

An Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing

Multi-Task Learning (MTL) aims at boosting the overall performance of ea...

0 Jianquan Li, et al. ∙

research

∙ 09/25/2022

Corpus-based Metaphor Analysis through Graph Theoretical Methods

As a contribution to metaphor analysis, we introduce a statistical, data...

0 Marie Teich, et al. ∙

research

∙ 07/25/2018

Evaluating Creativity in Computational Co-Creative Systems

This paper provides a framework for evaluating creativity in co-creative...

0 Pegah Karimi, et al. ∙

research

∙ 10/06/2020

A Survey on Recognizing Textual Entailment as an NLP Evaluation

Recognizing Textual Entailment (RTE) was proposed as a unified evaluatio...

0 Adam Poliak, et al. ∙

research

∙ 12/20/2022

Evaluation for Change

Evaluation is the central means for assessing, understanding, and commun...

0 Rishi Bommasani, et al. ∙

research

∙ 03/31/2022

On the Evaluation of NLP-based Models for Software Engineering

NLP-based models have been increasingly incorporated to address SE probl...

0 Maliheh Izadi, et al. ∙