WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

04/30/2020

∙

by   Anna Breit, et al.

∙

Iran University of Science and Technology

∙

Cardiff University

∙

0

∙

share

In this paper, we present WiC-TSV (Target Sense Verification for Words in Context), a new multi-domain evaluation benchmark for Word Sense Disambiguation (WSD) and Entity Linking (EL). Our benchmark is different from conventional WSD and EL benchmarks for it being independent of a general sense inventory, making it highly flexible for the evaluation of a diverse set of models and systems in different domains. WiC-TSV is split into three tasks (systems get hypernymy or definitional or both hypernymy and definitional information about the target sense). Test data is available in four domains: general (WordNet), computer science, cocktails and medical concepts. Results show that existing state-of-the-art language models such as BERT can achieve a high performance in both in-domain data and out-of-domain data, but they still have room for improvement. WiC-TSV task data is available at <https://competitions.codalab.org/competitions/23683>.