DeepAI AI Chat
Log In Sign Up

TMR: Evaluating NER Recall on Tough Mentions

by   Jingxuan Tu, et al.

We propose the Tough Mentions Recall (TMR) metrics to supplement traditional named entity recognition (NER) evaluation by examining recall on specific subsets of "tough" mentions: unseen mentions, those whose tokens or token/type combination were not observed in training, and type-confusable mentions, token sequences with multiple entity types in the test data. We demonstrate the usefulness of these metrics by evaluating corpora of English, Spanish, and Dutch using five recent neural architectures. We identify subtle differences between the performance of BERT and Flair on two English NER corpora and identify a weak spot in the performance of current models in Spanish. We conclude that the TMR metrics enable differentiation between otherwise similar-scoring systems and identification of patterns in performance that would go unnoticed from overall precision, recall, and F1.


page 1

page 2

page 3

page 4


Dutch Named Entity Recognition and De-identification Methods for the Human Resource Domain

The human resource (HR) domain contains various types of privacy-sensiti...

On the Strength of Character Language Models for Multilingual Named Entity Recognition

Character-level patterns have been widely used as features in English Na...

MANER: Mask Augmented Named Entity Recognition for Extreme Low-Resource Languages

This paper investigates the problem of Named Entity Recognition (NER) fo...

Generalisation in Named Entity Recognition: A Quantitative Analysis

Named Entity Recognition (NER) is a key NLP task, which is all the more ...

Entity Recognition at First Sight: Improving NER with Eye Movement Information

Previous research shows that eye-tracking data contains information abou...

NER-MQMRC: Formulating Named Entity Recognition as Multi Question Machine Reading Comprehension

NER has been traditionally formulated as a sequence labeling task. Howev...

Remedying BiLSTM-CNN Deficiency in Modeling Cross-Context for NER

Recent researches prevalently used BiLSTM-CNN as a core module for NER i...