Extensive Error Analysis and a Learning-Based Evaluation of Medical Entity Recognition Systems to Approximate User Experience

06/09/2020
by   Isar Nejadgholi, et al.
0

When comparing entities extracted by a medical entity recognition system with gold standard annotations over a test set, two types of mismatches might occur, label mismatch or span mismatch. Here we focus on span mismatch and show that its severity can vary from a serious error to a fully acceptable entity extraction due to the subjectivity of span annotations. For a domain-specific BERT-based NER system, we showed that 25 and overlapping span with gold standard entities. We collected expert judgement which shows more than 90 accepted by the user. Using the training set of the NER system, we built a fast and lightweight entity classifier to approximate the user experience of such mismatches through accepting or rejecting them. The decisions made by this classifier are used to calculate a learning-based F-score which is shown to be a better approximation of a forgiving user's experience than the relaxed F-score. We demonstrated the results of applying the proposed evaluation metric for a variety of deep learning medical entity recognition models trained with two datasets.

READ FULL TEXT

page 3

page 5

research
10/09/2022

Deep Span Representations for Named Entity Recognition

Span-based models are one of the most straightforward methods for named ...
research
09/04/2022

SCL-RAI: Span-based Contrastive Learning with Retrieval Augmented Inference for Unlabeled Entity Problem in NER

Named Entity Recognition is the task to locate and classify the entities...
research
04/21/2019

A Study on Agreement in PICO Span Annotations

In evidence-based medicine, relevance of medical literature is determine...
research
10/17/2022

SpanProto: A Two-stage Span-based Prototypical Network for Few-shot Named Entity Recognition

Few-shot Named Entity Recognition (NER) aims to identify named entities ...
research
08/09/2022

An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition

Named entity recognition (NER) is the task to detect and classify the en...
research
04/11/2023

An Entity-based Claim Extraction Pipeline for Real-world Biomedical Fact-checking

Existing fact-checking models for biomedical claims are typically traine...
research
04/06/2020

Building a Norwegian Lexical Resource for Medical Entity Recognition

We present a large Norwegian lexical resource of categorized medical ter...

Please sign up or login with your details

Forgot password? Click here to reset