A Caption Is Worth A Thousand Images: Investigating Image Captions for Multimodal Named Entity Recognition

10/23/2020
by   Shuguang Chen, et al.
9

Multimodal named entity recognition (MNER) requires to bridge the gap between language understanding and visual context. Due to advances in natural language processing (NLP) and computer vision (CV), many neural techniques have been proposed to incorporate images into the NER task. In this work, we conduct a detailed analysis of current state-of-the-art fusion techniques for MNER and describe scenarios where adding information from the image does not always result in boosts in performance. We also study the use of captions as a way to enrich the context for MNER. We provide extensive empirical analysis and an ablation study on three datasets from popular social platforms to expose the situations where the approach is beneficial.

READ FULL TEXT
research
02/16/2023

Dynamic Named Entity Recognition

Named Entity Recognition (NER) is a challenging and widely studied task ...
research
08/09/2022

Effects of Annotations' Density on Named Entity Recognition Models' Performance in the Context of African Languages

African languages have recently been the subject of several studies in N...
research
03/22/2021

MasakhaNER: Named Entity Recognition for African Languages

We take a step towards addressing the under-representation of the Africa...
research
06/29/2023

Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion

Multimodal Named Entity Recognition (MNER) is a crucial task for informa...
research
02/05/2021

RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER

Recently multimodal named entity recognition (MNER) has utilized images ...
research
04/20/2020

The Panacea Threat Intelligence and Active Defense Platform

We describe Panacea, a system that supports natural language processing ...
research
02/27/2021

Automated Generation of Interorganizational Disaster Response Networks through Information Extraction

When a disaster occurs, maintaining and restoring community lifelines su...

Please sign up or login with your details

Forgot password? Click here to reset