Prompt ChatGPT In MNER: Improved multimodal named entity recognition method based on auxiliary refining knowledge from ChatGPT

05/20/2023
by   Jinyuan Li, et al.
0

Multimodal Named Entity Recognition (MNER) on social media aims to enhance textual entity prediction by incorporating image-based clues. Existing research in this domain has primarily focused on maximizing the utilization of potentially relevant information in images or incorporating external knowledge from explicit knowledge bases (KBs). However, these methods either neglect the necessity of providing the model with relevant external knowledge, or the retrieved external knowledge suffers from high redundancy. To address these problems, we propose a conceptually simple two-stage framework called Prompt ChatGPT In MNER (PGIM) in this paper. We leverage ChatGPT as an implicit knowledge engine to acquire auxiliary refined knowledge, thereby bolstering the model's performance in MNER tasks. Specifically, we first utilize a Multimodal Similar Example Awareness module to select suitable examples from a small number of manually annotated samples. These examples are then integrated into a formatted prompt template tailored to the MNER task, guiding ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the raw text and inputted into the downstream model for further processing. Extensive experiments show that our PGIM significantly outperforms all existing state-of-the-art methods on two classic MNER datasets.

READ FULL TEXT

page 3

page 13

research
04/02/2019

Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

With massive explosion of social media such as Twitter and Instagram, pe...
research
02/22/2018

Multimodal Named Entity Recognition for Short Social Media Posts

We introduce a new task called Multimodal Named Entity Recognition (MNER...
research
10/23/2022

Improving Chinese Named Entity Recognition by Search Engine Augmentation

Compared with English, Chinese suffers from more grammatical ambiguities...
research
06/06/2023

TKDP: Threefold Knowledge-enriched Deep Prompt Tuning for Few-shot Named Entity Recognition

Few-shot named entity recognition (NER) exploits limited annotated insta...
research
05/15/2023

A Novel Framework for Multimodal Named Entity Recognition with Multi-level Alignments

Mining structured knowledge from tweets using named entity recognition (...
research
06/29/2023

Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion

Multimodal Named Entity Recognition (MNER) is a crucial task for informa...
research
08/03/2023

Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER

The challenge posed by multimodal named entity recognition (MNER) is mai...

Please sign up or login with your details

Forgot password? Click here to reset