Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

05/07/2022
by   Xiang Chen, et al.
0

Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. We further propose a dynamic gated aggregation strategy to achieve hierarchical multi-scaled visual features as visual prefix for fusion. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance. Code is available in https://github.com/zjunlp/HVPNeT.

READ FULL TEXT

page 1

page 8

research
08/24/2023

CARE: Co-Attention Network for Joint Entity and Relation Extraction

Joint entity and relation extraction is the fundamental task of informat...
research
11/14/2022

On Analyzing the Role of Image for Visual-enhanced Relation Extraction

Multimodal relation extraction is an essential task for knowledge graph ...
research
06/25/2023

Chain-of-Thought Prompt Distillation for Multimodal Named Entity and Multimodal Relation Extraction

Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extra...
research
04/05/2023

Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck

This paper studies the multimodal named entity recognition (MNER) and mu...
research
05/19/2023

Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling

Existing research on multimodal relation extraction (MRE) faces two co-e...
research
09/16/2020

Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation Extraction

Joint entity and relation extraction aims to extract relation triplets f...
research
10/05/2020

Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Neural models have achieved remarkable success on relation extraction (R...

Please sign up or login with your details

Forgot password? Click here to reset