Log In Sign Up

MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction

by   Guozhi Tang, et al.

Visual Information Extraction (VIE) task aims to extract key information from multifarious document images (e.g., invoices and purchase receipts). Most previous methods treat the VIE task simply as a sequence labeling problem or classification problem, which requires models to carefully identify each kind of semantics by introducing multimodal features, such as font, color, layout. But simply introducing multimodal features couldn't work well when faced with numeric semantic categories or some ambiguous texts. To address this issue, in this paper we propose a novel key-value matching model based on a graph neural network for VIE (MatchVIE). Through key-value matching based on relevancy evaluation, the proposed MatchVIE can bypass the recognitions to various semantics, and simply focuses on the strong relevancy between entities. Besides, we introduce a simple but effective operation, Num2Vec, to tackle the instability of encoded values, which helps model converge more smoothly. Comprehensive experiments demonstrate that the proposed MatchVIE can significantly outperform previous methods. Notably, to the best of our knowledge, MatchVIE may be the first attempt to tackle the VIE task by modeling the relevancy between keys and values and it is a good complement to the existing methods.


VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations

Document layout analysis is crucial for understanding document structure...

Distribution Matching for Rationalization

The task of rationalization aims to extract pieces of input text as rati...

Revising FUNSD dataset for key-value detection in document images

FUNSD is one of the limited publicly available datasets for information ...

Relation-aware Instance Refinement for Weakly Supervised Visual Grounding

Visual grounding, which aims to build a correspondence between visual ob...

One-shot Key Information Extraction from Document with Deep Partial Graph Matching

Automating the Key Information Extraction (KIE) from documents improves ...

OMG: Observe Multiple Granularities for Natural Language-Based Vehicle Retrieval

Retrieving tracked-vehicles by natural language descriptions plays a cri...

Relational Triple Extraction: One Step is Enough

Extracting relational triples from unstructured text is an essential tas...