Endowing Language Models with Multimodal Knowledge Graph Representations

by   Ningyuan Huang, et al.

We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG using a dense index. Given (possibly multilingual) downstream task data, e.g., sentences in German, we retrieve entities from the KG and use their multimodal representations to improve downstream task performance. We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn entity and relation representations that are grounded on the KG multimodal information. We demonstrate the usefulness of the learned entity representations on two downstream tasks, and show improved performance on the multilingual named entity recognition task by 0.3%–0.7% F1, while we achieve up to 2.5% improvement in accuracy on the visual sense disambiguation task. All our code and data are available in: <https://github.com/iacercalixto/visualsem-kg>.


page 1

page 2

page 3

page 4


mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Recent studies have shown that multilingual pretrained language models c...

PAI at SemEval-2023 Task 2: A Universal System for Named Entity Recognition with External Entity Information

The MultiCoNER II task aims to detect complex, ambiguous, and fine-grain...

BEKG: A Built Environment Knowledge Graph

Practices in the built environment have become more digitalized with the...

ParaNames: A Massively Multilingual Entity Name Corpus

This preprint describes work in progress on ParaNames, a multilingual pa...

Math-KG: Construction and Applications of Mathematical Knowledge Graph

Recently, the explosion of online education platforms makes a success in...

Are Multilingual Models Effective in Code-Switching?

Multilingual language models have shown decent performance in multilingu...

Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks

We present Bloom Library, a linguistically diverse set of multimodal and...

Please sign up or login with your details

Forgot password? Click here to reset