Generalization algorithm of multimodal pre-training model based on graph-text self-supervised training

02/16/2023
by   Zhangxiaobing, et al.
0

Recently, a large number of studies have shown that the introduction of visual information can effectively improve the effect of neural machine translation (NMT). Its effectiveness largely depends on the availability of a large number of bilingual parallel sentence pairs and manual image annotation. The lack of images and the effectiveness of images have been difficult to solve. In this paper, a multimodal pre-training generalization algorithm for self-supervised training is proposed, which overcomes the lack of visual information and inaccuracy, and thus extends the applicability of images on NMT. Specifically, we will search for many pictures from the existing sentences through the search engine, and then through the relationship between visual information and text, do the self-supervised training task of graphics and text to obtain more effective visual information for text. We show that when the filtered information is used as multimodal machine translation for fine-tuning, the effect of translation in the global voice dataset is 0.5 BLEU higher than the baseline.

READ FULL TEXT
research
06/08/2021

Self-supervised and Supervised Joint Training for Resource-rich Machine Translation

Self-supervised pre-training of text representations has been successful...
research
07/26/2022

Multimodal Neural Machine Translation with Search Engine Based Image Retrieval

Recently, numbers of works shows that the performance of neural machine ...
research
04/11/2022

Unified Speech-Text Pre-training for Speech Translation and Recognition

We describe a method to jointly pre-train speech and text in an encoder-...
research
09/08/2021

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Multimodal machine translation (MMT) systems have been shown to outperfo...
research
09/21/2020

Generative Imagination Elevates Machine Translation

There are thousands of languages on earth, but visual perception is shar...
research
05/20/2023

Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

In this work, we investigate a more realistic unsupervised multimodal ma...
research
09/17/2020

Code-switching pre-training for neural machine translation

This paper proposes a new pre-training method, called Code-Switching Pre...

Please sign up or login with your details

Forgot password? Click here to reset