Weakly supervised cross-domain alignment with optimal transport

08/14/2020
by   Siyang Yuan, et al.
9

Cross-domain alignment between image objects and text sequences is key to many visual-language tasks, and it poses a fundamental challenge to both computer vision and natural language processing. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities, under a weakly-supervised setup, improving performance over state-of-the-art solutions. Our method builds upon recent advances in optimal transport (OT) to resolve the cross-domain matching problem in a principled manner. Formulated as a drop-in regularizer, the proposed OT solution can be efficiently computed and used in combination with other existing approaches. We present empirical evidence to demonstrate the effectiveness of our approach, showing how it enables simpler model architectures to outperform or be comparable with more sophisticated designs on a range of vision-language tasks.

READ FULL TEXT

page 7

page 8

research
06/26/2020

Graph Optimal Transport for Cross-Domain Alignment

Cross-domain alignment between two sets of entities (e.g., objects in an...
research
03/31/2022

Partial Coupling of Optimal Transport for Spoken Language Identification

In order to reduce domain discrepancy to improve the performance of cros...
research
10/10/2022

Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment

Multimedia summarization with multimodal output (MSMO) is a recently exp...
research
09/28/2022

TokenFlow: Rethinking Fine-grained Cross-modal Alignment in Vision-Language Retrieval

Most existing methods in vision-language retrieval match two modalities ...
research
11/09/2021

Graph Matching via Optimal Transport

The graph matching problem seeks to find an alignment between the nodes ...
research
01/28/2020

Structural-Aware Sentence Similarity with Recursive Optimal Transport

Measuring sentence similarity is a classic topic in natural language pro...
research
11/06/2019

Unsupervised Hierarchy Matching with Optimal Transport over Hyperbolic Spaces

This paper focuses on the problem of unsupervised alignment of hierarchi...

Please sign up or login with your details

Forgot password? Click here to reset