DeepAI
Log In Sign Up

Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound

08/14/2020
by   Jianbo Jiao, et al.
24

In medical imaging, manual annotations can be expensive to acquire and sometimes infeasible to access, making conventional deep learning-based models difficult to scale. As a result, it would be beneficial if useful representations could be derived from raw data without the need for manual annotations. In this paper, we propose to address the problem of self-supervised representation learning with multi-modal ultrasound video-speech raw data. For this case, we assume that there is a high correlation between the ultrasound video and the corresponding narrative speech audio of the sonographer. In order to learn meaningful representations, the model needs to identify such correlation and at the same time understand the underlying anatomical features. We designed a framework to model the correspondence between video and audio without any kind of human annotations. Within this framework, we introduce cross-modal contrastive learning and an affinity-aware self-paced learning scheme to enhance correlation modelling. Experimental evaluations on multi-modal fetal ultrasound video and audio show that the proposed approach is able to learn strong representations and transfers well to downstream tasks of standard plane detection and eye-gaze prediction.

READ FULL TEXT

page 3

page 4

page 7

page 8

02/28/2020

Self-supervised Representation Learning for Ultrasound Video

Recent advances in deep learning have achieved promising performance for...
08/22/2022

Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound

Self-supervised contrastive representation learning offers the advantage...
03/31/2022

Video-Text Representation Learning via Differentiable Weak Temporal Alignment

Learning generic joint representations for video and text by a supervise...
07/18/2022

Contrastive Environmental Sound Representation Learning

Machine hearing of the environmental sound is one of the important issue...
07/21/2021

MG-NET: Leveraging Pseudo-Imaging for Multi-Modal Metagenome Analysis

The emergence of novel pathogens and zoonotic diseases like the SARS-CoV...
03/07/2019

Ultrasound Image Representation Learning by Modeling Sonographer Visual Attention

Image representations are commonly learned from class labels, which are ...
07/01/2019

Synchronising audio and ultrasound by learning cross-modal embeddings

Audiovisual synchronisation is the task of determining the time offset b...