Contrastive Learning of Medical Visual Representations from Paired Images and Text

10/02/2020 ∙ by Yuhao Zhang, et al. ∙ 39

Learning visual representations of medical images is core to medical image understanding but its progress has been held back by the small size of hand-labeled datasets. Existing work commonly relies on transferring weights from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. We propose an alternative unsupervised strategy to learn medical visual representations directly from the naturally occurring pairing of images and textual data. Our method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test our method by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that our method leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10 labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.



There are no comments yet.


page 1

page 2

page 3

page 4

Code Repositories


CLASP - Contrastive Language-Aminoacid Sequence Pretraining

view repo


Contrastive Learning Representations for Images and Text Pairs. Pytorch implementation of ConVIRT Paper.

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.