DeepAI AI Chat
Log In Sign Up

Inverse Problems Leveraging Pre-trained Contrastive Representations

by   Sriram Ravula, et al.
The University of Texas at Austin

We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.


page 2

page 23


Visual Representation Learning from Unlabeled Video using Contrastive Masked Autoencoders

Masked Autoencoders (MAEs) learn self-supervised representations by rand...

InverseNet: Solving Inverse Problems with Splitting Networks

We propose a new method that uses deep learning techniques to solve the ...

Improving Limited Angle CT Reconstruction with a Robust GAN Prior

Limited angle CT reconstruction is an under-determined linear inverse pr...

Solving Inverse Problems for Spectral Energy Distributions with Deep Generative Networks

We propose an end-to-end approach for solving inverse problems for a cla...

DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency

Diffusion models have established new state of the art in a multitude of...

Function Contrastive Learning of Transferable Representations

Few-shot-learning seeks to find models that are capable of fast-adaptati...

Equivariant Imaging: Learning Beyond the Range Space

In various imaging problems, we only have access to compressed measureme...