UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes

by   Alexander Kolesnikov, et al.

We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks; it requires no task-specific modifications which require extensive human expertise. The approach involves two components: (I) a base model (feed-forward) which is trained to directly predict raw vision outputs, guided by a learned discrete code and (II) a language model (autoregressive) that is trained to generate the guiding code. These components complement each other: the language model is well-suited to modeling structured interdependent data, while the base model is efficient at dealing with high-dimensional outputs. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks: panoptic segmentation, depth prediction and image colorization, where we achieve competitive and near state-of-the-art results. Our experimental results suggest that UViM is a promising candidate for a unified modeling approach in computer vision.


page 1

page 2

page 3

page 4

page 9

page 10

page 11

page 12


A Unified Sequence Interface for Vision Tasks

While language tasks are naturally expressed in a single, unified, model...

Tuning computer vision models with task rewards

Misalignment between model predictions and intended usage can be detrime...

Transframer: Arbitrary Frame Prediction with Generative Models

We present a general-purpose framework for image modelling and vision ta...

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

We propose Unified-IO, a model that performs a large variety of AI tasks...

InstructDiffusion: A Generalist Modeling Interface for Vision Tasks

We present InstructDiffusion, a unifying and generic framework for align...

Computer Vision with a Single (Robust) Classifier

We show that the basic classification framework alone can be used to tac...

Learning to Stop While Learning to Predict

There is a recent surge of interest in designing deep architectures base...

Please sign up or login with your details

Forgot password? Click here to reset