Image Compression with Product Quantized Masked Image Modeling

12/14/2022
by   Alaaeldin El-Nouby, et al.
0

Recent neural compression methods have been based on the popular hyperprior framework. It relies on Scalar Quantization and offers a very strong compression performance. This contrasts from recent advances in image generation and representation learning, where Vector Quantization is more commonly employed. In this work, we attempt to bring these lines of research closer by revisiting vector quantization for image compression. We build upon the VQ-VAE framework and introduce several modifications. First, we replace the vanilla vector quantizer by a product quantizer. This intermediate solution between vector and scalar quantization allows for a much wider set of rate-distortion points: It implicitly defines high-quality quantizers that would otherwise require intractably large codebooks. Second, inspired by the success of Masked Image Modeling (MIM) in the context of self-supervised learning and generative image models, we propose a novel conditional entropy model which improves entropy coding by modelling the co-dependencies of the quantized latent codes. The resulting PQ-MIM model is surprisingly effective: its compression performance on par with recent hyperprior methods. It also outperforms HiFiC in terms of FID and KID metrics when optimized with perceptual losses (e.g. adversarial). Finally, since PQ-MIM is compatible with image generation frameworks, we show qualitatively that it can operate under a hybrid mode between compression and generation, with no further training or finetuning. As a result, we explore the extreme compression regime where an image is compressed into 200 bytes, i.e., less than a tweet.

READ FULL TEXT

page 3

page 6

page 9

page 12

page 18

page 19

research
05/25/2023

NVTC: Nonlinear Vector Transform Coding

In theory, vector quantization (VQ) is always better than scalar quantiz...
research
08/27/2022

Lossy Image Compression with Quantized Hierarchical VAEs

Recent work has shown a strong theoretical connection between variationa...
research
07/17/2023

Extreme Image Compression using Fine-tuned VQGAN Models

Recent advances in generative compression methods have demonstrated rema...
research
03/03/2022

Autoregressive Image Generation using Residual Quantization

For autoregressive (AR) modeling of high-resolution images, vector quant...
research
08/04/2023

Frequency Disentangled Features in Neural Image Compression

The design of a neural image compression network is governed by how well...
research
04/07/2021

Learned transform compression with optimized entropy encoding

We consider the problem of learned transform compression where we learn ...
research
01/12/2020

Deep Optimized Multiple Description Image Coding via Scalar Quantization Learning

In this paper, we introduce a deep multiple description coding (MDC) fra...

Please sign up or login with your details

Forgot password? Click here to reset