Log In Sign Up

Image-to-Image MLP-mixer for Image Reconstruction

by   Youssef Mansour, et al.

Neural networks are highly effective tools for image reconstruction problems such as denoising and compressive sensing. To date, neural networks for image reconstruction are almost exclusively convolutional. The most popular architecture is the U-Net, a convolutional network with a multi-resolution architecture. In this work, we show that a simple network based on the multi-layer perceptron (MLP)-mixer enables state-of-the art image reconstruction performance without convolutions and without a multi-resolution architecture, provided that the training set and the size of the network are moderately large. Similar to the original MLP-mixer, the image-to-image MLP-mixer is based exclusively on MLPs operating on linearly-transformed image patches. Contrary to the original MLP-mixer, we incorporate structure by retaining the relative positions of the image patches. This imposes an inductive bias towards natural images which enables the image-to-image MLP-mixer to learn to denoise images based on fewer examples than the original MLP-mixer. Moreover, the image-to-image MLP-mixer requires fewer parameters to achieve the same denoising performance than the U-Net and its parameters scale linearly in the image resolution instead of quadratically as for the original MLP-mixer. If trained on a moderate amount of examples for denoising, the image-to-image MLP-mixer outperforms the U-Net by a slight margin. It also outperforms the vision transformer tailored for image reconstruction and classical un-trained methods such as BM3D, making it a very effective tool for image reconstruction problems.


page 8

page 13


Can Un-trained Neural Networks Compete with Trained Neural Networks at Image Reconstruction?

Convolutional Neural Networks (CNNs) are highly effective for image reco...

Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks

Deep neural networks, in particular convolutional neural networks, have ...

dAUTOMAP: decomposing AUTOMAP to achieve scalability and enhance performance

AUTOMAP is a promising generalized reconstruction approach, however, it ...

Manifold Modeling in Quotient Space: Learning An Invariant Mapping with Decodability of Image Patches

This study proposes a framework for manifold learning of image patches u...

Denoising-based image reconstruction from pixels located at non-integer positions

Digital images are commonly represented as regular 2D arrays, so pixels ...

A Neural-Network-Based Convex Regularizer for Image Reconstruction

The emergence of deep-learning-based methods for solving inverse problem...

MIST: Multiple Instance Spatial Transformer Network

We propose a deep network that can be trained to tackle image reconstruc...