Fourier Image Transformer

04/06/2021
by   Tim-Oliver Buchholz, et al.
0

Transformer architectures show spectacular performance on NLP tasks and have recently also been used for tasks such as image completion or image classification. Here we propose to use a sequential image representation, where each prefix of the complete sequence describes the whole image at reduced resolution. Using such Fourier Domain Encodings (FDEs), an auto-regressive image completion task is equivalent to predicting a higher resolution output given a low-resolution input. Additionally, we show that an encoder-decoder setup can be used to query arbitrary Fourier coefficients given a set of Fourier domain observations. We demonstrate the practicality of this approach in the context of computed tomography (CT) image reconstruction. In summary, we show that Fourier Image Transformer (FIT) can be used to solve relevant image analysis tasks in Fourier space, a domain inherently inaccessible to convolutional architectures.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 7

research
02/03/2015

Recovery of Piecewise Smooth Images from Few Fourier Samples

We introduce a Prony-like method to recover a continuous domain 2-D piec...
research
11/29/2022

Fourier-Net: Fast Image Registration with Band-limited Deformation

Unsupervised image registration commonly adopts U-Net style networks to ...
research
05/31/2023

Bayesian Image Analysis in Fourier Space

Bayesian image analysis has played a large role over the last 40+ years ...
research
10/28/2022

Contextual Learning in Fourier Complex Field for VHR Remote Sensing Images

Very high-resolution (VHR) remote sensing (RS) image classification is t...
research
07/06/2023

Fourier-Net+: Leveraging Band-Limited Representation for Efficient 3D Medical Image Registration

U-Net style networks are commonly utilized in unsupervised image registr...
research
12/13/2018

Fourier RNNs for Sequence Analysis and Prediction

Fourier methods have a long and proven track record in as an excellent t...
research
09/11/2018

Fourier-Domain Optimization for Image Processing

Image optimization problems encompass many applications such as spectral...

Please sign up or login with your details

Forgot password? Click here to reset