Iterative Patch Selection for High-Resolution Image Recognition

10/24/2022
by   Benjamin Bergner, et al.
22

High-resolution images are prevalent in various applications, such as autonomous driving and computer-aided diagnosis. However, training neural networks on such images is computationally challenging and easily leads to out-of-memory errors even on modern GPUs. We propose a simple method, Iterative Patch Selection (IPS), which decouples the memory usage from the input size and thus enables the processing of arbitrarily large images under tight hardware constraints. IPS achieves this by selecting only the most salient patches, which are then aggregated into a global representation for image recognition. For both patch selection and aggregation, a cross-attention based transformer is introduced, which exhibits a close connection to Multiple Instance Learning. Our method demonstrates strong performance and has wide applicability across different domains, training regimes and image sizes while using minimal accelerator memory. For example, we are able to finetune our model on whole-slide images consisting of up to 250k patches (>16 gigapixels) with only 5 GB of GPU VRAM at a batch size of 16.

READ FULL TEXT

page 9

page 18

page 19

page 20

research
04/07/2021

Differentiable Patch Selection for Image Recognition

Neural Networks require large amounts of memory and compute to process h...
research
07/02/2019

Multi-scale GANs for Memory-efficient Generation of High Resolution Medical Images

Currently generative adversarial networks (GANs) are rarely applied to m...
research
10/06/2020

Memory-efficient GAN-based Domain Translation of High Resolution 3D Medical Images

Generative adversarial networks (GANs) are currently rarely applied on 3...
research
07/29/2020

Foveation for Segmentation of Ultra-High Resolution Images

Segmentation of ultra-high resolution images is challenging because of t...
research
10/27/2017

Enhanced Biologically Inspired Model for Image Recognition Based on a Novel Patch Selection Method with Moment

Biologically inspired model (BIM) for image recognition is a robust comp...
research
02/07/2022

Patch-Based Stochastic Attention for Image Editing

Attention mechanisms have become of crucial importance in deep learning ...
research
03/08/2022

End-to-end Multiple Instance Learning with Gradient Accumulation

Being able to learn on weakly labeled data, and provide interpretability...

Please sign up or login with your details

Forgot password? Click here to reset