Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

12/12/2016
by   Sergey Zagoruyko, et al.
0

Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer

READ FULL TEXT

page 2

page 4

page 5

page 12

research
04/11/2019

Variational Information Distillation for Knowledge Transfer

Transferring knowledge from a teacher neural network pretrained on the s...
research
10/14/2022

Parameter-Free Average Attention Improves Convolutional Neural Network Performance (Almost) Free of Charge

Visual perception is driven by the focus on relevant aspects in the surr...
research
08/30/2022

DLDNN: Deterministic Lateral Displacement Design Automation by Neural Networks

Size-based separation of bioparticles/cells is crucial to a variety of b...
research
01/17/2020

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

Recent studies in image classification have demonstrated a variety of te...
research
05/20/2021

Biologically Inspired Semantic Lateral Connectivity for Convolutional Neural Networks

Lateral connections play an important role for sensory processing in vis...
research
08/23/2019

Assessing Knee OA Severity with CNN attention-based end-to-end architectures

This work proposes a novel end-to-end convolutional neural network (CNN)...
research
03/21/2022

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

Convolutional neural networks (CNNs) have gained significant popularity ...

Please sign up or login with your details

Forgot password? Click here to reset