Group Equivariant Stand-Alone Self-Attention For Vision

10/02/2020
by   David W. Romero, et al.
0

We provide a general self-attention formulation to impose group equivariance to arbitrary symmetry groups. This is achieved by defining positional encodings that are invariant to the action of the group considered. Since the group acts on the positional encoding directly, group equivariant self-attention networks (GSA-Nets) are steerable by nature. Our experiments on vision benchmarks demonstrate consistent improvements of GSA-Nets over non-equivariant self-attention networks.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

11/08/2021

E(2) Equivariant Self-Attention for Radio Astronomy

In this work we introduce group-equivariant self-attention models to add...
12/20/2020

LieTransformer: Equivariant self-attention for Lie Groups

Group equivariant neural networks are used as building blocks of group i...
11/18/2019

Affine Self Convolution

Attention mechanisms, and most prominently self-attention, are a powerfu...
12/07/2021

Hybrid Self-Attention NEAT: A novel evolutionary approach to improve the NEAT algorithm

This article presents a "Hybrid Self-Attention NEAT" method to improve t...
03/29/2022

Domain Invariant Siamese Attention Mask for Small Object Change Detection via Everyday Indoor Robot Navigation

The problem of image change detection via everyday indoor robot navigati...
04/23/2022

Visual Attention Emerges from Recurrent Sparse Reconstruction

Visual attention helps achieve robust perception under noise, corruption...
06/08/2020

The Lipschitz Constant of Self-Attention

Lipschitz constants of neural networks have been explored in various con...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.