LieTransformer: Equivariant self-attention for Lie Groups

12/20/2020
by   Michael Hutchinson, et al.
6

Group equivariant neural networks are used as building blocks of group invariant neural networks, which have been shown to improve generalisation performance and data efficiency through principled parameter sharing. Such works have mostly focused on group equivariant convolutions, building on the result that group equivariant linear maps are necessarily convolutions. In this work, we extend the scope of the literature to non-linear neural network modules, namely self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups. We demonstrate the generality of our approach by showing experimental results that are competitive to baseline methods on a wide range of tasks: shape counting on point clouds, molecular property regression and modelling particle trajectories under Hamiltonian dynamics.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/31/2023

A General Framework for Equivariant Neural Networks on Reductive Lie Groups

Reductive Lie Groups, such as the orthogonal groups, the Lorentz group, ...
research
02/25/2020

Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data

The translation equivariance of convolutional layers enables convolution...
research
04/19/2021

A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups

Symmetries and equivariance are fundamental to the generalization of neu...
research
10/02/2020

Group Equivariant Stand-Alone Self-Attention For Vision

We provide a general self-attention formulation to impose group equivari...
research
06/08/2020

The Lipschitz Constant of Self-Attention

Lipschitz constants of neural networks have been explored in various con...
research
12/19/2018

Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers

Sentiment Analysis has seen much progress in the past two decades. For t...
research
10/09/2022

LieGG: Studying Learned Lie Group Generators

Symmetries built into a neural network have appeared to be very benefici...

Please sign up or login with your details

Forgot password? Click here to reset