So3krates – Self-attention for higher-order geometric interactions on arbitrary length-scales

05/28/2022
by   J. Thorben Frank, et al.
0

The application of machine learning methods in quantum chemistry has enabled the study of numerous chemical phenomena, which are computationally intractable with traditional ab-initio methods. However, some quantum mechanical properties of molecules and materials depend on non-local electronic effects, which are often neglected due to the difficulty of modeling them efficiently. This work proposes a modified attention mechanism adapted to the underlying physics, which allows to recover the relevant non-local effects. Namely, we introduce spherical harmonic coordinates (SPHCs) to reflect higher-order geometric information for each atom in a molecule, enabling a non-local formulation of attention in the SPHC space. Our proposed model So3krates – a self-attention based message passing neural network – uncouples geometric information from atomic features, making them independently amenable to attention mechanisms. We show that in contrast to other published methods, So3krates is able to describe non-local quantum mechanical effects over arbitrary length scales. Further, we find evidence that the inclusion of higher-order geometric correlations increases data efficiency and improves generalization. So3krates matches or exceeds state-of-the-art performance on popular benchmarks, notably, requiring a significantly lower number of parameters (0.25–0.4x) while at the same time giving a substantial speedup (6–14x for training and 2–11x for inference) compared to other models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/26/2022

Exploring Self-Attention for Visual Intersection Classification

In robot vision, self-attention has recently emerged as a technique for ...
research
04/21/2023

Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers

We introduce a novel self-attention mechanism, which we call CSA (Chroma...
research
11/24/2022

A Self-Attention Ansatz for Ab-initio Quantum Chemistry

We present a novel neural network architecture using self-attention, the...
research
08/27/2020

Multi-scale approach for the prediction of atomic scale properties

Electronic nearsightedness is one of the fundamental principles governin...
research
08/21/2018

Machine learning non-local correlations

The ability to witness non-local correlations lies at the core of founda...
research
06/18/2022

An extended ordinary state-based peridynamics for non-spherical horizons

This work presents an extended ordinary state-based peridynamics (XOSBPD...
research
08/08/2021

Non-self adjoint impedance in Generalized Optimized Schwarz Methods

We present a convergence theory for Optimized Schwarz Methods that rely ...

Please sign up or login with your details

Forgot password? Click here to reset