Linear algebra with transformers

12/03/2021
by   François Charton, et al.
0

Most applications of transformers to mathematics, from integration to theorem proving, focus on symbolic computation. In this paper, we show that transformers can be trained to perform numerical calculations with high accuracy. We consider problems of linear algebra: matrix transposition, addition, multiplication, eigenvalues and vectors, singular value decomposition, and inversion. Training small transformers (up to six layers) over datasets of random matrices, we achieve high accuracies (over 90 problems. We also show that trained models can generalize out of their training distribution, and that out-of-domain accuracy can be greatly improved by working from more diverse datasets (in particular, by training from matrices with non-independent and identically distributed coefficients). Finally, we show that few-shot learning can be leveraged to re-train models to solve larger problems.

READ FULL TEXT
research
10/31/2022

What is my math transformer doing? – Three results on interpretability and generalization

This paper investigates the failure cases and out-of-distribution behavi...
research
12/02/2019

Deep Learning for Symbolic Mathematics

Neural networks have a reputation for being better at solving statistica...
research
04/13/2020

Pretrained Transformers Improve Out-of-Distribution Robustness

Although pretrained Transformers such as BERT achieve high accuracy on i...
research
05/16/2023

Mimetic Initialization of Self-Attention Layers

It is notoriously difficult to train Transformers on small datasets; typ...
research
08/29/2023

Can transformers learn the greatest common divisor?

I investigate the capability of small transformers to compute the greate...
research
06/12/2023

Transformers learn through gradual rank increase

We identify incremental learning dynamics in transformers, where the dif...
research
03/30/2023

Matrix diagonalization and singular value decomposition: Static SageMath and dynamic ChatGPT juxtaposed

We investigated some difficulties that students often face when studying...

Please sign up or login with your details

Forgot password? Click here to reset