Log In Sign Up

Analyzing the Nuances of Transformers' Polynomial Simplification Abilities

by   Vishesh Agarwal, et al.

Symbolic Mathematical tasks such as integration often require multiple well-defined steps and understanding of sub-tasks to reach a solution. To understand Transformers' abilities in such tasks in a fine-grained manner, we deviate from traditional end-to-end settings, and explore a step-wise polynomial simplification task. Polynomials can be written in a simple normal form as a sum of monomials which are ordered in a lexicographic order. For a polynomial which is not necessarily in this normal form, a sequence of simplification steps is applied to reach the fully simplified (i.e., in the normal form) polynomial. We propose a synthetic Polynomial dataset generation algorithm that generates polynomials with unique proof steps. Through varying coefficient configurations, input representation, proof granularity, and extensive hyper-parameter tuning, we observe that Transformers consistently struggle with numeric multiplication. We explore two ways to mitigate this: Curriculum Learning and a Symbolic Calculator approach (where the numeric operations are offloaded to a calculator). Both approaches provide significant gains over the vanilla Transformers-based baseline.


page 1

page 2

page 3

page 4


Normal and Triangular Determinantal Representations of Multivariate Polynomials

In this paper we give a new and simple algorithm to put any multivariate...

Improved Polynomial Remainder Sequences for Ore Polynomials

Polynomial remainder sequences contain the intermediate results of the E...

On the Computational Power of Transformers and Its Implications in Sequence Modeling

Transformers are being used extensively across several sequence modeling...

A fast, deterministic algorithm for computing a Hermite Normal Form of a polynomial matrix

Given a square, nonsingular matrix of univariate polynomials F∈K[x]^n × ...

Generating Intermediate Steps for NLI with Next-Step Supervision

The Natural Language Inference (NLI) task often requires reasoning over ...

Do Transformers use variable binding?

Increasing the explainability of deep neural networks (DNNs) requires ev...