Measuring Mathematical Problem Solving With the MATH Dataset

by   Dan Hendrycks, et al.

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.




Online learning from the convenience of one's own home grew popular throughout the outbreak. Because my child was having difficulty with arithmetic, we decided to try - online math . Surprisingly, I enjoy it in every way. They explain everything in an easy-to-understand way, allowing students to understand and enhance their skills from the convenience of their own homes. It's something I strongly advise!


page 3


Adventures in Mathematical Reasoning

"Mathematics is not a careful march down a well-cleared highway, but a j...

NaturalProofs: Mathematical Theorem Proving in Natural Language

Understanding and creating mathematics using natural mathematical langua...

Pretrained Language Models are Symbolic Mathematics Solvers too!

Solving symbolic mathematics has always been of in the arena of human in...

A stochastic model for Case-Based Reasoning

Case-Bsed Reasoning (CBR) is a recent theory for problem-solving and lea...

Automated Conjecturing VII: The Graph Brain Project & Big Mathematics

The Graph Brain Project is an experiment in how the use of automated mat...

Problem Solving and Computational Thinking in a Learning Environment

Computational thinking is a new problem soling method named for its exte...

Measuring Massive Multitask Language Understanding

We propose a new test to measure a text model's multitask accuracy. The ...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.