PSB2: The Second Program Synthesis Benchmark Suite

by   Thomas Helmuth, et al.

For the past six years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite to benchmark many aspects of automatic program synthesis systems. These problems have been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in the original benchmark suite have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible. In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. These new problems give plenty of room for improvement, pointing the way for the next six or more years of general program synthesis research.


page 1

page 2

page 3

page 4


Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming

GitHub Copilot, an extension for the Visual Studio Code development envi...

Recent Developments in Program Synthesis with Evolutionary Algorithms

The automatic generation of computer programs is one of the main applica...

Recent Advances in Neural Program Synthesis

In recent years, deep learning has made tremendous progress in a number ...

Inductive Program Synthesis Over Noisy Data

We present a new framework and associated synthesis algorithms for progr...

NAPS: Natural Program Synthesis Dataset

We present a program synthesis-oriented dataset consisting of human writ...

Knowledge-Driven Program Synthesis via Adaptive Replacement Mutation and Auto-constructed Subprogram Archives

We introduce Knowledge-Driven Program Synthesis (KDPS) as a variant of t...

Breakdown of a Benchmark Score Without Internal Analysis of Benchmarking Program

A breakdown of a benchmark score is how much each aspect of the system p...

Please sign up or login with your details

Forgot password? Click here to reset