PSB2: The Second Program Synthesis Benchmark Suite

by   Thomas Helmuth, et al.

For the past six years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite to benchmark many aspects of automatic program synthesis systems. These problems have been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in the original benchmark suite have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible. In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. These new problems give plenty of room for improvement, pointing the way for the next six or more years of general program synthesis research.



page 1

page 2

page 3

page 4


Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming

GitHub Copilot, an extension for the Visual Studio Code development envi...

Recent Advances in Neural Program Synthesis

In recent years, deep learning has made tremendous progress in a number ...

Recent Developments in Program Synthesis with Evolutionary Algorithms

The automatic generation of computer programs is one of the main applica...

Breakdown of a Benchmark Score Without Internal Analysis of Benchmarking Program

A breakdown of a benchmark score is how much each aspect of the system p...

Inductive Program Synthesis over Noisy Datasets using Abstraction Refinement Based Optimization

We present a new synthesis algorithm to solve program synthesis over noi...

New Directions in Cloud Programming

Nearly twenty years after the launch of AWS, it remains difficult for mo...

Code Building Genetic Programming

In recent years the field of genetic programming has made significant ad...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.