SynGuar: Guaranteeing Generalization in Programming by Example

by   Bo Wang, et al.

Programming by Example (PBE) is a program synthesis paradigm in which the synthesizer creates a program that matches a set of given examples. In many applications of such synthesis (e.g., program repair or reverse engineering), we are to reconstruct a program that is close to a specific target program, not merely to produce some program that satisfies the seen examples. In such settings, we wish that the synthesized program generalizes well, i.e., has as few errors as possible on the unobserved examples capturing the target function behavior. In this paper, we propose the first framework (called SynGuar) for PBE synthesizers that guarantees to achieve low generalization error with high probability. Our main contribution is a procedure to dynamically calculate how many additional examples suffice to theoretically guarantee generalization. We show how our techniques can be used in 2 well-known synthesis approaches: PROSE and STUN (synthesis through unification), for common string-manipulation program benchmarks. We find that often a few hundred examples suffice to provably bound generalization error below 5% with high (≥ 98%) probability on these benchmarks. Further, we confirm this empirically: SynGuar significantly improves the accuracy of existing synthesizers in generating the right target programs. But with fewer examples chosen arbitrarily, the same baseline synthesizers (without SynGuar) overfit and lose accuracy.


page 1

page 2

page 3

page 4


Learning to Combine Per-Example Solutions for Neural Program Synthesis

The goal of program synthesis from examples is to find a computer progra...

Programming-By-Example by Programming-By-Example: Synthesis of Looping Programs

Program synthesis has seen many new applications in recent years, in lar...

Program Synthesis using Abstraction Refinement

We present a new approach to example-guided program synthesis based on c...

Type-driven Neural Programming by Example

In this thesis we look into programming by example (PBE), which is about...

Efficient Pragmatic Program Synthesis with Informative Specifications

Providing examples is one of the most common way for end-users to intera...

EqFix: Fixing LaTeX Equation Errors by Examples

LaTeX is a widely-used document preparation system. Its powerful ability...

Amortizing Pragmatic Program Synthesis with Rankings

In program synthesis, an intelligent system takes in a set of user-gener...

Please sign up or login with your details

Forgot password? Click here to reset