Programming by Example (PBE) [10, 1] is an attractive means for end-user programming tasks, wherein the user provides the machine examples of a task she wishes to perform, and the machine infers a program to accomplish this. This paradigm has been used in a wide variety of domains;  gives a recent overview. We focus on text processing, a problem most computer users face (be it reformatting the contents of an email or extracting data from a log file), and for which several complete PBE systems have been designed, including LAPIS , SMARTedit , QuickCode [3, 13], and others [12, 14]. Such systems aim to provide a simpler alternative to the traditional solutions to the problem, which involve either tedious manual editing, or esoteric computing skills such as knowledge of awk or emacs.
A fundamental challenge in PBE is the following inference problem: given a set of base functions, how does one quickly search for programs composed of these functions that are consistent with the user-provided examples? One way is to make specific assumptions about the nature of the base functions, as is done by many existing PBE systems [11, 8, 3], but this is unsatisfying because it restricts the range of tasks a user can perform. The natural alternative, brute force search, is infeasible for even mildly involved programs . Thus, a basic question is whether there is a solution possessing both generality and efficiency.
This paper aims to take a step towards an affirmative answer to this question. We observe that there are often telling features in the user examples suggesting which functions are likely. For example, suppose that a user demonstrates their intended task through one or more input-output pairs of strings , where each is a permutation of . This feature provides a clue that when the system is searching for the such that , sorting functions may be useful. Our strategy is to incorporate a library of such clues, each suggesting relevant functions based on textual features of the input-output pairs. We learn weights telling us the reliability of each clue, and use this to bias program search. This bias allows for significantly faster inference compared to brute force search. Experiments on a prototype system demonstrate the effectiveness of feature-based learning.
To clarify matters, we step through a concrete example of our system’s operation.
1.1 Example of our system’s operation
Imagine a user has a long list of names with some repeated entries (say, the Oscar winners for Best Actor), and would like to create a list of the unique names, each annotated with their number of occurrences. Following the PBE paradigm, in our system, the user illustrates the operation by providing an example, which is an input-output pair of strings. Figure 1 shows one possible such pair, which uses a subset of the full list (in particular, the winners from ’91–’95) the user possesses.
One way to perform the above transformation is to first generate an intermediate list where each element of the input list is appended with its occurrence count – which would look like [ "Anthony Hopkins (1)", "Al Pacino (1)", "Tom Hanks (2)", "Tom Hanks (2)", "Nicolas Cage (1)"] – and then remove duplicates. The corresponding program may be expressed as the composition
The argument here represents the list of input lines that the user wishes to process, which may be much larger than the input provided in the example. We assume here a base language comprising (among others) a function dedup that removes duplicates from a list, concatLists that concatenates lists of strings elementwise, implicitly expanding singleton arguments, and count that finds the number of occurrences of the elements of one list in another.
While simple, this example is out of scope for existing text processing PBE systems. Most systems support a restricted, pre-defined set of functions that do not include natural tasks like removing duplicates; for example  only supports functions that operate on a line-by-line basis. These systems perform inference with search routines that are hand-coded for their supported functionality, and are thus not easily extensible. (Even if an exception could be made for specific examples like the one above, there are countless other text processing applications we would like to solve.) Systems with richer functionality are inapplicable because they perform inference with brute force search (or a similarly intractable operation ) over all possible function compositions. Such a naïve search over even a moderate sized library of base functions is unlikely to find the complex composition of our example. Therefore, a more generic framework is needed.
Our basic observation is that certain textual features can help bias our search by providing clues about which functions may be relevant: in particular, (a) there are duplicate lines in the input but not output, suggesting that dedup may be useful, (b) there are parentheses in the output but not input, suggesting the function concatLists("(",,")") for some list , (c) there are numbers on each line of the output but none in the input, suggesting that count may be useful, and (d) there are many more spaces in the output than the input, suggesting that " " may be useful. Our claim is that by learning weights that tell us the reliability of these clues – for example, how confident can we be that duplicates in the input but not the output suggests dedup – we can significantly speed up the inference process over brute force search.
In more detail, a clue is a function that generates rules in a probabilistic context free grammar based on features of the provided example. Each rule corresponds to a function111When we describe clues as suggesting functions, we implicitly mean the corresponding grammar rule.
(possibly with bound arguments) or constant in the underlying programming language. The rule probabilities are computed from weights on the clues that generate them, which in turn are learned from a training corpus of input-output examples. To learn, we now search through derivations of this grammar in order of decreasing probability. Table 1 illustrates what the grammar may look like for the above example. Note that the grammar rules and probabilities are example specific; we do not include a rule such as DELIM "$", say, because there is no instance of "$" in the input or output. Further, compositions of rules may also be generated, such as concatList("(",LIST,")").
Table 1 is of course a condensed view of the actual grammar our prototype system generates, which is based on a large library of about features and clues. With the full grammar, a naïve brute force search over compositions takes seconds to find the right solution to the example of Figure 1, whereas with learning the search terminates in just seconds.
To the best of our knowledge, ours is the first PBE system to exploit textual features for inference, which we believe is a step towards achieving the desiderata of efficiency and generality. The former will be demonstrated in an empirical evaluation of our learning scheme. For the latter, while the learning component is discussed in the context of text processing, the approach could possibly be adapted for different domains. Further, the resulting system is highly extensible. Through the use of clues, one only considers broadly relevant functions during the search for a suitable program: one is free to add functionality to process addresses, e.g. , without fear of it adversely affecting the performance of processing dates. Through the use of learning, we further sift amongst these broadly relevant functions, and determine which of them is likely to be useful in explaining the given data. A system designer need only write clues for any new functionality, and add relevant examples to the training corpus. Our system then automatically learns weights associated with these clues.
1.3 Comparison to previous learning systems
Most previous PBE systems for text processing handle a relatively small subset of natural text processing tasks. This is in order to admit efficient representation and search over consistent programs, e.g. using a version space , thus sidestepping the issue of searching for programs using general classes of functions. To our knowledge, every system designed for a library of arbitrary functions searches for appropriate compositions of functions either by brute force search, or a similarly intractable operation such as invoking a SAT solver .222 One could consider employing heuristic search techniques such as Genetic Programming. However, this requires picking a metric that captures meaningful search progress. This is difficult, since functions like sorting cause drastic changes on an input. Thus, standard metrics like edit distance may not be appropriate.
One could consider employing heuristic search techniques such as Genetic Programming. However, this requires picking a metric that captures meaningful search progress. This is difficult, since functions like sorting cause drastic changes on an input. Thus, standard metrics like edit distance may not be appropriate.Our learning approach based on textual features is thus more general and flexible than previous approaches.
Having said this, our goal in this paper is not to compete with existing PBE systems in terms of functionality. Instead, we wish to show that the fundamental PBE inference problem may be attacked by learning with textual features. This idea could in fact be applied in conjunction with prior systems. A specific feature of the data, such as the input and output having the same number of lines, may be a clue that a function corresponding to a system like QuickCode  will be useful.
2 Formalism of our approach
We begin a formal discussion of our approach by defining the learning problem in PBE.
2.1 Programming by example (PBE)
Let denote the set of strings. At inference time, the user provides a system input , where represents the data to be processed, and is the example input-output pair that represents the string transformation the user wishes to perform. In the example of the previous section, is the pair of strings represented in Figure 1, and is the list of all Oscar winners. While a typical choice for is some prefix of , this is not required in general333This is more general than the setup of e.g. , which assumes and have the same number of lines, each of which is treated as a separate example.. We assume that , for some unknown target function or program , from the set of functions that map strings to strings. Our goal is to recover .
We do so by defining a probability model over programs, parameterized by some . Given some , at inference time on input , we pick the most likely program under which is also consistent with . We do so by invoking a search function that depends on and an upper bound on search time. This produces our conjectured program computing a string-to-string transformation, or a trivial failure function if the search fails in the allotted time.
The parameters are learned at training time, where the system is given a corpus of training quadruples, , with representing the actual data and the example input-output pair, and the correct output on . Note that each quadruple here represents a different task; for example, one may represent the Oscar winners example of the previous section, another a generic email processing task, and so on. From these examples, the system chooses the parameters that maximize the likelihood . We now describe how we model the conditional distribution using a probabilistic context-free grammar.
2.2 PCFGs for programs
We maintain a probability distribution over programs with aProbabilistic Context-Free Grammar (PCFG) , as discussed in . The grammar is defined by a set of non-terminal symbols , terminal symbols (which may include strings and also other program-specific objects such as lists or functions), and rules . Each rule has an associated probability of being generated given the system input , where represents the unobserved parameters of the grammar. WLOG, each rule is also associated with a function , where NArgs denotes the number of arguments in the RHS of rule . A program444Two programs from different derivations may compute exactly the same function . However, determining whether two programs compute the same function is undecidable in general. Hence, we abuse notation and consider these to be different functions. is a derivation of the start symbol . The probability of any program is the probability of its constituent rules (counting repetitions):
We now describe how the distribution is parameterized using clues.
2.3 Features and clues for learning
The learning process exploits the following simple fact: the chance of a rule being part of an explanation for a string pair depends greatly on certain characteristics in the structure of and . For example, one interesting binary feature is whether or not every line of is a substring of . If true, it may suggest that the select_field rule should receive higher probability in the PCFG, and hence will be combined with other rules more often in the search. Another binary feature indicates whether or not “Massachusetts” occurs repeatedly as a substring in but not in
. This suggests that a rule generating the string “Massachusetts” may be useful. Conceptually, given a training corpus, we would like to learn the relationship between such features and the successful rules. However, there are an infinitude of such binary features as well as rules (e.g. a feature and rule corresponding to every possible constant string), but of course limited data and computational resources. So, we need a mechanism to estimate the relationship between the two entities.
We connect features with rules via clues. A clue is a function that states, for each system input , which subset of rules in (the infinite set of grammar rules), may be relevant. This set of rules will be based on certain features of , meaning that we search over compositions of instance-specific rules555As long as the functions generated by our clues library include a Turing-complete subset, the class of functions being searched amongst is always the Turing-computable functions, though having a good bias is probably more useful than being Turing complete.. For example, one clue might return if each line of is a substring of , and otherwise. Another clue might recognize the input string is a permutation of the output string, and generate rules , i.e., rules for sorting as well as introducing a nonterminal along with corresponding rules for various comparison functions. Note that a single clue can suggest a multitude of rules for different ’s (e.g. for every substring in the input), and “common” functions (e.g. concatenation of strings) may be suggested by multiple clues.
We now describe our probability model that is based on the clues formalism.
2.4 Probability model
Suppose the system has clues . For each clue , we keep an associated parameter . Let be the set of instance-specific rules (wrt ) in the grammar. While the set of all rules will be infinite in general, we assume there are a finite number of clues suggesting a finite number of rules, so that is finite. For each rule , we take , i.e. a rule that is not suggested by any clue is disregarded. For each rule , we use the probability model
where for each nonterminal , the normalizer ensures we get a valid probability distribution:
This is a log-linear model for the probabilities, where each clue has a weight , which is intuitively its reliability, and the probability of each rule is proportional to the product of the weights generating that rule. An alternative would be to make the probabilities be the (normalized) sums of corresponding weights, but we favor products for two reasons. First, as described shortly, maximizing the log-likelihood is a convex optimization problem in for products, but not for sums. Second, this formalism allows clues to have positive, neutral, or even negative influence on the likelihood of a rule, based upon the sign of .
3 System training and usage
We are now ready to describe in full the operation of the training and inference phases.
3.1 Training phase: learning
At training time, we wish to learn the parameter that characterizes the conditional probability of a program given the input, . We assume each training example is also annotated with the “correct” program that explains both the example and actual data pairs. We may attempt to discover these annotations automatically by bootstrapping: we start with a uniform parameter estimate . In iteration , we select to be the most likely program, based on , consistent with the system data. (If no program is found within the timeout, the example is ignored.) Then, parameters are learned, as described below. This is run until convergence.
Fix a single iteration . For notational convenience, we write target programs and parameters . We choose so as to minimize the negative log-likelihood of the data, plus a regularization term:
where is defined by equations (1) and (2), the regularizer is the norm , and is the regularization strength which may be chosen by cross-validation. If consists of rules (possibly with repetition), then
The convexity of the objective follows from the convexity of the regularizer and the log-sum-exp function. The parameters are optimized by gradient descent.
3.2 Inference phase: evaluating on new input
At inference time, we are given system input , clues , and parameters learned from the training phase. We are also given a timeout . The goal is to infer the most likely program that explains the data under a certain PCFG. This is done as follows:
We evaluate each clue on the system input . The underlying PCFG consists of the union of all suggested rules, .
Probabilities are assigned to these rules via Equation 2, using the learned parameters .
We enumerate over in order of decreasing probability, and return the first discovered that explains the string transformation, or if we exceed the timeout.
To find the most likely consistent program, we enumerate all programs of probability at least , for any given . We begin with a large , gradually decreasing it and testing all programs until we find one which outputs on (or we exceed the timeout ). (If more than one consistent program is found, we just select the most likely one.) Due to the exponentially increasing nature of the number of programs, this decreasing threshold approach imposes a negligible overhead due to redundancy – the vast majority of programs are executed just once.
To compute all programs of probability at least , a dynamic program first computes the maximal probability of a full trace from each nonterminal. Given these values, it is simple to compute the maximal probability completion of any partial trace. We then iterate over each nonterminal expansion, checking whether applying it can lead to any programs above the threshold; if so, we recurse.
4 Results on prototype system
4.1 Details of base functions and clues
As discussed in Section 2.2, we associated the rules in our PCFG with a set of base functions. In total we created around functions, such as dedup, concatLists, and count, as described in Section 1.1. For clues to connect these functions to features of the examples, we had one set of base clues that suggested functions we believed to be common, regardless of the system input (e.g. string concatenation). Other clues were designed to support common formats that we expected, such as dates, tabular and delimited data. Table 2 gives a sample of some of the clues in our system, in the form of grammar rules that certain textual features are connected to; in total we had approximately clues. The full list of functions and clues is available as part of our supplementary material.
|Substring appears in output but not input?||,|
|Duplicates in input but not output?|
|Numbers on each input line but not output line?|
4.2 Training set for learning
To evaluate the system, we compiled a set of examples with both an example pair and evaluation pair specified. These examples were partly hand-crafted, based on various common usage scenarios the authors have encountered, and partly based on examples used in . All examples are expressible as (possibly deep) compositions of our base functions; the median depth of composition on most examples is around . Like any classical learning model, we assume these are iid samples from the distribution of interest, namely over “natural’ text processing examples. It is hard to justify this independence assumption in our case, but we are not aware of a good solution to this problem in general; even examples collected from a user study, say, will tend to be biased in some way. Table 3 gives a sample of some of the scenarios we tested the system on. To encourage future research on this problem, our suite of training examples is ready for public release, and is available as part of our supplementary material.
|Adam Antn1 Ray Rd.nMAn90113||90113|
|28/6/2010||June the 28th 2010|
|612 Australia||case 612: return Australia;|
4.3 Does learning help?
The learning procedure aims to allow us to find the correct program in the shortest amount of time. We compare this method to a baseline, hoping to see quantifiable improvements in performance.
Baseline. Our baseline is to search through the grammar in order of increasing program size, attempting to find the shortest grammar derivation that explains the transformation. The grammar does use clues to winnow down the set of relevant rules, but does not use learned weights: we let for all , i.e. all rules that are suggested by a clue have the same constant probability. This method’s performance lets us measure the impact of learning. Note that pure brute force search would not even use clues to narrow down the set of feasible grammar rules, and so would perform strictly worse. Such a method is infeasible for the tasks we consider, because some of them involve e.g. constant strings, which cannot be enumerated.
Measuring performance. To assess a method, we look at its accuracy, as measured by the fraction of correctly discovered programs, and efficiency, as measured by the time required for inference. As every target program in the training set is expressible as a composition of our base functions, there are two ways in which we might fail to infer the correct program: (a) the program is not discoverable within the timeout set for the search, or (b) another program (one which also explains the example transformation) is wrongly given a higher probability. We call errors of type (a) timeout errors, and errors of type (b) ranking errors. Larger timeouts lead to fewer timeout errors.
Evaluation scheme. One possible pitfall in an empirical evaluation is having an overly specific set of clues for the training set: an extreme case would be a single clue for each training example, which automatically suggested the correct rules to compose. To ensure that the system is capable of making useful predictions on new data, we report the test error after creating random – splits of the training set. For each split, we compare the various methods as the inference timeout varies from seconds. For the learning method, we performed bootstrap iterations (see Section 3.1) with a timeout of seconds to get annotations for each training example.
Results. Figures 2(a) and 2(b) plot the timeout and ranking error rates respectively. As expected, for both methods, most errors arise due to timeout when the is small. To achieve the same timeout error rate, learning saves about two orders of magnitude in compared to the baseline. Learning also achieves lower mean ranking error, but this difference is not as pronounced as for timeout errors. This is not surprising, because the baseline generally finds few candidates in the first place (recall that the ranking error is only measured on examples that do not timeout); by contrast, the learning method opens the space of plausible candidates, but introduces a risk of some of them being incorrect.
Finally, Figure 3 compares the depths of programs (i.e. number of constituent grammar rules) discovered by learning and the baseline over all train-test splits, with an inference timeout of seconds. As expected, the learning procedure discovers many more programs that involve deep (depth ) compositions of rules, since the rules that are relevant are given higher probability.
5 Conclusion and future work
We propose a PBE system for repetitive text processing based on exploiting certain clues in the input data. We show how one can learn the utility of clues, which relate textual features to rules in a context free grammar. This allows us to speed up the search process, and obtain a meaningful ranking over programs. Experiments on a prototype system show that learning with clues brings significant savings over naïve brute force search. As future work, it would be interesting to learn correlations between rules and clues that did not suggest them, although this would necessitate enforcing some strong parameter sparsity. It would also be interesting to incorporate ideas like adaptor grammars  and learning program structure as in .
- Cypher et al.  Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, editors. Watch what I do: programming by demonstration. MIT Press, Cambridge, MA, USA, 1993. ISBN 0262032139.
- Gulwani  Sumit Gulwani. Dimensions in program synthesis. In PPDP, 2010.
- Gulwani  Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. In POPL, pages 317–330, 2011.
- Gulwani  Sumit Gulwani. Synthesis from examples. WAMBSE (Workshop on Advances in Model-Based Software Engineering) Special Issue, Infosys Labs Briefings, 10(2), 2012.
- Jha et al.  Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. Oracle-guided component-based program synthesis. In ICSE, pages 215–224, 2010.
- Johnson et al.  Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. Adaptor grammars: A framework for specifying compositional nonparametric bayesian models. In NIPS, pages 641–648, 2006.
- Lau et al.  Tessa Lau, Steven A. Wolfman, Pedro Domingos, and Daniel S. Weld. Programming by demonstration using version space algebra. Mach. Learn., 53:111–156, October 2003. ISSN 0885-6125.
- Lau et al.  Tessa A. Lau, Pedro Domingos, and Daniel S. Weld. Version space algebra and its application to programming by demonstration. In ICML, pages 527–534, 2000.
- Liang et al.  Percy Liang, Michael I. Jordan, and Dan Klein. Learning programs: A hierarchical Bayesian approach. In ICML, pages 639–646, 2010.
- Lieberman  H. Lieberman. Your Wish Is My Command: Programming by Example. Morgan Kaufmann, 2001.
- Miller and Myers  Robert C. Miller and Brad A. Myers. Lightweight structured text processing. In USENIX Annual Technical Conference, General Track, pages 131–144, 1999.
- Nix  Robert P. Nix. Editing by example. TOPLAS, 7(4):600–621, 1985.
- Singh and Gulwani  Rishabh Singh and Sumit Gulwani. Learning semantic string transformations from examples. PVLDB, 5(8), 2012.
- Witten and Mo  Ian H. Witten and Dan Mo. TELS: learning text editing tasks from examples, pages 183–203. MIT Press, Cambridge, MA, USA, 1993. ISBN 0-262-03213-9.