1 Introduction
We are interested in learning a semantic parser that maps natural language utterances into executable programs (e.g., logical forms). For example, in Figure 1, a program corresponding to the utterance transforms an initial world state into a new world state. We would like to learn from indirect supervision, where each training example is only labeled with the correct output (e.g. a target world state), but not the program that produced that output (Clarke et al., 2010; Liang et al., 2011; Krishnamurthy and Mitchell, 2012; Artzi and Zettlemoyer, 2013; Liang et al., 2017).
The process of constructing a program can be formulated as a sequential decisionmaking process, where feedback is only received at the end of the sequence when the completed program is executed. In the natural language processing literature, there are two common approaches for handling this situation: 1) reinforcement learning (RL), particularly the REINFORCE algorithm
(Williams, 1992; Sutton et al., 1999), which maximizes the expected reward of a sequence of actions; and 2) maximum marginal likelihood (MML), which treats the sequence of actions as a latent variable, and then maximizes the marginal likelihood of observing the correct program output (Dempster et al., 1977).While the two approaches have enjoyed success on many tasks, we found them to work poorly out of the box for our task. This is because in addition to the sparsity of correct programs, our task also requires weeding out spurious programs (Pasupat and Liang, 2016): incorrect interpretations of the utterances that accidentally produce the correct output, as illustrated in Figure 1.
We show that MML and RL optimize closely related objectives. Furthermore, both MML and RL methods have a mechanism for exploring program space in search of programs that generate the correct output. We explain why this exploration tends to quickly concentrate around short spurious programs, causing the model to sometimes overlook the correct program. To address this problem, we propose RandoMer, a new learning algorithm with two parts:
First, we propose randomized beam search, an exploration strategy which combines the systematic beam search traditionally employed in MML with the randomized offpolicy exploration of RL. This increases the chance of finding correct programs even when the beam size is small or the parameters are not pretrained.
Second, we observe that even with good exploration, the gradients of both the RL and MML objectives may still upweight entrenched spurious programs more strongly than correct programs with low probability under the current model. We propose a meritocratic parameter update rule, a modification to the MML gradient update, which more equally upweights all programs that produce the correct output. This makes the model less likely to overfit spurious programs.
We apply RandoMer to train a new neural semantic parser, which outputs programs in a stackbased programming language. We evaluate our resulting system on SCONE, the contextdependent semantic parsing dataset of long2016projections. Our approach outperforms standard RL and MML methods in a direct comparison, and achieves new stateoftheart results, improving over Long et al. (2016) in all three domains of SCONE, and by over 30% accuracy on the most challenging one.
2 Task
We consider the semantic parsing task in the SCONE dataset^{1}^{1}1 https://nlp.stanford.edu/projects/scone Long et al. (2016). As illustrated in Figure 1, each example consists of a world containing several objects (e.g., people), each with certain properties (e.g., shirt color and hat color). Given the initial world state and a sequence of natural language utterances , the task is to generate a program that manipulates the world state according to the utterances. Each utterance describes a single action that transforms the world state into a new world state . For training, the system receives weakly supervised examples with input and the target final world state .
The dataset includes 3 domains: Alchemy, Tangrams, and Scene. The description of each domain can be found in Appendix B. The domains highlight different linguistic phenomena: Alchemy features ellipsis (e.g., “throw the rest out”, “mix”); Tangrams features anaphora on actions (e.g., “repeat step 3”, “bring it back”); and Scene features anaphora on entities (e.g., “he moves back”, “…to his left”). Each domain contains roughly 3,700 training and 900 test examples. Each example contains 5 utterances and is labeled with the target world state after each utterance, but not the target program.
Spurious programs.
Given a training example , our goal is to find the true underlying program which reflects the meaning of . The constraint that must transform into , i.e. , is not enough to uniquely identify the true , as there are often many satisfying : in our experiments, we found at least 1600 on average for each example. Almost all do not capture the meaning of (see Figure 1). We refer to these incorrect ’s as spurious programs. Such programs encourage the model to learn an incorrect mapping from language to program operations: e.g., the spurious program in Figure 1 would cause the model to learn that “man in the yellow hat” maps to hasShirt(red).
Spurious programs in SCONE.
In this dataset, utterances often reference objects in different ways (e.g. a person can be referenced by shirt color, hat color, or position). Hence, any target programming language must also support these different reference strategies. As a result, even a single action such as moving a person to a target destination can be achieved by many different programs, each selecting the person and destination in a different way. Across multiple actions, the number of programs grows combinatorially.^{2}^{2}2The number of wellformed programs in Scene exceeds Only a few programs actually implement the correct reference strategy as defined by the utterance. This problem would be more severe in any more generalpurpose language (e.g. Python).
3 Model
We formulate program generation as a sequence prediction problem. We represent a program as a sequence of program tokens in postfix notation; for example, move(hasHat(yellow), leftOf(hasShirt(blue))) is linearized as yellow hasHat blue hasShirt leftOf move. This representation also allows us to incrementally execute programs from left to right using a stack: constants (e.g., yellow) are pushed onto the stack, while functions (e.g., hasHat) pop appropriate arguments from the stack and push back the computed result (e.g., the list of people with yellow hats). Appendix B lists the full set of program tokens, , and how they are executed. Note that each action always ends with an action token (e.g., move).
Given an input , the model generates program tokens from left to right using a neural encoderdecoder model with attention Bahdanau et al. (2015). Throughout the generation process, the model maintains an utterance pointer, , initialized to 1. To generate , the model’s encoder first encodes the utterance
into a vector
. Then, based on and previously generated tokens , the model’s decoder defines a distribution over the possible values of . The next token is sampled from this distribution. If an action token (e.g., move) is generated, the model increments the utterance pointer . The process terminates when all utterances are processed. The final probability of generating a particular program is .Encoder.
The utterance under the pointer is encoded using a bidirectional LSTM:
where is the fixed GloVe word embedding Pennington et al. (2014) of the th word in . The final utterance embedding is the concatenation .
Decoder.
Unlike bahdanau2015neural, which used a recurrent network for the decoder, we opt for a feedforward network for simplicity. We use and an embedding of the previous execution history (described later) as inputs to compute an attention vector :
Finally, after concatenating with , the distribution over the set of possible program tokens is computed via a softmax:
where is the embedding for token .
Execution history embedding.
We compare two options for , our embedding of the execution history. A standard approach is to simply take the most recent tokens and concatenate their embeddings. We will refer to this as Tokens and use in our experiments.
We also consider a new approach which leverages our ability to incrementally execute programs using a stack. We summarize the execution history by embedding the state of the stack at time , achieved by concatenating the embeddings of all values on the stack. (We limit the maximum stack size to 3.) We refer to this as Stack.
4 Reinforcement learning versus maximum marginal likelihood
Having formulated our task as a sequence prediction problem, we must still choose a learning algorithm. We first compare two standard paradigms: reinforcement learning (RL) and maximum marginal likelihood (MML). In the next section, we propose a better alternative.
4.1 Comparing objective functions
Reinforcement learning.
From an RL perspective, given a training example , a policy makes a sequence of decisions , and then receives a reward at the end of the episode: if executes to and 0 otherwise (dependence on and has been omitted from the notation).
We focus on policy gradient methods, in which a stochastic policy function is trained to maximize the expected reward. In our setup, is the policy (with parameters ), and its expected reward on a given example is
(1) 
where the sum is over all possible programs. The overall RL objective, , is the expected reward across examples:
(2) 
Maximum marginal likelihood.
The MML perspective assumes that is generated by a partiallyobserved random process: conditioned on , a latent program is generated, and conditioned on , the observation is generated. This implies the marginal likelihood:
(3) 
Note that since the execution of is deterministic, if executes to and 0 otherwise. The log marginal likelihood of the data is then
(4)  
(5) 
To estimate our model parameters
, we maximize with respect to .With our choice of reward, the RL expected reward (1) is equal to the MML marginal probability (3). Hence the only difference between the two formulations is that in RL we optimize the sum of expected rewards (2), whereas in MML we optimize the product (5).^{3}^{3}3 Note that the log of the product in (5) does not equal the sum in (2).
4.2 Comparing gradients
In both policy gradient and MML, the objectives are typically optimized via (stochastic) gradient ascent. The gradients of and are closely related. They both have the form:
(6)  
where equals
(7)  
(8)  
Taking a step in the direction of upweights the probability of
, so we can heuristically think of the gradient as attempting to upweight each rewardearning program
by a gradient weight . In Subsection 5.2, we argue why is better at guarding against spurious programs, and propose an even better alternative.4.3 Comparing gradient approximation strategies
It is often intractable to compute the gradient (6) because it involves taking an expectation over all possible programs. So in practice, the expectation is approximated.
In the policy gradient literature, Monte Carlo integration (MC) is the typical approximation strategy. For example, the popular REINFORCE algorithm (Williams, 1992)
uses Monte Carlo sampling to compute an unbiased estimate of the gradient:
(9) 
where is a collection of samples , and is a baseline (Williams, 1992)
used to reduce the variance of the estimate without altering its expectation.
In the MML literature for latent sequences, the expectation is typically approximated via numerical integration (NUM) instead:
(10) 
where the programs in come from beam search.
Beam search.
Beam search generates a set of programs via the following process. At step of beam search, we maintain a beam of at most search states. Each state represents a partially constructed program, (the first tokens of the program). For each state in the beam, we generate all possible continuations,
We then take the union of these continuations, . The new beam is simply the highest scoring continuations in , as scored by the policy, . Search is halted after a fixed number of iterations or when there are no continuations possible. is then the set of all complete programs discovered during beam search. We will refer to this as beam search MML (BSMML).
In both policy gradient and MML, we think of the procedure used to produce the set of programs as an exploration strategy which searches for programs that produce reward. One advantage of numerical integration is that it allows us to decouple the exploration strategy from the gradient weights assigned to each program.
5 Tackling spurious programs
In this section, we illustrate why spurious programs are problematic for the most commonly used methods in RL (REINFORCE) and MML (beam search MML). We describe two key problems and propose a solution to each, based on insights gained from our comparison of RL and MML in Section 4.
5.1 Spurious programs bias exploration
As mentioned in Section 4, REINFORCE and BSMML both employ an exploration strategy to approximate their respective gradients. In both methods, exploration is guided by the current model policy, whereby programs with high probability under the current policy are more likely to be explored. A troubling implication is that programs with low probability under the current policy are likely to be overlooked by exploration.
If the current policy incorrectly assigns low probability to the correct program , it will likely fail to discover during exploration, and will consequently fail to upweight the probability of . This repeats on every gradient step, keeping the probability of perpetually low. The same feedback loop can also cause already highprobability spurious programs to gain even more probability. From this, we see that exploration is sensitive to initial conditions: the rich get richer, and the poor get poorer.
Since there are often thousands of spurious programs and only a few correct programs, spurious programs are usually found first. Once spurious programs get a head start, exploration increasingly biases towards them.
As a remedy, one could try initializing parameters such that the model puts a uniform distribution over all possible programs. A seemingly reasonable tactic is to initialize parameters such that the model policy puts nearuniform probability over the decisions at each time step. However, this causes shorter programs to have orders of magnitude higher probability than longer programs, as illustrated in Figure
2and as we empirically observe. A more sophisticated approach might involve approximating the total number of programs reachable from each point in the programgenerating decision tree. However, we instead propose to reduce sensitivity to the initial distribution over programs.
Solution: randomized beam search
One solution to biased exploration is to simply rely less on the untrustworthy current policy. We can do this by injecting random noise into exploration.
In REINFORCE, a common solution is to sample from an greedy variant of the current policy. On the other hand, MML exploration with beam search is deterministic. However, it has a key advantage over REINFORCEstyle sampling: even if one program occupies almost all probability under the current policy (a peaky distribution), beam search will still use its remaining beam capacity to explore at least other programs. In contrast, sampling methods will repeatedly visit the mode of the distribution.
To get the best of both worlds, we propose a simple greedy randomized beam search. Like regular beam search, at iteration we compute the set of all continuations and sort them by their model probability . But instead of selecting the highestscoring continuations, we choose continuations one by one without replacement from . When choosing a continuation from the remaining pool, we either uniformly sample a random continuation with probability , or pick the highestscoring continuation in the pool with probability . Empirically, we find that this performs much better than both classic beam search and greedy sampling (Table 3).
5.2 Spurious programs dominate gradients
In both RL and MML, even if exploration is perfect and the gradient is exactly computed, spurious programs can still be problematic.
Even if perfect exploration visits every program, we see from the gradient weights in (7) and (8) that programs are weighted proportional to their current policy probability. If a spurious program has 100 times higher probability than as in Figure 2, the gradient will spend roughly 99% of its magnitude upweighting towards and only 1% towards even though the two programs get the same reward.
This implies that it would take many updates for to catch up. In fact, may never catch up, depending on the gradient updates for other training examples. Simply increasing the learning rate is inadequate, as it would cause the model to take overly large steps towards , potentially causing optimization to diverge.
Solution: the meritocratic update rule
To solve this problem, we want the upweighting to be more “meritocratic”: any program that obtains reward should be upweighted roughly equally.
We first observe that already improves over in this regard. From (6), we see that the gradient weight is the policy distribution restricted to and renormalized over only rewardearning programs. This renormalization makes the gradient weight uniform across examples: even if all rewardearning programs for a particular example have very low model probability, their combined gradient weight is always 1. In our experiments, performs significantly better than (Table 4).
However, while assigns uniform weight across examples, it is still not uniform over the programs within each example. Hence we propose a new update rule which goes one step further in pursuing uniform updates. Extending , we define a smoothed version:
(11) 
When , our weighting is completely uniform across all rewardearning programs within an example while recovers the original MML weighting. Our new update rule is to simply take a modified gradient step where .^{4}^{4}4 Also, note that if exploration were exhaustive,
would be equivalent to supervised learning using the set of all rewardearning programs as targets.
We will refer to this as the meritocratic update rule.5.3 Summary of the proposed approach
Method  Approximation of  Exploration strategy  Gradient weight 

REINFORCE  Monte Carlo integration  independent sampling  
BSMML  numerical integration  beam search  
RandoMer  numerical integration  randomized beam search 
We described two problems^{5}^{5}5 These problems concern the gradient w.r.t. a single example. The full gradient averages over multiple examples, which helps separate correct from spurious. E.g., if multiple examples all mention “yellow hat”, we will find a correct program parsing this as hasHat(yellow) for each example, whereas the spurious programs we find will follow no consistent pattern. Consequently, spurious gradient contributions may cancel out while correct program gradients will all “vote” in the same direction. and their solutions: we reduce exploration bias using greedy randomized beam search and perform more balanced optimization using the meritocratic parameter update rule. We call our resulting approach RandoMer. Table 1 summarizes how RandoMer combines desirable qualities from both REINFORCE and BSMML.
6 Experiments
Evaluation.
We evaluate our proposed methods on all three domains of the SCONE dataset. Accuracy is defined as the percentage of test examples where the model produces the correct final world state . All test examples have (5utts), but we also report accuracy after processing the first 3 utterances (3utts). To control for the effects of randomness, we train 5 instances of each model with different random seeds. We report the median accuracy of the instances unless otherwise noted.
Training.
Following Long et al. (2016), we decompose each training example into smaller examples. Given an example with 5 utterances, , we consider all length1 and length2 substrings of : (9 total). We form a new training example from each substring, e.g., where , and .
All models are implemented in TensorFlow
(Abadi et al., 2015). Model parameters are randomly initialized (Glorot and Bengio, 2010), with no pretraining. We use the Adam optimizer (Kingma and Ba, 2014) (which is applied to the gradient in (6)), a learning rate of 0.001, a minibatch size of 8 examples (different from the beam size), and train until accuracy on the validation set converges (on average about 13,000 steps). We use fixed GloVe vectors (Pennington et al., 2014) to embed the words in each utterance.Hyperparameters.
For all models, we performed a grid search over hyperparameters to maximize accuracy on the validation set. Hyperparameters include the learning rate, the baseline in REINFORCE,
greediness and meritocraticness. For REINFORCE, we also experimented with a regressionestimated baseline (Ranzato et al., 2015), but found it to perform worse than a constant baseline.6.1 Main results
Comparison to prior work.
Table 2 compares RandoMer to results from Long et al. (2016) as well as two baselines, REINFORCE and BSMML (using the same neural model but different learning algorithms). Our approach achieves new stateoftheart results by a significant margin, especially on the Scene domain, which features the most complex program syntax. We report the results for REINFORCE, BSMML, and RandoMer on the seed and hyperparameters that achieve the best validation accuracy.
We note that REINFORCE performs very well on Tangrams but worse on Alchemy and very poorly on Scene. This might be because the program syntax for Tangrams is simpler than the other two: there is no other way to refer to objects except by index.
We also found that REINFORCE required greedy exploration to make any progress. Using
greedy greatly skews the Monte Carlo approximation of
, making it more uniformly weighted over programs in a similar spirit to using meritocratic gradient weights . However, increases uniformity over rewardearning programs only, rather than over all programs.Alchemy  Tangrams  Scene  

system  3utts  5utts  3utts  5utts  3utts  5utts 
Long+16  56.8  52.3  64.9  27.6  23.2  14.7 
REINFORCE  58.3  44.6  68.5  37.3  47.8  33.9 
BSMML  58.7  47.3  62.6  32.2  53.5  32.5 
RandoMer  66.9  52.9  65.8  37.1  64.8  46.2 
Effect of randomized beam search.
Table 3 shows that greedy randomized beam search consistently outperforms classic beam search. Even when we increase the beam size of classic beam search to 128, it still does not surpass randomized beam search with a beam of 32, and further increases yield no additional improvement.
Alchemy  Tangrams  Scene  
random  beam  3utts  5utts  3utts  5utts  3utts  5utts 
classic beam search  
None  32  30.3  23.2  0.0  0.0  33.4  20.1 
None  128  59.0  46.4  60.9  28.6  24.5  13.9 
randomized beam search  
32  58.7  45.5  61.1  32.5  33.4  23.0  
32  61.3  48.3  65.2  34.3  50.8  33.5  
32  60.5  48.6  60.0  27.3  54.1  35.7 
Effect of meritocratic updates.
Table 4 evaluates the impact of meritocratic parameter updates (gradient weight ). More uniform upweighting across rewardearning programs leads to higher accuracy and fewer spurious programs, especially in Scene. However, no single value of performs best over all domains.
Choosing the right value of in RandoMer significantly accelerates training. Figure 3 illustrates that while and ultimately achieve similar accuracy on Alchemy, reaches good performance in half the time.
Since lowering reduces trust in the model policy, helps in early training when the current policy is untrustworthy. However, as it grows more trustworthy, begins to pay a price for ignoring it. Hence, it may be worthwhile to anneal towards 1 over time.
Alchemy  Tangrams  Scene  

3utts  5utts  3utts  5utts  3utts  5utts  
0.2  0.0  0.9  0.6  0.0  0.0  
61.3  48.3  65.2  34.3  50.8  33.5  
64.4  48.9  60.6  29.0  42.4  29.7  
63.6  46.3  54.0  23.5  61.0  42.4 
Effect of execution history embedding.
Table 5 compares our two proposals for embedding the execution history: Tokens and Stack. Stack performs better in the two domains where an object can be referenced in multiple ways (Scene and Alchemy). Stack directly embeds objects on the stack, invariant to the way in which they were pushed onto the stack, unlike Tokens. We hypothesize that this invariance increases robustness to spurious behavior: if a program accidentally pushes the right object onto the stack via spurious means, the model can still learn the remaining steps of the program without conditioning on a spurious history.
Alchemy  Tangrams  Scene  

3utts  5utts  3utts  5utts  3utts  5utts  
History  61.3  48.3  65.2  34.3  50.8  33.5 
Stack  64.2  53.2  63.0  32.4  59.5  43.1 
Fitting vs overfitting the training data.
Table 6 reveals that BSMML and RandoMer use different strategies to fit the training data. On the depicted training example, BSMML actually achieves higher expected reward / marginal probability than RandoMer, but it does so by putting most of its probability on a spurious program—a form of overfitting. In contrast, RandoMer spreads probability mass over multiple rewardearning programs, including the correct ones.
As a consequence of overfitting, we observed at test time that BSMML only references people by positional indices instead of by shirt or hat color, whereas RandoMer successfully learns to use multiple reference strategies.
7 Related work and discussion
Semantic parsing from indirect supervision. Our work is motivated by the classic problem of learning semantic parsers from indirect supervision (Clarke et al., 2010; Liang et al., 2011; Artzi and Zettlemoyer, 2011, 2013; Reddy et al., 2014; Pasupat and Liang, 2015). We are interested in the initial stages of training from scratch, where getting any training signal is difficult due to the combinatorially large search space. We also highlighted the problem of spurious programs which capture reward but give incorrect generalizations.
Maximum marginal likelihood with beam search (BSMML) is traditionally used to learn semantic parsers from indirect supervision.
Reinforcement learning. Concurrently, there has been a recent surge of interest in reinforcement learning, along with the wide application of the classic REINFORCE algorithm (Williams, 1992)—to troubleshooting (Branavan et al., 2009), dialog generation (Li et al., 2016), game playing (Narasimhan et al., 2015), coreference resolution (Clark and Manning, 2016), machine translation (Norouzi et al., 2016), and even semantic parsing (Liang et al., 2017). Indeed, the challenge of training semantic parsers from indirect supervision is perhaps better captured by the notion of sparse rewards in reinforcement learning.
The RL answer would be better exploration, which can take many forms including simple actiondithering such as greedy, entropy regularization (Williams and Peng, 1991), Monte Carlo tree search (Coulom, 2006), randomized value functions (Osband et al., 2014, 2016), and methods which prioritize learning environment dynamics (Duff, 2002) or underexplored states (Kearns and Singh, 2002; Bellemare et al., 2016; Nachum et al., 2016). The majority of these methods employ Monte Carlo sampling for exploration. In contrast, we find randomized beam search to be more suitable in our setting, because it explores lowprobability states even when the policy distribution is peaky. Our meritocratic update also depends on the fact that beam search returns an entire set of rewardearning programs rather than one, since it renormalizes over the rewardearning set. While similar to entropy regularization, meritocratic update is more targeted as it only increases uniformity of the gradient among rewardearning programs, rather than across all programs.
Our strategy of using randomized beam search and meritocratic updates lies closer to MML than RL, but this does not imply that RL has nothing to offer in our setting. With the simple connection between RL and MML we established, much of the literature on exploration and variance reduction in RL can be directly applied to MML problems. Of special interest are methods which incorporate a value function such as actorcritic.
Maximum likelihood and RL.
It is tempting to group our approach with sequence learning methods which interpolate between supervised learning and reinforcement learning
(Ranzato et al., 2015; Venkatraman et al., 2015; Ross et al., 2011; Norouzi et al., 2016; Bengio et al., 2015; Levine, 2014). These methods generally seek to make RL training easier by pretraining or “warmstarting” with fully supervised learning. This requires each training example to be labeled with a reasonably correct output sequence. In our setting, this would amount to labeling each example with the correct program, which is not known. Hence, these methods cannot be directly applied.Without access to correct output sequences, we cannot directly maximize likelihood, and instead resort to maximizing the marginal likelihood (MML). Rather than proposing MML as a form of pretraining, we argue that MML is a superior substitute for the standard RL objective, and that the meritocratic update is even better.
Simulated annealing. Our meritocratic update employs exponential smoothing, which bears resemblance to the simulated annealing strategy of Och (2003); Smith and Eisner (2006); Shen et al. (2015). However, a key difference is that these methods smooth the objective function whereas we smooth an expectation in the gradient. To underscore the difference, we note that fixing in our method (total smoothing) is quite effective, whereas total smoothing in the simulated annealing methods would correspond to a completely flat objective function, and an uninformative gradient of zero everywhere.
Neural semantic parsing.
There has been recent interest in using recurrent neural networks for semantic parsing, both for modeling logical forms
(Dong and Lapata, 2016; Jia and Liang, 2016; Liang et al., 2017) and for endtoend execution (Yin et al., 2015; Neelakantan et al., 2016). We develop a neural model for the contextdependent setting, which is made possible by a new stackbased language similar to Riedel et al. (2016).Acknowledgments.
This work was supported by the NSF Graduate Research Fellowship under No. DGE114747 and the NSF CAREER Award under No. IIS1552635.
Reproducibility.
Our code is made available at https://github.com/kelvinguu/lang2program. Reproducible experiments are available at https://worksheets.codalab.org/worksheets/0x88c914ee1d4b4a4587a07f36f090f3e5/.
References
 Abadi et al. (2015) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. 2015. Tensorflow: Largescale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 .
 Artzi and Zettlemoyer (2011) Y. Artzi and L. Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Empirical Methods in Natural Language Processing (EMNLP). pages 421–432.
 Artzi and Zettlemoyer (2013) Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL) 1:49–62.
 Bahdanau et al. (2015) D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR).
 Bellemare et al. (2016) M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. 2016. Unifying countbased exploration and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS). pages 1471–1479.
 Bengio et al. (2015) S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems (NIPS). pages 1171–1179.
 Branavan et al. (2009) S. Branavan, H. Chen, L. S. Zettlemoyer, and R. Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACLIJCNLP). pages 82–90.
 Clark and Manning (2016) K. Clark and C. D. Manning. 2016. Deep reinforcement learning for mentionranking coreference models. arXiv preprint arXiv:1609.08667 .
 Clarke et al. (2010) J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world’s response. In Computational Natural Language Learning (CoNLL). pages 18–27.
 Coulom (2006) R. Coulom. 2006. Efficient selectivity and backup operators in MonteCarlo tree search. In International Conference on Computers and Games. pages 72–83.
 Dempster et al. (1977) A. P. Dempster, L. N. M., and R. D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B 39(1):1–38.
 Dong and Lapata (2016) L. Dong and M. Lapata. 2016. Language to logical form with neural attention. In Association for Computational Linguistics (ACL).

Duff (2002)
M. O. Duff. 2002.
Optimal Learning: Computational procedures for Bayesadaptive Markov decision processes
. Ph.D. thesis, University of Massachusetts Amherst. 
Glorot and Bengio (2010)
X. Glorot and Y. Bengio. 2010.
Understanding the difficulty of training deep feedforward neural
networks.
In
International Conference on Artificial Intelligence and Statistics
.  Jia and Liang (2016) R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL).
 Kearns and Singh (2002) M. Kearns and S. Singh. 2002. Nearoptimal reinforcement learning in polynomial time. Machine Learning 49(2):209–232.
 Kingma and Ba (2014) D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .
 Krishnamurthy and Mitchell (2012) J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL). pages 754–765.
 Levine (2014) S. Levine. 2014. Motor Skill Learning with Local Trajectory Methods. Ph.D. thesis, Stanford University.
 Li et al. (2016) J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao. 2016. Deep reinforcement learning for dialogue generation. In Empirical Methods in Natural Language Processing (EMNLP).
 Liang et al. (2017) C. Liang, J. Berant, Q. Le, and K. D. F. N. Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Association for Computational Linguistics (ACL).
 Liang et al. (2011) P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependencybased compositional semantics. In Association for Computational Linguistics (ACL). pages 590–599.
 Long et al. (2016) R. Long, P. Pasupat, and P. Liang. 2016. Simpler contextdependent logical forms via model projections. In Association for Computational Linguistics (ACL).
 Nachum et al. (2016) O. Nachum, M. Norouzi, and D. Schuurmans. 2016. Improving policy gradient by exploring underappreciated rewards. arXiv preprint arXiv:1611.09321 .
 Narasimhan et al. (2015) K. Narasimhan, T. Kulkarni, and R. Barzilay. 2015. Language understanding for textbased games using deep reinforcement learning. arXiv preprint arXiv:1506.08941 .
 Neelakantan et al. (2016) A. Neelakantan, Q. V. Le, and I. Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In International Conference on Learning Representations (ICLR).
 Norouzi et al. (2016) M. Norouzi, S. Bengio, N. Jaitly, M. Schuster, Y. Wu, D. Schuurmans, et al. 2016. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems. pages 1723–1731.
 Och (2003) F. J. Och. 2003. Minimum error rate training in statistical machine translation. In Association for Computational Linguistics (ACL). pages 160–167.
 Osband et al. (2016) I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. 2016. Deep exploration via bootstrapped DQN. In Advances In Neural Information Processing Systems. pages 4026–4034.
 Osband et al. (2014) I. Osband, B. V. Roy, and Z. Wen. 2014. Generalization and exploration via randomized value functions. arXiv preprint arXiv:1402.0635 .
 Pasupat and Liang (2015) P. Pasupat and P. Liang. 2015. Compositional semantic parsing on semistructured tables. In Association for Computational Linguistics (ACL).
 Pasupat and Liang (2016) P. Pasupat and P. Liang. 2016. Inferring logical forms from denotations. In Association for Computational Linguistics (ACL).
 Pennington et al. (2014) J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP).
 Ranzato et al. (2015) M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 .
 Reddy et al. (2014) S. Reddy, M. Lapata, and M. Steedman. 2014. Largescale semantic parsing without questionanswer pairs. Transactions of the Association for Computational Linguistics (TACL) 2(10):377–392.
 Riedel et al. (2016) S. Riedel, M. Bosnjak, and T. Rocktäschel. 2016. Programming with a differentiable forth interpreter. CoRR, abs/1605.06640 .

Ross et al. (2011)
S. Ross, G. Gordon, and A. Bagnell. 2011.
A reduction of imitation learning and structured prediction to noregret online learning.
In Artificial Intelligence and Statistics (AISTATS).  Shen et al. (2015) S. Shen, Y. Cheng, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. 2015. Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433 .
 Smith and Eisner (2006) D. A. Smith and J. Eisner. 2006. Minimum risk annealing for training loglinear models. In International Conference on Computational Linguistics and Association for Computational Linguistics (COLING/ACL). pages 787–794.
 Sutton et al. (1999) R. Sutton, D. McAllester, S. Singh, and Y. Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS).
 Venkatraman et al. (2015) A. Venkatraman, M. Hebert, and J. A. Bagnell. 2015. Improving multistep prediction of learned time series models. In Association for the Advancement of Artificial Intelligence (AAAI). pages 3024–3030.
 Williams (1992) R. J. Williams. 1992. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. Machine learning 8(3):229–256.
 Williams and Peng (1991) R. J. Williams and J. Peng. 1991. Function optimization using connectionist reinforcement learning algorithms. Connection Science 3(3):241–268.
 Yin et al. (2015) P. Yin, Z. Lu, H. Li, and B. Kao. 2015. Neural enquirer: Learning to query tables. arXiv preprint arXiv:1512.00965 .
Appendix A Hyperparameters in Table 2
System  Alchemy  Tangrams  Scene 

REINFORCE 
Sample size Baseline
embed Tokens 
Sample size Baseline
embed Tokens 
Sample size Baseline
embed Tokens 
BSMML 
Beam size embed Tokens 
Beam size embed Tokens 
Beam size embed Tokens 
RandoMer 
embed Tokens 
embed Tokens 
embed Stack 
Appendix B SCONE domains and program tokens
token  type  semantics 
Shared across Alchemy, Tangrams, Scene  
1, 2, 3, …  constant  push: number 
1, 2, 3, …  
red, yellow, green,  constant  push: color 
orange, purple, brown  
allObjects  constant  push: the list of all objects 
index  function  pop: a list and a number 
push: the object (the index starts from 1; negative indices are allowed)  
prevArg ()  function  pop: a number 
push: the argument from the th action  
prevAction  action  pop: a number 
perform: fetch the th action and execute it using the arguments on the stack  
Additional tokens for the Alchemy domain An Alchemy world contains 7 beakers. Each beaker may contain up to 4 units of colored chemical.  
1/1  constant  push: fraction (used in the drain action) 
hasColor  function  pop: a color 
push: list of beakers with chemical color  
drain  action  pop: a beaker and a number or fraction 
perform: remove units of chemical (or all chemical if 1/1) from  
pour  action  pop: two beakers and 
perform: transfer all chemical from to  
mix  action  pop: a beaker 
perform: turn the color of the chemical in to brown  
Additional tokens for the Tangrams domain A Tangrams world contains a row of tangram pieces with different shapes. The shapes are anonymized; a tangram can be referred to by an index or a history reference, but not by shape.  
swap  action  pop: two tangrams and 
perform: exchange the positions of and  
remove  action  pop: a tangram 
perform: remove from the stage  
add  action  pop: a number and a previously removed tangram 
perform: insert to position  
Additional tokens for the Scene domain A Scene world is a linear stage with 10 positions. Each position may be occupied by a person with a colored shirt and optionally a colored hat. There are usually 15 people on the stage.  
noHat  constant  push: pseudocolor (indicating that the person is not wearing a hat) 
hasShirt, hasHat  function  pop: a color 
push: the list of all people with shirt or hat color  
hasShirtHat  function  pop: two colors and 
push: the list of all people with shirt color and hat color  
leftOf, rightOf  function  pop: a person 
push: the location index left or right of  
create  action  pop: a number and two colors , 
perform: add a new person at position with shirt color and hat color  
move  action  pop: a person and a number 
perform: move to position  
swapHats  action  pop: two people and 
perform: have and exchange their hats  
leave  action  pop: a person 
perform: remove from the stage 