1 Introduction
Sequential recurrent neural networks (RNNs) are remarkably effective models of natural language. In the last few years, language model results that substantially improve over longestablished stateoftheart baselines have been obtained using RNNs (Zaremba et al., 2015; Mikolov et al., 2010) as well as in various conditional language modeling tasks such as machine translation (Bahdanau et al., 2015), image caption generation (Xu et al., 2015), and dialogue generation (Wen et al., 2015). Despite these impressive results, sequential models are a priori inappropriate models of natural language, since relationships among words are largely organized in terms of latent nested structures rather than sequential surface order (Chomsky, 1957).
In this paper, we introduce recurrent neural network grammars (RNNGs; §2), a new generative probabilistic model of sentences that explicitly models nested, hierarchical relationships among words and phrases. RNNGs operate via a recursive syntactic process reminiscent of probabilistic contextfree grammar generation, but decisions are parameterized using RNNs that condition on the entire syntactic derivation history, greatly relaxing contextfree independence assumptions.
The foundation of this work is a topdown variant of transitionbased parsing (§3). We give two variants of the algorithm, one for parsing (given an observed sentence, transform it into a tree), and one for generation. While several transitionbased neural models of syntactic generation exist (Henderson, 2003, 2004; Emami and Jelinek, 2005; Titov and Henderson, 2007; Buys and Blunsom, 2015b), these have relied on structure building operations based on parsing actions in shiftreduce and leftcorner parsers which operate in a largely bottomup fashion. While this construction is appealing because inference is relatively straightforward, it limits the use of topdown grammar information, which is helpful for generation (Roark, 2001).^{2}^{2}2The leftcorner parsers used by Henderson (2003, 2004) incorporate limited topdown information, but a complete path from the root of the tree to a terminal is not generally present when a terminal is generated. Refer to Henderson (2003, Fig. 1) for an example. RNNGs maintain the algorithmic convenience of transitionbased parsing but incorporate topdown (i.e., roottoterminal) syntactic information (§4).
The topdown transition set that RNNGs are based on lends itself to discriminative modeling as well, where sequences of transitions are modeled conditional on the full input sentence along with the incrementally constructed syntactic structures. Similar to previously published discriminative bottomup transitionbased parsers (Henderson, 2004; Sagae and Lavie, 2005; Zhang and Clark, 2011, inter alia)
, greedy prediction with our model yields a lineartime deterministic parser (provided an upper bound on the number of actions taken between processing subsequent terminal symbols is imposed); however, our algorithm generates arbitrary tree structures directly, without the binarization required by shiftreduce parsers. The discriminative model also lets us use ancestor sampling to obtain samples of parse trees for sentences, and this is used to solve a second practical challenge with RNNGs: approximating the marginal likelihood and MAP tree of a sentence under the generative model. We present a simple importance sampling algorithm which uses samples from the discriminative parser to solve inference problems in the generative model (§
5).Experiments show that RNNGs are effective for both language modeling and parsing (§6). Our generative model obtains (i) the bestknown parsing results using a single supervised generative model and (ii) better perplexities in language modeling than stateoftheart sequential LSTM language models. Surprisingly—although in line with previous parsing results showing the effectiveness of generative models (Henderson, 2004; Johnson, 2001)—parsing with the generative model obtains significantly better results than parsing with the discriminative model.
2 RNN Grammars
Formally, an RNNG is a triple consisting of a finite set of nonterminal symbols (), a finite set of terminal symbols () such that , and a collection of neural network parameters . It does not explicitly define rules since these are implicitly characterized by . The algorithm that the grammar uses to generate trees and strings in the language is characterized in terms of a transitionbased algorithm, which is outlined in the next section. In the section after that, the semantics of the parameters that are used to turn this into a stochastic algorithm that generates pairs of trees and strings are discussed.
3 Topdown Parsing and Generation
RNNGs are based on a topdown generation algorithm that relies on a stack data structure of partially completed syntactic constituents. To emphasize the similarity of our algorithm to more familiar bottomup shiftreduce recognition algorithms, we first present the parsing (rather than generation) version of our algorithm (§3.1) and then present modifications to turn it into a generator (§3.2).
3.1 Parser Transitions
The parsing algorithm transforms a sequence of words into a parse tree using two data structures (a stack and an input buffer). As with the bottomup algorithm of Sagae and Lavie (2005), our algorithm begins with the stack () empty and the complete sequence of words in the input buffer (). The buffer contains unprocessed terminal symbols, and the stack contains terminal symbols, “open” nonterminal symbols, and completed constituents. At each timestep, one of the following three classes of operations (Fig. 1
) is selected by a classifier, based on the current contents on the stack and buffer:

introduces an “open nonterminal” X onto the top of the stack. Open nonterminals are written as a nonterminal symbol preceded by an open parenthesis, e.g., “(VP”, and they represent a nonterminal whose child nodes have not yet been fully constructed. Open nonterminals are “closed” to form complete constituents by subsequent reduce operations.

shift removes the terminal symbol from the front of the input buffer, and pushes it onto the top of the stack.

reduce repeatedly pops completed subtrees or terminal symbols from the stack until an open nonterminal is encountered, and then this open NT is popped and used as the label of a new constituent that has the popped subtrees as its children. This new completed constituent is pushed onto the stack as a single composite item. A single reduce operation can thus create constituents with an unbounded number of children.
The parsing algorithm terminates when there is a single completed constituent on the stack and the buffer is empty. Fig. 2 shows an example parse using our transition set. Note that in this paper we do not model preterminal symbols (i.e., partofspeech tags) and our examples therefore do not include them.^{3}^{3}3Preterminal symbols are, from the parsing algorithm’s point of view, just another kind of nonterminal symbol that requires no special handling. However, leaving them out reduces the number of transitions by and also reduces the number of action types, both of which reduce the runtime. Furthermore, standard parsing evaluation scores do not depend on preterminal prediction accuracy.
Our transition set is closely related to the operations used in Earley’s algorithm which likewise introduces nonterminals symbols with its predict operation and later completes them after consuming terminal symbols one at a time using scan (Earley, 1970). It is likewise closely related to the “linearized” parse trees proposed by Vinyals et al. (2015) and to the topdown, lefttoright decompositions of trees used in previous generative parsing and language modeling work (Roark, 2001, 2004; Charniak, 2010).
A further connection is to parsing which uses an unbounded lookahead (compactly represented by a DFA) to distinguish between parse alternatives in a topdown parser (Parr and Fisher, 2011); however, our parser uses an RNN encoding of the lookahead rather than a DFA.
Stack  Buffer  Open NTs  Action  Stack  Buffer  Open NTs 

shift  
reduce 
Input: The hungry cat meows .
Stack  Buffer  Action  

0  Thehungrycatmeows.  nt(S)  
1  (S  Thehungrycatmeows.  nt(NP) 
2  (S(NP  Thehungrycatmeows.  shift 
3  (S(NPThe  hungrycatmeows.  shift 
4  (S(NPThehungry  catmeows.  shift 
5  (S(NPThehungrycat  meows.  reduce 
6  (S(NP The hungry cat)  meows.  nt(VP) 
7  (S(NP The hungry cat)(VP  meows.  shift 
8  (S(NP The hungry cat)(VP meows  .  reduce 
9  (S(NP The hungry cat)(VP meows)  .  shift 
10  (S(NP The hungry cat)(VP meows).  reduce  
11  (S (NP The hungry cat) (VP meows) .) 
Stack  Terms  Open NTs  Action  Stack  Terms  Open NTs 

reduce 
Stack  Terminals  Action  

0  nt(S)  
1  (S  nt(NP)  
2  (S(NP  gen  
3  (S(NPThe  The  gen 
4  (S(NPThehungry  Thehungry  gen 
5  (S(NPThehungrycat  Thehungrycat  reduce 
6  (S(NP The hungry cat)  Thehungrycat  nt(VP) 
7  (S(NP The hungry cat)(VP  Thehungrycat  gen 
8  (S(NP The hungry cat)(VP meows  Thehungrycatmeows  reduce 
9  (S(NP The hungry cat)(VP meows)  Thehungrycatmeows  gen 
10  (S(NP The hungry cat)(VP meows).  Thehungrycatmeows.  reduce 
11  (S (NP The hungry cat) (VP meows) .)  Thehungrycatmeows. 
Constraints on parser transitions.
To guarantee that only wellformed phrasestructure trees are produced by the parser, we impose the following constraints on the transitions that can be applied at each step which are a function of the parser state where is the number of open nonterminals on the stack:

The operation can only be applied if is not empty and .^{4}^{4}4Since our parser allows unary nonterminal productions, there are an infinite number of valid trees for finitelength sentences. The constraint prevents the classifier from misbehaving and generating excessively large numbers of nonterminals. Similar constraints have been proposed to deal with the analogous problem in bottomup shiftreduce parsers (Sagae and Lavie, 2005).

The shift operation can only be applied if is not empty and .

The reduce operation can only be applied if the top of the stack is not an open nonterminal symbol.

The reduce operation can only be applied if or if the buffer is empty.
To designate the set of valid parser transitions, we write .
3.2 Generator Transitions
The parsing algorithm that maps from sequences of words to parse trees can be adapted with minor changes to produce an algorithm that stochastically generates trees and terminal symbols. Two changes are required: (i) there is no input buffer of unprocessed words, rather there is an output buffer (), and (ii) instead of a shift operation there are operations which generate terminal symbol and add it to the top of the stack and the output buffer. At each timestep an action is stochastically selected according to a conditional distribution that depends on the current contents of and . The algorithm terminates when a single completed constituent remains on the stack. Fig. 4 shows an example generation sequence.
Constraints on generator transitions.
The generation algorithm also requires slightly modified constraints. These are:

The operation can only be applied if .

The reduce operation can only be applied if the top of the stack is not an open nonterminal symbol and .
To designate the set of valid generator transitions, we write .
This transition set generates trees using nearly the same structure building actions and stack configurations as the “topdown PDA” construction proposed by Abney et al. (1999), albeit without the restriction that the trees be in Chomsky normal form.
3.3 Transition Sequences from Trees
Any parse tree can be converted to a sequence of transitions via a depthfirst, lefttoright traversal of a parse tree. Since there is a unique depthfirst, leftroright traversal of a tree, there is exactly one transition sequence of each tree. For a tree and a sequence of symbols , we write to indicate the corresponding sequence of generation transitions, and to indicate the parser transitions.
3.4 Runtime Analysis
A detailed analysis of the algorithmic properties of our topdown parser is beyond the scope of this paper; however, we briefly state several facts. Assuming the availability of constant time push and pop operations, the runtime is linear in the number of the nodes in the parse tree that is generated by the parser/generator (intuitively, this is true since although an individual reduce operation may require applying a number of pops that is linear in the number of input symbols, the total number of pop operations across an entire parse/generation run will also be linear). Since there is no way to bound the number of output nodes in a parse tree as a function of the number of input words, stating the runtime complexity of the parsing algorithm as a function of the input size requires further assumptions. Assuming our fixed constraint on maximum depth, it is linear.
3.5 Comparison to Other Models
Our generation algorithm algorithm differs from previous stackbased parsing/generation algorithms in two ways. First, it constructs rooted tree structures top down (rather than bottom up), and second, the transition operators are capable of directly generating arbitrary tree structures rather than, e.g., assuming binarized trees, as is the case in much prior work that has used transitionbased algorithms to produce phrasestructure trees (Sagae and Lavie, 2005; Zhang and Clark, 2011; Zhu et al., 2013).
4 Generative Model
RNNGs use the generator transition set just presented to define a joint distribution on syntax trees (
) and words (). This distribution is defined as a sequence model over generator transitions that is parameterized using a continuous space embedding of the algorithm state at each time step (); i.e.,and where actionspecific embeddings
and bias vector
are parameters in .The representation of the algorithm state at time , , is computed by combining the representation of the generator’s three data structures: the output buffer (), represented by an embedding , the stack (), represented by an embedding , and the history of actions () taken by the generator, represented by an embedding ,
where and are parameters. Refer to Figure 5 for an illustration of the architecture.
The output buffer, stack, and history are sequences that grow unboundedly, and to obtain representations of them we use recurrent neural networks to “encode” their contents (Cho et al., 2014). Since the output buffer and history of actions are only appended to and only contain symbols from a finite alphabet, it is straightforward to apply a standard RNN encoding architecture. The stack () is more complicated for two reasons. First, the elements of the stack are more complicated objects than symbols from a discrete alphabet: open nonterminals, terminals, and full trees, are all present on the stack. Second, it is manipulated using both push and pop operations. To efficiently obtain representations of under push and pop operations, we use stack LSTMs (Dyer et al., 2015). To represent complex parse trees, we define a new syntactic composition function that recursively defines representations of trees.
4.1 Syntactic Composition Function
When a reduce operation is executed, the parser pops a sequence of completed subtrees and/or tokens (together with their vector embeddings) from the stack and makes them children of the most recent open nonterminal on the stack, “completing” the constituent. To compute an embedding of this new subtree, we use a composition function based on bidirectional LSTMs, which is illustrated in Fig. 6.
The first vector read by the LSTM in both the forward and reverse directions is an embedding of the label on the constituent being constructed (in the figure, NP). This is followed by the embeddings of the child subtrees (or tokens) in forward or reverse order. Intuitively, this order serves to “notify” each LSTM what sort of head it should be looking for as it processes the child node embeddings. The final state of the forward and reverse LSTMs are concatenated, passed through an affine transformation and a nonlinearity to become the subtree embedding.^{5}^{5}5We found the many previously proposed syntactic composition functions inadequate for our purposes. First, we must contend with an unbounded number of children, and many previously proposed functions are limited to binary branching nodes (Socher et al., 2013b; Dyer et al., 2015). Second, those that could deal with ary nodes made poor use of nonterminal information (Tai et al., 2015), which is crucial for our task. Because each of the child node embeddings (, , in Fig. 6) is computed similarly (if it corresponds to an internal node), this composition function is a kind of recursive neural network.
4.2 Word Generation
To reduce the size of , word generation is broken into two parts. First, the decision to generate is made (by predicting gen as an action), and then choosing the word, conditional on the current parser state. To further reduce the computational complexity of modeling the generation of a word, we use a classfactored softmax (Baltescu and Blunsom, 2015; Goodman, 2001). By using classes for a vocabulary of size , this prediction step runs in time rather than the of the fullvocabulary softmax. To obtain clusters, we use the greedy agglomerative clustering algorithm of Brown et al. (1992).
4.3 Training
The parameters in the model are learned to maximize the likelihood of a corpus of trees.
4.4 Discriminative Parsing Model
A discriminative parsing model can be obtained by replacing the embedding of at each time step with an embedding of the input buffer . To train this model, the conditional likelihood of each sequence of actions given the input string is maximized.^{6}^{6}6For the discriminative parser, the POS tags are processed similarly as in (Dyer et al., 2015); they are predicted for English with the Stanford Tagger (Toutanova et al., 2003) and Chinese with Marmot (Mueller et al., 2013).
5 Inference via Importance Sampling
Our generative model defines a joint distribution on trees () and sequences of words (
). To evaluate this as a language model, it is necessary to compute the marginal probability
. And, to evaluate the model as a parser, we need to be able to find the MAP parse tree, i.e., the tree that maximizes. However, because of the unbounded dependencies across the sequence of parsing actions in our model, exactly solving either of these inference problems is intractable. To obtain estimates of these, we use a variant of importance sampling
(Doucet and Johansen, 2011).Our importance sampling algorithm uses a conditional proposal distribution with the following properties: (i) ; (ii) samples can be obtained efficiently; and (iii) the conditional probabilities of these samples are known. While many such distributions are available, the discriminatively trained variant of our parser (§4.4) fulfills these requirements: sequences of actions can be sampled using a simple ancestral sampling approach, and, since parse trees and action sequences exist in a onetoone relationship, the product of the action probabilities is the conditional probability of the parse tree under . We therefore use our discriminative parser as our proposal distribution.
Importance sampling uses importance weights, which we define as , to compute this estimate. Under this definition, we can derive the estimator as follows:
We now replace this expectation with its Monte Carlo estimate as follows, using samples from :
To obtain an estimate of the MAP tree , we choose the sampled tree with the highest probability under the joint model .
6 Experiments
We present results of our two models both on parsing (discriminative and generative) and as a language model (generative only) in English and Chinese.
Data.
For English, §2–21 of the Penn Treebank are used as training corpus for both, with §24 held out as validation, and §23 used for evaluation. Singleton words in the training corpus with unknown word classes using the the Berkeley parser’s mapping rules.^{7}^{7}7http://github.com/slavpetrov/berkeleyparser Orthographic case distinctions are preserved, and numbers (beyond singletons) are not normalized. For Chinese, we use the Penn Chinese Treebank Version 5.1 (CTB) (Xue et al., 2005).^{8}^{8}8§001–270 and 440–1151 for training, §301–325 development data, and §271–300 for evaluation. For the Chinese experiments, we use a single unknown word class. Corpus statistics are given in Table 1.^{9}^{9}9This preprocessing scheme is more similar to what is standard in parsing than what is standard in language modeling. However, since our model is both a parser and a language model, we opted for the parser normalization.
PTBtrain  PTBtest  CTBtrain  CTBtest  
Sequences  39,831  2,416  50,734  348 
Tokens  950,012  56,684  1,184,532  8,008 
Types  23,815  6,823  31,358  1,637 
UNKTypes  49  42  1  1 
Model and training parameters.
For the discriminative model, we used hidden dimensions of 128 and 2layer LSTMs (larger numbers of dimensions reduced validation set performance). For the generative model, we used 256 dimensions and 2layer LSTMs. For both models, we tuned the dropout rate to maximize validation set likelihood, obtaining optimal rates of 0.2 (discriminative) and 0.3 (generative). For the sequential LSTM baseline for the language model, we also found an optimal dropout rate of 0.3. For training we used stochastic gradient descent with a learning rate of 0.1. All parameters were initialized according to recommendations given by
Glorot and Bengio (2010).English parsing results.
Table 2 (last two rows) gives the performance of our parser on Section 23, as well as the performance of several representative models. For the discriminative model, we used a greedy decoding rule as opposed to beam search in some shiftreduce baselines. For the generative model, we obtained 100 independent samples from a flattened distribution of the discriminative parser (by exponentiating each probability by and renormalizing) and reranked them according to the generative model.^{10}^{10}10The value was chosen based on the diversity of the samples generated on the development set.
Model  type  F 

Vinyals et al. (2015) – WSJ only  D  88.3 
Henderson (2004)  D  89.4 
Socher et al. (2013a)  D  90.4 
Zhu et al. (2013)  D  90.4 
Petrov and Klein (2007)  G  90.1 
Bod (2003)  G  90.7 
Shindo et al. (2012) – single  G  91.1 
Shindo et al. (2012) – ensemble  G  92.4 
Zhu et al. (2013)  S  91.3 
McClosky et al. (2006)  S  92.1 
Vinyals et al. (2015) – single  S  92.1 
Discriminative,  D  89.8 
Generative,  G  92.4 
Chinese parsing results.
Chinese parsing results were obtained with the same methodology as in English and show the same pattern (Table 3).
Model  type  F 

Zhu et al. (2013)  D  82.6 
Wang et al. (2015)  D  83.2 
Huang and Harper (2009)  D  84.2 
Charniak (2000)  G  80.8 
Bikel (2004)  G  80.6 
Petrov and Klein (2007)  G  83.3 
Zhu et al. (2013)  S  85.6 
Wang and Xue (2014)  S  86.3 
Wang et al. (2015)  S  86.6 
Discriminative,  D  80.7 
Generative,  G  82.7 
Language model results.
We report heldout perword perplexities of three language models, both sequential and syntactic. Log probabilities are normalized by the number of words (excluding the stop symbol), inverted, and exponentiated to yield the perplexity. Results are summarized in Table 4.
Model  test ppl (PTB)  test ppl (CTB) 

IKN 5gram  169.3  255.2 
LSTM LM  113.4  207.3 
RNNG  102.4  171.9 
7 Discussion
It is clear from our experiments that the proposed generative model is quite effective both as a parser and as a language model. This is the result of (i) relaxing conventional independence assumptions (e.g., contextfreeness) and (ii) inferring continuous representations of symbols alongside nonlinear models of their syntactic relationships. The most significant question that remains is why the discriminative model—which has more information available to it than the generative model—performs worse than the generative model. This pattern has been observed before in neural parsing by Henderson (2004), who hypothesized that larger, unstructured conditioning contexts are harder to learn from, and provide opportunities to overfit. Our discriminative model conditions on the entire history, stack, and buffer, while our generative model only accesses the history and stack. The fully discriminative model of Vinyals et al. (2015) was able to obtain results similar to those of our generative model (albeit using much larger training sets obtained through semisupervision) but similar results to those of our discriminative parser using the same data. In light of their results, we believe Henderson’s hypothesis is correct, and that generative models should be considered as a more statistically efficient method for learning neural networks from small data.
8 Related Work
Our language model combines work from two modeling traditions: (i) recurrent neural network language models and (ii) syntactic language modeling. Recurrent neural network language models use RNNs to compute representations of an unbounded history of words in a lefttoright language model (Zaremba et al., 2015; Mikolov et al., 2010; Elman, 1990). Syntactic language models jointly generate a syntactic structure and a sequence of words (Baker, 1979; Jelinek and Lafferty, 1991). There is an extensive literature here, but one strand of work has emphasized a bottomup generation of the tree, using variants of shiftreduce parser actions to define the probability space (Chelba and Jelinek, 2000; Emami and Jelinek, 2005). The neuralnetwork–based model of Henderson (2004) is particularly similar to ours in using an unbounded history in a neural network architecture to parameterize generative parsing based on a leftcorner model. Dependencyonly language models have also been explored (Titov and Henderson, 2007; Buys and Blunsom, 2015a, b). Modeling generation topdown as a rooted branching process that recursively rewrites nonterminals has been explored by Charniak (2000) and Roark (2001). Of particular note is the work of Charniak (2010)
, which uses random forests and handengineered features over the entire syntactic derivation history to make decisions over the next action to take.
The neural networks we use to model sentences are structured according to the syntax of the sentence being generated. Syntactically structured neural architectures have been explored in a number of applications, including discriminative parsing (Socher et al., 2013a; Kiperwasser and Goldberg, 2016)
(Tai et al., 2015; Socher et al., 2013b), and sentence representation (Socher et al., 2011; Bowman et al., 2006). However, these models have been, without exception, discriminative; this is the first work to use syntactically structured neural models to generate language. Earlier work has demonstrated that sequential RNNs have the capacity to recognize contextfree (and beyond) languages (Sun et al., 1998; Siegelmann and Sontag, 1995). In contrast, our work may be understood as a way of incorporating a contextfree inductive bias into the model structure.9 Outlook
RNNGs can be combined with a particle filter inference scheme (rather than the importance sampling method based on a discriminative parser, §5) to produce a lefttoright marginalization algorithm that runs in expected linear time. Thus, they could be used in applications that require language models.
A second possibility is to replace the sequential generation architectures found in many neural network transduction problems that produce sentences conditioned on some input. Previous work in machine translation has showed that conditional syntactic models can function quite well without the computationally expensive marginalization process at decoding time (Galley et al., 2006; Gimpel and Smith, 2014).
A third consideration regarding how RNNGs, human sentence processing takes place in a lefttoright, incremental order. While an RNNG is not a processing model (it is a grammar), the fact that it is lefttoright opens up several possibilities for developing new sentence processing models based on an explicit grammars, similar to the processing model of Charniak (2010).
Finally, although we considered only the supervised learning scenario, RNNGs are joint models that could be trained without trees, for example, using expectation maximization.
10 Conclusion
We introduced recurrent neural network grammars, a probabilistic model of phrasestructure trees that can be trained generatively and used as a language model or a parser, and a corresponding discriminative model that can be used as a parser. Apart from outofvocabulary preprocessing, the approach requires no feature design or transformations to treebank data. The generative model outperforms every previously published parser built on a single supervised generative model in English, and a bit behind the bestreported generative model in Chinese. As language models, RNNGs outperform the best singlesentence language models.
Acknowledgments
We thank Brendan O’Connor, Swabha Swayamdipta, and Brian Roark for feedback on drafts of this paper, and Jan Buys, Phil Blunsom, and Yue Zhang for help with data preparation. This work was sponsored in part by the Defense Advanced Research Projects Agency (DARPA) Information Innovation Office (I2O) under the Low Resource Languages for Emergent Incidents (LORELEI) program issued by DARPA/I2O under Contract No. HR001115C0114; it was also supported in part by Contract No. W911NF1510543 with the DARPA and the Army Research Office (ARO). Approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. Miguel Ballesteros was supported by the European Commission under the contract numbers FP7ICT610411 (project MULTISENSOR) and H2020RIA645012 (project KRISTINA).
References
 Abney et al. (1999) Steven Abney, David McAllester, and Fernando Pereira. 1999. Relating probabilistic grammars and automata. In Proc. ACL.
 Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine transation by jointly learning to align and translate. In Proc. ICLR.
 Baker (1979) James K. Baker. 1979. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America, 65(S1):S132–S132.
 Baltescu and Blunsom (2015) Paul Baltescu and Phil Blunsom. 2015. Pragmatic neural modelling in machine translation. In Proc. NAACL.
 Bikel (2004) Dan Bikel. 2004. On the parameter space of generative lexicalized statistical parsing models. Ph.D. thesis, University of Pennsylvania.
 Bod (2003) Rens Bod. 2003. An efficient implementation of a new DOP model. In Proc. EACL.
 Bowman et al. (2006) Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2006. A fast unified model for parsing and sentence understanding. CoRR, abs/1603.06021.

Brown et al. (1992)
Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra,
and Jenifer C. Lai. 1992.
Classbased ngram models of natural language.
Computational Linguistics, 18(4):467–479.  Buys and Blunsom (2015a) Jan Buys and Phil Blunsom. 2015a. A Bayesian model for generative transitionbased dependency parsing. CoRR, abs/1506.04334.
 Buys and Blunsom (2015b) Jan Buys and Phil Blunsom. 2015b. Generative incremental dependency parsing with neural networks. In Proc. ACL.
 Charniak (2000) Eugene Charniak. 2000. A maximumentropyinspired parser. In Proc. NAACL.
 Charniak (2010) Eugene Charniak. 2010. Topdown nearlycontextsensitive parsing. In Proc. EMNLP.
 Chelba and Jelinek (2000) Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. Computer Speech and Language, 14(4).
 Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proc. EMNLP.
 Chomsky (1957) Noam Chomsky. 1957. Syntactic Structures. Mouton, The Hague/Paris.

Doucet and Johansen (2011)
Arnaud Doucet and Adam M. Johansen. 2011.
A tutorial on particle filtering and smoothing: Fifteen years later.
In
Handbook of Nonlinear Filtering
. Oxford. 
Dyer et al. (2015)
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith.
2015.
Transitionbased dependency parsing with stack long shortterm memory.
In Proc. ACL.  Earley (1970) Jay Earley. 1970. An efficient contextfree parsing algorithm. Communications of the ACM, 13(2):94–102.
 Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14:179–211.
 Emami and Jelinek (2005) Ahmad Emami and Frederick Jelinek. 2005. A neural syntactic language model. Machine Learning, 60:195–227.
 Galley et al. (2006) Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of contextrich syntactic translation models. In Proc. ACL.
 Gimpel and Smith (2014) Kevin Gimpel and Noah A. Smith. 2014. Phrase dependency machine translation with quasisynchronous treetotree features. Computational Linguistics, 40(2).
 Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. ICML.
 Goodman (2001) Joshua Goodman. 2001. Classes for fast maximum entropy training. CoRR, cs.CL/0108006.
 Henderson (2003) James Henderson. 2003. Inducing history representations for broad coverage statistical parsing. In Proc. NAACL.
 Henderson (2004) James Henderson. 2004. Discriminative training of a neural network statistical parser. In Proc. ACL.
 Huang and Harper (2009) Zhongqiang Huang and Mary Harper. 2009. Selftraining PCFG grammars with latent annotations across languages. In Proc. EMNLP.
 Jelinek and Lafferty (1991) Frederick Jelinek and John D. Lafferty. 1991. Computation of the probability of initial substring generation by stochastic contextfree grammars. Computational Linguistics, 17(3):315–323.
 Johnson (2001) Mark Johnson. 2001. Joint and conditional estimation of tagging and parsing models. In Proc. ACL.
 Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Easyfirst dependency parsing with hierarchical tree LSTMs. ArXiv:1603.00375.
 McClosky et al. (2006) David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective selftraining for parsing. In Proc. NAACL.
 Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proc. Interspeech.

Mueller et al. (2013)
Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013.
Efficient higherorder CRFs for morphological tagging.
In
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
, pages 322–332. Association for Computational Linguistics, Seattle, Washington, USA. URL http://www.aclweb.org/anthology/D131032.  Parr and Fisher (2011) Terence Parr and Kathleen Fisher. 2011. LL(*): The foundation of the ANTLR parser generator. In Proc. PLDI.
 Petrov and Klein (2007) Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proc. NAACL.
 Roark (2001) Brian Roark. 2001. Probabilistic topdown parsing and language modeling. Computational Linguistics, 27(2).
 Roark (2004) Brian Roark. 2004. Robust garden path parsing. JNLE, 10(1):1–24.
 Sagae and Lavie (2005) Kenji Sagae and Alon Lavie. 2005. A classifierbased parser with linear runtime complexity. In Proc. IWPT.
 Shindo et al. (2012) Hiroyuki Shindo, Yusuke Miyao, Akinori Fujino, and Masaaki Nagata. 2012. Bayesian symbolrefined tree substitution grammars for syntactic parsing. In Proc. ACL.
 Siegelmann and Sontag (1995) Hava T. Siegelmann and Eduardo D. Sontag. 1995. On the computational power of neural nets. Journal of Computer and System Sciences, 50.
 Socher et al. (2013a) Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013a. Parsing with compositional vectors. In Proc. ACL.

Socher et al. (2011)
Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and
Christopher D. Manning. 2011.
Dynamic pooling and unfolding recursive autoencoders for paraphrase detection.
In Proc. NIPS.  Socher et al. (2013b) Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
 Sun et al. (1998) GuoZheng Sun, C. Lee Giles, and HsingHen Chen. 1998. The neural network pushdown automaton: Architecture, dynamics and training. In Adaptive Processing of Sequences and Data Structures, volume 1387 of Lecture Notes in Computer Science, pages 296–345.
 Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from treestructured long shortterm memory networks. In Proc. ACL.
 Titov and Henderson (2007) Ivan Titov and James Henderson. 2007. A latent variable model for generative dependency parsing. In Proc. IWPT.
 Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Featurerich partofspeech tagging with a cyclic dependency network. In Proc. NAACL.
 Vinyals et al. (2015) Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Proc. ICLR.
 Wang et al. (2015) Zhiguo Wang, Haitao Mi, and Nianwen Xue. 2015. Feature optimization for constituent parsing via neural networks. In Proc. ACLIJCNLP.
 Wang and Xue (2014) Zhiguo Wang and Nianwen Xue. 2014. Joint POS tagging and transitionbased constituent parsing in Chinese with nonlocal features. In Proc. ACL.

Wen et al. (2015)
TsungHsien Wen, Milica Gašić, Nikola Mrkšić, PeiHao Su, David
Vandyke, and Steve Young. 2015.
Semantically conditioned LSTMbased natural language generation for spoken dialogue systems.
In Proc. EMNLP.  Xu et al. (2015) Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. ICML.
 Xue et al. (2005) Naiwen Xue, Fei Xia, Fudong Chiou, and Marta Palmer. 2005. The Penn Chinese TreeBank: Phrase structure annotation of a large corpus. Nat. Lang. Eng., 11(2).
 Zaremba et al. (2015) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In Proc. ICLR.

Zhang and Clark (2011)
Yue Zhang and Stephen Clark. 2011.
Syntactic processing using the generalized perceptron and beam search.
Computational Linguistics, 37(1).  Zhu et al. (2013) Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shiftreduce constituent parsing. In Proc. ACL.
RNNG Composition Function and Implementation Error
The composition function reduces a completed constituent into a single vector representation using a bidirectional LSTM (Figure 8) over embeddings of the constituent’s children as well as an embedding of the resulting nonterminal symbol type. The implementation error (Figure 8) composed the constituent (NP the hungry cat) by reading the sequence “NP the hungry NP”, that is, it discarded the rightmost child of every constituent and replaced it with a second copy of the constituent’s nonterminal symbol. This error occurs for every constituent and means crucial information is not properly propagated upwards in the tree.
Results after Correction
The implementation error affected both the generative and discriminative RNNGs.^{11}^{11}11The discriminative model can only be used for parsing and not for language modeling, since it only models . We summarize corrected English phrasestructure PTB §23 parsing result in Table 5, Chinese (CTB 5.1 §271–300) in Table 6 (achieving the the best reported result on both datasets), and English and Chinese language modeling perplexities in Table 7. The considerable improvement in parsing accuracy indicates that properly composing the constituent and propagating information upwards is crucial. Despite slightly higher language modeling perplexity on PTB §23, the fixed RNNG still outperforms a highly optimized sequential LSTM baseline.
Model  type  F 

Vinyals et al. (2015) – WSJ only  D  88.3 
Henderson (2004)  D  89.4 
Socher et al. (2013a)  D  90.4 
Zhu et al. (2013)  D  90.4 
Petrov and Klein (2007)  G  90.1 
Bod (2003)  G  90.7 
Shindo et al. (2012) – single  G  91.1 
Shindo et al. (2012) – ensemble  G  92.4 
Zhu et al. (2013)  S  91.3 
McClosky et al. (2006)  S  92.1 
Vinyals et al. (2015)  S  92.1 
Discriminative, – buggy  D  89.8 
Generative, – buggy  G  92.4 
Discriminative, – correct  D  91.7 
Generative, – correct  G  93.3 
Model  type  F 

Zhu et al. (2013)  D  82.6 
Wang et al. (2015)  D  83.2 
Huang and Harper (2009)  D  84.2 
Charniak (2000)  G  80.8 
Bikel (2004)  G  80.6 
Petrov and Klein (2007)  G  83.3 
Zhu et al. (2013)  S  85.6 
Wang and Xue (2014)  S  86.3 
Wang et al. (2015)  S  86.6 
Discriminative,  buggy  D  80.7 
Generative,  buggy  G  82.7 
Discriminative, – correct  D  84.6 
Generative, – correct  G  86.9 
Model  test ppl (PTB)  test ppl (CTB) 

IKN 5gram  169.3  255.2 
LSTM LM  113.4  207.3 
RNNG – buggy  102.4  171.9 
RNNG – correct  105.2  148.5 
Comments
There are no comments yet.