Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs

06/29/2016 ∙ by Swabha Swayamdipta, et al. ∙ Universitat Pompeu Fabra Carnegie Mellon University University of Washington 0

We present a transition-based parser that jointly produces syntactic and semantic dependencies. It learns a representation of the entire algorithm state, using stack long short-term memories. Our greedy inference algorithm has linear time, including feature extraction. On the CoNLL 2008--9 English shared tasks, we obtain the best published parsing performance among models that jointly learn syntax and semantics.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

We introduce a new joint syntactic and semantic dependency parser. Our parser draws from the algorithmic insights of the incremental structure building approach of Henderson:08, with two key differences. First, it learns representations for the parser’s entire algorithmic state, not just the top items on the stack or the most recent parser states; in fact, it uses no expert-crafted features at all. Second, it uses entirely greedy inference rather than beam search. We find that it outperforms all previous joint parsing models, including Henderson:08 and variants [Gesmundo et al.2009, Titov et al.2009, Henderson et al.2013] on the CoNLL 2008 and 2009 (English) shared tasks. Our parser’s multilingual results are comparable to the top systems at CoNLL 2009.

Joint models like ours have frequently been proposed as a way to avoid cascading errors in NLP pipelines; varying degrees of success have been attained for a range of joint syntactic-semantic analysis tasks [Sutton and McCallum2005, Henderson et al.2008, Toutanova et al.2008, Johansson2009, Lluís et al.2013, inter alia].

One reason pipelines often dominate is that they make available the complete syntactic parse tree, and arbitrarily-scoped syntactic features—such as the “path” between predicate and argument, proposed by Gildea:02a—for semantic analysis. Such features are a mainstay of high-performance semantic role labeling (SRL) systems [Roth and Woodsend2014, Lei et al.2015, FitzGerald et al.2015, Foland and Martin2015], but they are expensive to extract [Johansson2009, He et al.2013].

This study shows how recent advances in representation learning can bypass those expensive features, discovering cheap alternatives available during a greedy parsing procedure. The specific advance we employ is the stack LSTM [Dyer et al.2015]

, a neural network that continuously summarizes the contents of the stack data structures in which a transition-based parser’s state is conventionally encoded. Stack LSTMs were shown to obviate many features used in syntactic dependency parsing; here we find them to do the same for joint syntactic-semantic dependency parsing.

We believe this is an especially important finding for greedy models that cast parsing as a sequence of decisions made based on algorithmic state, where linguistic theory and researcher intuitions offer less guidance in feature design.

Our system’s performance does not match that of the top expert-crafted feature-based systems [Zhao et al.2009, Björkelund et al.2010, Roth and Woodsend2014, Lei et al.2015], systems which perform optimal decoding [Täckström et al.2015], or of systems that exploit additional, differently-annotated datasets [FitzGerald et al.2015]. Many advances in those systems are orthogonal to our model, and we expect future work to achieve further gains by integrating them.

Because our system is very fast— with an end-to-end runtime of seconds to parse the CoNLL 2009 English test data on a single core—we believe it will be useful in practical settings. Our open-source implementation has been released.111

all & are & expected & to & reopen & soon &

& & expect.01 & & reopen.01 & &

21sbj [edge unit distance=1.5ex]2root 23vc 34oprd 45im 56tmp [edge below]31A1 [edge below]34C-A1 [edge below, label style=below]56AM-TMP [edge below]51A1

Figure 1: Example of a joint parse. Syntactic dependencies are shown by arcs above the sentence and semantic dependencies below; predicates are marked in boldface. C- denotes continuation of argument A1. Correspondences between dependencies might be close (between expected and to) or not (between reopen and all).

2 Joint Syntactic and Semantic Dependency Parsing

We largely follow the transition-based, synchronized algorithm of Henderson:13 to predict joint parse structures. The input to the algorithm is a sentence annotated with part-of-speech tags. The output consists of a labeled syntactic dependency tree and a directed SRL graph, in which a subset of words in the sentence are selected as predicates, disambiguated to a sense, and linked by labeled, directed edges to their semantic arguments. Figure 1 shows an example.

2.1 Transition-Based Procedure

The two parses are constructed in a bottom-up fashion, incrementally processing words in the sentence from left to right. The state of the parsing algorithm at timestep is represented by three stack data structures: a syntactic stack , a semantic stack —each containing partially built structures—and a buffer of input words . Our algorithm also places partial syntactic and semantic parse structures onto the front of the buffer, so it is also implemented as a stack. Each arc in the output corresponds to a transition (or “action”) chosen based on the current state; every transition modifies the state by updating , , and to , , and , respectively. While each state may license several valid actions, each action has a deterministic effect on the state of the algorithm.

Initially, and are empty, and contains the input sentence with the first word at the front of and a special root symbol at the end.222This works better for the arc-eager algorithm [Ballesteros and Nivre2013], in contrast to Henderson:13, who initialized with root at the buffer front. Execution ends on iteration such that is empty and and contain only a single structure headed by root.

2.2 Transitions for Joint Parsing

There are separate sets of syntactic and semantic transitions; the former manipulate and , the latter and . All are formally defined in Table 1. The syntactic transitions are from the “arc-eager” algorithm of Nivre:08. They include:

  • S-Shift, which copies333Note that in the original arc-eager algorithm [Nivre2008], Shift and Right-Arc actions move the item on the buffer front to the stack, whereas we only copy it (to allow the semantic operations to have access to it). an item from the front of and pushes it on .

  • S-Reduce pops an item from .

  • S-Right creates a syntactic dependency. Let be the element at the top of and be the element at the front of . The new dependency has as head, as dependent, and label . is popped off , and the resulting structure, rooted at , is pushed on . Finally, is copied to the top of .

  • S-Left creates a syntactic dependency with label in the reverse direction as S-Right. The top of , , is popped. The front of , , is replaced by the new structure, rooted at .

The semantic transitions are similar, operating on the semantic stack.

  • M-Shift removes an item from the front of and pushes it on .

  • M-Reduce pops an item from .

  • M-Right creates a semantic dependency. Let be the element at the top of and , the front of . The new dependency has as head, as dependent, and label . is popped off , and the resulting structure, rooted at , is pushed on .

  • M-Left creates a semantic dependency with label in the reverse direction as M-Right. The buffer front, , is replaced by the new -rooted structure. remains unchanged.

Because SRL graphs allow a node to be a semantic argument of two parents—like all in the example in Figure 1M-Left and M-Right do not remove the dependent from the semantic stack and buffer respectively, unlike their syntactic equivalents, S-Left and S-Right. We use two other semantic transitions from Henderson:13 which have no syntactic analogues:

  • M-Swap swaps the top two items on , to allow for crossing semantic arcs.

  • M-Pred marks the item at the front of as a semantic predicate with the sense , and replaces it with the disambiguated predicate.

The CoNLL 2009 corpus introduces semantic self-dependencies where many nominal predicates (from NomBank) are marked as their own arguments; these account for 6.68% of all semantic arcs in the English corpus. An example involving an eventive noun is shown in Figure 2. We introduce a new semantic transition, not in Henderson:13, to handle such cases:

  • M-Self adds a dependency, with label between the item at the front of and itself. The result replaces the item at the front of .

Figure 2: Example of an SRL graph with an arc from predicate problem.01 to itself, filling the A2 role. Our Self(A2) transition allows recovering this semantic dependency.
Action Dependency
Table 1: Parser transitions along with the modifications to the stacks and the buffer resulting from each. Syntactic transitions are shown above, semantic below. Italic symbols denote symbolic representations of words and relations, and bold symbols indicate (learned) embeddings (§3.5

) of words and relations; each element in a stack or buffer includes both symbolic and vector representations, either atomic or recursive.

represents the set of syntactic transitions, and the set of semantic transitions.

Note that the syntactic and semantic transitions both operate on the same buffer, though they independently specify the syntax and semantics, respectively. In order to ensure that both syntactic and semantic parses are produced, the syntactic and semantic transitions are interleaved. Only syntactic transitions are considered until a transition is chosen that copies an item from the buffer front to the syntactic stack (either S-Shift or S-Right). The algorithm then switches to semantic transitions until a buffer-modifying transition is taken (M-Shift).444Had we moved the item at the buffer front during the syntactic transitions, it would have been unavailable for the semantic transitions, hence we only copy it. At this point, the buffer is modified and the algorithm returns to syntactic transitions. This implies that, for each word, its left-side syntactic dependencies are resolved before its left-side semantic dependencies. An example run of the algorithm is shown in Figure 3.

2.3 Constraints on Transitions

To ensure that the parser never enters an invalid state, the sequence of transitions is constrained, following Henderson:13. Actions that copy or move items from the buffer (S-Shift, S-Right and M-Shift) are forbidden when the buffer is empty. Actions that pop from a stack (S-Reduce and M-Reduce) are forbidden when that stack is empty. We disallow actions corresponding to the same dependency, or the same predicate to be repeated in the sequence. Repetitive M-Swap transitions are disallowed to avoid infinite swapping. Finally, as noted above, we restrict the parser to syntactic actions until it needs to shift an item from to , after which it can only execute semantic actions until it executes an M-Shift.

Asymptotic runtime complexity of this greedy algorithm is linear in the length of the input, following the analysis by Nivre:09.555The analysis in [Nivre2009] does not consider Swap actions. However, since we constrain the number of such actions, the linear time complexity of the algorithm stays intact.

Transition Dependency
[] [] [all, are, expected, to, reopen, soon, root]
S-Shift [all] [] [all, are, expected, to, reopen, soon, root]
M-Shift [all] [all] [are, expected, to, reopen, soon, root]
S-Left(sbj) [] [all] [are, expected, to, reopen, soon, root] all are
S-Shift [are] [all] [are, expected, to, reopen, soon, root]
M-Shift [are] [all, are] [expected, to, reopen, soon, root]
S-Right(vc) [are, expected] [all, are] [expected, to, reopen, soon, root] are expected
M-Pred(expect.01) [are, expected] [all, are] [expected, to, reopen, soon, root]
M-Reduce [are, expected] [all] [expected, to, reopen, soon, root]
M-Left(A1) [are, expected] [all] [expected, to, reopen, soon, root] all expect.01
M-Shift [are, expected] [all, expected] [to, reopen, soon, root]
***S-Right(oprd) [are, expected, to] [all, expected] [to, reopen, soon, root] expected to
M-Right(C-A1) [are, expected, to] [all, expected] [to, reopen, soon, root] expect.01 to
M-Reduce [are, expected, to] [all] [to, reopen, soon, root]
M-Shift [are, expected, to] [all, to] [reopen, soon, root]
S-Right(im) [are, expected, to, reopen] [all, to] [reopen, soon, root] to reopen
M-Pred(reopen.01) [are, expected, to, reopen] [all, to] [reopen, soon, root]
M-Reduce [are, expected, to, reopen] [all] [reopen, soon, root]
M-Left(A1) [are, expected, to, reopen] [all] [reopen, soon, root] all reopen.01
M-Reduce [are, expected, to, reopen] [] [reopen, soon, root]
M-Shift [are, expected, to, reopen] [reopen] [soon, root]
S-Right(tmp) [are, expected, to, reopen, soon] [reopen] [soon, root] reopen soon
M-Right(AM-TMP) [are, expected, to, reopen, soon] [reopen] [soon, root] reopen.01 soon
M-Reduce [are, expected, to, reopen, soon] [] [soon, root]
M-Shift [are, expected, to, reopen, soon] [soon] [root]
S-Reduce [are, expected, to, reopen] [soon] [root]
S-Reduce [are, expected, to] [soon] [root]
S-Reduce [are, expected] [soon] [root]
S-Reduce [are] [soon] [root]
S-Left(root) [] [soon] [root] are root
S-Shift [root] [soon] [root]
M-Reduce [root] [] [root]
M-Shift [root] [root] []
Figure 3: Joint parser transition sequence for the sentence in Figure 1, “all are expected to reopen soon.” Syntactic labels are in lower-case and semantic role labels are capitalized. *** marks the operation predicted in Figure 4.

3 Statistical Model

The transitions in §2

describe the execution paths our algorithm can take; like past work, we apply a statistical classifier to decide which transition to take at each timestep, given the current state. The novelty of our model is that it learns a finite-length vector representation of the entire joint parser’s state (

, , and ) in order to make this decision.

3.1 Stack Long Short-Term Memory (LSTM)

LSTMs are recurrent neural networks equipped with specialized memory components in addition to a hidden state 

[Hochreiter and Schmidhuber1997, Graves2013] to model sequences. Stack LSTMs [Dyer et al.2015] are LSTMs that allow for stack operations: query, push, and pop. A “stack pointer” is maintained which determines which cell in the LSTM provides the memory and hidden units when computing the new memory cell contents. Query provides a summary of the stack in a single fixed-length vector. Push adds an element to the top of the stack, resulting in a new summary. Pop, which does not correspond to a conventional LSTM operation, moves the stack pointer to the preceding timestep, resulting in a stack summary as it was before the popped item was observed. Implementation details [Dyer et al.2015, Goldberg2015] and code have been made publicly available.666

Using stack LSTMs, we construct a representation of the algorithm state by decomposing it into smaller pieces that are combined by recursive function evaluations (similar to the way a list is built by a concatenate operation that operates on a list and an element). This enables information that would be distant from the “top” of the stack to be carried forward, potentially helping the learner.

3.2 Stack LSTMs for Joint Parsing

Our algorithm employs four stack LSTMs, one each for the , , and data structures.Like Dyer:15, we use a fourth stack LSTM, , for the history of actions— is never popped from, only pushed to. Figure 4 illustrates the architecture. The algorithm’s state at timestep is encoded by the four vectors summarizing the four stack LSTMs, and this is the input to the classifier that chooses among the allowable transitions at that timestep.

Figure 4: Stack LSTM for joint parsing. The state illustrated corresponds to the ***-marked row in the example transition sequence in Fig. 3.

Let , , , and denote the summaries of , , , and , respectively. Let denote the allowed transitions given the current stacks and buffer. The parser state at time

is given by a rectified linear unit 

[Nair and Hinton2010] in vector :

where and are the parameters of the classifier. The transition selected at timestep is


where and are parameters for each transition type . Note that only allowed transitions are considered in the decision rule (see §2.3).

3.3 Composition Functions

To use stack LSTMs, we require vector representations of the elements that are stored in the stacks. Specifically, we require vector representations of atoms (words, possibly with part-of-speech tags) and parse fragments. Word vectors can be pretrained or learned directly; we consider a concatenation of both in our experiments; part-of-speech vectors are learned and concatenated to the same.

To obtain vector representations of parse fragments, we use neural networks which recursively compute representations of the complex structured output [Dyer et al.2015]. The tree structures here are always ternary trees, with each internal node’s three children including a head, a dependent, and a label. The vectors for leaves are word vectors and vectors corresponding to syntactic and semantic relation types.

The vector for an internal node is a squashed () affine transformation of its children’s vectors. For syntactic and semantic attachments, respectively, the composition function is:


where and are vectors corresponding to atomic words or composed parse fragments; and are learned vector representations for syntactic and semantic labels respectively. Syntactic and semantic parameters are separated (, and , , respectively).

Finally, for predicates, we use another recursive function to compose the word representation, with a learned representation for the dismabiguated sense of the predicate, :


where and are parameters of the model. Note that, because syntactic and semantic transitions are interleaved, the fragmented structures are a blend of syntactic and semantic compositions. Figure 5 shows an example.

Figure 5: Example of a joint parse tree fragment with vector representations shown at each node. The vectors are obtained by recursive composition of representations of head, dependent, and label vectors. Syntactic dependencies and labels are in green, semantic in blue.

3.4 Training

Training the classifier requires transforming each training instance (a joint parse) into a transition sequence, a deterministic operation under our transition set. Given a collection of algorithm states at time and correct classification decisions , we minimize the sum of log-loss terms, given (for one timestep) by:


with respect to the classifier and LSTM parameters. Note that the loss is differentiable with respect to the parameters; gradients are calculated using backpropagation. We apply stochastic gradient descent with dropout for all neural network parameters.

3.5 Pretrained Embeddings

Following Dyer:15, “structured skip-gram” embeddings [Ling et al.2015]

were used, trained on the English (AFP section), German, Spanish and Chinese Gigaword corpora, with a window of size 5; training was stopped after 5 epochs. For out-of-vocabulary words, a randomly initialized vector of the same dimension was used.

3.6 Predicate Sense Disambiguation

Predicate sense disambiguation is handled within the model (M-Pred transitions), but since senses are lexeme-specific, we need a way to handle unseen predicates at test time. When a predicate is encountered at test time that was not observed in training, our system constructs a predicate from the predicted lemma of the word at that position and defaults to the “01” sense, which is correct for 91.22% of predicates by type in the English CoNLL 2009 training data.

4 Experimental Setup

Our model is evaluated on the CoNLL shared tasks on joint syntactic and semantic dependency parsing in 2008 [Surdeanu et al.2008] and 2009 [Hajič et al.2009]. The standard training, development and test splits of all datasets were used. Per the shared task guidelines, automatically predicted POS tags and lemmas provided in the datasets were used for all experiments. As a preprocessing step, pseudo-projectivization of the syntactic trees [Nivre et al.2007] was used, which allowed an accurate conversion of even the non-projective syntactic trees into syntactic transitions. However, the oracle conversion of semantic parses into transitions is not perfect despite using the M-Swap action, due to the presence of multiple crossing arcs.777For of English sentences in the CoNLL 2009 English dataset, the transition sequence incorrectly encodes the gold-standard joint parse; details in Henderson:13.

The standard evaluation metrics include the syntactic labeled attachment score (LAS), the semantic

score on both in-domain (WSJ) and out-of-domain (Brown corpus) data, and their macro average (Macro ) to score joint systems. Because the task was defined somewhat differently in each year, each dataset is considered in turn.

4.1 CoNLL 2008

The CoNLL 2008 dataset contains annotations from the Penn Treebank [Marcus et al.1993], PropBank [Palmer et al.2005] and NomBank [Meyers et al.2004]. The shared task evaluated systems on predicate identification in addition to predicate sense disambiguation and SRL.

To identify predicates, we trained a zero-Markov order bidirectional LSTM two-class classifier. As input to the classifier, we use learned representations of word lemmas and POS tags. This model achieves an score of 91.43% on marking words as predicates (or not).


The input representation for a word consists of pretrained embeddings (size 100 for English, 80 for Chinese, 64 for German and Spanish), concatenated with additional learned word and POS tag embeddings (size 32 and 12, respectively). Learned embeddings for syntactic and semantic arc labels are of size 20 and predicates 100. Two-layer LSTMs with hidden state dimension 100 were used for each of the four stacks. The parser state and the composition function are of dimension 100. A dropout rate of 0.2 [Zaremba et al.2014] was used on all layers at training time, tuned on the development data from the set of values . The learned representations for actions are of size 100, similarly tuned from

. Other hyperparameters have been set intuitively; careful tuning is expected to yield improvements 

[Weiss et al.2015].

An initial learning rate of 0.1 for stochastic gradient descent was used and updated in every training epoch with a decay rate of 0.1 [Dyer et al.2015]. Training is stopped when the development performance does not improve for approximately 6–7 hours of elapsed time. Experiments were run on a single thread on a CPU, with memory requirements of up to 512 MB.

4.2 CoNLL 2009

Relative to the CoNLL 2008 task (above), the main change in 2009 is that predicates are pre-identified, and systems are only evaluated on predicate sense disambiguation (not identification). Hence, the bidirectional LSTM classifier is not used here. The preprocessing for projectivity, and the hyperparameter selection is the same as in §4.1.

In addition to the joint approach described in the preceding sections, we experiment here with several variants:


the set of syntactic transitions , the syntactic stack , and the syntactic composition function are discarded. As a result, the set of constraints on transitions is a subset of the full set of constraints in §2.3. Effectively, this model does not use any syntactic features, similar to Collobert:11 and Zhou:15. It provides a controlled test of the benefit of explicit syntax in a semantic parser.


all semantic transitions in , the semantic stack , and the semantic composition function are discarded. S-Shift and S-Right now move the item from the front of the buffer to the syntactic stack, instead of copying. The set of constraints on the transitions is again a subset of the full set of constraints. This model is an arc-eager variant of Dyer:15, and serves to check whether semantic parsing degrades syntactic performance.


the semantics parameters are trained using automatically predicted syntax from the syntax-only model. At test time, only semantic parses are predicted. This setup bears similarity to other approaches which pipeline syntax and semantics, extracting features from the syntactic parse to help SRL. However, unlike other approaches, this model does not offer the entire syntactic tree for feature extraction, since only the partial syntactic structures present on the syntactic stack (and potentially the buffer) are visible at a given timestep. This model helps show the effect of joint prediction.

5 Results and Discussion

CoNLL 2008 (Table 2)

Our joint model significantly outperforms the joint model of Henderson:08, from which our set of transitions is derived, showing the benefit of learning a representation for the entire algorithmic state. Several other joint learning models have been proposed [Lluís and Màrquez2008, Johansson2009, Titov et al.2009] for the same task; our joint model surpasses the performance of all these models. The best reported systems on the CoNLL 2008 task are due to Johansson:08, Che:08, Ciaramita:08 and Zhao:08, all of which pipeline syntax and semantics; our system’s semantic and overall performance is comparable to these. We fall behind only Johansson:08, whose success was attributed to carefully designed global SRL features integrated into a pipeline of classifiers, making them asymptotically slower.

Model LAS Sem. Macro
joint models:
Lluis:08 85.8 70.3 78.1
Henderson:08 87.6 73.1 80.5
Johansson:09 86.6 77.1 81.8
Titov:09 87.5 76.1 81.8
CoNLL 2008 best:
#3: Zhao:08 87.7 76.7 82.2
#2: Che:08 86.7 78.5 82.7
#2: Ciaramita:08 87.4 78.0 82.7
#1: J&N (2008) 89.3 81.6 85.5
Joint (this work) 89.1 80.5 84.9
Table 2: Joint parsers: comparison on the CoNLL 2008 test (WSJ+Brown) set.

CoNLL 2009 English (Table 3)

All of our models (Syntax-only, Semantics-only, Hybrid and Joint) improve over Gesmundo:09 and Henderson:13, demonstrating the benefit of our entire-parser-state representation learner compared to the more locally scoped model.

Given that syntax has consistently proven useful in SRL, we expected our Semantics-only model to underperform Hybrid and Joint, and it did. In the training domain, syntax and semantics benefit each other (Joint outperforms Hybrid). Out-of-domain (the Brown test set), the Hybrid pulls ahead, a sign that Joint overfits to WSJ. As a syntactic parser, our Syntax-only model performs slightly better than Dyer:15, who achieve LAS on this task. Joint parsing is very slightly better still.

The overall performance of Joint is on par with the other winning participants at the CoNLL 2009 shared task [Zhao et al.2009, Che et al.2009, Gesmundo et al.2009], falling behind only Zhao:09, who carefully designed language-specific features and used a series of pipelines for the joint task, resulting in an accurate but computationally expensive system.

State-of-the-art SRL systems (shown in the last block of Table 3) which use advances orthogonal to the contributions in this paper, perform better than our models. Many of these systems use expert-crafted features derived from full syntactic parses in a pipeline of classifiers followed by a global reranker [Björkelund et al.2009, Björkelund et al.2010, Roth and Woodsend2014]

; we have not used these features or reranking. Lei:15 use syntactic parses to obtain interaction features between predicates and their arguments and then compress feature representations using a low-rank tensor. Tackstrom:15 present an exact inference algorithm for SRL based on dynamic programming and their local and structured models make use of many syntactic features from a pipeline; our search procedure is greedy. Their algorithm is adopted by Fitzgerald:15 for inference in a model that jointly learns representations from a combination of PropBank and FrameNet annotations; we have not experimented with extra annotations.

Our system achieves an end-to-end runtime of seconds to parse the CoNLL 2009 English test set on a single core. This is almost 2.5 times faster than the pipeline model of Lei:15 ( seconds) on the same machine.888See; unlike other state-of-the-art systems, this one is publicly available.

Model LAS
CoNLL’09 best:
#3 G+ ’09 88.79 83.24 70.65 86.03
#2 C+ ’09 88.48 85.51 73.82 87.00
#1 Z+ ’09a 89.19 86.15 74.58 87.69
this work:
Syntax-only 89.83
Sem.-only 84.39 73.87
Hybrid 89.83 84.58 75.64 87.20
Joint 89.94 84.97 74.48 87.45
R&W ’14 86.34 75.90
L+ ’15 86.58 75.57
T+ ’15 87.30 75.50
F+ ’15 87.80 75.50
Table 3: Comparison on the CoNLL 2009 English test set. The first block presents results of other models evaluated for both syntax and semantics on the CoNLL 2009 task. The second block presents our models. The third block presents the best published models, each using its own syntactic preprocessing.

CoNLL 2009 Multilingual (Table 4)

We tested the joint model on the non-English CoNLL 2009 datasets, and the results demonstrate that it adapts easily—it is on par with the top three systems in most cases. We note that our Chinese parser relies on pretrained word embeddings for its superior performance; without them (not shown), it was on par with the others. Japanese is a small-data case (4,393 training examples), illustrating our model’s dependence on reasonably large training datasets.

We have not extended our model to incorporate morphological features, which are used by the systems to which we compare. Future work might incorporate morphological features where available; this could potentially improve performance, especially in highly inflective languages like Czech. An alternative might be to infer word-internal representations using character-based word embeddings, which was found beneficial for syntactic parsing [Ballesteros et al.2015].

Language #1 C+’09 #2 Z+ ’09a #3 G+ ’09 Joint
Catalan 81.84 83.01 82.66 82.40
Chinese 76.38 76.23 76.15 79.27
Czech 83.27 80.87 83.21 79.53
English 87.00 87.69 86.03 87.45
German 82.44 81.22 79.59 81.05
Japanese 85.65 85.28 84.91 80.91
Spanish 81.90 83.31 82.43 83.11
Average 82.64 82.52 82.14 81.96
Table 4: Comparison of macro scores on the multilingual CoNLL 2009 test set.

6 Related Work

Other approaches to joint modeling, not considered in our experiments, are notable. Lluis:13 propose a graph-based joint model using dual decomposition for agreement between syntax and semantics, but do not achieve competitive performance on the CoNLL 2009 task. Lewis:15 proposed an efficient joint model for CCG syntax and SRL, which performs better than a pipelined model. However, their training necessitates CCG annotation, ours does not. Moreover, their evaluation metric rewards semantic dependencies regardless of where they attach within the argument span given by a PropBank constituent, making direct comparison to our evaluation infeasible. Krishnamurthy:14 propose a joint CCG parsing and relation extraction model which improves over pipelines, but their task is different from ours. Li:10 also perform joint syntactic and semantic dependency parsing for Chinese, but do not report results on the CoNLL 2009 dataset.

There has also been an increased interest in models which use neural networks for SRL. Collobert:11 proposed models which perform many NLP tasks without hand-crafted features. Though they did not achieve the best results on the constituent-based SRL task [Carreras and Màrquez2005], their approach inspired Zhou:15, who achieved state-of-the-art results using deep bidirectional LSTMs. Our approach for dependency-based SRL is not directly comparable.

7 Conclusion

We presented an incremental, greedy parser for joint syntactic and semantic dependency parsing. Our model surpasses the performance of previous joint models on the CoNLL 2008 and 2009 English tasks, without using expert-crafted, expensive features of the full syntactic parse.


The authors thank Sam Thomson, Lingpeng Kong, Mark Yatskar, Eunsol Choi, George Mulcaire, and Luheng He, as well as the anonymous reviewers, for many useful comments. This research was supported in part by DARPA grant FA8750-12-2-0342 funded under the DEFT program and by the U.S. Army Research Office under grant number W911NF-10-1-0533. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the view of the U.S. Army Research Office or the U.S. Government. Miguel Ballesteros was supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).


  • [Ballesteros and Nivre2013] Miguel Ballesteros and Joakim Nivre. 2013. Going to the roots of dependency parsing. Computational Linguistics, 39(1):5–13.
  • [Ballesteros et al.2015] Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Proc. of EMNLP.
  • [Björkelund et al.2009] Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proc. of CoNLL.
  • [Björkelund et al.2010] Anders Björkelund, Bernd Bohnet, Love Hafdell, and Pierre Nugues. 2010. A high-performance syntactic and semantic dependency parser. In Proc. of COLING.
  • [Carreras and Màrquez2005] Xavier Carreras and Lluís Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proc. of CoNLL.
  • [Che et al.2008] Wanxiang Che, Zhenghua Li, Yuxuan Hu, Yongqiang Li, Bing Qin, Ting Liu, and Sheng Li. 2008. A cascaded syntactic and semantic dependency parsing system. In Proc. of CoNLL.
  • [Che et al.2009] Wanxiang Che, Zhenghua Li, Yongqiang Li, Yuhang Guo, Bing Qin, and Ting Liu. 2009. Multilingual dependency-based syntactic and semantic parsing. In Proc. of CoNLL.
  • [Ciaramita et al.2008] Massimiliano Ciaramita, Giuseppe Attardi, Felice Dell’Orletta, and Mihai Surdeanu. 2008. DeSRL: A linear-time semantic role labeling system. In Proc. of CoNLL.
  • [Collobert et al.2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch.

    Journal of Machine Learning Research

    , 12:2493–2537.
  • [Dyer et al.2015] Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proc. of ACL.
  • [FitzGerald et al.2015] Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labelling with neural network factors. In Proc. of EMNLP.
  • [Foland and Martin2015] William R. Foland and James Martin. 2015.

    Dependencybased semantic role labeling using convolutional neural networks.

    In Proc. of *SEM.
  • [Gesmundo et al.2009] Andrea Gesmundo, James Henderson, Paola Merlo, and Ivan Titov. 2009. A latent variable model of synchronous syntactic-semantic parsing for multiple languages. In Proc. of CoNLL.
  • [Gildea and Jurafsky2002] Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288.
  • [Goldberg2015] Yoav Goldberg. 2015. A primer on neural network models for natural language processing. arXiv:1510.00726.
  • [Graves2013] Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv:1308.0850.
  • [Hajič et al.2009]

    Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang.

    2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proc. of CoNLL.
  • [He et al.2013] He He, Hal Daumé III, and Jason Eisner. 2013.

    Dynamic feature selection for dependency parsing.

    In Proc. of EMNLP.
  • [Henderson et al.2008] James Henderson, Paola Merlo, Gabriele Musillo, and Ivan Titov. 2008. A latent variable model of synchronous parsing for syntactic and semantic dependencies. In Proc. of CoNLL.
  • [Henderson et al.2013] James Henderson, Paola Merlo, Ivan Titov, and Gabriele Musillo. 2013. Multi-lingual joint parsing of syntactic and semantic dependencies with a latent variable model. Computational Linguistics, 39(4):949–998.
  • [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780.
  • [Johansson and Nugues2008] Richard Johansson and Pierre Nugues. 2008. Dependency-based syntactic-semantic analysis with PropBank and NomBank. In Proc. of CoNLL.
  • [Johansson2009] Richard Johansson. 2009. Statistical bistratal dependency parsing. In Proc. of EMNLP.
  • [Krishnamurthy and Mitchell2014] Jayant Krishnamurthy and Tom M. Mitchell. 2014. Joint syntactic and semantic parsing with combinatory categorial grammar. In Proc. of ACL.
  • [Lei et al.2015] Tao Lei, Yuan Zhang, Lluís Màrquez i Villodre, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In Proc. of NAACL.
  • [Lewis et al.2015] Mike Lewis, Luheng He, and Luke Zettlemoyer. 2015. Joint A* CCG parsing and semantic role labelling. In Proc. of EMNLP.
  • [Li et al.2010] Junhui Li, Guodong Zhou, and Hwee Tou Ng. 2010. Joint syntactic and semantic parsing of Chinese. In Proc. of ACL.
  • [Ling et al.2015] Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In Proc. of NAACL.
  • [Lluís and Màrquez2008] Xavier Lluís and Lluís Màrquez. 2008. A joint model for parsing syntactic and semantic dependencies. In Proc. of CoNLL.
  • [Lluís et al.2013] Xavier Lluís, Xavier Carreras, and Lluís Màrquez. 2013. Joint arc-factored parsing of syntactic and semantic dependencies. Transactions of the ACL, 1:219–230.
  • [Marcus et al.1993] Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19(2):313–330.
  • [Meyers et al.2004] Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The NomBank project: An interim report. In Proc. of NAACL.
  • [Nair and Hinton2010] Vinod Nair and Geoffrey E. Hinton. 2010.

    Rectified linear units improve restricted Boltzmann machines.

    In Proc. of ICML.
  • [Nivre et al.2007] Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13:95–135.
  • [Nivre2008] Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553.
  • [Nivre2009] Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proc. of ACL.
  • [Palmer et al.2005] Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106.
  • [Roth and Woodsend2014] Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In Proc. of EMNLP.
  • [Surdeanu et al.2008] Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proc. of CoNLL.
  • [Sutton and McCallum2005] Charles Sutton and Andrew McCallum. 2005. Joint parsing and semantic role labeling. In Proc. of CoNLL.
  • [Täckström et al.2015] Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. Transactions of the ACL, 3:29–41.
  • [Titov et al.2009] Ivan Titov, James Henderson, Paola Merlo, and Gabriele Musillo. 2009. Online graph planarisation for synchronous parsing of semantic and syntactic dependencies. In Proc. of IJCAI.
  • [Toutanova et al.2008] Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34(2):161–191.
  • [Weiss et al.2015] David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In Proc. of ACL.
  • [Zaremba et al.2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv:1409.2329.
  • [Zhao and Kit2008] Hai Zhao and Chunyu Kit. 2008. Parsing syntactic and semantic dependencies with two single-stage maximum entropy models. In Proc. of CoNLL.
  • [Zhao et al.2009] Hai Zhao, Wenliang Chen, Jun’ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proc. of CoNLL.
  • [Zhou and Xu2015] Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proc. of ACL.