Constituency Chomsky (1956) and dependency grammars Mel’cuk (1988); Kübler et al. (2009) are the two main abstractions for representing the syntactic structure of a given sentence, and each of them has its own particularities (Kahane and Mazziotta, 2015). While in constituency parsing the structure of sentences is abstracted as a phrase-structure tree (see Figure 0(a)), in dependency parsing the tree encodes binary syntactic relations between pairs of words (see Figure 0(b)).
When it comes to developing natural language processing (nlp) parsers, these two tasks are usually considered as disjoint tasks, and their improvements therefore have been obtained separately Charniak (2000); Nivre (2003); Kiperwasser and Goldberg (2016); Dozat and Manning (2017); Ma et al. (2018); Kitaev and Klein (2018).
Despite the potential benefits of learning across representations, there have been few attempts in the literature to do this. klein2003fast considered a factored model that provides separate methods for phrase-structure and lexical dependency trees and combined them to obtain optimal parses. With a similar aim, ren2013combine first compute the n
best constituency trees using a probabilistic context-free grammar, convert those into dependency trees using a dependency model, compute a probability score for each of them, and finally rerank the most plausible trees based on both scores. However, these methods are complex and intended for statistical parsers. Instead, we propose a extremely simple framework to learn across constituency and dependency representations.
(i) We use sequence labeling for constituency Gómez-Rodríguez and Vilares (2018) and dependency parsing (Strzyz et al., 2019) combined with multi-task learning (mtl) Caruana (1997) to learn across syntactic representations. To do so, we take a parsing paradigm (constituency or dependency parsing) as an auxiliary task to help train a model for the other parsing representation, a simple technique that translates into consistent improvements across the board. (ii) We also show that a single mtl model following this strategy can robustly produce both constituency and dependency trees, obtaining a performance and speed comparable with previous sequence labeling models for (either) constituency or dependency parsing. The source code is available at https://github.com/mstrise/seq2label-crossrep
2 Parsing as Sequence Labeling
to denote an input sentence. We use bold style lower-cased and math style upper-cased characters to refer to vectors and matrices (e.g.and ).
Sequence labeling is a structured prediction task where each token in the input sentence is mapped to a label Rei and Søgaard (2018). Many nlp
tasks suit this setup, including part-of-speech tagging, named-entity recognition or chunkingSang and Buchholz (2000); Toutanova and Manning (2000); Tjong Kim Sang and De Meulder (2003). More recently, syntactic tasks such as constituency parsing and dependency parsing have been successfully reduced to sequence labeling Spoustová and Spousta (2010); Li et al. (2018); Gómez-Rodríguez and Vilares (2018); Strzyz et al. (2019). Such models compute a tree representation of an input sentence using tagging actions.
We will also cast parsing as sequence labeling, to then learn across representations using multi-task learning. Two are the main advantages of this approach: (i) it does not require an explicit parsing algorithm nor explicit parsing structures, and (ii) it massively simplifies joint syntactic modeling. We now describe parsing as sequence labeling and the architecture used in this work.
Constituency parsing as tagging
GomVilEMNLP2018 define a linearization method to transform a phrase-structure tree into a discrete sequence of labels of the same length as the input sentence. Each label is a three tuple where: is an integer that encodes the number of ancestors in the tree shared between a word and its next one (computed as relative variation with respect to ), is the non-terminal symbol shared at the lowest level in common between said pair of words, and (optional) is a leaf unary chain that connects to . Figure 0(a) illustrates the encoding with an example.111In this work we do not use the dual encoding by VilaresMTL2019, which combines the relative encoding with a top-down absolute scale to represent certain relations.
Dependency parsing as tagging
Strzyz et al. (2019) also propose a linearization method to transform a dependency tree into a discrete sequence of labels. Each label is also represented as a three tuple . If , ’s head is the th closest word with PoS tag to the right of . If , the head is the th closest word to the left of that has as a PoS tag . The element represents the syntactic relation between the head and the dependent terms. Figure 0(b) depictures it with an example.
Tagging with lstms
We use bidirectional lstms (bilstms) to train our models Hochreiter and Schmidhuber (1997); Schuster and Paliwal (1997). Briefly, let be an abstraction of a lstm that processes the input from left to right, and let be another lstm processing the input in the opposite direction, the output of a bilstm at a timestep is computed as: . Then, is further processed by a feed-forward layer to compute the output label, i.e. . To optimize the model, we minimize the categorical cross-entropy loss, i.e. . In Appendix A we detail additional hyperpameters of the network. In this work we use NCRFpp Yang and Zhang (2018) as our sequence labeling framework.
3 Learning across representations
To learn across representations we cast the problem as multi-task learning. mtl enables learning many tasks jointly, encapsulating them in a single model and leveraging their shared representation (Caruana, 1997; Ruder, 2017). In particular, we will use a hard-sharing architecture: the sentence is first processed by stacked bilstms shared across all tasks, with a task-dependent feed-forward network on the top of it, to compute each task’s outputs. In particular, to benefit from a specific parsing abstraction we will be using the concept of auxiliary tasks Plank et al. (2016); Bingel and Søgaard (2017); Coavoux and Crabbé (2017), where tasks are learned together with the main task in the mtl setup even if they are not of actual interest by themselves, as they might help to find out hidden patterns in the data and lead to better generalization of the model.222Auxiliary losses are usually given less importance during the training process. For instance, Hershcovich et al. (2018) have shown that semantic parsing benefits from that approach.
The input is the same for both types of parsing and the same number of timesteps are required to compute a tree (equal to the length of the sentence), which simplifies the joint modeling. In this work, we focus on parallel data (we train on the same sentences labeled for both constituency and dependency abstractions). In the future, we plan to explore the idea of exploiting joint training over disjoint treebanks Barrett et al. (2018).
3.1 Baselines and models
We test different sequence labeling parsers to determine whether there are any benefits in learning across representations. We compare: (i) a single-task model for constituency parsing and another one for dependency parsing, (ii) a multi-task model for constituency parsing (and another for dependency parsing) where each element of the 3-tuple is predicted as a partial label in a separate subtask instead of as a whole, (iii) different mtl models where the partial labels from a specific parsing abstraction are used as auxiliary tasks for the other one, and (iv) an mtl model that learns to produce both abstractions as main tasks.
Single-paradigm, single-task models (s-s)
For constituency parsing, we use the single-task model by Gómez-Rodríguez and Vilares (2018). The input is the raw sentence and the output for each token a single label of the form =. For dependency parsing we use the model by Strzyz et al. (2019) to predict a single dependency label of the form = for each token.
Single-paradigm, multi-task models (s-mtl)
For constituency parsing, instead of predicting a single label output of the form , we generate three partial and separate labels , and through three task-dependent feed-forward networks on the top of the stacked bilstms. This is similar to Vilares et al. (2019). For dependency parsing, we propose in this work a mtl version too. We observed in preliminary experiments, as shown in Table 1, that casting the problem as 3-task learning led to worse results. Instead, we cast it as a 2-task learning problem, where the first task consists in predicting the head of a word , i.e. predicting the tuple , and the second task predicts the type of the relation . The loss is here computed as =, where is the partial loss coming from the subtask .
Double-paradigm, multi-task models with auxiliary losses (d-mtl-aux)
We predict the partial labels from one of the parsing abstractions as main tasks. The partial labels from the other parsing paradigm are used as auxiliary tasks. The loss is computed as =, where is an auxiliary loss and its specific weighting factor. Figure 2 shows the architecture used in this and the following multi-paradigm model.
Double paradigm, multi-task models (d-mtl)
All tasks are learned as main tasks instead.
In the following experiments we use two parallel datasets that provide syntactic analyses for both dependency and constituency parsing.
We also use the spmrl datasets, a collection of parallel dependency and constituency treebanks for morphologically rich languages Seddah et al. (2014). In this case, we use the predicted PoS tags provided by the organizers. We observed some differences between the constituency and dependency predicted input features provided with the corpora. For experiments where dependency parsing is the main task, we use the input from the dependency file, and the converse for constituency, for comparability with other work. d-mtl models were trained twice (one for each input), and dependency and constituent scores are reported on the model trained on the corresponding input.
We use bracketing F-score from the originalevalb and eval_spmrl official scripts to evaluate constituency trees. For dependency parsing, we rely on las and uas scores where punctuation is excluded in order to provide a homogeneous setup for PTB and SPMRL.
Table 2 compares single-paradigm models against their double-paradigm mtl versions. On average, mtl models with auxiliary losses achieve the best performance for both parsing abstractions. They gain F1 points on average in comparison with the single model for constituency parsing, and uas and las points for dependency parsing. In comparison to the single-paradigm MTL models, the average gain is smaller: 0.05 f1 points for constituency parsing, and 0.09 uas and 0.21 las points for dependency parsing.
mtl models that use auxiliary tasks (d-mtl-aux) consistently outperform the single-task models (s-s) in all datasets, both for constituency parsing and for dependency parsing in terms of uas. However, this does not extend to las. This different behavior between uas and las seems to be originated by the fact that 2-task dependency parsing models, which are the basis for the corresponding auxiliary task and mtl models, improve uas but not las with respect to single-task dependency parsing models. The reason might be that the single-task setup excludes unlikely combinations of dependency labels with PoS tags or dependency directions that are not found in the training set, while in the 2-task setup, both components are treated separately, which may be having a negative influence on dependency labeling accuracy.
In general, one can observe different range of gains of the models across languages. In terms of uas, the differences between single-task and mtl models span between (Basque) and (Hebrew); for las, and (both for Hebrew); and for F1, (Hebrew) and (Korean). Since the sequence labeling encoding used for dependency parsing heavily relies on PoS tags, the result for a given language can be dependent on the degree of the granularity of its PoS tags.
In addition, Table 3 provides a comparison of the d-mtl-aux models for dependency and constituency parsing against existing models on the PTB test set. Tables 4 and 5 shows the results for various existing models on the SPMRL test sets.333Note that we provide these SPMRL results for merely informative purposes. While they are the best existing results to our knowledge in these datasets, not all are directly comparable to ours (due to not all of them using the same kinds of information, e.g. some models do not use morphological features). Also, there are not many recent results for dependency parsing on the SPMRL datasets, probably due to the popularity of UD corpora. For comparison, we have included punctuation for this evaluation.
Table 6 shows the speeds (sentences/second) on a single core of a CPU444Intel Core i7-7700 CPU 4.2 GHz.. The d-mtl setup comes at almost no added computational cost, so the very good speed-accuracy tradeoff already provided by the single-task models is improved.
We have described a framework to leverage the complementary nature of constituency and dependency parsing. It combines multi-task learning, auxiliary tasks, and sequence labeling parsing, so that constituency and dependency parsing can benefit each other through learning across their representations. We have shown that mtl models with auxiliary losses outperform single-task models, and mtl models that treat both constituency and dependency parsing as main tasks obtain strong results, coming almost at no cost in terms of speed. Source code will be released upon acceptance.
This work has received funding from the European Research Council (ERC), under the European Union’s Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150), from the ANSWER-ASAP project (TIN2017-85160-C2-1-R) from MINECO, and from Xunta de Galicia (ED431B 2017/01).
Miguel Ballesteros. 2013.
Effective morphological feature selection with maltoptimizer at the spmrl 2013 shared task.In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 63–70.
- Ballesteros et al. (2015) Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 349–359, Lisbon, Portugal. Association for Computational Linguistics.
- Barrett et al. (2018) Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 302–312.
- Bingel and Søgaard (2017) Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. CoRR, abs/1702.08303.
- Björkelund et al. (2013) Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (re) ranking meets morphosyntax: State-of-the-art results from the spmrl 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 135–145.
- Caruana (1997) Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75.
- Charniak (2000) Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pages 132–139. Association for Computational Linguistics.
- Chen and Manning (2014) Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics.
- Chomsky (1956) Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113–124.
- Coavoux and Crabbé (2016) Maximin Coavoux and Benoit Crabbé. 2016. Neural greedy constituent parsing with dynamic oracles. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 172–182.
- Coavoux and Crabbé (2017) Maximin Coavoux and Benoît Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), volume 2, pages 331–336. Association for Computational Linguistics.
- De La Clergerie (2013) Eric De La Clergerie. 2013. Exploring beam-based shift-reduce dependency parsing with dyalog: Results from the spmrl 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 53–62.
- De Marneffe et al. (2006) Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In Lrec, volume 6, pages 449–454.
- Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
- Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computational Linguistics.
- Fernández-González and Gómez-Rodríguez (2019) Daniel Fernández-González and Carlos Gómez-Rodríguez. 2019. Left-to-right dependency parsing with pointer networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), page to appear, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
- Fernández-González and Martins (2015) Daniel Fernández-González and André F. T. Martins. 2015. Parsing as reduction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1523–1533, Beijing, China. Association for Computational Linguistics.
- Gómez-Rodríguez and Vilares (2018) Carlos Gómez-Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1314–1324. Association for Computational Linguistics.
- Hershcovich et al. (2018) Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018. Multitask parsing across semantic representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 373–385.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
- Kahane and Mazziotta (2015) Sylvain Kahane and Nicolas Mazziotta. 2015. Syntactic polygraphs. a formalism extending both constituency and dependency. In Mathematics of Language.
- Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association for Computational Linguistics, 4:313–327.
- Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2676–2686.
- Klein and Manning (2003) Dan Klein and Christopher D Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in neural information processing systems, pages 3–10.
- Kübler et al. (2009) Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. Synthesis Lectures on Human Language Technologies, 1(1):1–127.
- Li et al. (2018) Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3203–3214, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Ma et al. (2018) Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1403–1414.
- Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330.
- Mel’cuk (1988) Igor Aleksandrovic Mel’cuk. 1988. Dependency syntax: theory and practice. SUNY press.
- Nivre (2003) Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proceedings of the Eighth International Workshop on Parsing Technologies (IWPT, pages 149–160, Nancy, France.
- Nivre et al. (2007) Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülşen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135.
- Plank et al. (2016) Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In The 54th Annual Meeting of the Association for Computational Linguistics, page 412.
- Rei and Søgaard (2018) Marek Rei and Anders Søgaard. 2018. Zero-shot sequence labeling: Transferring knowledge from sentences to tokens. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 293–302.
Ren et al. (2013)
Xiaona Ren, Xiao Chen, and Chunyu Kit. 2013.
Combine constituent and dependency parsing via reranking.
Twenty-Third International Joint Conference on Artificial Intelligence.
- Ruder (2017) Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098.
- Sang and Buchholz (2000) Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
- Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681.
- Seddah et al. (2014) Djamé Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the spmrl 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103–109.
- Spoustová and Spousta (2010) Drahomíra Spoustová and Miroslav Spousta. 2010. Dependency parsing as a sequence labeling task. The Prague Bulletin of Mathematical Linguistics, 94(1):7–14.
- Strzyz et al. (2019) Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Viable dependency parsing as sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), page to appear, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
- Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
- Toutanova and Manning (2000) Kristina Toutanova and Christopher D Manning. 2000. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics-Volume 13, pages 63–70. Association for Computational Linguistics.
- Vilares et al. (2019) David Vilares, Mostafa Abdou, and Anders Søgaard. 2019. Better, faster, stronger sequence tagging constituent parsers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), page to appear, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
- Vinyals et al. (2015) Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in neural information processing systems, pages 2773–2781.
- Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018. Ncrf++: An open-source neural sequence labeling toolkit. Proceedings of ACL 2018, System Demonstrations, pages 74–79.
- Zhu et al. (2013) Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 434–443.
Appendix A Model parameters
The models were trained up to 150 iterations and optimized with Stochastic Gradient Descent (SGD) with a batch size of 8. The best model for constituency parsing was chosen with the highest achievedF1 score on the development set during the training and for dependency parsing with the highest las
score. The best double paradigm, multi-task model was chosen based on the highest harmonic mean amonglas and F1 scores.