A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

03/07/2016 ∙ by Yangfeng Ji, et al. ∙ Georgia Institute of Technology 0

This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Natural language processing (NLP) has recently experienced a neural network “tsunami” [Manning2016]

. A key advantage of these neural architectures is that they employ discriminatively-trained distributed representations, which can capture the meaning of linguistic phenomena ranging from individual words 

[Turian et al.2010] to longer-range linguistic contexts at the sentence level [Socher et al.2013] and beyond [Le and Mikolov2014]. Because they are discriminatively trained, these methods can learn representations that yield very accurate predictive models (e.g., Dyer et al, 2015).

However, in comparison with the probabilistic graphical models that were previously the dominant machine learning approach for NLP, neural architectures lack flexibility. By treating linguistic annotations as random variables, probabilistic graphical models can marginalize over annotations that are unavailable at test or training time, elegantly modeling multiple linguistic phenomena in a joint framework 

[Finkel et al.2006]. But because these graphical models represent uncertainty for every element in the model, adding too many layers of latent variables makes them difficult to train.

In this paper, we present a hybrid architecture that combines a recurrent neural network language model with a latent variable model over shallow discourse structure. In this way, the model learns a discriminatively-trained distributed representation of the local contextual features that drive word choice at the intra-sentence level, using techniques that are now state-of-the-art in language modeling [Mikolov et al.2010]

. However, the model treats shallow discourse structure — specifically, the relationships between pairs of adjacent sentences — as a latent variable. As a result, the model can act as both a discourse relation classifier and a language model. Specifically:

  • [itemsep=0pt]

  • If trained to maximize the conditional likelihood of the discourse relations, it outperforms state-of-the-art methods for both implicit discourse relation classification in the Penn Discourse Treebank [Rutherford and Xue2015] and dialog act classification in Switchboard [Kalchbrenner and Blunsom2013]. The model learns from both the discourse annotations as well as the language modeling objective, unlike previous recursive neural architectures that learn only from annotated discourse relations [Ji and Eisenstein2015].

  • If the model is trained to maximize the joint likelihood of the discourse relations and the text, it is possible to marginalize over discourse relations at test time, outperforming language models that do not account for discourse structure.

In contrast to recent work on continuous latent variables in recurrent neural networks [Chung et al.2015]

, which require complex variational autoencoders to represent uncertainty over the latent variables, our model is simple to implement and train, requiring only minimal modifications to existing recurrent neural network architectures that are implemented in commonly-used toolkits such as Theano, Torch, and CNN.

We focus on a class of shallow discourse relations, which hold between pairs of adjacent sentences (or utterances). These relations describe how the adjacent sentences are related: for example, they may be in contrast, or the latter sentence may offer an answer to a question posed by the previous sentence. Shallow relations do not capture the full range of discourse phenomena [Webber et al.2012], but they account for two well-known problems: implicit discourse relation classification in the Penn Discourse Treebank, which was the 2015 CoNLL shared task [Xue et al.2015]; and dialog act classification, which characterizes the structure of interpersonal communication in the Switchboard corpus [Stolcke et al.2000], and is a key component of contemporary dialog systems [Williams and Young2007]. Our model outperforms state-of-the-art alternatives for implicit discourse relation classification in the Penn Discourse Treebank, and for dialog act classification in the Switchboard corpus.

2 Background

Our model scaffolds on recurrent neural network (RNN) language models [Mikolov et al.2010], and recent variants that exploit multiple levels of linguistic detail [Ji et al.2015, Lin et al.2015].

RNN Language Models

Let us denote token in a sentence by , and write to indicate the sequence of words in sentence

. In an RNN language model, the probability of the sentence is decomposed as,

(1)

where the probability of each word is conditioned on the entire preceding sequence of words through the summary vector . This vector is computed recurrently from and from the embedding of the current word, , where and is the dimensionality of the word embeddings. The language model can then be summarized as,

(2)
(3)

where the matrix defines the output embeddings, and is an offset. The function

is a deterministic non-linear transition function. It typically takes an element-wise non-linear transformation (e.g.,

) of a vector resulting from the sum of the word embedding and a linear transformation of the previous hidden state.

The model as described thus far is identical to the recurrent neural network language model (RNNLM) of mikolov2010recurrent. In this paper, we replace the above simple hidden state units with the more complex Long Short-Term Memory units 

[Hochreiter and Schmidhuber1997], which have consistently been shown to yield much stronger performance in language modeling [Pham et al.2014]. For simplicity, we still use the term RNNLM in referring to this model.

Figure 1: A fragment of our model with latent variable , which only illustrates discourse information flow from sentence to . The information from sentence affects the distribution of and then the words prediction within sentence .
(4)
Figure 2: Per-token generative probabilities in the discourse relation language model

Document Context Language Model

One drawback of the RNNLM is that it cannot propagate long-range information between the sentences. Even if we remove sentence boundaries, long-range information will be attenuated by repeated application of the non-linear transition function. ji2015document propose the Document Context Language Model (DCLM) to address this issue. The core idea is to represent context with two vectors: , representing intra-sentence word-level context, and , representing inter-sentence context. These two vectors are then linearly combined in the generation function for word ,

(5)

where is set to the last hidden state of the previous sentence. ji2015document show that this model can improve language model perplexity.

3 Discourse Relation Language Models

We now present a probabilistic neural model over sequences of words and shallow discourse relations. Discourse relations are treated as latent variables, which are linked with a recurrent neural network over words in a latent variable recurrent neural network [Chung et al.2015].

3.1 The Model

Our model (see Figure 1) is formulated as a two-step generative story. In the first step, context information from the sentence is used to generate the discourse relation between sentences and ,

(6)

where is a random variable capturing the discourse relation between the two sentences, and is a vector summary of the contextual information from sentence , just as in the DCLM (Equation 5). The model maintains a default context vector for the first sentences of documents, and treats it as a parameter learned with other model parameters during training.

In the second step, the sentence is generated, conditioning on the preceding sentence and the discourse relation :

(7)

The generative probability for the sentence decomposes across tokens as usual (Equation 7). The per-token probabilities are shown in Equation 4, in Figure 2. Discourse relations are incorporated by parameterizing the output matrices and ; depending on the discourse relation that holds between and , these matrices will favor different parts of the embedding space. The bias term is also parametrized by the discourse relation, so that each relation can favor specific words.

Overall, the joint probability of the text and discourse relations is,

(8)

If the discourse relations are not observed, then our model is a form of latent variable recurrent neural network (LVRNN). Connections to recent work on LVRNNs are discussed in § 6; the key difference is that the latent variables here correspond to linguistically meaningful elements, which we may wish to predict or marginalize, depending on the situation.

Parameter Tying

As proposed, the Discourse Relation Language Model has a large number of parameters. Let , and be the input dimension, hidden dimension and the size of vocabulary in language modeling. The size of each prediction matrix and is ; there are two such matrices for each possible discourse relation. We reduce the number of parameters by factoring each of these matrices into two components:

(9)

where and are relation-specific components for intra-sentential and inter-sentential contexts; the size of these matrices is , with . The larger matrices and are shared across all relations.

3.2 Inference

There are two possible inference scenarios: inference over discourse relations, conditioning on words; and inference over words, marginalizing over discourse relations.

Inference over Discourse Relations

The probability of discourse relations given the sentences is decomposed into the product of probabilities of individual discourse relations conditioned on the adjacent sentences . These probabilities are computed by Bayes’ rule:

(10)

The terms in each product are given in Equations 6 and 7. Normalizing involves only a sum over a small finite number of discourse relations. Note that inference is easy in our case because all words are observed and there is no probabilistic coupling of the discourse relations.

Inference over Words

In discourse-informed language modeling, we marginalize over discourse relations to compute the probability of a sequence of sentence , which can be written as,

(11)

because the word sequences are observed, decoupling each from its neighbors and . This decoupling ensures that we can compute the overall marginal likelihood as a product over local marginals.

3.3 Learning

The model can be trained in two ways: to maximize the joint probability , or to maximize the conditional probability . The joint training objective is more suitable for language modeling scenarios, and the conditional objective is better for discourse relation prediction. We now describe each objective in detail.

Joint likelihood objective

The joint likelihood objective function is directly adopted from the joint probability defined in Equation 8. The objective function for a single document with sentences or utterances is,

(12)

where represents the collection of all model parameters, including the parameters in the LSTM units and the word embeddings.

Maximizing the objective function will jointly optimize the model on both language language and discourse relation prediction. As such, it can be viewed as a form of multi-task learning [Caruana1997], where we learn a shared representation that works well for discourse relation prediction and for language modeling. However, in practice, the large vocabulary size and number of tokens means that the language modeling part of the objective function tends to dominate.

Conditional objective

This training objective is specific to the discourse relation prediction task, and based on Equation 10 can be written as:

(13)

The first line in Equation 13 is the same as , but the second line reflects the normalization over all possible values of . This forces the objective function to attend specifically to the problem of maximizing the conditional likelihood of the discourse relations and treat language modeling as an auxiliary task [Collobert et al.2011].

3.4 Modeling limitations

The discourse relation language model is carefully designed to decouple the discourse relations from each other, after conditioning on the words. It is clear that text documents and spoken dialogues have sequential discourse structures, and it seems likely that modeling this structure could improve performance. In a traditional hidden Markov model (HMM) generative approach 

[Stolcke et al.2000]

, modeling sequential dependencies is not difficult, because training reduces to relative frequency estimation. However, in the hybrid probabilistic-neural architecture proposed here, training is already expensive, due to the large number of parameters that must be estimated. Adding probabilistic couplings between adjacent discourse relations

would require the use of dynamic programming for both training and inference, increasing time complexity by a factor that is quadratic in the number of discourse relations. We did not attempt this in this paper; we do compare against a conventional HMM on the dialogue act prediction task in § 5.

ji2015document propose an alternative form of the document context language model, in which the contextual information impacts the hidden state , rather than going directly to the outputs . They obtain slightly better perplexity with this approach, which has fewer trainable parameters. However, this model would couple with all subsequent sentences , making prediction and marginalization of discourse relations considerably more challenging. Sequential Monte Carlo algorithms offer a possible solution [de Freitas et al., Gu et al.2015], which may be considered in future work.

4 Data and Implementation

We evaluate our model on two benchmark datasets: (1) the Penn Discourse Treebank [Prasad et al.2008, PDTB], which is annotated on a corpus of Wall Street Journal acticles; (2) the Switchboard dialogue act corpus [Stolcke et al.2000, SWDA], which is annotated on a collections of phone conversations. Both corpora contain annotations of discourse relations and dialogue relations that hold between adjacent spans of text.

The Penn Discourse Treebank (PDTB)

provides a low-level discourse annotation on written texts. In the PDTB, each discourse relation is annotated between two argument spans, Arg1 and Arg2. There are two types of relations: explicit and implicit. Explicit relations are signalled by discourse markers (e.g., “however”, “moreover”), and the span of Arg1 is almost totally unconstrained: it can range from a single clause to an entire paragraph, and need not be adjacent to either Arg2 nor the discourse marker. However, automatically classifying these relations is considered to be relatively easy, due to the constraints from the discourse marker itself [Pitler et al.2008]. In addition, explicit relations are difficult to incorporate into language models which must generate each word exactly once. On the contrary, implicit discourse relations are annotated only between adjacent sentences, based on a semantic understanding of the discourse arguments. Automatically classifying these discourse relations is a challenging task [Lin et al.2009, Pitler et al.2009, Rutherford and Xue2015, Ji and Eisenstein2015]. We therefore focus on implicit discourse relations, leaving to the future work the question of how to apply our modeling framework to explicit discourse relations. During training, we collapse all relation types other than implicit (explicit, EntRel, and NoRel) into a single dummy relation type, which holds between all adjacent sentence pairs that do not share an implicit relation.

As in the prior work on first-level discourse relation identification (e.g., Park and Cardie, 2012), we use sections 2-20 of the PDTB as the training set, sections 0-1 as the development set for parameter tuning, and sections 21-22 for testing. For preprocessing, we lower-cased all tokens, and substituted all numbers with a special token “NUM”. To build the vocabulary, we kept the 10,000 most frequent words from the training set, and replaced low-frequency words with a special token “UNK”. In prior work that focuses on detecting individual relations, balanced training sets are constructed so that there are an equal number of instances with and without each relation type [Park and Cardie, Biran and McKeown2013, Rutherford and Xue2014]. In this paper, we target the more challenging multi-way classification problem, so this strategy is not applicable; in any case, since our method deals with entire documents, it is not possible to balance the training set in this way.

The Switchboard Dialog Act Corpus (SWDA)

is annotated on the Switchboard Corpus of human-human conversational telephone speech [Godfrey et al.1992]. The annotations label each utterance with one of 42 possible speech acts, such as agree, hedge, and wh-question. Because these speech acts form the structure of the dialogue, most of them pertain to both the preceding and succeeding utterances (e.g., agree). The SWDA corpus includes 1155 five-minute conversations. We adopted the standard split from stolcke2000dialogue, using 1,115 conversations for training and nineteen conversations for test. For parameter tuning, we randomly select nineteen conversations from the training set as the development set. After parameter tuning, we train the model on the full training set with the selected configuration. We use the same preprocessing techniques here as in the PDTB.

4.1 Implementation

We use a single-layer LSTM to build the recurrent architecture of our models, which we implement in the cnn package.111https://github.com/clab/cnn Our implementation is available on https://github.com/jiyfeng/drlm. Some additional details follow.

Initialization

Following prior work on RNN initialization [Bengio2012], all parameters except the relation prediction parameters and are initialized with random values drawn from the range , where and are the input and output dimensions of the parameter matrix respectively. The matrix is initialized with random numbers from and is initialized to .

Learning

Online learning was performed using AdaGrad [Duchi et al.2011] with initial learning rate

. To avoid the exploding gradient problem, we used norm clipping trick with a threshold of

 [Pascanu et al.2012]. In addition, we used value dropout [Srivastava et al.2014] with rate , on the input , context vector and hidden state

, similar to the architecture proposed by pham2014dropout. The training procedure is monitored by the performance on the development set. In our experiments, 4 to 5 epochs were enough.

Hyper-parameters

Our model includes two tunable hyper-parameters: the dimension of word representation , the hidden dimension of LSTM unit . We consider the values for both and . For each corpus in experiments, the best combination of and is selected via grid search on the development set.

5 Experiments

Our main evaluation is discourse relation prediction, using the PDTB and SWDA corpora. We also evaluate on language modeling, to determine whether incorporating discourse annotations at training time and then marginalizing them at test time can improve performance.

5.1 Implicit discourse relation prediction on the PDTB

We first evaluate our model with implicit discourse relation prediction on the PDTB dataset. Most of the prior work on first-level discourse relation prediction focuses on the “one-versus-all” binary classification setting, but we attack the more general four-way classification problem, as performed by rutherford2015improving. We compare against the following methods:

[style=unboxed,leftmargin=0.4cm,itemsep=0cm]

rutherford2015improving

build a set of feature-rich classifiers on the PDTB, and then augment these classifiers with additional automatically-labeled training instances. We compare against their published results, which are state-of-the-art.

ji2015one

employ a recursive neural network architecture. Their experimental setting is different, so we re-run their system using the same setting as described in § 4.

Model Accuracy Macro
Baseline
1. Most common class 54.7
Prior work
2. [Rutherford and Xue2015] 55.0 38.4
3. [Rutherford and Xue2015] 57.1 40.5
     with extra training data
4. [Ji and Eisenstein2015] 56.4 40.0
Our work - DrLM
5. Joint training 57.1 40.5
6. Conditional training 59.5 42.3
significantly better than lines 2 and 4 with
Table 1: Multiclass relation identification on the first-level PDTB relations.

Results

As shown in Table 1, the conditionally-trained discourse relation language models (DrLM) outperforms all alternatives, on both metrics. While the jointly-trained DrLM is at the same level as the previous state-of-the-art, conditional training on the same model provides a significant additional advantage, indicated by a binomial test.

5.2 Dialogue Act tagging

Dialogue act tagging has been widely studied in both NLP and speech communities. We follow the setup used by stolcke2000dialogue to conduct experiments, and adopt the following systems for comparison:

[style=unboxed,leftmargin=0.4cm,itemsep=0cm]

stolcke2000dialogue

employ a hidden Markov model, with each HMM state corresponding to a dialogue act.

kalchbrenner2013recurrent

employ a complex neural architecture, with a convolutional network at each utterance and a recurrent network over the length of the dialog. To our knowledge, this model attains state-of-the-art accuracy on this task, outperforming other prior work such as [Webb et al.2005, Milajevs and Purver2014].

Results

As shown in Table 2, the conditionally-trained discourse relation language model (DrLM) outperforms all competitive systems on this task. A binomial test shows the result in line 6 is significantly better than the previous state-of-the-art (line 4). All comparisons are against published results, and Macro- scores are not available. Accuracy is more reliable on this evaluation, since no single class dominates, unlike the PDTB task.

1. Model Accuracy
Baseline
2. Most common class 31.5
Prior work
3. [Stolcke et al.2000] 71.0
4. [Kalchbrenner and Blunsom2013] 73.9
Our work - DrLM
5. Joint training 74.0
6. Conditional training 77.0
significantly better than line 4 with
Table 2: The results of dialogue act tagging.

5.3 Discourse-aware language modeling

As a joint model for discourse and language modeling, DrLM can also function as a language model, assigning probabilities to sequences of words while marginalizing over discourse relations. To determine whether discourse-aware language modeling can improve performance, we compare against the following systems:

[style=unboxed,leftmargin=0.4cm,itemsep=0cm]

RNNLM+LSTM

This is the same basic architecture as the RNNLM proposed by [Mikolov et al.2010]

, which was shown to outperform a Kneser-Ney smoothed 5-gram model on modeling Wall Street Journal text. Following pham2014dropout, we replace the Sigmoid nonlinearity with a long short-term memory (LSTM).

DCLM

We compare against the Document Context Language Model (DCLM) of ji2015document. We use the “context-to-output” variant, which is identical to the current modeling approach, except that it is not parametrized by discourse relations. This model achieves strong results on language modeling for small and medium-sized corpora, outperforming RNNLM+LSTM.

Results

The perplexities of language modeling on the PDTB and the SWDA are summarized in Table 3. The comparison between line 1 and line 2 shows the benefit of considering multi-sentence context information on language modeling. Line 3 shows that adding discourse relation information yields further improvements for both datasets. We emphasize that discourse relations in the test documents are marginalized out, so no annotations are required for the test set; the improvements are due to the disambiguating power of discourse relations in the training set.

Because our training procedure requires discourse annotations, this approach does not scale to the large datasets typically used in language modeling. As a consequence, the results obtained here are somewhat academic, from the perspective of practical language modeling. Nonetheless, the positive results here motivate the investigation of training procedures that are also capable of marginalizing over discourse relations at training time.

PDTB SWDA
Model pplx pplx
Baseline
1. RNNLM 96 128 117.8 128 96 56.0
2. DCLM 96 96 112.2 96 96 45.3
Our work
3. DrLM 64 96 108.3 128 64 39.6
Table 3: Language model perplexities (pplx), lower is better. The model dimensions and that gave best performance on the dev set are also shown.

6 Related Work

This paper draws on previous work in both discourse modeling and language modeling.

Discourse and dialog modeling

Early work on discourse relation classification utilizes rich, hand-crafted feature sets [Joty et al.2012, Lin et al.2009, Sagae2009]. Recent representation learning approaches attempt to learn good representations jointly with discourse relation classifiers and discourse parsers [Ji and Eisenstein2014, Li et al.2014]. Of particular relevance are applications of neural architectures to PDTB implicit discourse relation classification [Ji and Eisenstein2015, Zhang et al.2015, Braud and Denis2015]. All of these approaches are essentially classifiers, and take supervision only from the 16,000 annotated discourse relations in the PDTB training set. In contrast, our approach is a probabilistic model over the entire text.

Probabilistic models are frequently used in dialog act tagging, where hidden Markov models have been a dominant approach [Stolcke et al.2000]. In this work, the emission distribution is an -gram language model for each dialogue act; we use a conditionally-trained recurrent neural network language model. An alternative neural approach for dialogue act tagging is the combined convolutional-recurrent architecture of kalchbrenner2013recurrent. Our modeling framework is simpler, relying on a latent variable parametrization of a purely recurrent architecture.

Language modeling

There are an increasing number of attempts to incorporate document-level context information into language modeling. For example, mikolov2012context introduce LDA-style topics into RNN based language modeling. sordoni2015neural use a convolutional structure to summarize the context from previous two utterances as context vector for RNN based language modeling. Our models in this paper provide a unified framework to model the context and current sentence. wang2015larger and lin2015hierarchical construct bag-of-words representations of previous sentences, which are then used to inform the RNN language model that generates the current sentence. The most relevant work is the Document Context Language Model [Ji et al.2015, DCLM]; we describe the connection to this model in § 2. By adding discourse information as a latent variable, we attain better perplexity on held-out data.

Latent variable neural networks

Introducing latent variables to a neural network model increases its representational capacity, which is the main goal of prior efforts in this space [Kingma and Welling2014, Chung et al.2015]. From this perspective, our model with discourse relations as latent variables shares the same merit. Unlike this prior work, in our approach, the latent variables carry a linguistic interpretation, and are at least partially observed. Also, these prior models employ continuous latent variables, requiring complex inference techniques such as variational autoencoders [Kingma and Welling2014, Burda et al.2016, Chung et al.2015]. In contrast, the discrete latent variables in our model are easy to sum and maximize over.

7 Conclusion

We have presented a probabilistic neural model over sequences of words and shallow discourse relations between adjacent sequences. This model combines positive aspects of neural network architectures with probabilistic graphical models: it can learn discriminatively-trained vector representations, while maintaining a probabilistic representation of the targeted linguistic element: in this case, shallow discourse relations. This method outperforms state-of-the-art systems in two discourse relation detection tasks, and can also be applied as a language model, marginalizing over discourse relations on the test data. Future work will investigate the possibility of learning from partially-labeled training data, which would have at least two potential advantages. First, it would enable the model to scale up to the large datasets needed for competitive language modeling. Second, by training on more data, the resulting vector representations might support even more accurate discourse relation prediction.

Acknowledgments

Thanks to Trevor Cohn, Chris Dyer, Lingpeng Kong, and Quoc V. Le for helpful discussions, and to the anonymous reviewers for their feedback. This work was supported by a Google Faculty Research award to the third author. It was partially performed during the 2015 Jelinek Memorial Summer Workshop on Speech and Language Technologies at the University of Washington, Seattle, and was supported by Johns Hopkins University via NSF Grant No IIS 1005411, DARPA LORELEI Contract No HR0011-15-2-0027, and gifts from Google, Microsoft Research, Amazon and Mitsubishi Electric Research Laboratory.

References

  • [Bengio2012] Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437–478. Springer.
  • [Biran and McKeown2013] Or Biran and Kathleen McKeown. 2013. In Proceedings of the Association for Computational Linguistics (ACL), pages 69–73, Sophia, Bulgaria.
  • [Braud and Denis2015] Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 2201–2211, Lisbon, September.
  • [Burda et al.2016] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR).
  • [Caruana1997] Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75.
  • [Chung et al.2015] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Neural Information Processing Systems (NIPS), Montréal.
  • [Collobert et al.2011] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
  • [de Freitas et al.] João FG de Freitas, Mahesan Niranjan, Andrew H. Gee, and Arnaud Doucet. Sequential monte carlo methods to train neural network models. Neural computation, 12(4):955–993.
  • [Duchi et al.2011] John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159.
  • [Dyer et al.2015] Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the Association for Computational Linguistics (ACL), pages 334–343, Beijing, China.
  • [Finkel et al.2006] Jenny Rose Finkel, Christopher D Manning, and Andrew Y Ng. 2006.

    Solving the problem of cascading errors: Approximate bayesian inference for linguistic annotation pipelines.

    In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 618–626.
  • [Godfrey et al.1992] John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In ICASSP, volume 1, pages 517–520. IEEE.
  • [Gu et al.2015] Shixiang Gu, Zoubin Ghahramani, and Richard E Turner. 2015. Neural adaptive sequential monte carlo. In Neural Information Processing Systems (NIPS), Montréal.
  • [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
  • [Ji and Eisenstein2014] Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, MD.
  • [Ji and Eisenstein2015] Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributional semantics for discourse relations. Transactions of the Association for Computational Linguistics (TACL), June.
  • [Ji et al.2015] Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. In International Conference on Learning Representations, Poster Paper, volume abs/1511.03962.
  • [Joty et al.2012] Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP).
  • [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013.

    Recurrent convolutional neural networks for discourse compositionality.

    In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126, Sofia, Bulgaria, August. Association for Computational Linguistics.
  • [Kingma and Welling2014] Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR).
  • [Le and Mikolov2014] Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML).
  • [Li et al.2014] Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP).
  • [Lin et al.2009] Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 343–351, Singapore.
  • [Lin et al.2015] Rui Lin, Shujie Liu, Muyun Yang, Mu Li, Ming Zhou, and Sheng Li. 2015. Hierarchical recurrent neural network for document modeling. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 899–907, Lisbon, September.
  • [Manning2016] Christopher D. Manning. 2016.

    Computational linguistics and deep learning.

    Computational Linguistics, 41(4).
  • [Mikolov and Zweig2012] Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In Proceedings of Spoken Language Technology (SLT), pages 234–239.
  • [Mikolov et al.2010] Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH, pages 1045–1048.
  • [Milajevs and Purver2014] Dmitrijs Milajevs and Matthew Purver. 2014. Investigating the contribution of distributional semantic information for dialogue act classification. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 40–47.
  • [Park and Cardie] Joonsuk Park and Claire Cardie. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112, Seoul, South Korea, July. Association for Computational Linguistics.
  • [Pascanu et al.2012] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063.
  • [Pham et al.2014] Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, pages 285–290. IEEE.
  • [Pitler et al.2008] Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 87–90, Manchester, UK.
  • [Pitler et al.2009] Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Association for Computational Linguistics (ACL), Suntec, Singapore.
  • [Prasad et al.2008] Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of LREC.
  • [Rutherford and Xue2014] Attapol T Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the European Chapter of the Association for Computational Linguistics (EACL).
  • [Rutherford and Xue2015] Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. pages 799–808, May–June.
  • [Sagae2009] Kenji Sagae. 2009. Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT’09), pages 81–84, Paris, France, October. Association for Computational Linguistics.
  • [Socher et al.2013] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), Seattle, WA.
  • [Sordoni et al.2015] Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), Denver, CO, May.
  • [Srivastava et al.2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958.
  • [Stolcke et al.2000] Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373.
  • [Turian et al.2010] Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010.

    Word representations: A simple and general method for semi-supervised learning.

    pages 384–394.
  • [Wang and Cho2015] Tian Wang and Kyunghyun Cho. 2015. Larger-context language modelling. arXiv preprint arXiv:1511.03729.
  • [Webb et al.2005] Nick Webb, Mark Hepple, and Yorick Wilks. 2005. Dialogue act classification based on intra-utterance features. In Proceedings of the AAAI Workshop on Spoken Language Understanding.
  • [Webber et al.2012] Bonnie Webber, Markus Egg, and Valia Kordoni. 2012. Discourse structure and language technology. Journal of Natural Language Engineering, 1.
  • [Williams and Young2007] Jason D Williams and Steve Young. 2007.

    Partially observable markov decision processes for spoken dialog systems.

    Computer Speech & Language, 21(2):393–422.
  • [Xue et al.2015] Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol T Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Conference on Natural Language Learning (CoNLL).
  • [Zhang et al.2015] Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 2230–2235, Lisbon, September.