A Mention-Ranking Model for Abstract Anaphora Resolution

06/07/2017 ∙ by Ana Marasović, et al. ∙ University of Heidelberg 0

Resolving abstract anaphora is an important, but difficult task for text understanding. Yet, with recent advances in representation learning this task becomes a more tangible aim. A central property of abstract anaphora is that it establishes a relation between the anaphor embedded in the anaphoric sentence and its (typically non-nominal) antecedent. We propose a mention-ranking model that learns how abstract anaphors relate to their antecedents with an LSTM-Siamese Net. We overcome the lack of training data by generating artificial anaphoric sentence--antecedent pairs. Our model outperforms state-of-the-art results on shell noun resolution. We also report first benchmark results on an abstract anaphora subset of the ARRAU corpus. This corpus presents a greater challenge due to a mixture of nominal and pronominal anaphors and a greater range of confounders. We found model variants that outperform the baselines for nominal anaphors, without training on individual anaphor data, but still lag behind for pronominal anaphors. Our model selects syntactically plausible candidates and -- if disregarding syntax -- discriminates candidates using deeper features.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Current research in anaphora (or coreference) resolution is focused on resolving noun phrases referring to concrete objects or entities in the real world, which is arguably the most frequently occurring type. Distinct from these are diverse types of abstract anaphora (AA) (Asher, 1993) where reference is made to propositions, facts, events or properties. An example is given in (1) below.111Example drawn from ARRAU (Uryupina et al., 2016).

While recent approaches address the resolution of selected abstract shell nouns (Kolhatkar and Hirst, 2014), we aim to resolve a wide range of abstract anaphors, such as the NP this trend in (1), as well as pronominal anaphors (this, that, or it).

Henceforth, we refer to a sentence that contains an abstract anaphor as the anaphoric sentence (AnaphS), and to a constituent that the anaphor refers to as the antecedent (Antec) (cf. (1)).

Ever-more powerful desktop computers, designed with one or more microprocessors as their ”brains”, are expected to increasingly take on functions carried out by more expensive minicomputers and mainframes. ”[ The guys that make traditional hardware are really being obsoleted by microprocessor-based machines]”, said Mr. Benton. [ As a result of this trend, longtime powerhouses HP, IBM and Digital Equipment Corp. are scrambling to counterattack with microprocessor-based systems of their own.]

A major obstacle for solving this task is the lack of sufficient amounts of annotated training data. We propose a method to generate large amounts of training instances covering a wide range of abstract anaphor types. This enables us to use neural methods which have shown great success in related tasks: coreference resolution (Clark and Manning, 2016a), textual entailment (Bowman et al., 2016), learning textual similarity (Mueller and Thyagarajan, 2016), and discourse relation sense classification (Rutherford et al., 2017).

Our model is inspired by the mention-ranking model for coreference resolution (Wiseman et al., 2015; Clark and Manning, 2015, 2016a, 2016b) and combines it with a Siamese Net (Mueller and Thyagarajan, 2016), (Neculoiu et al., 2016) for learning similarity between sentences. Given an anaphoric sentence (AntecS in (1)) and a candidate antecedent (any constituent in a given context, e.g. being obsoleted by microprocessor-based machines in (1)), the LSTM-Siamese Net learns representations for the candidate and the anaphoric sentence in a shared space. These representations are combined into a joint representation used to calculate a score that characterizes the relation between them. The learned score is used to select the highest-scoring antecedent candidate for the given anaphoric sentence and hence its anaphor. We consider one anaphor at a time and provide the embedding of the context of the anaphor and the embedding of the head of the anaphoric phrase to the input to characterize each individual anaphor – similar to the encoding proposed by Zhou and Xu (2015) for individuating multiply occurring predicates in SRL. With deeper inspection we show that the model learns a relation between the anaphor in the anaphoric sentence and its antecedent. Fig. 1 displays our architecture.

In contrast to other work, our method for generating training data is not confined to specific types of anaphora such as shell nouns (Kolhatkar and Hirst, 2014) or anaphoric connectives (Stede and Grishina, 2016). It produces large amounts of instances and is easily adaptable to other languages. This enables us to build a robust, knowledge-lean model for abstract anaphora resolution that easily extends to multiple languages.

We evaluate our model on the shell noun resolution dataset of Kolhatkar et al. (2013b) and show that it outperforms their state-of-the-art results. Moreover, we report results of the model (trained on our newly constructed dataset) on unrestricted abstract anaphora instances from the ARRAU corpus (Poesio and Artstein, 2008; Uryupina et al., 2016). To our knowledge this provides the first state-of-the-art benchmark on this data subset.

Our TensorFlow

222Abadi et al. (2015) implementation of the model and scripts for data extraction are available at: https://github.com/amarasovic/neural-abstract-anaphora.

2 Related and prior work

Abstract anaphora has been extensively studied in linguistics and shown to exhibit specific properties in terms of semantic antecedent types, their degrees of abstractness, and general discourse properties (Asher, 1993; Webber, 1991). In contrast to nominal anaphora, abstract anaphora is difficult to resolve, given that agreement and lexical match features are not applicable. Annotation of abstract anaphora is also difficult for humans (Dipper and Zinsmeister, 2012), and thus, only few smaller-scale corpora have been constructed. We evaluate our models on a subset of the ARRAU corpus (Uryupina et al., 2016) that contains abstract anaphors and the shell noun corpus used in Kolhatkar et al. (2013b).333We thank the authors for making their data available. We are not aware of other freely available abstract anaphora datasets.

Little work exists for the automatic resolution of abstract anaphora. Early work (Eckert and Strube, 2000; Strube and Müller, 2003; Byron, 2004; Müller, 2008) has focused on spoken language, which exhibits specific properties. Recently, event coreference

has been addressed using feature-based classifiers

(Jauhar et al., 2015; Lu and Ng, 2016). Event coreference is restricted to a subclass of events, and usually focuses on coreference between verb (phrase) and noun (phrase) mentions of similar abstractness levels (e.g. purchase – acquire) with no special focus on (pro)nominal anaphora. Abstract anaphora typically involves a full-fledged clausal antecedent that is referred to by a highly abstract (pro)nominal anaphor, as in (0).

Rajagopal et al. (2016) proposed a model for resolution of events in biomedical text that refer to a single or multiple clauses. However, instead of selecting the correct antecedent clause(s) (our task) for a given event, their model is restricted to classifying the event into six abstract categories: this these changes, responses, analysis, context, finding, observation, based on its surrounding context. While related, their task is not comparable to the full-fledged abstract anaphora resolution task, since the events to be classified are known to be coreferent and chosen from a set of restricted abstract types.

More related to our work is Anand and Hardt (2016) who present an antecedent ranking account for sluicing

using classical machine learning based on a small training dataset. They employ features modeling distance, containment, discourse structure, and – less effectively – content and lexical correlates.

444Their data set was not publicized.

Closest to our work is Kolhatkar et al. (2013b) (KZH13) and Kolhatkar and Hirst (2014) (KH14) on shell noun resolution, using classical machine learning techniques. Shell nouns are abstract nouns, such as fact, possibility, or issue, which can only be interpreted jointly with their shell content (their embedded clause as in (1) or antecedent as in (2)). KZH13 refer to shell nouns whose antecedent occurs in the prior discourse as anaphoric shell nouns (ASNs) (cf. (2)), and cataphoric shell nouns (CSNs) otherwise (cf. (1)).555We follow this terminology for their approach and data representation.

Congress has focused almost solely on the fact that [special education is expensive - and that it takes away money from regular education.] Environmental Defense […] notes that [ “Mowing the lawn with a gas mower produces as much pollution […] as driving a car 172 miles.“] [ This fact may […] explain the recent surge in the sales of […] old-fashioned push mowers […]].

KZH13 presented an approach for resolving six typical shell nouns following the observation that CSNs are easy to resolve based on their syntactic structure alone, and the assumption that ASNs share linguistic properties with their embedded (CSN) counterparts. They manually developed rules to identify the embedded clause (i.e. cataphoric antecedent) of CSNs and trained SVM (Joachims, 2002) on such instances. The trained SVM model is then used to resolve ASNs. KH14 generalized their method to be able to create training data for any given shell noun, however, their method heavily exploits the specific properties of shell nouns and does not apply to other types of abstract anaphora.

Stede and Grishina (2016) study a related phenomenon for German. They examine inherently anaphoric connectives (such as demzufolge – according to which) that could be used to access their abstract antecedent in the immediate context. Yet, such connectives are restricted in type, and the study shows that such connectives are often ambiguous with nominal anaphors and require sense disambiguation. We conclude that they cannot be easily used to acquire antecedents automatically.

In our work, we explore a different direction: we construct artificial training data using a general pattern that identifies embedded sentence constituents, which allows us to extract relatively secure training data for abstract anaphora that captures a wide range of anaphora-antecedent relations, and apply this data to train a model for the resolution of unconstrained abstract anaphora.

Recent work in entity coreference resolution has proposed powerful neural network-based models that we will adapt to the task of abstract anaphora resolution. Most relevant for our task is the

mention-ranking neural coreference model proposed in Clark and Manning (2015), and their improved model in Clark and Manning (2016a)

, which integrates a loss function

(Wiseman et al., 2015) which learns distinct feature representations for anaphoricity detection and antecedent ranking.

Siamese Nets distinguish between similar and dissimilar pairs of samples by optimizing a loss over the metric induced by the representations. It is widely used in vision (Chopra et al., 2005), and in NLP for semantic similarity, entailment, query normalization and QA (Mueller and Thyagarajan, 2016; Neculoiu et al., 2016; Das et al., 2016).

3 Mention-Ranking Model

Given an anaphoric sentence with a marked anaphor (mention) and a candidate antecedent , the mention-ranking (MR) model assigns the pair a score, using representations produced by an LSTM-Siamese Net. The highest-scoring candidate is assigned to the marked anaphor in the anaphoric sentence. Fig. 1 displays the model.

We learn representations of an anaphoric sentence and a candidate antecedent

using a bi-directional Long Short-Term Memory

(Hochreiter and Schmidhuber, 1997; Graves and Schmidhuber, 2005). One bi-LSTM is applied to the anaphoric sentence and a candidate antecedent

, hence the term siamese. Each word is represented with a vector

constructed by concatenating embeddings of the word, of the context of the anaphor (average of embeddings of the anaphoric phrase, the previous and the next word), of the head of the anaphoric phrase666Henceforth we refer to it as embedding of the anaphor., and, finally, an embedding of the constituent tag of the candidate, or the S constituent tag if the word is in the anaphoric sentence. For each sequence or , the word vectors are sequentially fed into the bi-LSTM, which produces outputs from the forward pass, , and outputs from the backward pass. The final output of the i-th word is defined as . To get a representation of the full sequence, or

, all outputs are averaged, except for those that correspond to padding tokens.

Figure 1: Mention-ranking architecture for abstract anaphora resolution (MR-LSTM).

To prevent forgetting the constituent tag of the sequence, we concatenate the corresponding tag embedding with or (we call this a shortcut for the tag information). The resulting vector is fed into a feed-forward layer of exponential linear units (ELUs) (Clevert et al., 2016) to produce the final representation or of the sequence.

From and we compute a vector (Tai et al., 2015), where denotes the absolute values of the element-wise subtraction, and the element-wise multiplication. Then is fed into a feed-forward layer of ELUs to obtain the final joint representation, , of the pair . Finally, we compute the score for the pair that represents relatedness between them, by applying a single fully connected linear layer to the joint representation:

(1)

where W is a weight matrix, and the dimension of the vector .

We train the described mention-ranking model with the max-margin training objective from Wiseman et al. (2015), used for the antecedent ranking subtask. Suppose that the training set , where is the i-th abstract anaphor, the corresponding anaphoric sentence, the set of antecedents of and the set of candidates that are not antecedents (negative candidates). Let be the highest scoring antecedent of . Then the loss is given by

4 Training data construction

[.VP v [.S’ x [.S ]]]

Figure 2: A general pattern for artificially creating anaphoric sentence–antecedent pairs.

We create large-scale training data for abstract anaphora resolution by exploiting a common construction, consisting of a verb with an embedded sentence (complement or adverbial) (cf. Fig. 2). We detect this pattern in a parsed corpus, ’cut off’ the S constituent and replace it with a suitable anaphor to create the anaphoric sentence (AnaphS), while S yields the antecedent (Antec). This method covers a wide range of anaphora-antecedent constellations, due to diverse semantic or discourse relations that hold between the clause hosting the verb and the embedded sentence.

First, the pattern applies to verbs that embed sentential arguments. In (1), the verb doubt establishes a specific semantic relation between the embedding sentence and its sentential complement.

He doubts [[ a Bismarckian super state will emerge that would dominate Europe], but warns of ”a risk of profound change in the [..] European Community from a Germany that is too strong, even if democratic”].

From this we extract the artificial antecedent A Bismarckian super state will emerge that would dominate Europe, and its corresponding anaphoric sentence He doubts this, but warns of ”a risk of profound change … even if democratic”, which we construct by randomly choosing one of a pre-defined set of appropriate anaphors (here: this, that, it), cf. Table 1. The second row in Table 1 is used when the head of S is filled by an overt complementizer (doubts that), as opposed to (0). The remaining rows in Table 1 apply to adverbial clauses of different types.

Adverbial clauses encode specific discourse relations with their embedding sentences, often indicated by their conjunctions. In (1), for example, the causal conjunction as relates a cause (embedded sentence) and its effect (embedding sentence):

There is speculation that property casualty firms will sell even more munis [ as [ they scramble to raise cash to pay claims related to Hurricane Hugo [..] ]].

We randomly replace causal conjunctions because, as with appropriately adjusted anaphors, e.g. because of that, due to this or therefore that make the causal relation explicit in the anaphor.777In case of ambiguous conjunctions (e.g. as interpreted as causal or temporal), we generally choose the most frequent interpretation.

type head of S possible anaphoric phrase
empty this, that
general that, this that, this
causal because, as therefore, because of this/that,
temporal while, since, etc. during this/that
conditional if, whether if this/that is true
Table 1: S-heads and the anaphoric types and phrases they induce (most frequent interpretation).

Compared to the shell noun corpus of KZH13, who made use of a carefully constructed set of extraction patterns, a downside of our method is that our artificially created antecedents are uniformly of type S. However, the majority of abstract anaphora antecedents found in the existing datasets are of type S. Also, our models are intended to induce semantic representations, and so we expect syntactic form to be less critical, compared to a feature-based model.888This also alleviates problems with languages like German, where (non-)embedded sentences differ in surface position of the finite verb. We can either adapt the order or ignore it, when producing anaphoric sentence – antecedent pairs. Finally, the general extraction pattern in Fig. 2, covers a much wider range of anaphoric types.

Using this method we generated a dataset of artificial anaphoric sentence–antecedent pairs from the WSJ part of the PTB Corpus (Marcus et al., 1993), automatically parsed using the Stanford Parser (Klein and Manning, 2003).

5 Experimental setup

5.1 Datasets

We evaluate our model on two types of anaphora: (a) shell noun anaphora and (b) (pro)nominal abstract anaphors extracted from ARRAU.

a. Shell noun resolution dataset.

For comparability we train and evaluate our model for shell noun resolution, using the original training (CSN) and test (ASN) corpus of Kolhatkar et al. (2013a, b).999We thank the authors for providing the available data. We follow the data preparation and evaluation protocol of Kolhatkar et al. (2013b) (KZH13).

The CSN corpus was constructed from the NYT corpus using manually developed patterns to identify the antecedent of cataphoric shell nouns (CSNs). In KZH13, all syntactic constituents of the sentence that contains both the CSN and its antecedent were considered as candidates for training a ranking model. Candidates that differ from the antecedent in only one word or one word and punctuation were as well considered as antecedents101010We obtained this information from the authors directly.. To all other candidates we refer to as negative candidates. For every shell noun, KZH13 used the corresponding part of the CSN data to train SVM.

The ASN corpus serves as the test corpus. It was also constructed from the NYT corpus, by selecting anaphoric instances with the pattern ”this ” for all covered shell nouns. For validation, Kolhatkar et al. (2013a) crowdsourced annotations for the sentence which contains the antecedent, which KZH13 refer to as a broad region. Candidates for the antecedent were obtained by using all syntactic constituents of the broad region as candidates and ranking them using the SVM model trained on the CSN corpus. The top 10 ranked candidates were presented to the crowd workers and they chose the best answer that represents the ASN antecedent. The workers were encouraged to select None when they did not agree with any of the displayed answers and could provide information about how satisfied they were with the displayed candidates. We consider this dataset as gold, as do KZH13, although it may be biased towards the offered candidates.111111The authors provided us with the workers’ annotations of the broad region, antecedents chosen by the workers and links to the NYT corpus. The extraction of the anaphoric sentence and the candidates had to be redone.

b. Abstract anaphora resolution data set.

We use the automatically constructed data from the WSJ corpus (Section 4) for training.121212We excluded any documents that are part of ARRAU. Our test data for unrestricted abstract anaphora resolution is obtained from the ARRAU corpus (Uryupina et al., 2016). We extracted all abstract anaphoric instances from the WSJ part of ARRAU that are marked with the category abstract or plan,131313ARRAU distinguishes abstract anaphors and (mostly) pronominal anaphors referring to an action or plan, as plan. and call the subcorpus ARRAU-AA.

Candidates extraction.

Following KZH13, for every anaphor we create a list of candidates by extracting all syntactic constituents from sentences which contain antecedents. Candidates that differ from antecedents in only one word, or one word and punctuation, were as well considered as antecedents. Constituents that are not antecedents are considered as negative candidates.

Data statistics.

Table 2 gives statistics of the datasets: the number of anaphors (row 1), the median length (in tokens) of antecedents (row 2), the median length (in tokens) for all anaphoric sentences (row 3), the median of the number of antecedents and candidates that are not antecedents (negatives) (rows 4–5), the number of pronominal and nominal anaphors (rows 6–7). Both training sets, artificial and CSN, have only one possible antecedent for which we accept two minimal variants differing in only one word or one word and punctuation. On the contrary, both test sets by design allow annotation of more than one antecedent that differ in more than one word. Every anaphor in the artificial training dataset is pronominal, whereas anaphors in CSN and ASN are nominal only. ARRAU-AA has a mixture of nominal and pronominal anaphors.

shell noun abstract anaphora
CSN
darkgray train
ASN
darkgray test
artifical
darkgray train
ARRAU-AA
darkgray test
# shell nouns / anaphors 114492 2303 8527 600
median
# of tokens
Antec 12.75 13.87 11 20.5
AnaphS 11.5 24 19 28
median
#
Antec 2 4.5 2 1

negatives

44.5 39 15 48
# nominal 114492 2303 0 397
pronominal 0 0 8527 203
Table 2: Data statistics. For the ASN and CSN we report statistics over all shell nouns, but classifiers are trained independently.

Data pre-processing.

Other details can be found in Supplementary Materials.

5.2 Baselines and evaluation metrics

Following KZH13, we report success@n (s@n), which measures whether the antecedent, or a candidate that differs in one word141414We obtained this information in personal communication with one of the authors., is in the first ranked candidates, for . Additionally, we report the preceding sentence baseline (PS) that chooses the previous sentence for the antecedent and TAGbaseline (TAG) that randomly chooses a candidate with the constituent tag label in S, VP, ROOT, SBAR. For TAG we report the average of 10 runs with 10 fixed seeds. PS always performs worse than the KZH13 model on the ASN, so we report it only for ARRAU-AA.

5.3 Training details for our models

Hyperparameters tuning.

We recorded performance with manually chosen HPs and then tuned HPs with Tree-structured Parzen Estimators (TPE)

(Bergstra et al., 2011)151515https://github.com/hyperopt/hyperopt.. TPE chooses HPs for the next (out of ) trails on the basis of the s@1 score on the devset. As devsets we employ the ARRAU-AA corpus for shell noun resolution and the ASN corpus for unrestricted abstract anaphora resolution. For each trial we record performance on the test set. We report the best test s@1 score in trials if it is better than the scores from default HPs. The default HPs and prior distributions for HPs used by TPE are given below. The (exact) HPs we used can be found in Supplementary Materials.

Input representation. To construct word vectors as defined in Section 3, we used -dim. GloVe word embeddings pre-trained on the Gigaword and Wikipedia (Pennington et al., 2014), and did not fine-tune them. Vocabulary was built from the words in the training data with frequency in , and OOV words were replaced with an UNK

token. Embeddings for tags are initialized with values drawn from the uniform distribution

, where is the number of tags161616We used a list of tags obtained from the Stanford Parser. and the size of the tag embeddings.171717qlog- is the so-called qlog-uniform distribution. We experimented with removing embeddings for tag, anaphor and context.

Weights initialization. The size of the LSTMs hidden states was set to . We initialized the weight matrices of the LSTMs with random orthogonal matrices (Henaff et al., 2016), all other weight matrices with the initialization proposed in He et al. (2015). The first feed-forward layer size is set to a value in , the second to a value in . Forget biases in the LSTM were initialized with 1s (Józefowicz et al., 2015), all other biases with 0s.

Optimization. We trained our model in mini-batches using Adam (Kingma and Ba, 2015) with the learning rate of and maximal batch size . We clip gradients by global norm (Pascanu et al., 2013), with a clipping value in . We train for epochs and choose the model that performs best on the devset.

Regularization. We used the -regularization with . Dropout (Srivastava et al., 2014)

with a keep probability

was applied to the outputs of the LSTMs, both feed-forward layers and optionally to the input with .

6 Results and analysis

s @ 1 s @ 2 s @ 3 s @ 4
fact (train: 43809, test: 472) MR-LSTM 83.47 85.38 86.44 87.08
KZH13 70.00 86.00 92.00 95.00
TAG 46.99 - - -
reason (train: 4529, test: 442) MR-LSTM 71.27 77.38 80.09 80.54
+ tuning 87.78 91.63 93.44 93.89
KZH13 72.00 86.90 90.00 94.00
TAG 42.40 - - -
issue (train: 2664, test: 303) MR-LSTM 88.12 91.09 93.07 93.40
KZH13 47.00 61.00 72.00 81.00
TAG 44.92 - - -
decision (train: 42289, test: 389) MR-LSTM 76.09 85.86 91.00 93.06
KZH13 35.00 53.00 67.00 76.00
TAG 45.55 - - -
question (train: 9327, test: 440) MR-LSTM 89.77 94.09 95.00 95.68
KZH13 70.00 83.00 88.00 91.00
TAG 42.02 - - -
possibility (train: 11874, test: 277) MR-LSTM 93.14 94.58 95.31 95.67
KZH13 56.00 76.00 87.00 92.00
TAG 48.66 - - -
Table 3: Shell noun resolution results.

6.1 Results on shell noun resolution dataset

Table 3 provides the results of the mention-ranking model (MR-LSTM) on the ASN corpus using default HPs. Column 2 states which model produced the results: KZH13 refers to the best reported results in Kolhatkar et al. (2013b) and TAG is the baseline described in Section 5.2.

In terms of s@1 score, MR-LSTM outperforms both KZH13’s results and TAG

without even necessitating HP tuning. For the outlier

reason we tuned HPs (on ARRAU-AA) for different variants of the architecture: the full architecture, without embedding of the context of the anaphor (ctx), of the anaphor (aa), of both constituent tag embedding and shortcut (tag,cut), dropping only the shortcut (cut), using only word embeddings as input (ctx,aa,tag,cut), without the first (ffl1) and second (ffl2) layer. From Table 4 we observe: (1) with HPs tuned on ARRAU-AA, we obtain results well beyond KZH13, (2) all ablated model variants perform worse than the full model, (3) a large performance drop when omitting syntactic information (tag,cut) suggests that the model makes good use of it. However, this could also be due to a bias in the tag distribution, given that all candidates stem from the single sentence that contains antecedents. The median occurrence of the S tag among both antecedents and negative candidates is , thus the model could achieve s@1 by picking S-type constituents, just as TAG achieves for reason and for possibility.

Tuning of HPs gives us insight into how different model variants cope with the task. For example, without tuning the model with and without syntactic information achieves and (not shown in table) s@1 score, respectively, and with tuning: and . Performance of s@1 score indicates that the model is able to learn without syntactic guidance, contrary to the s@1 score before tuning.

reason
ctx aa tag cut ffl1 ffl2 s@1 s@2 s@ 3 s@ 4
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 87.78 91.63 93.44 93.89
darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 85.97 87.56 89.14 89.82
darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 86.65 88.91 91.18 91.40
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 68.10 80.32 85.29 89.37
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ 85.52 88.24 89.59 90.05
darkred ✗ darkred ✗ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 66.97 80.54 85.75 88.24
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ 87.56 91.63 92.76 94.12
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ 85.97 88.69 89.14 90.05
Table 4: Architecture ablation for reason.
all (600) nominal (397) pronominal (203)
ctx aa tag cut ffl1 ffl2 s@1 s@2 s@ 3 s@ 4 s@1 s@2 s@ 3 s@ 4 s@1 s@2 s@ 3 s@ 4
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 24.17 43.67 54.50 63.00 29.47 50.63 62.47 72.04 13.79 30.05 38.92 45.32
darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 29.67 52.50 66.00 75.00 33.50 58.19 72.04 80.86 22.17 41.38 54.19 63.55
darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 22.83 39.00 52.00 61.33 22.42 41.31 54.66 64.48 23.65 34.48 46.80 55.17
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 38.33 54.83 63.17 69.33 46.60 64.48 72.54 79.09 22.17 35.96 44.83 50.25
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 43.83 56.33 66.33 73.00 51.89 64.48 73.55 79.85 28.08 40.39 52.22 59.61
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 38.17 52.50 61.33 68.67 43.07 57.43 65.49 72.04 28.57 42.86 53.20 62.07
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ 30.17 48.00 57.83 67.33 30.73 50.88 61.21 71.54 29.06 42.36 51.23 59.11
darkred ✗ darkred ✗ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 26.33 40.50 50.67 58.67 28.46 41.81 52.14 59.70 22.17 37.93 47.78 56.65
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ 21.33 41.17 53.17 60.33 23.43 47.36 60.45 69.52 17.24 29.06 38.92 42.36
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ 12.00 24.67 33.50 41.50 13.35 27.20 37.28 45.84 9.36 19.70 26.11 33.00
PS 27.67 - - - 30.48 - - - 22.17 - - -
TAG 38.43 - - - 40.10 - - - 35.17 - - -
Table 5: Results table for the ARRAU-AA test set. Refer to text for explanation of duplicated rows.

6.2 Results on the ARRAU corpus

Table 5 shows the performance of different variants of the MR-LSTM with HPs tuned on the ASN corpus (always better than the default HPs), when evaluated on different subparts of the ARRAU-AA: all abstract anaphors, nominal and pronominal ones. HPs were tuned on the ASN corpus for every variant separately, without shuffling of the training data. For the best performing variant, without syntactic information (tag,cut), we report the results with HPs that yielded the best s@1 test score for all anaphors (row 4), when training with those HPs on shuffled training data (row 5), and with HPs that yielded the best s@1 score for pronominal anaphors (row 6).

The MR-LSTM is more successful in resolving nominal than pronominal anaphors, although the training data provides only pronominal ones. This indicates that resolving pronominal abstract anaphora is harder compared to nominal abstract anaphora, such as shell nouns. Moreover, for shell noun resolution in KZH13’s dataset, the MR-LSTM achieved s@1 scores in the range , while the best variant of the model achieves s@1 score for nominal anaphors in ARRAU-AA. Although lower performance is expected, since we do not have specific training data for individual nominals in ARRAU-AA, we suspect that the reason for better performance for shell noun resolution in KZH13 is due to a larger number of positive candidates in ASN (cf. Table 2, rows: antecedents/negatives).

We also note that HPs that yield good performance for resolving nominal anaphors are not necessarily good for pronominal ones (cf. rows 4–6 in Table 5). Since the TPE tuner was tuned on the nominal-only ASN data, this suggest that it would be better to tune HPs for pronominal anaphors on a different dataset or stripping the nouns in ASN.

Contrary to shell noun resolution, omitting syntactic information boosts performance in ARRAU-AA. We conclude that when the model is provided with syntactic information, it learns to pick S-type candidates, but does not continue to learn deeper features to further distinguish them or needs more data to do so. Thus, the model is not able to point to exactly one antecedent, resulting in a lower s@1 score, but does well in picking a few good candidates, which yields good s@2-4 scores. This is what we can observe from row 2 vs. row 6 in Table 5: the MR-LSTM without context embedding (ctx) achieves a comparable s@2 score with the variant that omits syntactic information, but better s@3-4 scores. Further, median occurrence of tags not in {S, VP, ROOT, SBAR} among top-4 ranked candidates is for the full architecture, and when syntactic information is omitted. The need for discriminating capacity of the model is more emphasized in ARRAU-AA, given that the median occurrence of S-type candidates among negatives is for nominal and even for pronominal anaphors, whereas it is for ASN. This is in line with the lower TAG in ARRAU-AA.

Finally, not all parts of the architecture contribute to system performance, contrary to what is observed for reason. For nominal anaphors, the anaphor (aa) and feed-forward layers (ffl1, ffl2) are beneficial, for pronominals only the second ffl.

6.3 Exploring the model

We finally analyze deeper aspects of the model: (1) whether a learned representation between the anaphoric sentence and an antecedent establishes a relation between a specific anaphor we want to resolve and the antecedent and (2) whether the max-margin objective enforces a separation of the joint representations in the shared space.

Figure 3: Visualizing the differences between outputs of the bi-LSTM over time for an anaphoric sentence containing two anaphors.

(1) We claim that by providing embeddings of both the anaphor and the sentence containing the anaphor we ensure that the learned relation between antecedent and anaphoric sentence is dependent on the anaphor under consideration. Fig. 3 illustrates the heatmap for an anaphoric sentence with two anaphors. The i-th column of the heatmap corresponds to absolute differences between the output of the bi-LSTM for the i-th word in the anaphoric sentence when the first vs. second anaphor is resolved. Stronger color indicates larger difference, the blue rectangle represents the column for the head of the first anaphor, the dashed blue rectangle the column for the head of the second anaphor. Clearly, the representations differ when the first vs. second anaphor is being resolved and consequently, joint representations with an antecedent will differ too.

(2) It is known that the max-margin objective separates the best-scoring positive candidate from the best-scoring negative candidate. To investigate what the objective accomplishes in the MR-LSTM model, we analyze the joint representations of candidates and the anaphoric sentence (i.e., outputs of ffl2) after training. For a randomly chosen instance from ARRAU-AA, we plotted outputs of ffl2 with the tSNE algorithm (v.d. Maaten and Hinton, 2008). Fig. 4 illustrates that the joint representation of the first ranked candidate and the anaphoric sentence is clearly separated from other joint representations. This shows that the max-margin objective separates the best scoring positive candidate from the best scoring negative candidate by separating their respective joint representations with the anaphoric sentence.

7 Conclusions

We presented a neural mention-ranking model for the resolution of unconstrained abstract anaphora, and applied it to two datasets with different types of abstract anaphora: the shell noun dataset and a subpart of ARRAU with (pro)nominal abstract anaphora of any type. To our knowledge this work is the first to address the unrestricted abstract anaphora resolution task with a neural network. Our model also outperforms state-of-the-art results on the shell noun dataset.

In this work we explored the use of purely artificially created training data and how far it can bring us. In future work, we plan to investigate mixtures of (more) artificial and natural data from different sources (e.g. ASN, CSN).

On the more challenging ARRAU-AA, we found model variants that surpass the baselines for the entire and the nominal part of ARRAU-AA, although we do not train models on individual (nominal) anaphor training data like the related work for shell noun resolution. However, our model still lags behind for pronominal anaphors. Our results suggest that models for nominal and pronominal anaphors should be learned independently, starting with tuning of HPs on a more suitable devset for pronominal anaphors.

We show that the model can exploit syntactic information to select plausible candidates, but that when it does so, it does not learn how to distinguish candidates of equal syntactic type. By contrast, if the model is not provided with syntactic information, it learns deeper features that enable it to pick the correct antecedent without narrowing down the choice of candidates. Thus, in order to improve performance, the model should be enforced to first select reasonable candidates and then continue to learn features to distinguish them, using a larger training set that is easy to provide.

In future work we will design such a model, and offer it candidates chosen not only from sentences containing the antecedent, but the larger context.

Figure 4: tSNE projection of outputs of ffl2. Labels are the predicted ranks and the constituent tag.

Acknowledgments

This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1. We would like to thank anonymous reviewers for useful comments and especially thank Todor Mihaylov for the model implementations advices and everyone in the Computational Linguistics Group for helpful discussion.

References

  • Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
  • Anand and Hardt (2016) Pranav Anand and Daniel Hardt. 2016. Antecedent selection for sluicing: Structure and content. In

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing

    , pages 1234–1243, Austin, Texas.
  • Asher (1993) Nicholas Asher. 1993. Reference to Abstract Objects in Discourse. Kluwer Academic Publishers.
  • Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NIPS), Granada, Spain.
  • Bowman et al. (2016) Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1466–1477, Berlin, Germany.
  • Byron (2004) Donna K. Byron. 2004. Resolving pronominal reference to abstract entities. Ph.D. thesis, University of Rochester, Rochester, New York.
  • Chopra et al. (2005) Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 539–546. IEEE.
  • Clark and Manning (2015) Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL), Beijing, China.
  • Clark and Manning (2016a) Kevin Clark and Christopher D. Manning. 2016a.

    Deep reinforcement learning for mention-ranking coreference models.

    In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, Texas.
  • Clark and Manning (2016b) Kevin Clark and Christopher D. Manning. 2016b.

    Improving coreference resolution by learning entity-level distributed representations.

    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berling, Germany.
  • Clevert et al. (2016) Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and accurate deep network learning by exponential linear units (elus). In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
  • Das et al. (2016) Arpita Das, Harish Yenala, Manoj Kumar Chinnakotla, and Manish Shrivastava. 2016. Together we stand: Siamese networks for similar question retrieval. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berling, Germany.
  • Dipper and Zinsmeister (2012) Stefanie Dipper and Heike Zinsmeister. 2012. Annotating Abstract Anaphora. Language Resources and Evaluation, 46(1):37–52.
  • Eckert and Strube (2000) Miriam Eckert and Michael Strube. 2000. Dialogue acts, synchronising units and anaphora resolution.
  • Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005. Framewise Phoneme Classification With Bidirectional LSTM And Other Neural Network Architectures. Neural Networks, 18:602–610.
  • He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.

    In 2015 IEEE International Conference on Computer Vision (ICCV).
  • Henaff et al. (2016) Mikael Henaff, Arthur Szlam, and Yann LeCun. 2016. Recurrent orthogonal networks and long-memory tasks. In Proceedings of the the 33rd International Conference on Machine Learning (ICML), New York City, USA.
  • Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
  • Jauhar et al. (2015) Sujay Kumar Jauhar, Raul Guerra, Edgar Gonzàlez Pellicer, and Marta Recasens. 2015. Resolving discourse-deictic pronouns: A two-stage approach to do it. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 299–308, Denver, Colorado.
  • Joachims (2002) Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the 8th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133–142.
  • Józefowicz et al. (2015) Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France.
  • Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, USA.
  • Klein and Manning (2003) Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 423–430. Association for Computational Linguistics.
  • Kolhatkar and Hirst (2014) Varada Kolhatkar and Graeme Hirst. 2014. Resolving shell nouns. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 499–510, Doha, Qatar.
  • Kolhatkar et al. (2013a) Varada Kolhatkar, Heike Zinsmeister, and Graeme Hirst. 2013a. Annotating anaphoric shell nouns with their antecedents. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 112–121, Sofia, Bulgaria.
  • Kolhatkar et al. (2013b) Varada Kolhatkar, Heike Zinsmeister, and Graeme Hirst. 2013b. Interpreting anaphoric shell nouns using antecedents of cataphoric shell nouns as training data. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 300–310, Seattle, Washington, USA.
  • Lu and Ng (2016) Jing Lu and Vincent Ng. 2016. Event Coreference Resolution with Multi-Pass Sieves. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pages 3996–4003, Portoroz.
  • v.d. Maaten and Hinton (2008) Laurens v.d. Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85.
  • Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330.
  • Mueller and Thyagarajan (2016) Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In

    Proceedings of the 13th Conference on Artificial Intelligence (AAAI)

    , pages 2786–2792, Phoenix, Arizona.
  • Müller (2008) Christoph Müller. 2008. Fully Automatic Resolution of It, This and That in Unrestricted Multi-Party Dialog. Ph.D. thesis, Universität Tübingen, Tübingen.
  • Neculoiu et al. (2016) Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. 2016. Learning Text Similarity with Siamese Recurrent Networks. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 148–157, Berlin, Germany.
  • Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013.

    On the difficulty of training recurrent neural networks.

    In Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, USA.
  • Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  • Poesio and Artstein (2008) Massimo Poesio and Ron Artstein. 2008. Anaphoric Annotation in the ARRAU Corpus. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco.
  • Rajagopal et al. (2016) Dheeraj Rajagopal, Eduard Hovy, and Teruko Mitamura. 2016. Unsupervised event coreference for abstract words. In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods, pages 22–26, Austin, Texas.
  • Rutherford et al. (2017) Attapol T. Rutherford, Vera Demberg, and Nianwen Xue. 2017. A Systematic Study of Neural Discourse Models for Implicit Discourse Relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.
  • Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958.
  • Stede and Grishina (2016) Manfred Stede and Yulia Grishina. 2016. Anaphoricity in Connectives: A Case Study on German. In Proceedings of the Coreference Resolution Beyond OntoNotes (CORBON) Workshop, San Diego, California.
  • Strube and Müller (2003) Michael Strube and Christoph Müller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 168–175, Sapporo, Japan.
  • Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL), Beijing, China.
  • Uryupina et al. (2016) Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Kepa J Rodriguez, and Massimo Poesio. 2016. ARRAU: Linguistically-Motivated Annotation of Anaphoric Descriptions. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pages 2058–2062, Portoroz.
  • Webber (1991) Bonnie Lynn Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive processes, 6(2):107–135.
  • Wiseman et al. (2015) Sam Joshua Wiseman, Alexander Matthew Rush, Stuart Merrill Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL), Beijing, China.
  • Zhou and Xu (2015) Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1127–1137, Beijing, China.

Appendix A Pre-processing details

The CSN corpus we obtained from the authors contained tokenized sentences for antecedents and anaphoric sentences. The number of instances differed from the reported numbers in KZH13 in to instances for training, and for testing. The given sentences still contained the antecedent, so we removed it from the sentence and transformed the corresponding shell into ”this ”. An example of this process is: The decision to disconnect the ventilator came after doctors found no brain activity.  This decision came after doctors found no brain activity.

To use pre-trained word embeddings we had to lowercase all the data. As we use an automatic parse to extract all syntactic constituents, due to parser errors, candidates with the same string appeared with different tags. We eliminated duplicates by checking which tag is more frequent for candidates which have the same POS tag of the first word as the duplicated candidate, in the whole dataset. In case duplicated candidates were still occurring, we chose any of them. If such duplicates occur in antecedents, we don’t take such instances in the training data to eliminate noise, or choose any of them for the test data. For the training data we choose instances with an anaphoric sentence length of at least tokens.

All sentences in the batch are padded with a PAD token up to the maximal sentence length in the batch and corresponding hidden states in the LSTM are masked with zeros. To implement the model efficiently in TensorFlow, batches are constructed in such a way that every sentence instance in the batch has the same number of positive candidates and the same number of negative candidates. Note that by this we do not mean that the ratio of positive and negative examples is 1:1.

ctx aa tag cut ffl1 ffl2 # param.
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 95 283 1115 49 2.13 9.40 0.60 0.99 0.71 1928557 3.86 9
darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 140 375 1193 83 7.44 4.41 0.62 0.80 0.82 2842489 3.83 5
darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 61 621 1485 81 8.27 3.27 0.56 0.94 0.99 3502713 3.71 6
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 39 722 1655 - 43.00 7.11 0.89 0.99 0.88 3624949 3.73 1
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ 79 359 1454 65 3.22 7.74 0.70 0.76 0.94 2459362 4.61 2
darkred ✗ darkred ✗ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 38 548 1997 - 82.00 7.14 0.52 0.98 0.80 3345859 4.41 4
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ 39 - 956 96 7.82 8.68 0.78 - 0.59 1567647 4.41 3
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ 71 305 - 94 9.40 5.42 0.52 0.83 - 1593880 4.62 8
Table 6: HPs used for the different architecture variants for the shell noun reason.
ctx aa tag cut ffl1 ffl1 # param.
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 37 684 1081 99 7.40 2.41 0.58 0.86 0.70 0.96 3716655 2.69 2
darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 59 520 520 71 3.06 3.59 0.70 0.83 0.56 0.89 2592937 2.62 1
darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ 45 782 447 31 1.50 6.20 0.85 0.90 0.52 0.87 2300531 2.62 1
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 36 423 417 - 46.00 3.64 0.57 0.86 0.65 0.63 2271652 8.24 2
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 36 423 417 - 46.00 3.64 0.57 0.86 0.65 0.63 2271652 8.24 2
darkgreen ✓ darkgreen ✓ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 70 221 620 - 98 5.15 0.90 0.87 0.84 0.75 2038202 8.26 1
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ darkgreen ✓ 121 355 1955 49 6.48 4.51 0.87 0.90 0.77 0.84 3584370 8.33 1
darkred ✗ darkred ✗ darkred ✗ darkred ✗ darkgreen ✓ darkgreen ✓ 44 622 633 - 96.00 4.29 0.92 0.90 0.53 0.63 2541217 6.62 1
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ darkgreen ✓ 134 - 1489 36 9.51 2.44 0.50 0.97 - 0.57 3575787 6.93 9
darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkgreen ✓ darkred ✗ 41 356 - 44 4.70 5.94 0.66 0.94 0.97 - 1700229 2.64 1
Table 7: HPs used for evaluation on the ARRAU-AA test set.

Appendix B Hyperparameter details

Tables 7 and 7 report the tuned HPs for resolution of the shell noun reason and resolution of abstract anaphors in ARRAU-AA for different model variants. Below is the list of all tunable HPs.

  • the dimensionality of the hidden states in the bi-LSTM,

  • the first feed-forward layer size,

  • the second feed-forward layer size,

  • the dimensionality of the tag embeddings,

  • frequency of words in vocabulary,

  • regularization coefficient,

  • keep probability of outputs of bi-LSTM,

  • keep probability of input,

  • keep probability of outputs of the first feed-forward layer,

  • keep probability of second of the first feed-forward layer,

We additionally report the number of trainable parameters (# param)̇, the average epoch training time using one Nvidia GeForce GTX1080 gpu () and the epoch after which the best score is achieved ().