Deep Relevance Ranking Using Enhanced Document-Query Interactions

09/05/2018 ∙ by Ryan McDonald, et al. ∙ 0

We explore several new models for document relevance ranking, building upon the Deep Relevance Matching Model (DRMM) of Guo et al. (2016). Unlike DRMM, which uses context-insensitive encodings of terms and query-document term interactions, we inject rich context-sensitive encodings throughout our models, inspired by PACRR's (Hui et al., 2017) convolutional n-gram matching features, but extended in several ways including multiple views of query and document inputs. We test our models on datasets from the BIOASQ question answering challenge (Tsatsaronis et al., 2015) and TREC ROBUST 2004 (Voorhees, 2005), showing they outperform BM25-based baselines, DRMM, and PACRR.



There are no comments yet.


This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Document relevance ranking, also known as ad-hoc retrieval Harman (2005), is the task of ranking documents from a large collection using the query and the text of each document only. This contrasts with standard information retrieval (ir) systems that rely on text-based signals in conjunction with network structure Page et al. (1999); Kleinberg (1999) and/or user feedback Joachims (2002). Text-based ranking is particularly important when (i) click-logs do not exist or are small, and (ii) the network structure of the collection is non-existent or not informative for query-focused relevance. Examples include various domains in digital libraries, e.g., patents Azzopardi et al. (2010) or scientific literature Wu et al. (2015); Tsatsaronis et al. (2015); enterprise search Hawking (2004); and personal search Chirita et al. (2005).

Figure 1: Illustration of drmm Guo et al. (2016) for a query of three terms and a document of terms.

We investigate new deep learning architectures for document relevance ranking, focusing on

term-based interaction models, where query terms (q-terms for brevity) are scored relative to a document’s terms (d-terms) and their scores are aggregated to produce a relevance score for the document. Specifically, we use the Deep Relevance Matching Model (drmm) of guo2016deep (Fig. 1), which was shown to outperform strong ir baselines and other recent deep learning methods. drmm

uses pre-trained word embeddings for q-terms and d-terms, and cosine similarity histograms (outputs of

in Fig. 1), each capturing the similarity of a q-term to all the d-terms of a particular document. The histograms are fed to an mlp (dense layers of Fig. 1) that produces the (document-aware) score of each q-term. Each q-term score is then weighted using a gating mechanism (topmost box nodes in Fig. 1) that examines properties of the q-term to assess its importance for ranking (e.g., common words are less important). The sum of the weighted q-term scores is the relevance score of the document. This ignores entirely the contexts where the terms occur, in contrast to recent position-aware models such as pacrr Hui et al. (2017) or those based on recurrent representations Palangi et al. (2016).

In order to enrich drmm with context-sensitive representations, we need to change fundamentally how q-terms are scored. This is because rich context-sensitive representations – such as input term encodings based on rnns or cnns – require end-to-end training and histogram construction is not differentiable. To account for this we investigate novel query-document interaction mechanisms that are differentiable and show empirically that they are effective ways to enable end-to-end training of context-sensitive drmm models. This is the primary contribution of this paper.

Overall, we explore several extensions to drmm, including: pacrr-like convolutional -gram matching features (3.1); context-sensitive term encodings (3.2); query-focused attention-based document representations (3.3

); pooling to reward denser term matches and turn rich term representations into fixed-width vectors (

3.4); multiple views of terms, e.g., context sensitive, insensitive, exact matches (3.5).

We test our models on data from the bioasq biomedical question answering challenge Tsatsaronis et al. (2015) and trec robust 2004 Voorhees (2005), showing that they outperform strong bm25-based baselines Robertson and Zaragoza (2009), drmm, and pacrr.111The code and data of our experiments, including word embeddings, are available at

2 Related Work

Document ranking has been studied since the dawn of ir; classic term-weighting schemes were designed for this problem Sparck Jones (1988); Robertson and Sparck Jones (1988). With the advent of statistical nlp and statistical ir, probabilistic language and topic modeling were explored Zhai and Lafferty (2001); Wei and Croft (2006), followed recently by deep learning ir methods Lu and Li (2013); Hu et al. (2014); Palangi et al. (2016); Guo et al. (2016); Hui et al. (2017).

Most document relevance ranking methods fall within two categories: representation-based, e.g., palangi2016deep, or interaction-based, e.g., lu2013deep. In the former, representations of the query and document are generated independently. Interaction between the two only happens at the final stage, where a score is generated indicating relevance. End-to-end learning and backpropagation through the network tie the two representations together. In the interaction-based paradigm, explicit encodings between pairs of queries and documents are induced. This allows direct modeling of exact- or near-matching terms (e.g., synonyms), which is crucial for relevance ranking. Indeed, guo2016deep showed that the interaction-based

drmm outperforms previous representation-based methods. On the other hand, interaction-based models are less efficient, since one cannot index a document representation independently of the query. This is less important, though, when relevance ranking methods rerank the top documents returned by a conventional ir engine, which is the scenario we consider here.

One set of our experiments ranks biomedical texts. Several methods have been proposed for the bioasq challenge Tsatsaronis et al. (2015), mostly based on traditional ir techniques. The most related work is of mohan2017deep, who use a deep learning architecture. Unlike our work, they focus on user click data as a supervised signal, and they use context-insensitive representations of document-query interactions. The other dataset we experiment with, trec robust 2004 Voorhees (2005), has been used extensively to evaluate traditional and deep learning ir methods.

Document relevance ranking is also related to other nlp tasks. Passage scoring for question answering Surdeanu et al. (2008) ranks passages by their relevance to the question; several deep networks have been proposed, e.g., tan2015lstm. Short-text matching/ranking is also related and has seen recent deep learning solutions Lu and Li (2013); Hu et al. (2014); Severyn and Moschitti (2015). In document relevance ranking, though, documents are typically much longer than queries, which makes methods from other tasks that consider pairs of short texts not directly applicable.

Our starting point is drmm, to which we add richer representations inspired by pacrr. Hence, we first discuss drmm and pacrr further.

2.1 Drmm

We have already presented an overview of drmm. For gating (topmost box nodes of Fig. 1), guo2016deep use a linear self-attention:

where is the embedding of the -th q-term, or its idf, , and is a weights vector. Gating aims to weight the (document-aware) score of each q-term (outputs of dense layers in Fig. 1) based on the importance of the term. We found that , where ‘;’ is concatenation, was optimal for all drmm-based models.

The crux of the original drmm are the bucketed cosine similarity histograms (outputs of nodes in Fig. 1), each capturing the similarity of a q-term to all the d-terms. In each histogram, each bucket counts the number of d-terms whose cosine similarity to the q-term is within a particular range. Consider a document with three terms, with cosine similarities, s, to a particular q-term , , , respectively. If we used two buckets and , then the input to the dense layers for would be . The fixed number of buckets leads to a fixed-dimension input for the dense layers and makes the model agnostic to different document and query lengths – one of drmm’s main strengths. The main disadvantage is that bucketed similarities are independent of the contexts where terms occur. A q-term ‘regulated’ will have a perfect match with a d-term ‘regulated’, even if the former is ‘up regulated’ and the latter is ‘down regulated’ in context. Also, there is no reward for term matches that preserve word order, or multiple matches within a small window.

2.2 Pacrr

Figure 2: pacrr Hui et al. (2017) and pacrr-drmm.

In pacrr Hui et al. (2017), a query-document term similarity matrix sim is computed (Fig. 2A). Each cell of sim contains the cosine similarity between the embeddings of a q-term and a d-term . To keep the dimensions of sim

fixed across queries and documents of varying lengths, queries are padded to the maximum number of q-terms

, and only the first terms per document are retained.222We use pacrr-firstk, which hui2017pacrr recommend when documents fit in memory, as in our experiments. Then, convolutions (Fig. 2A) of different kernel sizes () are applied to sim to capture -gram query-document similarities. For each size

, multiple kernels (filters) are used. Max pooling is then applied along the dimension of the filters (max value of all filters), followed by row-wise

-max pooling to capture the strongest signals between each q-term and all the d-terms. The resulting matrices are concatenated into a single matrix where each row is a document-aware q-term encoding; the idf of the q-term is also appended, normalized by applying a softmax across the idfs of all the q-terms. Following hui2018copacrr, we concatenate the rows of the resulting matrix into a single vector, which is passed to an mlp (Fig. 2A, dense layers) that produces a query-document relevance score.333hui2017pacrr used an additional lstm, which was later replaced by the final concatanation Hui et al. (2018).

The primary advantage of pacrr over drmm is that it models context via the -gram convolutions, i.e., denser -gram matches and matches preserving word order are encoded. However, this context-sensitivity is weak, as the convolutions operate over the similarity matrix, not directly on terms or even term embeddings. Also, unlike drmm, pacrr

requires padding and hyperparameters for maximum number of q-terms (

) and d-terms (, since the convolutional and dense layers operate over fixed-size matrices and vectors. On the other hand, pacrr is end-to-end trainable – though hui2017pacrr use fixed pre-trained embeddings – unlike drmm where the bucketed histograms are not differentiable.

3 New Relevance Ranking Models

3.1 Pacrr-Drmm

In a drmm-like version of pacrr, instead of using an mlp (dense layers, Fig. 2A) to score the concatenation of all the (document-aware) q-term encodings, the mlp independently scores each q-term encoding (the same mlp for all q-terms, Fig. 2B); the resulting scores are aggregated via a linear layer. This version, pacrr-drmm, performs better than pacrr, using the same number of hidden layers in the mlps. Likely this is due to the fewer parameters of its mlp, which is shared across the q-term representations and operates on shorter input vectors. Indeed, in early experiments pacrr-drmm was less prone to over-fitting.

In pacrr-drmm, the scores of the q-terms (outputs of dense layers, Fig. 2B) are not weighted by a gating mechanism, unlike drmm (Fig. 1). Nevertheless, the idfs of the q-terms, which are appended to the q-term encodings (Fig. 2B), are a form of term-gating (shortcut passing on information about the terms, here their idfs, to upper layers) applied before scoring the q-terms. By contrast, in drmm (Fig. 1) term-gating is applied after q-term scoring, and operates on .

3.2 Context-sensitive Term Encodings

In their original incarnations, drmm and pacrr use pre-trained word embeddings that are insensitive to the context of a particular query or document where a term occurs. This contrasts with the plethora of systems that use context-sensitive word encodings (for each particular occurrence of a word) in virtually all nlp tasks Bahdanau et al. (2014); Plank et al. (2016); Lample et al. (2016). In general, this is achieved via rnns, e.g., lstms Gers et al. (2000), or cnns Bai et al. (2018).

In the ir literature, context-sensitivity is typically viewed through two lenses: term proximity Büttcher et al. (2006) and term dependency Metzler and Croft (2005). The former assumes that the context around a term match is also relevant, whereas the latter aims to capture when multiple terms (e.g., an

-gram) must be matched together. An advantage of neural network architectures like

rnns and cnns is that they can capture both.

In the models below (3.33.4), an encoder produces the context-sensitive encoding of each q-term or d-term from the pre-trained embeddings. To compute this we use a standard bilstm encoding scheme and set the context-sentence encoding as the concatenation of the last layer’s hidden states of the forward and backward lstms at each position. As is common for cnns and even recent rnn term encodings Peters et al. (2018), we use the original term embedding as a residual and combine it with the bilstm encodings. Specifically, if and are the last layer’s hidden states of the left-to-right and right-to-left lstms for term , respectively, then we set the context-sensitive term encoding as:


Since we are adding the original term embedding to each lstm hidden state, we require the dimensionality of the hidden layers to be equal to that of the original embedding. Other methods were tried, including passing all representations through an mlp, but these had no effect on performance.

This is an orthogonal way to incorporate context into the model relative to pacrr. pacrr creates a query-document similarity matrix and computes -gram convolutions over the matrix. Here we incorporate context directly into the term encodings; hence similarities in this space are already context-sensitive. One way to view this difference is the point at which context enters the model – directly during term encoding (Eq. 1) or after term similarity scores have been computed (pacrr, Fig. 2).

3.3 Abel-Drmm

Using the context-sensitive q-term and d-term encodings of 3.2 (Eq. 1), our next extension to drmm is to create document-aware q-term encodings that go beyond bucketed histograms of cosine similarities, the stage in Fig. 1 indicated by . We focus on differentiable encodings to facilitate end-to-end training from inputs to relevance scores.

Figure 3 shows the sub-network that computes the document-aware encoding of a q-term in the new model, given a document of d-terms. We first compute a dot-product444Dot-products have a larger range than other similarity functions, encouraging low entropy attention distributions. attention score for each relative to :


where is the context-sensitive encoding of (Eq. 1). We then sum the context-sensitive encodings of the d-terms, weighted by their attention scores, to produce an attention-based representation of document from the viewpoint of :


The Hadamard product (element-wise multiplication, ) between the (L2-normalized) document representation and the q-term encoding is then computed and used as the fixed-dimension document-aware encoding of (Fig. 3):


The nodes and lower parts of the drmm network of Fig. 1 are now replaced by (multiple copies of) the sub-network of Fig. 3 (one copy per q-term), with the nodes replacing the nodes. We call the resulting model Attention-Based ELement-wise drmm (abel-drmm).

Figure 3: abel-drmm sub-net. From context-aware q-term and d-term encodings (Eq. 1), it generates fixed-dimension document-aware q-term encodings to be used in drmm (Fig. 1, replacing nodes).

Intuitively, if the document contains one or more terms that are similar to , the attention mechanism will have emphasized mostly those terms and, hence, will be similar to , otherwise not. This similarity could have been measured by the cosine similarity between and , but the cosine similarity assigns the same weight to all the dimensions, i.e., to all the (L2 normalized) element-wise products in , which cosine similarity just sums. By using the Hadamard product, we pass on to the upper layers of drmm (the dense layers of Fig. 1), which score each q-term with respect to the document, all the (normalized) element-wise products of , allowing the upper layers to learn which element-wise products (or combinations of them) are important when matching a q-term to the document.

Other element-wise functions can also be used to compare to , instead of the Hadamard product (Eq. 4). For example, a vector containing the squared terms of the Euclidean distance between and could be used instead of . This change had no effect on abel-drmm’s performance on development data. We also tried using instead of , but performance on development data deteriorated.

abel-drmm is agnostic to document length, like drmm. abel-drmm, however, is trainable end-to-end, unlike the original drmm. Still, both models do not reward higher density matches.

3.4 Posit-Drmm

Ideally, we want models to reward both the maximum match between a q-term and a document, but also the average match (between several q-terms and the document) to reward documents that have a higher density of matches. The document-aware q-term scoring of abel-drmm does not account for this, as the attention summation hides whether a single or multiple terms were matched with high similarity. We also want models to be end-to-end trainable, like abel-drmm.

Figure 4 (context-sensitive box) outlines a simple network that produces document-aware q-term encodings, replacing the abel-drmm sub-network of Fig. 3 in the drmm framework. We call the resulting model POoled SImilariTy drmm (posit-drmm). As in abel-drmm, we compute an attention score for each relative to , now using cosine similarity (cf. Eq. 2):


However, we do not use the scores to compute a weighted average of the encodings of the d-terms (cf. Eq. 3), which is also why there is no softmax in above (cf. Eq. 2).555The s still need to be normalized for input to the upper layers, but they do not need to be positive summing to 1. This is why we use cosine similarity in Eq. 5 instead of dot-products combined with softmax of Eq. 2. Instead, we concatenate the attention scores of the d-terms:

and we apply two pooling steps on to create a 2-dimensional document-aware encoding of the q-term (Fig. 4). First max-pooling, which returns the single best match of in the document. Then average pooling over a -max-pooled version of , which represents the average similarity for the top matching terms:

Figure 4: posit-drmm with multiple views (+mv). Three two-dimensional document-aware q-term encodings, one from each view, are produced, concatenated, and used in drmm (Fig. 1, replacing nodes).

posit-drmm has many fewer parameters than the other models. The input to the upper q-term scoring dense layers of the drmm framwork (Fig. 1) for abel-drmm has the same dimensionality as pre-trained term embeddings, on the order of hundreds. By contrast, the input dimensionality here is 2. Hence, posit-drmm does not require deep dense layers, but uses a single layer (depth 1). More information on hyperparameters is provided in Appendix A (supplementary material).

posit-drmm is closely related to pacrr (and pacrr-drmm). Like posit-drmm, pacrr first computes cosine similarities between all q-terms and d-terms (Fig. 2). It then applies -gram convolutions to the similarity matrix to inject context-awareness, and then pooling to create document-aware q-term representations. Instead, posit-drmm relies on the fact that the term encodings are now already context sensitive (Eq. 1) and thus skips the -gram convolutions. Again, this is a choice of when context is injected – during term encoding or after computing similarity scores.

Mohan et al.’s work (2017) is related in the sense that for each q-term, document-aware encodings are built over the best matching (Euclidean distance) d-term. But again, term encodings are context-insensitive pre-trained word embeddings and the model is not trained end-to-end.

3.5 Multiple Views of Terms (+MV)

An extension to abel-drmm and posit-drmm (or any deep model) is to use multiple views of terms. The basic posit-drmm produces a two-dimensional document-aware encoding of each q-term (Fig. 4, context-sensitive box) viewing the terms as their context-sensitive encodings (Eq. 1). Another two-dimensional document-aware q-term encoding can be produced by viewing the terms directly as their pre-trained embeddings without converting them to context-sensitive encodings (Fig. 4, context-insensitive box). A third view uses one-hot vector representations of terms, which allows exact term matches to be modeled, as opposed to near matches in embedding space. Concatenating the outputs of the 3 views, we obtain 6-dimensional document-aware q-term encodings, leading to a model dubbed posit-drmm+mv. An example of this multi-view document-aware query term representation is given in Fig. 5 for a query-document pair from bioasq’s development data.

The multi-view extension of abel-drmm (abel-drmm+mv) is very similar, i.e., it uses context-sensitive term encodings, pre-trained term embeddings, and one-hot term encodings in its three views. The resulting three document-aware q-term embeddings can be summed or concatenated, though we found the former more effective.

3.6 Alternative Network Structures

The new models (3.13.5) were selected by experimenting on development data. Many other extensions were considered, but not ultimately used as they were not beneficial empirically, including deeper and wider rnns or cnn encoders Bai et al. (2018); combining document-aware encodings from all models; different attention mechanisms, e.g., multi-head Vaswani et al. (2017).

Pointer Networks Vinyals et al. (2015) use the attention scores directly to select an input component. posit-drmm does this via max and average pooling, not argmax. We implemented Pointer Networks – argmax over abel-drmm attention to select the best d-term encoding – but empirically this was similar to abel-drmm. Other architectures considered in the literature include the k-nrm model of xiong2017end. This is similar to both abel-drmm and posit-drmm in that it can be viewed as an end-to-end version of drmm. However, it uses kernels over the query-document interaction matrix to produce features per q-term.

The work of pang2017deeprank is highly related and investigates many different structures, specifically aimed at incorporating context-sensitivity. However, unlike our work, Pang et al. first extract contexts (-grams) of documents that match q-terms. Multiple interaction matrices are then constructed for the entire query relative to each of these contexts. These document contexts may match one or more q-terms allowing the model to incorporate term proximity. These interaction matrices can also be constructed using exact string match similar to posit-drmm+mv.

4 Experiments

We experiment with ad-hoc retrieval datasets with hundreds of thousands or millions of documents. As deep learning models are computationally expensive, we first run a traditional ir system666We used Galago (, v.3.10). We removed stop words and applied Krovetz’s stemmer Krovetz (1993). using the bm25 score Robertson and Zaragoza (2009) and then re-rank the top returned documents.

4.1 Methods Compared

All systems use an extension proposed by severyn2015learning, where the relevance score is combined via a linear model with a set of extra features

. We use four extra features: z-score normalized

bm25 score; percentage of q-terms with exact match in the document (regular and idf weighted); and percentage of q-term bigrams matched in the document. The latter three features were taken from mohan2017deep.

In addition to the models of 2.1, 2.2, 3.13.5, we used the following baselines: Standard Okapi bm25 (BM25); and bm25 re-ranked with a linear model over the four extra features (BM25+extra). These ir baselines are very strong and most recently proposed deep learning models do not beat them.777See, for example, Table 2 of guo2016deep. drmm and pacrr are also strong baselines and have shown superior performance over other deep learning models on a variety of data Guo et al. (2016); Hui et al. (2017).888For pacrr/pacrr-drmm, we used/modified the code released by Hui et al. (2017, 2018). We use our own implementation of drmm, which performs roughly the same as guo2016deep, though the results are not directly comparable due to different random partitions of the data.

All hyperparameters were tuned on development data and are available in Appendix A. All models were trained using Adam Kingma and Ba (2014) with batches containing a randomly sampled negative example per positive example999We limit positive examples to be in the top documents. and a pair-wise loss. As the datasets contain only documents marked as relevant, negative examples were sampled from the top documents (returned by bm25) that had not been marked as relevant.

We evaluated the models using the trec ad-hoc retrieval evaluation script101010 (v9.0) focusing on map, Precision@20 and nDCG@20 Manning et al. (2008)

. We trained each model five times with different random seeds and report the mean and standard deviation for each metric on test data; in each run, the model selected had the highest

map on the development data. We also report results for an oracle, which re-ranks the documents returned by bm25 placing all human-annotated relevant documents at the top. To test for statistical significance between two systems, we employed two-tailed stratified shuffling Smucker et al. (2007); Dror et al. (2018) using the model with the highest development map over the five runs per method.

4.2 BioASQ Experiments

Our first experiment used the dataset of the document ranking task of bioasq Tsatsaronis et al. (2015), years 1–5.111111 It contains 2,251 English biomedical questions, each formulated by a biomedical expert, who searched (via PubMed121212 for, and annotated relevant documents. Not all relevant documents were necessarily annotated, but the data includes additional expert relevance judgments made during the official evaluation.131313Our results are, thus, not comparable to those of participating systems, since experts did not consider our outputs.

average over five runs with std. dev. System MAP P@20 nDCG@20 Traditional IR Baselines BM25 BM25+extra Deep Learning Baselines PACRR DRMM Deep Learning with Enhanced Interactions PACRR-DRMM ABEL-DRMM +MV POSIT-DRMM +MV Oracle best run with stat. sig. System MAP P@20 nDCG@20 Traditional IR Baselines BM25 BM25+extra Deep Learning Baselines PACRR DRMM Deep Learning with Enhanced Interactions PACRR-DRMM ABEL-DRMM +MV POSIT-DRMM +MV Oracle
Table 1: Performance on bioasq test data. Statistically significant () difference from bm25 ; bm25+extra; pacrr ; drmm ; pacrr-drmm ; abel-drmm ; abel-drmm+mv .
average over five runs with std. dev. System MAP P@20 nDCG@20 Traditional IR Baselines BM25 BM25+extra Deep Learning Baselines PACRR DRMM Deep Learning with Enhanced Interactions PACRR-DRMM ABEL-DRMM +MV POSIT-DRMM +MV Oracle best run with stat. sig. System MAP P@20 nDCG@20 Traditional IR Baselines BM25 BM25+extra Deep Learning Baselines PACRR DRMM Deep Learning with Enhanced Interactions PACRR-DRMM ABEL-DRMM +MV POSIT-DRMM +MV Oracle
Table 2: Performance on trec robust test data. Statistically significant () difference from bm25 ; bm25+extra; pacrr ; drmm ; pacrr-drmm ; abel-drmm ; abel-drmm+mv .

The document collection consists of approx. 28M ‘articles’ (titles and abstracts only) from the ‘medline/PubMed Baseline 2018’ collection.141414See We discarded the approx. 10M articles that contained only titles, since very few of these were annotated as relevant. For the remaining 18M articles, a document was the concatenation of each title and abstract. Consult Appendix B for further statistics of the dataset. Word embeddings were pre-trained by applying word2vec Mikolov et al. (2013) (see Appendix A for hyper-parameters) to the 28M ‘articles’ of the medline/PubMed collection. idf values were computed over the 18M articles that contained both titles and abstracts.

The 1,751 queries of years 1–4 were used for training, the first 100 queries of year 5 (batch 1) for development, and the remaining 400 queries of year 5 (batches 2–5) as test set. We set , since even using only the top 100 documents of bm25, the oracle scores are high. PubMed articles published after 2015 for the training set, and after 2016 for the development and test sets, were removed from the top (and replaced by lower ranked documents up to ), as these were not available at the time of the human annotation.

Table 1 reports results on the bioasq test set, averaged over five runs as well as the single best run (by development map) with statistical significance. The enhanced models of this paper perform better than bm25 (even with extra features), pacrr, and drmm. There is hardly any difference between pacrr and drmm, but our combination of the two (pacrr-drmm) surpasses them both on average, though the difference is statistically significant () only when comparing to pacrr. Models that use context-sensitive term encodings (abel-drmm, posit-drmm) outperform other models, even pacrr-style models that incorporate context at later stages in the network. This is true both on average and by statistical significance over the best run. The best model on average is posit-drmm+mv, though it is not significantly different than posit-drmm.

4.3 TREC Robust 2004 Experiments

Our primary experiments were on the bioasq dataset as it has one of the largest sets of queries (with manually constructed relevance judgments) and document collections, making it a particularly realistic dataset. However, in order to ground our models in past work we also ran experiments on trec robust 2004 Voorhees (2005), which is a common benchmark. It contains 250 queries151515We used the ‘title’ fields of the queries. and 528K documents. As this dataset is quite small, we used a 5-fold cross-validation. In each fold, approx.  of the queries were used for training, for development, and for testing. We applied word2vec to the 528K documents to obtain pre-trained embeddings. idf values were computed over the same corpus. Here we used , as the oracle scores for were low.

Table 2 shows the trec robust results, which largely mirror those of bioasq. posit-drmm+mv is still the best model, though again not significantly different than posit-drmm. Furthermore, abel-drmm and posit-drmm are clearly better than the deep learning baselines,161616The results we report for our implementation of drmm are slightly different than those of guo2016deep. There are a number of reasons for why this might be the case: there is no standard split of the data; non-standard preprocessing of the documents; the original drmm paper reranks the top documents returned by Query Likelihood and not BM25. but unlike bioasq, there is no statistically significant difference between pacrr-drmm and the two deep learning baselines. Even though the scores are quite close (particularly map) both abel-drmm and posit-drmm are statistically different from pacrr-drmm, which was not the case for bioasq. abel-drmm+mv is significantly different than abel-drmm on the best run for map and nDCG@20, unlike bioasq where there was no statistically significant difference between the two methods. However, on average over 5 runs, the systems show little difference.

5 Discussion

Figure 5: Left: Cosine similarities (posit-drmm attention) of query and document terms, with context-sensitive, context-insensitive, and exact match views of the terms (top to bottom). Document truncated to 50 words. White is stronger. Right: Corresponding posit-drmm+mv 6-dimensional document-aware query term encodings.
Figure 6: posit-drmm+mv 6-dimensional document-aware q-term encodings for ‘Does autophagy induce apoptosis defense?’ and the same document as Fig. 5.

An interesting question is how well the deep models do without the extra features. For bioasq, the best model’s (posit-drmm+mv) map score drops from 48.1 to 46.2 on the development set, which is higher than the bm25 baseline (43.7), but on-par with bm25+extra (46.0). We should note, however, that on this set, the drmm baseline without the extra features (which include bm25) is actually lower than bm25 (map 42.5), though it is obviously adding a useful signal, since drmm with the extra features performs better (46.5).

We also tested the contribution of context-sensitive term encodings (Eq. 1). Without them, i.e., using directly the pre-trained embeddings, map on bioasq development data dropped from 47.6 to 46.3, and from 48.1 to 47.0 for abel-drmm and posit-drmm, respectively.

Fig. 5 shows the cosine similarities (attention scores, Eq. 5) between q-terms and d-terms, using term encodings of the three views (Fig. 4), for a query “Does Vitamin D induce autophagy?” and a relevant document from the bioasq development data. posit-drmm indeed marks this as relevant. In the similarities of the context-insensitive view (middle left box) we see multiple matches around ‘vitamin d’ and ‘induce autophagy’. The former is an exact match (white squares in lower left box) and the latter a soft match. The context-sensitive view (upper left box) smooths things out and one can see a straight diagonal white line matching ‘vitamin d induce autophagy’. The right box of Fig. 5 shows the 6 components (Fig. 4) of the document-aware q-term encodings. Although some terms are not matched exactly, the context sensitive max and average pooled components (two left-most columns) are high for all q-terms. Interestingly, ‘induce’ and ‘induces’ are not an exact match (leading to black cells for ‘induce’ in the two right-most columns) and the corresponding context-insensitive component of (third cell from left) is low. However, the two components of the context-sensitive view (two left-most cells of ‘induce’) are high, esp. the max-pooling component (left-most).Finally, ‘vitamin d’ has multiple matches leading to a high average k-max pooled value, which indicates that the importance of that phrase in the document.

Fig. 6 shows the 6 components of the document-aware q-term encodings for another query and the same document, which is now irrelevant. In the max pooling columns of the exact match and context-insensitive view (columns 3, 5), the values look quite similar to those of Fig. 5. However, posit-drmm scores this query-document pair low for two reasons. First, in the average-k-max pooling columns (columns 2, 4, 6) we get lower values than Fig. 5, indicating that there is less support for this pair in terms of density. Second, the context sensitive values (columns 1, 2) are much worse, indicating that even though many exact matches exist, in context, the meaning is not the same.

We conclude by noting there is still quite a large gap between the current best models and the oracle re-ranking scores. Thus, there is head room for improvements through more data or better models.


We thank the reviewers for their constructive feedback that greatly improved this work. Oscar Täckström gave thorough input on an early draft of this work. Finally, aueb’s nlp group provided many suggestions over the course of the work.


  • Azzopardi et al. (2010) Leif Azzopardi, Wim Vanderbauwhede, and Hideo Joho. 2010. Search system requirements of patent analysts. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 775–776, Geneva, Switzerland.
  • Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  • Bai et al. (2018) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.
  • Büttcher et al. (2006) Stefan Büttcher, Charles LA Clarke, and Brad Lushman. 2006. Term proximity scoring for ad-hoc retrieval on very large text collections. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 621–622. ACM.
  • Chirita et al. (2005) Paul Alexandru Chirita, Rita Gavriloaie, Stefania Ghita, Wolfgang Nejdl, and Raluca Paiu. 2005. Activity based metadata for semantic desktop search. In European Semantic Web Conference, pages 439–454, Heraklion, Greece.
  • Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.

    The hitchhiker’s guide to testing statistical significance in natural language processing.

    In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, Australia.
  • Gers et al. (2000) Felix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451–2471.
  • Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 55–64, Indianapolis, IN.
  • Harman (2005) Donna K Harman. 2005. The TREC ad hoc experiments. Technical report, National Institute of Standards and Technology (NIST).
  • Hawking (2004) David Hawking. 2004. Challenges in enterprise search. In Proceedings of the 15th Australasian Database Conference-Volume 27, pages 15–24, Dunedin, New Zealand.
  • Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems 27, pages 2042–2050.
  • Hui et al. (2017) Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2017. PACRR: A position-aware neural IR model for relevance matching. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1049–1058, Copenhagen, Denmark.
  • Hui et al. (2018) Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2018. Co-PACRR: A context-aware neural IR model for ad-hoc retrieval. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pages 279–287, Marina Del Rey, CA.
  • Joachims (2002) Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 133–142, Edmonton, Canada.
  • Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  • Kleinberg (1999) Jon M. Kleinberg. 1999. Authoritative sources in a hyperlinked environment. Journal of the ACM (JACM), 46(5):604–632.
  • Krovetz (1993) Robert Krovetz. 1993. Viewing morphology as an inference process. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 191–202, Pittsburgh, PA.
  • Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.

    Neural architectures for named entity recognition.

    In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270, San Diego, CA.
  • Lu and Li (2013) Zhengdong Lu and Hang Li. 2013. A deep architecture for matching short texts. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, pages 1367–1375, Lake Tahoe, NV.
  • Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press.
  • Metzler and Croft (2005) Donald Metzler and W Bruce Croft. 2005. A markov random field model for term dependencies. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 472–479. ACM.
  • Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, pages 3111–3119, Lake Tahoe, Nevada.
  • Mohan et al. (2017) Sunil Mohan, Nicolas Fiorini, Sun Kim, and Zhiyong Lu. 2017. Deep learning for biomedical information retrieval: Learning textual relevance from click logs. In BioNLP 2017, pages 222–231.
  • Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
  • Palangi et al. (2016) Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016.

    Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval.

    IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 24(4):694–707.
  • Pang et al. (2017) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. 2017. Deeprank: A new deep architecture for relevance ranking in information retrieval. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 257–266. ACM.
  • Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227–2237.
  • Plank et al. (2016) Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin, Germany.
  • Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.
  • Robertson and Sparck Jones (1988) Stephen E. Robertson and Karen Sparck Jones. 1988. Document retrieval systems. chapter Relevance Weighting of Search Terms, pages 143–160. Taylor Graham Publishing, London, UK.
  • Severyn and Moschitti (2015) Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 373–382, Santiago, Chile.
  • Smucker et al. (2007) Mark D Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 623–632. ACM.
  • Sparck Jones (1988) Karen Sparck Jones. 1988. Document retrieval systems. chapter A Statistical Interpretation of Term Specificity and Its Application in Retrieval, pages 132–142. Taylor Graham Publishing, London, UK.
  • Surdeanu et al. (2008) Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2008. Learning to rank answers on large online QA collections. In Proceedings of the 46th Annual Meeting for the Association for Computational Linguistics: Human Language Technologies (ACL-08: HLT), pages 719–727, Columbus, OH.
  • Tan et al. (2015) Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. LSTM-based deep learning models for non-factoid answer selection. arXiv preprint arXiv:1511.04108.
  • Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015. An overview of the BioASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16(1):138.
  • Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008, Long Beach, CA.
  • Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems 28, pages 2692–2700, Montreal, Canada.
  • Voorhees (2005) Ellen M Voorhees. 2005. The TREC robust retrieval track. ACM SIGIR Forum, 39(1):11–20.
  • Wei and Croft (2006) Xing Wei and W Bruce Croft. 2006. LDA-based document models for ad-hoc retrieval. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 178–185, Seattle, WA.
  • Wu et al. (2015) Jian Wu, Kyle Mark Williams, Hung-Hsuan Chen, Madian Khabsa, Cornelia Caragea, Suppawong Tuarob, Alexander G Ororbia, Douglas Jordan, Prasenjit Mitra, and C Lee Giles. 2015. Citeseerx: AI in a digital library search engine. AI Magazine, 36(3):35–48.
  • Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 55–64. ACM.
  • Zhai and Lafferty (2001) Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 334–342, New Orleans, LA.