Learning to rank is fundamental to information retrieval, E-commerce, and many other applications, for ranking items . In this work we focus on document retrieval without loss of generality. Document retrieval (i.e., document ranking) has applications in large-scale item search engines, which can generally be described as follows: There is a collection of documents (items). Given a query (e.g. a query entered by a user in the search engine), the ranking function assigns a score to each document, quantifying the relative relevancy of the document to the query. The documents are ranked in the descending order based on these scores and the top ranked ones are returned.
Traditional approaches rank documents based on unsupervised models of words appearing in the documents and query and do not need any training 
. The rise of using machine learning to learn ranking models has been due to the availability of more signals related to relevance of documents, such as click items or search log data.
The bulk of machine learning methods for learning to rank can roughly be categorized as pointwise, pairwise and listwise methods. Pointwise methods cast the ranking problem as a regression problem for predicting relevance scores  or a multiple ordinal classification to predict categorical relevance levels . Pairwise approaches take document pairs as instances in learning, and formalize the learning-to-rank problem as that of classification. More precisely, they collect document pairs to query the relative ranking from the underlying unknown ranking lists. They then train a classification model with the labeled data and make use of the classification model in ranking 
. Finally, listwise methods use ranked document lists instead of document pairs as instances in learning and define an optimization loss function over the entire ranked list(s).
In this paper, we propose a new framework for supervised learning to rank
. Specifically, we define a scoring function that maps the input vector of features for a document to the probability parameters of a categorical distribution, where each category represents the relative relevance of the input document to the query. We then define the objective function of learning-to-rank as the expectation of a loss function, which determines the distance between predicted and true relevance labels of the input document, with respect to the scoring function categorical distribution. To achieve a rich family of ranking algorithms, we employ neural networks as scoring functions.
Due to its novel discrete structure, we exploit stochastic gradient based optimization to learn the parameters of the scoring function. The main difficulty arises when backpropagating the gradients through categorical variables. The recently proposed augment-REINFORCE-swap-merge (ARSM) gradient estimator provides a natural solution with low varaince unbiased gradient updates during the training of our proposed learning-to-rank framework. ARSM first uses variable augmentation, REINFORCE , and Rao-Blackwellization  to re-express the gradient as an expectation under the Dirichlet distribution, then uses variable swapping to construct differently expressed but equivalent expectations, and finally shares common random numbers between these expectations to achieve significant variance reduction.
The proposed framework, hereby referred to as ARSM-L2R, has a main advantage over the existing learning-to-rank methods. More precisely, due to the utilization of ARSM gradient estimator, the loss function assessing the distance between predicted and true document relevance labels needs not to be differentiable. This significantly increase the choices of loss functions that can be employed. Specifically, in our experiments, we optimize the truncated normalized discounted cumulative gain (NDCG) .
Comprehensive experiments conducted on benchmark datasets demonstrate that our proposed ARSM-L2R method achieves better or comparable results with pairwise and listwise approaches in terms of common ranking metrics such as truncated NDCG and mean average precision (MAP).
The remainder of this paper is organized as follows. In Section 2, we present the methodology, including the new formulation of ARSM-L2R for supervised learning to rank, and its parameter estimation using Monte Carlo gradient estimates. Section 3 provides comprehensive experimental results for comparison with several existing learning-to-rank methods. The paper is concluded in Section 4.
2.1 Supervised learning to rank
In the supervised learning-to-rank setting, a set of queries is given. Each query is associated with a list of documents , where and denote the th document and size of respectively. In addition, a list of scores is available for each list of documents . The score represents the relevance degree of document to query , and can be a judgment score explicitly or implicitly given by humans . Higher scores imply more relevant documents.
For each query-document pair , a -dimensional vector of features is constructed. The training set is represented as . The objective of learning is to create ranking functions that map the input query-document features to scores resembling the true relevant scores. In the following discussions, we drop the query index to avoid cluttering the notations.
In this paper, we formulate the supervised learning-to-rank problem as maximizing an objective, expressed as an expectation over multivariate categorical variables. More specifically, given documents for a query, let denote the relevance label for th document, where is the number of possible levels of relevance for each document. In our proposed generative model, each is distributed according to a categorical distribution whose probabilities are constructed based on a scoring function parameterized by :
is the softmax function. We use multi-layer perceptrons (MLPs) as scoring functions, thuscorresponds to the collection of weight matrices of MLPs. For each realization of categorical variables , we employ a loss function to determine their distance from the true labels . We then define the learning-to-rank optimization problem as finding:
where can be any loss function measuring the dissimilarity of two vectors of ordinal labels. We resort to stochastic gradient based methods to solve the optimization problem in (2). Backpropagating the gradient through discrete latent variables have been recently studied extensively [17, 20, 8]. For optimizing (2
), the challenge lies in developing a low-variance and unbiased estimator for its gradient with respect to, which is denoted by .
2.2 ARSM gradient estimator
We employ Augment-REINFORCE-Swap-Merge (ARSM) gradient estimator for training the scoring functions described in the previous section. To describe this algorithm, we start by the simple objective function with respect to a univariate categorical variable, where is the reward function and . In the augmentation step, the gradient of can be expressed as an expectation under a Dirichlet distribution as
Given the vector , we denote the vector obtained after swapping th and th elements of as , where , and for we have . Exploiting the symmetrical property , and sharing common random numbers between different expectations to potentially significantly reduce Monte Carlo integration variance leads to another unbiased estimator referred as ARS estimator:
where and is the reference category. Finally, the ARS estimator can be further improved by considering all swap operations, and adding a merge step to construct the ARSM estimator as
2.3 ARSM for learning to rank
To employ ARSM for learning to rank, we need to consider the optimization problem with respect to the multivariate categorical variables . Let denote the multivariate swapping whose elements are defined, similar to those in (4) and (5), as . Then the multivariate extension of ARSM gradient estimator for the learning-to-rank objective in (2) can be expressed as :
where . Since we define the categorical distribution parameter in terms of a neural network with parameters
, the final gradients are computed using the chain rule as
The estimated gradients are then utilized in a stochastic optimization process to learn the model parameters. Algorithm 1 summarizes the parameter learning for ARSM-L2R.
2.4 Loss function and rank prediction
The loss function in (2) measures the dissimilarity between predicted categorical labels and the true labels . In this work, we utilize the negative truncated NDCG as the loss function of ARSM-L2R. The calculation of NDCG only relies on the sorting of the predicted labels , and the true labels . Furthermore, our experiments show that setting the number of possible levels of relevance to be higher than the number of true levels in improves the performance of ARSM-L2R. Hence, for all experiments in this paper we set .
After the parameters of the scoring function are learned in the training phase, the probability of different levels of relevance for the test documents can be calculated by simply passing the documents features through the scoring function. We then construct the final scores of the test documents by a weighted combination of these probabilities, and sort the documents based on these scores. More precisely, given the probability of different labels for a test document, we calculate its overall ranking score as , where and higher values of correspond to more relevant levels. Our experiments show that the performance of ARSM-L2R is not sensitive to the choice of the weight combination scheme.
We evaluate the performance of ARSM-L2R on two widely tested benchmark datasets, including a query set from Million Query track of TREC 2007, denoted as MQ2007 , as well as the OHSUMED dataset . Each dataset consists of queries, corresponding retrieved documents and labels provided by human experts. The possible relevance labels for each document are relevant, partially relevant, and not relevant. We use the 5-fold partitions provided in the original dataset for 5-fold cross validation in the experiments. In each fold, there are three subsets for learning: training set, validation set and testing set. The properties of these learning to rank datasets are presented in Table 1.
3.3 Evaluation metrics
We use two popular learning-to-rank scoring functions to compare the predicted rankings of the test documents with their true rankings: truncated Normalized Discounted Cumulative Gain (NDCG@)  and Mean Average Precision (MAP) . NDCG (DCG) has the effect of giving high scores to the ranking lists in which relevant documents are ranked high. Average Precision (AP) represents the averaged precision over all the positions of documents with relevant label for query . Denoting the ranking list on , MAP is defined as
where . NDCG@ is calculated by
where if represents the true ranking list of , then . here represents the truncation level.
3.4 Implementation details
For the scoring function neural network, we employ a fully connected neural network with one hidden layer of 500 units and the tanh
nonlinear activation function. We initialize the wights of the neural network byGlorot method , and train ARSM-L2R using the Adam optimizer  with a learning rate of
. The algorithm is run for a total of 2000 epochs, and the ranking metrics on the validation sets are monitored for choosing the best performing neural network weights. ARSM-L2R is implemented inTensorflow .
3.5 Results and discussions
, respectively. Our ARSM-L2R achieves the highest NDCG@1 and NDCG@3 on the MQ2007 dataset. On the OHSUMED dataset ARSM-L2R has a significantly higher NDCG@1 compared with all the other methods tested. It also shows the best MAP on this dataset. It is worth mentioning that NDCG@1 is one of the most important metrics for ranking systems, since it quantifies the relevance of the top ranked item. It is interesting to note that our method only optimizes a rough approximation of the evaluation metric NDCG, but shows the best performance on both two metrics for each dataset and comparable results for the rest of the metrics on the datasets. Previous works have shown that pointwise approaches cannot achieve as good performance as listwise approaches. But our proposed method achieves better or comparable performance due to utilizing a loss function more directly related to ranking performance and also taking advantage of unbiased and low-variance gradient estimation.
We have developed a new supervised learning-to-rank model—ARSM-L2R—that generates relevance labels based on a categorical model with probabilities estimated by a MLP. The training objective is optimized with respect to the multivariate categorical variables with an unbiased and low-variance gradient estimator, ARSM. The experimental results show that ARSM-L2R achieves better or comparable results with pairwise and listwise approaches.
-  (2016) Tensorflow: a system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI’16, pp. 265–283. External Links: Cited by: §3.4.
-  (1999) Modern information retrieval. Vol. 463, ACM press New York. Cited by: §3.3.
-  (2007) Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129–136. Cited by: §1, §2.1, §3.2.
-  (1996) Rao-Blackwellisation of sampling schemes. Biometrika 83 (1), pp. 81–94. Cited by: §1.
Subset ranking using regression.
International Conference on Computational Learning Theory, pp. 605–619. Cited by: §1.
-  (2010) Search engines: Information retrieval in practice. Vol. 520, Addison-Wesley Reading. Cited by: §1.
Understanding the difficulty of training deep feedforward neural networks.
Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256. Cited by: §3.4.
-  (2017) Backpropagation through the void: optimizing control variates for black-box gradient estimation. arXiv preprint arXiv:1711.00123. Cited by: §2.1.
-  (2000) IR evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 41–48. Cited by: §1, §3.3.
-  (2002) Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 133–142. Cited by: §1.
-  (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.4.
-  (2014) Large-scale linear ranksvm. Neural computation 26 (4), pp. 781–817. Cited by: §3.2.
Mcrank: learning to rank using multiple classification and gradient boosting. In Advances in neural information processing systems, pp. 897–904. Cited by: §1.
-  (2009) Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval 3 (3), pp. 225–331. Cited by: §1, §1.
-  (2010) LETOR: a benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13 (4), pp. 346–374. Cited by: §3.1.
-  (2013) Introducing LETOR 4.0 datasets. CoRR abs/1306.2597. External Links: Cited by: §3.1.
-  (2017) Rebar: low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Information Processing Systems, pp. 2627–2636. Cited by: §2.1.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 (3-4), pp. 229–256. Cited by: §1.
-  (2007) AdaRank: A boosting algorithm for information retrieval. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 391–398. Cited by: §3.2.
-  (2019) ARSM: augment-reinforce-swap-merge estimator for gradient backpropagation through categorical variables. arXiv preprint arXiv:1905.01413. Cited by: §1, §2.1, §2.3.