1 Introduction
Classical supervised machine learning is built on the assumption that the joint probability distribution that features and labels are sourced from does not change during the life cycle of the predictive model: from training to testing and deployment. However, in reality this assumption is broken more often than not: medical diagnostic classifiers are often trained with an oversampling of diseasepositive instances, surveyors are often biased to collecting labelled samples from certain segments of a population, user demographics and preferences change over time on social media and ecommerce sites, etc.
While these are all examples of dataset shift, the nature of these shifts can be quite different. There have been several efforts to create taxonomies of dataset shift [14, 11]. The field of transfer learning offers many methods of learning models for scenarios with awareness of the shift during training. However, often the shift is not yet known during training and it is either too expensive or even impossible to retrain once the shift happens. There are several reasons for it: original training data or training infrastructure might not be available; shift happens so frequently that there is no time to retrain; the kind of shift is such that without having labels in the shifted context there is no hope of learning a better model than the original.
In this work we address multiclass classification scenarios where training a classifier for the shifted deployment context is not possible (due to any of the above reasons), and the only possibility is to postprocess the outputs from an existing classifier that was trained before the shift happened. To succeed, such postprocessing must be guided by some information about the shifted deployment context. In the following, we will assume that we know the overall expected class distribution in the shifted context, at least approximately. For example, consider a medical diagnostic classifier of disease subtypes, which has been trained on the cases of country A, and gets deployed to a different country B. It is common that the distribution of subtypes can vary between countries, but in many cases such information is available. So here many labels are available but not the feature values (country B has data about subtypes in past cases, but no diagnostic markers were measured back then), making training of a new model impossible. Still, the model adjustment methods proposed in this paper can be used to adjust the existing model to match the class distribution in the deployment context. As another example, consider a bank’s fraud detection classifier trained on one type of credit cards and deployed to a new type of credit cards. For new cards there might not yet be enough cases of fraud to train a new classifier, but there might be enough data to estimate the class distribution, that is the prevalence of fraud. The old classifier might predict too few or too many positives on the new data, so it must be adjusted to the new class distribution.
In many application domains, including the above examples of medical diagnostics and fraud detection, it is required that the classifiers would output confidence information in addition to the predicted class. This is supported by most classifiers, as they can be requested to provide for each instance the class probabilities instead of a single label. For example, the feedforward neural networks for classification typically produce class probabilities using the final softmax layer. Such confidence information can then be interpreted by a human expert to choose the action based on the prediction, or feeded into an automatic costsensitive decisionmaking system, which would use the class probability estimates and the misclassification cost information to make costoptimal decisions. Probabilistic classifiers are typically evaluated using
Brier score or logloss (also known as squared error and crossentropy, respectively). Both measures belong to the family of proper scoring rules: measures which are minimized by the true posterior class probabilities produced by the Bayesoptimal model. Proper losses also encourage the model to produce calibrated probabilities, as every proper loss decomposes into calibration loss and refinement loss [9].Our goal is to improve the predictions of a given model in a shifted deployment context, using the information about the expected class distribution in this context, without making any additional assumptions about the type of dataset shift. The idea proposed by Kull et al. [9] is to take advantage of a property that many dataset shift cases share: a difference in the classifier’s average prediction and the expected class distribution of the data. They proposed two different adjustment procedures which transform the predictions to reequalise the average prediction with the expected class distribution, resulting in a theoretically guaranteed reduction of Brier score or logloss. Interestingly, it turned out that different loss measures require different adjustment procedures. They proved that their proposed additive adjustment (additively shifting all predictions, see Section 2 for the definitions) is guaranteed to reduce Brier score, while it can increase logloss in some circumstances. They also proposed multiplicative adjustment (multiplicatively shifting and renormalising all predictions) which is guaranteed to reduce logloss, while it can sometimes increase Brier score. It was proved that if the adjustment procedure is coherent with the proper loss (see Section 2), then the reduction of loss is guaranteed, assuming that the class distribution is known exactly. They introduced the term adjustment loss to refer to the part of calibration loss which can be eliminated by adjustment. Hence, adjustment can be viewed as a weak form of calibration. In the end, it remained open: (1) whether for every proper scoring rule there exists an adjustment procedure that is guaranteed to reduce loss; (2) is there a general way of finding an adjustment procedure to reduce a given proper loss; (3) whether this reduction of loss from adjustment materializes in practice where the new class distribution is only known approximately; (4) how to solve algorithm convergence issues of the multiplicative adjustment method; (5) how to solve the problem of additive adjustment sometimes producing predictions with negative ’probabilities’.
The contributions of our work are the following: (1) we construct a family called BGA (Bounded General Adjustment) of adjustment procedures, with one procedure for each proper loss, and prove that each BGA procedure is guaranteed to reduce the respective proper loss, if the class distribution of the dataset is known; (2) we show that each BGA procedure can be represented as a convex optimization task, leading to a practical and tractable algorithm; (3) we demonstrate experimentally that even if the new class distribution is only known approximately, the proposed BGA methods still usually improve over the unadjusted model; (4) we prove that the BGA procedure of logloss is the same as multiplicative adjustment, thus solving the convergence problems of multiplicative adjustment; (5) we construct another family called UGA (Unbounded General Adjustment) with adjustment procedures that are dominated by the respective BGA methods according to the loss, but are theoretically interesting by being coherent to the respective proper loss in the sense of Kull et al. [9], and by containing the additive adjustment procedure as the UGA for Brier score.
Section 2 of this paper provides the background for this work, covering the specific types of dataset shift and reviewing some popular methods of adapting to them. We also review the family of proper losses, i.e. the loss functions that adjustment is designed for. Section 3 introduces the UGA and BGA families of adjustment procedures and provides the theoretical results of the paper. Section 4 provides experimental evidence for the effectiveness of BGA adjustment in practical settings. Section 5 concludes the paper, reviewing its contributions and proposing open questions.
2 Background and Related Work
2.1 Dataset Shift and Prior Probability Adjustment
In supervised learning, dataset shift can be defined as any change in the joint probability distribution of the feature vector
and label between two data generating processes, that is , where and are the probability distributions before and after the shift, respectively. While the proposed adjustment methods are in principle applicable for any kind of dataset shift, there are differences in performance across different types of shift. According to MorenoTorres et al [11]there are 4 main kinds of shift: covariate shift, prior probability shift, concept shift and other types of shift.
Covariate shift is when the distribution of the covariates/features changes, but the posterior class probabilities do not. At first, this may not seem to be of much interest since the classifiers output estimates of posterior class probabilities and these remain unshifted. However, unless the classifier is Bayesoptimal, then covariate shift can still result in a classifier underperforming [14]. Many cases of covariate shift can be modelled as sample selection bias [8], often addressed by retraining the model on a reweighted training set [15, 13, 7]. Prior probability shift is when the prior class probabilities change, but the likelihoodsdo not. An example of this is down or upsampling of the instances based on their class in the training or testing phase. Given the new class distribution, the posterior class probability predictions can be modified according to Bayes’ theorem to take into account the new prior class probabilities, as shown in
[12]. We will refer to this procedure as the Prior Probability Adjuster (PPA) and the formal definition is as follows:In other types of shift
both conditional probability distributions
and change. The special case where or remains unchanged is called concept shift. Concept shift and other types of shift are in general hard to adapt to, as the relationship between and has changed in an unknown way.2.2 Proper Scoring Rules and Bregman Divergences
The best possible probabilistic classifier is the Bayesoptimal classifier which for any instance outputs its true posterior class probabilities
. When choosing a loss function for evaluating probabilistic classifiers, it is then natural to require that the loss would be minimized when the predictions match the correct posterior probabilities. Loss functions with this property are called proper scoring rules
[5, 10, 9]. Note that throughout the paper we consider multiclass classification with classes and represent class labels as onehot vectors, i.e. the label of class is a vector of zeros and a single at position .Definition 1 (Proper Scoring Rule (or Proper Loss))
In a class classification task a loss function is called a proper scoring rule (or proper loss), if for any probability vectors with and the following inequality holds:
where
is a onehot encoded label randomly drawn from the categorical distribution over
classes with class probabilities represented by vector . The loss function is called strictly proper if the inequality is strict for all .This is a useful definition, but it does not give a very clear idea of what the geometry of these functions looks like. Bregman divergences [4] were developed independently of proper scoring rules and have a constructive definition (note that many authors have the arguments and the other way around, but we use this order to match proper losses).
Definition 2 (Bregman Divergence)
Let be a strictly convex function defined on a convex set such that is differentiable on the relative interior of , . The Bregman divergence is defined as
Previous works [1] have shown that the two concepts are closely related. Every Bregman divergence is a strictly proper scoring rule and every strictly proper scoring rule (within an additive constant) is a Bregman divergence. Best known functions in these families are squared Euclidean distance defined as and KullbackLeiblerdivergence . When used as a scoring rule to measure loss of a prediction against labels, they are typically referred to as Brier Score , and logloss , respectively.
2.3 Adjusted Predictions and Adjustment Procedures
Let us now come to the main scenario of this work, where dataset shift of unknown type occurs after a probabilistic class classifier has been trained. Suppose that we have a test dataset with instances from the postshift distribution. We denote the predictions of the existing probabilistic classifier on these data by , where is the predicted class probability on the th instance, and hence . We further denote the hidden actual labels in the onehot encoded form by , where if the th instance belongs to class , and otherwise . While the actual labels are hidden, we assume that the overall class distribution is known, where . The following theoretical results require to be known exactly, but in the experiments we demonstrate benefits from the proposed adjustment methods also in the case where is known approximately. As discussed in the introduction, examples of such scenarios include medical diagnostics and fraud detection. Before introducing the adjustment procedures we define what we mean by adjusted predictions.
Definition 3 (Adjusted Predictions)
Let be the predictions of a probabilistic class classifier on instances and let be the actual class distribution on these instances. We say that predictions are adjusted on this dataset, if the average prediction is equal to the class proportion for every class , that is .
Essentially, the model provides adjusted predictions on a dataset, if for each class its predicted probabilities on the given data are on average neither under nor overestimated. Note that this definition was presented in [9]
using random variables and expected values, and our definition can be viewed as a finite case where a random instance is drawn from the given dataset.
Consider now the case where the predictions are not adjusted on the given test dataset, and so the estimated class probabilities are on average too large for some class(es) and too small for some other class(es). This raises a question of whether the overall loss (as measured with some proper loss) could be reduced by shifting all predictions by a bit, for example with additive shifting by adding the same constant vector to each prediction vector . The answer is not obvious as in this process some predictions would also be moved further away from their true class. This is in some sense analogous to the case where a regression model is on average over or underestimating its target, as there also for some instances the predictions would become worse after shifting. However, additive shifting still pays off, if the regression results are evaluated by mean squared error. This is well known from the theory of linear models where mean squared error fitting leads to an intercept value such that the average predicted target value on the training set is equal to the actual mean target value (unless regularisation is applied). Since Brier score is essentially the same as mean squared error, it is natural to expect reduction of Brier score after additive shifting of predictions towards the actual class distribution. This is indeed so, and [9] proved that additive adjustment guarantees a reduction of Brier score. Additive adjustment is a method which adds the same constant vector to all prediction vectors to achieve equality between average prediction vector and actual class distribution.
Definition 4 (Additive Adjustment)
Additive adjustment is the function which takes in the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, and outputs adjusted predictions defined as where , , and for each class .
It is easy to see that additive adjustment procedure indeed results in adjusted predictions, as . Note that even if the original predictions are probabilities between and , the additively adjusted predictions can sometimes go out from that range and be negative or larger than . For example, if an instance is predicted to have probability to be in class and at the same time on average the overall proportion of class is overestimated, then and the adjusted prediction is negative. While such predictions are no longer probabilities in the standard sense, these can still be evaluated with Brier score. So it is always true that the overall Brier score on adjusted predictions is lower than on the original predictions, , where the equality holds only when the original predictions are already adjusted, that is . Note that whenever we mention the guaranteed reduction of loss, it always means (even if we do not reemphasise it) that there is no reduction in the special case where the predictions are already adjusted, since then adjustment has no effect.
Additive adjustment is just one possible transformation of unadjusted predictions into adjusted predictions, and there are infinitely many other such transformations. We will refer to these as adjustment procedures. If we have explicitly required the output values to be in the range then we use the term bounded adjustment procedure, otherwise we use the term unbounded adjustment procedure, even if actually the values do not go out from that range.
Definition 5 (Adjustment Procedure)
Adjustment procedure is any function which takes as arguments the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, such that for any and the output predictions are adjusted, that is for each class .
In this definition and also in the rest of the paper we assume silently, that contains valid predictions of a probabilistic classifier, and so for each instance the predicted class probabilities add up to , that is . Similarly, we assume that contains a valid class distribution, with .
Definition 6 (Bounded Adjustment Procedure)
An adjustment procedure is bounded, if for any and the output predictions are in the range , that is for all .
An example of a bounded adjustment procedure is the multiplicative adjustment method proposed in [9], which multiplies the prediction vector componentwise with a constant weight vector and renormalizes the result to add up to 1.
Definition 7 (Multiplicative Adjustment)
Multiplicative adjustment is the function which takes in the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, and outputs adjusted predictions defined as , where are realvalued weights chosen based on and such that the predictions would be adjusted, and are the renormalisation factors defined as .
As proved in [9], the suitable class weights are guaranteed to exist, but finding these weights is a nontrivial task and the algorithm based on coordinate descent proposed in [9] can sometimes fail to converge. In the next Section 3 we will propose a different and more reliable algorithm for multiplicative adjustment.
It turns out that the adjustment procedure should be selected depending on which proper scoring rule is aimed to be minimised. It was proved in [9] that Brier score is guaranteed to be reduced with additive adjustment and logloss with multiplicative adjustment. It was shown that when the ’wrong’ adjustment method is used, then the loss can actually increase. In particular, additive adjustment can increase logloss and multiplicative adjustment can increase Brier score. A sufficient condition for a guaranteed reduction of loss is coherence between the adjustment procedure and the proper loss corresponding to a Bregman divergence. Intuitively, coherence means that the effect of adjustment is the same across instances, where the effect is measured as the difference of divergences of this instance from any fixed class labels and . The exact definition is the following:
Definition 8 (Coherence of Adjustment Procedure and Bregman Divergence [9])
Let be an adjustment procedure and be a Bregman divergence. Then is called to be coherent with if and only if for any predictions and class distribution the following holds for all and :
where is a quantity not depending on , and where and is a onehot encoded vector corresponding to class (with at position and everywhere else).
The following result can be seen as a direct corollary of Theorem 4 in [9].
Theorem 2.1 (Decomposition of Bregman Divergences [9])
Let be a Bregman divergence and let be an adjustment procedure coherent with . Then for any predictions , onehot encoded true labels and class distribution (with ) the following decomposition holds:
(1) 
Due to nonnegativity of this theorem gives a guaranteed reduction of loss, that is the loss on the adjusted probabilities (average divergence between and ) is less than the loss on the original unadjusted probabilities (average divergence between and ), unless the probabilities are already adjusted (). As additive adjustment can be shown to be coherent with the squared Euclidean distance and multiplicative adjustment with KLdivergence [9], the respective guarantees of loss reduction follow from Theorem 8.1.
However, from [9] it remained open: (1) whether for every proper loss there exists an adjustment procedure that is guaranteed to reduce loss; (2) is there a general way of finding an adjustment procedure to reduce a given proper loss; (3) whether this reduction of loss from adjustment materializes in practice where the new class distribution is only known approximately; (4) how to solve algorithm convergence issues of the multiplicative adjustment method; (5) how to solve the problem of additive adjustment sometimes producing predictions with negative ’probabilities’. We will answer these questions in the next section.
3 General Adjustment
The contributions of our work are the following: (1) we construct a family called BGA (Bounded General Adjustment) of adjustment procedures, with one procedure for each proper loss, and prove that each BGA procedure is guaranteed to reduce the respective proper loss, if the class distribution of the dataset is known; (2) we show that each BGA procedure can be represented as a convex optimization task, leading to a practical and tractable algorithm; (3) we demonstrate experimentally that even if the new class distribution is only known approximately, the proposed BGA methods still usually improve over the unadjusted model; (4) we prove that the BGA procedure of logloss is the same as multiplicative adjustment, thus solving the convergence problems of multiplicative adjustment; (5) we construct another family called UGA (Unbounded General Adjustment) with adjustment procedures that are dominated by the respective BGA methods according to the loss, but are theoretically interesting by being coherent to the respective proper loss in the sense of Kull et al. [9], and by containing the additive adjustment procedure as the UGA for Brier score.
Our main contribution is a family of adjustment procedures called BGA (Bounded General Adjustment). We use the term ’general’ to emphasise that it is not a single method, but a family with exactly one adjustment procedure for each proper loss. We will prove that every adjustment procedure of this family is guaranteed to reduce the respective proper loss, assuming that the true class distribution is known exactly. To obtain more theoretical insights and answer the open questions regarding coherence of adjustment procedures with Bregman divergences and proper losses, we define a weaker variant of BGA called UGA (Unbounded General Adjustment). As the name says, these methods can sometimes output predictions that are not in the range . On the other hand, the UGA procedures turn out to be coherent with their corresponding divergence measure, and hence have the decomposition stated in Theorem 8.1 and also guarantee reduced loss. However, UGA procedures have less practical value, as each UGA procedure is dominated by the respective BGA in terms of reductions in loss. We start by defining the UGA procedures, as these are mathematically simpler.
3.1 Unbounded General Adjustment (UGA)
We work here with the same notations as introduced earlier, with denoting the matrix with the outputs of a class probabilistic classifier on a test dataset with instances, and denoting the matrix with the same shape containing onehot encoded actual labels. We denote the unknown true posterior class probabilities on these instances by , again a matrix with the same shape as and .
Our goal is to reduce the loss knowing the overall class distribution , while not having any other information about labels . Due to the defining property of any proper loss, the expected value of this quantity is minimised at . As we know neither nor , we consider instead the set of all possible predictions that are adjusted to , that is . Note that here we do not require , as in this subsection we are working to derive unbounded adjustment methods which allow predictions to go out from the range .
The question is now whether there exists a prediction matrix that is better than (i.e. has lower divergence from ) regardless of what the actual labels are (as a sidenote, also belongs to ). It is not obvious that such exists, as one could suspect that for any there exists some bad such that the original would be closer to than the ‘adjusted’ is.
Now we will define UGA and prove that it outputs adjusted predictions that are indeed better than , regardless of what the actual labels are.
Definition 9 (Unbounded General Adjuster (UGA))
Consider a class classification task with a test dataset of instances, and let be a Bregman divergence. Then the unbounded general adjuster corresponding to is the function defined as follows:
The definition of UGA is correct in the sense that the optimisation task used to define it has a unique optimum. This is because it is a convex optimisation task, as will be explained in Section 3.3. Intuitively,
can be thought of as an infinite hyperplane of adjusted predictions, also containing the unknown
. The original prediction is presumably not adjusted, so it does not belong to . UGA essentially ‘projects’ to the hyperplane , in the sense of finding in the hyperplane which is closest from according to , see the diagram in Figure 1.The following theorem guarantees that the loss is reduced after applying UGA by showing that UGA is coherent with its Bregman divergence.
Theorem 3.1
Let be the unbounded general adjuster corresponding to the Bregman divergence . Then is coherent with .
Proof
The proof of this and following theorems are given in the Supplementary.
The next theorem proves that UGA is actually the one and only adjustment procedure that decomposes in the sense of Theorem 8.1. Therefore, UGA coincides with additive and multiplicative adjustment on Brier score and logloss, respectively.
Theorem 3.2
Let be a Bregman divergence, let be a set of predictions, and be a class distribution over classes. Suppose is such that for any the decomposition of Eq.(2) holds. Then .
As explained in the example of additive adjustment (which is UGA for Brier score), some adjusted predictions can get out from the range . It is clear that a prediction involving negative probabilities cannot be optimal. In the following section we propose the Bounded General Adjuster (BGA) which does not satisfy the decomposition property but is guaranteed to be at least as good as UGA.
3.2 Bounded General Adjustment
For a given class distribution , let us constrain the set of all possible adjusted predictions further, by requiring that all probabilities are nonnegative:
We now propose our bounded general adjuster (BGA), which outputs predictions within .
Definition 10 (Bounded General Adjuster (BGA))
Consider a class classification task with a test dataset of instances, and let be a Bregman divergence. Then the bounded general adjuster corresponding to is the function defined as follows:
Similarly as for UGA, the correctness of BGA is guaranteed by the convexity of the optimisation task, as shown in Section 3.3. BGA solves almost the same optimisation task as UGA, except that instead of considering the whole hyperplane it finds the closest within a bounded subset within the hyperplane. Multiplicative adjustment is the BGA for logloss, because logloss is not defined at all outside the bounds, and hence the UGA for logloss is the same as the BGA for logloss. The following theorem shows that there is a guaranteed reduction of loss after BGA, and the reduction is at least as big as after UGA.
Theorem 3.3
Let be a Bregman divergence, let be a set of predictions, and be a class distribution over classes. Then for any the following holds:
Note that the theorem is even more general than we need and holds for all , not only those which represent label matrices. A corollary of this theorem is that the BGA for Brier score is a new adjustment method dominating over additive adjustment in reducing Brier score. In practice, all practitioners should prefer BGA over UGA when looking to adjust their classifiers. Coherence and decomposition are interesting from a theoretical perspective but from a loss reduction standpoint, BGA is superior to UGA.
3.3 Implementation
Both UGA and BGA are defined through optimisation tasks, which can be shown to be convex. First, the objective function is convex as a sum of convex functions (Bregman divergences are convex in their second argument [2]). Second, the equality constraints that define are linear, making up a convex set. Finally, the inequality constraints of make up a convex set, which after intersecting with remains convex. These properties are sufficient [3] to prove that both the UGA and BGA optimisation tasks are convex.
UGA has only equality constraints, so Newton’s method works fine with it. For Brier score there is a closed form solution [9] of simply adding the difference between the new distribution and the old distribution for every set of k probabilities. BGA computations are a little more difficult due to inequality constraints, therefore requiring interior point methods [3]. While multiplicative adjustment is for logloss both BGA and UGA at the same time, it is easier to calculate it as a UGA, due to not having inequality constraints.
4 Experiments
4.1 Experimental Setup
While our theorems provide loss reduction guarantees when the exact class distribution is known, this is rarely something to be expected in practice. Therefore, the goal of our experiments was to evaluate the proposed adjustment methods in the setting where the class distribution is known approximately. For loss measures we selected Brier score and logloss, which are the two most well known proper losses. As UGA is dominated by BGA, we decided to evaluate only BGA, across a wide range of different types of dataset shift, across different classifier learning algorithms, and across many datasets. We compare the results of BGA with the prior probability adjuster (PPA) introduced in Section 2, as this is to our knowledge the only existing method that inputs nothing else than the predictions of the model and the shifted class distribution. As reviewed in [17], other existing transfer learning methods need either the information about the features or a limited set of labelled data from the shifted context.
To cover a wide range of classifiers and datasets, we opted for using OpenML [16], which contains many datasets, and for each dataset many runs of different learning algorithms. For each run OpenML provides the predicted class probabilities in 10fold crossvalidation. As the predictions in 10fold crossvalidation are obtained with 10 different models trained on different subsets of folds, we compiled the prediction matrix from one fold at a time. From OpenML we downloaded all usersubmitted sets of predictions for both binary and multiclass (up to eight classes) classification tasks, restricting ourselves to tasks with the number of instances in the interval of . Then we discarded every dataset that included a predicted score outside the range . To emphasize, we did not include runs which contain a or a anywhere in the predictions, since logloss becomes infinite in case of errors with full confidence. We discarded datasets with less than instances and sampled datasets with more than instances down to instances. This left us with 590 sets of predictions, each from a different model. These 590 sets of predictions come from 59 different runs from 56 different classification tasks. The list of used datasets and the source code for running the experiments is available in the Supplemental Material.
Shifting.
For each dataset we first identified the majority class(es). After sorting the classes by size decreasingly, the class(es) were considered as majority class(es), where was the smallest possible integer such that . We refer to other class(es) as minority class(es). We then created variants of each dataset by artificially inducing shift in four ways. Each of those shifts has a parameter quantifying the amount of shift, and was chosen uniformly randomly and independently for each adjustment task.
The first method induces prior probability shift by undersampling the majority class(es), reducing their total proportion from to . The second method induces a variety of concept shift by selecting randomly a proportion of instances from majority class(es) and changing their labels into uniformly random minority class labels. The third method induces covariate shift by deleting within class the proportion of the instances with the lowest values of the numeric feature which correlates best with this class label. The fourth method was simply running the other three methods all one after another, which produces an other type of shift.
Approximating the New Class Distribution.
It is unlikely that a practitioner of adjustment would know the exact class distribution of a shifted dataset. To investigate this, we ran our adjustment algorithms on our shifted datasets with not only the exact class distribution, but also eight ‘estimations’ of the class distribution obtained by artificially modifying the correct class distribution into , where was one of eight values , and was chosen to ensure that the sum of class proportions adds up to . If any resulting class proportion left the [0,1] bounds, then the respective adjustment task was skipped.In total, we recorded results for 17527 adjustment tasks resulting from combinations of dataset fold, shift amount, shift method, and estimated class distribution.
Adjustment.
For every combination of shift and for the corresponding nine different class distribution estimations, we adjusted the datasets/predictions using the three abovementioned adjusters: BrierscoreminimizingBGA, loglossminimizingBGA, and PPA. PPA has a simple implementation, but for the general adjusters we used the CVXPY library [6] to perform convex optimization. For BrierscoreminimizingBGA, the selected method of optimization was OSQP (as part of the CVXPY library). For loglossminimizingBGA, we used the ECOS optimizer with the SCS optimizer as backup (under rare conditions the optimizers could numerically fail, occurred 30 times out of 17527). For both Brier score and log loss, we measured the unadjusted loss and the loss after running the dataset through the aforementioned three adjusters.
4.2 Results
On different datasets the effects of our shifting procedures vary and thus we have categorized the shifted datasets into 3 equalsized groups by the amount of squared Euclidean distance between the original and new class distributions (high, medium and low shift). Note that these are correlated to the shift amount parameter , but not determined by it. Figures 2 and 3 both visualise the loss reduction after adjustment in proportion to the loss before adjustment. In these violin plots the part of distributions above stands for reduction of loss and below for increased loss after adjustment. For example, proportional reduction value means that 20% of the loss was eliminated by adjustment. The left side of the leftmost violins in Figure 2 show the case where BGA for Brier score is evaluated on Brier score (with high shift at the top row and low at the bottom). Due to guaranteed reduction in loss the left sides of violins are all above . In contrast, the right side of the same violins shows the effect of PPA adjustment, and PPA can be seen to sometimes increase the loss, while also having lower average reduction of loss (the horizontal black line marking the mean is lower). When the injected estimation error in the class distribution increases (next 4 columns of violins), BGA adjustment can sometimes increase the loss as well, but is on average still reducing loss more than PPA in all of the violin plots. Similar patterns of results can be seen in the right subfigure of Figure 2, where BGA for logloss is compared with PPA, both evaluated on logloss. The mean proportional reduction of loss by BGA is higher than by PPA in 13 out of 15 cases. The bumps in some violins are due to using 4 different types of shift.
Figure 3 demonstrates the differences between BGA aiming to reduce Brier score (left side of each violin) and BGA to reduce log loss (right side of each violin), evaluated on Brier score (left subfigure) and logloss (right subfigure). As seen from the right side of the leftmost violins, BGA aiming to reduce the wrong loss (logloss) can actually increase loss (Brier score), even if the class distribution is known exactly. Therefore, as expected, it is important to adjust by minimising the same divergence that is going to be used to test the method.
5 Conclusion
In this paper we have constructed a family BGA of adjustment procedures aiming to reduce any proper loss of probabilistic classifiers after experiencing dataset shift, using knowledge about the class distribution. We have proved that the loss is guaranteed to reduce, if the class distribution is known exactly. According to our experiments, BGA adjustment to an approximated class distribution often still reduces loss more than prior probability adjustment.
References
 [1] Banerjee, A., Guo, X., Wang, H.: On the optimality of conditional expectation as a Bregman predictor. IEEE Trans. on Information Theory 51(7), 2664–2669 (2005)
 [2] Bauschke, H.H., Borwein, J.M.: Joint and separate convexity of the Bregman distance. In: Studies in Computational Mathematics, vol. 8, pp. 23–36. Elsevier (2001)
 [3] Boyd, S., Vandenberghe, L.: Convex optimization. Cambridge Univ. Press (2004)
 [4] Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics 7(3), 200–217 (1967)
 [5] Dawid, A.P.: The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematics 59(1), 77–93 (2007)
 [6] Diamond, S., Boyd, S.: CVXPY: A Pythonembedded modeling language for convex optimization. Journal of Machine Learning Research 17(83), 1–5 (2016)
 [7] Gretton, A., Smola, A.J., Huang, J., Schmittfull, M., Borgwardt, K.M., Schölkopf, B.: Covariate shift by kernel mean matching. Dataset shift in machine learning pp. 131–160 (2009)
 [8] Hein, M.: Binary classification under sample selection bias. Dataset Shift in Machine Learning (J. Candela, M. Sugiyama, A. Schwaighofer and N. Lawrence, eds.). MIT Press, Cambridge, MA pp. 41–64 (2009)
 [9] Kull, M., Flach, P.: Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In: Joint European Conf. on Machine Learning and Knowledge Discovery in Databases. pp. 68–85. Springer (2015)
 [10] Merkle, E.C., Steyvers, M.: Choosing a strictly proper scoring rule. Decision Analysis 10(4), 292–304 (2013)

[11]
MorenoTorres, J.G., Raeder, T., AlaizRodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern Recognition
45(1), 521–530 (2012)  [12] Saerens, M., Latinne, P., Decaestecker, C.: Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural Comp. 14(1), 21–41 (2002)
 [13] Shimodaira, H.: Improving predictive inference under covariate shift by weighting the loglikelihood function. J. of Stat. Planning and Inference 90(2), 227–244 (2000)
 [14] Storkey, A.: When training and test sets are different: characterizing learning transfer. Dataset shift in machine learning pp. 3–28 (2009)
 [15] Sugiyama, M., Krauledat, M., Müller, K.R.: Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research 8(May), 985–1005 (2007)
 [16] Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: Networked science in machine learning. SIGKDD Explorations 15(2), 49–60 (2013)
 [17] Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. Journal of Big Data 3(1), 9 (2016)
6 Introduction
This supplemental material of the paper Shift Happens: Adjusting Classifiers first lists all the definitions of the paper, followed by theorems and proofs and also the lemmas needed to complete these proofs. The source code and the list of IDs of OpenML tasks and runs that enter our experiments have been provided as file source_code.zip together with the current document.
7 Definitions
Definition 11 (Proper Scoring Rule (or Proper Loss))
In a class classification task a loss function is called a proper scoring rule (or proper loss), if for any probability vectors with and the following inequality holds:
where is a onehot encoded label randomly drawn from the categorical distribution over classes with class probabilities represented by vector . The loss function is called strictly proper if the inequality is strict for all .
Definition 12 (Bregman Divergence)
Let be a strictly convex function defined on a convex set such that is differentiable on the relative interior of , . The Bregman divergence is defined as
Definition 13 (Adjusted Predictions)
Let be the predictions of a probabilistic class classifier on instances and let be the actual class distribution on these instances. We say that predictions are adjusted on this dataset, if the average prediction is equal to the class proportion for every class , that is .
Definition 14 (Additive Adjustment)
Additive adjustment is the function which takes in the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, and outputs adjusted predictions defined as where , , and for each class .
Definition 15 (Adjustment Procedure)
Adjustment procedure is any function which takes as arguments the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, such that for any and the output predictions are adjusted, that is for each class .
Definition 16 (Bounded Adjustment Procedure)
An adjustment procedure is bounded, if for any and the output predictions are in the range , that is for all .
Definition 17 (Multiplicative Adjustment)
Multiplicative adjustment is the function which takes in the predictions of a probabilistic class classifier on instances and the actual class distribution on these instances, and outputs adjusted predictions defined as , where are realvalued weights chosen based on and such that the predictions would be adjusted, and are the renormalisation factors defined as .
Definition 18 (Coherence of Adjustment Procedure and Bregman Divergence [9])
Let be an adjustment procedure and be a Bregman divergence. Then is called to be coherent with if and only if for any predictions and class distribution the following holds for all and :
where is a quantity not depending on , and where and is a onehot encoded vector corresponding to class (with at position and everywhere else).
Definition 19 (Unbounded General Adjuster (UGA))
Consider a class classification task with a test dataset of instances, and let be a Bregman divergence. Then the unbounded general adjuster corresponding to is the function defined as follows:
Definition 20 (Bounded General Adjuster (BGA))
Consider a class classification task with a test dataset of instances, and let be a Bregman divergence. Then the bounded general adjuster corresponding to is the function defined as follows:
8 Theorems and Lemmas with Proofs
Theorem 8.1 (Decomposition of Bregman Divergences [9])
Let be a Bregman divergence and let be an adjustment procedure coherent with . Then for any predictions , onehot encoded true labels and class distribution (with ) the following decomposition holds:
(2) 
Proof
Proof given in cited article [9].
Lemma 1
Let be a Bregman divergence. Then for any the following holds:
where notates the gradient with respect to vector .
Proof
By the definition of Bregman divergence,
The required result follows by taking of each side and simplifying:
Lemma 2
Let be a Bregman divergence. Then for any and the following holds:
Proof
Simplifying from the definition of Bregman divergence gives:
Using Lemma 1 to rewrite the third term yields:
Lemma 3
Let be a Bregman divergence, let be a set of predictions, and be a class distribution over classes. Denoting , the following holds for any :
Proof
In the following we will use a simplified notation and write , , instead of , , . Using Lemma 2 we can write
If we can prove that
then the proof will be complete. So we begin by using the method of Lagrange multipliers to define what each is for each . We rewrite the original argument minimization problem. Keep note our new function will have variables from , and variables from our first constraint, and variables from our second constraint.
Minimum is when
Let’s expand the gradient.
Let’s expand the first term. For simplicity’s sake we will represent as a matrix, but it is a vector in actuality.
We can now see that for each entry in to equal , then each
Since is at the minimum, this implies
We can now write out
We know from the constraints that each row and column of sums to 0.
Comments
There are no comments yet.