Journal ranking should depend on the level of aggregation

04/08/2019
by   László Csató, et al.
Corvinus University of Budapest
0

Journal ranking, that is, placing journals within their respective fields, is an important research evaluation tool. Several indices have been suggested for this purpose, typically on the basis of citation graph. We follow an axiomatic approach and find an impossibility theorem: any self-consistent ranking method, which satisfies a natural monotonicity property, should depend on the level of aggregation. Our result presents a trade-off between two axiomatic properties and reveals a dilemma of aggregation.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

06/09/2020

The utilization of paper-level classification system on the evaluation of journal impact

CAS Journal Ranking, a ranking system of journals based on the bibliomet...
06/29/2021

Disturbance of questionable publishing to academia

Questionable publications have been accused of "greedy" practices; howev...
05/31/2022

Weight Set Decomposition for Weighted Rank Aggregation: An interpretable and visual decision support tool

The problem of interpreting or aggregating multiple rankings is common t...
11/14/2019

Beyond Pairwise Comparisons in Social Choice: A Setwise Kemeny Aggregation Problem

In this paper, we advocate the use of setwise contests for aggregating a...
01/15/2020

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

Despite the increasing use of citation-based metrics for research evalua...
03/06/2018

Impact Factors and the Central Limit Theorem: Why Citation Averages Are Scale Dependent

Citation averages, and Impact Factors (IFs) in particular, are sensitive...
07/27/2019

Coping with the delineation of emerging fields: Nanoscience and Nanotechnology as a case study

Proper field delineation plays an important role in scientometric studie...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Scientometrics, the measurement of the quality and quantity of academic research, plays an increasing role in the evaluation of researchers and research proposals. It is not surprising that a number of indices have been suggested to assess intellectual influence, and now a plethora of ranking methods are available to measure the performance of journals and scholars (Palacios-Huerta and Volij, 2014). Consequently, an axiomatic approach is worth applying: the introduction of some reasonable axioms or conditions narrows the set of appropriate methods, reveals their crucial properties, and allows for their comparison.

An important contribution of similar analyses can be an axiomatic characterisation, meaning that a set of properties uniquely determine a preference vector. For example,

Palacios-Huerta and Volij (2004) give a characterisation of the invariant method, while Demange (2014)

provides a characterisation of the handicap method, both of them used to rank journals. Results for citation indices are probably even more abundant, including characterisations of the

-index (Kongo, 2014; Marchant, 2009; Miroiu, 2013; Quesada, 2010, 2011a, 2011b; Woeginger, 2008b), the -index (Woeginger, 2008a; Quesada, 2011a), the Euclidean index (Perry and Reny, 2016), or a class of step-based indices (Chambers and Miller, 2014), among others. de la Vega and Volij (2018) characterise scholar rankings admitting a measure representation. There are also axiomatic comparisons of bibliometric indices (Bouyssou and Marchant, 2014, 2016).

However, the above works seldom uncover the inevitable trade-offs between different natural requirements, which can be achieved mainly by impossibility theorems. Similar results are well-established in social choice theory since Arrow’s impossibility theorem (Arrow, 1950) and the Gibbard-Satterthwaite theorem (Gibbard, 1973; Satterthwaite, 1975; Duggan and Schwartz, 2000) but not so widely used in scientometrics.

We provide an impossibility result in journal ranking. In particular, it will be proved that two axioms, invariance to aggregation and self-consistency, cannot be satisfied simultaneously even on a substantially restricted domain of citation graphs. Invariance to aggregation means that the order of two journals is not influenced by the level of aggregation among the remaining journals, while self-consistency, introduced by Chebotarev and Shamis (1997), is a kind of monotonicity property, responsible for some impossibility theorems in ranking from paired comparisons (Csató, 2019a, b).

The paper is organised as follows. Our setting and notations are introduced in Section 2. Section 3 motivates and defines the two axioms, which turn out to be incompatible in Section 4. Section 5 summarises the main findings and concludes.


2 The journal ranking problem

A journal ranking problem consists of a group of journals and their respective citation records (Palacios-Huerta and Volij, 2014). Let , be a non-empty finite set of journals and be a nonnegative citation matrix for . The entry can be directly the number of citations that journal received from journal , or any transformation of this value, for example, by using exponentially decreasing weights for older citations. is assumed for all .

The pair is called a journal ranking problem. The set of journal ranking problems with journals () is denoted by .

The aim is to aggregate the opinions given in the citation matrix into an “objective judgement”. Formally, a scoring procedure is a function that takes a journal ranking problem and returns a rating for each journal , representing this objective judgement.

A scoring method immediately induces a ranking for the journals of (a transitive and complete weak order on the set of ): means that journal is ranked weakly above , denoted by . The symmetric and asymmetric parts of are denoted by and , respectively: if both and hold, while if holds but does not hold.

A journal ranking problem has the symmetric matches matrix such that is the number of the citations between the journals and in both directions, which can be called the number of matches between them in the terminology of sports (Kóczy and Strobel, 2010; Csató, 2015).

It is sometimes convenient to consider not a general problem, arising from complicated networks of citations, but only a simple one.

A journal ranking problem is called balanced if for all . The set of balanced journal ranking problems is denoted by . In a balanced journal ranking problem, all journals have the same number of matches.


A journal ranking problem is called unweighted if for all . The set of unweighted journal ranking problems is denoted by . In an unweighted journal ranking problem, either there is no citations, or there exists only one citation between any pair of journals.

The subsets of balanced and unweighted problems restrict the matches matrix .

A journal ranking problem is called extremal if for all . The set of extremal journal ranking problems is denoted by . In an extremal journal ranking problem, only three cases are allowed in the comparison of journals and : there are citations only for or , or they are tied with respect to mutual citations.

Any intersection of these special classes can be considered, too.


3 Axioms of journal ranking

In this section two properties, a natural axiom of aggregation and a variant of monotonicity, are introduced.


3.1 Invariance to aggregation

The first condition aims to regulate the ranking if two journals are aggregated into one.

Axiom 1.

Invariance to aggregation (): Let be a journal ranking problem and be two different journals. Journal ranking problem is given by and such that

  • if ;

  • for all ;

  • for all .

Scoring procedure is called invariant to aggregation if implies for all .

The idea behind invariance to aggregation is that any journal ranking problem can be transformed into a reduced problem by defining the union of journals and as follows: all citations between them are deleted, while any citations by/to these journals are summed up for the “aggregated” journal . This transformation is required to preserve the order of the journals not affected by the aggregation.

Such an aggregation makes sense, for example, if one is interested only in the ranking of journals from a given field (e.g. economics journals) when journals from other disciplines can be considered as one entity.

Invariance to aggregation is somewhat related to the consistency axiom of Palacios-Huerta and Volij (2004), which also uses the notion of reduced problem. However, our property is probably more straightforward than consistency because the latter has been devised to play a crucial role in the characterisation of the invariant method (Palacios-Huerta and Volij, 2004), so it uses the information from the missing journal in a more complicated and unintuitive way.


3.2 Self-consistency

This axiom, originally introduced in Chebotarev and Shamis (1997) to operators used for aggregating preferences, may require a longer explanation.

First, some reasonable conditions are formulated for the ranking derived from any journal ranking problem. In particular, journal is judged better than journal if one of the following holds:

  1. [label = ⬠0), ref = ⬠0]

  2. has more favourable citation records against the same journals;

  3. has more favourable citation records against journals with the same quality;

  4. has the same citation records against higher quality journals;

  5. has more favourable citation records against higher quality journals.

In addition, journals and should get the same rank if one of the following holds:

  1. [resume, label = ⬠0), ref = ⬠0]

  2. they have the same citation records against the same journals;

  3. they have the same citation records against journals with the same quality.


Principles 2-4 and 2 can be applied only after measuring the quality of the journals. The name of the property, self-consistency, refers to the fact that it is provided by the scoring procedure itself.

The meaning of the requirements above is illustrated by an example.

Figure 1: The journal ranking problem of Example 3.1
Example 3.1.

Consider the journal ranking problem with the following citation matrix:

This is shown in Figure 1 where a directed edge from node to indicates that journal has received a citation from journal .

Self-consistency has the following implications for the journal ranking problem presented in Example 3.1:

  • due to rule 1.

  • because of rule 1 as and .

  • Assume for contradiction that . Then and , as well as and , so rule 4 leads to , which is impossible. Consequently, .

  • Assume for contradiction that . Then and , as well as and , so rule 4 leads to , which is impossible. Consequently, .

To conclude, self-consistency demands the ranking to be in Example 3.1.

It is clear that self-consistency does not guarantee the uniqueness of the ranking in general (Csató, 2019a).

Now we turn to the mathematical formulation of this axiom.

Definition 3.1.

Competitor set: Let be an unweighted journal ranking problem. The competitor set of journal is .

Journals of the competitor set are called the competitors of . Note that for all if and only if the ranking problem is balanced.


The competitor set is defined only for unweighted journal ranking problem, but self-consistency may have implications for journals which have the same number of matches. The generalisation is based on a decomposition of journal ranking problems.

Definition 3.2.

Sum of journal ranking problems: Let be two journal ranking problems with the same set of journals . The sum of these journal ranking problems is the journal ranking problem .

The sum of journal ranking problems has a number of reasonable interpretations. For instance, they can reflect the citations from different years, or by authors from different countries.

According to Definition 3.2, any journal ranking problem can be derived as the sum of unweighted journal ranking problems. However, it might have a number of possible decompositions.

Notation 3.1.

Let be an unweighted journal ranking problem. The competitor set of journal is . Let be two different journals and be a one-to-one correspondence between the competitors of and . Then is given by .

Finally, we are able to introduce conditions 1-2 with mathematical formulas.

Axiom 2.

Self-consistency () (Chebotarev and Shamis, 1997): Scoring procedure is called self-consistent if the following implication holds for any journal ranking problem and for any journals : if there exists a decomposition of the journal ranking problem into unweighted journal ranking problems – that is, and is an unweighted journal ranking problem for all – together with the existence of a one-to-one mapping from onto such that and for all and , then . Furthermore, if or for at least one and .

In a nutshell, self-consistency implies that if journal does not show worse performance than journal on the basis of the citation matrix, then it is not ranked lower, in addition, it is ranked strictly higher when it becomes clearly better.


Chebotarev and Shamis (1998, Theorem 5) gives a necessary and sufficient condition for self-consistent scoring procedures, while Chebotarev and Shamis (1998, Table 2) presents some scoring procedures that satisfy this requirement. See also Csató (2019a) for an extensive discussion of self-consistency.


4 The incompatibility of the two axioms

In the following, it will be proved that no scoring procedure can meet axioms and .

Journal ranking problem

Journal ranking problem
Figure 2: The journal ranking problems of Example 4.1
Example 4.1.

Let and be the journal ranking problems with the citation matrices

Journal ranking problem is obtained by uniting journals and .

This is shown in Figure 2 where a directed edge from node to indicates that journal has received a citation from journal , and an undirected edge between the nodes means that the two journals are tied by mutual citations.

Theorem 4.1.

There exists no scoring procedure that is invariant to aggregation and self-consistent.

Proof.

The contradiction of the two properties can be proved by Example 4.1. Take first the journal ranking problem , which has the competitor sets and . Assume for contradiction the existence of a scoring procedure satisfying invariance to aggregation and self-consistency.

Self-consistency has several implications for the scoring procedure as follows:

  1. [label=)]

  2. Consider the (identity) one-to-one correspondence such that and . Then satisfies condition 1 of due to and , thus .

  3. Consider the (identity) one-to-one correspondence such that and . Then satisfies condition 1 of due to and , thus .

  4. Suppose that , which implies according to the inequalities derived in 1 and 2. Consider the one-to-one correspondence such that and . Then satisfies condition 3 of due to and , thus , a contradiction.


Therefore should hold, when invariance to aggregation results in . However, self-consistency leads to in the journal ranking problem because of the one-to-one mapping such that and : the assumption of implies due to condition 3 (the competitors of are more prestigious), while would result in due to condition 3 (the competitors of are more prestigious) again.

Hence a scoring procedure cannot meet and at the same time. ∎

Since Example 4.1 contains balanced, unweighted, and extremal journal ranking problems, there is few hope to avoid the impossibility of Theorem 4.1 by plausible domain restrictions.

Remark 4.1.

and are logically independent axioms because there exist scoring procedures that satisfy one of the two properties: the least squares method is self-consistent (Chebotarev and Shamis, 1998, Theorem 5), and the flat scoring procedure, which gives for all and , is invariant to aggregation.

5 Conclusions

We have shown an impossibility theorem in journal ranking: a reasonable method cannot be invariant to the aggregation of journals, even in the case of a substantially restricted domain of citation graphs. As a consequence, the choice of the set of journals to be compared is an important aspect of every empirical study which aims to measure intellectual influence.

It is clear that our axiomatic analysis has a number of limitations as it is able to consider indices from only one point of view (Glänzel and Moed, 2013). For example, the citation graph is assumed to be known, that is, the issue of choosing an adequate time window is neglected. In addition, this paper has not addressed several important problems of scientometrics such as the comparability of distant research areas, or the proper treatment of different types of publications.

Nevertheless, the derivation of similar impossibility results may contribute to a better understanding of the inevitable trade-offs between various properties, and it means a natural subject of further studies besides axiomatic characterisations.


Acknowledgements

We are grateful to György Molnár and Dóra Gréta Petróczy for inspiration.

The research was supported by OTKA grant K 111797 and by the MTA Premium Postdoctoral Research Program.

References

  • Arrow (1950) Arrow, K. J. (1950). A difficulty in the concept of social welfare. Journal of Political Economy, 58(4):328–346.
  • Bouyssou and Marchant (2014) Bouyssou, D. and Marchant, T. (2014). An axiomatic approach to bibliometric rankings and indices. Journal of Informetrics, 8(3):449–477.
  • Bouyssou and Marchant (2016) Bouyssou, D. and Marchant, T. (2016). Ranking authors using fractional counting of citations: An axiomatic approach. Journal of Informetrics, 10(1):183–199.
  • Chambers and Miller (2014) Chambers, C. P. and Miller, A. D. (2014). Scholarly influence. Journal of Economic Theory, 151:571–583.
  • Chebotarev and Shamis (1997) Chebotarev, P. and Shamis, E. (1997). Constructing an objective function for aggregating incomplete preferences. In Tangian, A. and Gruber, J., editors, Constructing Scalar-Valued Objective Functions, volume 453 of Lecture Notes in Economics and Mathematical Systems, pages 100–124. Springer, Berlin-Heidelberg.
  • Chebotarev and Shamis (1998) Chebotarev, P. Yu. and Shamis, E. (1998). Characterizations of scoring methods for preference aggregation. Annals of Operations Research, 80:299–332.
  • Csató (2015) Csató, L. (2015). A graph interpretation of the least squares ranking method. Social Choice and Welfare, 44(1):51–69.
  • Csató (2019a) Csató, L. (2019a). An impossibility theorem for paired comparisons. Central European Journal of Operations Research, 27(2):497–514.
  • Csató (2019b) Csató, L. (2019b). Some impossibilities of ranking in generalized tournaments.

    International Game Theory Review

    , in press.
    DOI: 10.1142/S0219198919400024.
  • de la Vega and Volij (2018) de la Vega, C. L. and Volij, O. (2018). Ranking scholars: A measure representation. Journal of Informetrics, 12(2):510–517.
  • Demange (2014) Demange, G. (2014). A ranking method based on handicaps. Theoretical Economics, 9(3):915–942.
  • Duggan and Schwartz (2000) Duggan, J. and Schwartz, T. (2000). Strategic manipulability without resoluteness or shared beliefs: Gibbard-Satterthwaite generalized. Social Choice and Welfare, 17(1):85–93.
  • Gibbard (1973) Gibbard, A. (1973). Manipulation of voting schemes: A general result. Econometrica, 41(4):587–601.
  • Glänzel and Moed (2013) Glänzel, W. and Moed, H. F. (2013). Opinion paper: thoughts and facts on bibliometric indicators. Scientometrics, 96(1):381–394.
  • Kóczy and Strobel (2010) Kóczy, L. Á. and Strobel, M. (2010). The world cup of economics journals: A ranking by a tournament method. IEHAS Discussion Papers 1018, Institute of Economics, Hungarian Academy of Sciences.
  • Kongo (2014) Kongo, T. (2014). An alternative axiomatization of the Hirsch index. Journal of Informetrics, 8(1):252–258.
  • Marchant (2009) Marchant, T. (2009). An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics, 80(2):325–342.
  • Miroiu (2013) Miroiu, A. (2013). Axiomatizing the Hirsch index: Quantity and quality disjoined. Journal of Informetrics, 7(1):10–15.
  • Palacios-Huerta and Volij (2004) Palacios-Huerta, I. and Volij, O. (2004). The measurement of intellectual influence. Econometrica, 72(3):963–977.
  • Palacios-Huerta and Volij (2014) Palacios-Huerta, I. and Volij, O. (2014). Axiomatic measures of intellectual influence. International Journal of Industrial Organization, 34:85–90.
  • Perry and Reny (2016) Perry, M. and Reny, P. J. (2016). How to count citations if you must. American Economic Review, 106(9):2722–2741.
  • Quesada (2010) Quesada, A. (2010). More axiomatics for the Hirsch index. Scientometrics, 82(2):413–418.
  • Quesada (2011a) Quesada, A. (2011a). Axiomatics for the Hirsch index and the Egghe index. Journal of Informetrics, 5(3):476–480.
  • Quesada (2011b) Quesada, A. (2011b). Further characterizations of the Hirsch index. Scientometrics, 87(1):107–114.
  • Satterthwaite (1975) Satterthwaite, M. A. (1975). Strategy-proofness and Arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of Economic Theory, 10(2):187–217.
  • Woeginger (2008a) Woeginger, G. J. (2008a). An axiomatic analysis of Egghe’s -index. Journal of Informetrics, 2(4):364–368.
  • Woeginger (2008b) Woeginger, G. J. (2008b). An axiomatic characterization of the Hirsch-index. Mathematical Social Sciences, 56(2):224–232.