Characterisation of the χ-index and the rec-index

06/24/2019 ∙ by Mark Levene, et al. ∙ Bar-Ilan University Birkbeck, University of London 0

Axiomatic characterisation of a bibliometric index provides insight into the properties that the index satisfies and facilitates the comparison of different indices. A geometric generalisation of the h-index, called the χ-index, has recently been proposed to address some of the problems with the h-index, in particular, the fact that it is not scale invariant, i.e., multiplying the number of citations of each publication by a positive constant may change the relative ranking of two researchers. While the square of the h-index is the area of the largest square under the citation curve of a researcher, the square of the χ-index, which we call the rec-index (or rectangle-index), is the area of the largest rectangle under the citation curve. Our main contribution here is to provide a characterisation of the rec-index via three properties: monotonicity, uniform citation and uniform equivalence. Monotonicity is a natural property that we would expect any bibliometric index to satisfy, while the other two properties constrain the value of the rec-index to be the area of the largest rectangle under the citation curve. The rec-index also allows us to distinguish between influential researchers who have relatively few, but highly-cited, publications and prolific researchers who have many, but less-cited, publications.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Axiomatic characterisation of a bibliometric index [Mar09, BM14] provides insight into the properties that the index satisfies and facilitates the comparison of different indices. (Axiomatic characterisation is used in a number of areas for the same purpose.)

Since Garfield’s foundational work in bibliometrics [Gar79], a plethora of bibliometric indices have been suggested [RB15, TB16], from a simple count of the total number of citations to the more sophisticated -index [Hir05]. These have often been motivated by the ongoing debate between quantity (as measured by the number of publications) and quality (as measured by the number of citations to those publications). A review of many of these [WSL14], where a comparison of 108 bibliometric indicators was presented, concluded that, in order to gauge the overall impact of a researcher, several indicators should be used. In particular, many variants of the -index have been proposed and, in [BMHD11], it was shown that generally there is high correlation between the -index and 37 of its variants. Moreover, a critical view of the -index and its variants, which was provided in [Sch18], argued that it is not a good indicator of recent impact and suggested instead a timed -index over a sliding time window. Another criticism of the -index and its variants is that it treats all citations in the same way, ignoring the relevance and impact of citations. This issue can be addressed by Markovian methods, such as PageRank [Lev10], giving rise to indices based on author or publication-level citation networks, such as the Eigenfactor score [WJD13], which can model influence between researchers and publications. However, Markovian indicators have the disadvantage of being significantly more complex to compute than those based on citation counts, and there is no conclusive evidence that they actually outperform rankings based on citation counts [FS̆Z̆B15].

We mention two other issues concerning bibliometic indices that have been recently addressed in a formal setting. The first is that of normalising citation counts across different fields. For example, in [BM16], the authors investigated fractional counting of citations, where the value of a citation is inversely proportional to the numbers papers being cited. They provide a characterisation of the ranking induced by such fractional counting. The second issue is aggregation of bibliometric indices when there are several conflicting indices and there is no compelling reason to chose one over another. For example, in [SAP18], the authors investigated ranking methods from social choice theory in order to provide an axiomatic analysis of aggregation of bibliometric indices.

In this paper we focus on a (two-dimensional) geometric generalisation of the -index, called the -index [FHLB18], which has recently been proposed to address some of the problems with the -index, in particular the fact that it is not scale invariant [PR16].

The -index is determined by the largest square that fits under the citation curve of a researcher when plotting the number of citations to individual publications in decreasing order. On the other hand, the -index is determined by the largest area rectangle that fits under the citation curve. The -index (or rectangle-index) is defined to be the area of this rectangle and is the square of the -index. The -index is thus a member of the class of geometric indices that approximate the area under the citation curve of a researcher. Such indices can address problems attached to, for example, the citation count, which takes into account all citations. The -index penalises both highly-cited publications and publications with only a few citations. On the other hand, the -index is more balanced than the -index in this respect, as it allows us to cater for both influential researchers with a few very highly-cited publications and prolific researchers who may have many publications but relatively few citations per publication.

Our main contribution here is to provide a first characterisation of the -index via three properties: monotonicity, uniform citation and uniform equivalence. Monotonicity is a natural property that we would expect any bibliometric index to satisfy, while the other two properties relate to the rectangle under the citation curve that determines the index. Uniform citation specifies that when the shape of the citation curve is uniform, i.e. rectangular, then the value of the index is the total citation count of all the publications. Complementing this property, uniform equivalence specifies that when the shape of the citation curve is not uniform, i.e. not rectangular, the value of the index is equal to that of some uniform citation curve that can obtained by omitting some number of citations.

We note that many different properties may be used to characterise a bibliometric index, and there is no general agreement on which are the most compelling. For example, there are a number of distinct characterisations of the -index, such as those presented in [Woe08b] and [Que11a]. However, if a number of properties are proved to characterise a given index, the acceptance of any other index would necessitate the violation of at least one of the properties used to characterise the given index.

The rest of the paper is organised as follows. In Section 2, we introduce the -index. In Section 3, we define and discuss properties of the -index and other indices. Then, in Section 4, we present an axiomatic characterisation of the -index. Finally, in Section 5, we give our concluding remarks.

2 The -index and related bibliometric indices

We assume that a researcher publishes publications, where , which are represented by a citation vector of positive integers, , where is the number of citations to publication , sorted in descending order, i.e. for

. (We note that one could consider only a subset of a researcher’s publications in the citation vector, for example, by only allowing journal publications, or publications in high-impact venues.)

The citation curve is the curve arising from plotting the number of citations against the ranking of the publications as a histogram specified by the citation vector.

A bibliometric index is a function that maps citation vectors to the set of non-negative real numbers. As in [Woe08b], we assume the baseline condition that, for the empty citation vector , we have .

In this paper, we concentrate on characterising the -index directly, following the approach adopted in [Woe08b, Woe08a, Que11a, Que11b], rather than characterising the bibliometric ranking induced by the index, as was done in [Mar09, BM14]. This stems from our particular interest in the properties of geometric indices. It is evident that any two indices, such as the -index and the -index, that are monotonic transformations of each other are equivalent with respect to the induced rankings.

The citation count index for a citation vector is the L1 norm of , denoted by and defined by

(1)

We say that a citation vector is dominated by a citation vector , written as , if and for all , .

It is worth recalling some known bibliometric indices [WSL14, TB16]: the citation count, as defined in (1); the publication count ; the maximum citation index ; the Euclidean index [PR16], which is the Euclidean norm of , i.e., ; the -index [Hir05]; and the -index [Egg06], which is a variant of the -index giving extra weight to highly-cited publications.

The -index [Hir05], in particular, has gained popularity due to its relative simplicity, ease of calculation, and its ingenious method of combining the quality and quantity of a researcher’s outputs. It is defined as the maximum number of the researcher’s publications such that each has at least citations, i.e. for a citation vector, the -index is the largest for which .

To motivate the -index, consider the following three citation vectors, the citation curves of which are depicted in Figure 1: (i) , i.e. 1 publication with 100 citations, (ii) , i.e. 10 publications with 10 citations each, and (iii) , i.e. 100 publications with 1 citation each. (Note that the diagram in Figure 1 is not drawn to scale.)

Figure 1: Example of the geometric interpretation of the and indices.

The -index is defined as the square root of the maximum area rectangle that can fit under the citation curve, while the -index is the square root of the maximum area square that can fit under the citation curve.

Formally, we first define the -index (or rectangle-index) of a researcher with citation vector by

(2)

The -index [FHLB18] is then defined by .

Returning to our example shown in Figure 1, we see that all three researchers have a -index of , while researcher (ii) has an -index of , but researchers (i) and (iii) both have an -index of only . The -index may be seen as balancing quality, on the one hand, by favouring publications with a higher number of citations and quantity, on the other hand, by taking into account all publications with a sufficient number of citations. However, such an approach disadvantages a researcher, such as (i), with a few very highly-cited publications, who may have carried out some influential seminal research, and it also disadvantages a prolific researcher, such as (iii), who may have many publications but fewer citations per publication. Now, if we let denote a value of that maximises in (2), the -index can distinguish between more influential researchers for which , such as (i), and more prolific researchers for which , such as (iii). In this sense the -index avoids the debate of number of citations versus number of publications by awarding all three researchers the same -index of .

The -index is a member of the class of (two-dimensional) geometric indices, as is the square of the -index, and also the half the square of the -index [Woe08b], which is the area of the maximal isosceles right-angled triangle under the citation curve; formally, the -index is the largest integer such that the citation vector contains distinct publications with at least citations, respectively. In a more general setting, the dimension of an index can be formally defined [Pra17], and is related to the requirement of dimensional homogeneity from physics, that we may only compare quantities that have the same units.

Geometric indices are actually quite natural, as their goal is to consider the area under the citation curve in order to encapsulate the essential citations for a set of core publications that in some sense represent the output of a researcher. We note that the -index, the square of the -index, the publication count, and the maximum citation index all include the same number of citations for each core publication.

The citation count, which includes all publications with at least one citation in the core, is of course a reasonable bibliometric index. However, it is often argued that the citation count is problematic; in particular, it may be inflated by a small number of publication having a large number of citations or it may be sensitive to a long tail of publications each having only few citations. The axiomatic characterisation we describe here can be viewed as contributing to this debate by discussing several characteristics of geometric indices.

3 Some properties of bibliometric indices

In this section we define a variety of properties of bibliometric indices, almost all of which are satisfied by the -index, and then, in Section 4, we show that monotonicity combined with uniform citation and uniform equivalence characterise the -index.

We assume throughout this section that is the index under consideration. The first property, monotonicity, is a natural requirement for any bibliometric index, stating that adding citations to the citation vector should not decrease the value of the index:

  1. Monotonicity (M): If citation vector is dominated by citation vector , i.e. , then .

It is easy to verify that all the indices we consider are monotonic. We note that the citation count satisfies the stronger property of strict monotonicity (SM), in which when . It also satisfies several other desirable properties, such as rank independence (adding a new publication with a given number of citations to two citation vectors does not change their relative ranking) and rank scale invariance (multiplying the number of citations of each publication by a positive constant does not change the relative ranking of two citation vectors) [PR16]. Neither the -index nor the -index are rank independent. However, the -index is rank scale invariant, but the -index is not. Moreover, the -index satisfies the following stronger property of (linear) scale invariance:

  1. Scale invariance (SI): , for any positive constant .

It is easy to see that scale invariance implies rank scale invariance.

A natural form of symmetry can be attained via the conjugate partition of a citation vector , which is the publication vector , where and is the number of publications with at least citations [Woe08b]. Geometrically, the publication vector is obtained by reflecting the geometric representation of the citation vector along the main diagonal. This is shown in Figure 2 for a citation vector , on the left, and its conjugate partition publication vector , on the right. This motivates the following property.

Figure 2: The citation vector is shown on the left and its conjugate partition, the corresponding publication vector, is shown on the right.
  1. Self-conjugacy (SC): Let be the conjugate partition of the citation vector , then .

Clearly the and indices, as well as the citation count, are all self-conjugate. On the other hand, we note that the publication count and the maximum citation index are conjugates of each other. Self-conjugacy implies a balanced approach between influence (quality) and prolificity (quantity).

Some indices tend to emphasise influence, for example, the maximum citation index, Euclidean index and -index, whereas others, such as the publication count, emphasise prolificity. Should we wish to emphasise influence rather than prolificity, we may define a version of the -index, the -index, in which we restrict the maximum in (2) to be over those for which . Conversely, should we wish to emphasise prolificity, we may instead define the conjugate index, the -index, by using the corresponding publication vector and restricting so that .

Many people, probably the majority, tend to favour indices that emphasise influence. It is therefore worth noting our findings in

[FHLB18], where the citations of a large number of researchers, from a Google Scholar data set made available by Radicchi and Castellano [RC13], were analysed and their -indices calculated. Table 11 in [FHLB18] shows that of the researchers for which the -index was significantly larger than the -index were more influential than prolific, i.e. for these researchers . This indicates that, in general, the -index satisfies the tendency to favour influence.

Following the conclusion in [WSL14] that several bibliometric indicators should be used to gauge the overall impact of a researcher, we suggest that both quality and quantity may be assessed by using the pair of indices .

We now concentrate on the -index. A typical citation curve, corresponding to the citation vector , is shown in Figure 3, with circles indicating where a new citation can be added. We observe that each addition of a new citation completes a rectangle. If the new citation is added to publication , then the newly formed rectangle has width and height . For example, if we add a citation to the third publication, producing the citation vector , then the -index will increase from to , but if we add it to the fourth publication, producing the citation vector , then the -index will not increase.

Figure 3: A citation vector with circles indicating where a new citation may be added.

For any citation vector , we write for the citation vector obtained from by adding a single citation to publication , where , thereby increasing its citation count from to . If , we assume that . We note that must be the smallest index for which .

By the definition of the -index, it is straightforward to see that , and thus

(3)

The following property encapsulates this observation.

  1. Rectangle completion (RC): For any citation vector ,

Noting that rectangle completion implies monotonicity, it is then straightforward to verify that a bibliometric index satisfies rectangle completion (together with the baseline condition) if and only if it is the -index. Obviously, as it essentially encapsulates the definition of the -index, rectangle completion is not particularly useful as a characterisation of the -index; however, it provides an alternative and constructive definition of the index.

Consider the situation when . We note that (i) if , the -index is equal to the square of the -index, (ii) if , the researcher tends towards being more influential, and (iii) if , the researcher tends towards being more prolific. Thus the shape of the maximum area rectangle will indicate an interpretation of the index value. We also note that, when the histogram of the citation curve is a rectangle, the distribution of citations is uniform. In this case the core includes all publications.

More formally, we say that a citation vector is uniform if . It follows that if and only if is uniform. This observation suggests the following weaker form of this property.

  1. Uniform citation (UC): If the citation vector is uniform then .

This property makes the reasonable assertion that, if all publications have the same number of citations, they should all be in the core and all citations to them included in the index.

As stated above rectangle completion is too contrived in the sense that it mimics the definition of the -index. So we now explore a way to replace it with uniform citation, which gives a lower bound on the index, together with another property that gives an upper bound on the index. This additional property is:

  1. Uniform equivalence (UE): For any citation vector , there exists a uniform citation vector dominated by , i.e. , for which .

This property asserts that the same number of citations should be included for each publication in the core.

The following proposition, which defines a property that is similar to Axiom D in [Woe08b], can easily be shown to follow from monotonicity, uniform citation and uniform equivalence.

Proposition 3.1.

If satisfies properties M, UC and UE then it also satisfies the following property:

  1. Citation increase (CI): If we add a single citation to each publication in , resulting in the citation vector , then .

However, citation increase together with monotonicity and uniform citation do not imply uniform equivalence, as is the case for the bibliometric index,

(4)

which satisfies citation increase but not uniform equivalence.

4 Axiomatic characterisation of the -index

The rankings induced by several bibliometric indices, including the -index and -index, were characterised in [Mar09, BM14], whereas, in [Woe08b, Woe08a] and [Que11a, Que11b], the authors concentrated on characterising the -index and -index directly. These characterisations, as well as that presented here, are based on properties that address three types of issues. First, the inclusion of fundamental properties like baseline and monotonicity should be considered. The second issue is concerned with the conditions under which the value of the index increases (see, for example, citation increase). The third issue considers what changes to the citation vector leave the index unchanged (for example, the -index satisfies the property of independence of irrelevant citations [Que11b], cf. [Kon14], which essentially states that adding a single citation to a core publication does not increase the index). Another category of properties that are important for characterising bibliometric indices are invariants, such as scale invariance and self-conjugacy, which give transformations that change the index in a predictable manner or do not change the value at all.

The main result in this section is Theorem 4.1, which provides an axiomatic characterisation of the -index. We also show in Proposition 4.5 that the -index satisfies the desirable property that, subsequent to the addition of a single citation to the citation vector, the -index cannot increase by more than one.

Theorem 4.1.

A bibliometric index satisfies the three properties of monotonicity, uniform citation and uniform equivalence if and only if it is the -index.

Proof.

It is clear from Section 3 that the -index satisfies these properties, so it remains to prove that they are sufficient.

Let be a citation vector and let be a uniform citation vector such that is maximal for all . Clearly, . So, by monotonicity and uniform citation, we have . By uniform equivalence, we may let be a uniform citation vector such that and , and therefore by uniform citation. Since by the definition of , it follows that . ∎

The following two corollaries show that we may replace two of the properties in Theorem 4.1 by two simpler properties together with the intuitive property of scale invariance.

Corollary 4.2.

The result of Theorem 4.1 holds if we replace monotonicity by the following more restrictive property.

  1. Uniform monotonicity (UM): If is uniform and , then .

Corollary 4.3.

The result of Theorem 4.1 holds if we replace uniform citation by the following more restrictive property together with scale invariance.

  1. Uniform single citation (USC): If for the citation vector , then .

Proof.

Clearly USC and SI imply UC. ∎

It is not difficult to demonstrate that the three properties of Theorem 4.1 characterising the -index are independent, i.e. omitting any one of them would render the theorem false.

Proposition 4.4.

The three properties of Theorem 4.1 characterising the -index are independent.

Proof.

The following examples justify this claim.

  1. The index defined in (4) satisfies M and UC but not UE.

  2. The square of the -index, the publication count , the maximum citation index , and all satisfy M and UE but not UC.

  3. The product of the number of publications and the minimum number of citations satisfies UC and UE but not M, nor UM.

To summarise the properties that characterise the -index: UM or M implies that adding new citations will not decrease the value of the index, while UC and UE provide, respectively, lower and upper bounds on its value. We recall that, since the -index is not strictly monotonic, some citations may not contribute to the value of the index; however, all core publications contribute an equal number of citations.

We now present a construction that gives rise to a property equivalent to UE. While indices, such as the citation count (1), that include the full set of citations are strictly monotonic, indices, such as the -index, that include just a core set of publications typically increase only after a batch of citations has been added. The construction we now present captures this aspect of the -index.

Consider a sequence of citation vectors

(5)

where and . When and , for , we say that is a constructive sequence for .

We are interested in constructive sequences satisfying the property that, for each , if then is a uniform citation vector; we call such a constructive sequence -incremental. This suggests the following property of a citation index.

  1. Uniform increment (UI): For any citation vector , there exists an -incremental constructive sequence for .

It is not difficult to prove that UI implies UE. Moreover, the -index satisfies UI, and we now present one method for constructing a -incremental sequence for a citation vector .

  1. Start from and construct a sequence of uniform citation vectors as follows.

  2. From the uniform citation vector , add citations one-by-one to obtain a new uniform citation vector, either by adding a new column (i.e. a new publication) or by adding a new row (cf. property CI) to the rectangle corresponding to .

  3. Repeat step (ii) until we obtain a citation vector for which .

  4. Add the remaining citations one-by-one in any order until we obtain .

It is straightforward to show that it is always possible, in step (ii) above, to choose between adding a column or a row in such a way that the sequence is -incremental.

It may be argued that it is natural that a one-dimensional bibliometric index should not increase by more than one when a single citation is added to the citation vector. We now prove that this holds for the -index.

Proposition 4.5.

Let be a citation vector and let be a citation vector obtained by adding a single citation to . Then .

Proof.

We recall that is obtained from by adding a single citation to publication , thereby increasing its citation count to . If the result holds trivially. So we may assume that , in which case by (3).

Now, since is a citation vector, we must have

Therefore,

and thus

It follows that

.

Finally, we note that if it is required that the -index be an integer, then the ceiling function, which maps to the least integer greater than or equal to , can be employed.

5 Concluding remarks

Geometric indices, such as the -index, capture a core set of the publications that represent a researcher’s total output. The -index (which is equal to the square root of the -index) can be viewed as a generalisation of the -index, and has the advantage that it allows us to distinguish between more influential and more prolific researchers, depending on whether the height of the largest area rectangle under the citation curve is greater than or less than its width, respectively.

We presented several properties that are satisfied by the -index and proved, in Theorem 4.1, that the three properties of monotonicity, uniform citation and uniform equivalence characterise the -index. While monotonicity is a very natural property for any bibliometric index, uniform citation and uniform equivalence are natural when it is required to include the same number of citations for each publication in the core.

Geometric indices, such as the -index, give better insight into the relationship between influence (quality) and prolificity (quantity) than indices, such as the -index, that are more constrained in this respect.

Acknowledgements

The authors would like to thank the reviewers for their constructive comments, which have helped us to improve the paper.

References

  • [BM14] D. Bouyssou and T. Marchant. An axiomatic approach to bibliometric rankings and indices. Journal of Informetrics, 8:449–477, 2014.
  • [BM16] D. Bouyssou and T. Marchant. Ranking authors using fractional counting of citations: An axiomatic approach. Journal of Informetrics, 10:183–199, 2016.
  • [BMHD11] L. Bornmann, R. Mutz, S.E. Hug, and H.-D. Daniel. A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5:346––359, 2011.
  • [Egg06] L. Egghe. Theory and practise of the g-index. Scientometrics, 69:131–152, 2006.
  • [FHLB18] T. Fenner, M. Harris, M. Levene, and J. Bar-Ilan. A novel bibliometric index with a simple geometric interpretation. PLoS ONE, 13(7):e0200098, 2018.
  • [FS̆Z̆B15] D. Fiala, L. S̆ubelj, S. Z̆itnik, and M. Bajec. Do PageRank-based author rankings outperform simple citation counts? Journal of Informetrics, 9:334–348, 2015.
  • [Gar79] E. Garfield. Is citation analysis a legitimate evaluation tool? Scientometrics, 1:359–375, 1979.
  • [Hir05] J.E. Hirsch. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 98:16569–16572, November 2005.
  • [Kon14] T. Kongo. An alternative axiomatization of the Hirsch index. Journal of Informetrics, 8:252–258, 2014.
  • [Lev10] M. Levene. An Introduction to Search Engines and Web Navigation. John Wiley & Sons, Hoboken, New Jersey, second edition, 2010.
  • [Mar09] T. Marchant. An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics, 80:325–342, 2009.
  • [PR16] M. Perry and P.J. Reny. How to count citations if you must. The American Economic Review, 106:2722–2741, 2016.
  • [Pra17] G. Prathap. Citation indices and dimensional homogeneity. Current Science, 113:853–855, 2017.
  • [Que11a] A. Quesada. Axiomatics for the Hirsch index and the Egghe index. Journal of Informetrics, 5:476–480, 2011.
  • [Que11b] A. Quesada. Further characterizations of the Hirsch index. Scientometrics, 87:107–114, 2011.
  • [RB15] R.C. Roemer and R. Borchardt. Meaningful Metrics: A 21st-Century Librarian’s Guide to Bibliometrics, Altmetrics, and Research Impact. Association of College and Research Libraries, Chicago, Il., 2015.
  • [RC13] F. Radicchi and C. Castellano. Analysis of bibliometric indicators for individual scholars in a large data set. Scientometrics, 97:627––637, 2013.
  • [SAP18] A. Subochev, F. Aleskerov, and V. Pislyakov. Ranking journals using social choice theory methods: A novel approach in bibliometrics. Journal of Informetrics, 12:416–429, 2018.
  • [Sch18] M. Schreiber. A skeptical view on the Hirsch index and its predictive power. Physica Scripta, 93(10):102501, 2018.
  • [TB16] R. Todeschini and A. Baccini. Handbook of Bibliometric Indicators: Quantitative Tools for Studying and Evaluating Research. Wiley-VCH, Weinheim, Germany, 2016.
  • [WJD13] J.D. West, M.C. Jensen, R.J. Dandrea, G.J. Gordon, and C.T. Bergstrom. Author-level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the American Society for Information Science and Technology, 64:787–801, 2013.
  • [Woe08a] G.J. Woeginger. An axiomatic analysis of Egghe’s g-index. Journal of Informetrics, 2:364––368, 2008.
  • [Woe08b] G.J. Woeginger. A symmetry axiom for scientific impact indices. Journal of Informetrics, 2:298–303, 2008.
  • [WSL14] L. Wildgaard, J.W. Schneider, and B. Larsen. A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics, 101:125–158, 2014.