Characterising authors on the extent of their paper acceptance: A case study of the Journal of High Energy Physics

06/12/2020 ∙ by Rima Hazra, et al. ∙ Abdus Salam International Centre for Theoretical Physics (ICTP) IIT Kharagpur 0

New researchers are usually very curious about the recipe that could accelerate the chances of their paper getting accepted in a reputed forum (journal/conference). In search of such a recipe, we investigate the profile and peer review text of authors whose papers almost always get accepted at a venue (Journal of High Energy Physics in our current work). We find authors with high acceptance rate are likely to have a high number of citations, high h-index, higher number of collaborators etc. We notice that they receive relatively lengthy and positive reviews for their papers. In addition, we also construct three networks – co-reviewer, co-citation and collaboration network and study the network-centric features and intra- and inter-category edge interactions. We find that the authors with high acceptance rate are more `central' in these networks; the volume of intra- and inter-category interactions are also drastically different for the authors with high acceptance rate compared to the other authors. Finally, using the above set of features, we train standard machine learning models (random forest, XGBoost) and obtain very high class wise precision and recall. In a followup discussion we also narrate how apart from the author characteristics, the peer-review system might itself have a role in propelling the distinction among the different categories which could lead to potential discrimination and unfairness and calls for further investigation by the system admins.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Publishing new research in journals/conferences is a common practice in the scientific community. It is noticed that papers of few authors consistently get accepted in journals whereas papers of certain other authors get rarely accepted111https://www.sciencemag.org/careers/2018/12/yes-it-getting-harder-publish-prestigious-journals-if-you-haven-t-already. An intriguing question thus is what makes the papers of certain authors almost always eligible for acceptance. Is there a special recipe that they follow in preparing their manuscripts? Does it depend on their position in the collaboration/citation network? Does their experience or their -index matter? Does the diversity in the topics that they work on help escalate the acceptance? The present paper attempts to delve into some of these questions and characterise authors based on their paper acceptance profile. We base our investigations on a dataset obtained from the Journal of High Energy Physics that has information about authors, papers written by them, citations obtained by them and the review reports written by expert referees for each of their accepted paper. The overall peer review workflow for this journal is illustrated in Figure 1. In a nutsell the workflow is as follows – once an author submits a paper, the system allocates the submission to an editor based on a simple keyword matching technique. The editor then handles the paper and chooses one or more competent referees who are experts in the area and can judge the technical merit of the paper. The referee(s) in turn read the paper and send their review report(s) to the editor. The editor reads the review(s) and takes a decision to either accept, reject or invite the authors to revise and resubmit. The revise and resubmit decision re-instantiates the same workflow described above once again and the cycle continues until the paper is eventually accepted or rejected.

We categorize the authors in this dataset into three classes based on the fraction of their papers accepted to the journal. We calculate the acceptance rate () of an author as the ratio of the number of papers accepted to the number of papers submitted by the author to the journal. For each of the three categories (discussed below) we analyse a bunch of interesting features that are drawn from the collaboration/citation network of an author as well as the peer reviews received by the different accepted papers of the authors. We find that these features are considerably different across the three classes.

Figure 1. The JHEP peer review workflow.

1.1. Our contributions

We categorize the authors into three classes based on their acceptance rate. Authors whose papers are consistently accepted for publication and have high are placed in the class ; authors whose papers are rarely accepted and have low are placed in the class and authors who are neither in , nor in and have moderate are placed in . We explain the process of author categorization in details in section 3. Our main contributions are threefold.

  1. Rigorous analysis of the profile and peer review based features of authors belonging to each category.

  2. Analyzing inter-category and intra-category interaction and network-centric properties obtained from three different networks – (i) the co-reviewer network (), (ii) the collaboration network () and (iii) the co-citation network ().

  3. Early prediction of an author’s category based on the profile, peer review data and network-centric features.

Toward the first objective, we extract various features representing an author. These features are divided into two types – (i) author’s profile based features () and (ii) features based on peer review data (). Author’s profile based features () comprises citation count (), topic diversity (), experience () and -index (Hirsch:2005) (). Peer review based features () consists of sentiment of review text (), length of the review text (), reviewer diversity () and editor diversity ().

In addition, we extract various features – centrality values, clustering coefficient, core-periphery structure etc. from the three different type of networks mentioned above. These networks are defined below.

(i) Co-reviewer network (): In this article, we introduce a co-reviewer network. Each author is considered as a node in the network and two authors are connected by an edge if their papers are reviewed by the same reviewer. In addition, we also prepare an induced co-reviewer graph for the three different author categories.
(ii) Collaboration network (): Each author in this network is considered as a node and two authors are connected by an edge if they co-authored in a paper. We also prepare the induced collaboration networks of the authors of each category.
(iii) Co-citation network (): In this directed network, each author is considered as a node and two authors are connected by an edge () if author has cited an article authored by . There is bidirectional edge () if author and author cites each other.

For our experiments, we consider the authors who have submitted their paper to the Journal of High Energy Physics (JHEP) between 1997 to 2015. We consider approx. 29k papers and more than 24k authors. We also have approx. 70k unique review reports.

1.2. Key results

A nuanced analysis shows that authors in the class usually receive more citations than the other two categories. We also note that papers of the authors receive more citation if they coauthored with authors in some paper. authors always receive more positive reviews than the other two categories. An intriguing observation is that the set of referees and editors to whom the papers of the class are assigned are found to be less diverse than the other two classes. The authors are more ‘central’ in all the networks. We finally make early predictions of the category of an author and obtain 0.82 - 0.95 precision and 0.82 - 0.91 recall. In a followup discussion we narrate how apart from the author characteristics, the peer review system itself can potentially facilitate discrimination in the editing and the reviewing process of papers in the three categories which could reinforce the distinction between the authors of these categories and calls for further investigation by the system admins.

1.3. Outline

The rest of the paper is organised as follows. Section 2 describes the dataset used in this paper. Section 3 details the method for author categorization. Section 4 and 5 demonstrate the author profile features and peer review based features respectively. In section 6, we discuss the network features of the three category of authors. In section 7 we predict the category of the authors. In section 8 discuss the potential role of the peer review system in enhancing the distinction among the three categories of authors. Section 9 presents a brief literature review. Finally, we conclude in section 10.

2. Dataset Description

In our article, we consider papers submitted to the Journal of High Energy Physics (JHEP)222https://jhep.sissa.it/jhep/ in between 1997 and 2015. JHEP is one of the leading journals in the domain of high energy physics. In JHEP, the identity of the referee remains confidential. This dataset contains a total of 28871 papers, where the number of accepted and rejected papers are 20384 and 6190 respectively. We also have 70000 unique peer review reports. For each paper we have the title, author names, broad topics that the paper is on, publication date (in case it was accepted) and the number of citations for the accepted papers. In addition, this dataset contains the review text, number of review rounds, editor and reviewer ids (anonymised) of each paper. We also have the citation link among the papers. For the rejected papers, we collected the arXiv333http://arxiv.org id using the Inspire444https://inspirehep.net search engine. We consider the cumulative number of citations obtained at the end of 2015. We present a brief statistics of the dataset in Table 1.

width=0.35 Basic Information Count #papers 26574 #unique authors 24868 #papers (accepted) 20384 #papers (rejected) 6190 Average #citations (accepted papers) 31.88 Average #citations (rejected papers) 9.45

Table 1. Dataset description.

3. Author Categorization

In this section, we categorize authors’ profile into three categories based on their articles’ acceptance rate () – (i) authors with high acceptance () (ii) authors with moderate acceptance () (iii) authors with low acceptance (). Acceptance rate of an author is calculated as the ratio of the number of papers accepted to number of papers submitted by that author. We calculate article acceptance rate of each author for every year. In case of category, we consider only those authors who have high acceptance rate in at least 70% of the years over all the years. category contains authors who have very low acceptance rate in at least 80% of the years. We keep the rest of the authors (not falling in the other two categories) in category. Statistics of the unique authors are given in Table 2. The number of accepted and rejected papers in each class are noted in Figure 2. The papers of authors in the class almost always get accepted.

width=0.3 Author Categories #Authors 3688 10359 9644

Table 2. Statistics of author categorization.
Figure 2. Percentage of accepted and rejected papers of (High), (Moderate) and (Low) authors.
Figure 3. (Left) This collaboration network includes , and authors. (Right) This collaboration network includes and authors only for better visualisation.

4. Author profile based features ()

4.1. Citation index ()

Citation count of each author is computed by considering the total number of citations an author received in their active period. For each category, we define citation index as the standard deviation of citation counts of all the authors. We compute for three categories , and (see Figure see Figure 4(d)). There is a stark difference in the values of among the three categories. Authors in the class have low (approx. 50) whereas authors in the class have high (approx. 101). Thus, the citation counts in the class are far more uniform across the authors compared to the class.

4.2. Experience ()

Experience of an author is defined in terms of number of papers he has published. For each category, we compute experience of all the authors and consider the mean of these s. We observe has highest mean experience (see Figure 4(a)). has moderate value of mean experience whereas has very low mean experience (see Figure 4(a)). From this, it is clear that category contains those authors who are either new in research or has very few publications.

4.3. Topic diversity ()

We consider a topic set for each author. This topic set contains all the topics on which an author published their papers. For each author we compute topic ratio as the ratio of the total number of topics on which he/she has written a paper to the total number of papers he/she published. For each category, we consider mean over topic ratio of all the authors to compute topic diversity () (see Figure 4(e)). Interestingly, category authors have less (1.03) than the other two categories ( has 1.36 and has 1.57). We observe that category authors publish papers on a lot of topics whereas authors focus on a relatively less number of topics and publish a large number of papers in those topics.

4.4. -index ()

The -index (Hirsch:2005) is defined as the maximum value of such that an author has published papers that have each been cited at least times. For all the three categories, we consider mean of the of all the authors. From Figure 4(b), it is clear that have very high mean compared to the other two categories. Thus the class usually comprises the high impact authors.

4.5. Team size ()

Team size of an author is calculated as the number of contributing co-authors averaged across all the papers that the particular author has written. We examine mean team size for each category (see Figure 4(c)). and authors have mean team sizes of 2.44 and 2.15. The typical team sizes for both these classes are very similar. On the other hand, we find that the mean team size of is which is quite low compared to the other two classes.

Figure 4. (a) The mean experience () of (High), (Moderate) and (Low). (b) The mean -index for the three categories. (c) The mean team size () for the three categories. (d) Citation index () of the three categories. (e) Topic diversity () for the three categories.

5. Peer review text based features ()

5.1. Sentiment of review text ()

We compute the sentiment score of each review text for each paper555https://textblob.readthedocs.io/en/dev/. For every author we compute the average review sentiment across all the papers (s)he has written. For every class, we take the mean of these average values across all the authors in that class (see Figure 5(a)). Among the three classes, the review text bears the highest positive sentiment (0.15) in the class. This is followed by class where the overall sentiment is 0.05. Finally, the review texts corresponding to the class indicate the presence of high negative sentiment .

5.2. Length of review text ()

Length of review text is computed as the number of words present in the review text except stop-words (see Figure 5(b)). Surprisingly, we find that category receive relatively lengthier reviews (2368) compared to (1305). It is therefore quite clear that papers in the class typically receive more detailed feedback from the referees compared to the class.

5.3. Reviewer diversity ()

We use Shannon index (Spellerberg:2003) to calculate the reviewer diversity. For each author in a particular category, we extract the reviewer ids of all his/her published papers and add it to a global list. Thus we have three global lists for each of the three categories. Next, for each category, we compute the entropy of this global list. Let the size of the global list for a category be and let the number of occurrences of a reviewer in the list be . Then the entropy would be . If the value of this entropy is low then this would mean that the number of reviewers to whom the papers of a class go for review are very limited. In contrast, if this value is high for a class then it would mean that many reviewers are assigned as referees for the papers in the class (see Figure 5(c)). Surprisingly, we notice that has less reviewer diversity than the other two categories. and categories have reviewer diversity 7.36 and 7.34 respectively. This possibly indicates that for the class the set of referees are relatively more fixed and papers of authors from this group usually go to other peer authors (in the role of referees) mostly from this group itself for a review. This, we believe, is a sign of unhealthy reviewing practice. We shall discuss more about this in section 8.

5.4. Editor diversity ()

Once again we use Shannon index (Spellerberg:2003) to calculate editor diversity. We compute this metric exactly as with the exception that here the three global lists are composed of editor ids to whom the papers are assigned (as opposed to reviewer ids in the previous case). Here also we observe that editor diversity of is quite low ; on the other hand, the editor diversity of and classes are relatively higher and respectively (see Figure 5(d)). It seems that the same set of editors handle the papers of the class.

5.5. Linguistic quality indicator ()

Here we analyze the different emotions (positive, optimism, cheerfulness, confusion and contentment) reflected by each word present in the review text666https://github.com/Ejhfast/empath-client. Then we take the mean of the emotion values of words present in a particular review text and average it over all authors in a class. We find quite a few interesting results. There are more positive emotion words in the review texts of the class (0.018) compared to the class (0.015). Further, there are more optimism related words in the review texts of the class (0.01) compared to the class (0.004). There are more cheerfulness related words present in the review texts of the class (0.0017) compared to the class (0.0014). There are less confusion words in the review texts of the class (0.0026) compared to the (0.0036) class. Last, there are more contentment related words in the review texts of the class (0.0079) compared to the class (0.0059).

Figure 5. (a) The sentiment of review text () of (High), (Moderate) and (Low). (b) The length of the review text () for the three categories. (c) Reviewer diversity () of the three categories. (d) Editor diversity () for the three categories.

6. Network analysis based features ()

Figure 6. (a) The average degree centrality of (High), (Moderate) and (Low) for the three networks (, and ). (b) The average betweenness centrality of (High), (Moderate) and (Low) for three networks.

In this section, we study the properties of the three different networks in details.

6.1. Analysis of the co-reviewer network ()

Recall that in a co-reviewer network each node corresponds to an author and two authors are connected if their papers have been co-reviewed by the same referee. We run series of analysis on this network to investigate the differences between the three categories.

6.1.1. Centrality measures

Here we compute four centrality measures of the whole co-reviewer network.

Degree centrality: We compute the average degree centrality of the authors (see Figure 6(a)) for each category. We observe that the average degree centrality of authors of category is high (0.019) whereas the average degree centrality of the authors for () and () are relatively lower. (see Figure 6(a)).

Betweenness centrality: We compute the average betweenness centrality of the authors of each category. The average betweenness centrality (see Figure 6(b)) of category is marginally higher () than the other two categories.

Closeness centrality: We calculate the average closeness centrality of authors for each category. The average closeness centrality (see Figure 7(a)) of category is higher () than the other two categories.

PageRank: We calculate the average PageRank score of the authors for each category. The average PageRank (see Figure 7(b)) of category is marginally higher () than the other two categories.

6.1.2. Core periphery analysis

Here we perform a -shell decomposition of the network and inspect four different shells – the innermost (), the inner-mid (), the outer-mid () and the outermost (). As noted in Table 3, we observe that the innermost and inner-mid shells contain a larger fraction of nodes from the and classes compared to the class. In contrast, the outermost shell contains the largest fraction of nodes from the class.

width=0.45 Shell # Authors % % % Innermost (180) 167 29.9 49.1 20.3 Inner-mid (140) 37 13.5 78.3 8.1 Outer-mid (90) 116 11.2 48.2 36.2 Outermost (1) 227 7 14.5 62

Table 3. Core periphery analysis of the co-reviewer network.

6.1.3. Induced co-reviewer network

Here we construct three induced co-reviewer networks comprising the authors in the classes , and respectively.

Density: We calculate the density of each induced graph to observe how densely the authors are connected among themselves through the common reviewers. Density of the induced graph is higher (0.047) than others. Density of and are 0.016 and 0.001 respectively.

Assortativity coefficient: We compute the assortativity coefficient of the three induced networks. While this coefficient for the induced graph is as high as 0.82, the same for the and the induced graphs are 0.66 and 0.24 respectively. This indicates that the induced graph is much more homophilic compared to the other two graphs.

Edge transitions: We finally study the edge transitions among the three induced graphs, i.e., given a pair of induced graphs we find the fraction of edges going from one of them to the other from the original co-reviewer network. We find that and share almost 34.7% edges whereas and share only 4.3% edges. The fraction of edges between and is around 10.4%.

Figure 7. (a) The average closeness centrality of (High), (Moderate) and (Low) for three networks (, and ). (b) The average PageRank of (High), (Moderate) and (Low) for three networks.

6.2. Analysis of co-citation network ()

Recall the the co-citation network has authors as its nodes and there is an edge from author to if cites a paper of . If both and cite each other in some of their papers then there is a bidirectional edge between them.

6.2.1. Centrality measures

We compute four centrality measures in the co-citation network.

Degree centrality: We compute the average degree centrality of the authors (see Figure 6(a)) for each category. We observe that the average degree centrality of the authors of category is high compared to the average degree centrality of authors for and categories.

Betweenness centrality: We compute the average betweenness centrality of the authors for each category. The average betweenness centrality (see Figure 6(b)) of the authors of category is marginally higher () than the other two categories.

Closeness centrality: We calculate the average closeness centrality of the authors of each category. The average closeness centrality (see Figure 7(a)) of category authors is higher () than the other two categories.

PageRank: We calculate the average PageRank score of the authors of each category. The average PageRank (see Figure 7(b)) of and categories are marginally higher than the category.

6.2.2. Induced co-citation network

Here again we construct three induced co-citation networks comprising the authors from the three classes – , and .

Cross citations: We find the fraction of citations running in between the classes. Notably, the largest fraction of citation edges run between and induced graphs (45%). Fraction of citation edges running between and induced graphs, on the other hand, is the least (1%).

Self citations: Fraction of citation edges running within the induced graph is the highest . This fraction for the and are 17.9% and 0.4% respectively.

Reciprocity: We compute the reciprocity within and across all the three induced networks. Reciprocity within the induced network is the highest (0.61); reciprocity in the induced network is 0.20 and the same for the induced network is 0.08 which is the least among the three.

Reciprocity in between and induced networks is 0.34 which is higher than between and (0.11) as well as and (0.12).

6.2.3. authors that are cited by authors

Although a rare case, here, we observe how the citations coming from the authors affect the fate of the papers written by the authors. We separately consider those papers which are cited by authors and observe the author characteristics of such papers. We find that the mean citation of papers written by authors and cited by authors is roughly double the mean citation of papers written by authors that are never cited by the authors.

We further notice that the mean citation of those authors whose papers are cited by authors is higher than the mean citation of the other authors .

6.2.4. authors cited by authors

In this section, we investigate the characteristics of those authors whose papers are cited by authors. Once again, we observe that the mean citation of papers written by authors and cited by authors is much higher than the mean citation of papers written by authors but never cited by the authors.

6.3. Analysis of collaboration network ()

Recall that the in the collaboration network each node is an author and two authors are connected if they have co-authored a paper together. We present a visualisation of the collaboration network in Figure 3. The left sub-figure shows the authors in the three categories as nodes of different colours. The blue nodes correspond to the authors in the category, the red nodes correspond to the authors in the category and the yellow nodes correspond to the authors in the category. The blue nodes are concentrated mostly in the center of the network while the red and the yellow nodes are scattered all across the network. This is more clear when we draw the network of the authors corresponding to the and the category. The blue nodes are largely concentrated at the center of the network.

6.3.1. Centrality measures

We compute four centrality measures from the collaboration network.

Degree centrality: We compute the average degree centrality of the authors (see Figure 6(a)) for each category. We observe that the average degree centrality of the authors in the category is higher (0.0039) than the average degree centrality of the authors in the other two categories.

Betweenness centrality: We compute the average betweenness centrality of the authors of each category. The average betweenness centrality (see Figure 6(b)) of category is higher () than the other two categories.

Closeness centrality: We calculate the average closeness centrality of the authors of each category. The average closeness centrality (see Figure 7(a)) of category is higher () than the other two categories.

PageRank: We calculate the average PageRank score of the authors of each category. The average PageRank (see Figure 7(b)) of category is marginally higher () than the other two categories.

6.3.2. Class wise collaborations

The fraction of collaboration edges between the and authors is 38.9% which is much higher than either the fraction of collaboration edges between and authors (1.0%) or and authors (0.2%).

On the other hand, the fraction of collaboration edges within the authors is 26.4%, while this is 31.3% for the authors and 0.7% for the authors.

6.3.3. authors collaborating in papers primarily written by authors

In this section, we focus on those authors who get a chance to collaborate with authors. In particular, we consider those papers which are written by a mix of 20% authors and 80% authors (i.e., papers predominantly written by authors with high acceptance ratio).

We compute various features discussed earlier for this 20% authors when they write papers with authors and when they write papers without them. The feature values are noted in Table 4. Collaborations with the authors seems to heavily benefit the authors in terms of accrued citations as well as review sentiments obtained from the referees.

width=0.48 Features Collaborated Not collaborated with with Mean #papers 1.1 1.9 Team size () 4.3 3.1 Citation () 30 12 Review text sentiment () 0.23 -0.13

Table 4. Properties of authors who collaborate with a high number authors.

6.3.4. authors collaborating in papers primarily written by authors

In this section, we analyze such cases where papers are written by 80% and 20% authors. We analyze profile features of these 80% authors when they write papers with authors as well as when they write without them. Table 5 enumerates the important features and shows that even having a small fraction of authors in their paper can increase the citation count and reduce the negative sentiment in the reviews of the authors.

width=0.48 Features Collaborated Not collaborated with with Mean #papers 1.0 1.8 Team size () 4.12 2.85 Citation () 53.7 27.7 Review text sentiment () -0.39 -0.54

Table 5. Analysis of authors who collaborate with a low number of authors.

7. Author category prediction

7.1. Classification model

In our classification model, we consider , and

features for the first three years of career of each author as the training data. For example, if an author published his first paper in 1996 then we consider papers published in between 1996 and 1998 for training purpose. We compute all the features of an author based on the first three years of career information. For testing, we leave a gap of two years to prevent any data leakage. After five years, we predict their category. We use two different classifiers – XGBoost 

(Chen:2016) and random forest (Breiman:2001). In order to evaluate the model, we compute class wise precision and recall. In addition, we also compute F1-score. We calculate precision as the fraction of authors who are correctly classified out of all the predicted authors. Recall is the fraction of relevant authors correctly classified by the classifier.

Features: We use the author profile features (), peer review based features () as well as network features ().

Results: The class wise precision and recall for the XGBoost model are noted in Table 6

. The F1-score for the model is 0.84. The confusion matrix is tabulated in Table 

7.

The class wise precision and recall for the random forest model are noted in Table 8. F1-score for this model is 0.89. We report the confusion matrix in Table 9. The random forest model outperforms the XGBoost model.

width=0.35 Categories Precision Recall 0.78 0.75 0.84 0.88 0.92 0.88

Table 6. Class wise precision and recall of the XGBoost model.

width=0.35 Categories 2622 718 174 715 8669 502 18 987 7389

Table 7. Confusion matrix of the XGBoost model.

width=0.35 Categories Precision Recall 0.82 0.82 0.87 0.91 0.95 0.91

Table 8. Class wise precision and recall of the random forest model.

width=0.35 Categories 2889 527 98 604 9006 276 20 727 7647

Table 9. Confusion matrix of the random forest model.

Feature importance: Some of the important features for both the models are degree centrality of , sentiment of review text (), PageRank of , citation count, team size (), degree centrality of , core number, PageRank of , reciprocity of , experience, -index (), closeness centrality of , betweenness centrality of , reviewer diversity (). The individual set of features that are important for the two models are noted in Figure 8 (random forest) and Figure 9 (XGBoost).

Figure 8. Important features for the random forest model.
Figure 9. Important features for XGBoost model.

8. The role of the peer review system

So far we have investigated author characteristics that could act as early indicators of the acceptance rate of the authors. However, recall, the reviewer and editor diversity measures presented in sections 5.3 and 5.4 respectively. In fact these features are also found to have strong predictive power in section 7. Although we have used these features in profiling the authors, it can be easily reasoned that they are based on the functioning of the peer review system itself. In this section we shall therefore discuss the role of the peer review system (in any) in reinforcing the distinction among the three categories of authors.

To this purpose, we characterize the authors of different categories in terms of the set of editors and reviewers who have ever edited/reviewed their paper. We consider pairs of authors from each category and compute the Jaccard overlap () of the reviewer and the editor sets respectively. Next for each category, we calculate the average pairwise values. Interestingly, for the reviewer set we observe that the average value of for authors is relatively higher (0.0202) compared to (0.0016) and (0.0008) authors. For the editor set, the average value of for is 0.0302 whereas the average value for and are similar (0.0137 and 0.0105 respectively). This potentially again indicates that there is less diversity in the editors and reviewers who are assigned to the category. However, one might argue that this could as well be an artefact of the authors in the category collaborating more heavily among themselves compared to the other two categories and therefore it is obvious that they would tend to have more overlap in the reviewer and editor sets. In order to verify if this is actually an artefact, we next consider for each category the pairs of authors who have never collaborated (i.e., never co-authored a paper together). For such pairs of authors in a category, we calculate the of their editor and reviewer sets again. In particular, we identify the % of author pairs having in the range and author pairs having exactly 1. We note the percentage overlap values in Table 10. For both the editor and the reviewer sets we observe that even if the authors have never collaborated they tend to get more similar referees and editors in the category compared to the other two categories. This result indicates that the initial observation that we made was not an artefact and that the peer-review system indeed enables a less diverse referee and editor set for the authors. We present a visualisation of this phenomenon in Figure 10. In Figure 10 (Up), the green coloured nodes represent the reviewers and the blue, the red and the yellow nodes correspond to the authors in the , and categories respectively. There is a directed edge from a reviewer to an author if the reviewer had reviewed one or more papers of the author (i.e., a directed bipartite network). The visualisation again indicates that there are ‘patches’ of clusters of unique reviewers around authors of the category. Similarly, in Figure 10 (Down) the sky blue colored nodes represent the editors and the blue, the red and the yellow nodes correspond to the authors in the , and categories respectively. There is a directed edge from an editor to an author if the editor had edited one or more papers of the author. Similar patches of clusters also appear here. Overall, we believe that this might lead to potential discrimination and unfairness and should therefore be further investigated by the system admins.

Figure 10. (Up) This network shows the relationship between hundred top cited authors and their reviewer from three different categories. There is a directed edge from a reviewer to an author if the reviewer had reviewed one or more papers of the author. (Down) This network shows the relationship between hundred top cited authors and their editors from three different categories. There is a directed edge from an editor to an author if the editor had edited one or more papers of the author.

width=0.45 Categories Editor set Reviewer set () () () () 1.94% 1.86% 0.46% 0.31% 1.36% 1.31% 0.32% 0.13% 1.03% 1.02% 0.04% 0.03%

Table 10. Percentage of author pairs having Jaccard overlap of editor and reviewer set in and exactly 1.

As an additional investigation we choose author pairs across categories and observe how their editor sets overlap. If we choose author pairs with one from and another from the value in for the editor set is 0.13%. Similarly, if we choose author pairs with one from and another from the value in for the editor set is 0.57%. However, what is most intriguing is that if we choose author pairs with one from and another from the value in for the editor set is 0%. This indicates that the editors who are assigned to the category of authors are almost never assigned to the category users. Once again this could be indicative of a potential unfairness situation in the peer-review system and needs to be carefully investigated further.

9. Related Work

Peer review system plays an important role in the acceptance of a research paper in a journal. Quality peer review system helps authors to improve themselves. There are lots of debates on the quality (Jefferson:2002) and bias777https://www.nature.com/news/let-s-make-peer-review-scientific-1.20194 in a peer review system (Huisman:2017; Falkenberg:2018; Sikdar:2016). Jefferson et al. (Jefferson:2002) investigated the quality of editorial peer review. They claimed that measuring the quality of peer review require huge co-operation of authors. Sikdar et al. (Sikdar:2017) studied reviewer-reviewer interaction network to predict the long term citation of a paper. They also studied whether the peer review system can be improved. In (Sikdar:2016), the authors investigated anomalies in a peer review system. They computed different features from the editor and the reviewer information available. In (helmer17) the authors investigated the existence of gender bias in a peer review system. Another interesting study by Tomkins et al. (tomkins17) showed that a single blind reviewing system gives disproportionate advantage to the papers of famous authors and authors from highly reputed institutions. In similar lines the authors in (alina19) proposed how to improve a single blind review process.

Earlier research also explored various author profile based features such as experience, citation count, -index, research topic diversity to quantify research productivity/success of an author (Bu:2018). The productivity of an author (Abramo:2018) had been defined as the extent of his/her contribution (publications) to the scientific community. Most of the earlier research focused on whether such author profile based features are sufficient to justify ones research productivity. In (Bremholm:2005), the authors explored the productivity of authors and their citations considering publications in the Proceedings of the Oklahoma Academy of Science (POAS). They found that authors with high productivity are not highly cited. Bayer et al. (Alan:1966) computed citation count to measure the productivity and found that it is less correlated with the quality of researcher’s academic career but there is no correlation with his/her IQ.

Our work is very different from the above studies. We utilise author profile information, peer review information and three different networks to predict the class of an author based on his/her acceptance rate.

10. Conclusion

We categorize the authors into three classes based on their acceptance rate in the journal. We characterise these classes of authors based on their profile, the peer reviews their papers received and three different networks. The authors with high acceptance rate seem to be markedly different in terms of many of these characteristic features. Finally, using these features we show that it is possible to predict the acceptance rate class early for any author.

In future we would like to investigate in more details the reasons for the differences in the reviewer and editor diversities across the classes. In specific this problem can be posed as an anomaly/bias detection where we plan to use state-of-the-art techniques to understand the precise reasons for such uneven diversity across the classes.

11. Acknowledgements

We thank Media Lab SISSA for providing us with the necessary JHEP data for the analysis. RH and AM thank Simons Foundation for financial support through the Simons Associateship Programme.

References