Original Investigation

Relationship between an editor in chief's commentary publications and the impact factor of cardiovascular medicine journals

Alex Salazar¹, Michael Joseph Berry²

doi: http://dx.doi.org/10.5195/jmla.2021.1145

Volume 109, Number 3: 459-463
Received 10 2020: Accepted 02 2021

ABSTRACT

Journal impact factor (IF) inflation is suggested as a problem resulting from commentaries published by the editors in chief (EiCs) of their respective journals. However, it is unclear whether this is a systemic problem across the top thirty cardiovascular medicine journals. Therefore, the purpose of this investigation was to examine the relationship between the number of commentaries written by an EiC and their journal's IF and Eigenfactor (Ef). Utilizing Spearman rank partial correlations controlling for length of service as the EiC, significant moderate correlations were found between the number of commentaries and the number of first-author commentaries by the EiC and the IF of their journal (r=0.568, p=0.001 and r=0.504, p=0.005; respectively). A weak but still significant correlation was found between the number of commentaries by the EiC and the Ef of their journal (r=0.431, p=0.020). The reason for these correlations is unclear, and whether the methodology used to compute the IF and Ef should be modified needs further research.

Keywords: impact factor; Eigenfactor; editorial comment; cardiology.

INTRODUCTION

A common method for determining the quality of a journal is the Journal Citation Reports' impact factor (IF) [1]. Developed in the 1960s, the IF was used as a tool for selecting journals to include in the Science Citation Index, the predecessor to the Clarivate Web of Science database, and was eventually used as an acquisitions tool by libraries [2, 3]. The IF is the number of times a journal's articles have been cited in other articles in the two previous years divided by the total number of articles published by the journal during the same time period. For example, an IF of ten means that, on average, each article published by a journal in the previous two years had been cited ten times [1].

Due to the simplicity of calculating the IF, it is possible to manipulate the score. Because the IF is simply a mean score, it can be skewed by outliers. Even one highly cited paper can significantly boost a journal's IF [4, 5]. Another manipulation of the IF is through author or journal self-citation or both, which allows authors and journals to bolster the reputation of their works through repeated self-referencing. A study examining the relationship between self-citation rate and journal IF in the field of plastic surgery found that the two were significantly correlated [6]. Self-citation is also problematic within cardiovascular journals [7]. The Eigenfactor (Ef) is another bibliometric indicator that is touted as a solution to problems with the IF. The Ef, which is calculated based on the number of times articles published in the last five years have been cited [8], accounts for self-citation by removing the influence of citations to other articles in the same journal [9, 10].

While there is evidence of journal self-citation in the cardiovascular sciences, it is not clear what role editors in chiefs (EiCs) have in this problem [11]. EiCs often publish commentary pieces, typically drawing attention to or criticizing other academic works. These pieces are typically short, lack an abstract, and serve as a form of communication between the EiC and readers. Commentaries written by EiCs may increase their journal's IF by being cited frequently and citing other articles from the parent journal [12, 13]. The use of commentaries or editorials written by EiCs has been identified as a technique that could manipulate journal IFs. These pieces are not counted in the denominator but are counted in the numerator of the IF calculation, allowing potential manipulation of the IF [14].

The primary aim of this study was to determine the relationship between the number of commentaries written by an EiC and the IF of their journal. A secondary aim was to determine the relationship between the total number of publications by an EiC and the IF of their journal to account for the full influence of an individual editor's contributions to their field on their journal's IF. Tertiary aims were to determine whether the EiC being a first author influenced these correlations and to determine if an EiC's publication record also correlated with their journal's Ef.

METHODS

Utilizing the Journal Citation Reports database from Clarivate Analytics, we selected the top thirty cardiac and cardiovascular systems journals based on their IF in the year 2019. Cardiology was selected because of previous research indicating problems with self-citation in this field [7, 11]. Table 1 shows the journals included in this analysis, the IF and Ef at the time of this analysis (2020), and the number of commentaries by EiCs published in their journal.

The EiCs of journals were found using the journals' websites. PubMed was used to determine the following values for each EiC: number of commentaries published, number of commentaries for which the EiC was the first author, total number of publications, and total number of publications for which the EiC was the first author. The number of commentaries published was restricted to the EiC's journal, whereas total number of publications included all items published regardless of the journal. Commentaries were defined as any article written by the EiC published in their own journal that specifically said “no abstract” in PubMed. The approximate date of the appointment of each EiC was also determined to control for variation in publication counts based on the length of time each EiC served. If a journal had co-EiCs, the first EiC that appeared alphabetically was used.

Data were analyzed using SPSS version 26. Data were tested for normality using the Shapiro-Wilk test. Partial correlations were computed using Spearman rank correlations. EiCs' publications were correlated with their journals' IF and Ef while controlling for the number of years the EiC had served to statistically remove the influence of length of EiC experience. Comparisons between correlation coefficients were performed by converting each correlation coefficient into a z-score using Fisher's r-to-z transformation. Asymptotic covariance of the estimates were computed, and these quantities were used in an asymptotic z-test [15]. Significance was set at p<0.05.

RESULTS

For the journals included in this analysis, IFs ranged from 4.642 to 23.603, with a median IF of 6.730 and an interquartile range of 5.338 to 12.171. Efs ranged from 0.00164 to 0.20502, with a median Ef of 0.01884 and an interquartile range of 0.01132 to 0.02905. Spearman rank correlations between journal IF and Ef and their EiC's publication records are shown in Table 2.

Considering the IF, no significant correlation was found between the IF of a journal and the total number of publications by its EiC. However, weak to moderate significant correlations were found between the IF and the numbers of first-author publications by the EiC, commentaries by the EiC, and first-author commentaries by the EiC. Considering the Ef, the only significant correlation was between the Ef of a journal and the number of commentaries by its EiC. The magnitudes of the IF and Ef correlations with the number of commentaries and first-author commentaries by the EiC were not significantly different (p=0.261 and 0.193, respectively).

DISCUSSION

In this study, we analyzed correlations between various categories of EiCs' publications and scientometric indices (i.e., IF and Ef) of their journal while controlling for the number of years served by the EiC. We found weak to moderate, statistically significant correlations between the number of commentaries published by EiCs and these scientometric indices.

Previous literature shows an association between journal self-citation and the IF [6]. Journal self-citation typically focuses on a single paper that can boost an IF when published. Opthof explains that something as mundane as the Journal of the American College of Cardiology releasing a “Highlights of the Year” paper would allow their IF to rise by 0.309. This rise in IF would allow them to overtake Circulation, which at the time had the highest IF among cardiovascular journals, establishing the Journal of the American College of Cardiology as the journal with the highest IF in 2010 [7]. While journal self-citation has been identified as a technique that can be used to intentionally inflate the IF of journals, the fact that we found positive correlations between the EiCs' commentaries and IF and Ef does not provide evidence that EiCs are using self-citation or journal self-citation to manipulate the IF or Ef of their journals. Rather, these correlations may simply be a result of the fact that EiCs of journals with higher IFs publish more commentaries in an attempt to provide greater insight into the articles published in their journals.

Heneberg performed a citation analysis of publications appearing in eleven arbitrarily selected prominent general science and biomedical journals [16], which showed that original data-based research accounted for 12% to 79% of items published by the journals, whereas 11% to 44% of items published were considered editorial materials. Heneberg reports that these editorial materials, along with commissioned opinion articles in the parent journal and those published in other journals, accounted for up to 30% of the total citations in these top tier journals, thus affecting their IF and Ef. Furthermore, over a three-year period from 2009 to 2011, these journals reported a similar percentage of citations for data-based articles and reviews versus other editorial materials. Heneberg recommends excluding journal self-citations and certain document types (including editorial comments) from the calculation of the IF and Ef in an effort to limit both intentional and unintentional manipulation of these scientometric indices.

Because of the problems of journal self-citation and the use of different types of published items to calculate the numerator and denominator of the IF, Liu et al. suggest a number of solutions [14]. These solutions include devising new scientometric indices, such as the SCImago Journal Rank and Science Normalized Impact Per Paper indicators based on the Scopus database; computing the IF based on the type of article or item published by a given journal; and redefining the types of articles or items to be included in the numerator and denominator of the IF [14]. They evaluated the effect of adjusting the numerator and denominator of the IF based on the type of item published in ophthalmologic or mathematical journals. Their results showed that changes in the way in which the IF is calculated had an effect for ophthalmologic journals and thus may be valuable; however, the change did not seem to be useful when evaluating mathematic journals. Whether changes in the computation of the IF for cardiovascular medicine journals would be valuable is unclear.

Some limitations of the current study should be noted. Correlations were used to examine the relationship between EiCs' publications and their journals' IF and Ef. As such, a causal relationship between the two cannot be established. A few of the journals had two EiCs, and only the first one alphabetically listed was used for data collection. Due to the large number of publications analyzed, individual confirmation of whether each article was in fact a commentary was not performed—rather, whether the article had “no abstract” listed in PubMed was the criteria for denoting a commentary piece. Thus, a more precise and efficient method of enumerating commentaries is needed, such as developing a coding script to automate data collection. We examined the top cardiovascular medicine journals, but our analysis was not sensitive enough to determine whether particular journals engaged in excessive self-citation. A more detailed analysis of each individual journal would be required to obtain this information.

In conclusion, we found significant moderate correlations between a journal's IF and the number of all commentaries and first-author commentaries published by its EiC. Also, a weak but statistically significant correlation was found between a journal's Ef and the number of commentaries published by its EiC. Whether the methodology used to compute the IF and Ef for cardiovascular medicine journals should be modified requires further research.

DATA AVAILABILITY STATEMENT

Data associated with this article are available in the Figshare repository at: https://figshare.com/articles/dataset/JMLA_1145_Data_ sav/13725835.

REFERENCES

1. Carpenter CR, Cone DC, Sarli CC. Using publication metrics to highlight academic productivity and research impact. Acad Emerg Med Off J Soc Acad Emerg Med. 2014 Oct;21(10):1160–72.

2. Garfield E. Citation indexes for science. 1955;122:4.

3. Garfield E. The history and meaning of the Journal Impact Factor. JAMA. 2006 Jan 4;295(1):90.

4. Reuters T. What does it mean to be #2 in Impact?—EndNote Community [Internet]. [cited 2020 Aug 17]. <https://community.endnote.com/t5/Citation-Impact-Center/What-does-it-mean-to-be-2-in-Impact/ba-p/11386>.

5. Schutte HK, Svec JG. Reaction of folia phoniatrica et logopaedica on the current trend of impact factor measures. FOLIA Phoniatr Logop. 2007;59(6):281–5.

6. Miyamoto S. Self-citation rate and impact factor in the field of plastic and reconstructive surgery. J Plastic Surg Hand Surg. 2018 Feb:52(1):40–6. DOI: http://dx.doi.org/10.1080/2000656X.2017.1319847
.

7. Opthof T. Inflation of impact factors by journal self-citation in cardiovascular science. Neth Heart J. 2013 Apr;21(4):163–5.

8. Roldan-Valadez E, Orbe-Arteaga U, Rios C. Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: a linear mixed design model analysis of radiology, nuclear medicine and medical Imaging journals. Radiol Med (Torino). 2018 Jul 1;123(7):524–34.

9. Bergstrom CT, West JD. Assessing citations with the Eigenfactor^™ metrics. Neurology. 2008 Dec 2;71(23):1850.

10. Bergstrom CT, West JD, Wiseman MA. The Eigenfactor^™ metrics. J Neurosci. 2008 Nov 5;28(45):11433–4.

11. Opthof Tobias. Comparison of the impact factors of the most-cited cardiovascular journals. Circ Res. 2019 Jun 7;124(12):1718–24.

12. Reynolds JC, Menegazzi JJ, Yealy DM. Emergency medicine journal impact factor and change compared to other medical and surgical specialties. Acad Emerg Med. 2012;19(11):1248–54.

13. Smart P. Is the impact factor the only game in town? Ann R Coll Surg Engl. 2015 Aug 30;97(6):405–8.

14. Liu X-L, Gai S-S, Zhou J. Journal Impact Factor: do the numerator and denominator need correction? PLoS ONE [Internet]. 2016 Mar;11(3):e0151414. DOI: http://dx.doi.org/10.1371/journal.pone.0151414
.

15. Steiger JH. Testing pattern hypotheses on correlation matrices: alternative statistics and some empirical results. Multivar Behav Res. 1980 Jul 1;15(3):335–52.

16. Heneberg P. Parallel worlds of citable documents and others: inflated commissioned opinion articles enhance scientometric indicators. J Assoc Inf Sci Technol. 2014;65(3):635–43.

Alex Salazar, 1 salaas17@wfu.edu, Department of Health and Exercise Science, Wake Forest University, Winston-Salem, NC

Michael Joseph Berry, 2 berry@wfu.edu, Professor of Health and Exercise Science, Department of Health and Exercise Science, Wake Forest University, Winston-Salem, NC

Copyright © 2021 Alex Salazar, Michael Joseph Berry

This work is licensed under a Creative Commons Attribution 4.0 International License.

Journal of the Medical Library Association, VOLUME 109, NUMBER 3, July 2021