Amanda Ross-White, AHIP, Christina M. Godfrey, Kimberley A. Sears, Rosemary Wilson
doi: http://dx.doi.org/10.5195/jmla.2019.491
Received 01 April 2018: Accepted 01 September 2018
ABSTRACT
Objectives
The number of predatory journals is increasing in the scholarly communication realm. These journals use questionable business practices, minimal or no peer review, or limited editorial oversight and, thus, publish articles below a minimally accepted standard of quality. These publications have the potential to alter the results of knowledge syntheses. The objective of this study was to determine the degree to which articles published by a major predatory publisher in the health and biomedical sciences are cited in systematic reviews.
Methods
The authors downloaded citations of articles published by a known predatory publisher. Using forward reference searching in Google Scholar, we examined whether these publications were cited in systematic reviews.
Results
The selected predatory publisher published 459 journals in the health and biomedical sciences. Sixty-two of these journal titles had published a total of 120 articles that were cited by at least 1 systematic review, with a total of 157 systematic reviews citing an article from 1 of these predatory journals.
Discussion
Systematic review authors should be vigilant for predatory journals that can appear to be legitimate. To reduce the risk of including articles from predatory journals in knowledge syntheses, systematic reviewers should use a checklist to ensure a measure of quality control for included papers and be aware that Google Scholar and PubMed do not provide the same level of quality control as other bibliographic databases.
Since librarian Jeffery Beall first coined the term “predatory publishers” in 2010, listing nine journal publishers he believed engaged in questionable practices, the numbers of questionable academic publications have increased considerably [1]. Despite being explicit that he used the term “predatory” with caution, it is the name most frequently used to describe a nebulous concept of research journal publishers who use unethical business practices, minimal or no peer review, or limited editorial oversight to publish articles that are below a minimally accepted standard of quality [2]. In 2012, Beall began publishing regular blog postings on journals and publishers he deemed predatory, a list of journals to exclude that quickly grew in size, commonly called “Beall’s list.” Although he shut down the blog in January 2017, several archived versions are available.
There are no generally accepted criteria of what makes a journal predatory. This can pose a problem as both Beall’s list and its various replacements are not always clear in their definition of what makes a journal predatory or what a specific journal or publisher has done to gain entry to the list [3]. Predatory journals are often, but not exclusively, linked to open access publishing models, and an earlier reference to the proliferation of these journals linked the gold open access model of publishing to the rise in these questionable journals [4].
These journals are increasing rapidly in number and have the potential to alter the results of research syntheses. It is unknown how often predatory publications are cited in systematic reviews and other research syntheses. If the open access model used by many of these predatory publications makes them more accessible to systematic reviewers and their low standards lead to the publication of flawed or fraudulent research, future research could be compromised by prior errors. The authors sought to determine the degree to which articles from predatory publications are cited in systematic reviews.
As there is no consensus on criteria or strict definition of predatory journals, we selected a list of health and biomedical sciences journals from a publisher considered by many to be predatory, including the US Federal Trade Commission [5–8]. This journal publisher was chosen for two reasons: it was included on the now-defunct Beall’s list and legal action was taken against it by the US Federal Trade Commission [9]. From the list of journal titles on this publisher’s website, which numbered over 1,000, we created a list of 459 journals with health and biomedical sciences titles.
Using information from Google Scholar, we searched each predatory journal title individually and downloaded citations of articles published in these journals into EndNote x7 citation management software. We then used Google Scholar’s forward reference searching feature to determine whether the articles were cited in systematic reviews or meta-analyses. Only articles that used the words “systematic review” or “meta-analysis” in the title or clearly indicated in the abstract that a systematic process was used in the review methodology were included. Google Scholar was chosen to determine the number of citations of the indexed papers and to identify the citing systematic reviews, because most predatory journals are not indexed in traditional bibliometric databases and can only be found using Google and Google Scholar. Both index articles and citing reviews are available as an Excel spreadsheet in our institutional repository <http://dx.doi.org/10.5683/SP2/VMMBNH>. Searches were conducted from June to August 2017.
Of the 459 journals investigated, 145 (31.6%) appeared to have never published any articles but to exist in title only. A search of the publisher’s websites revealed no papers published in these 145 journals but only solicitations for submissions. Sixty-four journals (13.9%) had published at least 1 article, but none of these articles had been cited by another publication, peer-reviewed or otherwise. The remaining 250 journals (54.5%) had 6,302 articles that had been cited at least once. Of these, 120 articles were cited by at least 1 systematic review. These 120 articles were published in 62 unique journals. As each systematic review could cite more than 1 article, a total of 157 systematic reviews cited an article from a journal from this predatory publisher. Of these 157 systematic reviews that cited a paper from a predatory journal, only 16 were themselves published in a predatory journal, whereas 4 were book chapters or doctoral dissertations and 137 were published in journals believed to be reputable and published by major international publishers (e.g., Wiley, Springer, Elsevier, Taylor & Francis, Sage).
Of the 459 journal titles from our initial list, only 1 title was indexed in MEDLINE. Another 7 were indexed in Embase, and 2 were indexed in CINAHL. Nine of the 10 journal titles indexed in bibliographic databases were journals that had previously been published by reputable scientific organizations that had been bought by the predatory publisher after the decision to index the journals [5, 6]. In addition, 39 of the journals with articles cited in a systematic review or meta-analysis had select publications in PubMed Central (PMC) in compliance with public access policies requiring authors of National Institutes of Health–funded research to deposit completed manuscripts in PMC.
The purpose of peer review is to establish and maintain a standard for research in a particular field. This standard not only keeps researchers honest, but also allows readers (or authors of a systematic review) to feel secure that the rigor of the research and interpretation of the findings has been addressed and maintained by the publishing journal and that the article can be considered to make a valuable contribution to the state of knowledge. Given that not all readers are experts in content or methodology, the contribution of a manuscript without the standards provided by peer review is suspect at best.
The publisher we selected for this examination is allegedly engaged in questionable business practices and is known to have limited, or even nonexistent, peer review. In particular, the publisher has been found in a US federal court injunction to not engage in peer review, to fraudulently misrepresent their impact factor, and to engage in deception by failing to adequately disclose publication fees [9]. For this reason, we are concerned to see continued citing of research published in these journals, particularly in research syntheses, without knowing whether the authors of the reviews are aware of this problematic history of the journals.
Systematic review authors need to be vigilant for articles published in predatory journals that can appear to be legitimate. Using a checklist or critical appraisal tool to ensure that only papers of high quality are included in a research synthesis will reduce the risk of including poor-quality research and potentially changing outcomes on that basis. This is the case for both poor-quality research published in high-quality journals and poor-quality research published in predatory journals. A plethora of critical appraisal tools are available for evaluating research by employing many different research methodologies [10–13]. Systematic review publishers such as Cochrane, Campbell, the Evidence for Policy and Practice Information (EPPI), and the Joanna Briggs Institute all recommend the use of critical appraisal tools to evaluate the quality of research cited in a systematic review [14, 15].
While the criteria for what makes a publication predatory are fluid, there are some options for both researchers publishing articles and authors of knowledge syntheses for checking the validity of publications. This list of options is by no means exhaustive and is subject to change:
Check to see if the publisher is a member of reputable publishing organizations, of which some examples are World Association of Medical Editors (WAME), Committee on Publication Ethics (COPE), and the International Academy of Nursing Editors (INANE) [16].
Confirm that journal websites contain accurate and current information from an independent source (e.g., if the website indicates that the journal is indexed in PubMed, check MEDLINE to confirm).
Consult Jeffery Beall’s original published criteria [17], or use Think. Check. Submit.
To reduce the likelihood that systematic reviewers engage with predatory publications, the producers of bibliographic databases—including MEDLINE, Embase, and CINAHL—should maintain awareness of the changing ownership of journals, as nine of the ten predatory journals indexed in bibliographic databases in this study were formerly legitimate journals that had been sold to a predatory publisher [5]. To further limit the impact on research syntheses of flawed and potentially fraudulent research, awareness on the part of researchers that Google Scholar and PubMed do not provide the same level of quality control as other bibliographic databases is critical. Authors may have themselves added the articles to PubMed Central under the public access policy.
Other researchers have found similar instances in which predatory journals were being included in high-quality bibliographic databases [18]. Researchers also need to be aware of the differences between PubMed and MEDLINE and what it means for a journal to be indexed in MEDLINE as opposed to appearing in PubMed, in some instances due to author-submitted manuscripts.
It is also important to note that not everything published in a predatory journal is fraudulent or of poor quality. Ethical researchers can also be caught in the predatory trap. It is also true that many systematic reviews do not limit their inclusion criteria to the peer-reviewed literature, including, for example, grey literature. However, if systematic review authors include predatory publications on the presumption of peer review, this poses a problem. Future research should evaluate the quality of the indexed papers using validated tools. It is unknown what percentage of articles in predatory publications is of insufficient quality to be published elsewhere. We also encourage PMC to ensure the quality of author-supplied submissions.
When systematic reviewers are searching for studies, they should exercise caution when an article is only found in Google Scholar and not through a search of bibliographic databases. This can be an indication that either the search in bibliographic databases needs improvement (because known indexed articles are not being found) or the articles may be in a predatory journal that is not included in reputable bibliographic databases. Other researchers have similarly expressed caution about the use of Google Scholar as a source in systematic reviews and meta-analyses [19, 20], as well as its use as a research tool in general [21, 22]. As only 10 of the 459 investigated journals were indexed in bibliographic databases—9 of which because they were small society publishers that had been purchased by a larger predatory organization—it would appear that these predatory publications are consistently failing to meet indexing criteria for these databases. While there may be a myriad of reasons that legitimate research is not indexed by any given bibliographic database, such as scope of the research or publication date, a failure to meet the criteria of all of the most significant bibliographic databases should raise some concerns.
Both bibliographic databases and journals are in a continual state of change, so these results are not generalizable and subject to change in future. However, predatory publications are not likely to disappear any time soon. Systematic reviewers must be cautious both in evaluating research by carefully considering its source and in seeking publication themselves.
The authors thank our research assistant Yasmeen Maurice for her help in checking citations. This research has been funded by the Medical Library Association’s Research, Development, and Demonstration Project Grant. As no human participants were included in this research, ethics approval was neither required nor sought. This research was previously presented in poster form at the Global Evidence Summit, Cape Town, South Africa; September 2017.
1 Beall J. “Predatory” open-access scholarly publishers. Charleston Advisor. 2010 Apr;11(4):10–7.
2 Bartholomew RE. Science for sale: the rise of predatory journals. J R Soc Med. 2014 Oct;107(10):384–5.
3 Singh Chawla D. The undercover academic keeping tabs on ‘predatory’ publishing. Nature. 2018;555(7697):422–3.
4 Harnad S. The dot-gold rush for open access. Open Access Archivangelism [Internet]. 15 Jul 2008 [cited 29 Mar 2018]. <http://openaccess.eprints.org/index.php?/archives/432-The-Dot-Gold-Rush-for-Open-Access.html>.
5 Brown C. Alleged predatory publisher buys medical journals. CMAJ. 2016 Nov 1;188(16):E398.
6 Brown C. Medical society journals face publishing woes. CMAJ. 2016 Nov 1;188(16):1143.
7 Masic I. Predatory publishing - experience with OMICS International. Med Arch. 2017 Oct;71(5):304–7.
8 Molteni M. The FTC is cracking down on predatory science journals. Wired [Internet]. 19 Sep 2016 [cited 27 Sep 2018]. <https://www.wired.com/2016/09/ftc-cracking-predatory-science-journals/>.
9 Complaint for Permanent Injunction and Other Equitable Relief, Federal Trade Commission v. OMICS Group, (2016), no. 2:16-cv-02022 (Nev. Dist. Ct. Aug. 25, 2016). (Available from: <https://www.ftc.gov/system/files/documents/cases/160826omicscmpt.pdf>. [cited 27 Sep 2018].)
10 Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. Int J Health Policy Manag. 2014 Aug;3(3):123–8.
11 Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005 May 7;330(7499):1053.
12 Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013 Aug;19(4):722.
13 Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004 Sep 16;4:22.
14 Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions [Internet]. Version 5.1.0. The Cochrane Collaboration; 2011 [updated Mar 2011; cited 27 Sep 2018]. <http://www.handbook.cochrane.org>.
15 Aromataris E, Munn Z, eds. Joanna Briggs Institute reviewers’ manual [Internet]. Adelaide, Australia: Joanna Briggs Institute; 2017 [cited 27 Sep 2018]. <https://reviewersmanual.joannabriggs.org>.
16 Laine C, Winker MA. Identifying predatory or pseudo-journals. Biochem Med (Zagreb). 2017 Jun 15;27(2):285–91.
17 Beall J. Criteria for determining predatory open-access publishers [Internet]. 2015 [updated 1 Jan 2015; cited 27 Sep 2018]. <https://beallslist.weebly.com/uploads/3/0/9/5/30958339/criteria-2015.pdf>.
18 Dadkhah M, Lagzian M, Borchardt G. Questionable papers in citation databases as an issue for literature review. J Cell Commun Signal. 2017 Jun;11(2):181–5.
19 Bramer WM, Giustini D, Kramer BM. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev. 2016;5:39.
20 Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Methodol. 2013;13:131.
21 Halevi G, Moed H, Bar-Ilan J. Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation-review of the literature. J Informetr. 2017 Aug;11(3):823–34.
22 Fagan JC. An evidence-based review of academic web search engines, 2014–2016: implications for librarians’ practice and research agenda. Inf Technol Libr. 2017;36(2):7–47.
Alison Amanda Ross-White, AHIP, amanda.ross-white@queensu.ca, https://orcid.org/0000-0003-4737-0968, Associate Librarian, Bracken Health Sciences Library, Queen’s University, Kingston, ON, Canada
Christina M. Godfrey, christina.godfrey@queensu.ca, Associate Professor, School of Nursing, Queen’s Joanna Briggs Collaboration, Queen’s University, Kingston, ON, Canada
Kimberley A. Sears, kim.sears@queensu.ca, Associate Professor and Associate Director (Healthcare Quality), School of Nursing, Queen’s University, Kingston, ON, Canada
Rosemary Wilson, rosemary.wilson@queensu.ca, https://orcid.org/0000-0003-3262-243X, Associate Professor, School of Nursing, Queen’s University, Kingston, ON, Canada
Articles in this journal are licensed under a Creative Commons Attribution 4.0 International License.
This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press.
Journal of the Medical Library Association, VOLUME 107, NUMBER 1, January 2019