Citation bias in otolaryngology systematic reviews
DOI:
https://doi.org/10.5195/jmla.2021.736Keywords:
Citation Bias, Otolaryngology, Systemic Review, Hand-Search, Reference ListsAbstract
Objective: Reproducibility of systemic reviews (SRs) can be hindered by the presence of citation bias. Citation bias may occur when authors of SRs conduct hand-searches of included study reference lists to identify additional studies. Such a practice may lead to exaggerated SR summary effects. The purpose of this paper is to examine the prevalence of hand-searching reference lists in otolaryngology SRs.
Methods: The authors searched for systematic reviews published in eight clinical otolaryngology journals using the Cochrane Library and PubMed, with the date parameter of January 1, 2008, to December 31, 2017. Two independent authors worked separately to extract data from each SR for the following elements: whether reference lists were hand-searched, other kinds of supplemental searching, PRISMA adherence, and funding source. Following extraction, the investigators met to review discrepancies and achieve consensus.
Results: A total of 539 systemic reviews, 502 from clinical journals and 37 from the Cochrane library, were identified. Of those SRs, 72.4% (390/539) hand-searched reference lists, including 97.3% (36/37) of Cochrane reviews. For 228 (58.5%) of the SRs that hand-searched reference lists, no other supplemental search (e.g., search of trial registries) was conducted.
Conclusions: These findings indicate that hand-searching reference lists is a common practice in otolaryngology SRs. Moreover, a majority of studies at risk of citation bias did not attempt to mitigate the bias by conducting additional supplemental searches. The implication is that summary effects in otolaryngology systematic reviews may be biased toward statistically significant findings.References
Higgins JPT, Green S, ed. Cochrane handbook for systematic reviews of interventions: online version (5.1.0, Mar 2011) [Internet]. Cochrane Collaboration; 2011 [cited 1 Oct 2020]. <https://handbook-5-1.cochrane.org/>.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ. GRADE guidelines: 1. introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011 Apr;64(4):383–94.
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland S. Searching for and selecting studies. In: Higgins J, Thomas J, eds. Cochrane handbook for systematic reviews of interventions. Cochrane Collaboration; 2019.
Sterne J, Egger M, Moher D. 10.2.2.3 Citation bias. In: Higgins JPT, Green S, ed. Cochrane handbook for systematic reviews of interventions. Version 5.1.0. Cochrane Collaboration; 2011.
Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064–5.
Brooks TA. Private acts and public objects: an investigation of citer motivations. J Am Soc Inf Sci. 1985 Jul;36(4):223–9.
Gøtzsche PC. Reference bias in reports of drug trials. Br Med J (Clin Res Ed). 1987 Sep 12;295(6599):654–6.
Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992 Jul 4;305(6844):15–9.
Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. J Med Libr Assoc. 2016 Oct;104(4):302–4. DOI: http://dx.doi.org/10.3163/1536-5050.104.4.009.
Higgins J, Thomas J, eds. Preparing a Cochrane review [Internet]. Version 6.1. Cochrane Collaboration; 2020 [cited 1 Oct 2020]. <https://training.cochrane.org/handbook/current>.
Panic N, Leoncini E, de Belvis G, Ricciardi W, Boccia S. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013 Dec 26;8(12):e83138.
Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges Team. Optimal search strategies for retrieving systematic reviews from MEDLINE: analytical survey. BMJ. 2005 Jan 8;330(7482):68.
National Library of Medicine. MEDLINE Data changes—2019 [Internet]. The Library; 27 Nov 2018 [cited 2019 Jul 19]. <https://www.nlm.nih.gov/pubs/techbull/nd18/nd18_medline_data_changes_2019.html>.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015 Jan 1;4(1):1.
Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011 Aug 10;(8):MR000026.
Cooper C. Improving literature searching in systematic reviews: the application of tailored literature searching compared to “the conventional approach” [Internet]. University of Exeter; 2019 [cited 29 Jun 2020]. <https://ore.exeter.ac.uk/repository/handle/10871/35637>.
Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2012 Jan 3;344:d7762.
Rietjens JA, Bramer WM, Geijteman EC, van der Heide A, Oldenmenger WH. Development and validation of search filters to find articles on palliative care in bibliographic databases. Palliat Med. 2019 Apr;33(4):470–4.
