Original Investigation

Search methods for prognostic factor systematic reviews: a methodologic investigation

Leah Boulos¹, Rachel Ogilvie², Jill A. Hayden³

doi: http://dx.doi.org/10.5195/jmla.2021.939

Volume 109, Number 1: 23-32
Received 01 2020: Accepted 09 2020

ABSTRACT

Objective:

This study retroactively investigated the search used in a 2019 review by Hayden et al., one of the first systematic reviews of prognostic factors that was published in the Cochrane Library. The review was designed to address recognized weaknesses in reviews of prognosis by using multiple supplementary search methods in addition to traditional electronic database searching.

Methods:

The authors used four approaches to comprehensively assess aspects of systematic review literature searching for prognostic factor studies: (1) comparison of search recall of broad versus focused electronic search strategies, (2) linking of search methods of origin for eligible studies, (3) analysis of impact of supplementary search methods on meta-analysis conclusions, and (4) analysis of prognosis filter performance.

Results:

The review ’s focused electronic search strategy resulted in a 91% reduction in recall, compared to a broader version. Had the team relied on the focused search strategy without using supplementary search methods, they would have missed 23 of 58 eligible studies that were indexed in MEDLINE; additionally, the number of included studies in 2 of the review ’s primary outcome meta-analyses would have changed. Using a broader strategy without supplementary searches would still have missed 5 studies. The prognosis filter used in the review demonstrated the highest sensitivity of any of the filters tested.

Conclusions:

Our study results support recommendations for supplementary search methods made by prominent systematic review methodologists. Leaving out any supplemental search methods would have resulted in missed studies, and these omissions would not have been prevented by using a broader search strategy or any of the other prognosis filters tested.

INTRODUCTION

Systematic reviews of prognosis studies are being published at an increasing rate. Such reviews typically address one or more of the following aims: identify the most likely course for a specific condition (overall prognosis) [1]; identify what characteristics are associated with, or predict, a given outcome (prognostic factors) [2]; identify cohorts with specific characteristics who are more or less likely to experience a given outcome (predictive models) [3]; and/or identify the characteristics or factors that impact the effectiveness of a specific treatment (treatment effect modification) [4]. Each of these aims synthesizes different aspects of prognosis or clinical prediction studies, and therefore, each requires distinct search approaches.

Prognosis studies can be difficult to retrieve in traditional searches of electronic databases [5]. Although several search filters have been developed and validated to identify prognosis studies [6–9], the sensitivity of these filters tends to be low [9, 10]. There is also a lack of consensus on the best approach to search for prognosis studies [5, 10]. This is in large part due to the variability in language that is used to describe prognosis in research articles. Without consistent prognosis terminology, it is difficult for filter developers to determine the optimal combination of search terms. Prognosis-related controlled vocabulary tends to be quite broad, and prognosis studies are often poorly and/or inconsistently indexed [5].

Prognostic factor studies pose additional challenges for searchers. They present similar issues with inconsistent terminology, and the addition of search terms that are related to the prognostic factors of interest can significantly reduce retrieval while also reducing sensitivity, increasing the likelihood of missing relevant studies. Riley et al. published a guide to systematic review and meta-analysis of prognostic factor studies in 2019 [11], but its guidance on searching was specific to clinical prediction studies, rather than prognostic factor studies or prognosis studies in general. Searches for clinical prediction studies require their own unique language that can be found in existing search filters [12–16]. This leaves a gap in the understanding of best practices for searching for prognostic factor studies and prognosis studies in general.

Greenhalgh and Peacock found that “protocol-driven” database searches and hand searches alone resulted in only 30% of sources being found in a systematic review of complex evidence [17]. Later, in an overview of 17 reviews of low back pain prognosis, Hayden et al. found many methodological shortcomings in the reviews that they studied, including inadequate search approaches [18]. These findings were corroborated in a subsequent Cochrane review protocol by Hayden et al. [19], which reported that of the more than 35 prospective cohort studies that were likely to be included in their review, less than 30% were included in 3 existing systematic reviews on the same topic [20–22]. Furthermore, those existing reviews included only a small number of overlapping studies, suggesting that each employed different approaches to searching.

The authors posit that traditional, highly sensitive searches of electronic databases are inadequate to systematically retrieve prognosis studies and that combining a more focused electronic database search with supplementary search methods can be a more effective approach. We define supplementary search methods as any means of locating studies other than query-based database searching. This most often includes hand searching, reference searching and citation tracking (sometimes called backward and forward searching), liaising with subject area experts, and consulting researchers ’ personal files. These methods are demonstrably effective approaches in several case examples [17, 23–26] and are recommended in the searching chapter of the Cochrane Handbook [27]. This approach harnesses the human capacity for evaluation and judgment, which can be difficult to communicate through traditional electronic database search language—especially when search concepts are difficult to define and use inconsistent terminology. Additionally, by extending beyond the scope of a database search alone, supplementary search methods help mitigate the risk of bias introduced by using prognostic factor terms to focus an electronic search [28].

To test our hypothesis and make recommendations for future practice, we conducted a methodologic investigation of Hayden et al. ’s review on the association between recovery expectations and disability outcomes in adults with low back pain [28]. This review is one of the first systematic reviews of prognostic factor studies in the Cochrane Library. It is notable for featuring a novel search strategy that combined a focused electronic database search with extensive supplementary search methods. Our objective was to retrospectively investigate and assess the search methods used in Hayden et al. ’s review, highlighting experiences and lessons learned in order to provide guidance for future reviewers of prognostic factor studies.

METHODS

Using data from Hayden et al. ’s example review [28], we used four approaches to comprehensively assess aspects of systematic searching for prognostic factor studies: (1) comparison of search recall of broad versus focused electronic search strategies, (2) linking of search methods of origin for eligible studies, (3) analysis of impact of supplementary search methods on meta-analysis conclusions, and (4) analysis of prognosis filter performance.

Comparison of search recall of broad versus focused electronic search strategies

The first component of Hayden et al. ’s search strategy was a focused electronic search for prognostic factor studies (supplemental Appendix A) that combined search terms for low back pain, a prognosis filter, and terms related to the prognostic factor of interest, in this case, recovery expectations. To assess the feasibility of running a broad electronic search versus a focused search combined with supplementary methods, in November 2019, we used Ovid MEDLINE to rerun Hayden et al. ’s focused search (low back pain AND prognosis filter AND recovery expectations) and compare its recall with that of a broader search without recovery expectations terms (low back pain AND prognosis filter).

Linking of search methods of origin for eligible studies

In addition to database searching, Hayden et al. used multiple supplementary search methods: a reference search of previously published systematic reviews, which were retrieved using a broader electronic search that did not include the prognostic factor concept; a forward search of publications citing identified prognostic factor measures; and hand searching, reference searching of included studies, and consultation of personal files to identify additional studies. To determine which studies would have been missed had any of the search strategy components been left out, we linked each included study in Hayden et al. to its search methods of origin, focusing only on those studies indexed in Ovid MEDLINE (n=58 of 60 total included studies).

To confirm which studies were retrieved by the focused electronic search strategy, we created a search string comprising the PubMed identifiers (PMIDs) of each of the fifty-eight included MEDLINE-indexed studies and combined it with the search that the team used. We consulted the team ’s search records to determine which studies had been retrieved by supplementary search methods. Finally, in addition to tracing the origin of each included study back to the search methods used in the review, we explored a hypothetical scenario in which only the broad electronic search strategy was used.

Analysis of impact of supplementary search methods on meta-analysis conclusions

Hayden et al. ’s review included unadjusted and adjusted meta-analyses (MAs) of the effect of recovery expectations on four outcomes: work participation, important recovery outcomes, functional limitations, and pain intensity. To gauge the impact of supplementary search methods on the MAs ’ conclusions, we repeated them with only those studies retrieved by the focused electronic search strategy. We then ran a similar experiment to see if the hypothetical broader search would have made any difference to the MAs ’ conclusions.

Analysis of prognosis filter performance

We conducted a performance analysis of the prognosis filter used in Hayden et al. ’s review (Irvin filter) [29] compared with three other known filters: the optimized version of a filter developed by the Hedges Team at McMaster University (“Hedges Optimized” filter) [9]; an inclusive, general filter developed by Parker et al. (“Inclusive General” filter) [30]; and another filter developed by Parker et al. combining all of the Hedges Team ’s filters [9] plus the keywords “natural history” (“Combined Hedges + Natural History” filter) [30]. Table 1 shows the details of all filters we analyzed in Ovid MEDLINE format. Although the four filters contain similar terms, they demonstrate the slight variance in language that is often used when searching for prognosis studies. Through our analysis, we explored whether these minute differences affected recall.

The performance analysis measured operating characteristics for each filter against a reference standard of records in Ovid MEDLINE using formulae outlined by Kok et al. [31] and Gehanno et al. [32]. To provide richer ground for analysis, we wanted to expand our reference standard beyond the sixty studies included in the final stage of Hayden et al. ’s review. As detailed in the original Hayden et al. review, to mitigate the risk of introducing bias by including prognostic factor terms in the electronic search, the team advanced all citations—retrieved by any of the search methods detailed thus far—of low back pain prognostic factor studies, regardless of the presence or absence of the specific prognostic factor of interest. We, therefore, drew the reference standard from prognosis studies that were included at the first stage of screening in the review. We used the reference standard to calculate each filter ’s sensitivity, precision, number needed to read (NNR), specificity, and accuracy.

RESULTS

Comparison of search recall of broad versus focused electronic search strategies

The broad electronic search strategy (no prognostic factor component) retrieved 15,242 records in MEDLINE (run November 2019). The focused electronic search strategy (prognostic factor—i.e., “expectations”—terms included) retrieved 1,332 records. Including prognostic factor terms resulted in a 91% reduction in the number of retrieved records.

Linking of search methods of origin for eligible studies

Figure 1 shows the search methods of origin for each included study indexed in MEDLINE (n=58 of 60 included studies). The numbers in each column do not add up to 100, as some studies were retrieved by multiple methods. The underlying data for this figure are found in supplemental Appendix B.

* Search recall and search filter analyses in this methodologic investigation were limited to Ovid MEDLINE. Incidentally, the two studies not located in Ovid MEDLINE were not included in the review ’s meta-analyses.

† A broad electronic database search for primary studies was not undertaken as part of the review ’s actual search methods; this is a hypothetical search to show the difference in recall.

Figure 2 visualizes the overlap in search results from each component of the search strategy. The focused electronic database search for primary studies retrieved thirty-five records; the systematic review reference search retrieved twenty; the forward search of publications citing identifying prognostic factor measures retrieved four; and hand searching, reference searching, and consulting personal files retrieved eighteen. There was overlap between some components: seventeen records were retrieved by both of the first two methods, and two were retrieved by both the focused electronic search and the forward search for prognostic factor measures. The portions of the diagram that do not overlap with any other portions represent studies that would not have been found by any other method. Because each method features a portion without any overlap, the diagram demonstrates that none of the methods could have been left out. Most notable of these was hand searching, which did not overlap with any of the other methods, as the purpose and approach of hand searching is to locate studies that are not retrieved by other previous methods.

PF=prognostic factor.

* Figures offer visual estimates for illustrative purposes only and are not mathematically proportional.

In the hypothetical scenario in which only a broad electronic search strategy was used, fifty-three of fifty-eight studies indexed in Ovid MEDLINE would have been retrieved. Figure 3 visualizes this hypothetical scenario. Although the portions with overlap are larger than in Figure 2, the portions without overlap demonstrate that even with a broader electronic database search, supplementary search methods would still have been necessary to retrieve all included studies.

* Figures offer visual estimates for illustrative purposes only and are not mathematically proportional.

Analysis of impact of supplementary search methods on meta-analysis conclusions

Removing studies that were not found by the focused electronic search resulted in changes in two of the four primary outcome MAs: work participation and important recovery outcomes. The missed studies did not report data for MAs of functional limitations or pain intensity outcomes, so there were no changes to these MAs. For the former, Table 2 shows the comparison between the results of the original MAs (unadjusted and adjusted) with the results of the MAs of only those studies retrieved by the focused electronic search (“Partial [focused search] meta-analysis [MA]”). In the original MA on work participation outcomes, ten studies were included in the unadjusted MA and twelve were included in the adjusted MA. In the partial MA, those numbers would have dropped to three studies (unadjusted) and four studies (adjusted). In the recovery outcomes MA, the number of included studies would have dropped from three to two (unadjusted) and five to four (adjusted).

* Dichotomous measure of expectations (follow-up closest to 12 months).

†Hayden et al. included study results from the best adjusted model available considering 5 domains of covariates: individual demographics, social support, work factors and environment, psychological factors, and low back pain complaint factors.

MA=meta-analysis; CI=confidence interval.

In the case of the meta-analysis with the largest number of studies available (work participation), the number of participants included in the original MA (n=4,528 in unadjusted MA; n=4,777 in adjusted MA) decreased significantly in the partial MA (n=525 in unadjusted MA; n=731 in adjusted MA), leaving out data on 4,003 potential participants in the unadjusted MA and 4,046 potential participants in the adjusted MA. In our second experiment, limiting to the hypothetical broad electronic search on its own resulted in changes to 1 of the 4 primary outcome MAs: work participation. As shown in Table 2, the number of included studies dropped from 10 to 9 (unadjusted) and 12 to 10 (adjusted). The underlying data for this analysis are found in supplemental Appendix C.

Interpretation of the results did not change in either experiment, as the studies included in the review reported consistently positive associations between expectations and outcomes. In the partial MA, however, confidence intervals (CIs) would have been larger in every case except the adjusted MA of important recovery outcomes. In the work participation outcomes partial MA, the odds ratio (95% CI) would have changed from the original 4.11 (3.46, 4.89) to 4.38 (2.48, 7.74) in the unadjusted MA, and from 2.43 (1.64, 3.62) to 2.94 (1.86, 4.64) in the adjusted MA. In the important recovery outcomes partial MA, the odds ratio (95% CI) would have changed from the original 2.40 (1.32, 4.37) to 2.83 (1.24, 6.45) in the unadjusted MA but remained similar in the adjusted MA (original MA: 1.89 (1.49, 2.41); partial MA: 1.88 (1.46, 2.41)).

Analysis of prognosis filter performance

The reference standard, drawn from the first stage of screening in the original review, numbered 272 citations. Performance analysis results for each prognosis filter are outlined in Table 3. The Irvin and Combined Hedges + Natural History filters demonstrated similar performances across the board, particularly in sensitivity (90.4% and 90.1%, respectively). The NNR was identical (n=5) for the Irvin, Hedges Optimized, and Inclusive General filters; for the Combined Hedges + Natural History filter, the NNR increased by 1 (n=6).

NNR=number needed to read.

DISCUSSION

To the best of our knowledge, this methodologic investigation is the first to explore supplementary search methods in the context of searching for prognosis studies. It is also the first effort to directly examine search methods used in a Cochrane review of prognosis studies. This study responds directly to recognized weaknesses in traditional approaches to searching for prognosis studies and aligns with suggestions for supplementary search methods that prominent systematic review methodologists have previously made [17, 26]. Although this is a single case study, the supplementary search methods designed for Hayden et al. ’s review can be adapted to suit the purposes of other reviews of prognosis studies.

Each of the individual search components used in the original review retrieved records that were not retrieved by any other component. Leaving out any of the search components (i.e., only running a focused database search) would have resulted in missed studies and would have had an impact on the number of studies included in some of the MAs. Although the observed changes to the MAs ’ conclusions were not very remarkable in our investigation, it is important to note that the original review consistently found strong, positive associations between the prognostic factor and outcomes of interest; in other words, the review ’s conclusions were neither ambiguous nor surprising. This made it unlikely that changes to the number of studies that were included would change the outcomes of these MAs. In typical reviews, where the evidence base includes much more uncertainty, it is likely that missing up to 70% of studies would have a far more significant impact on MA conclusions.

Could using a single, broader electronic search have prevented any of these issues? Our exploration of the recall of the focused electronic search strategy that the team used (n=1,332) versus that of a hypothetical broader search (n=15,242) suggests that the broader search, while more sensitive, would have been less feasible to screen. We also found that even the broad search would not have found all studies without also using supplementary search methods. While some review teams may feel more secure running a broader search that leaves fewer stones unturned, we argue that in this case, running a more focused search and putting more comprehensive efforts into supplementary search methods is the more efficient way to locate prognosis studies. Although using prognostic factor terms to focus the search increased the risk of bias in the search results, using supplementary search methods—along with broad inclusion criteria as described by Hayden et al. [28]—helps to mitigate that risk.

The choice of search approaches, however, is highly dependent on the context of the review. While some supplemental search methods (e.g., hand searching, reference searching, forward searching, and consultation of personal files) are possible no matter what the subject matter, some of the supplementary search methods that were investigated in this study would not have been possible if the review had not been in such an active field of research. Reference searching of existing systematic reviews is only possible if there have been previous attempts to synthesize the evidence on a topic. Similarly, identifying relevant prognostic factor measures can be more straightforward if the research team is already aware of literature on a topic. Thus, the supplementary search methods we investigated may be best applied to large reviews of well-established prognosis topics, and those conducting reviews in newer research areas may be better advised to run a broad electronic search.

The large size of Hayden et al. ’s review affects our analysis in other ways. The difference in recall between a focused and broad electronic search strategy might not be so stark in a smaller review, potentially making supplementary search methods less productive. However, even in smaller reviews, supplementary searching remains necessary to retrieve all relevant studies, at least until prognosis studies become more findable (e.g., by applying better indexing and using more consistent prognosis terminology) or until databases improve controlled vocabulary and synonymy for prognosis concepts.

Our prognosis filter analysis found that the filter that the original review team used (the Irvin filter [29]) had the highest sensitivity (90.4%) of any of the other filters we tested and had an identical or near identical NNR (n=5). Even though the Irvin filter ’s sensitivity was the best in our test group, it was still not sensitive enough to retrieve all of the studies in our reference standard. This again highlights the importance of not relying solely on a prognosis filter to locate prognosis studies. Furthermore, our findings suggest it may be unreasonable for searchers to expect that a single, ideal prognosis filter could possibly be created to apply in all prognosis searching contexts. However, our ad hoc filter testing is not a substitute for systematically testing every prognosis filter that has been validated in the literature—our team is aware of one such effort currently being undertaken [5]—nor can it take the place of a formal filter validation study.

It remains unknown whether the focused electronic search strategy used in this example is the “ideal” strategy. While the comprehensiveness of a search strategy can be difficult to absolutely ascertain, the team may have benefitted from using a more iterative, “pearl growing,” search development approach to harvest additional relevant search terms [33]. Iterative search development is particularly important in the prognostic factor context, in which terminology can vary from topic to topic. Using a more iterative approach might have increased the sensitivity of the focused search without adding greatly to recall and could have been very effective in this case, where a large body of evidence—including previous syntheses—was already in existence.

Our analysis was done retrospectively, and it was impossible to make firm time estimates for each of the search components used. Future research on prognosis search methods could conduct a more rigorous, real-time comparison of traditional versus supplementary search methods. Future research could also explore iterative search methods in a prognosis context. Finally, methodologic investigations of reviews of prognosis on topics other than low back pain would add greatly to the field of prognosis search methods research.

DATA AVAILABILITY STATEMENT

All data associated with this article are available in supplemental Appendix D.

REFERENCES

1. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KGM, Steyerberg EW, Roberts I, Schroter S, Altman DG, Riley RD; PROGRESS Group. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013 Feb 5;346:e5595-e. DOI: http://dx.doi.org/10.1136/bmj.e5595
.

2. Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, Hemingway H; PROGRESS Group. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380. DOI: http://dx.doi.org/10.1371/journal.pmed.1001380
.

3. Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG; PROGRESS Group. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381. DOI: http://dx.doi.org/10.1371/journal.pmed.1001381
.

4. Hingorani AD, van der Windt DA, Riley RD, Abrams K, Moons KGM, Steyerberg EW, Schroter S, Sauerbrei W, Altman DG, Hemingway H; PROGRESS Group. Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ. 2013 Feb 5;346:e5793-e. DOI: http://dx.doi.org/10.1136/bmj.e5793
.

5. Corp N, Jordan JL, Hayden JA, Irvin E, Parker R, Smith A, van der Windt DA. Protocol: a systematic review of studies developing and/or evaluating search strategies to identify prognosis studies. Syst Rev. 2017 Apr 20;6(1):88. DOI: http://dx.doi.org/10.1186/s13643-017-0482-y
.

6. Yale University. Finding evidence in PubMed [Internet]. Newhaven, CT: Yale University; 2012 [updated 16 Jan 2015; cited 13 Jan 2020]. <https://library.medicine.yale.edu/tutorials/577>.

7. Walker-Dilks C, Wilczynski NL, Haynes RB; Hedges Team. Cumulative Index to Nursing and Allied Health Literature search strategies for identifying methodologically sound causation and prognosis studies. Appl Nurs Res. 2008 May;21(2):98–103. DOI: http://dx.doi.org/10.1016/j.apnr.2006.04.004
.

8. Wilczynski NL, Haynes RB. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. J Am Med Inform Assoc. 2005 Jul;12(4):481–5. DOI: http://dx.doi.org/10.1197/jamia.M1752
.

9. Wilczynski NL, Haynes RB; Hedges Team. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004 Jun 9;2:23. DOI: http://dx.doi.org/10.1186/1741-7015-2-23
.

10. Chatterley T, Dennett L. Utilisation of search filters in systematic reviews of prognosis questions. Health Inf Libr J. 2012 Dec;29(4):309–22. DOI: http://dx.doi.org/10.1111/hir.12004
.

11. Riley RD, Moons KGM, Snell KIE, Ensor J, Hooft L, Altman DG, Hayden J, Collins GS, Debray TPA. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019 Jan 30;364:k4597-k. DOI: http://dx.doi.org/10.1136/bmj.k4597
.

12. Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons KGM. Search filters for finding prognostic and diagnostic prediction studies in MEDLINE to enhance systematic reviews. PLoS One. 2012;7(2):e32844. DOI: http://dx.doi.org/10.1371/journal.pone.0032844
.

13. Holland JL, Wilczynski NL, Haynes RB; Hedges Team. Optimal search strategies for identifying sound clinical prediction studies in EMBASE. BMC Med Inform Decis Mak. 2005 Apr 29;5:11. DOI: http://dx.doi.org/10.1186/1472-6947-5-11
.

14. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001 Jul;8(4):391–7. DOI: http://dx.doi.org/10.1136/jamia.2001.0080391
.

15. Keogh C, Wallace E, O ’Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a web-based register. J Clin Epidemiol. 2011 Aug;64(8):848–60. DOI: http://dx.doi.org/10.1016/j.jclinepi.2010.11.011
.

16. Wong SSL, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges T. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc. 2003:728–32.

17. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064–5. DOI: http://dx.doi.org/10.1136/bmj.38636.593461.68
.

18. Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results—guidance for future prognosis reviews. J Clin Epidemiol. 2009 Aug;62(8):781–96.e1. DOI: http://dx.doi.org/10.1016/j.jclinepi.2008.09.004
.

19. Hayden JA, Tougas ME, Riley R, Iles R, Pincus T. Individual recovery expectations and prognosis of outcomes in non-specific low back pain: prognostic factor exemplar review. Cochrane Database Syst Rev. 2014(9). DOI: http://dx.doi.org/10.1002/14651858.CD011284
.

20. Fadyl J, McPherson K. Return to work after injury: a review of evidence regarding expectations and injury perceptions, and their influence on outcome. J Occup Rehabil. 2008 Dec;18(4):362–74. DOI: http://dx.doi.org/10.1007/s10926-008-9153-0
.

21. Hallegraeff JM, Krijnen WP, van der Schans CP, de Greef MHG. Expectations about recovery from acute non-specific low back pain predict absence from usual work due to chronic low back pain: a systematic review. J Physiother. 2012 Sep;58(3):165–72. DOI: http://dx.doi.org/10.1016/S1836-9553(12)70107-8
.

22. Iles RA, Davidson M, Taylor NF. Psychosocial predictors of failure to return to work in non-chronic non-specific low back pain: a systematic review. Occup Environ Med. 2008 Aug;65(8):507–17. DOI: http://dx.doi.org/10.1136/oem.2007.036046
.

23. Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It ’s in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health (Oxf). 2005 Dec;27(4):388–91. DOI: http://dx.doi.org/10.1093/pubmed/fdi056
.

24. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007 Apr 18(2):MR000001. DOI: http://dx.doi.org/10.1002/14651858.MR000001.pub2
.

25. Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015 Apr;68(4):412–7. DOI: http://dx.doi.org/10.1016/j.jclinepi.2014.10.008
.

26. Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement. Res Synth Methods. 2018 Jun;9(2):195–223. DOI: http://dx.doi.org/10.1002/jrsm.1286
.

27. Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS; Cochrane Information Retrieval Methods Group. Chapter 4: Searching for and selecting studies. In: Cochrane handbook for systematic reviews of interventions [Internet]. Version 6.0. Cochrane; Jul 2019 [cited 13 Jan 2020]. <https://training.cochrane.org/handbook/current/chapter-04>.

28. Hayden JA, Wilson MN, Riley RD, Iles R, Pincus T, Ogilvie R. Individual recovery expectations and prognosis of outcomes in non-specific low back pain: prognostic factor review. Cochrane Database Syst Rev. 2019 Nov 25;2019(11):CD011284. DOI: http://dx.doi.org/10.1002/14651858.CD011284.pub2
.

29. Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis. Presented at: 14th Cochrane Colloquium; Dublin, Ireland; 23–26 Oct 2006.

30. Parker R, Tougas ME, Hayden JA. Validating prognosis search filters using relative recall based on prognosis systematic reviews. Poster presented at: 21st Cochrane Colloquium; Quebec City, QC, Canada; 19–23 Sep 2013.

31. Kok R, Verbeek JAHM, Faber B, van Dijk FJH, Hoving JL. A search strategy to identify studies on the prognosis of work disability: a diagnostic test framework. BMJ Open. 2015 May 19;5(5):e006315-e. DOI: http://dx.doi.org/10.1136/bmjopen-2014-006315
.

32. Gehanno JF, Rollin L, Le Jean T, Louvel A, Darmoni S, Shaw W. Precision and recall of search strategies for identifying studies on return-to-work in MEDLINE. J Occup Rehabil. 2009 Sep;19(3):223–30. DOI: http://dx.doi.org/10.1007/s10926-009-9177-0
.

33. Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Inf Libr J. 2008 Dec;25(4):313–7. DOI: http://dx.doi.org/10.1111/j.1471-1842.2008.00825.x
.

SUPPLEMENTAL FILES

Appendix A: Focused Ovid MEDLINE search strategy used in Hayden et al. ’s Cochrane review on the association between recovery expectations and disability outcomes in adults with low back pain

Appendix B: Supplemental tables

Appendix C: Data used in meta-analyses of association of expectations with outcomes

Appendix D: Associated data

Leah Boulos, LeahM.Boulos@nshealth.ca, https://orcid.org/0000-0002-9849-383X, Evidence Synthesis Coordinator, Maritime SPOR SUPPORT Unit, Halifax, NS, Canada

Rachel Ogilvie, Rachel.Ogilvie@dal.ca, https://orcid.org/0000-0001-6789-7017, Research Program Coordinator, Department of Community Health and Epidemiology, Dalhousie University, Halifax, NS, Canada

Jill A. Hayden, JHayden@dal.ca, https://orcid.org/0000-0001-7026-144X, Associate Professor, Department of Community Health and Epidemiology, Dalhousie University, Halifax, NS, Canada

Copyright © 2021 Leah Boulos, Rachel Ogilvie, Jill A. Hayden

This work is licensed under a Creative Commons Attribution 4.0 International License.

Journal of the Medical Library Association, VOLUME 109, NUMBER 1, January 2021