Background
Evidence-Based Medicine (EBM) is a critical skill to be
developed in medical education training, essential for both school
accreditation[1] and performance in USMLE exams[2]. In addition, medical
students are required to demonstrate EBM competence prior to residency[3].
Librarians, with their expertise in question formulation, information
retrieval, and critical appraisal, are well-positioned to teach EBM. Their
involvement in curricula ranges from embedded librarianship in larger
courses[4, 5] and leading longitudinal, embedded courses[6-9] to delivering
discrete skills workshops[10-12] and offering elective courses[13].
EBM training typically includes instruction on the five
steps of EBM: question formulation, evidence retrieval and selection, critical
appraisal, application to the scenario, and reflection on performance in the
whole EBM task. Each of these steps is necessary for ensuring that medical
practice is informed by high-quality evidence [14].
While each step is individually important, researchers
suggest that whole-task approaches, in which all the steps are combined in a
single task, benefit learning in EBM [15].
In developing instruction and assessment of EBM skills,
we sought to understand the relationship between the quality of evidence
searches and the quality of the evidence ultimately retrieved, as the ability
to effectively search for and select relevant articles is a critical component
of EBM. This study examines potential correlations between the quality of
students’ searches and the quality of their article selections for a whole-task
assignment within an EBM course. We hypothesize that search quality (hereafter
“search”) and article selection quality (hereafter “selection”) for preclinical
medical students are positively correlated.
Case Presentation
At NYU Grossman School of Medicine, librarians
implemented a capstone assignment after a curriculum-integrated EBM course for
preclinical, first-year medical students. Although medical students may have
previous experience with research, the EBM course introduces new skills such as
how to frame a clinical question, develop effective search strategies, and
search biomedical databases efficiently. During the course, they are given
instructions and opportunities to conduct searches in different databases.
Curriculum and Setting
The pre-clinical EBM curriculum at NYU Grossman School of
Medicine consists of five required sessions, part of a year-long
multidisciplinary course, Foundational Clinical Skills. The sessions are taught
about once a month in the second half of the first-year curriculum. In
designing the course, we employed different pedagogical strategies proven to
help in knowledge retention, including a distributed practice of learning
rather than stand-alone courses[16], a whole-task EBM capstone assignment in
which students practice all five steps of EBM (Ask, Acquire, Appraise, Apply,
Assess) using several clinical scenarios [15], and small-group learning
activities.
The first module introduces students to EBM, the second
delves into finding evidence, and the remaining modules are devoted to critical
appraisal. The three critical appraisal classes are dedicated to the appraisal
of cohort studies, randomized controlled trials, and systematic reviews and
meta-analyses, all taught in a small group, flipped classroom setting. Before
and after this curriculum, students complete a pre- and post-test. After the
last critical appraisal class, students complete the whole-task assignment,
part of which is the focus of this research. This assignment was embedded in a
learning management system, where students submitted their answers and received
individual feedback on their performance.
The course lead is a medical librarian and the content
director for EBM in the preclinical undergraduate medical education curriculum,
having designed and taught these classes for five years. All other facilitators
are medical librarians with experience in teaching EBM to a variety of
audiences.
Assignment
As a whole-task activity, this assignment was built so
that learners would have the opportunity to practice the constituent components
of EBM in an integrated manner. This assignment was mandatory and was set up in
the learning management system Brightspace[17] as a quiz. Although students
received a grade, this was a formative assessment and did not count towards
their final Foundational Clinical Skills course grade. Learners had about ten
days to complete both this assignment and the post-test. After completion,
students were given narrative individual feedback.
Since this is taught in the preclinical curriculum,
learners practice with clinical scenarios instead of real patients. Based on
the scenario provided below, they were free to formulate their own questions
and select any article they believed to be appropriate to answer their
question. The goal was to select the best evidence to answer the question.
Your patient is 17 years old and recently has been
struggling with suicidal ideation. They are a recreational marijuana user with
no history of mental health disorders. You are wondering whether marijuana use
might be associated with suicidal thoughts and behavior.
Follow the five steps of EBM to examine the best and
most current evidence regarding this concern.
This scenario was used for two reasons. First, the EBM
classes in the NYU Grossman School of Medicine curriculum are integrated with
the organ systems content. In this case, the assignment aligned with the Brain
and Behavior content. Second, the specific topic was based on the number of
relevant PubMed articles that address the relationship between suicidal
ideation and marijuana use in adolescents. To ensure an efficient evaluation
process, we chose a topic that would yield a smaller number of articles when
following best practices in database searching. This would prevent assessors
from having to review an overwhelming amount of literature, which would
significantly delay the feedback response time. During the four years we used
this scenario, cohorts of just over 100 students selected only 10 to 15 unique
articles. These few articles on the topic have remained relatively consistent
from year to year.
To investigate the proposed research question, we focused
on analyzing two questions from the assignment:
1. List
your final search strategy (keywords AND filters used, if any).
2. Select
one article to answer your question.
With these questions, we sought to assess whether there
was a correlation between search strategy and the quality of the articles
selected.
Population
The population comprised 105 first-year medical students.
From this total, 82 students completed this assignment in spring 2023 and
agreed to have their data used for quality improvement reports, such as this
one. The data analyzed includes only de-identified datasets from consenting
students.
Answer Scoring
To help reduce bias and improve consistency, we developed
a rubric (appendix 1) specifically for this study. This rubric was developed
after scoring these assignments for a few years for previous cohorts and
applied to the questions described above, and it was revised by the three
authors. Additionally, we decided to assess the answers for search and
selection separately. By evaluating the two questions independently, we aimed
to prevent potential bias from students’ responses in the searching questions
influencing the assessment of the selection answers.
Only one rater (JM) scored the assignment. Each of the
two questions analyzed received a score from 1 (does not meet expectations) to
4 (exceeds expectations). Note that this rubric was developed for this specific
scenario and should not be used for other scenarios without adaptations. As
mentioned, the literature regarding the relationship between marijuana use and
suicidal ideation in adolescents is sparse. In addition, because of the nature
of this topic, certain study designs are not feasible. Hence, the available
systematic reviews and meta-analyses would constitute stronger evidence than
one observational study alone — we want to underscore that this is not the case
for all clinical questions.
Statistical analysis
Because the data consists of ordinal data, we used
Spearman's rank correlation coefficient to test whether there was a
relationship between the two datasets. The test's goal is to assess whether the
change in the magnitude of one variable is associated with a change in
another's, either in the same (positive correlation) or the opposite (negative
correlation) direction[18].
Results
Eighty-two students completed this assignment. The
possible scores for searching were posed as a scale ranging from one (does not
meet expectations) to four (exceed expectations). As the rubric shows (appendix
1), there was the possibility of a 0.5 points deduction for pre-established
events. Most learners (about 59%) received a score of three, while only 11%
received the maximum score of four (figure 1).
The same grading scale (1-4, with possible deductions)
was applied to the evidence selection rubric. For this question, most learners
(about 77%) received the maximum score of four, exceeding expectations. These
students selected a recent systematic review related to the scenario.
Table 1 lists the articles chosen by these students and
the number of times that title was selected: 82 students chose 16 different
articles. Thirty-two students (39%) selected the two most current systematic
reviews on this topic at the time. Interestingly, learners also selected book
chapters, cross-sectional studies, and older articles, although less frequently
— these types of articles do not correspond to the best, most current evidence.
(denoted as ⊙,
blue) and search scores higher than selection scores (denoted as -, pink).
Fifty-one students (62%) who scored 3 or more in their search had an equal or
higher score in their article selection (*). For students with lower search
scores, there was also a trend where their article selection scores were equal
or higher than their searches (⊙).
Only nine students (11%) had a search score higher than their selection score
(-).
The table shows that 62% of high scoring students (green
* cells) scored equal or higher in article selection compared to searching,
while 26% (blue ⊙
cells) of low scoring students had an equal or higher score in article
selection. Among students who scored 3 or higher in their search, 86% (51
students) also scored 3 or higher on article selection. Conversely, 69% (16
students) who scored 2.5 or lower in searching achieved 2.5 or higher in
article selection, with an absolute difference of 17%. To assess the
relationship between these variables, we calculated the Spearman’s correlation
coefficient. The result (r=0.19, p-value = 0.086) indicates a weak, positive
correlation that is not statistically significant.
Discussion
This study attempted to determine whether there was a
correlation between the quality of searches and the quality of article
selection for a whole-task EBM assignment for preclinical medical students. We
hypothesized that there would be a trend where higher scores in searches would
lead to higher scores in selection. After analysis, the data showed that there
was a weak, positive correlation between these two questions, but the findings
were not statistically significant (p-value = 0.086).
The contingency table, however, provides some insight
into the relationship between search quality and article selection based on
this assignment. The fact that a portion of students with lower search scores
(≤2.5) still managed to select appropriate articles suggests that
students may be compensating for poor search skills through browsing, searching
in other databases, or using other strategies. In addition, many students had
article selection scores equal to or higher than their search scores. This
could indicate that students are better at selecting appropriate articles than
they are at searching. It is also possible that the assignment design
facilitated good selection even with suboptimal searches, as we selected a
topic with narrow available evidence on purpose.
Several practical implications arise from this study.
First, instructors should explicitly address the issue of students selecting
older articles, book chapters, or cross-sectional studies when more current and
less prone to bias evidence is available. Instructors should consider providing
specific instruction on how to identify the most up-to-date and high-quality
evidence, which should be reinforced throughout the curriculum if possible.
This is a change we implemented in our curriculum following this assessment.
Second, even though the correlation between search quality and article
selection was not statistically significant, the data suggests that search
skills may influence article selection. Therefore, instructors should continue
to emphasize and reinforce searching skills: this should help both in student
learning and in providing more robust data for future studies. Third, educators
should be mindful of the scope of the topic chosen for EBM assignments, as this
can impact the results. A broader topic with more diverse evidence could
provide a more realistic and challenging learning experience. Based on this
reflection, we plan to change the assignment topic in future cohorts. Fourth,
we encourage educators to systematically assess students’ searching patterns.
We observed certain behaviors that could impact the database results negatively
(e.g., use of quotation marks for all keywords in PubMed, disabling Automatic
Term Mapping, improper use of Boolean operators, lack of appropriate synonyms,
etc.), which are now addressed in advance. Lastly, we note that both this
capstone assignment and the post-test were released and due on the same day
through the Learning Management System, causing some confusion among students
who thought that there was only one activity to complete. As a result, several
students were missing either the assignment or the post-test. Therefore, we
recommend that separate tasks should have different release and due dates to
avoid confusion.
This study has several strengths. First, it addresses a
gap in the literature by focusing on the relationship between search quality
and article selection. Researchers have noted that the focus on critical
appraisal rather than the other EBM steps is a shortcoming of this
literature[14] and this study adds evidence to the assessment of earlier steps.
Second, the use of a whole-task assignment, where students were free to
formulate their own questions and searches, adds to the validity of the study.
Third, a rubric was used to assess search quality and article selection,
helping reduce bias and improve consistency. Finally, the study is based on a
real-world EBM assignment, part of a medical school curriculum, and highlights
that continuous assessment and reflection improves curriculum, as discussed in
the paragraph above.
This study also has several limitations that should be
considered. We assessed only one cohort of students from a single institution,
which may limit the generalizability of our findings. Furthermore, the
assignment did not include direct observation, making it challenging to
understand the context, idea generation, browsing patterns, and search
formulation and reformulation tactics that students might have employed. An
additional limitation is that the rubric was not formally validated and was
designed based on the topic chosen. This narrow topic may have influenced the
range of articles available for student selection. Finally, there could be potential confounding variables that might have
influenced the results, such as prior knowledge or experience with database
searching, which were not measured as a baseline.
Future research should incorporate direct observation of
the search and selection process in order to better capture student
performance. Although challenging, conducting longitudinal studies that follow
students throughout their medical education could help provide insights into
how their search and selection skills develop over time. Such studies could
assess the long-term impact of EBM training on students' abilities and identify
key points in the curriculum where reinforcement or additional instruction is
most needed. Another helpful assessment would be to evaluate students’ prior
knowledge and experience with database searching before starting EBM classes.
With this type of data, researchers could also investigate whether certain
groups of students benefit more from specific types of training. Another
possible future direction is analyzing the performance of students with high
search scores but low selection scores, and vice versa. This could elucidate
factors that could contribute to efficient or inefficient evidence selection in
the EBM process. Finally, with the increasing use of AI in literature searches,
future studies could examine how the use of AI tools can impact students’
search and article selection compared to traditional methods.
Acknowledgments
This research was developed through the training and
support provided by the MLA Research Training Institute (RTI).
Data Availability Statement
The data that support the findings of this study are
openly available in Zenodo at https://doi.org/10.5281/zenodo.15683435.
Author Contributions Statement
Juliana Magro: Conceptualization; data curation; formal
analysis; investigation; methodology; visualization; writing—original draft. Caitlin
Plovnick: Conceptualization; writing—review and editing. Joey Nicholson:
Conceptualization; writing—review and editing.
References
1. The
Liaison Committee on Medical Education. Standards for the Accreditation of
Medical Education Programs Leading to the M.D. Degree [internet]. [cited 18
Jun 2025]. https://lcme.org/publications/
2. The
United States Medical Licensing Examination. USMLE Content Outline [internet].
[cited 18 Jun 2025]. https://www.usmle.org/sites/default/files/2022-01/USMLE_Content_Outline_0.pdf
3. Obeso
V, Brown D, Aiyer M, Barron B, Bull J, Carter T, Emery M, Gillespie C, Hormann
M, Hyderi A. Core entrustable professional activities for entering residency:
toolkits for the 13 core EPAs. Washington, DC: Association of American Medical
Colleges. 2017.
4. Putnam
J, Faltermeier D, Riggs CJ, Pulcher K, Kitts R. Conquering evidence-based
practice using an embedded librarian and online search tool. Journal of
Nursing Education. 2011;50(1):60-60. DOI: https://doi.org/10.3928/01484834-20101220-03
5. Phelps
SF, Hyde L, Wolf JP. Introducing information literacy competency standards for
nursing. Nurse Educator. 2015;40(6):278-280. DOI: https://doi.org/10.1097/nne.0000000000000170
6. Menard
L, Blevins AE, Trujillo DJ, Lazarus KH. Integrating evidence-based medicine
skills into a medical school curriculum: a quantitative outcomes assessment. BMJ
Evidence-Based Medicine. 2021;26(5):249-250. DOI: https://doi.org/10.1136/bmjebm-2020-111391
7. Magro
J, Plovnick C, Laynor G, Nicholson J. From Questions to Answers: Teaching
Evidence-Based Medicine Question Formulation and Literature Searching Skills to
First-Year Medical Students. MedEdPORTAL. 2025;21:11496. DOI: https://doi.org/10.15766/mep_2374-8265.11496
8. Muellenbach
JM, Houk KM, E. Thimons D, Rodriguez B. Integrating information literacy and
evidence-based medicine content within a new school of medicine curriculum:
process and outcome. Medical Reference Services Quarterly.
2018;37(2):198-206. DOI: https://doi.org/10.1080/02763869.2018.1439225
9. Maggio LA, Durieux N, Tannery NH. Librarians in Evidence-Based
Medicine Curricula: A Qualitative Study of Librarian Roles, Training, and
Desires for Future Development. Medical Reference Services Quarterly.
2015;34(4):428-40. DOI: https://doi.org/10.1080/02763869.2015.1082375
10. Eldredge
J, Schiff MA, Langsjoen JO, Jerabek RN. Question formulation skills training
using a novel rubric with first-year medical students. Journal of the
Medical Library Association: JMLA. 2021;109(1):68. DOI: https://doi.org/10.5195/jmla.2021.935
11. Eldredge
JD, Bear DG, Wayne SJ, Perea PP. Student peer assessment in evidence-based
medicine (EBM) searching skills training: an experiment. Journal of the
Medical Library Association: JMLA. 2013;101(4):244. DOI: https://doi.org/10.3163/1536-5050.101.4.003
12. Eldredge
JD, Schiff MA, Langsjoen JO. Effectiveness of a question formulation rubric
with second-year medical students: a randomized controlled trial. Journal of
the Medical Library Association: JMLA. 2023;111(1-2):591. DOI: https://doi.org/10.5195/jmla.2023.1529
13. Gagliardi
JP, Stinnett SS, Schardt C. Innovation in evidence-based medicine education and
assessment: an interactive class for third-and fourth-year medical students. Journal
of the Medical Library Association: JMLA. 2012;100(4):306. DOI: https://doi.org/10.3163/1536-5050.100.4.014
14. Albarqouni
L, Hoffmann T, Glasziou P. Evidence-based practice educational intervention
studies: a systematic review of what is taught and how it is measured. BMC
Med Educ. 2018 Aug 1;18(1):177. DOI: https://doi.org/10.1186/s12909-018-1284-1
15. Maggio
LA, Capdarest-Arest N. Practising evidence-based medicine (EBM): a descriptive
analysis of medical students' whole-task EBM assignments. Evid Based Med.
2017 Apr;22(2):41-44. DOI: https://doi.org/10.1136/ebmed-2016-110593
16. Maggio
LA, Kung JY. How are medical students trained to locate biomedical information
to practice evidence-based medicine? A review of the 2007-2012 literature. J
Med Libr Assoc. 2014 Jul;102(3):184-91. DOI: https://doi.org/10.3163/1536-5050.102.3.008
17. D2L
Corporation. Brightspace: So Much More Than a Learning Management System
[internet]. [cited 7 Jan 2025]. https://www.d2l.com/brightspace/
18. Schober P, Boer C, Schwarte LA. Correlation Coefficients:
Appropriate Use and Interpretation. Anesth Analg. 2018
May;126(5):1763-1768. DOI:10.1213/ane.0000000000002864.
19. Gobbi
G, Atkin T, Zytynski T, Wang S, Askari S, Boruff J, Ware M, Marmorstein N,
Cipriani A, Dendukuri N, Mayo N. Association of Cannabis Use in Adolescence and
Risk of Depression, Anxiety, and Suicidality in Young Adulthood: A Systematic
Review and Meta-analysis. JAMA Psychiatry. 2019 Apr 1;76(4):426-434.
DOI:10.1001/jamapsychiatry.2018.4500.
20. Fresán
A, Dionisio-García DM, González-Castro TB, Ramos-Méndez M, Castillo-Avila RG,
Tovilla-Zárate CA, Juárez-Rojop IE, López-Narváez ML, Genis-Mendoza AD,
Nicolini H. Cannabis smoking increases the risk of suicide ideation and suicide
attempt in young individuals of 11-21 years: A systematic review and
meta-analysis. J Psychiatr Res. 2022;153:90-98.
DOI:10.1016/j.jpsychires.2022.06.053.
21. Flores
MW, Granados S, Cook BL. US trends in the association of suicide
ideation/behaviors with marijuana use among adolescents ages 12-17 and
differences by gender and race/ethnicity. Front Psychiatry.
2022;13:1057784. DOI:10.3389/fpsyt.2022.1057784.
22. Kahn
GD, Wilcox HC. Marijuana Use Is Associated With Suicidal Ideation and Behavior
Among US Adolescents at Rates Similar to Tobacco and Alcohol. Arch Suicide
Res. 2022 Apr-Jun;26(2):520-533. DOI:10.1080/13811118.2020.1804025.
23. Zhang
X, Wu LT. Suicidal ideation and substance use among adolescents and young
adults: a bidirectional relation? Drug Alcohol Depend. 2014 Sep
1;142:63-73. DOI:10.1016/j.drugalcdep.2014.05.025.
24. Lydiard
JB, Patel H, Strugatsky Y, Thompson WK, Pelham WE, 3rd, Brown SA. Prospective
associations between cannabis use and depressive symptoms across adolescence
and early adulthood. Psychiatry Res. 2023 Jul;325:115190.
DOI:10.1016/j.psychres.2023.115190.
25. Athanassiou
M, Dumais A, Zouaoui I, Potvin S. The clouded debate: A systematic review of
comparative longitudinal studies examining the impact of recreational cannabis
legalization on key public health outcomes. Front Psychiatry.
2022;13:1060656. DOI:10.3389/fpsyt.2022.1060656.
26. Carvalho
JV, Souza LS, Moreira EC. Association between cannabis use and suicidal
behavior: A systematic review of cohort studies. Psychiatry Res. 2022
Jun;312:114555. DOI:10.1016/j.psychres.2022.114555.
27. Basith
SA, Nakaska MM, Sejdiu A, Shakya A, Namdev V, Gupta S, Mathialagan K, Makani R.
Substance Use Disorders (SUD) and Suicidal Behaviors in Adolescents: Insights
From Cross-Sectional Inpatient Study. Cureus. 2021 Jun;13(6):e15602.
DOI:10.7759/cureus.15602.
28. Bolanis
D, Orri M, Castellanos-Ryan N, Renaud J, Montreuil T, Boivin M, Vitaro F,
Tremblay RE, Turecki G, Côté SM, Séguin JR, Geoffroy MC. Cannabis use,
depression and suicidal ideation in adolescence: direction of associations in a
population based cohort. J Affect Disord. 2020 Sep 1;274:1076-1083.
DOI:10.1016/j.jad.2020.05.136.
29. Sellers
CM, McRoy RG, O'Brien KHM. Substance use and suicidal ideation among child
welfare involved adolescents: A longitudinal examination. Addict Behav.
2019 Jun;93:39-45. DOI:10.1016/j.addbeh.2019.01.021.
30. Bahorik AL, Sterling SA, Campbell CI, Weisner C, Ramo D, Satre DD. Medical
and non-medical marijuana use in depression: Longitudinal associations with
suicidal ideation, everyday functioning, and psychiatry service utilization. J
Affect Disord. 2018 Dec 1;241:8-14. DOI:10.1016/j.jad.2018.05.065.
31. Gilder
DA, Ehlers CL. Depression symptoms associated with cannabis dependence in an
adolescent American Indian community sample. Am J Addict. 2012
Nov-Dec;21(6):536-43. DOI:10.1111/j.1521-0391.2012.00281.x.
32. Adverse
effects of cannabis. Prescrire Int. 2011 Jan;20(112):18-23.
33. Pedersen
W. Does cannabis use lead to depression and suicidal behaviours? A
population-based longitudinal study. Acta Psychiatr Scand. 2008
Nov;118(5):395-403. DOI:10.1111/j.1600-0447.2008.01259.x.
34. Resnick
MD, Bearman PS, Blum RW, Bauman KE, Harris KM, Jones J, Tabor J, Beuhring T,
Sieving RE, Shew M, Ireland M, Bearinger LH, Udry JR. Protecting adolescents
from harm. Findings from the National Longitudinal Study on Adolescent Health. JAMA.
1997 Sep 10;278(10):823-32. DOI:10.1001/jama.278.10.823.
Supplemental Files
Appendix A
Authors’ Affiliations
Juliana Magro, juliana.magro@nyulangone.org, https://orcid.org/0000-0002-9679-9501,
NYU Health Sciences Library, NYU Grossman School of Medicine, NYU Langone
Health, New York, NY
Caitlin Plovnick, caitlin.plovnick@nyulangone.org,
https://orcid.org/0000-0001-7340-2332,
NYU Health Sciences Library, NYU Grossman School of Medicine, NYU Langone
Health, New York, NY
Joey Nicholson, joey.nicholson@nyulangone.org, https://orcid.org/0000-0001-8658-5879,
NYU Health Sciences Library, NYU Grossman School of Medicine, NYU Langone
Health, New York, NY
Received April 2025; accepted June 2025
