"I still have not mastered that skill!" Medical student perspectives on a simulation-based evidence-based medicine competency assessment
DOI:
https://doi.org/10.5195/jmla.2025.2023Keywords:
Evidence Based Practice, Competency-Based Assessment, Medical Students, Evidence-Based LibrarianshipAbstract
Objective: We expect medical students to be able to apply evidence-based medicine (EBM) skills in the context of the clinical care of patients. Previous assessments of this domain have primarily utilized decontextualized knowledge tests, which provide limited insights into students’ understanding of EBM skills in the context of patient care. New performance-based EBM competence assessments using Objective Structured Clinical Examinations (OSCEs) are being developed and tested. Understanding how students experience and interact with a simulation-based assessment of EBM competence would enable us to improve the modality.
Methods: We recruited 13 graduating medical students from one medical school who had recently completed an immersive multi station readiness-for-residency OSCE (Night onCall) which included a case-based EBM assessment. We conducted individual interviews to explore their perceptions of participating in this OSCE as a method of EBM assessment. The interviews were transcribed, coded, and analyzed using Dedoose by three health science librarians.
Results: Students discussed their experience and perceptions in six main areas: connection to clinical practice, curricular timing and content coverage, feedback, station instructions, awareness of their own limitations, and an OSCE as a format for assessing EBM.
Conclusion: Medical students appreciated the EBM OSCE because it enhanced their learning about how to integrate EBM into clinical practice. They proposed implementing multiple such opportunities throughout medical school because it would improve their competence and provide highly impactful opportunities to build toward EBM mastery. They endorsed that this would be well-accepted by medical students.
References
Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency: Faculty and Learners’ Guide. Washington, D.C.: AAMC, 2014.
Lomis K, Amiel JM, Ryan MS, Esposito K, Green M, Stagnaro-Green A, Bull J, Mejicano GC. Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot. Acad Med. 2017 Jun;92(6):765-70.
Maggio LA. Educating physicians in evidence based medicine: current practices and curricular strategies. Perspect Med Educ. 2016 Dec;5(6):358-61.
Maggio LA, ten Cate O, Chen HC, Irby DM, O'Brien BC. Challenges to Learning Evidence-Based Medicine and Educational Approaches to Meet These Challenges: A Qualitative Study of Selected EBM Curricula in U.S. and Canadian Medical Schools. Acad Med. 2016 Jan;91(1):101-6.
Buljan I, Jeroncic A, Malicki M, Marusic M, Marusic A. How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures. BMC Med Educ. 2018 Dec 4;18(1):290. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6278026/pdf/12909_2018_Article_1391.pdf.
Lai NM, Teng CL, Nalliah S. Assessing undergraduate competence in evidence based medicine: a preliminary study on the correlation between two objective instruments. Educ Health (Abingdon). 2012 Jul;25(1):33-9. http://www.educationforhealth.net/article.asp?issn=1357-6283;year=2012;volume=25;issue=1;spage=33;epage=39;aulast=Lai.
Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003 Feb 8;326(7384):319-21. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC143529/pdf/319.pdf.
Ilic D. Assessing competency in Evidence Based Practice: strengths and limitations of current tools in practice. BMC Med Educ. 2009 Aug 6;9:53. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2728711/pdf/1472-6920-9-53.pdf.
Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002 Dec 7;325(7376):1338-41. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC137813/pdf/1338.pdf.
Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E. Development and validation of the ACE tool: assessing medical trainees' competency in evidence based medicine. BMC Med Educ. 2014 Jun 9;14:114. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4062508/pdf/1472-6920-14-114.pdf.
Tilson JK, Kaplan SL, Harris JL, Hutchinson A, Ilic D, Niederman R, Potomkova J, Zwolsman SE. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ. 2011 Oct 5;11:78.
Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J. Sicily statement on evidence-based practice. BMC Med Educ. 2005 Jan 5;5(1):1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC544887/pdf/1472-6920-5-1.pdf.
Kalet A, Zabar S, Szyld D, Yavner SD, Song H, Nick MW, Ng G, Pusic MV, Denicola C, Blum C, Eliasz KL, Nicholson J, Riles TS. A simulated "Night-onCall" to assess and address the readiness-for-internship of transitioning medical students. Adv Simul (Lond). 2017 2:13.
Nicholson J, Kalet A, van der Vleuten C, de Bruin A. Understanding medical student evidence-based medicine information seeking in an authentic clinical simulation. J Med Libr Assoc. 2020 Apr;108(2):219-28.
Kumaravel B, Stewart C, Ilic D. Development and evaluation of a spiral model of assessing EBM competency using OSCEs in undergraduate medical education. BMC Med Educ. 2021 Apr 10;21(1):204. https://bmcmededuc.biomedcentral.com/counter/pdf/10.1186/s12909-021-02650-7.pdf.
Bradley P, Humphris G. Assessing the ability of medical students to apply evidence in practice: the potential of the OSCE. Med Educ. 1999 Nov;33(11):815-7.
Burrows SC, Tylman V. Evaluating medical student searches of MEDLINE for evidence-based information: process and application of results. Bull Med Libr Assoc. 1999 Oct;87(4):471-6.
Frohna JG, Gruppen LD, Fliegel JE, Mangrulkar RS. Development of an evaluation of medical student competence in evidence-based medicine using a computer-based OSCE station. Teach Learn Med. 2006 Summer;18(3):267-72. http://www.tandfonline.com/doi/pdf/10.1207/s15328015tlm1803_13?needAccess=true.
Fliegel JE, Frohna JG, Mangrulkar RS. A computer-based OSCE station to measure competence in evidence-based medicine skills in medical students. Acad Med. 2002 Nov;77(11):1157-8. http://ovidsp.tx.ovid.com/ovftpdfs/FPDDNCLBFHHOID00/fs046/ovft/live/gv025/00001888/00001888-200211000-00022.pdf.
Varkey P, Natt N, Lesnick T, Downing S, Yudkowsky R. Validity evidence for an OSCE to assess competency in systems-based practice and practice-based learning and improvement: a preliminary investigation. Acad Med. 2008 Aug;83(8):775-80.
Tudiver F, Rose D, Banks B, Pfortmiller D. Reliability and validity testing of an evidence-based medicine OSCE station. Fam Med. 2009 Feb;41(2):89-91.
Kumaravel B, Stewart C, Ilic D. Development and evaluation of a spiral model of assessing EBM competency using OSCEs in undergraduate medical education. BMC Med Educ. 2021 2021/04/10;21(1):204. https://doi.org/10.1186/s12909-021-02650-7.
Nicholson J, Spak JM, Kovar-Gough I, Lorbeer ER, Adams NE. Entrustable professional activity 7: opportunities to collaborate on evidence-based medicine teaching and assessment of medical students. BMC Med Educ. 2019 Sep 3;19(1):330.
Hennink M, Kaiser BN. Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Soc Sci Med. 2022 Jan;292:114523.
Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health. 2019 2019/08/08;11(4):589-97. https://doi.org/10.1080/2159676X.2019.1628806.
Braun V, Clarke V, Hayfield N, Terry G. Thematic Analysis. In: Pranee Liamputtong, editor. Handbook of Research Methods in Health Social Sciences. Singapore: Springer Singapore; 2018. p. 1-18.
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006 2006/01/01;3(2):77-101. https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa.
Bierer SB, Dannefer EF, Taylor C, Hall P, Hull AL. Methods to assess students’ acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum. Med Teach. 2008 30(7):e171-e7.
Kulasegaram K, Manzone JC, Ku C, Skye A, Wadey V, Woods NN. Cause and effect: testing a mechanism and method for the cognitive integration of basic science. Acad Med. 2015 90(11):S63-S9.
Young JQ, Van Merrienboer J, Durning S, Ten Cate O. Cognitive Load Theory: implications for medical education: AMEE Guide No. 86. Med Teach. 2014 May;36(5):371-84. https://www.ncbi.nlm.nih.gov/pubmed/24593808.
Paas F, Renkl A, Sweller J. Cognitive load theory and instructional design: Recent developments. Educ Psychol. 2003 38(1):1-4.
van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010 Jan;44(1):85-93.
Kalyuga S, Chandler P, Tuovinen J, Sweller J. When problem solving is superior to studying worked examples. J Educ Psychol. 2001 93(3):579.
Salden RJ, Aleven VA, Renkl A, Schwonke R. Worked examples and tutored problem solving: redundant or synergistic forms of support? Top Cogn Sci. 2009 Jan;1(1):203-13.
Renkl A, Atkinson RK. Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educ Psychol. 2003 38(1):15-22.
Atkinson RK, Renkl A, Merrill MM. Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. J Educ Psychol. 2003 95(4):774.
Mezirow J. An overview on transformative learning. Lifelong learning. 2008:40-54.
Pollock E, Chandler P, Sweller J. Assimilating complex information. Learning and instruction. 2002 12(1):61-86.
Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR. Core principles of assessment in competency-based medical education. Med Teach. 2017 Jun;39(6):609-16.
Downloads
Additional Files
Published
Issue
Section
License
Copyright (c) 2025 Joey Nicholson, Caitlin Plovnick, Juliana Magro, Cees van der Vleuten, Anique de Bruin, Adina Kalet
This work is licensed under a Creative Commons Attribution 4.0 International License.
