Theme
Assessment: Written and Feedback
INSTITUTION
David Geffen School of Medicine at UCLA
The New Medicine
Medical Education is at the cusp of a paradigm shift, with an emphasis on competencies, milestones, reflective practice, self-directed learning, professionalism, integrated curricula, clinical competence and adaptive learning. Assessments must change to reflect these shifts.
What is the future of Web-Based Assessments?
Driven by a need to promote learning. “Assessment by itself is insufficient as a driver of competence. It would beneficial to integrate assessment into the learning process in order to optimize the presentation of information. Such an integrated assessment can be used to determine whether the learner has reached the desired standard and, if not, to extend the learning module until this standard is attained.”
“Scoring rubrics should emphasize non-analytical clinical reasoning -pattern recognition ability- rather than completeness of information or algorithmic approaches.” (Cook and Treola, 2009)
Allow extensive possibilities in the design and delivery of assessments. Web-Based Assessments can be customized to be adaptive, visual, immersive, interactive, reflective, given the sophistication of tools available in online environments.
Customized to reflect an optimum continuum from formative to summative assessment. Customized designs of assessment range from exclusively formative to solely summative, and anything in between. Varying ways of delivering feedback alters the nature of assessments.
Learner-centered in nature. Online assessments are flexible and adaptive
Directed by theoretical1 and evidence-based foundations. Principles of deliberate practice, self-directed learning, perceptual learning, and theories of expertise constitute the underpinning of web-based assessment.
Can assessment methods driven by innovations in technology provide an impetus for new ways of training future physicians?
COMPUTER-BASED AND ONLINE SUMMATIVE ASSESSMENTS AT UCLA
1. A Home-Grown Computer-Based, Summative Assessment Tool for Anatomy, 2000
2. Summative Assessments on a Learning Management System (ANGEL), 2001-
3. Summative Assessments on Dedicated Assessment Systems: ExamSoft, 2012-
WEB-BASED FORMATIVE ASSESSMENTS
Weekly Formative Assessments via a Learning Management System, 2002-Multiple-choice, open-ended (open-book) and USMLE type questions. Students receive brief feedback or elaborate explanations on their responses. A runaway success, these became the most highly rated component of the curriculum. We investigated the value of closed and open-book formative assessments via Structural Equation Modeling2, empirically illustrating the value of formative assessments on summative performance.
Online Problem Based Learning Assessment via LMS, 2003- . Allows students controlled progress through a video-based case while eliciting diagnostic reasoning process, serving as a satisfactory replacement for a face-to-face faculty session.
ASSESSMENT FOR LEARNING: A FORMATIVE AND SUMMATIVE CONTINUUM
PALMS: Perceptive and Adaptive Learning Modules, custom built, 2012-Customized modules combining Ericsson’s deliberate practice theory3 and perceptual learning theory for developing expertise in pattern recognition. Topics covered via these modules include diagnosing rashes, analyzing ECGs, interpreting radiographs, discriminating different heart sounds, and interpreting histopathology images. The underlying “adaptive” algorithm, developed by a cognitive psychologist4, is based on speed and accuracy to train clinical pattern recognition, via an assessment of these attributes among learners. 
Assessing Diagnostic Reasoning: Ontology-Based5Assessment Engine, conceptual level
Originated from problems with transfer of clinical skills, noted when students work on an acute care, full body, simulated case. Thus, a) while students perform adequately on a written examination, many are unable to apply the tested knowledge during a clinical performance test on a simulated patient; b) a single simulation session is insufficient to comprehend reliably the depth and breadth of students’ clinical performance skills; and c) debriefing sessions, undertaken to identify cognitive paths taken during patient management, pose a financial and logistical challenge with 170 students.
Video Assessments, 2013-
The power of video for visual assessment of patient cues, behavior, communication, empathy is being increasingly recognized6. We are using a tool Zaption (Zaption.com) to assess these learning outcomes.
EPortfolio Assessments7, future…
EPortfolios allow qualitative assessments which give students opportunities to participate in their own assessment. We are conceptualizing a clinical EPortfolio which will extend our current patient documentation tool used in third year clerkships.
Web-based assessments are flexible, allowing different formats even in standard learning management systems. Items need not be confined to multiple choice and essay questions; self-directed learning skills employed in the curriculum, for example, in problem-based learning can be effectively assessed.
- The use of PALMS demonstrates that students learn via repeated practice and assessment presented in an adaptive, evidence-based algorithm.
- A larger pool of skills in clinical competence and knowledge can be assessed via web-based instruction with new tools available for use.
- In an integrated curriculum, where test items should be representative of interdisciplinary content, web-based tools facilitate the process of collaboration in test construction.
-
Students rate formative assessments highly- automated immediate feedback and category reports have received high ratings by students.
-
We will evaluate ontologies, video assessment, and ePortfoilo-based assessments as these are integrated into the curriculum.
New online technologies and tools have extended ways in which medical students can be assessed across the spectrum of competencies. Learner-centered, adaptive assessment environments are now feasible, which can have a great impact on how medical education is delivered. Web-Based Assessments can become an agent of change in implementing training for the New Medicine.
Theoretically driven web-based assessments have yielded best learning outcomes.
- Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783-97.
- Krasne S, Wimmers PF, Relan A, Drake TA. Differential effects of two types of formative assessment in predicting performance of first-year medical students. Adv Health Sci Educ Theory Pract. 2006 May;11(2):155-71.
- Ericsson, K., Krampe, R. & Tesch-Römer, C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993, 100:363-40.
4. Kellman PJ and Garrigan P. Perceptual learning and human expertise. Phys Life Rev. 2009, 6:53–84
5. Chung, K.W.K, Baker, E.L., Delacruz, G.C., Elmore, J.J., Bewley, W.L., & Seely, B. (2006). An architecture for a problem-solving assessment authoring and delivery system. (Deliverable to the Office of Naval Research). Los Angeles, University of California, CRESST.
6. Webb EA, Davis L, Muir G, Lissauer T, Nanduri V, Newell SJ. Improving postgraduate clinical assessment tools: the introduction of video recordings to assess decision making. Med Teach. 2012;34(5):404-10.
7. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 2007 May;82(5):493-502.
CRESST Team, UCLA
Instructional Design and Technology Unit
Dr. Randy Steadman
Post your comments
Send Email