| Theme: 9JJ Admission to Medicine and Postgraduate Training Programmes | |||
 |
 |
|||||
| The gap between first impression and multiple mini-interview performance ratings: A comparison between different rater groups |
 |
|||||
|
||||||
The Multiple Mini-Interview (MMI) is currently the most popular method to measure non-academic attributes in candidates for medical school.
Although MMI scores rely on behavioral observations by multiple raters the problem of rater bias should still be considered.
Possibilities to reduce rater bias include:
- the use of standardized, anchored rating scales by trained raters
- and a balanced assignment of raters to stations according to gender, profession or other characteristics.
Objectives
Since these measures require a lot of preparatory effort our goal was to display their effects by comparing two different rating types:
- first impression ratings
- and MMI performance ratings
formed by raters of different
- profession
- and gender.
Design
- 2012 MMI for admission to medical school at the Hamburg University Medical Center
- 192 candidates
- 1 day, 5 sets, 4 simultaneous circuits
- 8 different stations with 2-3 raters per station
Raters were either physicians, medical students, psychologists, psychology students or of other professional backgrounds (e.g. sociologists or dentists).
Data analysis
In addition to descriptive statistics we analyzed the effects of
- rating type (first impression, MMI performance)
- rater characteristics (gender, profession)
- and candidate characteristics (gender, age)
on performance ratings in a linear mixed model.
- Overall mean ratings significantly dropped from first impression to MMI performance ratings across all groups
- First impression: M = 3.64 (95% CI: 3.62-3.67, SD = 0.85)
- MMI performance: M = 3.48 (95% CI: 3.44-3.50, SD = 0.98)
- The change in ratings was influenced by rater (profession, gender) but not by candidate characteristics. Most notably the three-way interaction between rater’s gender, rater’s profession and rating type was significant.
- The mean difference varied between rater groups with male psychologists showing the largest gap of 0.3 points between ratings.
- Most rater groups adjust their initial first impression rating after using the standardized, anchored rating scheme.
- Rater groups vary in their
- levels of severity for both ratings
- and in their adjustment from one to the other rating.
These findings implicate differences in rating behavior. Therefore, we advise a balanced assignment of raters of different profession and gender.
Observed differences between rating types and rater groups support costly or time-consuming measures like the development and use of standardized, anchored rating scales by trained raters and a balanced assignment of raters according to rater characteristics.
Post your comments
Send Email