Comparison of Psychometrics when using Teaching Assistants vs Standardized Patients in an OSCE Exam

Authors

Dr. I. Hammel
Dr. E. Fernandez
N. Irias

Theme

Simulation and Simulated Patients

Category

Simulated Patients

INSTITUTION

Ross University School of Medicine

Background

 

Using standardized patients (SP) is an excellent approach for performance evaluation in medical education. Effective SP programs require significant financial investment, which may prove difficult for many medical schools to provide. Ross University’s Advanced Introduction to Clinical Medicine Department uses teaching assistants (TA) as SPs in an effort to optimize resources.

 

Our hypothesis is that the use of  TAs who are medical school graduates as SPs influences the psychometrics of the examination, since their medical knowledge plays a role in the answers they give to random and unscripted student questions, thus influencing students’ decision making process.  

Conclusion

  

Use of medically trained faculty as SPs may impact the scoring of standardized patient encounters.

 

We need to consider other factors that may have impacted the results, such as TAs’ prior knowledge of each student’s abilities and differences in leniency among graders.

 

Although statistically significant, the small variance leads to the conclusion that future studies need to be conducted to clarify the benefit/risk ratio of using medical professionals as standardized patients.

Summary of Work

 

Data was collected utilizing two groups of students and graders: the control group consisted of 135 students assigned to SPs and the study group consisted of 137 students assigned to TAs. Each instructor varied in group size, prompting standard deviation analysis for both groups prior to testing hypothesis.

Take-home Messages

  

Even though it may be more cost-effective to use the program’s own medical staff and avoid incurring the higher costs associated with a standardized patient program, the psychometric impact and grade accuracy should be considered.

Summary of Results

 

 

Statistically significant differences were found through a one tail T- test. The test indicates the TA group scored students higher and had a smaller standard deviation than the SP group. 

Acknowledgement

 

We would like to akcnowledge Dean Flaherty's leadership and support of our research initiatives and Dr. Vivian Shayne's contribution to the statistical analysis.

References

 

1. Wang WD, Yang PC, Chen CY, Lue BH, Yang PM: Using senior residents as standardized patients for evaluating basic skills of medical students. J Med Assoc. 2004 Jul; 103 (7):519-25

 

2. Poenaru D, Morales D, Richards A, O'Connor HM: Running an objective structured clinical examination on a shoestring budget. Am J Surg. 1997 Jun;173(6):538-41.

 

3. Sibbald D, Regehr G: Impact on the psychometric properties of a pharmacy OSCE: using 1st-year students as standardized patients. Teach Learn Med 2003; 15(3):180-5.

 

Background

One study on this topic looked at the utilization of senior residents as standardized patients for evaluating basic clinical skills of medical students and found it to be cost-effective and feasible1. Another study evaluated the cost of OSCE examinations and found that OSCEs can be set up with reasonable cost and limited resources if the use of standardized patients, faculty and staff is done judiciously2. The psychometric implications of using medically trained individuals were not considered. A University of Toronto study on the psychometric impact of using first year pharmacy students as SPs for an OSCE exam concluded it was psychometrically feasible3. However, first year students’ knowledge level may not have been sufficient to impact the results, when compared to that of medical school graduates as standardized patients.

 

 

 

 

Conclusion
Summary of Work

 


Take-home Messages
Summary of Results
  TA SP
Std. Deviation 1.89502 2.41301
Std. Error Mean 0.1619 0.20768

 

Levene's Test for Equality of Variances t-test for Equality of Means
    95% Confidence Interval of the Difference
F Sig. t df Sig. (2-tailed) Mean Difference Std. Error Difference Lower Upper
6.579 .011 2.867 270 .004 .75369 .26287 .23615 1.27123
    2.862 253.951 .005 .75369 .26333 .23510 1.27228

 

Acknowledgement
References
Send ePoster Link