Title

Psychometric properties of a novel questionnaire assessing seminar teacher performance in undergraduate medical education

Theme

5AA Evaluation of the teacher

Authors

Sonia Sippel (1)
Joy Backhaus (1)
Katharina Belting (2)
Sven Anders (2)
Nicole von Steinbuechel (1)
Tobias Raupach (3)

Institutions

(1) Georg-August-University Goettingen - Institute of Medical Psychology and Medical Sociology

(2) University Medical Center Hamburg-Eppendorf

(3) Georg-August-University Goettingen - Department of Cardiology and Pneumology

Background

There is currently no concise evaluation instrument assessing seminar teacher performance in undergraduate medical education. The aim of the study was to develop a comprehensive but rather short questionnaire with good psychometric characteristics.

Summary of Work

Existing and new items were combined to create an item pool (N=29) covering all aspects of an established framework for evaluating medical educators. These were included in a pilot questionnaire version completed by 482 undergraduate students at two German medical schools.

Summary of Results

Table 1. Criteria analysed for questionnaires [1-6] used.
Theoretical background	Reliability	Number of items
Year	Factor analysis	Number of (sub-) scales
Languages	Validation (correlation with external criteria)	Number of items per subscale
Scope	Probabilistic Analysis	Short version
Form of response	Published in peer reviewed journal	Specific feature(s)
Evaluated online and/or print	Sample size (N)	Accessible or is there a fee for usage

The questionnaire was revised using feedback from Subject Matter Experts and students, which led to the exclusion of 14 further items. The 15 remaining items were piloted in another student cohort (N= 535).

Table 2. Descriptive Analysis.
Content of item	Mean	SD	Skew
1. Learning atmosphere	4.47	.76	-1.41
2. Rapport between teacher and students	4.78	.51	-2.91
3. Realistic examples	4.47	.78	-1.57
4. Questions are taken seriously	4.63	.69	-2.46
5. Learning material	4.24	.89	-1.19
6. Feedback on student contributions	4.38	.86	-1.68
7. Enhancing student interest in subject matter	3.86	1.12	-.84
8. Comprehensibility	4.64	.68	-2.16
9. Using questions to check learning outcome	4.28	.95	-1.34
10. Workload	4.32	.95	-1.47
11. Goal communication	3.60	1.39	-.64
12. Pitching of teaching to the student level	4.40	.84	-1.40
13. Session structure	4.34	.86	-1.28
14. Balance between didactic teaching and student activity	4.36	.88	-1.50
15. Feedback on student learning outcome	4.29	.99	-1.39

Note. Values for skew all fall within a tolerable range of ±2, which is considered acceptable.

Principal Component Analysis (PCA) using Promax rotation was conducted. The final questionnaire contained 15 items and yielded two factors. The two-factor model explained 60% of variance in the data.

Table 3. Results of the Principal Component Analysis (PCA).
Item	Content of item	λ₁	λ₂
1	Learning atmosphere	.91
2	Rapport between teacher and students	.80
3	Realistic examples	.76
4	Questions are taken seriously	.75
5	Learning material	.63
6	Feedback on student contributions	.63
7	Enhancing student interest in subject matter	.62
8	Comprehensibility	.60
9	Using questions to check learning outcome	.51	.31
10	Workload		.85
11	Goal communication		.84
12	Pitching of teaching to the student level		.79
13	Session structure		.58
14	Balance between didactic teaching and student activity		.57
15	Feedback on student learning outcome		.50

Note. Cronbach’s alpha values for the two factors were α =.88 and α =.86, respectively. The original factor structure was verified by Confirmatory Factor Analysis.

Conclusion

Future studies should address construct validity of the new tool by comparing it to existing questionnaires. Preliminary probabilistic analyses indicate that the questionnaire may be used for computerized adaptive testing which could pave the way for more efficient and accurate evaluation procedures. To enable comparisons between different teaching settings we are working on the development and validation of two corresponding questionnaires which focus on evaluating teachers in lectures and bedside-teaching.

Take-home Messages

The tool has excellent psychometric properties and lends itself to innovative approaches to evaluation. This is the first report of a short questionnaire covering significant areas of seminar teacher performance in medical education.

Acknowledgement

The development of this questionnaire was enabled by the enormous cooperation and input of students and teachers. We would like to thank all students and teachers who contributed to this project.

References

1. Beckman, T. J. et al. (2003). Evaluating an instrument for the peer review of inpatient teaching. In: Medical Teacher, Vol. 25, No. 2, (pp. 131–135).

2. Centra, J. A. & Gaubatz, N. B. (2005). Student perceptions of learning and instructional effectiveness in college courses a validity study of SIR II.

3. Gollwitzer, M. & Schlotz, W. (2003). Das Trierer Inventar zur Lehrveranstaltungsevaluation (TRIL): Entwicklung und erste testtheoretische Erprobungen. In G. Krampen & H. Zayer, Psychologiedidaktik und Evaluation IV (pp. 114-128). Bonn: Deutscher Psychologen Verlag.

4. Litzelmann, D.K. et al. (1998). Factorial validation of a widely disseminated educational framework for evaluation clinical teachers. In: Academic Medicine, Vol. 73, No. 6, (pp. 688-695).

5. Marsh, H. W. (1982). SEEQ: a reliable, valid, and useful instrument for collecting students’ evaluations of university teaching. In: British Journal of Educational Psychology, 52, (pp. 77-95).

6. Staufenbiel, T. (2000). Fragebogen zur Evaluation von universitären Lehrveranstaltungen durch Studierende und Lehrende. In: Diagnostica, Vol. 46, No. 4, (pp. 169-181).