ePoster Manager

ePoster

A randomised study to explore the effect of tutor training on Objective Structured Clinical Examination (OSCE) performance in a large Peer Assisted Learning (PAL) project

Authors

MILNER Kate
RUGHANI Guy
CLIFTON Laura
RAE Valerie
COLLINS Adam
BAHARIN Amira

Theme

Student as teacher

INSTITUTION

University of Edinburgh

Background

Feedback is a hot topic in medical education. Students seek high quality feedback and tutors want to be able to provide it. Peer assisted learning (see 'more detail' button below), is a useful way to deliver both teaching and learning opportunities to students, and whilst the model is used internationally, there has been little research on the importance of training peers to give feedback and its effect on tutees.

The images above show fourth year tutors and third year tutees completing a cardio station (left) and a shoulder examination station (right) during the OSCE PAL event. (Used with the permission of participants).

more detail

Summary of Work

Aim

To assess whether training Year 4 students on how to deliver good feedback would make a difference to the mock exam performance or perceptions of Year 3 students who received this feedback during a peer-assisted learning event.

Method

Year 4 students at the University of Edinburgh ran a mock OSCE for year 3 students to help them prepare for their professional exam.

119 year 4 students volunteered to be tutors, and 234 year 3's (91% of the year) signed up to attend the event over the course of 3 nights.

Each year 3 student sat 8 stations of 5 minutes and received 1 minute of feedback from a year 4 tutee after each station.

The images above show fourth year tutors and third year tutees completing a resus station (left) and a locomotor station (right) during the OSCE PAL event. (Used with the permission of participants).

Tutor training (see video at top)

Prior to the event, half of the tutors were randomly allocated to attend training on how to give good feedback. This focused on generic feedback skills. Those allocated who could not attend the session were sent a secure link to a video of the same session. This was group A.

The other half, group B,received a generic advice sheet on feedback.

Tutee allocation

During the PAL, tutees were allocated to group A or B alternatively as they arrived. This meant half of the tutees were consistently advised by trained year 4 tutors (group A), and the other half consistently had year 4 tutors who had not been trained (group B).

The tutees were blind to the training status of their tutors.

Outcome measures

After 8 stations, the tutees sat a validated mock exam station on lower back pain. All tutors acted as examiners on this station and marked tutee performance according to a consultant-approved mark scheme.

At the end of the evening, tutees were asked about the feedback they had received from the tutors and their perceptions of its quality to investigate if there were any differences between those who had trained (group A) versus untrained tutors (group B).

Summary of Results

Key findings:

No statistically significant difference in the marked exam station performance between group A and B tutees, with those in group B performing marginally better. (Table 1)
Trend towards more group A tutees finding the feedback useful and having increased confidence in passing their upcoming exams than those in group B. (Table 2)
Tutees were unable to consistently identify trained Vs non-trained tutors. (Table 3)

Table 1: Analysis of the marked locomotor 'red flag' station

	Group A	Group B
Mean mark	26.42	26.73
Variance	17.20	23.10
Observations	83	74
df	155
t stat	-0.43
P(T<=t) two-tail	0.67

Table 2: Tutee feedback responses (figures are percentages)

Key: SD=strongly disagree, D=disagree, N=neutral, A=agree, SA=strongly agree

Statement	Group A					Group B
Statement	SD	D	N	A	SA	SD	D	N	A	SA
I enjoyed the evening	0.0	0.0	1.1	23.7	75.3	2.2	0.0	0.0	31.5	66.3
In general, I found the station topics fair	0.0	0.0	0.0	20.7	79.3	2.2	0.0	0.0	23.9	73.9
I found the evening useful in my exam preparation	0.0	0.0	1.1	7.6	91.3	2.2	0.0	0.0	15.2	82.6
I feel more confident about my upcoming exams	0.0	1.1	3.3	22.8	72.8	2.2	0.0	3.3	31.5	63.0
The feedback I received from tutors helped me improve my performance from station to station	0.0	0.0	4.3	37.6	58.1	2.2	0.0	0.0	40.2	57.6
The feedback from the tutors will help me improve my performance in the exam	0.0	0.0	2.2	32.6	65.2	4.4	0.0	1.1	35.2	59.3
The feedback from tutors has not been of benefit	61.1	26.7	3.3	1.1	7.8	59.3	25.3	1.1	2.2	12.1
The amount of time for feedback at the end of each station was not enough to make any difference to my performance	16.3	44.6	13.0	18.5	7.6	18.5	30.4	26.1	16.3	8.7
The feedback I received in OSCE 1 and 2 helped my performance in the final exam style station (Loco. Red Flag station)	3.7	2.4	18.3	42.7	32.9	3.6	10.7	16.7	41.7	27.4
I think peer group teaching is effective	1.1	1.1	2.2	24.7	70.8	2.2	0.0	0.0	24.7	73.0

Table 3: Number of tutee responses to the question 'Do you think your tutor received additional training in giving good feedback?'

	Tutees assigned to Group A tutors	Tutees assigned to Group B tutors
Trained	51	57
Untrained	22	12
Unsure	19	23
Total returned forms	92	92

For further detail of the results and discussion, click the 'more detail' tab below.

more detail

Conclusion

Student tutees enjoyed PAL and generally felt encouraged by the feedback they received whether their tutors were or were not trained in giving feedback.
Tutees who had trained tutors seemed to leave with more confidence and were more satisfied that they had received meaningful feedback even in the short 1 minute timeframe.
It was not possible to demonstrate a significant difference in performance during a mock exam station following the feedback training intervention.
Tutees were unable to consistently identify whether they had received training from trained or untrained tutors.

Take-home Messages

Tutees generally value feedback given by peers.
Very limited tutor training in feedback can improve the perception of peer assisted learning experiences. It would be interesting to see whether increased tutor training or more time for feedback increased the benefits.
We do not have evidence that tutor training improves performance in actual exams or performance across different learning domains.

The image above shows a fourth year tutor and third year tutee completing a resus station. (Used with the permission of participants).

Acknowledgement

Thanks to Dr Ed Mellanby for his advice and the efforts of the fourth year volunteer tutors. We are grateful to our supporters, Wesleyan Medical Sickness, the MPS and the MDU who helped fund the OSCE PAL event and AMEE conference fees.

Summary of Results

Discussion

The study benefitted from its large scale, high rates of feedback sheet return, and the high quality of the intervention (tutor training) delivered by an experienced Fellow in Medical Education. Considerable efforts were also made to maximise the possibility of Group A tutors receiving the training despite logistical difficulties. For those who could not attend in person, the secure video of the session allowed missing Group A tutors to catch up whilst keeping the content away from those allocated to Group B.

Marked exam station (table 1)

It is perhaps unsurprising that there was no significant difference found between those students who received feedback from trained tutors and those who received feedback from untrained tutors as the station was arguably centred more around the students' knowledge than skills. For example, however useful the tutor feedback in the preceding stations, most of the points available in the marked station were dependent on the tutees having some core knowledge of the ‘red flag’ back pain symptoms.

There are other reasons why the intervention does not seem to have had a significant effect. Some possibilities include:

Too minimal an intervention:

o The one-off training session was too brief to make a difference to the tutors’ ability to give feedback

o The time given for feedback from tutors to tutees was too brief – only 1 minute after each station – to make a difference

The generic feedback sheet was too similar to the feedback skills taught at the tutor training session, negating any difference in quality of feedback given by Group A and B tutors
Testing the tutees in the marked exam station occurred too quickly after the intervention to detect any difference, as the purpose of good feedback is arguably to give students something to work on before being retested at a later date.

Tutee feedback (tables 2 and 3)

Although the comparability of the tutees in group A and B was not formally compared, one could infer from the result that very similar proportions found the stations ‘fair’ in each group (bearing in mind the stations sat by group A and B were the same) that all tutees were starting from a broadly similar baseline level of knowledge/experience. This may help to validate differences found elsewhere.

A higher proportion of tutees in Group A strongly agreed that the evening was useful and that they felt more confident about their upcoming exams than tutees in group B. Similarly, compared to group B, more group A tutees felt that feedback would help them improve their performance in their exams and more of them agreed or strongly agreed that the feedback helped their performance in the marked locomotor exam station. It is interesting then that their mean marked exam station performance was marginally worse than those in group B. Perhaps good feedback improves tutee confidence so that they now know what they need to go and work on, rather than immediately improving examined performance i.e. the benefits are more subtle than simple improvements in score.

Despite the suggestion above that there was not enough time for feedback to make any difference, tutees in both groups seemed to think the time was adequate. Slightly more tutees in group A seemed to think that there was enough time (disagreed or strongly disagreed with the statement), perhaps evidence of trained tutors offering more adequate feedback within the 1 minute than group B tutors.

Interestingly, more tutees in group B thought that their tutors were trained than tutees in group A, with only 13% of group B tutees successfully guessing the training status of their tutors. It is difficult to explain this in light of the differences found in the rest of the feedback. There is likely to be a ‘bias of positivity’ towards all of these results with students just happy to be receiving extra teaching and feedback, almost regardless of actual quality. The fact the teaching is peer based may mean that tutees feel uncomfortable honestly appraising its quality. This could explain the tendency for group B tutees to say that their tutors were trained when they were not, trying to express their gratefulness through the feedback sheets and for students to say that 1 minute of feedback is adequate, or even that they enjoyed the evening!