Abstract Title
SCRIPT eLearning: a multi-method evaluation

Authors

Hannah L Brooks
Sarah K Thomas
James Hodson
Nicholas Blackwell
John F Marriott
Christine Hirsch
Elizabeth Hughes
Jamie J Coleman

Theme

4AA eLearning courses

INSTITUTION

University of Birmingham - United Kingdom
Health Education West Midlands - United Kingdom
OCB Media Ltd - United Kingdom

Background

The General Medical Council’s EQUIP study found that in UK hospitals the majority of prescribing errors are caused by Foundation trainee doctors[1] (i.e. those in their first two years following medical school). Recommendations from the study were:

  • Education in practical prescribing should be part of Foundation education.
  • Foundation trainees should be given explicit feedback regarding their prescribing practice.

In response to the report, Health Education West Midlands commissioned a web-based eLearning programme with an aim to provide standardised prescribing education in the Foundation training programme[2] and promote safe, effective and rational prescribing by Foundation trainees.

SCRIPT eLearning was developed by a team of clinical pharmacists and clinical pharmacologists working in both education and healthcare. The programme consists of 44 modules relating to prescribing and therapeutics. Each module takes approximately 30-40 minutes to complete.

In the West Midlands, it is required that all Foundation Year 1 (F1) trainees complete 16 specified modules, and all Foundation Year 2 (F2) trainees complete a further 15 modules of their choice.

It is also required that Foundation trainees receive three hours protected study time per week during the Foundation training programme[2]. Despite being a mandatory part of trainees’ Foundation education, SCRIPT modules are not necessarily completed within this protected time.

Summary of Work

To ensure the programme is achieving its intended objectives, we conducting a multi-method evaluation of SCRIPT eLearning based on Kirkpatrick’s evaluation framework[3]. We are investigating trainees’ attitudes towards SCRIPT and the impact of SCRIPT on trainees’ knowledge, clinical prescribing behaviours and organisational outcomes.

Here we present an examination of the learning behaviours of trainees, as well as their perceptions of the SCRIPT eLearning programme and perceived impact of SCRIPT on their prescribing in clinical practice.

Learning behaviours:

Data were captured from the SCRIPT Content Management System (CMS) for each of the 16 mandatory modules completed by each of the 688 F1 trainees in the West Midlands from 01 August 2013 to 31 July 2014, including:

  • Pre/post-test scores
  • Date module completed and time taken to complete modules
  • Module factors (word count; number of images; number of  external web links available)

Data were analysed using Generalised Estimating Equations (GEE) with an exchangeable correlation structure, in order to account for the within-doctor correlations in the times taken to complete modules.

Reaction and behaviours:

We explored trainees’ perceptions of SCRIPT and its perceived impact of SCRIPT on prescribing in clinical practice, focus groups and interviews conducted at three hospital sites across the West Midlands region.

Thematic analysis was completed by two researchers on the transcribed data, who first independently developed and subsequently collectively reviewed a coding scheme.  Codes were then reviewed by an expert group of clinical practitioners  comprising medics and pharmacists. Each transcript was viewed by both researchers and at least two clinicians.

Conclusion

The SCRIPT eLearning programme standardises prescribing education for Foundation trainee doctors. However, it is not always used as intended, which may be due to a lack of available time during working hours in which trainees can complete SCRIPT modules.

We suggest that Foundation trainees should be provided with a structured learning plan throughout their two-year training period to encourage regular use of the programme and to prevent bi-annual progression reviews being the motivating factor to complete the modules. If possible trainees should be provided with protected time during paid employment to complete SCRIPT modules.

These findings will be used to inform future research streams, commissioned by Health Education West Midlands, in which we will:

  • Examine changes in trainees’ prescribing knowledge using test scores as a surrogate measure of knowledge acquisition.
  • Investigate the rate of medication errors and serious untoward incidents relating to medicines in the hospital setting using patient safety incident reports.
Take-home Messages
  • It is important to evaluate eLearning programmes to help ensure they are effective in achieving their intended objectives.
     
  • This multi-method evaluation allows us to provide recommendations for future developments of the programme and optimise its integration into the Foundation training curriculum.
     
  • It is hoped that improvements to the programme and its integration will help to improve prescribing in clinical practice, to optimise the use of medicines and benefit patient care.
References
  1. Dornan T, Ashcroft D, Heathfield H, et al. (2009). An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical education. EQUIP study. [online]. http://www.gmc-uk.org/about/research/research_commissioned_4.asp
  2. UK Foundation Programme Office (2010). The UK Foundation Programme Curriculum. Available online at www.foundationprogramme.nhs.uk
  3. Kirkpatrick DL, Kirkpatrick JD (2006). Evaluating Training programmes: The Four Levels. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers.
Acknowledgement

We would like to thank:

  • Ms Sonal Shah
  • Dr Hannah Batchelor
  • Mr David Smith
  • Health Education West Midlands
  • OCB Media Ltd
  • Our dedicated SCRIPT authors and peer reviewers
Summary of Results

Learning behaviours:

Over the 12-month study period, there were 10,255 interactions made by 688 F1 trainees with the 16 mandatory modules (approx. 200 per week). The average time taken to complete a module was 32.9 minutes.

The data revealed a peak in the frequency of module completion and a trough in the time taken to complete modules directly prior to trainees’ bi-annual progression review meetings in late February and early June (Figure 1). This suggests that trainees appeared to be extrinsically motivated to complete SCRIPT modules.

Compared to weekdays, there was an increase in the frequency of modules completed (a total of 2,107 modules compared to a total of 1,500 modules) and time spent on the learning (38 minutes compared to 35 minutes) on a Sunday. This suggests that trainees were still prepared to dedicate more time to modules over weekends, which was likely to be during their free time, rather than during protected study time.


Figure 1: Graph to show frequency of module completion and time taken completing modules


Reaction and behaviours:

A schematic representing the main themes from the analysis is displayed in Figure 2. Some examples of themes and quotes are presented below.

Participants experienced formal and informal undergraduate and postgraduate prescribing education. SCRIPT was viewed as a supplementary part of formal learning.

“They’re a good supplement”

In general, participants had a positive user experience.

“As an overview I think SCRIPT’s wonderful and I think it did give me a level of teaching through all the modules”

Most of the module content was perceived to be relevant, in particular clinically-focused modules. The knowledge gained from these modules could subsequently  be applied to trainees’ prescribing in clinical practice.

 “It’s much better to have all the more clinically related ones – you’re actually using it on the wards, using it every day, and it’s much more useful to you”

However the importance of considering individual learning needs at different points in time was discussed.

“The job you’re on might make it useful and you’re willing to dedicate the time to it and go through it thoroughly, but it also might mean you know a bit more about it so you don’t have to spend as much time going through it”

It was also agreed that the programme was easy to use however various modules were too long.

“I think that is the only criticism I have of SCRIPT, that if anything, they probably try and include too much into a single session”

The mandatory nature of modules and available time to complete modules affected engagement with the programme.

“I think the problem with SCRIPT modules is that because they are mandatory, it can almost become a tick box exercise”

A number of improvements to the structure and integration of SCRIPT were suggested, primarily relating to designing shorter modules, allowing self-selection of modules, and providing protected time during working hours during which modules can be completed.

Figure 2: Schematic display of main themes in reaction and behaviours study

Background
Summary of Work
Conclusion
Take-home Messages
References
Acknowledgement
Summary of Results
Send ePoster Link