Simulation-based training in critical resuscitation procedures improves residents’ competence

EM Advances

CJEM 2009;11(6):535-539

Abstract

Résumé

This article has been peer reviewed.

INTRODUCTION

The 1999 Institute of Medicine report entitled To Err Is Human1 highlighted the diagnostic and cognitive aspects of medical error, and emphasized the role of simulation in procedural skills training.2-4 Subsequent national patient safety agendas have challenged licensing and governing bodies to improve physician competence and patient safety. In addition to concerns for patient safety, issues regarding liability of faculty, graduates and institutions have led to educational reform. Both the American College of Cardiology and the American Board of Internal Medicine supported the American Medical College’s Medical School Objective Project, which stated, “The successful medical school graduate should be able to effectively utilize various instructional tools including electronic tutorials and patient simulations.”5

Research in simulation-based medical education has been dominated by the fields of obstetrics and anesthesia, which have pioneered the use of simulation to improve performance while reducing patient risk.2-4,6 Simulators allow trainees to acquire knowledge and competence in a controlled environment.7 They incorporate reality-resembling options, allow students to work through a clinical problem with limited direction, permit life-threatening mistakes to evolve without harming a real patient and provide immediate feedback on performances that are difficult to evaluate effectively in other circumstances.2-4,6-12

The importance of procedural skills training in the context of clinical competency has become recognized as an important component of residency in-training programs. In clinical medicine, many of the potentially life-saving interventions that are needed to care for patients require mastery of procedural skills. Although traditionally taught through a process informally referred to as “see one, do one, teach one,”13 educational opportunities to perform such procedures during clinical rotations are limited.14 Inconsistent and inadequate exposure may lead to physicians lacking competence in their abilities to perform key procedures on completion of their residency. We sought to assess the impact of a simulation-based procedural skills training course on residents' competence in the performance of critical resuscitation procedures.

METHODS

Study design

This study was reviewed and approved by the Calgary Health Region Ethics Board. The design was a prospective, nonblinded convenience sample study of residents.

At the beginning of the study, participants were asked to complete a self-assessment questionnaire about their knowledge and clinical skills competence in resuscitation procedures. Participants were subsequently exposed to an 8-hour intensive simulation-based course focused on acquisition of procedural skills. Skill stations included the following: insertion of a central line into the right internal jugular vein, insertion of a central line into the subclavian vein and placement of a temporary pacemaker.

The impact of the simulation-based training course was determined by comparing preintervention self-assessed competence with self-assessed competence immediately after the course and at 3 months after the course. In addition to the self-assessment measures, participants were directly observed performing skills by an expert observer using an Objective Structured Clinical Examination (OSCE) station. The expert observer was one of the study designers (T.L.) with postgraduate training in medical education and evaluation. A standardized assessment tool was used at one station for all study participants. Expert assessment through the use of the OSCE was undertaken to concurrently test the validity of the self-assessed measures of competence.

Participants

Thirty-five residents were approached to participate in the study. A total of 28 residents (76% of those ap proached) completed the 8-hour Resuscitation Procedures Course and all 3 self-assessment questionnaires. Participants were postgraduate trainees from 4 different clinical specialties (FRCPC emergency medicine, CCFP emergency medicine, family medicine and internal medicine).

Intervention

The study intervention was an 8-hour simulation-based course on the 3 aforementioned core procedures for the resuscitation of critically ill patients. The course focused on practical application of skills, with time dedicated to actual procedure performance rather than on case discussions. The goal of the study intervention was to maximize “hands-on” time for all participants. A brief didactic summary preceded each skill station, after which the study participants were directly observed attempting procedural skills.

Feedback was provided in a manner that allowed participants multiple attempts at the procedures. Guided instruction was provided to augment skill retention. At the conclusion of each skill station, formative feedback was provided to reinforce desired behaviours and to rectify incorrect or dangerous techniques.

Assessment

The self-assessment questionnaire was designed to measure residents’ perception of their knowledge of and clinical skills in the procedures under investigation. In this context, “knowledge” referred to the theoretical information required to perform the skills, such as indications, contraindications and expected complications for each procedure, and “clinical skills” referred to the psychomotor task of performing the procedure. The questionnaire comprised 25 questions with responses provided using a 5-point Likert scale. The questionnaires were completed by the study participants at 3 periods: 1) at the beginning of the course, 2) immediately after the course and 3) at 3 months after the course.

At the conclusion of the intervention and before the completion of the self-assessment questionnaire, participants were evaluated for their ability to cannulate the right internal jugular vein, one of the key technical skills taught during the intervention. One expert observer completed a checklist for each participant reflecting the curriculum of the intervention than included a global assessment score from 1 = poor to 5 = excellent.

Data analysis

Standard descriptive statistics were generated. Concurrent validity was determined using Pearson product-moment correlation to compare self-assessed com petence with expert assessment OSCE scores. Repeated-measures multivariate analysis of variance was used to evaluate changes in self-assessed competence measures over the 3 periods.

RESULTS

There were an almost equal number of male (n = 15, 54%) and female (n = 13, 46%) residents. Participating residents were distributed between PGY [postgraduate year]-1 (n = 10), PGY-2 (n = 9) and PGY-3 (n = 9). The specialty training program distribution of the resident sample is indicated in Table 1.

Table 1. Characteristics of residents who participated in the study

Characteristic	No. (%) of participants*
Approached for study	35
Declined participation	7	(20)
Completed 3-month follow-up	28	(80)
Participant demographic
Mean (SD) age, yr	30.2	(3.8)
Male sex	15	(54)
PGY
1	10	(36)
2	9	(32)
3	9	(32)
Postgraduate training program
CCFP	12	(43)
FRCPC-IM	4	(14)
CCFP-EM	4	(14)
FRCPC-EM	8	(29)
CCFP = Canadian College of Family Practice; EM = emergency medicine; FRCPC = Fellow Royal College of Physicians and Surgeons of Canada; IM = internal medicine; PGY = postgraduate year; SD = standard deviation. *Unless otherwise indicated.

The internal reliability of the 25-item self-assessment questionnaire was excellent (0.94 using Cronbach α). A repeated-measures multivariate analysis of variance showed a significant improvement in both residents’ knowledge and clinical skills ratings over the 3 self-assessment periods (p <0.001). Improvements in self-assessed knowledge and clinical skills were significant for all 3 PGY groups. A between-participants analysis found that residents’ year of postgraduate training influenced both self-assessment of knowledge (F2,25 = 4.91, p <0.01) and clinical skills (F2,25 = 10.89, p <0.001). As shown in Figure 1 and Figure 2, a post hoc analysis revealed that significant differences existed between the PGY-1 and PGY-3 residents on self-assessment scores of both knowledge and clinical skills. At the 3-month follow-up, junior residents (PGY-1 and -2) maintained significantly higher scores than at the beginning of the course, but indicated that decreases in both their knowledge and clinical skills had occurred after their postcourse self-assessment. In comparison, senior residents (PGY-3) indicated ongoing increases in both knowledge and clinical skills on self-assessment at the 3-month follow-up.

The OSCE checklist completed by the expert ob server at the completion of the course had an internal reliability coefficient of α= 0.79. The correlation between the 16-item checklist and the 5-point global rating scale was r = 0.78 (p <0.001). The expert assessment total rating scores correlations with residents’ self-assessed knowledge and clinical skills scores were r = 0.40 (p <0.05) and r = 0.51 (p <0.01), respectively.

DISCUSSION

We sought to examine the impact of a course dedicated to procedural skills on residents’ competence in critical resuscitation procedures. Our results suggest the following: resident self-assessment has a fairly good correlation with OSCE evaluation, there is a significant improvement in residents’ self-assessment ratings for both knowledge and skills 3 months after simulation training, and, although maintenance of knowledge and clinical skills appears lower for junior residents (PGY-1 and -2) at 3-month follow-up, senior residents (PGY-3) indicate ongoing growth in both areas.

Our findings support the use of simulation as a training tool for critical resuscitation procedures. The re sults suggest an increased benefit for senior learners after this intervention. The reasons for this are likely multifactorial, but it is possible that senior learners have a more secure foundation of knowledge and clinical experience, leading to greater motivation to learn skills because of a perceived relevance to their current or future practice. Also, senior-level trainees may have had greater opportunity for reinforcement of the material through actual clinical experience. This has potential implications for future curricula planning. Procedural skills are traditionally considered “fundamental” skills taught to junior learners. It is possible that more targeted teaching for senior-level learners may provide better results.

Limitations

Our study has several limitations. We used self-assessed competence as a proxy for objectively assessed procedural skill competence. There is a large body of literature that identifies potential inaccuracies in self-assessed measures.15 We attempted to account for the “miscalibration” of self-assessment by using a relevant OSCE station and correlating self- to expert-assessment of clinical competence to determine concurrent validity of our self-assessment instrument. The OSCE scores suggest reliability at one point; however, the measure was not repeated when the follow-up questionnaires were completed. An additional limitation is that the intervention had a strong emphasis on simulation using task trainers, but also included didactic and small-group sessions. It is therefore not possible to determine the effect of the individual components of the course.

Bias may have entered our assessment in that the expert assessment was only carried out on the right internal jugular station. We did not assess participants’ competence in stations other than this. Our primary intent was to determine interval change in resident self-assessment. The OSCE station was used as a one-time indicator of concurrent validity with participant self-assessments. Formal objective evaluation by experts at each station was beyond the scope of our investigation.

Our inability to control for learning that took place before or after the intervention is a potential source of bias. It is possible that exposure to other learning opportunities or clinical scenarios mandating the use of the skills may have confounded our findings. This limitation also applies to the experience each learner brought with them at the outset of the intervention. The study participants were self-motivated, and may have sought independent study on topics of interest or in subjects in which they felt weak. We cannot account for clinical experience or independent study performed during the study period.

We did not evaluate the effect of the intervention on actual patient care management or objective outcomes. Our findings support the benefits of simulation training for residents and suggest that there may be benefit in matching the timing of clinical skills training to the clinical responsibilities of learners. Further study on the knowledge translation of skills learned on simulators into the clinical arena, however, must be pursued before the broad implementation of simulation training for physicians and other health care professionals.

CONCLUSION

Residents can achieve significant improvement in self-assessed theoretical knowledge and procedural skill competence through participation in a simulation-based resuscitation course. Gains in perceived competence appear to be stable over time, with senior learners gaining further confidence at 3-month follow-up. Our findings support the benefits of simulation training for residents and suggest that there may be a benefit in matching the timing of clinical skills training to the clinical responsibilities of learners.