The effectiveness of training with an emergency department simulator on medical student performance in a simulated disaster

EM Advances

CJEM 2010;12(1):27-32

Abstract

Résumé

This article has been peer reviewed.

Introduction

The subspecialty of disaster medicine, although still in its infancy, has become increasingly important, particularly in emergency medicine. Unfortunately, training in disaster medicine remains scant, or even nonexistent, in many medical schools.1 Unlike most other areas of medicine, the ability to practise disaster management skills in a real setting is usually limited. Although didactic teaching is worthwhile, simulation may offer an opportunity for more training and practice in this area.

A recent review of simulation training in emergency medicine by the Society of Academic Emergency Medicine Technology in Medical Education Committee found that simulation appears to be effective in teaching specific skills or protocols such as intubation, adherence to safety protocols, and surgical skills.2 This review also noted there is "limited but growing evidence" that simulation training may translate to improved patient care.2

Although competences such as leadership, organization and command and control are important for disaster management, objective means to measure these are scant; we could identify only 1 published report of an objective measurement tool.3 The disastermed.ca (soft ware package developed by the principal author, J.F.L.) website includes a collection of simulated patients who have been involved in a large disaster, and includes a simulated hospital tracking system. Participants manage the care of patients on the simulator as they would in real life, including making triage, investigation, procedure and disposition decisions in real time. The simulator has been used previously, and has been found to be an effective method of simulating the emergency department response to a major disaster.4 The software allows analysis of numerous markers of patient flow. In addition, the software has been used as a teaching tool for the European Master in Disaster Medicine program at the Università degli Studi del Piemonte Orientale "Amedeo Avogadro" Novara, Italy.

The purpose of this study was to determine whether exposure to the disastermed.ca simulation program would improve the ability of medical students to manage a simulated disaster.

Methods

Study design

This was a prospective observational cohort study.

Study setting and population

Students in their final 2 years of medical school who were enrolled in an optional course in disaster medicine at the Università degli Studi del Piemonte Orientale in Novara, Italy, were invited to participate in the study.

Study protocol

The Faculty of Medicine at the Università degli Studi del Piemonte Orientale approved the study protocol. As the study was limited to an educational tool, the faculty waived the requirement for ethics approval. Following an explanation of the study, students provided written consent. We matched students by year of study to ensure that the student level of education was equal in the intervention and control groups. We randomly selected 50% of the students from each education category to be the intervention group, and the remaining 50% of each group formed the control group.

Before the simulations, all participants were provided a didactic curriculum that included 8 hours of lectures on general aspects of disaster medicine, triage, command and control and various specific disaster related injuries such as burns, crush injuries and blast injuries.

We performed training and evaluation using the disastermed.ca emergency department simulator. The groups performed simulation exercises, with each of the 2 groups working as a team to manage the simulated disaster. Command and control, job assignment and organizational structure were left to the discretion of the participants. During the simulations, disaster patients arrived at the simulated hospital. We based the distribution of patient acuity and time on the ergometric work of de Boer and Debacker,5 which described expected patient flow based on an analysis of a large number of disasters. Students were asked to manage the patients, including triage, physician assignment, laboratory testing, radiographic studies and disposition, as they would during a true event. All patient management manoeuvres took place in real time, with delay times for procedures and investigations, which were determined electronically by the disastermed.ca database to represent realistic real time delays. All management, investigations and procedures took place directly on the participants' computers.

When required, students were able to request assistance from management staff who used the written disastermed.ca protocols to guide students. These protocols were based on previously published guidelines for use in disaster exercises.6 Briefly, these guidelines divide the management staff into higher, lower and flank control, and specifically address questions that may be asked by the participants regarding the roles of other resources in the emergency management team who were not physically present during the simulation. The guidelines help to standardize the management team's responses, and facilitate the scenario being carried to completion. We obtained aggregate markers of emergency department management, including mean time for patients to achieve specific markers of flow (i.e., triage, bed assignment, physician assignment and disposition). In addition, we captured the number of patients to reach each of the above care milestones. We also evaluated accuracy of triage, but again only in aggregate with no data collection regarding individual participant performance.

During the training phase, both groups were given a 30 minute tutorial in the use of the software. The 2 groups of students then participated in a training session using the disastermed.ca emergency department simulation. Both groups used the same simulated hospital configuration and both groups were given copies of the disaster plan for the Geyserville_2008 hospital from the disastermed.ca software package. The disaster plan is based on the generic Hospital Emergency Incident Command System template, which includes a model of command and control, detailed job descriptions for key roles in the hospital disaster response team, and a collection of standardized forms.7 For this training phase the simulation scenario was different between the 2 groups. The control group received patient flow and patient pro files typical for a hospital of this size, meant to represent normal "day to day" emergency department activity of the hospital. This included a variety of patient presentations. In contrast, in the intervention group, the simulation scenario was a disaster that clearly overwhelmed the hospital's resources. Again, students were asked to man age the department exactly as they would in real life. Use of the disaster plan, command and control and triage were left to the discretion of the participants. Both groups were given the same amount of time (120 min) during the training scenarios.

Subsequently, each of the 2 groups participated in an evaluative simulation using the disastermed.ca emergency department simulation. Here both groups received the same disaster scenario and patients. In this case, the scenario and patient volume was sufficient to overwhelm the hospital's resources. The patient data set and disaster scenario for the evaluative exercise was different from that given to either of the groups during the training sessions.

Measurements

The primary outcome measures were difference in patient flow parameters between the 2 groups, difference in accuracy of triage between the 2 groups, and difference in score of performance indicators for command and control between the 2 groups.

Since the evaluation simulation lasted less than 2 hours, we included only triage code "red" (i.e., immediate) and "yellow" (i.e., delayed) patients in the final analysis, as it is reasonable to conclude that triage code "green" (i.e., ambulatory) patients do not require, nor would they receive, any management in the first 2 hours. Patient flow measurements included the number of patients who were triaged, the number of patients who were assigned to emergency department beds, the number of patients who were seen by a physician and the number of patients who were assigned a disposition. In addition, we compared the difference in time to triage, time to bed assignment, time to physician assessment and time to disposition. We assessed triage accuracy for the 2 groups by comparison with the standard codes from the disastermed.ca database, which had been assigned using the Canadian Emergency Department Triage and Acuity Score with translation to the 4 START (Simple Triage and Rapid Treatment) colours of black, red, yellow and green using the standard START triage assignment algorithm.8

We also compared a previously studied command and control performance indicator scoring system between groups.3 Trained observers scored the participants' performances using the observational tool of Rüter and coworkers.3 The tool involves a set of standardized performance indicators that evaluate the skills of command, coordination and control during incidents and disasters. The indicators assess decision making and management skills, with some performance indicators including a set time for performance of procedures. participants received 2 points for accomplishing the performance indicator within the designated time, 1 point for accomplishing the indicator after the designated time and 0 points if the indicator was not accomplished.

We based secondary outcomes on participant evaluation forms detailing the students' opinions of the training exercise.

Data analysis

We performed statistical analysis using the "R" statistics package (The R Foundation) for SUSE Linux (Novell). We compared differences in time intervals between groups using confidence intervals (CIs), which we calculated at the 99.5% level to allow for an experiment wise error rate of 0.1% (Bonferroni correction with n = 10 contrasts). We assessed differences in proportions of patients to reach care milestones, differences between groups in triage accuracy and command and control scoring using the χ2 test. We considered p < 0.01 significant for all statistical tests, again to allow for a 0.1% experiment wise error rate using the Bonferroni correction with n = 10 contrasts.

Results

Twenty two students agreed to participate in the study, (2 groups of 11 students). All 22 participants were present for both the training and evaluation simulations.

During the training simulations, the intervention group activated the disaster plan and the control group did not. The intervention group used the written disaster plan to organize a command and control structure. However, during the training phase we obtained no markers of performance.

During the evaluation exercise, each group received 97 patients in 90 minutes. This included 16 "red," 16 "yellow" and 65 "green" patients categorized by the START criteria.

There were significant differences in patient flow between the 2 groups during the evaluation exercise. The intervention group triaged patients more quickly than the control group (mean difference 43 s, 99.5% CI 12 to 75 s). Nonsignificant improvements in rapidity of assessment (mean difference 6.3 min, 99.5% CI -2.4 to 14.9 min), rapidity of room assignment (mean difference 2.9 min, 99.5% CI -10.7 to 16.5 min), and time to disposition (mean difference 8.9 min, 99.5%CI -11.6 to 29.3 min) were found in the intervention group.

Table 1 shows a comparison of the number of patients to reach each care milestone. There was no significant difference in triage accuracy between the 2 groups (p > 0.70).

Table 2 presents the results for performance indicators. The score for performance indicators was significantly higher in the intervention group (p < 0.001).

Beyond our numeric findings, staff managing the disaster exercise described subjective differences in the groups' approaches to patient management during the evaluation exercise. For example, the intervention group immediately elected an incident commander, immediately consulted the hospital's disaster plan, quickly delegated duties and responsibilities to the remaining staff and briefed each additional physician resource on arrival. In contrast, the control group did not elect an incident commander until 44 minutes into the simulation. Furthermore, the control group did not develop an obvious organizational structure, and did not usually brief additional staff resources on arrival.

Following the evaluation exercise, 21 of 22 students returned evaluation forms, although not all questions were answered by each student. Of the 21 who responded, all stated that they preferred the simulation based curriculum to a lecture based curriculum. Of those who responded to the question, 11/20 (55%) felt that the time spent on the simulations was appropriate, and 9/20 (45%) stated that time spent on the simulations was too short. All students who responded to the question (20/20) felt the simulation was a valuable learning experience. When asked to rate the exercise overall on a 10 point modified Likert scale, the median score was 8 (range 6-9). There was no significant difference in the evaluation scores between the control and intervention groups.

Discussion

We found that time to triage and command and control markers improved following exposure to a disaster simulation, which suggests that exposure to a disaster simulation using the disastermed.ca software package improves the ability of medical students to manage a simulated disaster. As both groups spent equal time using the simulator during the training phase, differences between groups because of skills using the soft ware were minimized.

Unlike other aspects of emergency medicine in which exposure to real situations is the primary method of teaching, disaster medicine does not allow for such methods. Simulation may present a practical alternative. Although it may be difficult to quantify such education, our study suggests improvement in command and control performance indicators following exposure to a disaster simulation. Simulation appears to be a valuable tool for teaching these competencies.

In our study, we found a correlation between scores on the command and control competencies and patient flow; that is, the intervention group scored higher on the command and control markers with higher patient flow markers. Aside from suggesting that the training simulation had a beneficial effect on both of these indicators, our findings are consistent with, although do not prove, the intuitive notion that improvement in command and control competencies correlates with an increase in patient flow.

Student satisfaction with the exercise was very high, with all students stating that the exercise was a valuable experience and preferable to a lecture based curriculum. Furthermore, because the disaster medicine curriculum for medical students is optional at the study university, providing an experience that is highly rated by students may lead to greater participation in the program.

In the future, further evaluation of the same scenario and simulations with other participants may be valuable. This may allow development of objective bench marks of patient flow, by mean time to reach care mile stones, or by the number of patients to reach each care milestone. If validated, such an initiative may provide a more objective method to measure the success of training programs in disaster medicine.

Limitations

Several design limitations should be considered when interpreting our results. Unfortunately, staff managing the disaster exercise were not blinded to the 2 groups. Ideally, 2 separate management groups would have been preferable for the training and evaluative phase, so that management staff could be blinded to which students had had previous disaster training on the simulator. Practically, however, since blinding of the participants would be impossible, we felt it would not be possible to guarantee that blinding the management groups would be successful; any discussion among the students with regard to their previous experience overheard by any of the management staff would effectively lead to unblinding.

Because of a limited number of management volunteers, a single reviewer assessed command and control indicators. It would be preferable to have multiple reviewers and to assess interrater agreement.

The number of participants was modest, as there were only 22 volunteer students. This led to small sample sizes in each group and relatively wide CIs. In addition, both student groups had requested additional personnel for the simulation beyond the number of students available. Had additional personnel been available to both groups, the outcome of the simulations may have changed.

Because of the busy student curriculum, only 90 min utes of simulation time was available for the evaluation simulations. This limited the number of patients for the simulations, and could have undermined our ability to find statistical differences between some variables.

Although we found increases in patient flow in the intervention group, the validity of the disastermed.ca evaluative instrument has not been assessed. Although it may be reasonable to suppose that increases in patient flow during the simulation would indicate an overall improvement in management, the answer to whether this would translate to "real world" differences in patient management is unknown. Validation of the simulator as an evaluative tool is likely to be very difficult. Comparison to a true disaster is effectively impossible, and we are not aware of any other publications describing validated instruments. Repeatedly using the same tool under similar circumstances is likely to be the only method of validation.

We found minimal published literature regarding assessment tools for command and control competency. Furthermore, no studies have proven that command and control competencies translate to improved patient outcomes. Further research is clearly necessary to develop a standardized tool for these competencies.

Although student satisfaction was very high, all students in the study were volunteers. Other studies that were performed using the same simulation software showed comparable levels of participant satisfaction (J.F.L., unpublished data, 2007-2009). However, in all cases, study participation was voluntary. Whether similar satisfaction would occur with mandatory simulation training is not known.

Conclusion

Participation in an electronic disaster simulation using the disastermed.ca software package appears to increase the speed at which medical students triage simulated patients and their score on a structured command and control performance indicator instrument. Participants indicated that the simulation based curriculum in disaster medicine is preferable to a lecture based curriculum. Overall student satisfaction with the simulation based curriculum was high.