The effect of training on nurse agreement using an electronic triage system

EM Advances

CJEM 2007;9(4):260-266

Abstract

Résumé

Introduction

In the emergency department (ED), patients are prioritized by triage staff for urgency based on a brief initial clinical assessment. The Canadian Triage and Acuity Scale (CTAS), a 5-level acuity scale (1 = Resuscitation, 2 = Emergent, 3 = Urgent, 4 = Semi-urgent, 5 = Non- urgent), is the nationally recognized ED triage standard in Canada.1-3 Research has shown that CTAS is a valid measure based on admission rates4 and prediction of ED resource use.5,6

One problem with traditional triage methods is their reliance on memory by the individual performing triage, which may be influenced by a lack of time, triage complexity and recall bias. Moreover, ED overcrowding and duties at the triage desk not related to triaging may distract triage nurses and interfere with the triage process, potentially affecting patient safety.7

Memory enhancements (e.g., reminders, card prompts and electronic decision support tools, etc.) may improve triage reliability. Computerized triage tools are now able to display the key elements for each standard complaint to assist in the identification of the highest level acuity modifier(s) applicable to each patient, thus decreasing triage variability.8-10 Information technology in the ED has become so important that Academic Emergency Medicine recently made it a consensus topic.11 With the continuing evolution of software design, standardized decision support systems will likely become more common.

A Web-based triage decision support tool (eTRIAGE), compliant with CTAS, has been developed in Canada and is now employed in a number of EDs. The application eTRIAGE was designed to support the written CTAS guidelines and to support the autonomy of nursing clinical judgment. It requires the user to select from a standardized complaint set,12 which then displays a complaint-specific CTAS-based template to assist with the assignment of the appropriate triage level. If the user's judgment differs from the eTRIAGE generated score, the user is permitted to override the computer score and provide an explanation (e.g., "impression of higher acuity," "pain score not equal to presentation," etc).

Previous research has demonstrated that eTRIAGE is easy to learn even for novice computer users, that it does not increase triage nurse assessment time and that it is widely accepted by triage nurses.13 Moreover, triage nurses using eTRIAGE have higher agreement with a consensus standard than do nurses using memory-based triage.10

The current study describes the implementation of eTRIAGE in a busy, urban Canadian tertiary care ED. The objective of this study was to determine the agreement between duty nurses and study nurses, both using eTRIAGE, the influence of enhanced software training on performance and the frequency of overrides by triage nurses in both training schemes.

Methods

Study population and setting

This study was conducted in a large Canadian, urban, tertiary care, teaching hospital with an annual volume of approximately 67 000 ED visits. CTAS was implemented without decision support in May 1996; eTRIAGE was deployed as the standard method of triage in July 2003.

Training approaches

The study department employed a total of 77 triage nurses. In phase 1, a 3-hour training course was provided to 24 (of the 77) triage nurses, who then shared their knowledge with their untrained triage colleagues during regular triage shifts. All triage staff members (77 nurses) were called the standard trained nurses (STNs). In phase 2, 8 volunteer emergency triage nurses (targeted triage nurses [TTNs]) were recruited to participate in further training and evaluation. Recruitment to this group was based on triage experience (median = 7 years; inter-quartile range [IQR]: 3, 12). Six of the TTNs were members of the initial 24 triage nurses trained in phase 1. In addition to the initial deployment training by the department, each TTN received an additional 3 hours of teaching on a training version of eTRIAGE.

Study evaluation

Two study nurses were employed to perform the second independent eTRIAGE assessment. They were trained to use eTRIAGE during a 3-hour training session identical to the TTNs; however, they were trained at a different time. In phase 1, study nurses were paired with STNs on afternoon and evening duty shifts over a 6-week period between October and November 2003. In phase 2, the study nurses were paired with TTNs during their triage shifts (each triage shift was 4 hours long) during afternoons and evenings over a 9-week period between April and June 2004. The time of day was chosen to maximize the number of patient observations per study shift.

Study protocol

All adult patients (≥ 17 years of age) presenting to the ED during a scheduled study-nurse shift were eligible for inclusion. For each phase, the volunteer duty triage nurse (STN in phase 1 or TTN in phase 2) assessed patients who presented during each study shift using eTRIAGE. The patients were then placed either in the waiting room or directed to the patient-care area, based on the triage score and ED volume. After verbal consent was obtained from the patient, the study nurse completed a second, independent assessment using eTRIAGE. This assessment took place in a separate triage area or at the bedside, and the nurse performing it was blinded to the initial assessment and CTAS score. If the patient was critically ill, consent was waived providing the study nurse did not interfere with patient care. In the study ED, the triage nurse or nurse supervisor routinely assigned critically ill patients directly to a bed.

Measurements

Each patient's vital signs and discriminating triage modifiers were entered in real time into the eTRIAGE database. Data from the STNs and TTNs were recorded in the ED's database and the study nurses' data were recorded in a separate database that did not appear on the patients' charts or influence patient care. The databases also captured the number of times the nurses elected to override the computer's assigned score.

Overrides

Whenever an STN, a TTN, or a study nurse assigned a different CTAS score than the one generated by eTRIAGE, this was recorded. The override rate and the "direction" of each override (to a higher or lower acuity CTAS score) were documented.

Data analysis

The primary outcome measure was a comparison of inter-rater reliability between triage score assignment by STNs and TTNs, compared with study nurses. The triage score used in the agreement calculations for each patient included all user overrides. Unweighted, linear weighted and quadratic weighted kappa statistics were calculated for each phase.14 A priori, a comparison of the quadratic weighted kappas was the primary outcome for this study. Although some triage studies report unweighted kappa scores,8,15 most use weighted kappa.16-21 Unweighted kappa scores reflect only exact agreement, while weighted kappa take into account close agreement ("near misses") and the relative values of close agreement versus marked disagreement. Quadratic weighted kappa assigns greater relative credit to closer "near misses" than linear weighted kappa. Kappa agreement was defined a priori as excellent (κ ≥ 0.8), good (0.6 = κ < 0.8), moderate (0.4 = κ < 0.6), fair (0.2 = κ < 0.4) or poor (κ < 0.2).22 Override frequencies between STN and TTN, and the study nurse during phase 1 and phase 2 were compared using chi-squared tests. Statistical calculations were conducted with SPSS (SPSS Inc., 2006, Chicago, Ill.), and SAS (SAS Institue, 2005, Cary, NC) software packages.

Ethics

The study was approved by the Health Research Ethics Board at the University of Alberta. All patients were initially triaged by the duty triage nurse to avoid delays in care. All volunteer nurses provided written informed consent to participate in the study.

Results

Sample

In phase 1 of the study, 569 patients were triaged by both an STN and a study nurse. Incomplete data were obtained in 56 cases (41 were caused by a technical computer problem and 15 had missing triage forms), leaving 513 (90.2%) complete data pairs for comparison (Fig. 1).

In phase 2 of the study, 577 patients were triaged by a TTN and a study nurse. Complete data were available for 555 (96.2%) patient encounters (Fig. 2); for 22 patients, the triage forms were missing.

Table 1 illustrates the demographics, time of presentation and disposition of the patients in each phase.

Agreement

In phase 1, agreement between STNs and study nurses was moderate (quadratic weighted κ = 0.55, 95% confidence interval [CI] 0.49–0.62). In phase 2, agreement between TTNs and study nurses was good (quadratic weighted κ = 0.65; 95% CI 0.60–0.70). The unweighted and linear weighted kappa values are listed in Table 2. Although the point estimate difference in agreement between the 2 phases improved, the CIs for these estimates overlapped.

Overrides

Table 3 illustrates the frequency and direction (to a higher or lower acuity CTAS score) of overrides by the duty triage nurses (STNs in phase 1 and TTNs in phase 2) and study nurses in each phase. The duty triage nurses did not significantly differ in overrides between phases, nor was there a difference in overrides by the study nurses between phases. Within each phase, the study nurses used the override function more frequently than the duty nurses (p < 0.01 in both phase 1 and phase 2). In each phase, the study nurses more often used the override function to lower the acuity score rather than to raise it (p < 0.01 in both phase 1 and phase 2). There was no statistical difference in the override direction for the duty nurses in either phase (p = 0.75 in phase 1, and p = 0.19 in phase 2).

Table 4 lists the eTRIAGE complaints and their frequency of occurrence during the study by combining the phase 1 and 2 data sets. It also illustrates the frequency of overrides by duty nurses and study nurses.

Discussion

This study describes 2 implementations of a computerized ED CTAS-based triage decision support tool, using differing educational strategies. Although CTAS had been used in the busy, urban, high-acuity study ED for more than 7 years, computerized triage was a new and unique approach designed to reduce variability and standardize triage. Initial training involved educating a proportion of the triage nurses, and then relying on them to share their knowledge with the remainder of the triage staff. For the second phase, a cohort of experienced triage nurses volunteered for additional targeted education. During this phase, in which only the targeted cohort with enhanced eTRIAGE education was studied, there was better agreement between the study nurses and triage nurses. Finally, the study explored the issue of triage overrides and found they decreased slightly with additional training, although the overall rate was low (less than 1 in 10).

The developers of CTAS1 and Peds CTAS23 and the CTAS National Working Group have consistently advocated that nurse judgment must be included in the final assignment of the triage score. It is conceivable that the changes introduced in the 2004 CTAS guidelines,3 which were released following the completion of this study, may lower the need for nurses to override CTAS scores in systems using eTRIAGE.

The study nurses used the override function more often than the duty nurses. Moreover, most overrides by the study nurses were to a lower acuity CTAS score. A debriefing with the study nurses identified several possible reasons for this finding. For example, the duty nurses may have felt more "pressure" performing triage in an overcrowded ED and were less willing to expend time to perform an override; or the limited number of study nurses (2) in this study may have resulted in their feeling a greater expertise and comfort using eTRIAGE and more comfort in performing overrides. Further study focusing on user overrides to determine whether they are related to specific complaints, user bias or inconsistencies in CTAS would be useful in order to optimize eTRIAGE, education and guide future enhancements.

This was a prospective study conducted in real time in a busy ED environment. Previously, most triage agreement studies have been limited to simulated patient scenarios.16,17,24 In simulated scenarios, the same "patient" data, including vital signs, are provided to both assessors. In this study, 2 different nurses independently assessed each patient within a short time period. As a result, the history was not scripted and the vital signs were not always the same, even between minimally separated assessments. Further, the sometimes chaotic activity of a real ED environment cannot be simulated in the case-based scenarios. While studies like ours are more complicated to perform and may produce different kappa scores compared with simulations, the real time implementation and testing we report is more generalizable to a real world triage environment.

In addition to eTRIAGE, several other computerized triage programs have been reported. Early work on a computerized triage system in a military setting demonstrated the ability to triage personnel to either an acute care clinic or an ED.25 This system is not applicable to the vast majority of civilian settings and essentially uses a 2-level triage system (acute care clinic v. the ED). More recently, Grafstein and colleagues found high reliability with PC-linked triage, in which each presenting complaint was linked to one or more specific CTAS levels, thus directing the triage nurse toward the preferred triage levels for each complaint type.8,26 Using this approach, the study nurse performed the second triage while observing the duty nurse's triage assessment; they independently input their determination of the triage score into a computer. Although blinded to the duty nurse's triage score, the study nurse did not perform an independent assessment. This difference in study design may account for the difference in inter-rater agreement. Maningas and colleagues found excellent reliability among independent users of the Soterion Rapid Triage System, a complaint-driven, algorithm-based computerized triage system.26

Currently, clinical applications of all major nationally recognized triage systems, including CTAS, the Australasian Triage Scale,27 the Emergency Severity Index in the United States19,20 and the Manchester Triage Scale in the United Kingdom,28 are based on training, memory and experience. In the current ED environment, triage nurses cannot be expected to refer to the paper version during actual clinical shifts or to accurately recall the entire contents from memory. This may lead to subjectivity and therefore inconsistency in the triage process. Decision support tools, such as electronic triage systems, are designed to assist those performing triage by displaying the modifiers for each complaint that define the criteria for each triage level. These tools are not intended to replace clinical judgment and should not be permitted to promote total dependence. The goal is to develop trustworthy tools that permit and even encourage overrides when indicated by clinical judgment. Moreover, these clinical overrides can be used to adjust the source reference used to develop the tool, in this case the CTAS guidelines. The principles of iterative feedback, clinical efficiency, end-user sensibility and implementer flexibility have ensured success of such computer information systems.29

Limitations

This study has several limitations. As the study took place in 1 centre, our results may only be generalizable to large urban Canadian tertiary-care EDs, serving predominantly inner-city populations. Performance in smaller and non- urban locations should be evaluated.

The observed improvement in agreement may partially be explained by increased experience with the software and the selection of motivated nurses. Phase 1 was conducted 4 months after the study ED started to use eTRIAGE in an effort to standardize the application of CTAS. If triage nurses only worked 1 or 2 triage shifts per week, each nurse may only have had up to 8 shifts to become proficient with the program. Phase 2 was conducted 10 months after eTRIAGE deployment. The increased experience with the system may have accounted for some of the improvement in agreement. Similarly, the increase in experience may explain the reduction in the number of manual overrides that the duty triage nurse exercised in phase 2.

In this study, the duty triage nurse (STN or TTN) performed the initial assessment using eTRIAGE. The study nurse performed an independent assessment using eTRIAGE after the duty nurse's assessment. For patient safety reasons, the duty triage nurse was ethically required to perform the initial assessment. Another method described in the literature is to have 2 nurses perform the triage assessment at the same time, blinded to each other's computer screens. Only 1 of the 2 nurses would be directing the assessment while the other nurse listened.8 The method used in the current study involved an independent triage assessment by 2 users; however, it has the disadvantage of potentially obtaining different patient responses to each assessor. This may have contributed to a lower level of agreement. Other, non-triage studies, involving such topics as decision rules and physical examinations, have also used independent observers in sequential order.30,31

The sequence of triage assessments (duty nurse, then study nurse) in this study may have been a source of bias if the study nurse conducted the second triage assessment at the bedside or patient care area. If a patient was critically ill and immediately placed in a resuscitation bed by the duty nurse, the study nurse might have been biased to assign a higher acuity triage score. Similarly, patients who were assessed by the study nurse in a non-monitored area of a "fast track" area might have biased the study nurse to assign a lower acuity score. These factors may have contributed to a higher level of agreement.

This study evaluated 2 sets of duty triage nurses using eTRIAGE, each set having received 2 different training regimens. The training regimen in phase 1 was inconsistent, because two-thirds of the nurses received no formal training on the application and were forced to learn on the job, whereas all the nurses in phase 2 were trained and comfortable with the application before study data was collected. The amount and type of training and optimum nursing or triage experience required to efficiently use eTRIAGE have not been determined. As emergency departments implement more clinical computer applications, the need for research in this area will take on greater urgency.

Conclusions

This study evaluated the implementation of a Web-based triage decision support tool using complaint-based templates derived from CTAS. Agreement between study nurses and duty triage nurses, both using eTRIAGE, was moderate to good, with a trend toward improvement with additional training. Triage overrides occurred in approximately 10% of cases, and study nurses tended to "down-triage" patients more often than duty nurses. Efforts to improve the triage process and identify optimal training and skill retention warrant further research.

References

1. Beveridge R, Clarke B, Janes L, et al. Canadian Emergency Department Triage and Acuity Scale: implementation guidelines. Can J Emerg Med. 1999;1(3 Supplement):S1-S24.
2. Beveridge R. CAEP issues. The Canadian Triage and Acuity Scale: a new and critical element in health care reform. Canadian Association of Emergency Physicians. J Emerg Med 1998;16:507-11.
3. Murray MJ, Bullard MJ, Grafstein E. Revisions to the Canadian Emergency Department Triage and Acuity Ccale implementation guidelines. Can J Emerg Med. 2004;6:421-7.
4. Spence JM, Beaton DE, Murray MJ, et al. Does the Canadian emergency department triage and acuity scale correlate with admission to the hospital from the emergency department? Can J Emerg Med. 2003;6:180.
5. Stenstrom R, Grafstein GE, Innes G, et al. The predictive validity of the Canadian Triage and Acuity Scale (CTAS). Can J Emerg Med. 2003;5:184.
6. Murray MJ, Levis G. Does triage level (Canadian Triage and Acuity Scale) correlate with resource utilization for emergency department visits? Can J Emerg Med. 2003;6:180.
7. Jimenez JG, Murray MJ, Beveridge R, et al. Implementation of the Canadian Emergency Department Triage and Acuity Scale in the principality of Andorra: Can triage parameters serve as emergency department quality indicators? Can J Emerg Med. 2003;5:315-22.
8. Grafstein E, Innes G, Westman J, et al. Inter-rater reliability of a computerized presenting-complaint-linked triage system in an urban emergency department. Can J Emerg Med. 2003;5:323-9.
9. Dong SL, Bullard MJ, Meurer DP, et al. Reliability of computerized emergency triage. Acad Emerg Med 2006;13:269-75.
10. Dong SL, Bullard MJ, Meurer DP, et al. Emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med 2005;12:502-7.
11. Handler JA, Adams JG, Feied CF, et al. Emergency medicine information technology consensus conference: executive summary. Acad Emerg Med 2004;11:1112-3.
12. Grafstein E, Bullard MJ, Unger B. The Canadian Emergency Department Information System (CEDrosoph Inf Serv) presenting complaint list version 1.0. Can J Emerg Med. 2003;5:27-34.
13. Bullard MJ, Meurer D, Pratt S, et al. Evaluation of triage nurse satisfaction with training and use of an electronic triage tool. Can J Emerg Med. 2003;5:183-4.
14. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions, 3rd Edition. New Jersey (NY): J. Wiley; 2003.
15. Goransson K, Ehrenberg A, Marklund B, et al. Accuracy and concordance of nurses in emergency department triage. Scand J Caring Sci 2005;19:432-8.
16. Beveridge R, Ducharme J, Janes L, et al. Reliability of the Canadian emergency department triage and acuity scale: interrater agreement. Ann Emerg Med 1999;34:155-9.
17. Manos D, Petrie DA, Beveridge R, et al. Inter-observer agreement using the Canadian Emergency Department Triage and Acuity Scale. Can J Emerg Med. 2002;4:16-22.
18. Worster A, Gilboy N, Fernandes CM, et al. Assessment of inter-observer reliability of two five-level triage and acuity scales: a randomized controlled trial. Can J Emerg Med. 2004;6:240-5.
19. Wuerz RC, Milne LW, Eitel DR, et al. Reliability and validity of a new five-level triage instrument. Acad Emerg Med 2000; 7:236-42.
20. Eitel DR, Travers DA, Rosenau AM, et al. The Emergency Severity Index Triage Algorithm version 2 is reliable and valid. Acad Emerg Med 2003;10:1070-80.
21. Tanabe P, Gimbel R, Yarnold PR, et al. Reliability and validity of scores on the Emergency Severity Index version 3. Acad Emerg Med 2004;11:59-65.
22. Kramer MS, Feinstein AR. Clinical biostatistics. LIV. The biostatistics of concordance. Clin Pharmacol Ther 1981;29:111-23.
23. Warren D, Jarvis A, Leblanc L. Canadian Pediatric Triage and Acuity Scale: implementation guidelines for emergency departments. Can J Emerg Med. 2001;3(4 Supplement):S1-S27.
24. Fernandes CM, Wuerz R, Clark S, et al. How reliable is emergency department triage? Ann Emerg Med 1999;34:141-7.
25. Berman DA, Coleridge ST, McMurry TA. Computerized algorithm-directed triage in the emergency department. Ann Emerg Med 1989;18:141-4.
26. Maningas PA, Hime DA, Parker DE, et al. The Soterion Rapid Triage System: evaluation of inter-rater reliability and validity. J Emerg Med; 2006;30:461-9.
27. Australasian College for Emergency Medicine. National Triage Scale. Emerg Med 1994;6:145-6.
28. Mackway-Jones K, Manchester Triage Group. Emergency triage. London (UK): BMJ; 1997.
29. Lapointe L, Rivard S. Getting physicians to accept new information technology: insights from case studies. CMAJ 2006; 174:1573-8.
30. Rodger MA, Maser E, Stiell I, et al. The interobserver reliability of pretest probability assessment in patients with suspected pulmonary embolism. Thromb Res 2005;116:101-7.
31. Stiell IG, Wells GA, Vandemheen K, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet 2001; 357:1391-6.