ESI and CTAS

Letters

CJEM 2006;6(6):395-396

To the Editor: We read with interest Grafstein's commentary¹ on Worster and colleagues' article,² "Assessment of inter-observer reliability of two five-level triage and acuity scales: a randomized controlled trial" in CJEM's July 2004 issue.

Worster and colleagues sought to compare the inter-observer reliability of ESI [Emergency Severity Index] and CTAS [Canadian Emergency Department Triage and Acuity Scale] using weighted quadratic kappas. Grafstein disagreed with the use of weighted quadratic kappas and stated that "agreement is agreement" and that "weighted kappa scores tend to overestimate the level of agreement between observers." He suggests that "unweighted kappa value and the raw agreement on exact triage level" as well as "unweighted kappa values between adjacent triage levels" be reported.

According to statistical principles, reliability in this instance is a measure of the extent to which a triage scale gives the same acuity level over different situations (e.g., different observers or days) and a measure of the extent to which one can differentiate among patients on acuity levels. This is calculated as the true variance among the patient acuity divided by the sum of the true variance and the error variance divided by the number of levels in the triage scale. Inter-observer reliability specifically examines the degree of agreement of the acuity levels between different observers using the same triage scale.³ One way of expressing inter-observer reliablity is with weighted or unweighted kappa levels. Using these principles, there are 3 flaws to Grafstein's commentary.

Firstly, "agreement is [only] agreement" when one can convince oneself that a true Level 2/II (ESI/CTAS) being mis-designated as a Level 3/III versus a Level 5/V has the same clinical implications. The purpose of weighting is to evaluate the degree of disagreement of triage levels between different observers. Therefore, a weighted kappa more closely reflects the clinical implications of disagreements among triage levels, whereas unweighted kappas do not.

Secondly, triage reliability as depicted by raw agreement instead of an inter-observer reliability, as expressed by a kappa level, will not account for the play of chance.³ This will cause an overestimate of triage reliability. Kappa levels remove the degree of chance-related agreement and provide a more conservative estimate of reliablity. Therefore, expressing raw agreements adds no useful information.

Thirdly, it is also incorrect that kappa values between adjacent triage levels should be reported, because reliability increases when the error variance is small -- which is achieved by increasing numbers of levels on a triage scale.³ To reduce the scale to 2 levels will only decrease the reliability and will provide no assessment of the true discriminative value of the entire triage scale.

As a closing note, Grafstein might be correct in his assumption that paper scenarios do not produce the same inter-observer reliability as real-time scenarios; however, he provides no evidence to support this. Furthermore, previous inter-observer reliability studies including the first such assessment of CTAS by Beveridge and colleagues⁴ were based on paper scenarios while the more recent study by Manos and associates⁵ also reported inter-observer reliability using weighted quadratic kappas. This controversy may be an opportunity for further research.

Jerome Fan, MD
Suneel Upadhye, MD
Karen Woolfrey, MD
Division of Emergency Medicine
McMaster University
Hamilton, Ont.

References

1. Grafstein E. Close only counts in horseshoes and ... triage? Can J Emerg Med 2004;6(4):288-9.
2. Worster A, Gilboy N, Fernandes CM, Eitel D, Eva K, Geisler R, Tanabe P. Assessment of inter-observer reliability of two five-level triage and acuity scales: a randomized controlled trial. Can J Emerg Med 2004;6(4):240-5.
3. Streiner DL, Norman GR. PDQ Epidemiology. 2nd ed. Philadelphia: Decker Inc; 1998. p. 107, 111-3.
4. Beveridge R, Ducharme J, Janes L, Beaulieu S, Walter S. Reliability of the Canadian emergency department triage and acuity scale: interrater agreement. Ann Emerg Med 1999;34(2):155-9.
5. Manos D, Petrie DA, Beveridge RC, Walter S, Ducharme J. Inter-observer agreement using the Canadian Emergency Department Triage and Acuity Scale. Can J Emerg Med 2002;4(1):16-22.