Accuracy of clinical diagnosis versus the World Health Organization case definition in the Amoy Garden SARS cohort

EM Advances

CJEM 2003;5(6):384-391

Abstract

Résumé

Fast-tracked article. Early release, published at www.caep.ca/cjem on Oct. 22, 2003.

See related article: Commentary

Introduction | Methods | Results | Discussion | Conclusion | References

Introduction

Severe acute respiratory syndrome (SARS) was recognized in China in November 2002, and the culprit agent, a new strain of coronavirus (SARS-associated CoV [SARS-CoV]), was identified in April 2003.^1,2 As of Aug. 7, 2003, World Health Organization (WHO) statistics indicate that SARS has infected 8422 people and caused 916 deaths worldwide. In Hong Kong alone, it has infected 1755 people and killed 300.

Amoy Garden, the site of the largest community SARS outbreak in the world, is a densely populated residential complex in Hong Kong, comprised of 5016 households in several blocks of high-rise buildings. The Amoy Garden outbreak began on Mar. 25, 2003, and affected 323 residents, producing 18% of all Hong Kong cases within a short period of time. In this cohort there were 37 deaths, accounting for 12% of Hong Kong SARS mortality. Several factors, including a defective sewage system, inadequate ventilation and poor building design contributed to the high SARS infection rate seen in Amoy Garden.^3,4 Most of the Amoy Garden residents who developed SARS-like symptoms were managed at the United Christian Hospital (UCH) in Hong Kong.

Early accurate diagnosis is critical in preventing spread and avoidable mortality, but SARS may present like a nonspecific viral illness, making it a diagnostic challenge for emergency physicians. Even with our experience to date, there are still uncertainties about the possible range of SARS presentations. During the 2003 SARS outbreak, the WHO case definition⁵ (Box 1) was the recommended guideline for case identification and classification, but this case definition is not 100% accurate, and clinicians should exercise caution when basing critical decisions on imperfect tools. Clinical judgement is an indispensable tool for emergency physicians and one that cannot be ignored in addressing this evolving clinical challenge.

Our primary objective was to compare the diagnostic accuracy of emergency department (ED) physician judgement to that of the WHO case definition. We hypothesized that physician clinical judgement would be more accurate than the WHO case definition.

Methods

Setting and patients

In the early phase of the SARS outbreak, the UCH established an ED-based screening clinic for patients with SARS-like symptoms. This cohort study includes all patients from the Amoy Garden complex who presented to the ED SARS screening clinic between Mar. 10 and May 10, 2003. For study purposes, we defined cases as "Amoy Garden patients with a final diagnosis of confirmed SARS," and non cases as "Amoy Garden patients with a final diagnosis of non-SARS." Patients who presented to the ED but did not live in Amoy Garden were excluded from the study.

Data collection

On arrival, all patients were screened for WHO case definition criteria. In addition, presenting symptoms, physical findings, vital signs, investigation results, ED diagnoses and subsequent disposition were documented on standard charts, which were scanned and stored in the hospital's electronic database. After the outbreak, trained research assistants retrospectively reviewed the clinical data and collated the following information: patient age, gender, presence of chronic illness (defined as any medical problem requiring regular follow-up), primary symptoms, presence or absence of subjective fever, measured temperature at the time of ED arrival, type of contact (none, social, close, clustering, or health care worker), results of laboratory tests and chest x-rays, and ED diagnosis. Data were encoded and input into Statistical Data Base System (Statistical Package for Social Science 11.5) for subsequent analysis.

Emergency department diagnosis

The ED diagnosis was based on the clinical judgement of physicians who were aware of the WHO case definition. For study purposes, we considered the ED diagnosis to be "SARS" if the ED physician diagnosis recorded on the chart was "suspected SARS," "clinical SARS" or "SARS." We considered the ED diagnosis to be "non-SARS" if the diagnosis recorded on the ED chart was unrelated to SARS. Research assistants were blinded to the final outcome diagnosis when they abstracted charts and recorded ED diagnosis.

Patient follow-up

Patients without any features of SARS were discharged with information pamphlets about SARS, general guidelines for household hygiene measures, and numbers for contact hotlines. Patients who had some features of SARS, but who were not ill enough to require initial hospitalization, received detailed advice regarding home quarantine, personal isolation and strict hygiene measures. Daily ED follow-up was arranged with senior doctors monitoring for symptom progression and changes in the blood picture or chest x-ray. All patients in Hong Kong who received a diagnosis of confirmed SARS were recorded in the Hong Kong Authority eSARS system and the Department of Health's Master List. This included patients who were discharged from the ED with a non-SARS diagnosis and ultimately developed SARS.

Outcome diagnosis

Final outcome diagnoses for the study cohort were retrieved from the eSARS system and the Department of Health's Master List, which contained all Hong Kong patients with suspected or confirmed SARS. Final diagnoses were made by Hong Kong Public Health experts according to WHO recommendations for interpreting SARS-related laboratory tests⁶ (Box 2). Patients were defined as confirmed SARS if they had clinical SARS and virology confirmation (antibody to SARS-CoV or SARS-CoV ribonucleic acid [RNA] reverse transcription polymerase chain reaction [RT-PCR] positivity). Patients were defined as undetermined SARS if they had clinical SARS without virology confirmation (i.e., laboratory testing was not performed or incomplete). Patients were defined as non-SARS if their final diagnosis was unrelated to SARS.

Data analysis

Descriptive statistics including means, standard deviations and ranges were used to characterize the study population. Missing data are reported in tables and text. ED diagnoses were compared to final outcome diagnoses using 2 | 2 tables. Standard formulae were used to calculate sensitivity, specificity and accuracy for physician judgement and WHO case definition. Only patients with confirmed SARS and non-SARS were included in the diagnostic accuracy comparisons (those with undetermined SARS were excluded). The statistical significance of observed differences in categorical variables was assessed using chi-squared analysis or, where appropriate, Fisher's exact test. The statistical significance of observed differences in interval data was assessed using Student's t-test or analysis of variance (ANOVA), as appropriate. Intervals of 95% confidence were calculated around critical proportions.

Results

During the study period, 821 patients from Amoy Garden were evaluated in the UCH ED SARS screening clinic. Table 1 shows that, at baseline, patients with and without SARS were similar with respect to age, gender and comorbid illness prevalence. The final diagnosis was confirmed SARS in 205 cases, undetermined SARS in 35 cases, and non-SARS in 581 cases, for a disease prevalence of 26% in the study population. Overall, 281 patients were admitted after their index visit, 430 were discharged with unspecified follow-up and 110 were discharged but asked to return for ED follow-up. Figure 1 shows the distribution of SARS cases in these 3 patient groups.

Fig. 1. Emergency department disposition and diagnostic outcome. See Table 1 for definitions of "Confirmed," "Undetermined" and "Non-SARS."

Table 2 shows the prevalence of the WHO case definition criteria in patients with and without SARS. In the SARS group, 197 patients (96%) reported a history suggestive of a fever, but only 129 (63%) had a fever >38°C at the time of presentation. In addition, 187 (91%) had an identifiable contact history, and 101 (49.3%) had one or more respiratory complaints. In the non-SARS group, 11% had a fever over 38°C, 81% had a contact history and 41.2% had one or more respiratory complaints. In the SARS group, 87 patients (42%) met all 3 WHO criteria (as defined in item 1 of Box 1), fulfilling the case definition for suspected SARS, while in the non-SARS group, 78 patients (14%) met the case definition. Eighty-six percent of SARS patients and 20% of non-SARS patients had an abnormal chest x-ray.

Table 3 and Table 4 show that the ED physicians were 91% sensitive, 96% specific and 94% accurate in detecting confirmed SARS, while the WHO case definition was 42% sensitive, 86% specific and 75% accurate. Positive predictive value (PPV) and negative predictive value (NPV) were 88.2% and 96.7% for physician judgement, compared to 52.7% and 80.8% for the WHO case definition. Positive likelihood ratios (LR+) were 21.1 for physician judgement and 3.1 for the WHO criteria. Negative likelihood ratios (LR-) were 0.10 for physician judgement and 0.67 for the WHO criteria, indicating that clinician judgement was a much more powerful predictor than the WHO criteria.

Discussion

Physicians around the world have been advised to use the WHO case definition to guide SARS screening decisions. The data from this study show that ED physician judgement was more accurate than the WHO case definition, and that the WHO case definition was neither sensitive nor specific. Most important, at the time of ED presentation, only 42% of subsequently confirmed SARS cases met WHO case definition criteria. Although 96% reported a subjective fever when they arrived in the ED, only 63% had a measured temperature over 38°C. A minority had respiratory symptoms at this stage (46% with cough, 9% with sputum and 5% with shortness of breath). At the same time, the specificity of the WHO case definition was only 86%, with a PPV of 53%, meaning that 47% of patients who met the WHO case definition in the ED ultimately proved not to have SARS. These findings concur with those from another recent study showing that the WHO case definition criteria were 26% sensitive, 96% specific and 83% accurate.⁷ Relying on the current WHO case definition to screen for SARS in an ED or primary health care setting is therefore likely to lead to an unacceptably high rate of misdiagnosis.

Physician judgement

Despite the fact that most physicians have been advised to use the WHO case definition to guide SARS screening decisions, emergency physician clinical judgement was substantially better than the WHO criteria, with 91% sensitivity, 96% specificity, and 94% accuracy. Clinicians' positive predictive value was 88.2% and negative predictive value, 96.7%. This high level of diagnostic accuracy may be, in part, relate to the high disease prevalence in our setting, which provided clinicians with much valuable experience. The situation may be different for physicians working in non-endemic areas where disease prevalence is low and clinical judgement less reliable.

Our data do not prove that the WHO case definition is bad. They do suggest, however, that it is far from perfect, that it is not an adequate screening tool, that rigid adherence to it will lead to potentially disastrous over- and under-diagnosis and that, while physicians should consider the WHO criteria, they should not ignore their clinical judgement. Physicians must also be aware that many patients who do not meet WHO criteria when they present to the ED do, in fact, have SARS and that many who fulfill WHO criteria prove ultimately not to have SARS. Given the potential for tragedy if infectious SARS patients are released into the community or admitted to hospital wards without meticulous infection control, physicians should adopt a cautious approach and err on the side of proper isolation and contact management in cases of concern -- even for patients who do not meet case definition criteria

ED follow-up system

Recognizing that there are cases of diagnostic uncertainty, it is important to develop a defined discharge and follow-up strategy for potential SARS patients who present with atypical features and do not warrant hospitalization. Our strategy included detailed advice regarding home quarantine and personal hygiene, isolation and quarantine, public health linkage and a reliable follow-up mechanism. This follow-up system enhanced the efficiency of SARS screening and enabled us to discharge clinically stable patients knowing that "missed" cases would not fall through the cracks. This system proved effective, and of 205 patients with confirmed SARS, 180 were recognized during the initial ED visit and 25 (12%) during scheduled follow-up.

The need to revise the SARS case definition

According to WHO recommendations, patients being screened for SARS are classified as "suspect cases" if 3 criteria -- fever, contact history and respiratory symptoms -- are fulfilled. Patients are considered "probable cases" if, in addition, they have pulmonary infiltrates on x-ray, a positive test for SARS-CoV, or autopsy findings of respiratory distress syndrome without an identifiable cause. As is clear, the WHO case definition in use at the time did not mandate testing for SARS-CoV. But without the need for virology confirmation, clinicians using this "purely clinical" case definition are likely to diagnose SARS in many patients without CoV infection (e.g., those with atypical pneumonia). In our study cohort, where the prevalence of confirmed SARS was 26%, the WHO case definition's PPV for "suspected SARS" was only 53%. In a more typical setting with lower SARS prevalence (e.g., 1%), its PPV would be in the range of 3%, setting the stage for many inappropriate SARS diagnoses and exposing patients to the social consequences of isolation and stigmatization, as well as the medical consequences of incorrect therapy.

The current case definition may be acceptable for infection control and community surveillance purposes but clinicians need a tool that offers greater accuracy and specificity. Our data suggest that the incorporation of rapid and accurate diagnostic tests in the case definition is therefore advisable.

As of April 2003, the US Centers for Disease Control and Prevention (CDC) has revised their surveillance case definitions for SARS to include laboratory criteria for evidence of infection.⁸ Since July 2003, the definition was further updated to exclude cases with negative SARS-CoV virology. These changes in the case definition (i.e., with requirements for virological testing) will increase specificity, but it is not clear they will improve the sensitivity, feasibility or safety of applying the WHO case definition as an ED screening tool.

A good recent example involved the use of a rapid SARS-CoV test to clarify the status of 97 residents and 46 health care workers in a July 2003 nursing home outbreak in British Columbia, Canada. These patients developed nonspecific flu-like and respiratory symptoms without prominent fever, and were suspected of having SARS. In this situation, virology study rapidly identified a virus similar but not identical to SARS-CoV, which may represent a newly identified but less virulent variant of CoV.

Recommendations

Based on our experience in the management of this large community SARS outbreak, we recommend revision of the SARS case definition, development of a better clinical screening tool for EDs and other primary care settings, rapid development and implementation of an accurate serological test, and widespread influenza vaccination to reduce the number of "confounding" cases of febrile respiratory syndromes presenting to EDs. In addition, we suggest liberal use of chest radiography for patients presenting with potential SARS symptoms and the institution of a reliable follow-up system like that described here, to avoid public health disasters related to cases that will, inevitably, be missed at first presentation. During possible SARS outbreaks, EDs should consider establishing a "fever clinic" or "SARS screening clinic," which segregates patients with fever or other SARS symptoms in a designated area to minimize secondary outbreaks. This should be managed according to a biohazard model that protects non-SARS patients and staff. We staffed our clinic with relatively senior doctors, and this might be the reason for the high diagnostic accuracy seen in our series.

Limitations

Although key predictors were collected prospectively, research assistants were appropriately blinded and final diagnoses were made independently of ED diagnoses, this was a retrospective study and it is possible that there were missing data with respect to some of the secondary clinical predictors. In addition, the Amoy Garden outbreak, which was attributed to a defective sewage drainage system that enabled extensive viral spread in a densely populated community,^3,4 was characterized by a very high infection rate within a relatively short period of time. This may have been a unique situation; therefore readers should be cautious in generalizing our conclusions to other settings. The prevalence of SARS in this study cohort was much higher than that in most settings, and physicians should remember that diagnostic tools like the WHO case definition have different performance characteristics -- particularly predictive value -- when prevalence changes. This raises a concern with respect to external validity.

There have been minor changes to the WHO case definition over the past few months. The version we used was the updated version (i.e., the May 1, 2003 version), which may be slightly different from previous versions, although the basic inclusion and exclusion criteria remained unchanged. We believe that it will pose minimal, if any, problems if our data are compared to other studies using the WHO case definition of a slightly different version.

Conclusion

In this study cohort, physician clinical judgement was more accurate than the WHO case definition. Reliance on the WHO case definition as a SARS screening tool may lead to an unacceptable rate of misdiagnosis, and our data suggest a strong need to revise the current case definition or refrain from using it as a screening tool in EDs and primary care settings. 

References

1. Drosten C, Gunther S, Preiser W, van der Werf S, Brodt HR, Becker S, et al. Identification of a novel coronavirus in patients with severe acute respiratory syndrome. N Engl J Med 2003;348:1967-76. Epub 2003 Apr 10.
2. Ksiazek TG, Erdman D, Goldsmith CS, Zaki SR, Peret T, Emery S, et al. A novel coronavirus associated with severe acute respiratory syndrome. N Engl J Med 2003;348:1953-66. Epub 2003 Apr 10.
3. World Health Organization, Regional Office for the Western Pacific. WHO environmental health team reports on Amoy Gardens [press release]. 2003 May 16. Available: www.info.gov.hk/gia/general/200305/16/0516114.htm (accessed 2003 Oct 1).
4. Outbreak of severe acute respiratory syndrome (SARS) at Amoy Gardens, Kowloon Bay, Hong Kong: main findings of the investigation. Hong Kong: Dept of Health; 2003 Apr 17. Available: www.info.gov.hk/info/ap/pdf/amoy_e.pdf (accessed 2003 Oct 1).
5. World Health Organization. Case definitions for surveillance of severe acute respiratory syndrome (SARS). Geneva: The Organization; 2003. Available: www.who.int/csr/sars/casedefinition/en (accessed 2003 Oct 1).
6. World Health Organization. Use of laboratory methods for SARS diagnosis. Geneva: The Organization; 2003. Available: www.who.int.csr/sars/labmethods/en (accessed 2003 Oct 1).
7. Rainer TH, Cameron PA, Smit D, Ong KL, Hung AN, Nin DC, et al. Evaluation of WHO criteria for identifying patients with severe acute respiratory syndrome out of hospital: prospective observational study. BMJ 2003;326:1354-8.
8. US Centers for Disease Control and Prevention. CDC updates interim U.S. case definition of severe acute respiratory syndrome (SARS) [press release]. 30 Apr 2003. Available: www.cdc.gov/od/oc/media/pressrel/r030430.htm (accessed 2003 Oct 1).