External validation of the San Francisco Syncope Rule in the Australian context

EM Advances

CJEM 2007;9(3):157-161

Abstract

Résumé

Introduction

Syncope accounts for 0.35%-5% of all emergency department (ED) presentations.1-4 It is, however, a symptom rather than a disease, and it has a large differential diagnosis with possible underlying causes ranging from common physiologic states to life threatening conditions.1-4 Some syncope patients have been reported to have 1-year mortality rates as high as 30%.4

To assist decision making regarding the need for hospital admission for syncope patients, a number of clinical decision rules and risk stratification tools have been developed.4-7 The San Francisco Syncope Rule (SFSR)4 aims to predict which patients presenting to the ED with syncope are at high risk of short-term (7 d) serious adverse outcomes. These outcomes were defined as death, stroke, myocardial infarction, arrhythmia, pulmonary embolism, subarachnoid hemorrhage, significant hemorrhage, any related event causing an unplanned return ED visit resulting in hospitalization and any acute intervention for an inpatient that would have prompted re-presentation to the ED if the patient had not already been admitted. The SFSR defined a patient as "high risk" if he or she had any of the following 5 features: abnormal electrocardiogram (ECG), hematocrit < 30%, shortness of breath, history of congestive heart failure or systolic blood pressure of < 90 mmHg. In the initial chart review, the SFSR was found to have a sensitivity of 96% (95% confidence interval [CI] 92%-100%) and a specificity of 62% (95% CI 58%-66%).4 Quinn and colleagues reported that strict application of the SFSR in their derivation set would have reduced the admission rate by 10%, from 55% to 45%. On internal validation, the rule was found to be 98% sensitive (95% CI 89%-100%) and 58% specific overall (95% CI 52%-60%).7 Moreover, 52% of patients were classified as "high risk" and application of the rule was reported to have reduced overall admissions by 7%.

The objective of this study was to validate the SFSR for the prediction of serious outcomes in an external, non-US setting.

Methods

This prospective, observational cohort study was conducted at Western Hospital, Footscray, Victoria, Australia, an adult teaching hospital, with an annual ED census of about 32000 patients.

Patients were eligible for enrollment if they presented to the ED between August 1, 2005 and February 23, 2006 with syncope or near syncope, which was defined as near or full loss of consciousness with return to pre-existing neurologic function. The treating ED doctor confirmed the episode as syncope or near syncope before data collection. Patients were excluded if they declined to participate; were unable to communicate in English and an interpreter was not available; or had persistent altered mental or neurologic status, confusion or loss of consciousness due to seizure, head trauma, alcohol or illicit drug use. The SFSR was not being used in the ED at the time of the study.

All data was collected by a researcher who was not a part of the clinical team caring for the patient. Patients were recruited for study participation in 2 ways:

1.Eligible patients identified during their ED presentation

We obtained informed consent from patients to collect data. We collected data during patients' ED stay following initial medical assessment to confirm eligibility but before a disposition decision had been made. We collected data on demographics, the date and time of the ED presentation, the presence or absence of components of the SFSR, and ED and hospital discharge diagnoses. Follow-up information was obtained by scripted telephone interview at 7 days for patients discharged home, or by inpatient interview or explicit inpatient admission record review for those still hospitalized.

2.Eligible patients identified after ED presentation

These patients were identified by searching the ED clinical database for the terms "collapse," "faint" or "syncope." We used an explicit medical record review to extract study information as described above. Reviews were undertaken within 1 week of the index visit so that a valid 7-day follow-up could occur. Consent was sought by telephone, as was follow-up information regarding adverse events. Missing data for historical aspects of the medical record review (i.e., history of congestive heart failure or shortness of breath) were treated as absent criteria.

If follow-up could not be completed within 9 days of the index visit, patients were deemed to be lost to follow-up. This choice was made to minimize recall errors.

The researcher who was not part of the clinical team caring for the patient undertook all follow-up. We did not collect data on patients who refused to participate in the study.

An abnormal ECG was defined as one showing a nonsinus rhythm or new changes, as defined by Quinn and others in the SFSR derivation study.4 All ECGs were assessed by 2 researchers with experience in ECG interpretation. Serious outcome was defined as the occurrence within 7 days of the index presentation of death, myocardial infarction, arrhythmia, pulmonary embolism, stroke, subarachnoid hemorrhage, significant hemorrhage, any condition causing return ED visit and hospitalization, or any acute intervention for an inpatient that would have prompted re-presentation to the ED if the patient was not already admitted. These definitions are in line with the SFSR derivation study.4

The accuracy of the rule, as measured by its sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for the prediction of short-term serious adverse outcome in syncope patients, was the primary outcome of interest. The potential impact of application of the SFSR on admission rates was a secondary outcome.

Data were analyzed using SPSS (Release 11.0.1. 15 SPSS Inc., 2001, Chicago, Ill.) and for descriptive statistics (numbers, percentages, 95% CIs) and sensitivity, specificity, PPV and NPV for the application of the SFSR on this cohort.

Two independent researchers performed interrater agreement analysis for the classification of risk according to the SFSR and adverse outcome on a randomly selected 13% of the sample (15 cases). The kappa statistic was used to quantify the interrater agreement using Analyse-IT (Leeds, UK, http://www.analyse-it.com/). The Melbourne Health Research Directorate granted ethics approval for this study.

As this study was a student-researcher project under the Advanced Medical Science Program of the University of Melbourne, data collection was limited to the time available under that program. A formal sample size calculation was not included in the study protocol.

Results

We identified 155 patients who presented to the Western Hospital ED during the study period with near or complete loss of consciousness. Of these patients, 40 were excluded for reasons outlined in Table 1, and 2 patients could not be classified for the SFSR because no hematocrit was obtained. Therefore, 113 patients were included in the study. Forty of these patients (35%) were admitted from the ED.

For the sample, there were 26 different diagnoses at ED discharge. The most common ED diagnoses were a vasovagal episode (16%), dehydration (10%) and hypotension (10%). Cause of collapse was not diagnosed at ED discharge for 36 patients (32%).

Follow-up information was obtained from 89 of the 113 (79%) study participants. Summary data for the follow-up cohort is shown in Table 2. We compared the demographic and SFSR-related data of the 89 patients who were followed up with that of the 24 patients who were not followed up to ensure that the follow-up group was representative of the total sample. We found no statistically significant differences for any of the variables.

The follow-up group (89 patients) comprised the final study sample. Of these, 37 (42%) were male and the admission rate was 39%. The median age was 74 years (range 20-93 yr). Data on 77 patients (87%) were collected at the time of their ED presentation, and only 12 (13%) had data collected by medical record review. Seventy patients had 7-day follow-up by telephone and 19 were followed up in hospital because they remained inpatients.

In our sample, we identified 12 adverse events in 10 patients (11%). One patient suffered both an acute myocardial infarction and subsequently died, and another suffered an arrhythmia that led to the insertion of a pacemaker, which is a defined acute intervention. The types and frequencies of the different adverse events suffered by this group are summarized in Table 3.

When applied to our sample, the SFSR predicted that 43 patients would be at high risk of experiencing a serious event within 7 days. Nine of these patients (21%) experienced a serious event as defined by the SFSR criteria. The other patient who experienced poor outcome was categorized by the SFSR as low risk. The SFSR, as applied in our study, demonstrated a sensitivity of 90% (95% CI 60%-98%), a specificity of 57% (95% CI 46%-67%), a PPV of 21% (95% CI 11%-35%) and an NPV of 98% (95% CI 89%-99%) (Table 4).

The case the SFSR failed to identify was of an 80-year-old man who suffered from sick sinus syndrome and required pacemaker insertion.

Of the 89 patients who were followed up, 35 (39%) were admitted. Of those admitted, 23 (66%) were SFSR high-risk patients. One patient discharged from the ED suffered a defined adverse outcome. This 84-year-old woman would have been classified as high risk by the SFSR on the basis of a low hematocrit. She returned to the ED 24 hours after her initial discharge with cognitive impairment and was admitted to the hospital.

The doctors' decisions to admit had a sensitivity of 90% (95% CI 60%-98%) and a specificity of 60% (95% CI 49%-69%) for predicting patients at risk of poor outcome within 7 days. If the SFSR had been strictly applied to the study group, it would have increased the admission rate by 9% (8/89; 95% CI 4%-17%).

Interrater agreement was very good for applying the SFSR to classify patients in high or low risk (93.3% agreement and kappa statistic = 0.81) and for determining patients who experienced serious adverse events (93.3% agreement and kappa statistic = 0.86).

Discussion

In this study, the SFSR was found to have a sensitivity of 90% and a specificity of 57% for the prediction of patients at risk of defined serious adverse outcome. Had it been strictly applied, the SFSR would have missed 1 patient who experienced a serious event. Strict application of the rule would also have increased the proportion of patients with syncope who were admitted to hospital by 9%. In reality, the doctors' clinical decision-making performed at least as well as the SFSR — the doctors also missed only 1 patient with a defined event and admitted fewer patients than the SFSR would have recommended.

There have been studies assessing the usefulness of the SFSR for outcomes at 7 days,4,7,8,9 30 days,7,10 6 months and 1 year.11 This was the first known study of the SFSR to be conducted outside of the United States. The 90% sensitivity of the SFSR in our study is lower than the derivation and the other validation studies that looked at 7-day outcome and reported the rule to have sensitivities of 96%,4 98%7 and 91%11; however, our study reported higher sensitivity than the 76.5% (95% CI 62.2%-86.8%) reported by Schladenhaufen and colleagues in their study of an elder cohort.9 Possible explanations for these differences include the sample size, differences in study population (in particular, our cohort was older than most of the previous studies), and differences in case mix and medical practice.

In our cohort, the SFSR predicted that the proportion of high-risk patients would be similar to that seen in other studies, which suggests that the overall risk profile of patients in the US and Australia is similar, although the distribution of underlying syncope causes may be very different. Despite the similarity in the proportions of high-risk patients that were predicted by the rule, application would have reduced admission rates by 10% in the derivation4 and validation7 sets but would have increased the admission rate by a similar amount in this study. The explanation for this likely lies in the much higher admission rates at the US centres (55%4 and 59.4%7), compared with the admission rate of 39% at Western Hospital. The explanation for the difference in admission rates likely reflects the differences in practice and perhaps a higher tolerance of risk. It is unlikely that our study was affected by the Hawthorne effect; emergency physicians were not specifically informed that our study was occurring, were not using the SFSR at the time and had relatively low awareness of the SFSR.

If the SFSR is consistently found to have high sensitivity, it could be a useful rule in countries like the United States where admission rates are high. It follows that it may not be as useful in settings with lower admission rates unless it can be shown to result in better clinical outcome.

In light of such differences in admission rates, it is reasonable to question whether the admission rate at Western Hospital is too low. The results of this study suggest that this is not the case as doctors only missed 1 patient with defined poor outcome (sensitivity 90%).

Diagnosing the specific cause of syncope is uncommon. Some studies have found that more than 50% of these patients are discharged from the ED without a specific diagnosis.12 Our findings support this conclusion.

Before being broadly accepted, the SFSR requires further evaluation, particularly in locations outside the United States and in age-stratified cohorts. Our study provides evidence that application in external settings may increase admission rates without benefit in outcomes. Currently, the derivation and validation studies based in San Francisco have performed best. This may have been owing more to the local patient population and medical processes than to the rule alone.

This study has some limitations that must be considered when interpreting the results. The sample size was small, resulting in wider CIs. Some data were collected from medical records despite their well-known limitations. Researchers were not blinded to the study hypotheses, which may have introduced bias. The sample was not consecutive and patients who did not speak English were excluded for logistic reasons; this may have introduced bias. It was a single site study, which limits its generalizability. A moderate proportion of patients were lost to follow-up.

Conclusion

The SFSR demonstrated 90% sensitivity in this validation study. Strict application of the SFSR would have increased hospital admissions and would not have identified all adverse outcomes. In our setting, clinician judgement performed as well as the SFSR, with a baseline admission rate of 36%.

References

1. Suzuki M, Hori S, Nakamura I, et al. Long-term survival of Japanese patients transported to an emergency department because of syncope. Ann Emerg Med 2004;44:215-21.
2. Ammirati F, Colivicchi F, Santini M. Diagnosing syncope in clinical practice. Implementation of a simplified diagnostic algorithm in a multicentre prospective trial—the OESIL 2 study (Osservatorio Epidemiologico della Sincope nel Lazio). Eur Heart J 2000;21:935-40.
3. Schillinger M, Mullner M, Meron G, et al. Causes and outcome of syncope. Wien Klin Wochenschr [The Middle European Journal of Medicine] 1999;111:512-6.
4. Quinn JV, Stiell IG, McDermott DA, et al. Derivation of the San Francisco Syncope Rule to predict patients with short-term serious outcomes. Ann Emerg Med 2004;43:224-32.
5. Colivicchi F, Ammirati F, Melina D, et al. Development and prospective validation of a risk stratification system for patients with syncope in the emergency department: the OESIL risk score. Eur Heart J 2003;24:811-9.
6. Sarasin FP, Hanusa BH, Perneger T, et al. A risk score to predict arrhythmias in patients with unexplained syncope. Acad Emerg Med 2003;10:1312-7.
7. Quinn JV, Stiell IG, McDermott DA, et al. Prospective validation of the San Francisco Syncope Rule (SFSR) to predict patients with serious outcomes. Ann Emerg Med 2006;47:448-54.
8. Shen WK, Decker WW, Smars PA, et al. Syncope Evaluation in the Emergency Department Study (SEEDS). Circulation 2004;110:3636-45.
9. Schladenhaufen R, Benenson R, Feilinger S, et al. Application of San Francisco Syncope Rule to elderly emergency department patients. Eur J Emerg Med 2006;13:A7.
10. Quinn JV, Stiell IG, McDermott DA, et al. The San Francisco Syncope Rule vs physician judgement and decision making. Am J Emerg Med 2005;23:782-6.
11. Stracner DL, Kass LE. Validation of the San Francisco Syncope Rule. [Abstract]. Acad Emerg Med 2005;12(Suppl 1):87-8.
12. Schillinger M, Domanovits H, Mullner M, et al. Admission for syncope: evaluation, cost and prognosis. Wien Klin Wochenschr [The Middle European Journal of Medicine] 2000;112:835-41.