Emergency physician estimates of the probability of acute coronary syndrome in a cohort of patients enrolled in a study of coronary computed tomographic angiography

Original Research

CJEM 2012;14(3):147-156

Abstract

Résumé

Coronary artery disease (CAD) is the leading cause of death in the United States and accounts for over 6 million emergency department (ED) visits.¹ Up to 80% of patients admitted to rule out acute coronary syndrome (ACS) turn out to have non-ACS causes for their presenting symptoms.² Overtriaging of patients for a “rule-out” admission results in an annual expense of $8 billion in the United States.¹ At least 2% of patients with ACS are inappropriately discharged to home from the ED.^2–5 Misdiagnoses of acute myocardial infarction (AMI) account for the highest dollar amount of emergency physician (EP) malpractice claims.⁶ Despite advances in medical knowledge and diagnostics, the search for triage perfection remains elusive.^4,7–10 To that end, newer technologies, such as the Vancouver Chest Pain Rule have been tested.¹¹

Because of this ongoing under- and overtriage, many studies have focused on the creation of prediction rules to help EPs more accurately triage patients.^4,11–13 Goldman and colleagues created a model for ED patients presenting with chest pain that predicted the need for an intensive care unit admission.¹² Selker and colleagues developed the Acute Cardiac Ischemia Time-Insensitive Predictive Instrument (ACI-TIPI), which was intended to reduce admissions to the coronary care unit, telemetry unit, and hospital.¹³ The seven-variable Thrombolysis-in-Myocardial-Infarction (TIMI) risk score aids in prognostication of adverse events in patients with ACS, including undifferentiated ED populations, but has not been found to have sufficient sensitivity to be used as a triage tool.^14–16 The Vancouver Chest Pain Rule identified a group of patients with undifferentiated chest pain who could be discharged early from the ED with minimal testing, but this rule remains to be validated in a multicentre study.¹¹ The common components of these clinical rules are objective patient-oriented variables such as demographics, risk factor burden, and cardiac biomarkers. They are typically incorporated with the physician's interpretation of the clinical history and initial electrocardiogram (ECG).^7,10 However, some have suggested that the clinical history has a limited role in the diagnosis of ACS.¹⁷ Furthermore, ECG interpretation is often difficult in the ED setting.^18–22 There are few data on how EPs use available data to estimate the probability of ACS in an individual patient or their accuracy.

The ability of physicians to estimate ACS has been tested in several studies.^23–25 These were mainly cross-sectional scenario-based studies. Using simulated cases, Reilly and colleagues compared physician predictions with the Goldman model, a widely known prediction rule.²⁶ In this study, they found triage decisions based on physician predictions to be both less sensitive and less specific than those based on the prediction rule. Mitchell and colleagues conducted a prospective multicentre study designed to measure the diagnostic accuracy of three methods of assessment of low-risk ED patients, one of which was the physician's written unstructured estimate.²⁷ Other studies have investigated characteristics of physicians that may affect the decision-making process, such as their attitudes toward risk, fear of malpractice, and bias.^28–30

The objective of our study was to determine the accuracy with which EPs predict the presence of ACS in a prospectively identified cohort of patients with chest pain meeting enrolment criteria for a study of coronary computed tomographic angiography (CCTA) and admitted for a “rule out ACS” protocol.

Materials And Methods

Study Design

This was a substudy of the previously published Rule Out Myocardial Infarction/Ischemia Using Computer Assisted Tomography (ROMICAT) study. The ROMICAT study was a study of CCTA in low-risk patients admitted with chext pain, in which all subjects received CCTA; the results were not provided to clinicians and did not influence in-patient management decisions.^31,32 The Institutional Review Board of Partners Health Care approved the study. Patients provided written consent, and the consent of providers answering our research questionnaire was implicit.

Study Setting And Population

This study was performed in the ED of the Massachusetts General Hospital, an academic medical centre with 78,000 patient visits a year. Approximately 60% of adults presenting with chest pain or other symptoms consistent with ACS are admitted to an observation unit or in-patient service after an initial ED evaluation. The initial ECG and troponin level for such patients are obtained in the ED, but subsequent ECGs and troponin levels are obtained after admission. Patients entered the study if they met the ROMICAT inclusion criteria.³⁰ Specifically, they were included if they complained of chest pain, had an inconclusive evaluation in the ED, and were awaiting admission to rule out ACS. This admission was to either an observation unit staffed by ED physicians or an in-patient hospital floor covered by Internal Medicine house staff supervised by attending internists and cardiologists. The admission typically consisted of vital sign monitoring, a second serum troponin T level obtained at least 8 hours after onset of symptoms, and a risk stratification test, which was most often an exercise stress test with or without nuclear imaging. This risk stratification evaluation typically occurred within 12 to 24 hours but could occur as long as several days after the index ED visit.

Patients who had an ST elevation myocardial infarction or abnormal cardiac biomarkers in the ED were excluded from the study. Additional exclusion criteria included a history of CAD, arrhythmia, creatinine elevation, contrast allergy, hemodynamic instability, metformin therapy, hyperthyroidism, and inability to provide informed consent. Of 2,014 patients screened, 368 (18%) were enrolled in the study. Of these, 324 (88%) had EP probability estimates and constituted the study population. All study patients received standard treatment and evaluation for ACS both in the ED and in the hospital. The serum biomarker used was troponin T (Cobas-Roche, Roche Diagnostics Limited, West Sussex, England), with a level above 0.09 ng/mL considered elevated.

The physicians involved in the study were board-certified faculty EPs and emergency medicine and internal medicine residents under their supervision. Faculty EPs see all ED patients at the study site and write complete notes on them, as do residents.

Study Protocol

Trained research assistants were stationed in the ED to screen patients who presented with a chief complaint of chest pain. Study intake occurred weekdays from 7 am to 7 pm from May to July 2005 and March 2006 to May 2007. Physicians were interviewed after their patient encounter was completed and they had reviewed the initial ECG and the results of the first troponin measurement. A standardized intake questionnaire was used to record patients' age, gender, risk factors for and history of CAD, medications, chest pain history, and physician estimates of ACS. Estimates from attending physicians and residents were recorded separately.

Measurement

Physicians

The primary outcome was the accuracy of physicians' estimates of ACS for the study population. This was defined as the sensitivity and specificity of the physician's prediction of ACS compared to the criterion standard of presence of ACS as determined by explicit criteria and adjudication. Physicians' estimates of ACS were provided as a continuous variable ranging from 0 to 100%. These estimates were then categorized based on the Goldman criteria: very low (< 1%), low (1–7%), intermediate (8–20%), and high (≥ 21%).¹² ACS was defined as an AMI or unstable angina.

Clinical Outcomes

Clinical outcomes were determined by the ROMICAT adjudication committee consisting of two physicians: one cardiologist and one EP. The committee discussed all cases until consensus was reached. In cases where consensus was not reached, a third physician (a cardiologist) completed the adjudication. Because the original two physicians performed adjudications together in real time rather than independently, a kappa statistic was not calculated. All adjudication decisions were based on the American Heart Association/American College of Cardiology/European Society of Cardiology guidelines for AMI and the Braunwald criteria for unstable angina.^33–35 Reviewers were blinded to the CCTA results. All patient data during the index hospitalization were available to the reviewers. The definitions of risk factors for CAD and the history of CAD have been published with the ROMICAT data.^32,36

Sensitivity Analysis

Given that the physicians providing estimates were a combination of experienced attending physicians and less experienced residents, test characteristics using only attending physicians' estimates were calculated. We then compared the sensitivity of these estimates for varying cutoffs to those of residents and those of residents plus unidentified physicians.

Data Analysis

Data were collected by trained research assistants during physician interviews and were then entered into an Excel 2007 (Microsoft Corp., Redmond, WA) spreadsheet. Analysis of data was done using SAS version 9.2 (SAS Institute, Cary, NC). Characteristics of subjects with and without ACS were compared using two-sample t-tests for continuous variables and chi-square tests for categorical variables. The test characteristics of the physicians' “high predictions” and “low predictions” were reported along with 95% confidence intervals. The Cochran-Armitage test for trend was used to assess proportions across ACS groupings with the levels of physicians' estimated risk. Chi-square tests were used to compare the test characteristics between attending and other physicians.

Results

Complete data were available and estimates were provided for 334 patients. Of this group, 35 (10%) had ACS. Demographic information and cardiac risk factors for the patient groups are provided in Table 1.

Table 1: Patient demographics and cardiac risk factors for patient groups, with comparisons of patients with and without ACS

ACS  =  acute coronary syndrome; CAD  =  coronary artery disease. *Comparing patients with and without ACS. †First-degree relative age 55 years old or less with a history of myocardial infarction or sudden death.
Characteristic	All patients	ACS	Non-ACS	p*
	(n  =  334)	(n  =  35; 10%)	(n  =  299; 90%)
Age, yr (mean ± SD)	54.2 ± 11.9	62.6 ± 12.4	53.2 ± 11.5	< 0.001
Male gender, n (%)	211 (63)	27 (77)	184 (62)	0.07
Diabetes, n (%)	45 (13)	8 (23)	37 (12)	0.086
Hypertension, n (%)	150 (45)	25 (71)	125 (42)	< 0.001
Hyperlipidemia or statin use, n (%)	144 (43)	23 (66)	121 (40)	0.004
Family history of CAD, n (%)†	86 (26)	10 (29)	76 (25)	0.69
Current smoking, n (%)	179 (54)	21 (60)	158 (53)	0.42

Estimates of the probability of ACS for these patients were obtained from the participating physicians. Among these, there were 136 (41%) estimates given by 25 attending physicians. The remaining 198 (59%) were provided by residents or unidentified physicians. The source of these estimates and the characteristics of the physicians who made them are provided in Table 2.

Table 2: Characteristics and contributions of the participating physicians

EM  =  emergency medicine; IM  =  internal medicine.
Provider group	Estimate counts, n (% of all patients)
	EM	IM
Attending physicians	136 (41)	0
Unidentified physicians	115 (34)
Residents	63 (19)	20 (6)
Total	314 (94)	20 (6)

The 334 physicians' estimates were categorized into four groups: very low (< 1%), low (1–7%), intermediate (8–20%), and high (≥ 21%).¹² The distribution and characteristics of each group are provided in Table 3.

Table 3: Distribution and characteristics of each group for all physician estimates

ACS  =  acute coronary syndrome; IQR  =  interquartile range.
Risk group	Likelihood estimate of ACS (%)	n	Median estimate of ACS (%)	IQR estimate of ACS (%)
Very low	< 1	7	0	0, 0
Low	1–7	68	5	4, 5
Intermediate	8–20	131	15	10, 20
High	21–90	128	40	30, 60

Figure 1 shows the number of patients and ACS outcomes within each risk group. Physicians were able to predict ACS better than at random. Increasingly higher physician estimates were associated with a higher incidence of ACS (p  =  0.004). The percentage of patients with ACS was 0%, 6%, 7%, and 17% for very low, low, intermediate, and high probability groups, respectively. Table 4 shows test characteristics by combinations of very low, low, intermediate, and high probability estimates for all physicians. If the very low, low, and intermediate probability estimates (≤ 20%) were considered a “negative test” for ACS and high probability estimate (> 20%) was considered a “positive test,” the test characteristics that would result are shown in the left-hand columns of Table 4. Six percent (13 of 206) of patients in this negative test group had ACS compared to 17% (22 of 128) in the positive test group (p  =  0.003). The diagnostic test defined by this cutoff would have a sensitivity of 63 and a specificity of 65%. Among the 299 subjects without ACS, only 35% had a positive test. If the very low and low probability estimates (≤ 7%) were combined to define a negative test and the intermediate and high probability estimates (> 7%) of ACS were combined and considered a positive test, the test characteristics that would result are shown in the right-hand columns of Table 4. Five percent (4 of 75) of patients in this negative test group had ACS compared to 12% (31 of 259) in the positive test group (p  =  0.13). With this threshold, the sensitivity of the physician estimates would increase to 89%, but the specificity would fall to 24%.

Table 4: Test characteristics by combinations of very low, low, intermediate, and high probabilities (all physicians)

*Physician estimate ≤ 20% was considered a negative test for acute coronary syndrome (ACS) and > 20% was considered a positive test. †Physician estimate ≤ 7% was considered a negative test for ACS and > 7% was considered a positive test.
Test characteristic	Very low, low, and intermediate versus high*		Very low and low versus intermediate and high†
	Estimate	95% CI	Estimate	95 % CI
Sensitivity	0.63	0.45–0.79	0.89	0.73–0.97
Specificity	0.65	0.59–0.70	0.24	0.19–0.29
Positive predictive value	0.17	0.11–0.25	0.12	0.08–0.17
Negative predictive value	0.94	0.89–0.97	0.95	0.87–0.99
Accuracy	0.36	0.30–0.41	0.69	0.64–0.74
Likelihood ratio positive	1.77	1.32–2.39	1.16	1.02–1.33
Likelihood ratio negative	0.58	0.37–0.89	0.48	0.19–1.24

The distribution and characteristics of the 136 likelihood estimates given by 26 attending physicians are shown in Table 5.

Table 5: Distribution and characteristics of each group for attending physician–only estimates

ACS  =  acute coronary syndrome; IQR  =  interquartile range.
Risk group	Likelihood estimate of ACS (%)	n	Median estimate of ACS (%)	IQR estimate of ACS (%)
Very low	< 1	4	0	0, 0
Low	1–7	32	5	4, 5
Intermediate	8–20	55	15	10, 20
High	21–90	45	35	30, 55

As with the overall group, increasingly higher physician estimates were associated with a higher incidence of ACS (p  =  0.004). The percentage of patients with ACS was 0%, 0%, 4%, and 16% for the very low, low, intermediate, and high probability groups, respectively. These are illustrated in Figure 2.

The test characteristics limited to the 136 attending physician predictions are shown in Table 6. As with the likelihood estimates from all providers, decreasing the cutoff from > 20% to > 7% increased the sensitivity of the estimates at the cost of specificity. We compared the sensitivity of attending physician predictions to that of the rest of the cohort. When compared to those of residents plus unidentified physicians, the sensitivity for attending physicians was higher with the cutoff of > 7% (100% v. 85%) and with the cutoff of > 20% (78% v. 58%). Likely due to the small sample size of ACS patients in this cohort, neither comparison reached statistical significance (p  =  0.55 and 0.43, respectively). In comparing the sensitivity of the estimates for the same cutoffs of attending physicians to that of residents, the p values were 0.49 and 0.64, respectively.

Table 6: Test characteristics by combinations of very low, low, intermediate, and high probabilities for attending physician–only estimates

*Physician estimate ≤ 20% was considered a negative test for acute coronary syndrome (ACS) and > 20% was considered a positive test. †Physician estimate ≤ 7% was considered a negative test for ACS and > 7% was considered a positive test. ‡Not available because negative predictive value  =  1.
Test characteristic	Very low, low, and intermediate versus high*		Very low and low versus intermediate and high†
	Estimate	95% CI	Estimate	95% CI
Sensitivity	0.78	0.40–0.97	1.00	0.66–1.00
Specificity	0.70	0.61–0.78	0.28	0.21–0.37
Positive predictive value	0.16	0.06–0.29	0.09	0.04–0.16
Negative predictive value	0.98	0.92–1.00	1.00	0.90–1.00
Accuracy	0.29	0.22–0.38	0.67	0.58–0.75
Likelihood ratio positive	2.60	1.68–4.03	1.40	1.25–1.56
Likelihood ratio negative	0.32	0.09–1.08	NA‡

Discussion

This prospective observational cohort suggests that EPs are able to predict ACS better than random chance would dictate. There was also a suggestion of a relationship between the physicians' predictions and the incidence of ACS. The sensitivity of estimates from EM faculty was not significantly different from that of residents. Although all physicians were able to predict ACS, their predictions were both insensitive and nonspecific. This lack of sensitivity of physician likelihood estimates is particularly important because it speaks to the issue of underadmitting patients with ACS with subsequent poor outcomes. Recent data describe models to predict absence of major adverse cardiac events among patients diagnosed with non–ST elevation myocardial infarction–ACS.³⁷ But these analyses are predicated on physicians recognizing with a high degree of accuracy that the patient is experiencing an ACS event. Without accurate probability prediction by physicians, risk stratification to predict major adverse cardiac outcomes becomes problematic. Using the Goldman categorization of the probability of ACS led to the creation of four risk groups for physician estimates: very low, low, intermediate, and high. Combining the very low, low, and intermediate probabilities led to physician estimates of ACS of ≤ 20% to constitute a negative test. Not surprisingly, this cutoff point created a test that was reasonably specific (65%) but insensitive (63%). When only the very low and low probabilities were used to define a negative test, this led to a cutoff point (≤ 7%) that was more sensitive (89%) but less specific (24%). If we were to use 2% as a cutoff for a positive test, we would have achieved a sensitivity of 94% (95% CI 81–99). Mitchell and colleagues concluded that a physician estimate of 2% or less in ED patients being evaluated for ACS conferred a sensitivity of 96.1% (95% CI 86.5–99.5), a result similar to ours.²⁷

There are three ways to achieve triage perfection in the diagnosis of ACS. The first is to have a diagnostic test that has almost perfect test characteristics, particularly sensitivity. The second approach would be a perfect decision rule. Recent efforts have been largely focused on these two approaches.^11,38,39 Although some decision rules have been shown to predict ACS more accurately than physicians, none that have been validated have extremely high sensitivity and specificity.^24–26 ECGs at presentation have been used, but up to 18% of patients presenting to the ED with an AMI have normal or only nonspecific ECG changes.^40,41 Furthermore, EPs may not accurately interpret these ECGs.⁴² The accuracy of new-generation troponins has improved dramatically, but they remain imperfect to rule out ACS.^43,44 In summary, no existing technology distinguishes ACS from non-ACS with complete accuracy.⁴⁵ The last possible way to achieve near-perfect triage decisions is to teach physicians to make extremely accurate predictions. However, how physicians make predictions and their accuracy remain a largely unexplored field.

Limited data from studies conducted on the process of physician decision making indicate that this comprises three general steps: gathering data, interpreting data, and comparing the physician's general impression to a “classic” or “textbook” norm.²⁵ The history of chest pain has a significant impact on the physician triage decision, although this aspect has been shown to be unreliable.¹⁷ Interestingly, one study suggests that experience and specialty do not have any impact on the accuracy of physicians' predictions,²⁴ a conclusion consistent with our finding that attending physicians' predictions were no more accurate than those of residents. As one would likely predict, physicians' triage processes are heavily influenced by social bias, fear of malpractice, and individual risk-taking behaviour.^28,30 Our study represents an effort to learn how accurately physicians make predictions on the presence of ACS in ED patients with chest pain. Given that physicians' predictions appear to be both insensitive and nonspecific, future research should involve learning how physicians make predictions and how this can be improved.

The findings of our study suggest that in an ED population with a nondiagnostic evaluation for ACS, even very low physicians' estimates of the probability of ACS are not sufficient to safely discharge patients to home. Given that all patients in our study, even those with a very low and low probability of ACS, were deemed by their providers to require hospital admission, it is possible that the providers considered them at higher risk for ACS than their likelihood estimates would suggest. Three possibilities arise in the interpretation of this finding. The first is that physicians felt the need to admit patients despite their low prediction of the probability of ACS. The second is that the probabilities physicians offered in a research protocol do not match those in their thoughts when they make a home versus admission decision. The third is that the physician giving the estimate may not be the sole decision maker on admission. For example, resident admission decisions may have been affected by ED faculty and ED faculty decisions by primary care providers.

Limitations

Our study had several limitations. It was a single-centre study with a small number of outcomes. The exclusion of many patients who failed to meet CCTA eligibility criteria not only limits the ability to generalize our findings but could also lead to selection bias. Attending physicians gave many, but not all, probability estimates. The number of physician estimates we had did not result in us having sufficient power to test the hypothesis that more experienced physicians provide more accurate predictions. Even though we used an accepted risk stratification score, it was one of several that we could have chosen.

Conclusions

Our data suggest that for a selected ED cohort meeting eligibility criteria for a study of CCTA, EPs predict ACS better than by chance, with an increasing proportion of patients proving to have ACS with increasing probability estimates. Lowering the estimate threshold does not result in an overall sensitivity level that is sufficient to send patients home from the ED and is associated with a poor specificity.