Clinical utility of novel cardiac markers: Let the buyer beware

Commentary

CJEM 2006;8(1):32-36

In this issue, Lippi and colleagues, from the University of Verona, provide an optimistic overview of new biochemical markers, concluding that a new era is dawning in the diagnostic approach to acute coronary syndromes (ACS).¹ This is a dramatic conclusion. Conceptually, markers of inflammation, ischemia and myocardial dysfunction hold promise, and emergency physicians should know about them; they are being studied and marketed heavily. But it is also important for physicians to recognize the difference between an exciting concept and clinical usefulness.

During the evolution of an ACS, inflammation leads to plaque rupture, platelet aggregation, coronary thrombosis, cardiac ischemia and, finally, myocardial necrosis. Necrosis markers currently in use, notably CK MB, the troponins and myoglobin, do not detect ACS prior to the onset of irreversible injury; therefore researchers have set their sights on new markers that detect inflammation (the postulated cause of plaque rupture), ischemia and myocardial dysfunction. But are these markers ready for clinical practice and will they help us?

To enhance our performance in a meaningful way, a diagnostic test should measure the phenomenon it purports to, it should accurately distinguish patients with and without disease, it should add to clinical judgement, and it should prompt a change in management that improves patient outcome. Few diagnostic tests, and none of those discussed by these authors, fulfill these expectations. In addressing why they do not, it is helpful to consider several important diagnostic concepts: those of association, predictive value, sensitivity, specificity, and test utility.

Association

Countless recent articles report impressive sounding and statistically significant associations between new diagnostic tests and clinical outcomes. Lippi and colleagues describe the strong association between B-type natriuretic peptide (BNP) levels and outcomes of patients with ACS. They also note that BNP is an independent predictor of future cardiovascular events. But what do the terms strong association and independent predictor really mean? Do they mean the test is accurate? Do they mean it is a useful diagnostic test for ACS? The answer to both questions is No.

To say that a test is statistically associated with an outcome means only that there is some non-random relationship between the test result and what ultimately happens to the patient. It does not mean the test is sensitive, specific or accurate; nor that the test adds anything to clinical judgement; nor that it can be used to make diagnostic or therapeutic decisions. When researchers say that a test is an independent predictor of outcome, it means only that, if patients are matched on all other parameters, a group of patients with abnormal test results will have more outcome events than a group with normal results. It does not mean that patients who have a negative test can be discharged or that those with a positive test require additional investigation. By itself, association is of no value in making diagnostic or therapeutic decisions.

To put this in perspective, there are hundreds of independent outcome predictors in ACS, including age, comorbidity, pain characteristics, diaphoresis, pulmonary crackles, previous heart disease, cholesterol level and ASA use.^2-4 Five of the 7 predictor variables incorporated in the thrombolysis in myocardial infarction (TIMI) risk score are historical variables (age, risk factors, prior coronary disease, number of episodes of rest pain, aspirin use), one is an ECG finding (ST-segment deviation) and one is a test result (abnormal necrosis marker).⁴ In fact, clinical predictors are more strongly associated with outcomes than many of the laboratory tests we put faith in, and the most powerful independent predictor of all is the clinical gestalt of an experienced clinician. Recent research suggests that, in emergency department (ED) patients with undiagnosed chest pain, the most powerful predictors of 30-day outcome were age, initial ECG, prior history of myocardial infarction, angina or nitroglycerine use, pain characteristics and initial CK MB. Of note, this study did not find ischemia modified albumin (IMA) or BNP to be among the useful predictors incorporated in a derived ACS clinical prediction rule.³

Predictive value

Lippi and colleagues conclude that IMA may be a superior screening method to rule out ACS, especially when associated with standard markers of myocyte necrosis. They note that "an IMA increase at the time of ED admission might be interpreted as an indicator of ischemia prior to necrosis, displaying up to 90% negative predictive value." The fact is, a negative predictive value (NPV) of 90% tells us almost nothing about test performance. It does not mean the test is a good test or that it has any practical value.

Predictive value tells us what proportion of patients with a positive test will ultimately prove to have the disease in question and what proportion of patients with a negative test are disease free.⁵ If a test has 90% positive predictive value (PPV), this means 90% of patients with a positive test will have the disease. Similarly, if a test has 90% NPV, 90% of those with a negative test will turn out to be disease free. Unfortunately, predictive value changes dramatically depending on the type of patients the test is applied to; consequently, predictive values derived in published studies cannot necessarily be translated into clinical practice. In reality, predictive value is less a measure of test performance than it is a reflection of disease prevalence in the population being tested.

To illustrate, let's consider using a novel diagnostic test — the Canadian Loonie — in a group of patients with chest pain. We know from previous research that the Loonie is 50% sensitive ("heads" will come up 50% of the time in patients with ACS) and 50% specific ("tails" will come up 50% of the time in patients without ACS). Yet Table 1 shows that, if we apply this useless test in a typical cohort of ED chest pain unit patients with a 5% prevalence of ACS, it has excellent NPV — better in fact than IMA. Unfortunately the Loonie, like IMA, has poor PPV and will generate many false positives.

Table 2 shows that, if we introduce the Loonie as a point-of-care test in the cardiac care unit, where the prevalence of ACS is 80%, predictive values actually reverse; now it has strong PPV (80%) and poor NPV (20%). These tables illustrate the fact that useless tests have excellent NPV in low prevalence populations, and excellent PPV in high prevalence populations. They also show that predictive value changes when a test is used in different settings, that predictive value is not truly a diagnostic test parameter, and that physicians cannot use NPVs published in the literature as evidence that a new test will be helpful in their clinical setting.

Sensitivity, specificity and accuracy

While association and predictive value do not help us determine the potential value of a diagnostic test, sensitivity, specificity and accuracy tell us much more. Sensitivity tells us what proportion of tests will be positive in patients ultimately proven to have the disease in question; specificity tells us the proportion of tests that will be negative in patients who do not have the disease in question; and accuracy tells us, overall, what proportion of tests will be "correct."⁵

In a recent study, Christenson and coworkers reported real-life data showing that a group of ED physicians were 94.7% sensitive and 74% specific in identifying patients who subsequently had an ACS diagnosis made within 30 days of their ED visit.⁶ Their 5.5% "miss" rate is slightly higher than other authors have reported, and it illustrates why physicians are increasingly tempted to incorporate new diagnostic tests to maximize early sensitivity. The question is, how good are these tests?

When patients arrive in the ED with chest pain, necrosis markers like CK MB and troponin have poor sensitivity, ranging from 28%-76% depending on the duration of chest pain.⁷ Ischemia modified albumin may have better early sensitivity, ranging from 70%-92% depending on the test cut-off used; unfortunately, IMA has very poor specificity, generating a high proportion of false-positive results.^8-11 Worster and colleagues found that an IMA assay drawn at the time of patient presentation was 70% sensitive and 24% specific for serious cardiac outcomes. Table 3 shows that if such a test is applied in a cohort of ED patients with 5% prevalence of ACS, 757 of 1000 patients tested would have a positive test, but only 4.6% of positive tests (less than 1 in 20) would be true positives. At the same time the test would "miss" 30% of patients with ACS — more than 10 times the acceptable standard. Indeed, likelihood ratios* derived in the study by Worster and colleagues ranged from 0.9 to 1.75, suggesting that IMA is very poor test that does not substantially modify the post-test probability of disease.⁸

In a recent ED study of chest pain patients, Bassan and associates reported that arrival BNP was 70.8% sensitive and 68.9% specific for acute myocardial infarction (AMI).¹² They also concluded that BNP is an independent predictor of AMI and a useful adjunct to standard cardiac markers for the investigation of patients with chest pain. This conclusion is incorrect for several reasons. First, the authors did not study usefulness; they studied diagnostic accuracy (which was only marginal). Second, they considered only patients with AMI, but in real life we are also expected to identify patients with unstable angina. Finally, they jumped to the same interesting and illogical conclusion that many others have — that a test that is demonstrably less sensitive, less specific and less accurate than clinical practice will somehow improve our diagnostic accuracy.

Specificity costs

Their relatively dismal performance characteristics mean that IMA and BNP cannot stand on their own as diagnostic tests for ACS, but this is not what Lippi and colleagues (or even the test manufacturers) recommend.¹ Rather, they propose that, by adding the results of the IMA assay to a traditional panel including myoglobin, CK MB and troponins, we can increase diagnostic sensitivity for cardiac ischemia. Of course they are correct: It is a basic principle of diagnostic testing that adding tests together increases sensitivity, even if the tests have no actual diagnostic value. Adding IMA to traditional markers will increase sensitivity, but so would adding a white blood cell count, a blood glucose or a serum ferritin. By adding enough tests together, we can force sensitivity as high as we desire, but combining tests in this manner also reduces specificity, and combining IMA assays with traditional necrosis markers would drive the false-positive testing rate even higher than the 95% (false-positive rate) illustrated in Table 3.

Assuming that physicians must act upon positive tests, false positives exact a high cost on our health care system and potentially on our patients, by triggering unnecessary hospitalization, prolonged ED observation and serial testing, and diversion of scarce resources such as ED stretcher time, exercise treadmill tests, nuclear scans, cardiac catheterization slots, and hospital beds away from patients who really need them. False-positive tests also lead to downstream invasive procedures, exposing patients to a small but real risk of iatrogenic morbidity.

Test utility

If "association" does not make a test useful, if predictive value is not actually a test parameter, and if sensitivity by itself does not assure added value, then how should we decide whether to incorporate new diagnostic tests in our clinical practice? The answer is simple. We should evaluate diagnostic tests the same way we evaluate therapeutic interventions — by subjecting them to well-designed randomized clinical trials (Fig. 1).

The Basel study is an example of using clinical trial methodology to estimate the utility of a diagnostic test.¹³ In this study, 452 ED patients with acute dyspnea were randomly exposed to standard care versus standard care plus a rapid bedside BNP assay. After randomizing dyspneic patients to 1 of the 2 study arms, investigators tracked several important outcomes, including mortality rates, hospitalization, total length of stay, and overall diagnostic and treatment costs. This methodology is not perfect, and it is difficult to eliminate potential biases from a trial of a diagnostic test (for example, treating physicians given access to the new test results are clearly unblinded, which can lead to unbalanced co-intervention or even compromise outcome adjudication); nevertheless, the clinical trial approach provides much more valuable information than mere descriptions of association, predictive value and sensitivity. It helps clarify what the new test might add to clinical judgement, as well as its likely impact on health, utilization and cost outcomes.

New markers will eventually play a role in ACS diagnosis. Future strategies may involve combining markers of several distinct processes — for example, an inflammatory marker, a marker of platelet activation or coagulation, an ischemia marker, a necrosis marker, and a hemodynamic marker. The billion dollar question is whether there is a marker combination that provides better sensitivity and specificity than current clinical performance, and whether that combination will actually lead to better outcomes for our patients. To date, this question has not been answered. For now, don't feel guilty if you have been slow to incorporate IMA, C-reactive protein, BNP, PAPP-A (pregnancy-associated plasma protein-A), lactate or interleukin-6 into your practice. Broader use of these tests will increase downstream utilization, invasive testing and cost, but there is no evidence they will improve patient outcomes. If you want to optimize ACS diagnosis in the year 2006, then fix your systems and processes, implement a chest pain pathway, improve your patient handovers and negotiate timely access to provocative tests for your patients.

Conclusion

A statistically significant association is sufficient to publish a paper, but it doesn't help make a diagnosis. Negative predictive value is often used, inappropriately, to convince physicians that a diagnostic test is useful. Sensitivity is great, but not without specificity. If physicians are naive enough to change our practice based on the unconvincing data published to date, industry will have no need to conduct meaningful studies addressing utility, outcomes and cost-effectiveness. It may be true that a new era is dawning, but that faint pink glow on the horizon isn't bright enough to guide us anywhere just yet.

*Likelihood ratios are the most useful measure of a tests's diagnostic strength, but beyond the scope of this article.

Fig. 1. (enlarge) A clinical trial approach to the evaluation of a new diagnostic test.

References

1. Lippi G, Montagnana M, Salvagno GL, et al. Potential value for new diagnostic markers in the early recognition of acute coronary syndromes. Can J Emerg Med 2006;8(1):27-31.
2. Panju AA, Hemmelgarn BR, Guyatt GH, et al. Is this patient having a myocardial infarction? JAMA 1998;280:1256-63.
3. Christenson J, Innes, GD, Mcknight D, et al. A clinical prediction rule for early discharge of patients with chest pain. Ann Emerg Med 2005. In press.
4. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 2000;284:835-42.
5. Worster A, Abu-Laban R, Innes GD. Diagnostic testing in emergency medicine. Can J Emerg Med 2002;4:348-55.
6. Christenson J, Innes G, McKnight D, et al. Safety and efficiency of emergency department assessment of chest discomfort. CMAJ 2004;170:1803-7.
7. Innes G, Christenson J, Weaver WD, et al. Diagnostic parameters of CK-MB and myoglobin related to chest pain duration. Can J Emerg Med 2002;4(5):322-30.
8. Worster A, Devereaux PJ, Heels-Ansdell D, et al. Capability of ischemia-modified albumin to predict serious cardiac outcomes in the short term among patients with potential acute coronary syndrome. CMAJ 2005;172:1685-90.
9. Roy D, Quiles J, Aldama G, et al. Ischemia modified albumin for the assessment of patients presenting to the emergency department with acute chest pain but normal or non-diagnostic 12-lead electrocardiograms and negative cardiac troponin T. Int J Cardiol 2004;97:297-301.
10. Anwaruddin S, Januzzi Jr JL, Baggish AL, et al. Ischemia-modified albumin improves the usefulness of standard cardiac biomarkers for the diagnosis of myocardial ischemia in the emergency department setting. Am J Clin Pathol 2005;123:140-5.
11. Sinha MK, Roy D, Gaze DC, et al. Role of ischemia modified albumin, a new biochemical marker of myocardial ischaemia, in the early diagnosis of acute coronary syndromes. Emerg Med J 2004;21:29-34.
12. Bassan R, Potsch A, Maisel A, et al. B-type natriuretic peptide: a novel early marker of acute myocardial infarction in patients with chest pain and no ST-segment elevation. Europ Heart J 2005;26:234-40.
13. Mueller C, Scholer A, Laule-Kilian K, et al. Use of B-type natriuretic peptide in the evaluation and management of acute dyspnea. N Engl J Med 2004;350:647-54.