Does outcome feedback make you a better emergency physician? A systematic review and research framework proposal

Review Articles

CJEM 2009;11(6):545-552

Abstract

Résumé

SEE RELATED ARTICLES ON PAGES 523 AND 566

This article has been peer reviewed.

INTRODUCTION

Emergency medicine (EM) is a 24-7, on-demand, high-pressure discipline that provides care for a wide spectrum of illness and the sickest of patients, yet it includes very little opportunities for follow-up to learn the outcomes of these patients. Put another way, emergency physicians almost never see their patients again and rarely hear what happens to them. However, this is not the case for other medical specialties. From primary care medicine to tertiary care surgical services, most physicians have the benefit of scheduled follow-up visits, if not extended observation, after diagnostic or therapeutic interventions. It is through this follow-up that these practitioners receive outcome information that confirms or challenges their initial decisions and their decision-making processes.1 Moreover, it is in this way that physicians learn from their patients and gain clinical experience. The consequences of the lack of outcome feedback for emergency physicians concern us, and as a result we sought to systematically review the available evidence concerning this phenomenon.

Concerns about a lack of outcome feedback in EM are not new. Croskerry1 suggested that emergency physicians have great difficulty finding out what happens to their patients and that most “lament” this loss of information. However, he noted that with time emergency physicians come to accept this as part of the development of what he calls an “adaptive myopia.” Croskerry states that the problem is that such information is crucial to calibrating the diagnostic tools of physicians and without it they tend to become overconfident in their diagnostic strategies.1

There are many types of feedback in the practice of medicine, and as such, the concept of outcome feedback in EM requires clarification. Outcome feedback, as we define it, is limited to the natural process of finding out what happens to one's patients after their evaluation and treatment in the emergency department (ED). Did they improve, or did they deteriorate? Did another doctor make a new diagnosis at a later time? Did anything unexpected happen? We specifically exclude the outcome of tests that were performed during the ED visit in our definition (e.g., bacterial cultures that are reported later or radiographs done at the time of consultation that are later interpreted by a radiologist) since these do not reflect evolution or rediagnosis of a pa tient's condition. We also exclude evaluative types of feedback in our definition, whereby individuals might critique the care provided to patients. This might occur during educational activities and training (by mentors or supervisors), during audits by peers or other professionals, or by surveying patients and families themselves after the visit. Although these types of feedback all have value, they are quite different from the natural phenomenon of simply finding out what happens to our patients after they leave our care.

We undertook this review to determine what is currently known about outcome feedback in EM. This includes but is not limited to the incidence, the impact and the modifiers of outcome feedback in EM. In other words, how much outcome feedback do emergency physicians receive now, what effect does it have on practice, and what factors make it more or less useful?

METHODS

Study design

We developed a comprehensive search strategy and applied it to Ovid’s MEDLINE database for articles from 1950 until November 2008 (Box 1). We developed the search strategy using medical subject headings and text words for the general concepts of 1) emergency medicine, 2) patient/treatment outcomes and discharge and 3) feedback and decision support techniques. This was subsequently adapted for and applied to the following databases: PreMEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), CENTRAL (Cochrane Central Register of Controlled Trials), PsycINFO, DARE (Database of Abstracts of Reviews of Effects), Health Technology Assessment Database, and AMED (Allied and Complementary Medicine Database). There were no restrictions placed on the EM context, trainee versus attending physician status, language of publication, pub li cation status (e.g., unpublished evidence) or experimental design for intervention studies.

Other sources reviewed were reference lists of relevant reports and selected review articles; the literature files of 3 experts (Dr. Chii-Hwa Chern, Veterans General Hospital, Taipei and National Yang-Ming University, Taipei, Taiwan: personal communication, 2006; Dr. Pat Croskerry, Department of Emergency Medicine, Dartmouth General Hospital, Dartmouth, NS: personal communication, 2006; Dr. Louis Graff, New Britain General Hospital, New Britain, Conn.: personal communication, 2006); the websites of key health information and research foundations;2-10 and 4 key international EM websites.11-14 In addition, we manually searched the 1998-2008 issues of the Canadian Journal of Emergency Medicine (CJEM), and the abstracts published by the Society for Academic Emergency Medicine15 (2003-2008) and the Canadian Association of Emergency Physicians (CJEM 2003-2008). We entered search results into Endnote citation management software, version 5.0 (Thompson Reuters), and removed duplicates.

Selection process

Eligibility criteria for this review included any research addressing outcome feedback (defined as “physicians knowing the outcomes of their patients after discharge from their care”) provided to emergency physicians or trainees, excluding any feedback on the “results of diagnostic tests performed during care” or feedback involving “a critique” (i.e., a comparison of the outcome or management of a case to a standard of care or an expert opinion). Two reviewers (C.L. and A.S.) underwent a calibration exercise that employed 10 preselected bibliographic records to ensure conformity to the review objectives. Reviewers then performed a dual independent review of the search results. The first level of screening, using only the bibliographic records including abstracts, excluded only those records for which there was clear and unequivocal agreement as to their irrelevance. We retrieved the corresponding full-text reports for the remainder of the records and performed a second screening using the same eligibility criteria. We resolved disagreements via forced consensus. We calculated interrater agreement at both levels of screening. Reports were included in this review if they passed both levels of screening and reported original research.

Quality and relevance assessment

We informally assessed all relevant studies for quality of reporting, study design and relevance. For reporting quality we assigned labels of “adequate,” “inadequate” or “unable to assess,” based on the clarity of the methods and results in the paper and their ability to permit replication of the study. The same labels were assigned in evaluating the study design of survey and intervention reports. For surveys, this determination was based on the ability of the methodology to elicit true frequency data. To do this we considered sampling bias, response rate and whether “outcome feedback” was well defined in the survey. In evaluating the study design of intervention reports we assessed relevance by looking for specific biases including “nonoutcome contamination bias.” Since outcome feedback refers only to information on what actually happens to the patients, feedback of evaluative information such as found in audits, advice or similar information, would constitute a contamination of the intervention. The heterogeneity of the reports, the novelty of the topic and the uniformly low quality prevented a more formal assessment using standardized evaluation tools.

RESULTS

The database search yielded 1128 unique references that underwent primary screening. This process identified 45 potentially relevant reports, and we found an additional 9 reports through review of reference lists and a manual search of journals. Of the 54 potentially relevant reports, we obtained the original articles for 53 and, from these, we deemed 7 relevant for review. The interobserver agreement (κ) was 0.97 during the primary screening and 0.81 during the secondary screening. Five reports described efforts concerning the prevalence of outcome feedback in EM or the perceived need for more outcome feedback (Table 116-20) and 2 reports addressed the impact of outcome feedback interventions (Table 221,22). Quality assessments were generally hampered by the method of reporting: 2 were only available as abstracts16,17 and 1 only as a letter to the editor.18 We were unable to contact authors to obtain more information. As a result we are only able to present a qualitative and descriptive appraisal of the results.

Outcome feedback: mechanisms, prevalence and perceptions

Table 1 summarizes 5 reports of physician surveys concerning the availability and perceptions of outcome feedback systems. None of the studies reported the actual percentage of patients for whom outcome feedback was available.

The first 4 reports were surveys of EM residents and program directors, and largely investigated compliance with a requirement for outcome feedback by the Residency Review Committee in Emergency Medicine in the United States. This requirement has been in place since 1980,22 and states that EM residency programs must “assure that the residents follow-up on a representative sample of patients so as to learn about the results of diagnostic studies, the outcome of interventions, and the final patient diagnosis.”23 The fifth study examined the habits of attending physicians.

Three of these studies17,19,20 surveyed EM residency program directors, over several years. The results indicate that outcome feedback systems for EM residents have been increasingly available since 1984.

Four of these studies looked at the availability of specific outcome feedback mechanisms, including one which surveyed attending emergency physicians. For admitted patients, automatic mailing of discharge summaries and case conferences (or quality assurance activities) were reported to be the most commonly available mechanisms for EM residents,16,19,20 as well as the most popular for attending emergency physicians.18 Only 1 study looked at mechanisms of outcome feedback for discharged patients. This reported that the use of direct telephone calls to patients was the outcome feedback mechanism most commonly available for EM residents.20 Three studies surveyed EM residents or residency program directors about their perceptions of these systems of outcome feedback and all found a 60% or higher rate of dissatisfaction or desire for improvement.16,17,19 The reporting and study quality of all reports were deemed to be “inadequate” or “unable to assess.”

The impact of outcome feedback

Table 2 summarizes the articles that address the impact of outcome feedback in EM. The first study by Chern and colleagues21 was a controlled intervention study that found a significant reduction in adverse events in response to telephone outcome feedback. This was a before-and-after study of the effect of outcome feedback obtained by telephone calls to patients on the rate of adverse patient outcomes in a Taiwanese hospital. In the 15-week control period (Apr. 15-Jul. 31, 2001), 2 senior emergency physicians reviewed all emergency charts every second day and telephoned all discharged patients flagged as “high-risk” within 24 hours of discharge. The criteria for high-risk flagging included a total of 40 neurologic, cardiovascular, pulmonary, gastrointestinal and infectious conditions, as well as any patients who were intoxicated or had abnormal laboratory values. During the 15-week intervention period (Aug. 1-Nov. 15), this process continued and the outcome information obtained was given back (within 2 days) to the resident involved in the initial patient care. Using a review of admissions and repeated telephone calls to patients who did not improve, the authors monitored the rate of clinically significant adverse events (CSAEs), which were defined as return visits with serious management error (misdiagnosis or erroneous treatment plan) resulting in death, or admission for longer than 3 days.”21

During this outcome feedback intervention, Chern and coworkers21 observed a drop in the rate of CSAEs in high-risk cases from 4.1% in the control period to 1.5% during the intervention period (a reduction of 2.6%, 95% confidence interval 0.3%-4.8%), and relative reduction of the overall CSAE rate of more than 50%, from 0.9% to 0.4% (reduction of 0.5%, 95% confidence interval 0.1%-0.9%).21 However, a number of limitations of this study merit consideration. First, the outcome information provided to the residents was combined with other feedback including a discussion of “the pitfalls in diagnosis or management, [and] appropriate performance that might avoid adverse event recurrence and the possible error.” This nonoutcome information may have affected physician performance and therefore patient outcomes. Second, the control and intervention periods had potentially significant differences: they occurred during different seasons, did not have equivalently trained residents and did not have equivalent teaching rounds. Furthermore, “shift change-over rounds” (i.e., lectures that “focused on the need to understand the uncertain nature of many ED conditions, the causes of return visits with adverse events, and the utility of using resources available to manage uncertain ED conditions”) were introduced during the intervention period. Finally, neither the high-risk criteria nor the CSAE definition were validated, and no independent or blinded review of outcomes was obtained. These various potential confounders and sources of bias are likely to have obscured the true relationship between pure outcome feedback and adverse events.

In the final study, Sadosty and colleagues22 evaluated a Web-based outcome follow-up system for EM residents. These trainees followed 1 patient per shift and documented the details on a Web-based e-form. Both the trainee and supervisor were then asked to rate the educational value of the case follow-ups. The vast majority of cases were deemed to be educational by both the resident and faculty reviewer. However, the definition and interpretation of “educational” was not provided, and no other outcomes were measured.

DISCUSSION

Finding out what happens to patients after they leave an emergency physician’s care is a phenomenon we have called “outcome feedback.” In this systematic review we found sparse literature of generally inadequate quality, from which only limited conclusions can be drawn. First, there is evidence to suggest that US EM residency programs have increased the availability of systems for outcome feedback to residents since 1984, but that these systems were perceived as inadequate by respondents. Second, the mechanisms of outcome feedback used most commonly by physicians include automatic routing of admission discharge summaries, case conferences and telephone calls to discharged patients.

Outcome feedback: a research framework proposal

Croskerry1 suggests that increasing outcome feedback to emergency physicians would improve calibration and overall effectiveness of their practice. From this review we can conclude that this important hypothesis has yet to be tested. As such, we propose a framework which centres on 3 broad questions to move forward with this topic of research.

First, what is the incidence and nature of outcome feedback occurrences within EM? In other words, how much outcome information is currently provided to practitioners, how is outcome information obtained, how is it provided and what are the barriers to its provision or reception? As a first step, understanding the current state of outcome feedback in EM is paramount. Is there indeed an overall deficit as Croskerry suggests?1 If so what are the barriers? Croskerry suggests several barriers to outcome feedback: a lack of awareness of its importance, time pressures and workloads, shift work structure (no contact with other providers), action- outcome delays, scarcity of diagnoses, specialist follow-up deficiency, shift change-over, deficient reporting systems and numerous communication problems (e.g., handwriting). Other barriers may exist and all need to be investigated.

Second, what impact does outcome feedback have on diagnostic accuracy, treatment outcomes, clinical efficiency and job satisfaction? Based on Croskerry’s suggestions,1 it is reasonable to hypothesize that diagnostic accuracy would improve through calibration. However knowing the results of treatment choices might also improve future treatment choices, resulting in better treatment outcomes. Greater clinical efficiency might result from the ability to make more rapid diagnoses, also through calibration. And job satisfaction? Consider that by not knowing the outcome of the diagnoses and treatment choices they make, emergency physicians may fail to fully understand the impact they are having on their community. We believe that job satisfaction is tied to a sense of one’s role in society and hypothesize that by increasing outcome feedback, pride in one’s job increases. In addition to these possible benefits, there may be negative effects to practitioners such as increased workload, overly cautious practice, emotional stress and the risks of inaccurate feedback. These would need to be determined and investigated before recommendations can be made.

Third, what are the modifiers? In other words, what are the factors that prevent or ensure any beneficial impacts while also minimizing any possible negative effects? For example, if outcome feedback is to be helpful it likely needs to be accurate and objective. It also needs to occur soon enough after the emergency visit that the physician still recalls details of the case (thus reporting delay must be avoided), yet long enough after the visit to allow for evolution in the illness or results from the treatment (thus the observation interval must be appropriate). Also the outcome information should not require excessive time, effort or expense to obtain. Furthermore the presence of subjective critiques of the physician (evaluative contamination), increased medicolegal responsibility or excessive nonoutcome information would likely diminish the utility of outcome feedback. These and other factors should be investigated.

Future research in outcome feedback will require multiple approaches. Surveys and focus groups of EM practitioners could inform on its frequency, barriers benefits and drawbacks. A controlled simulation study using either real or fictitous patient cases would inform on its impact, as well as allow manipulation of potentially modifying factors. Eventually a large real-world trial would be the most informative source of information with the greatest potential impact. Practical barriers to this may be substantial: the intervention needs to be specific to outcome information, patient confidentiality needs to be assured, and study outcomes need to be clear and objectively measured. Other studies that are limited to certain presenting complaints, diagnoses, mechanisms or patient outcomes would also be helpful in further elucidating this phenomenon.

CONCLUSION

Does finding out what happens to your patients make you a better doctor? Because of poor design and reporting quality in the available literature, there is insufficient evidence to answer this overriding question. We propose a framework for future research in outcome feedback: first, to understand the status quo (the nature, incidence and barriers); second, to investigate the impact of increased outcome feedback; and third, to assess the factors that modify this impact. Based on a model of calibration suggested by Croskerry1 we hypothesize that increased outcome feedback to emergency physicians could significantly improve the delivery of emergency medical care.