"I never hear what happens, even if they die": a survey of emergency physicians about outcome feedback

EM Advances

CJEM 2009;11(6):523-528

Abstract

Résumé

SEE RELATED ARTICLES ON PAGES 545 AND 566

This article has been peer reviewed.

INTRODUCTION

Emergency medicine (EM) is a quixotic profession: emergency physicians (EPs) deliver one-time, episodic care to random patients during unpredictable health crises at all hours of the day; and then, most commonly, practitioners never see their patients again. As a result of the current organization of EM care, EPs have difficulty finding out what happens to their patients after discharge, a process we term "outcome feedback." In contrast, pediatricians, internists and family physicians follow up their patients for years and thus have a plethora of opportunities for outcome feedback. Even surgeons, who typically also provide one-time interventions, have the benefit of several days of hospital observation of their patients after procedures, and at least 1 follow-up visit. A question central to this discussion is, Is outcome feedback important?

A systematic review on outcome feedback in EM, however, revealed a paucity of research on the subject. The purpose of this study was to determine how much outcome feedback is currently taking place in EM, what the EP desire is for more and EPs' perception of the impact of having more outcome feedback. We hypothesized that EPs currently receive outcome feedback on only a small proportion of their patients and that the majority of EPs would like to receive more. The over arching purpose of this hypothesis-generating study was to assist in the launch of the field of outcome feedback research in EM.

METHODS

Study design

We developed a survey instrument in French and English for distribution to EPs. It was designed to elucidate responses in the following 4 domains: 1) the role of EP experience and outcome feedback, 2) the prevalence and merit of various mechanisms of passive outcome feedback, 3) the prevalence and merit of various mechanisms of active outcome feedback, and 4) the perceived effects — both negative and positive — of getting more outcome feedback.

We defined passive outcome feedback as "outcome feedback that arrives automatically without actively seeking it out," whereas active outcome feedback was defined as "outcome feedback that is actively sought." Examples of passive outcome feedback include reply letters to consultation requests, discharge summaries on admissions, information on patient transfers, direct feedback from another physician or family member, information on returns to the ED, and autopsy reports. Examples of active feedback include arranging a follow-up visit with the patient, checking on patients who are admitted, looking up charts on patients, contacting receiving physicians or hospitals, contacting the coroner and calling patients or family.

We used a variety of open-ended, Likert and num eric response questions and solicited demographic information concerning age, sex, language group, training, type of practice, professional membership and location of practice. We piloted the survey among 10 EPs for readability and time to completion, and incorporated their feedback into the final instrument. We obtained ap proval from the Research Ethics Board of the Children's Hospital of Eastern Ontario and engaged 2 certified translators, one to translate the original English questionnaire into French, and the other to back-translate the survey into English. We then compared the latter English translation with the original to ensure the accuracy of the French translation. In addition, posters, reminder postcards and email messages were prepared and translated into French for promotion of the study.

Setting and selection of participants

We felt that face-to-face contact and discussion with medical directors and staff at hospital sites, and on-site distribution of posters would improve participation and commitment to the study. To permit this face-toface contact and still maximize rural participation, we used a geographic criterion and surveyed only hospitals within 100 km of Ottawa, Ont. We established a list of all hospitals within this radius using a hospital registry and a map of Ontario and Quebec. The primary investigator (PI) contacted the medical director of each ED to seek participation of the site. The PI then travelled to each participating hospital to establish physician eligibility, distribute the survey, meet participants and raise enthusiasm. Physicians were deemed eligible for inclusion in the study if their name appeared on the ED duty roster of an eligible hospital for the month of February 2006 (the month the study took place), or if their absence was due to parental leave. Surveys were coded on-site by the PI and deposited directly into physicians' mailboxes when possible, or by mail to physicians' offices. Promotional posters were placed in strategic locations in or around the ED. The PI subsequently forwarded the codes to the research assistant, and the PI remained blinded to the survey assignment.

Data collection and processing

To maximize the response rate, we used a modified Dillman method.2 Within 2 weeks of initial contact, a promotional email message was forwarded via the individual directors to all physicians who had email distributions already in use (all but 2 rural hospitals). Within 3 weeks of the survey distribution, we sent reminder postcards to all nonrespondents and followed up within another 2 weeks with a second mailing of the entire questionnaire to persistent nonrespondents, this time to their office address as indicated in the Canadian Medical Directory.3 We did a final mailing of the survey to the remaining nonrespondents 2 weeks later. Data from the survey results were entered into and analyzed using SPSS Version 14.0 (SPSS Inc.). We assessed reliability by a re-entry of 10% of the data set.

Primary data analysis

We generated descriptive summaries of each survey question and summarized dichotomous variables using percentages. We summarized normally distributed continuous variables using means together with standard deviations, and we summarized continuous variables that were not normally distributed, using medians together with range. For several questions, we collapsed responses across similar categories for the purpose of analysis and to simplify the presentation of results.

We compared study objectives across physicians based on years in practice (dichotomized into groups of ≥ or < 10 yr), sex, language group, urban versus rural location and the clinical proportion of pediatrics (dichotomized into groups of ≥ or < 50%). We used the Fisher exact test, χ2, Student t test or the Mann-Whitney test as appropriate for the data in question. We obtained demographic data for all eligible physicians from the publicly available Canadian Medical Directory3 and used the data to compare nonrespondents to respondents. When possible, we generated 95% confidence intervals (CIs). We deemed p values of < 0.05 statistically significant.

RESULTS

Characteristics of study participants

Twenty-two hospital EDs were eligible for inclusion in the study and 297 physicians were identified on the February 2006 ED schedules for these institutions. Of these, 231 (77.8%) completed the survey. Respondent demographics are presented in Table 1.

Attitudes about experience and outcome feedback

The first survey question addressed the relative contribution of various sources of information to clinical decision-making in EM. The average contribution of "experience" was 46.0% (95% CI 43.6%-48.4%), which was significantly greater than "scientific evidence" (27.0%, 95% CI 24.1%-29.1%) and "tradition or opinion" (26.7%, 95% CI 24.7%-28.7%). This response distribution persisted when rural and urban physicians, or male and female practitioners, were compared. Experience was a significantly greater factor among those with 10 or more years of practice (51.4%, 95% CI 48.2%-54.6%) compared with those who had practised for less than 10 years (39.2%, 95% CI 36.0%- 43.0%). More experienced physicians also reported less contribution from direct scientific evidence (24.0% v. 30.0%) and opinion or tradition (24.1% v. 29.7%).

We assessed the role of patient outcome information, as it contributes to experience, by asking participants to choose 1 of 4 possible answers to complete the phrase, "Knowing treatment outcomes is [blank] to gaining clinical experience." Most of the respondents (144/230, 62.6%) indicated that it was "essential," with the rest indicating that it was "beneficial" (86/230, 37.4%). No respondents stated that knowing outcomes was "of no value" or "detrimental" to gaining clinical experience. We further explored the role of outcome feedback by asking respondents to "estimate what proportion of the most important clinical patient outcomes occur … during your care" and "after leaving your care." The mean responses were 47.1% (95% CI 44.0%-50.6%) during care and 52.8% (95% CI 49.3%-55.8%) after leaving care.

Prevalence and desire for outcome feedback

Participants reported receiving passive outcome feedback in a median of 10% (range 0.5%-90%) of cases and seeking out active outcome feedback in a median of 5% (range 0%-80%) of cases. This distribution was similar when we compared male and female as well as rural and urban physicians. A total of 96.8% (215/222) of physicians desired more passive outcome feedback, and 82.8% (173/222) desired more active outcome feedback.

Respondents who desired more outcome feedback were asked to list the most valuable means of obtaining it and how often they currently received it (Table 2). For passive outcome feedback the most commonly indicated mechanisms included consultant reply letters and discharge summaries. For active outcome feedback, the respondents valued direct phone calls to patients and family, as well as looking up the chart of admitted patients and contacting a second EP (e.g., after an end-of-shift patient transfer). In a subquestion on the subject of phone calls to patients for outcome feedback, respondents indicated they called patients or families after discharge in a median of 5% (range 0%-75%) of cases and that the best time to call was 2 to 3 (median 2.5, range 0-30) days after discharge.

Positive and negative effects of more outcome feedback

A large majority of respondents believed that greater outcome feedback would have positive effects on the practice of EM, including effects on diagnostic accuracy, clinical efficiency, treatment outcomes and job satisfaction (Table 3).

When asked to indicate "any negative effects that might arise from getting more outcome feedback," 144 re spondents (62.1%) indicated none. The most common negative effect suggested was the possibility of a negative emotional effect, including sadness, loss of confidence or even depression, and possibilities of this nature were suggested by 43 respondents (18.5%). A loss of time or increased workload was suggested by 27 respondents (11.6%), an increase in overcautiousness by 11 (4.7%) and an increased liability by 4 (1.7%). One percent or fewer respondents suggested other negative possibilities including impacting the environment (through paper use), a loss of confidentiality, and an increase in errors if inaccurate information were given or pertinent scientific evidence were ignored as a result of outcome feedback.

DISCUSSION

We report the results of a large regional survey across a wide demographic of Canadian EPs on the prevalence and importance of outcome feedback. Our results support the role of outcome feedback as an important factor in the development of expertise and satisfaction, and suggest that improving it would be largely beneficial to the practice of EM. To our knowledge, this is the first published research on this subject.

Our findings support assertions made by Croskerry in 20004 in a review concerning the need for outcome feedback in EM. Croskerry's thesis is that outcome feedback is an essential mechanism for calibrating EP's clinical skills and gaining clinical acumen. Indeed our respondents indicated that outcome feedback is perceived as beneficial, if not essential, for gaining experience, and that this is EPs' greatest source of clinical knowledge for decision-making. Of interest is the fact that senior physicians (who have been in practice for more than 10 years) indicated they rely more heavily on experience as a source of knowledge when compared with less experienced physicians. Recent changes in medical curricula stressing scientific evidence-based decision-making may be a factor in this. Our finding may also be a reflection of a natural tendency of physicians to rely more and more, over time, on skills and knowledge acquired during practice, rather than on external sources of knowledge.

Croskerry also asserts that EPs rarely know the outcomes of their patients, and that they "lament" this "feedback sanction."4 Emergency physicians in our study indicated that they receive outcome feedback on only a small minority of their patients, and the vast majority of EPs would like more outcome feedback. Croskerry also asserts that EPs have come to accept this loss of information in what he terms an "adaptive myopia." However, far from accepting it, our results suggest that EPs make considerable efforts to work around system impediments to know their patients' outcomes. This interpretation of our findings is supported by a number of things. First, the practice of seeking such information is not reimbursed. Second, the time to track down patients, caregivers, charts or other physicians can easily take more time than the original patient encounter. Finally, it is really difficult to know in advance which patients will have "unexpected outcomes" so the yield of endeavours of this nature for such outcomes is likely to be low.

Croskerry4 also suggests that clinical acumen is improved with outcome feedback, and to date there is no original research to support this in EM. An overwhelming proportion of EPs in our study indicated that increased outcome feedback would have a positive impact on their practice.

Limitations

This study attained a relatively high response rate, as well as a sampling of a range of sex, language and professional groups of physicians practising EM in Canada, albeit in a limited geographic area. Although it would have been ideal to sample more broadly, we chose a smaller region in order to establish a comprehensive census population and facilitate face-to-face contact, which we believed would enhance the response rate and thus the validity of our results. Moreover, the membership list of the Canadian Association of Emergency Physicians (CAEP), the only national database of EPs, may not appropriately reflect the entire population of EM providers. In correspondence with the CAEP Research Consortium, their chair reports that membership lists underrepresent rural, part-time, and non-EM certified EPs, all of whom were of interest to us (Dr. Brian Rowe, Edmonton, Alta.: personal communication, 2005).

Reliance on self-reporting may have limited the validity of these results because of recall bias and/or social desirability bias. This is possible because although little research has been done on outcome feedback, following up on one's patients may be generally perceived as the "right thing to do." As such, physicians, despite anonymity, may have overestimated the actual amount of outcome feedback they obtain, as well as their "desire for more." In interpreting our results, it is also important to consider that this study was exploratory in nature and involved multiple comparisons.

CONCLUSION

The overwhelming majority of EPs receive very little outcome feedback. Most would like more outcome feedback, and believe it would improve the practice of EM. However, elucidating the actual merits of outcome feedback will require further research using both qualitative techniques and intervention studies. A clinical trial with a cluster randomized design may be the best way to determine the true effects — both positive and negative — of outcome feedback. We believe the time has come to find out whether knowing what happens to our patients would make us better physicians.