The appropriateness of referrals to a pediatric emergency department via a telephone health line

EM Advances

CJEM 2009;11(2):139-148

Abstract

Résumé

Introduction

Telephone health lines have become increasingly widespread.1 These health lines vary in availability from "after hours" to coverage 24 hours per day, 7 days per week. Nurses usually staff the lines and triage the callers using algorithms and clinical judgment to provide health care advice. In late 2001, the Ontario Ministry of Health and Long-Term Care implemented a free 24-hour telephone health line known as "Telehealth Ontario," which is staffed by registered nurses who use protocols to direct patients to self-care or appropriate medical resources.2

Implementing a telephone health line requires a substantial investment of resources,3 making its evaluation important. Although no peer-reviewed evaluation of Telehealth Ontario has been published, similar health lines have been evaluated. Overall, it appears that user satisfaction is high, ranging from 95% to 99%,4,5 although one study showed a satisfaction rate of 55%.6

Of the calls concerning pediatric problems, studies consistently report that about 20% of patients are referred for urgent medical care.7-10,11 Of these referrals, 63%-83% of callers comply with the recommendation.9-12 Only a few studies have examined the appropriateness of these referrals. These studies, using a variety of methods, determined that approximately 78%-90% of emergency department (ED) referrals were appropriate.5,7,13 In only one study, however, was the appropriateness of health line- referred patient visits compared with that of other ED patient visits,13 and in no study were the reviewers who were determining the appropriateness blinded to the purpose of the study.

Our study had 3 objectives: 1) to determine the appropriateness of patients referred by a province-wide telephone health service to a pediatric ED, using published criteria applied by a pediatrician blinded to the study purpose, and to compare these patients with self- or parent-referred and physician-referred patients; 2) to compare the examining physician's rating of appropriateness for the 3 groups of patients; and 3) to compare the triage category, clinical characteristics, investigations performed and outcomes of the 3 groups of patients. To our knowledge, ours is the first study to explicitly compare pediatric patients who were referred to the ED via a telephone health line, with self- or parent-referred, and physician-referred patients.

Methods

Study design

This prospective, observational study compared the appropriateness of ED visits among 3 groups of children — those who were physician-referred, telephone health line-referred, and self- or parent-referred. The Children's Hospital of Eastern Ontario (CHEO) Research Ethics Committee approved this study.

Study setting and population

We conducted our study in the ED of CHEO, which is an academic, tertiary care children's hospital in Ottawa, Ont., with an annual ED census of 55 000. All children who were referred to the ED by a provincial telephone health line (Telehealth Ontario) or by a physician during 1 of 4 randomly chosen weeks between Sep. 1, 2003, and Aug. 31, 2004, were eligible for inclusion in the sample. For each child referred by the health line, the next patient who was self- or parent-referred was also enrolled. Only children who resided in the province of Ontario were eligible for inclusion in the study.

Telehealth Ontario

Telehealth Ontario is a province-wide program established by the Ontario Ministry of Health and Long-Term Care.2 It is a free telephone service that offers health advice to the residents of Ontario (with a population of approximately 11.5 million); the health advice is provided by a team of registered nurses. The availability of the health line has been widely advertised by the provincial government and most EDs refer patients seeking general medical advice to Telehealth or their family physician. Based on the information provided by the caller, protocols are used to advise self-care, recommend a visit to a health care provider or suggest community resources. If a visit to a health care provider is recommended, the caller is given guidance about which provider to visit (e.g., their family physician or an ED) and the time frame for this visit (e.g., proceed immediately to an ED). Telehealth calls are placed into 5 categories: priority (call 911 immediately), emergent (see a physician within hours), urgent (see a physician within 24 h), referral (see a physician within 72 h) and self-care. Within the region served by CHEO, any pediatric patient under 18 years of age who is deemed emergent is advised to attend the CHEO ED, and a fax is immediately sent notifying the ED of the patient's referral. Although several other EDs in the region will see children, only CHEO admits children, and families are encouraged by Teleheath to attend CHEO. If the families report they will be attending other EDs, the fax is sent to the appropriate ED. The nurses receive no specific pediatric training, but may contact a senior nurse supervisor to discuss cases for which they have questions.

Study protocol

To reflect seasonal variations in ED usage and patterns of injury or illness, 1 week in each season was randomly chosen using a computer-based sequence for a total of 4 weeks of data collection. During the study periods, a registration clerk identified each patient's route of referral upon presentation to the ED. Patients were defined as health line-referred if they reported being referred by Telehealth, and this was subsequently confirmed by receipt of a fax from Telehealth. For each such patient enrolled in the study, the next self- or parent-referred patient was also enrolled. All physician- referred patients were enrolled. Patients were confirmed to be physician-referred if a letter from the referring physician, who had recently seen the patient, accompanied the patient. Patients were excluded if they were referred from physicians at other EDs. There was no follow-up of patients referred by Telehealth who did not present to the ED.

The most senior emergency physician examining the patient was asked to give his or her opinion of the need for the ED visit in a simple question format. Emergency physicians could not be blinded to the route of referral or to the fact that they were examining visit appropriateness. To minimize physician bias, we phrased the question as "Did this patient need to be seen in the emergency today?" Answer categories included "No," "Yes, for medical reasons," "Yes, for social reasons" and "Yes, for other reasons." Parents who identified Telehealth as the route of referral were also asked if they would have come to the ED had Telehealth not recommended a visit. A chart review was completed for all enrolled patients to allow collection of data regarding baseline characteristics, triage categories, final diagnosis, investigations and treatments.

After the ED visit, a pediatrician who was blinded to the purpose of the study reviewed patients' charts and completed a standardized checklist14 on a data form that contained no mention of visit appropriateness (Box 1). We did not remove from the patients' chart details that the physician or triage nurse may have written in their notes regarding how the patient came to the ED, but Telehealth faxes do not form part of the hospital chart. Telehealth faxes were reviewed daily to identify any eligible patients who were missed at registration; these cases were enrolled in the study, but only a chart review was possible for them. As these health line-referred patients were retrospectively identified, it was not feasible to identify the next self- or parent-referred patient.

Outcome measurements

The primary outcome was the appropriateness of referral, as determined using specific criteria published previously (Box 1).14 These criteria were divided into 3 main categories: 1) specific diagnoses and dispositions, 2) symptoms and complaints and 3) parental concerns. Equal weight was given to the 3 categories and a visit was deemed appropriate if any of the items on the checklist applied to the patient's visit. This checklist was chosen because it has good interobserver reliability, was developed specifically for pediatric patients and has been used in 2 pediatric ED appropriateness studies.14,15 Secondary outcome measures included physician determination of the appropriateness of the referral, triage categories, investigations, treatments, final diagnosis, disposition, specialty referral, length of ED stay and parental attitudes toward Telehealth referral. The visit was deemed appropriate if the physician responded that the patient needed to be seen urgently in the ED for any reason.

Data analysis

Descriptive statistics were generated to summarize patient characteristics for each of the 3 study groups. Chi-square or Fisher exact tests were used to assess whether the proportion of patients deemed appropriately referred was homogeneously distributed over the 3 referral route categories. Estimates of visit appropriateness were computed along with their 95% confidence interval (CI) using the Wilson score method.16 Secondary outcomes were compared between study groups using χ² tests (or Fisher exact tests when appropriate). Differences in the length of ED stay were assessed using the Kruskal-Wallis test and the Mann-Whitney test for post hoc analyses. Assuming a χ² test with 2 df, an effect size of 0.20 (which is between a medium and small effect size according to Cohen,17) and using an α of 0.05 and a power of 80%, we aimed to enroll a minimum of 81 patients per group.

Results

Sample characteristics

Five hundred seventy-eight patients were enrolled in the study, with 164 (28.4%) in the autumn, 190 (32.9%) in the winter, 124 (21.4%) in the spring and 100 (17.3%) in the summer. There were 347 subjects in the physician-referred group, 102 in the self-referred group and 129 in the health line-referred group. The final sample included 27 health line-referred patients who were missed at registration, but were enrolled on subsequent review of the health line faxes. There were 74 patients who identified themselves as health line-referred, but were subsequently excluded because no health line fax was received. Fifteen patients who registered in the ED and then left without being seen were not included in the final sample for determination of appropriateness by checklist or emergency physician. Patient and referral characteristics are shown in Table 1. Health line-referred subjects were younger than patients in the other 2 groups. Physician referrals were most common during the day on weekdays, but there was little difference between health line-referral and self- or parent-referral in terms of the time of day or the day of the week.

Visit appropriateness and triage categories

Visit appropriateness was determined using the checklist criteria proposed by DeAngelis and colleagues14 and according to the examining physician, and are described in Table 2. Using the checklist criteria, 406 (72.1%, 95% CI 68.3%-75.7%) visits were deemed appropriate. There was no statistically significant difference in appropriateness by referral route.

The examining physician determined visit appropriateness for 407 (72.3%) patients (Table 2). The proportion of visits that was deemed appropriate was 72.7% (95% CI 68.2%-76.8%) for physician-referred patients, which was significantly greater than visits by patients who were self- or parent-referred (p < 0.001) or health line-referred (p = 0.002). There was no significant difference in the physician's rating of visit appropriateness between parent- or self-referred patients and those who were referred by the health line (p = 0.50). Among visits deemed appropriate, almost all were considered necessary for "medical reasons." Table 2 also compares the ED triage categories for the 3 groups. There are clear differences: a significantly higher number of patients who were referred by a physician were triaged as "urgent," "emergent" or "resuscitation."

Given that physicians did not determine appropriateness for 30% of enrolled patients, we compared the triage categories and discharge disposition for those patients assessed by a physician for visit appropriateness with those who were not assessed for visit appropriateness (Table 3). Overall, patients for whom physicians did not indicate visit appropriateness were more likely to be admitted to hospital. If we assume that physicians considered all admitted patients to have appropriate ED visits, we still find a significant difference in appropriateness between groups (p < 0.001). Visits by physician-referred patients were still deemed to be significantly more appropriate than visits by patients who were self- or parent-referred (p < 0.001) or health line-referred (p < 0.001); there was no significant difference between self- or parent-referred and health line-referred patients (p = 0.33).

Treatment, investigations, disposition and diagnosis

Table 4describes the disposition, treatment, and length of stay for patients in the 3 study groups. In keeping with their greater acuity, physician-referred patients were significantly more likely to have an investigation, receive some treatment, be admitted to hospital and have a longer length of stay. The 5 most frequent discharge diagnoses for the physician-referred patients were fracture, gastroenteritis, abdominal pain, pneumonia and upper respiratory tract infection. For the parent- or self-referred patients, they were viral illness, upper respiratory tract infection, fracture, laceration and croup. For those referred by the health line, the 5 most frequent diagnoses included gastroenteritis, viral illness, asthma, croup and abdominal pain.

Parental views about Telehealth

Seventy-one (55%) of the parents whose children were referred by Telehealth and enrolled in the study completed surveys in the ED. They were asked to report whether they would have come to the ED had it not been recommended by Telehealth. Of these parents, 25 (35.2%) stated they would not have come to the ED, 45 (63.4%) stated they would have come and 1 (1.4%) was unsure. Among the 45 patients who would have come to the ED, 29 (64.4%) visits were deemed appropriate using checklist criteria. Among the 25 parents who would not have visited the ED, 56% of these visits were deemed appropriate using checklist criteria.

Discussion

Telephone health lines provide a potentially valuable service, but concerns have been expressed that they may not reduce unnecessary use of the ED, and may even recommend visits in cases in which the parents called for reassurance, but would not otherwise have visited the ED. We found no significant difference in visit appropriateness between physician-referred, parent- or self-referred, or health line-referred patients using explicit set criteria that gave equal weight to diagnosis, symptoms and parental concerns. However, when one compares physician opinion, triage categories and overall resource use, physician- referred patient visits had the highest level of appropriateness and health line-referred patient visits were no more appropriate than self-referred patient visits. As well, the majority of patients referred by the health line stated that they would have attended the ED had the health line not been available. Interestingly, we identified a large number of patients (74 patients over 4 weeks) who reported being referred by the health line to the ED but for whom no record of referral could be found. Although it is possible that some of the parents who said they had been referred by Telehealth had in fact been referred, and the fax was misplaced or the parents had changed the hospital they planned to visit en route, it is unlikely that this represents a significant number of patients. The majority of these patients presented on weekdays when other sources of care, other than the ED, were available. It is possible that these patients misunderstood the advice from the health line and thus influenced the perception that health line patients are inappropriately referred.

To our knowledge, ours is the first study to compare health line-referred patient visits with both self- or parent-referred, and physician-referred patient visits. Previous studies found levels of appropriateness for health line- referred patients at 78%-90%,5,7,13 which is higher than the 72% found in our study. Similar to our study, these included only patients who were compliant with the urgent care recommendation made by the telephone health service and used a variety of methods to assess appropriateness. For example, one required the examining physician to complete a questionnaire after the patient visit,7 whereas another had several physician reviewers retrospectively apply a combination of set criteria and clinical judgment,13 and the third had a single reviewer retrospectively apply set criteria.5 Although the lower level of appropriateness found in our study may relate to differences in measuring appropriateness, the criteria used in these previous studies did consider a similar combination of historical features, physical examination findings, investigation, disposition and parental concerns. Only 1 of these studies13 compared health line-referred patients with a control group of ED patients (including both physician- and self-referred patients). Unlike our study, they found health line-referred patients were more likely to have been appropriately referred when compared with other ED patients (80.2% v. 60.5%). Interestingly, although registered nurses, who used standardized protocols, staffed all the health lines examined, those in studies other than ours were staffed specifically by pediatric nurses and were affiliated with children's hospitals. Telehealth is staffed by registered nurses, but not specifically by pediatric nurses, and is not affiliated with a children's hospital. It may be that additional pediatric experience is an important factor, as the Telehealth Ontario service is designed for the population at large.

Appropriateness research is fraught with difficulties. Physicians may disagree on which ED visits are necessary,18 nurses and physicians may not agree19 and there may be poor correlation between methods of assessing appropriateness.20 As well, since there is no gold standard,21 the best method to assess appropriateness remains equivocal. Current methods include using prespecified explicit criteria, subjective, implicit criteria (e.g., "Would this patient's outcome have been worse had he or she not been seen for 24 hours?"), some form of acuity staging (e.g., triage category) and examining resource use (e.g., investigations completed, treatments given or disposition). A strength of our study is that we used all these methods to assess appropriateness.

Evaluations of new health care programs are important for understanding, verifying or increasing the impact of the program for the target population. This is particularly important in the case of a nonprofit or government program, as we cannot count on profits to measure client satisfaction or the accountability of the program. Such evaluations can also improve delivery mechanisms resulting in greater efficiency, lower cost and identification of program strengths and weaknesses. One of the stated goals of Telehealth Ontario is to improve "the use of the health care system."2 In our study, most measures of appropriateness did not show that ED visits by health line-referred patients were any more appropriate than those visits by patients who were self- or parent-referred, although they were no less appropriate. Interestingly, during the study period, 3-4 families per day presented with the understanding that they had been referred to the ED by Telehealth, when, indeed, no confirmation of such a referral was received. These findings suggest that clearer communication of appropriate sources of medical care may need to be given and that algorithms used, at least with regard to pediatric patients, may need to be reassessed. It must be acknowledged, however, that this is not an evaluation of all Telehealth patients but only for those referred to the ED. We cannot comment on how many ED visits may have been averted (i.e., families planning on visiting the ED with their child, who chose not to after communicating with Telehealth) and the appropriateness overall of Telehealth triage decisions and advice.

Limitations

There are several limitations to our study. Only about 70% of eligible patients were assessed for appropriateness by the emergency physician. If admission to hospital is considered as a surrogate for the degree of illness, those patients who were not assessed were sicker than those assessed, but this was true across all study groups, so would not have biased the existing comparison. As well, the treating physician may have been aware of the referral route and study purpose and thus been biased in his or her determination of visit appropriateness. We attempted to address this issue by promoting the study as an assessment of appropriateness of ED patient visits in general rather than for those referred in a specific manner. We were also unable to compare the agreement of visit appropriateness between our explicit checklist criteria and physician opinion, given that 30% of patients did not have both measures of appropriateness completed. We also did not capture all health line-referred patients on their initial presentation to the ED. We were, however, able to identify the 27 "missed" health line-referred patients through a review of all health line faxes and thus complete their chart review. We may have also missed some physician-referred patients at registration, but we believe this to be a small number, as billing data show that on average there are 90 physician-referred patients per week at our institution (and we enrolled 346 over 4 weeks).

As with any observational study, there is a theoretical possibility of selection bias that could confound results. In the case of our study, there is a possibility that patients who were referred to the ED by the 3 modes of referral under study could differ if certain groups of patients (or their parents) self-selected into a particular route of referral. Rather than being a limitation of the study per se, this merely reflects the reality of the ED. Moreover, because our study was conducted in a Canadian province that provides access to a universal health care system, patients' routes of referral to the ED should not be confounded by those factors that are associated with access to care in countries without universal systems. It is not surprising that the physician-referred patients were, on many measures, deemed to be the most appropriate visits, as these patients had been recently examined by a physician. However, one would have expected the health line-referred patients to be more appropriate than self-referred patients as they, too, had received some medical screening and advice, but this was not the case. Although slightly fewer health line-referred patients had family physicians than self- referred patients, the self-referred patients reported that they had not sought their family physicians' advice before ED presentation. As well, given the universal health care in Canada, any patient, with or without a family physician, can access walk-in clinics (staffed by physicians) during daytime and evening hours. Consequently, having a regular family physician should not be a factor in the use of a telephone health line service. Given that the health line- referred patients were younger than parent- or self-referred patients, it is possible that the health line staff had difficulty assessing the child's degree of illness, but one could expect such difficulty to be encountered by the parent- of self-referred patients as well.

Conclusion

On the basis of physician opinion, triage categories and overall resource use, physician-referred patients had the highest level of appropriateness, and health line-referred patients were no more appropriate than self- or parent- referred patients. As well, a large number of families reported being referred by the health line to the ED although we could find no evidence of this. Re-evaluation of communication strategies and algorithms may be needed to improve the effectiveness of health line triage, particularly for pediatric concerns.

References

1. Pediatric call centres. Strategies for practice management. Telehealth Care. Grove Village (IL): American Academy of Pediatrics; 1998. Available: www.aap.org/sections/telecare/11_98.pdf (accessed 2006 Mar 1).
2. Ministry of Health and Long-Term Care. Telehealth Ontario. Toronto (ON): Government of Ontario. Available: www.health.gov.on.ca/english/public/program/telehealth/telehealth_mn.html (accessed 2009 Jan 28).
3. Melzer SM, Poole SR. Computerized pediatric telephone triage and advice programs at children's hospitals: operating and financial characteristics. Arch Pediatr Adolesc Med 1999;153: 858-63.
4. Hagan L, Morin D, Lepine R. Evaluation of telenursing outcomes: satisfaction, self-care practices, and cost savings. Public Health Nurs 2000;17:305-13.
5. Poole SR, Schmitt BD, Carruth T, et al. After-hours telephone coverage: the application of an area-wide telephone triage and advice system for pediatric practices. Pediatrics 1993;92:670-9.
6. Lee TJ, Guzy J, Johnson D, et al. Caller satisfaction with after-hours telephone advice: nurse advice service versus on-call pediatricians. Pediatrics 2002;110:865-72.
7. Kempe A, Dempsey C, Whitefield J, et al. Appropriateness of urgent referrals by nurses at a hospital-based pediatric call center. Arch Pediatr Adolesc Med 2000;154:355-60.
8. Kempe A, Dempsey C, Hegarty T, et al. Reducing after-hours referrals by an after-hours call center with second-level physician triage. Pediatrics 2000;106:226-30.
9. Belman S, Chandramouli V, Schmitt BD, et al. An assessment of pediatric after-hours telephone care: a 1-year experience. Arch Pediatr Adolesc Med 2005;159:145-9.
10. Scarfone RJ, Luberti AA, Mistry RD. Outcomes of children referred to an emergency department by an after-hours call center. Pediatr Emerg Care 2004;20:367-72.
11. Kempe A, Bunik M, Ellis J, et al. How safe is triage by an after-hours telephone call center? Pediatrics 2006;118:457-63.
12. Kempe A, Luberti AA, Hertz AR, et al. Delivery of pediatric after-hours care by call centers: a multicenter study of parental perceptions and compliance. Pediatrics 2001;108:E111.
13. Barber JW, King WD, Monroe KW, et al. Evaluation of emergency department referrals by telephone triage. Pediatrics 2000; 105: 819-21.
14. DeAngelis C, Fosarelli P, Duggan AK. Use of the emergency department by children enrolled in a primary care clinic. Pediatr Emerg Care 1985;1:61-5.
15. Berns SD, Linakis JG, Lewander WJ, et al. Appropriate use of a pediatric emergency department: Is the paediatrician called before the visit? Pediatr Emerg Care 1994;10:13-7.
16. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998; 17: 857-72.
17. Cohen J. Statistical power analysis for the behavioral sciences. Philadelphia (PA): Lawrence Erlbaum Associates; 1988.
18. O'Brien GM, Shapiro MJ, Woolard RW, et al. "Inappropriate" emergency department use: a comparison of three methodologies for identification. Acad Emerg Med 1996;3:252-7.
19. Gill JM, Reese CL, Diamond JJ. Disagreement among health care professionals about the urgent care needs of emergency department patients. Ann Emerg Med 1996;28:474-9.
20. O'Brien GM, Shapiro MJ, Fagan MJ, et al. Do internists and emergency physicians agree on the appropriateness of emergency department visits? J Gen Intern Med 1997;12:188-91.
21. Lowe RA, Bindman AB, Ulrich SK, et al. Refusing care to emergency department of patients: evaluation of published triage guidelines. Ann Emerg Med 1994;23:286-93.