Impact of a multifaceted pediatric sedation course: self-directed learning versus a formal continuing medical education course to improve knowledge of sedation guidelines

EM Advances

CJEM 2007;9(2):93-100

Abstract

Résumé

Introduction

Over the past decade there has been an increased recognition of the importance of providing safe and effective management of painful procedures in the emergency department (ED). However, children often receive inadequate sedation and pain management due to fear of adverse reactions.1,2 Many of the drugs used for sedation and analgesia have the potential to cause central nervous system, respiratory and cardiac depression.3-5 Adverse outcomes from sedation tend to occur because individuals who have the appropriate airway management and resuscitation skills fail to recognize patients who are developing adverse events and intervene in a timely manner.6 Adequate attention to procedural pain and anxiety will improve quality of care and patient satisfaction by facilitating procedures and minimizing patient suffering. Hoffman and colleagues found that applying a structured model to the practice of pediatric procedural sedation reduced the occurrence of adverse events.7 Although Canadian pediatric emergency physicians have provided sedation for invasive and noninvasive procedures for several years, there has been no formal training or certification for physicians.

Guidelines for pediatric procedural sedation have been established.8-15 Although physicians often report good compliance with guidelines, studies have shown relatively poor understanding of or compliance with them.16-20 Dissemination and implementation of guidelines is more effective when they are locally developed or approved, actively deployed by community-based opinion leaders, have qualities compatible with existing beliefs and values, and can be readily incorporated into current practice.21-24

We established pediatric emergency procedural sedation guidelines for a tertiary care pediatric ED based on existing hospital-wide sedation guidelines and published guidelines.8-15 We then developed an educational intervention to help disseminate and implement these guidelines. The learning needs of our target audience were identified with surveys and focus group discussions. Davis and colleagues demonstrated that multifaceted interventions incorporating 2 or more education strategies more effectively changes physician performance when compared with single interventions. Based on their work, we designed a multi-faceted sedation course.25-27

Although many educational programs have shifted toward self-directed learning, formal continuing medical education (CME) activities rely heavily on traditional expert-led teaching.28 Adult learning principles suggest that self-directed learning is more likely to result in "deep' learning and the development of critical thinking skills.29 However, if the knowledge gap is unperceived, then self-directed learning may not be effective. Emergency physicians who are already involved in procedural sedation without specific institutional guidelines may not be aware of potential gaps in their knowledge and may not perceive a need for further education. The purpose of this study was to compare the effectiveness of a formal procedural sedation course with unstructured, self-directed learning in terms of improvements in physician knowledge of pediatric procedural sedation practices and guidelines.

Methods

Research design

This randomized, post-test, controlled trial was approved by the Hospital for Sick Children Research Ethics Board.

Subjects

Between June and September 2003, we recruited physicians affiliated with the ED at the Hospital for Sick Children to participate in an educational intervention. Full-time and part-time staff emergency physicians as well as physicians with advanced emergency or pediatric training, including pediatric emergency medicine fellows, emergency medicine residents (Royal College of Physicians and Surgeons of Canada, Canadian College of Family Practice) and fourth year pediatric residents, were eligible to participate in the study. Participation was voluntary for all groups. Physicians involved in course development and design, as well as the participants who had not received the study material 2 weeks before the learning assessment, were excluded from the study. All participants completed demographic information questionnaires and provided informed consent.

Interventions

The sedation learning package included detailed learning objectives, a sedation booklet outlining the sedation guidelines, the pharmacology of sedative agents, airway management and complications, as well as a sedation flow sheet, a sedation pocket card and discharge instructions for parents.

The sedation course was developed as a 4-hour, multifaceted teaching program consisting of didactic and small group learning sessions. We used results from focus groups held before the study to develop the intervention and to identify barriers to change. The 2-hour didactic portion encompassed a review of pediatric emergency sedation guidelines, pharmacology of sedative and analgesic agents used in the ED, and a review of potential complications of sedation. The 1.5-hour small group sessions were case-based discussions designed to reinforce and apply the material from the didactic sessions. Enabling methods included the introduction of a procedural sedation record. Members of the Divisions of Pediatric Critical Care, Anesthesiology, and the Drugs and Therapeutics Committee at the Hospital for Sick Children developed a sedation pocket card to reinforce and review lessons. Those attending the course were asked to fill out an evaluation to assess teaching effectiveness and course content.

Using computer generated random numbers, participants were randomly allocated to either a self-study or a formal teaching group. All participants received the sedation learning package with clearly defined learning objectives and were asked to study the material. Those assigned to the self-study group (group 1) received no formal teaching, while those in the formal teaching group (group 2) attended a 4-hour, multifaceted sedation course. Prior to the assessment date, participants were unaware of their assigned group. Both groups took the same 30-item, 45-minute, closed-book multiple-choice examination (Figure 1).

Assessment tool: multiple-choice examination

To ensure content validity of the test items, we used a representative collection of items and a "sensible" method of test construction.30 Test questions corresponding to each objective were composed by course faculty or modified from other published sources on pediatric sedation. There was at least one action item and one concept item question corresponding to each learning objective. The action items corresponded as closely as possible to the practice behaviour that the program was designed to promote. Course faculty members reviewed all the test items. The test was piloted on volunteer, third-year pediatric residents at the tertiary care institution to ensure item difficulty and discrimination. We compared the multiple choice examination scores of the 2 groups.

We conducted a pilot study with the first 9 physicians in each group. We calculated the mean scores and standard deviations (SDs) of the 2 groups to determine the sample size needed to achieve a power of 0.8 with an alpha of 0.05, 2-tailed. Based on this data, only 9 physicians in each group were needed to achieve a statistically significant difference between the 2 groups. However, 24 physicians were enrolled in each group to account for potential drop-out and to obtain adequate evaluation of the course.

Data analysis

We calculated mean, median and range of multiple choice exam scores for both groups. We used a Wilcoxon 2- sample test, as opposed to a a t test, to compare the exam scores of the 2 groups because the assumption of normality was not verified. We also compared exam scores by subgroup (ED staff physicians compared with trainees), by level of subspecialty training (pediatric emergency medicine subspecialty training compared with other subspecialty training or no subspecialty training) and for years of clinical experience in the ED. We used Cohen's d to calculate the effect size.

Results

There were 51 physicians eligible for the study; however, 2 weeks before the assessment, 3 physicians who had agreed to participate had not received their course material and were therefore excluded (see Figure 1). Table 1 shows that there were 24 physicians with similar baseline characteristics in each group. Most were staff emergency physicians (54% in the self-directed group, compared with 62% in the formal teaching group), although a minority had emergency subspecialty training, reflecting the recent emergence of the pediatric emergency medicine subspecialty in Canada. Of participants in both groups, 79% had been in clinical practice for less than 10 years. Both groups indicated that a sedation course would be useful to their practice (100% in the self-directed group, compared with 92% in the formal teaching group) and most participants indicated a need for greater knowledge about pediatric sedation (79% in the self-directed group, compared with 54% in the formal teaching group). Overall, only 8% of participants indicated that self-directed learning was their preferred method of learning.

Table 2 summarizes both groups' examination scores, showing that subjects in the formal teaching group scored significantly higher (p < 0.0001). The median and mean examination scores of participants in the self-study group were 73.3% (range 43.5%-86.6%) and 71.1%, SD 11.6, respectively. The median and mean examination scores of participants in the formal teaching group were 83.3% (range 75.8%-96.5%) and 85.5% SD 5.76, respectively. This is shown graphically in Figure 2. An effect size of 0.6 was calculated from Cohen's d of 1.52, which indicates that the percentage nonoverlap between the 2 groups was 38.2%.

There were no significant exam score differences between staff emergency physicians and trainees (p = 0.83), between physicians with emergency medicine subspecialty training and no subspecialty training or other subspecialty training, or based on years of experience in the ED (see Table 3). There was no significant difference in examination scores based on sex.

Discussion

This study suggests that a multifaceted sedation course was more effective at improving physicians' knowledge of pediatric sedation guidelines and practices than self- directed learning. The success of this sedation program was based on several factors:

• to ensure compatibility with the learning needs of the target audience, prior to the study we completed a careful needs assessment;
• the learning objectives were clearly outlined at the beginning of the course;
• the sedation course was developed using evidence-based educational strategies and was based on adult learning principles;
• the multifaceted intervention applied predisposing methods (including didactic lectures and small group case-based discussions), enabling methods (a sedation flow record) and reinforcing methods (a sedation pocket card for easy reference);
• course material was carefully prepared to ensure it was practical and relevant to the learning needs of the target audience;
• to ensure that practitioners had confidence in the recommendations, we consulted local opinion leaders about guideline development and dissemination processes;
• prior to the study, we used surveys and focus groups to identify increased knowledge of sedation practices and guidelines as important and relevant to the target audience; and
• an active approach to guideline dissemination (i.e., development of a sedation course) ensured that the practitioners had the equipment and skills needed to implement the guidelines.

Physician learning strategies

Increasingly, physicians are being asked to demonstrate clinical competence. Multiple-choice examinations are commonly used as measures of competence for high-risk clinical scenario. Procedural sedation in the ED is considered a high-risk procedure because it has the potential to result in serious adverse events. Courses, such as the Pediatric Life Support Course (PALS) from the American Heart Association and the Advanced Pediatric Life Support Course (APLS) from the American Academy of Pediatrics and American College of Emergency Physicians, support the use of multiple-choice examinations with minimum pass scores to obtain certification.

In addressing the need to increase clinical competence, physicians most often choose to learn by individual reading; however, when the focus involves technical, procedural or communication skills, physicians are more likely to choose formal CME.31 In this study, the formal CME event might have been perceived as a more practical, efficient and effective method of learning highly desirable information. Unlike other CME events that are generally designed to meet the broad needs of individuals, this event was developed based on the specific needs of the target audience. We provided in-depth learning of emergency procedural sedation along with case-based scenarios that were directly applicable to practice. The clinical decision models included options to broaden the applicability of the clinical practice guidelines to individual practice.

Guideline implementation

Improved knowledge of sedation practices and guidelines will ultimately improve the quality of ED care by reducing complications and improving patient satisfaction. However, successful implementation must also include strategies to facilitate the adoption of improved practices.32-35 Ockene and colleagues demonstrated that education alone is insufficient for implementing guidelines.36 Strategies, that enable clinicians to implement the recommended clinical processes and reinforce their use, are needed. Clinical systems and procedures that will aid in this process include patient tracking and provider reminders. Although we introduced measures, such as the procedural sedation record and sedation pocket cards, chart audits and feedback to monitor physician compliance and health care outcomes are currently being evaluated.

Study strengths

The self-study group was randomly selected from the study population and we assumed it was similar to the formal teaching group. Participants were unaware of their group allocation until the day of assessment, at which time they either participated in the course or wrote the multiple-choice examination. As a result, participants in both groups had equal opportunity to review the course material. Participants randomized to take the course all had the same instructors, ensuring that all participants were exposed to the same information during the course. We made every attempt to ensure construct and discriminate validity of the examination. Examination questions were based on the learning objectives and measured knowledge and understanding of sedation guidelines, as well as problem solving and decision making skills in clinical scenarios.

Limitations

The examination was not designed to determine predictive validity, although further studies may determine its effect on future examinations or physician performance. Although competence may not be guaranteed by a multiple-choice examination, it does measure knowledge transfer, which is a prerequisite for professional competence.

We did not ask participants to record how much time they spent reviewing the study material before the assessment. It is conceivable that the self-study group spent less time reviewing the material due to an unperceived knowledge gap. In addition, although it is conceivable that the formal teaching group simply had more knowledge of sedation, we believe that the study participants were a relatively homogeneous group of physicians with experience in the ED and exposure to the same sedation practices.

This study did not analyze the effect of repeated- measurement testing because we did not administer a pre- and post-test to the same individual. A pre-test in the formal teaching group could result in the recall of items in the post-test rather than knowledge gained in the educational intervention. The physicians in this study were from a tertiary care centre and are more accustomed to regular formal teaching sessions. Physicians in the community setting may be more isolated and thus may be more motivated to undertake self-directed learning, which may affect the generalizabililty of our results to other populations.

Despite excellent course evaluations and a significant improvement in physician knowledge of sedation practices following the sedation course, this may not necessarily translate into change in physician behaviour or patient outcomes. Evaluation of behaviour involves the transfer of learning to the workplace. Future studies will focus on change in physician behaviour in adhering to sedation guidelines.

Conclusion

The multifaceted sedation course was more effective in improving physician knowledge and understanding of sedation guidelines and practices than unstructured self- directed learning.

References

1. Petrack EM, Christopher NC, Kriwinsky J. Pain management in the emergency department: patterns of analgesic utilization. Pediatrics 1997;99:711-4.
2. Alexander J, Manno M. Underuse of analgesia in very young pediatric patients with isolated painful injuries. Ann Emerg Med 2003; 42:197-205.
3. Newman DH, Azer MM, Pitetti RD, et al. When is a patient safe for discharge after procedural sedation? The time of adverse effect events in 1367 pediatric procedural sedations. Ann Emerg Med 2003;42:627-35.
4. Malviya S, Voepel-Lewis T, Tait AR. Adverse events and risk factors associated with the sedation of children by non- anesthesiologists. Anesth Analg 1997;85:1207-13.
5. Pitetti R, Singh S, Pierce MC. Safe and efficacious use of procedural sedation and analgesia by non-anesthesiologists in a pediatric emergency department. Arch Pediatr Adolesc Med 2003; 157: 1090-6.
6. Cote CJ, Notterman DA, Karl HW, et al. Adverse sedation events in pediatrics. A critical incident analysis of contributing factors. Pediatrics 2000;105:805-14.
7. Hoffman GM, Nowakowski R, Troshynski TJ, et al. Risk reduction in pediatric procedural sedation by application of an American Academy of Pediatrics/American Society of Anesthesiologists process model. Pediatrics 2002;109:236-43.
8. American College of Emergency Physicians. Clinical policy for procedural sedation and analgesia in the emergency department. Ann Emerg Med 1998;31:663-7.
9. Americian College of Emergency Physicians. Clinical policy: procedural sedation and analgesia in the emergency department. Ann Emerg Med 2005;45:177-96.
10. American Academy of Pediatrics Committee on Drugs. Guidelines for monitoring and management of pediatric patients during and after sedation for diagnostic and therapeutic procedures. Pediatrics 1992;89:1110-5.
11. American Academy of Pediatrics Committee on Drugs. Guidelines for monitoring and management of pediatric patients during and after sedation for diagnostic and therapeutic procedures: addendum. Pediatrics 2002;110:836-8.
12. American Society of Anesthesiologists Task Force on Sedation and Analgesia by Non-Anesthesiologist. Practice guidelines for sedation and analgesia by non-anesthesiologists. Anesthesiology 1996; 84:459-71.
13. Practice guidelines for sedation and analgesia by non-anesthesiologists: a report by the American Society of Anesthesiologists Task Force on Sedation and Analgesia by Non-Anesthesiologists. Anesthesiology 2002;96:1004-17.
14. Innes G, Murphy M, Nijssen-Jordan C, et al. Procedural sedation and analgesia in the emergency department. Canadian consensus guidelines. Can J Emerg Med 1999;17:145-56.
15. Roy R, Griffiths K., eds. Sedation and analgesia guidelines. In: The 2003-2004 Formulary of Drugs. The Hospital for Sick Children. Toronto (ON): The Graphic Centre, HSC; 2003.
16. Lomas J, Anderson GM, Dominick-Pierre K, et al. Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med 1989;321:1306-11.
17. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999;282:1458-65.
18. Davis DA, Taylor-Vaisey A. Translating guidelines into practice: a systematic review of theoretical concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ 1997;157:408-16.
19. Ward MM, Vaughn TE, Uden-Holman T, et al. Physician knowledge, attitudes and practices regarding a widely implemented guideline. J Eval Clin Pract 2002;8:155-62.
20. Worrall G, Chaulk P, Freake D. The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. CMAJ 1997;156:1705-12.
21. Grilli R, Lomas J. Evaluating the message: the relationship between compliance rate and the subject of practice guideline. Med Care 1994;32:202-13.
22. Grol R, Dalhuijsen J, Thomas S, et al. Attributes of clinical guidelines in general practice: observational study. BMJ 1998; 317:858-61.
23. Hayward RSA. Clinical practice guidelines on trial. CMAJ 1997; 156:1725-7.
24. Rogers EM. Lessons for guidelines from the diffusion of innovations. Jt Comm J Qual Improv 1995;21:324-8.
25. Davis DA, Thompson MA, Oxman AD, et al. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274:700-5.
26. Davis DA, Thomson MA, Oxman AD, et al. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268:1111-7.
27. Oxman AD, Thomson MA, Davis DA, et al. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995;153:1423-31.
28. Sectish TC, Floriani V, Badat MC, et al. Continuous professional development: raising the bar for pediatricians. Pediatrics 2002; 110:152-6.
29. Knowles MS. Introduction: the art and science of helping adults learn. In Andragogy in action. Applying modern principles of adult learning. San Francisco (CA): Jossey-Bass; 1984:1-21.
30. Nunnally JC. Validity. In: Psychometric Theory. New York (NY): McGraw-Hill;1978: 87-113.
31. McClaran J, Snell L, Franco E. Type of clinical problem is a determinant of physicians' self-selected learning methods in their practice settings. J Contin Educ Health Prof 1998;18:107-18.
32. Graham ID, Beardall S, Carter AO, et al. The state of the science and art of practice guidelines development, dissemination and evaluation in Canada. J Eval Clin Pract 2003;9:195-202.
33. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993; 342:1317-22.
34. Grimshaw JM, Shirran L, Thomas R, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001; 39(8 Suppl 2):II2-45.
35. Gross PA, Greenfield S, Cretin S, et al. Optimal methods for guideline implementation: conclusions from Leeds Castle meeting. Med Care 2001;39(8 Suppl 2): II85-92.
36. Ockene JK, Zapka JG. Provider education to promote implementation of clinical practice guidelines. Chest 2000;118:S33-9.