Self-reported experience and competence in core procedures among Canadian pediatric emergency medicine fellowship trainees

Student Teaching and Evaluation

CJEM 2008;10(6):533-538

Abstract

Résumé

Introduction

Pediatric emergency medicine (PEM) is a rapidly evolving subspecialty in Canada, the United States, Australia and the United Kingdom.1 A PEM practitioner requires expertise in specific procedures; however, guidelines pertaining to the nature of training necessary to acquire these skills have not been created in Canada. The PEM practitioner is further challenged in acquiring the necessary skills because many involve procedures that are rarely performed on pediatric patients.

The Royal College of Physicians and Surgeons of Canada oversees all Canadian training programs in PEM. The college recognized PEM as a subspecialty in 2000. To acquire PEM certification, a trainee must complete a 2-year postgraduate fellowship program in PEM after completing either the FRCP emergency medicine residency program or a pediatrics residency program. A mandatory PEM fellowship examination commenced in the fall of 2008; however, procedural proficiency is not expected to be a component of the evaluation.2

Within pediatrics there are few standardized, objective, validated, uniformly accepted methods of assessing competence in specific procedures. A list of 125 competencies for PEM was recently published by the college.2 Researchers evaluating large lists of procedural skills in other countries found many of the skills to be inappropriate; some skills were deemed better acquired in medical school, some did not reflect the current standard of care and others involved procedures that were too infrequently performed to establish or maintain competence during training.3 A PEM curriculum subcommittee in the United States evaluated a list of 120 skills. Graduates were found to have limited exposure to 49 PEM skills and no exposure to 22 PEM skills. The authors of the study also found that the skills recorded in documented lists were not appropriately maintained by new graduates.4

To determine the current state of PEM training in Canada, as well as the directions for future training, we sought to ascertain the frequency with which fellows in accredited Canadian PEM fellowships perform specific procedures, the level of confidence fellows have in their abilities and whether there are differences in self-perceived success between first- and second-year fellows.

Methods

We reviewed a list of 120 core competences identified by the 1993 curriculum subcommittee for PEM in the United States.5 A focus group consisting of the Postgraduate Education Committee for PEM fellowship at the Children's Hospital at the London Health Sciences Centre identified 24 PEM procedures perceived to be commonly performed and to have more relative importance in the acute resuscitation setting. The list of these 24 PEM procedures was then distributed to the PEM consultants at the Children's Hospital for review and feedback. A survey was then created to determine the number of times each of the 24 procedures were performed by fellows within the past 12 months and within the past 3 years (Box 1). The survey also addressed the confidence of the respondents in successfully performing each procedure. The survey included demographic data as well, such as age, sex, year of training and certifications.

Approval was obtained from the Children's Hospital research ethics board, which considered completion of the survey an implication of consent. The survey was pilot tested on 15 fellows from across Canada who were attending a PEM conference in 2005. Based on feedback obtained from the pilot group, we revised the survey and then redistributed it by mail in April 2005 to all 46 PEM fellows in accredited PEM fellowship programs across Canada using a modified Dillman technique.6 Reminders were sent out twice in 2-week intervals after the initial survey.

Data were analyzed in SPSS version 14.0 (SPSS Inc.). Means and standard deviations (SDs) were used to report normally distributed continuous variables. The median and range was used to report skewed continuous variables. Categorical variables were expressed as percentages. The Mann-Whitney test was used to compare differences in skewed continuous outcomes between first- and second-year fellows. P < 0.05 was considered statistically significant.

Results

Of the 46 surveys sent to PEM fellows, 32 (70%) were returned. The demographic characteristics of the respondents are listed in Table 1. The mean age of respondents was 31.3 (SD 2.3) years and 68.8% of respondents were male. Most respondents were in their second year of training and had prior training in pediatrics.

The 24 procedures included in the survey are reported in Table 2. Closed reduction of fractures, followed by peripheral intravenous insertion, complex laceration repair and endotracheal intubation where the most commonly performed procedures. Second-year fellows reported performing significantly more upper airway foreign-body removals, oropharyngeal/nasopharyngeal airway insertions and intraosseous line insertion procedures than did the first-year fellows (p < 0.05).

Perceived success rates for the 24 procedures are reported in Table 3. The success rate was highest for oropharyngeal/nasopharyngeal airway insertion: second-year fellows reported a median success rate of 100% (range 90%-100%) and first-year fellows reported a success rate of 92.5% (range 0%-100%, p = 0.01). Second-year fellows also reported a significantly higher median success rate for intraosseous line insertion at 95.0% (range 80%-100%) compared with 80% for first-year fellows (range 0%-100%, p < 0.001).

Discussion

PEM requires both medical and procedural expertise, yet skill competency is expected to be part of the formal examination of graduates in the current curriculum of the Royal College of Physicians and Surgeons of Canada. All PEM graduates must have a signed "Final in training evaluation report" form that evaluates procedural skills that were observed during training in order to take the exam.

The documentation and evaluation of procedure proficiency is an evolving area in the educational literature. Many subspecialties, including internal medicine and surgery, have undertaken the evaluation of procedural skills as part of their training requirements.7 Within pediatrics there are few standardized, objective, validated and uniformly accepted methods of assessing competence in technical skills.8

Current US emergency medicine trainees are required to track procedures performed in a procedure log. However, there are no minimum numbers of procedures required for board certification.9,10 In Australia, emergency medicine trainees are required to complete a pediatric log book which details their experience with pediatric patients and procedures.11

Regardless of training requirements set by governing bodies, the actual effect of program size, program type and year of training on procedural skill acquisition has not been extensively studied. Paul and King12 evaluated US trainees for the effect of various residency and hospital settings to determine the factors that influenced the range of skills performed. The authors concluded that there were few statistically significant differences in procedure experience between the different program formats and that there were large variations in the type and number of procedures recorded by emergency medicine programs.

Furthermore, there is little consensus regarding a standardized list of skills that would be appropriate to evaluate. Farion and Morrison3 attempted to refine an emergency medicine procedure skill list by undertaking a national survey to determine procedural frequency and self-assessment of competence. They undertook a rigorous 3-step approach to define a "core" group of 127 procedures that were appropriate for postgraduate acquisition and for emergency medicine certification. These were based on the endorsement of actively practising emergency physicians certified by the Royal College of Physicians and Surgeons of Canada. The authors noted that many previously defined procedural skills were inappropriate because some skills were better acquired in medical school, some skills did not reflect the current standard of care and other procedures were too infrequently performed to establish or maintain competence during training.

In 1993, a large list of 120 technical skills was published by a curriculum subcommittee for PEM in the United States, and a study evaluating 63 graduates was performed. The study had a survey completion rate of 51% and the authors found that of 117 skills analyzed, graduates had limited exposure to 49 skills and no exposure to 22 skills. The authors of the study also found that the skills recorded in documented lists were not being maintained by these new graduates.4

We are not aware of any previously published studies of a procedural skill list for PEM within a training program setting. We felt that the first step to establishing such a list was to survey the trainees in PEM to determine their perceived success rates and frequency of exposure to procedures. Since the completion of our survey, the Royal College of Physicians and Surgeons Subspecialty Standards in Pediatric Emergency Medicinepublished a comprehensive list specifying the procedural standards in the objectives of training in 2007.2This list is composed of 125 items ranging from the interpretation of radiographs to the placement of central lines.

The acquisition of procedural skills occurs in many ways, often at various stages of one's career. Nevertheless, the acquisition of core life-saving skills deemed necessary for PEM should be acquired primarily in thefellowshiptraining program.

It could be argued that skills acquisition is often obtained through different modalities including didactic lectures, skills stations, animal laboratories and model simulation. There have been numerous studies reporting novel procedural skill techniques involving animal laboratory settings and computer-based technologies.13-17 It is noteworthy that the 5 procedures we identified in which trainees felt least competent were also among the 8 least frequently encountered procedures. The development of novel techniques for the acquisition of these particular skills should be an important priority for PEM.

For some procedures we found a wide range of results among respondents. For example, exposure to intravenous insertion and dislocation reductions varied widely. The variation in exposure to these skills may be related to a number of factors including the trainees' elective rotations (orthopedics and anesthesia may allow for greater or lesser exposure), the frequency with which these procedures are carried out in their local department and who primarily performs the procedures. Further research in this area may lead to opportunities for improving exposure to infrequent PEM procedures. Our results suggest that training programs need to ensure both the teaching of procedures and assessment of learners' skills.17-20

Limitations and future questions

Our study had a number of limitations, in large part because it was a retrospective survey. Although we obtained a response rate of 70%, the sample size of just 32 PEM fellows may limit generalizability. In addition, the reported frequency of procedure performance was based only on recall, which was impossible to verify objectively, thus global estimates were subject to all the vagaries of memory over time and recall bias. Although our findings are a starting point, these data do not necessarily translate directly into the current state of PEM training or skill competence in Canada.

The 24 procedural skills we surveyed were developed as a starting point for a discussion of this aspect of PEM training. There was no systematic review leading to agreement as to the "core" competences with regard to procedural skills within PEM. In addition, a list of 125 competences defined by the Royal College of Physicians and Surgeons of Canada now exists.

The scope of this study was narrow: it was limited to Canadian fellows undertaking pediatric fellowship training at the time of this survey. Therefore, its relevance and generalizability to training programs in other countries may be limited.

Further research is required to establish the core group of competences for the specialty of PEM. Moreover, the frequency and type of exposure necessary for competence in each procedure should be defined.

Conclusion

In surveying PEM trainees across Canada, we have described the frequency and self-perceived success rate for 24 important procedures. This information may be helpful for program directors in evaluating future directions and opportunities for training of their PEM trainees.

References

1. Babl FE, Weiner DL, Bhanji F, et al. Advanced training in pediatric emergency medicine in the United States, Canada, United Kingdom, and Australia: an international comparison and resources guide. Ann Emerg Med 2005;45:269-75.
2. Royal College of Physicians and Surgeons of Canada. Information by specialty or subspecialty. Ottawa (ON): The College; 2008. Available: http://rcpsc.medical.org/information/index.php?specialty=462&submit=Select (accessed 2008 Sept 29).
3. Farion K, Morrison L. Redefining emergency medicine procedures: Canadian competence and frequency survey. Acad Emerg Med 2001;8:731-8.
4. Hayden SR, Panacek EA. Procedural competency in emergency medicine. Acad Emerg Med 1999;6:728-35.
5. Pediatric Emergency Medicine (PEM) Fellowship Curriculum Statement. Curriculum Subcommittee, Section of Emergency Medicine, American Academy of Pediatrics. Pediatr Emerg Care 1993; 9:60-6.
6. Dillman DA. Mail and telephone surveys: the total design method. Washington (DC): John Wiley and Sons; 1978.
7. American Board of Internal Medicine. Guide to evaluation of residents in internal medicine, 1991-1992. Philadelphia (PA): The Board; 1991.
8. Quan L, Shugerman R, Kunkel N, et al. Evaluation of resuscitation skills in new residents before and after pediatric advanced life support course. Pediatrics 2001;108:e110.
9. Resident Review Committee for Emergency Medicine. Program information forms. Chicago IL: American Board of Emergency Medicine, 1995.
10. Task Force on the Core Content for Emergency Medicine Revision. Core content for emergency medicine. Acad Emerg Med 1997;4:628-42.
11. Australasia College for Emergency Medicine. Handbook of training in emergency medicine. Victoria (AU): The College; 2007.
12. Paul RI, King L. Technical skills experiences in pediatric emergency medicine fellowship programs. Pediatr Emerg Care 1996;12:10-2.
13. Olshaker JS, Brown CK, Arthur DC, et al. Animal procedure laboratory surveys: use of the animal laboratory to improve physician confidence and ability. J Emerg Med 1989;7:593-7.
14. Sternbach GL, Rosen P. Use of laboratory animals in the teaching of emergency procedures. JACEP 1977;6:543-5.
15. Chapman DM, Marx JA, Honigman B, et al. Emergency thoracotomy: comparison of medical students, resident and faculty performance on written, computer and animal-model assessments. Acad Emerg Med 1994;1:373-81.
16. Chapman DM. Use of computer-based technologies in teaching emergency procedural skills. Acad Emerg Med 1994;1:404-7.
17. Chapman DM, Marx JA, Honigman B, et al. Final report: computer-based teaching and evaluation of critical emergency medicine procedure skills. Dallas (TX): Emergency Medicine Foundation; 1990.
18. Frank J, Mann K, Neely E. Portfolios and logbooks. In: Bandiera G, Sherbino J, Frank JR. The CanMEDS assessment tools handbook. An introductory guide to assessment methods for the CanMEDS competencies. Ottawa (ON): The Royal College of Physicians and Surgeons of Canada; 2006. p. 32-5.
19. Penciner R, Siddiqui S, Lee S. Emergency medicine clerkship encounter and procedure logging using handheld computers. Acad Emerg Med 2007;14:727-31.
20. Kovacs G. Procedural skills in medicine: linking theory to practice. J Emerg Med 1997;15:387-91.