Performance criteria for emergency medicine residents: a job analysis

Student Teaching and Evaluation

CJEM 2008;10(6):539-544

Abstract

Résumé

Introduction

Surveys of program directors in multiple specialties have shown that personal interviews are regarded as the most important screening tool in resident selection.1-7 Some of the major roles of admission interviews are to assess candidates' noncognitive skills and compatibility with a residency program.4,7-9 The reliability and validity of structured interviews are consistently reported to be higher than those of unstructured interviews.5,10-15

The content of a structured interview needs to be built around the dimensions of performance that are deemed necessary to succeed in a particular line of work. In other fields, a formal job analysis is conducted to determine these dimensions.5,11,13,16-18 The dimensions of performance that are essential to succeed as a resident in emergency medicine (EM) have not been identified. This study endeavours to determine these essential dimensions by conducting a formal job analysis of the work done by EM residents.

Methods

Study design

In our prospective study, a focus group of EM faculty members performed an analysis of the work of EM residents to extract the essential dimensions of performance.

Study setting and population

The study took place at Queen's University, Kingston, Ont. The study focused on critical incidents involving the resident population enrolled in the 5-year EM training program accredited by the Royal College of Physicians and Surgeons of Canada. At the time of the study, the EM department comprised 20 full-time members and 7 adjunct faculty members.

Study protocol

The critical incident technique as described by Flanagan19 was used to perform the job analysis. In this technique, subject experts reflect on critical incidents. These critical incidents are actual events that revealed successful and unsuccessful behaviours, attitudes and performances.

Five of our full-time faculty members representing different interest and academic orientations volunteered to form the subject expert group for the study. These faculty members all work clinically with EM residents. The principal investigator facilitated a 2-hour focus group discussion with the subject expert group. Participants were asked to reflect on the previous year and recall incidents of extremely good or extremely poor clinical performance involving EM residents. Participants described each incident and commented on the components that made the performance good or poor. The group met until no further incidents could be generated. The entire discussion was audiotaped. After the incidents had been transcribed as descriptions and edited to remove any identifying features, 2 separate faculty members who were not involved in the generation of these incidents but were members of our department reviewed all incidents and labelled each with 1 or several dimensions of performance that they felt to be represented in the incident. The choice of the actual labels was left with these reviewers; no predetermined set of labels was provided. Labels were transposed into their positive aspect (e.g., "lack of confidence" became "confidence"). All labels applied to an incident were then pooled and the list of merged labels for each incident was submitted to the same reviewers to achieve consensus. As recommended by Flanagan,19 any incident for which the 2 reviewers could not agree on a common label was eliminated. The result was a set of critical incidents that reflected the dimensions of performance deemed important for an EM resident to succeed. The study was approved by our Research Ethics Board.

Outcome measures

The outcomes of interest were the labels given by the 2 reviewers. These were accepted as representative of the performance dimensions essential for success as an EM resident at our institution.

Data analysis

A descriptive analysis was performed, with measures of frequency. Each dimension was weighted according to its relative frequency within the set of critical incidents.

Results

The 5 subject experts generated 51 critical incidents during a 2-hour focus group meeting. Ten incidents were removed from the final analysis because the reviewers could not agree on at least 1 common label. Figure 1 provides an example of a scenario where reviewers reached agreement on common labels. Figure 2 represents a scenario where agreement was not attained. A total of 156 performance dimension labels were applied to the remaining 41 incidents (mean of 3.8 labels/incident). Some labels were common to several incidents; the number of distinct labels was 50 (Table 1). The most frequently cited dimension of performance, "professionalism," was reflected in 24 incidents, more than double the number of incidents reflecting the next most frequently cited dimension.

Discussion

Our study presents the essential dimensions of performance for EM residents as identified through a formal job analysis. This type of analysis has been described for other fields of medicine. Gilbart and colleagues20 mailed a questionnaire to program directors and selection committee members of all Canadian orthopedic residency programs asking them to list resident characteristics felt to be essential for success. Although described as a critical incident technique, this method did not involve the generation of critical incidents and was not based on a formal analysis of the work expected of a resident in that specialty. Using Flanagan's technique, Tarico and coworkers21,22 contrasted the essential dimensions of performance for radiology residents from job analyses performed by both the radiology residents and the attending radiologists. Differences were found in the relative emphasis faculty and residents placed on attitudinal characteristics and interpersonal skills in the residency, with higher rankings of these dimensions from residents. Based on these works, multiple points of view may offer a more complete and representative job analysis. Altmaier and colleagues applied the Flanagan technique to multisite radiology23 and anesthesiology24 residency programs. In these studies, faculty members were interviewed individually rather than as a group. Comparison between sites revealed a differential distribution of incidents among performance categories for the anesthesiology program only; the pattern of distribution did not differ between the radiology training sites. The differences seen in the anesthesiology program suggest that the results of a job analysis at one site may not be generalizable to another site.

Two previous articles report on the attributes felt to be desirable for EM residents25 and pediatric EM fellows.26 Table 2 compares these attributes across studies. Bandiera and Regher25 developed a semistructured interview for assessment of candidates to an EM program. The particular attributes that composed the instrument were selected after review of national and international discussion group documents, previous interview protocols and input from the program's residents and faculty members. Poirier and Pruitt26 surveyed 43 pediatric EM program directors with a 42-question, self-administered questionnaire, which listed several factors used in ranking applicants. Respondents rated each factor on a 5-point Likert scale as to its relative importance, 1 being unimportant and 5 being critical. The factors selected were not based on formal analyses but rather adapted from a previous study, which was itself based on several other studies in which the performance criteria were identified through literature reviews and surveys of program directors.27 Balentine and coworkers,28 Crane and Ferraro,29 and Hayden and colleagues30 have also reported on the predictors of success for EM residents. The focus of their studies, however, was on those factors available to the selection committee at the time of application (e.g., medical school attended, dean's letter, EM rotation grades, interview scores) rather than on the personal characteristics of the applicants. Our results show that professionalism far outweighs any other dimension as the most desirable attribute for applicants to our program. Professionalism was also thought to be primordial in Bandiera and Regher's study,25 and each of their 4 interview panels rated each candidate on this particular domain, effectively giving professionalism 4 times the weight of other criteria. Professionalism ranked 12th in Poirier and Pruitt's26 important attributes for pediatric EM fellows, and teamwork ranked first. In our study, teamwork ranked ninth. Some overlap exists between the dimensions expressed by our faculty and those selected in Bandiera and Regher's25 and Poirier and Pruitt's26 studies ("professionalism," "insight," "teamwork"). The consistent ranking of these specific domains suggests that they represent the core performance dimensions shared by faculty in all EM programs. Other dimensions such as "self-confidence," "humility" and "compassion" are unique to specific program assessments, and their relative ranking between programs could signal the specific flavour of a particular program.

Limitations

Our study has several limitations. First, the incidents were generated by a small group of faculty members. Although this small group comprises 25% of our total faculty, their opinion might not represent that of the remaining 75%, or of faculty at other institutions. Second, although a minimum of 50 incidents is reported as acceptable for a job analysis, it might not have been sufficient to saturate the dimensions of performance for EM residents.19 A 2-hour discussion produced 51 incidents and effectively saturated our expert group discussion; repeating the same exercise with another group of faculty might have identified additional desirable dimensions of performance. For complex job situations, Flanagan19 suggests that up to 4000 incidents might be necessary. Third, the total number of individual labels reached 50. Some labels relate very closely to others, such as compassion and caring; collapsing these separate labels into 1 dimension might have given more weight and a higher final ranking to that particular dimension. Some authors have provided their reviewers with a set of predesignated labels.22,23 We chose to integrally apply Flanagan's description of the critical incident technique and not select labels a priori. One could repeat the study choosing the CanMeds roles31 or the ACGME competencies32 as labels. The restricted number of labels would yield higher relative frequencies but might fail to expose important dimensions not captured in the preselected labels. Finally, our results may be inadequately generalizable. As seen in the studies by Altmaier and colleagues,23,24 the relative importance of performance dimensions varies across different training sites, even within a single program. It is conceivable that individual programs need to perform their own job analysis to extract the specific dimensions of performance felt to be essential by their own faculty.

Conclusion

"Professionalism," "self-confidence," "experience" and "knowledge" were identified as the performance dimensions essential to succeed as an EM resident based on our formal job analysis using the critical incident technique. Performing a formal job analysis may assist directors of training programs in developing admission interviews.

References

1. Galazka SS, Kikano GE, Zyzanski S. Methods of recruiting and selecting residents for U.S. family practice residencies. Acad Med 1994;69:304-6.
2. Davis JL, Platt LD, Sandhu M, et al. Evaluating factors in the selection of residents. Acad Med 1995;70:176-7.
3. Provan JL, Cuttress L. Preferences of program directors for evaluation of candidates for postgraduate training. CMAJ 1995;153: 919-23.
4. Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ 1986;61:10-21.
5. Baker HG, Spier MS. The employment interview: guaranteed improvement in reliability. Public Pers Manage 1990;19:85-90.
6. Garden FH, Smith BS. Criteria for selection of physical medicine and rehabilitation residents. A survey of current practices and suggested changes. Am J Phys Med Rehabil 1989;68:123-7.
7. Taylor CA, Weinstein L, Mayhew HE. The process of resident selection: a view from the residency director's desk. Obstet Gynecol 1995;85:299-303.
8. Slone RM. Resident selection: part 3. The interview. Invest Radiol 1991;26:396-9.
9. Johnson EK, Edwards JC. Current practices in admission interviews at U.S. medical schools. Acad Med 1991;66:408-12.
10. Hermelin E, Robertson IT. A critique and standardization of meta-analytic validity coefficients in personnel selection. J Occup Organ Psychol 2001;74:253-77.
11. Patrick LE, Altmaier EM, Kuperman S, et al. A structured interview for medical school admission, phase 1: initial procedures and results. Acad Med 2001;76:66-71.
12. Altmaier EM, Smith WL, O'Halloran CM, et al. The predictive utility of behavior-based interviewing compared with traditional interviewing in the selection of radiology residents. Invest Radiol 1992;27:385-9.
13. Latham GP, Saari LM, Pursell ED, et al. The situational interview. J Appl Psychol 1980;65:422-7.
14. Campion MA, Palmer DK, Campion JE. A review of structure in the selection interview. Person Psychol 1997;50:655-702.
15. Campion MA, Pursell ED, Brown BK. Structured interviewing: raising the psychometric properties of the employment interview. Person Psychol 1988;41:25-42.
16. Lowry PE. The structured interview: An alternative to the assessment center? Public Pers Manage 1994;23:201-15.
17. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med 1990;65:167-77.
18. Latham GP, Sue-Chan C. A meta-analysis of the situational interview: an enumerative review of reasons for its validity. Can Psychol 1999;40:56-67.
19. Flanagan JC. The critical incident technique. Psychol Bull 1954; 51:327-58.
20. Gilbart MK, Cusimano MD, Regehr G. Evaluating surgical resident selection procedures. Am J Surg 2001;181:221-5.
21. Tarico V, Smith WL, Altmaier E, et al. Critical incident interviewing in evaluation of resident performance. Radiology 1984;152:327-9.
22. Tarico VS, Altmaier EM, Smith WL, et al. A resident perspective on the radiology residency. The critical incident technique. Invest Radiol 1986;21:877-80.
23. Altmaier E, Smith WL, Wood P, et al. Cross-institutional stability of behavioral criteria desirable for success in radiology residency. Invest Radiol 1989;24:249-51.
24. Altmaier EM, From RP, Pearson KS, et al. A prospective study to select and evaluate anesthesiology residents: phase I, the critical incident technique. J Clin Anesth 1997;9:629-36.
25. Bandiera G, Regehr G. Reliability of a structured interview scoring instrument for a Canadian postgraduate emergency medicine training program. Acad Emerg Med 2004;11:27-32.
26. Poirier MP, Pruitt CW. Factors used by pediatric emergency medicine program directors to select their fellows. Pediatr Emerg Care 2003;19:157-61.
27. DeLisa JA, Jain SS, Campagnolo DI. Factors used by physical medicine and rehabilitation residency training directors to select their residents. Am J Phys Med Rehabil 1994;73:152-6.
28. Balentine J, Gaeta T, Spevack T. Evaluating applicants to emergency medicine residency programs. J Emerg Med 1999;17: 131-4.
29. Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med 2000;7:54-60.
30. Hayden SR, Hayden M, Gamst A. What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident? Acad Emerg Med 2005;12:206-10.
31. The Royal College of Physicians and Surgeons of Canada. The CanMEDS Physician Competency Framework. Ottawa (ON): The College; 2007. Available: http://rcpsc.medical.org/canmeds (accessed 2008 Sept 22).
32. Accreditation Council for Graduate Medical Education. Outcome Project: Enhancing Residency Education Through Outcomes Assessment. Available: www.acgme.org/outcome/comp/GeneralCompetenciesStandards21307.pdf (accessed 2008 Apr 22).