Assessing competence in emergency medicine trainees: an overview of effective methodologies

Student Teaching and Evaluation

CJEM 2008;10(4):365-371

Abstract

Résumé

Introduction

Ensuring competence to effectively and safely practise the breadth and depth of emergency medicine (EM) is a critical task for EM educators. However, the idea of the "competent physician" is a dynamic one. One of the most significant changes in medical education in recent years is the adoption of a multidimensional competence model. Kane defines competence as "the degree to which the individual can use the knowledge, skills and judgment associated with the profession to perform effectively in the domain of possible encounters defining the scope of professional practice.^"1

Thus emergency physician competence is 1) based on abilities; 2) derived from a set of domains that define the field of EM; 3) measurable in terms of observable behaviour; and 4) specific to the EM context.2 Competence is currently regarded as a dynamic, evolving status that must be sought and maintained throughout a professional career, rather than a single "milestone" achievement to be attained at a given point in time (e.g., completion of residency).3,4

The multiple dimensions that define competence evolved out of a number of projects in the 1990s, including the Educating Future Physicians for Ontario project5 and later the Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS framework. The CanMEDS roles include Medical Expert (the central role), Communicator, Collaborator, Health Advocate, Manager, Scholar and Professional.6 The College of Family Physicians of Canada (CFPC) has a similar multifaceted approach to competency that parallels these domains in the Four Principles of Family Medicine: Skilled Clinician, Community-Based Discipline, Resource to a Population, and Patient-Physician Relationship.7 These approaches imply that learning and evaluation can be tied to discreet, definable domains of performance that collectively embody the specialty of EM.

Our contemporary era is one of increasing public accountability for all professions.8 Around the world, the medical profession has been buffeted by controversies that threaten professional autonomy.9 Licensing bodies in Canada (e.g., provincial colleges) require the ongoing assurance of competence and are exploring revalidation programs.10 Accreditation standards for residency programs are evolving to mandate the inclusion of both competency-based curricula and associated assessment tools.11 Finally, international medical graduates seeking certification also require assessment of specific competencies to determine suitability for practice in the Canadian setting.12

Recognizing that EM educators should be knowledgeable about the role of competency-based education as well as the contextual nature of effective assessment tools for EM residency programs,2 we performed a detailed MEDLINE literature review encompassing the period from 1950 to January 2007. This search failed to reveal a single article that discussed a global approach to comprehensive competency-based resident assessment in EM. Accordingly, we provide a narrative review of available tools for assessing resident performance specific to the diverse practice and unique environment of EM.

Assessment of competence

Determining an EM resident is competent requires that the multiple domains of EM practice be assessed. Assessment can be defined as the process of collecting, synthesizing and interpreting information to aid decision-making.13 The complexity of global competence in medicine means that no single assessment tool adequately addresses all relevant domains.14 Thus comprehensive assessment of competence requires the use of multiple tools in a longitudinal fashion.

Assessment can take 2 forms: formative and summative. Formative assessment (i.e., the ongoing formation of knowledge, skills and attitudes) is intended to inform both the learner and the teacher about the learner's progress toward the attainment of a certain level of performance (e.g., end-of-shift feedback). Summative assessment (i.e., the summation of knowledge, skills and attitudes), on the other hand, measures whether a certain level of performance has been achieved at a given point in time (e.g., successful completion of rotation). Assessment information can also be the basis for ongoing curricular modifications to meet learner needs and adapt to evolutions in the specialty.15

Assessment tools must be matched to the domain being assessed and be representative of the curriculum blueprint. A curriculum blueprint provides an educational map connecting the goals and objectives of a rotation (or residency) to the instructional methods and the assessment tools.

Selection of the most appropriate tools must take into account the purpose of the assessment (i.e., formative v. summative, high stakes v. low stakes), practicality (e.g., environmental and resource constraints), reliability (i.e., the degree to which an assessment tool generates reproducible results) and validity (i.e., the degree to which an assessment tool measures what it is purported to measure).16,17 Competency-based education is predicated on the principles of achievable objectives and definable standards of performance, both of which must be readily accessible to teachers, learners and assessors. Assessment tools can use a multitude of scoring methods ranging from very concrete (e.g., checklists) to more general overall ratings (e.g., global assessment scores).18

The following sections are meant to assist in the selection of appropriate assessment tools to meet the demands of a competency-based education mandate. Table 1 provides a comparison of commonly used assessment tools, including common strengths, limitations and the psychometric qualities of the tools. Table 2 suggests assessment tools best suited to assess a particular element of a domain of competence. An example behaviour is included to provide context; however, it should be noted that these examples do not cover the entire range of behaviours included within each domain of competence. In Table 2, no ranking of the assessment tools is implied by the order in which they are listed. Other assessment tools not listed may also be adaptable and appropriate to specific behaviours.

Table 1. Comparing commonly used assessment tools*

Tool	Reliability	Validity	Feasibility	Strengths	Limitations
MCQ	+	+	+	· Universal familiarity· Easy to score· Extensive research	· Random guessing· Difficult to write well· Trivial content
Essay	-	+/-	+/-	· Assess problem solving· Easy to compose	· Difficult to score· Inefficient
SAQ	+	+	+	· Easy to write	· Difficult to score
Oral	+/-	+	+/-	· Assess higher order reasoning· Trainee can clarify answers	· Time intensive for faculty· Examiner cueing· Standardization issues
OSCE	+	+	-	· Fidelity· Scheduled· Assess behaviour in critical events	· Deconstructs patient-physician dynamic· Potential scenarios limited· Expensive to run
ITER	-	+	+	· Assess ‘soft’ domains· Flexible· Longitudinal assessment	· Student characteristics influence assessment · Poor discrimination between domains· Reliability issues
MSF	+/-	+	-	· Assess ‘soft’ domains· Interprofessional· Multiple raters	· Time intensive· May affect patient-physician dynamic· Observer training
Portfolios	-	+/-	-	· Self-reflection· Promotes remediation· Longitudinal assessment	· Time intensive· Intentional misrepresentation· Time consuming
Logs	NA	+/-	+	· Increases exposure to technical skills· Tracking	· Assumes repetition equals competence· Time consuming
ITER = in-training evaluation report; MCQ = multiple choice question examination; MSF = multisource feedback; OSCE = objective structured clinical examination; SAQ = short answer question examination. *Adapted with permission from Bandiera G, Sherbino J, Frank JR, editors. The CanMEDS assessment tools handbook: an introductory guide to assessment methods for the CanMEDS competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.18

Table 2. Assessment tools for CanMEDS and the Principles of Family Medicine

Domain of competence	Sample EM behaviours	Potential assessment tools
Medical Expert (CanMEDS) / Skilled Clinician (PFM)	Knowledge content (e.g., list a differential diagnosis of chest pain)	SAQ; MCQ; Oral
	History taking (e.g., take a history from a patient with chest pain)	ITER / DEC; Mini CEX; OSCE
	Physical exam (e.g., perform a cardiovascular examination)	ITER / DEC; Mini CEX; OSCE
	Clinical decision-making (e.g., manage a myocardial infarction)	Oral; OSCE; ITER / DEC; CRR
	Technical skills performance (e.g., place a central line)	ITER; Log; Simulated mode
Communicator (CanMEDS) / Patient-Physician Relationship (PFM)	Patient and family interaction (e.g., establish rapport and explain a diagnosis of myocardial infarction)	Mini CEX; OSCE; MSF; ITER / DEC
	Writing records (e.g., complete an ED chart)	CRR; OSCE; Portfolio
Collaborator (CanMEDS) / Community-Based Discipline (PFM)	Interprofessional interaction (e.g., effectively interact with an RT during an intubation)	MSF; ITER / DEC; HF simulation
	Ability to manage conflict (e.g., secure admission for a patient between 2 disagreeing services)	ITER / DEC; OSCE; Portfolio
Manager (CanMEDS) / Community-Based Discipline (PFM)	Ability to lead (e.g., coordinate nurses in resuscitating a patient)	MSF; ITER / DEC; HF simulation
	Utilization of health care resources (e.g., incorporate cost into the choice of antihypertensives)	CRR; Oral; SAQ; MCQ
	Manage a practice and career (e.g., describe 3 remuneration systems)	Portfolio; MCQ; SAQ
Health Advocate (CanMEDS) / Resource to a Population (PFM)	Describe determinants of health (e.g., describe the influence of poverty on ED visits)	Essay; SAQ; Portfolio; Oral
	Advocate for an individual patient (e.g., secure early follow-up when the circumstances are exceptional)	MSF; ITER / DEC; Portfolio
Scholar (CanMEDS) / Resource to a Population (PFM)	Application of EBM (e.g., critically appraise the literature on fibrinolysis for myocardial infarction)	SAQ; Oral; Portfolio
	Self-directed learning (e.g., research a clinical question related to a patient presentation)	Portfolio; Oral
	Teaching ability (e.g., teach a medical student to suture)	ITER / DEC; MSF; Mini CEX; OSCE
Professional (CanMEDS) / Patient-Physician Relationship	Ethical practice (e.g., respect a patient’s autonomy to refuse a test)	ITER / DEC; MSF
	Ensures a sustainable practice (e.g., describe the negative effects of shift work)	Essay; SAQ; Portfolio; Oral
CEX = clinical examination; CRR = clinical record review; DEC = daily encounter card; EBM = evidence-based medicine; ED = emergency department; EM = emergency medicine; HF = high fidelity; ITER = in-training evaluation report; MCQ = multiple choice questionnaire; MSF = multisource feedback; OSCE = objective structured clinical examination; PFM = Principles of Family Medicine; PtS = patient survey; RT = respiratory therapist; SAQ = short answer question examination.

Assessment tools

Written exams: selected response

Selected-response written exams require the trainee to recognize the correct answer, which is present on the exam page. The term "written" can be misleading, as these exam formats can also be completed at a computer terminal. Selected-response questions include multiple choice, matching item, true or false, and alternate choice.19,20 These exam formats are most appropriate for assessing factual knowledge across multiple domains but can, if properly constructed, also assess problem-solving.21 Selected response exams tend to be easily standardized, easy to mark and amenable to question-banking, but they are difficult to construct because of the sensitive nature of the answer lists.

Written exams: constructed response

Constructed-response written exams require the trainee to recall information in response to a question. Typically, information recall is a more difficult task than recognition of the right answer. Constructed-response formats include essay questions20 and short answer questions.22 Questions tend to be easier to develop and can be banked but are harder to mark given the multitude of ways that learners can express themselves. Exams should be standardized, provide explicit instructions and blank spaces for each answer, and be supported with a marking template developed a priori. Constructed-response exams can be used to assess both factual knowledge and problem-solving related to a number of competencies. For example, a question about an elderly patient who had a fall might focus on injury complications (Medical Expert, Skilled Clinician) or risk factors for a fall (Health Advocate, Resource to a Population).

Oral exams

Standardized oral examination

Standardized oral examinations23 involve real-time dialogue between an examiner and a trainee. A clinical scenario stem is verbally presented and questions probing investigation and management of the case are asked. They require physician examiners who are trained in providing data, questioning the trainee and scoring the responses consistently using structured answer keys. The questions should be realistic and emphasis placed on patient management, rather than in-depth knowledge probing. RCPSC competencies such as Manager, Professional, Scholar and Collaborator, or CCFP principles such as Resource to a Population or Physician-Patient communication can be assessed through skillful development of the question scenario and focused questioning around these areas.

Chart-stimulated recall oral exam

A chart-stimulated oral examination24 differs from a standardized oral examination in that real charts, selected from the resident's clinical experience, serve as the basis for exam questions. This form of examination is more practice-based, as the actual performance of the trainee can be explored.

Observed clinical behaviour

Just as patient care is the cornerstone of EM teaching, direct observation is critical in assessment. Actual (and not simulated or self-reported) behaviour provides the highest fidelity sample for assessment. Direct observation provides the opportunity to make multiple assessments over time and across a breadth of clinical circumstances. Any domain of competence can be assessed to some degree via direct observation.

In-training evaluation reports

An in-training evaluation report (ITER)25 typically consists of a series of scales to assess multiple domains of competence. Multiple observations (from 1 or more observers) of previously observed clinical behaviour inform the global assessment for each domain. An ITER is of particular value in assessing complex, non-Medical Expert competencies. Items on ITERS should reflect the goals and objectives set out for the experience in question and be designed according to commonly accepted guidelines for maximum reliability (i.e., concrete descriptive anchors, a 5-7 point scale with an obvious centre score and feasible extreme options).26-29

Encounter cards

An encounter card (or daily encounter card, DEC) is a small-scale variant of an ITER. Typically containing a scale to assess various domains of competence, the DEC is completed at the end of a clinical shift by a single assessor based on direct observation of the trainee. Providing immediate formative feedback, encounter cards can also be used in a summative fashion to inform the rotation ITER. Some evidence exists that properly designed DECs can provide accurate constructive feedback to trainees, overcoming strong biases among faculty to be lenient or overly positive when documenting trainee performance.29

Mini clinical exam

A mini clinical exam (CEX)30 involves an expert observing a trainee conduct a single clinical skill or procedure in a real-patient encounter. Assessment can be carried out with a variety of tools, including scales and checklists, with or without numerical grades.

Simulation

Simulation is a broad term often loosely applied to a range of assessment modalities, which include low- fidelity models, complex task trainers (high-fidelity devices that mimic a clinical procedure, e.g., bronchoscopy), standardized patients, screen-based computer simulators and computer-driven robotic mannequins.31 Such skills as verbal communication, teamwork, problem-solving, leadership and decision-making can be incorporated into a simulation plan.

Objective structured clinical examination

An objective structured clinical examination (OSCE)32 is a method of assessment where trainees progress through a circuit of constructed stations consisting of standardized patients (SPs), technical procedures, interpretation of clinical investigations (e.g., laboratory tests), etc., in order to sample their abilities in multiple discreet areas.33 This assessment format has been alternatively labelled an OSPRE (objective structured performance-related exam) by some educators. In the classical definition, an OSCE is a circuit of stations only involving skill assessment using SPs. The format can be adapted to multiple competencies and standardized through the use of proper rating tools to allow for reliable assessment of trainees.

Standardized patient examination

An SP examination34 uses actual patients or trained actors to present a history or clinical signs, or both, in a standardized, consistent manner. The trainee's behaviour is assessed by either the SP or an expert rater.

Peer and patient assessment

Multisource feedback

Multisource feedback35 (sometimes referred to as 360° assessment) consists of an aggregate assessment of behaviour based on multiple inputs, from multiple individuals, who form the trainee's sphere of influence, including peer assessment. This assessment technique is appealing to educators wishing to get accurate assessments of teamwork, professionalism and collaboration skills, but requires extensive training of assessors and ongoing monitoring to ensure reliable results. To maintain face validity, each assessor should only comment on areas where they have reasonable expertise and exposure to learner behaviour.

Patient surveys

Patient surveys36 can be used to assess a trainee's skills with respect to history-taking, communication, professionalism and patient advocacy.

Self-assessment

Self-assessment has been suggested to be the first step in self-directed learning, an important practice in EM.37 The issue is best defined by the question, "What aspects of my performance need work?" and not, "How good am I?" The ability of professionals to self-assess is a skill requiring development and, even at the highest levels, is unlikely to provide a completely accurate picture of one's abilities.38,39 Self-assessment, therefore, should complement but not replace other means of assessment in an overall program.

Portfolios

Portfolios40 are a collection of a learner's evidence of the competencies that he or she has achieved over time, and can have an additional component of self-analysis that addresses further required learning. Thus a portfolio consists of selective, objective documents from the individual's clinical practice. A faculty member may mentor the trainee's assessment of existing strengths and weaknesses. Portfolios can address a wide range of competencies. The reflective nature provides insights into a trainees' abilities to self-assess and learn from experiences, both essential qualities of the life-long learning obligation of professionals. If a portfolio only serves as a repository of required submissions and does not include self-reflection, it is not effective as an assessment tool.

Assessment using recorded data

Logs

Logs41 are paper or electronic records of training experiences, such as procedures performed or cases managed. The data are recorded in a real-time, sequential manner. Typically, logs do not have a direct assessment component. Often, assumptions regarding competency are made based on achieving a minimum number of procedures or cases. The cumulative sum technique (CUSUM) is a statistical method that can correlate the number of logged technical procedures with competent performance.42

Clinical record review

A clinical record review43 uses a trained data abstractor to assess a trainee's documentation of clinical encounters (e.g., chart). The abstracted data are compared against standard practice guidelines.

Conclusion

In an era of professional accountability, rigorous training standards and competency-based education, trainee assessment plays a prominent role in medical education. As EM physicians, our role in the assessment of trainees is important to the profession and fundamental to our specialty. No single assessment tool can determine competence. Rather, multiple inputs that use multiple tools in multiple settings and that measure different domains are necessary to ensure competent EM physicians for tomorrow.

There is a paucity of educational research regarding assessment tools in the context of EM. We would encourage this research agenda to assist in the ongoing promotion of excellence in EM training.

References

1. Kane MT. The assessment of professional competence. Eval Health Prof 1992;15:163-82.
2. Epstein RM. Assessment in medical education. N Engl J Med 2007; 356:387-96.
3. Gray JA. Continuing medical education: retooling and renaissance. Lancet 1986;1:1261-2.
4. Leach DC. Competence is a habit. JAMA 2002;287:243-4.
5. Neufeld VR, Maudsley R, Pickering J, et al. Educating future physicians for Ontario. Acad Med 1998;73:1133-48.
6. The CanMEDS Project. Canadian Medical Education Directions for Specialists. Royal College of Physicians and Surgeons, Canada. Available: http://rcpsc.medical.org (accessed 2007 Feb 8).
7. Four Principles of Family Medicine. Available: www.cfpc.ca (accessed 2007 Feb 8).
8. Murray EL, Gruppen P, Catton R, et al. The accountability of clinical education; its definition and assessment. Med Educ 2000; 23:871-9.
9. Irvine D. Doctors in the UK: their new professionalism and its regulatory framework. Lancet 2001;358:1807-10.
10. CPSA. Physician Achievement Review Program. Available: www.cpsa.ab.ca (accessed 2007 Feb 8).
11. Brown N, Parry E, Oyebode F. Appraisal for consultant medical staff. Adv Psychiatr Treat 2003;9:152-8.
12. Alternative routes to certification for IMGs. The Royal College of Physicians and Surgeons of Canada. Available: http://rcpsc.medical.org/residency/certification/img_e.php (accessed 2007 Mar 4).
13. Airasian P. Classroom assessment. 3rd ed. New York (NY): McGraw-Hill; 1997.
14. Epstein RM, Hundert E. Defining and assessing professional competence. JAMA 2002;287:226-34.
15. Leinster S. Assessment in medical training. Lancet 2003; 362: 1770.
16. Abrahamson S, Nyquist J. Deciding how to evaluate competence. In: Lloyd J, Langsely D, Editors. How to evaluate residents. Chicago (Il): American Board of Medical Specialties; 1986. p. 45-56.
17. Wass V, Van der Vleuten C, Shatzer J, et al. Assessment of clinical competence. Lancet 2001;357:945-9.
18. Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.
19. Wood T, Cole G. Developing multiple choice questions for the RCPSC certification examinations. Ottawa (ON): Office of Education, Royal College of Physicians and Surgeons of Canada; 2004.
20. Downing S. Assessment of knowledge with written test forms, in international handbook of research in medical education. In: Norman G, van der Vleuten C, Newble D, Editors. Dordrecht (Holland): Kluwer Academic Publishers; 2002. p. 647-72.
21. Coderre SP, Harasym P, Mandin H, et al. The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts. BMC Med Educ 2004;4:23.
22. Short Answer Questions. Guidelines for their development. Educational Research and Development Unit. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2000.
23. Assessing clinical reasoning: the oral examination and alternative methods. In Mancall E, Bashook P, Editors. Evanston (IL): American Board of Medical Specialties; 1995.
24. Cunnington JP, Hanna E, Turnhbull J, et al. Defensible assessment of the competency of the practicing physician. Acad Med 1997;72:9-12.
25. Turnbull J, Van Barneveld C. Assessment of clinical performance: in-training evaluation, in international handbook of research in medical education. In: Norman G, van der Vleuten C, Editors. Dordrecht (Holland): Kluwer Academic Publishers; 2002. p. 793-810.
26. Gray JD. Global rating scales in residency education. Acad Med 1996; 71(Suppl):S55-63.
27. Streiner DL. Global Rating Scales. In: Neufeld VR, Norman GR, Editors. Assessing Clinical Competence. New York (NY): Springer Verlag; 1985. p. 119-41.
28. Bandiera GW, Morrison LJ, Regehr G. Predictive validity of the Global Assessment Form used in a final-year undergraduate rotation in emergency medicine. Acad Emerg Med 2002;9:889-95.
29. Weaver CS, Humbert AJ, Besinger BR, et al. A more explicit grading scale decreases grade inflation in a clinical clerkship. Acad Emerg Med 2007;14:283-6.
30. Norcini JJ, Blank L, Duffy F, et al. The mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003;138:476-81.
31. Ziv A, Wolpe P, Small S, et al. Simulation-based medical education: an ethical imperative. Acad Med 2003;78:783-8.
32. Lee C. Objective structured clinical examinations (OSCEs) and objective structured performance-related examinations (OSPREs). In: Bandiera G, Sherbino J, Frank J, Editors. The CanMEDS Assessment Tools Handbook: an introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006. p. 23-5.
33. Hodges BD. The objective structured clinical examination: three decades of development. J Vet Med Educ 2006;33:571-7.
34. van der Vleuten C, Swanson D. Assessment of clinical skills with standardized patients: state of the art. Teach Learn Med 1990; 2:58-76.
35. Rodgers KG, Manifold C. 360-degree feedback: possibilities for assessment of the ACGME core competencies for emergency medicine residents. Acad Emerg Med 2002;9:1300-4.
36. Matthews D, Feinstein A. A new instrument for patients' ratings of physician performance in the hospital setting. J Gen Intern Med 1989;4:4-22.
37. Parboosingh J. Role of self-assessment in identification of learning needs. J Contin Educ Health Prof 1998;18:213-9.
38. Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 2005; 80(Suppl): S46-54.
39. Regehr G, Eva K. Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res 2006;449:34-8.
40. O'Sullivan P, Greene C. Portfolios: possibilities for addressing emergency medicine resident competencies. Acad Emerg Med 2002; 9:1305-9.
41. Hunskaar S, Heim SH. Asessment of student's experiences in technical procedures in a medical clerkship. Med Educ 1983; 17: 300-9.
42. Lim TO, Soraya A, Ding L, et al. Assessing doctors' competence: application of CUSUM technique in monitoring doctors' performance. Int J Qual Health Care 2002;14:251-8.
43. Tugwell P, Dok C. Medical record review. In: Berk R, editor. Performance assessment: methods and applications. Baltimore (MD): Johns Hopkins University; 1986. p. 51-70.