Daily evaluation cards for trainees: "Make it so"

Commentary

CJEM 2001;3(3):228

See related Education article: McGraw & Verma

In this issue McGraw and Verma¹ discuss the challenges associated with the "trainee in difficulty." As they rightly point out, we find it extremely difficult to objectively identify and provide feedback on poor performance, so it often goes unreported. In emergency medicine, trainees typically work only once or twice with multiple preceptors, so a pattern of difficulty may not be noticed or reported. The result may be a trainee who reaches the end of his or her training with a long history of difficulty, none of which has been clearly documented and for which he or she has received little feedback or specific help. The trainee has "slipped through the cracks." This is a multi-faceted problem, and the solutions are similarly complex.

One element of the solution -- daily evaluation cards -- is mentioned only briefly by McGraw and Verma. This simple but powerful tool merits more attention. It is probably the key to successful evaluation of clinical performance of all trainees in the emergency department, whether they are in difficulty or not. This brief commentary describes why and how you should set up a daily evaluation system in your emergency department.

Although there are anecdotal reports of wide use of daily evaluations in emergency medicine training programs across Canada, none of this experience has yet appeared in the literature. The most useful article comes from the anesthesia experience. This is perhaps not surprising, as they share with emergency medicine the characteristic of direct and immediate supervision, in theory at least, of trainee activity. Rhoton described the use of unstructured index cards to gather comments (without any rating scales or categories) on resident performance on a daily basis.² Analysis of the comments over the whole training period provided an extremely powerful quantitative predictor of overall clinical competence, much better than any of the traditional in-training objective tests. The comments were extremely useful in adapting subsequent learning to specifically identified strengths and weaknesses. In a later study, Rhoton demonstrated that comments on noncognitive skills, rather than on knowledge, were the most powerful independent predictors of overall performance.³ For the trainee in difficulty this is exactly what we need.

How do you set up such a system, and what are the keys to success?

1. How many cards are necessary?
You should have a card for every shift (with the names of the trainee and preceptor, and the date), even though not all will be returned. In Rhoton's experience 75% were returned, and only 25% of these had comments. In our centre trainees are given one card for each ED shift in their rotation, and must return at least 75% of them with comments if they wish to get credit for the rotation. Compliance is excellent.

2. Who manages the cards?
The trainees do it best. They have a vested interest in getting the cards filled in and will prompt their preceptors to do it at the pertinent time. Departmental policy should expect preceptors to fill in a card for every shift, except in unusual circumstances. The trainees keep the cards and give them to the departmental office toward the end of the rotation. They are kept on file and used as necessary for documentation in any subsequent evaluation discussions. We use the comments on the cards in a qualitative fashion; quantitative analysis can be done, though we do none at this time.

3. What form should the card take, and what should be on it?
This is perhaps the point of greatest variance around the country. At one extreme we find the full-size sheet of paper, covered on both sides with categories and scales and, at the other, a 4x6 index card with nothing on it except the names and date. Format and structure actually matter much less than the fact that a card exists and is used on a daily basis. Having said this, there is good evidence that case-specific comments, both positive and negative, are much more useful than broad skill categories and normative scales. It is easier to give and to take constructive criticism when it is based on recent concrete examples -- we all have "bad days" or missed cases from time to time -- than to pass a global judgement on a trainee based on a very limited sample of his or her work. Negative feedback is still difficult to give, however. It is helpful to have a box or category on the card, which can be ticked easily and which indicates in a diplomatic way that performance in a specified area is below an expected level.

Index cards are a good format to start with -- they travel well in pockets, are easy to handle and file well. I would use an open format card, with the names and date and only a few general headings (cases seen; positive comments; problems noted or things that need improvement) and one box to indicate a sub-par performance in a particular domain. Traditional evaluation categories and scales can be added, as long as you leave space for case-specific comments as well.

4. When should the cards be filled in, and how longdoes it take?
Cards are best filled in at the end of a shift. The trainee lists cases seen with you, then you spend 5 to 10 minutes making and writing comments and suggestions on the work that was done. That is all the time it takes. One reviewer of McGraw and Verma's article stated that he agreed in principle with daily structured feedback but in his department it is a practically unattainable goal. This suggested to me, on the contrary, that daily evaluation cards are exactly what his department needs: they encourage trainee/preceptor interaction, the feedback generated is high quality (specific, immediate, constructive), they provide clear documentation of trainee performance, they are managed by the trainees, and it only takes 10 minutes of the preceptor's time. If this cannot be integrated into the clinical activities of even the busiest ED then I think we should seriously question the pertinence of a trainee's presence in this department.

5. How are the cards used for subsequent summative evaluations?
Although the daily cards are most useful as a formative evaluation tool, for the trainee in difficulty they are also particularly useful for summative evaluation, and for subsequent training and promotion decisions. Occasional difficulties are normal for any trainee, but repeated difficulties from multiple situations with multiple preceptors are a serious sign. Daily evaluation cards permit you to clearly identify and document such a pattern, and to plan appropriate action.

To summarize, daily evaluation cards are an extremely effective evaluation and teaching tool in busy EDs. They are simpler to use than you might expect and usually permit you do to a better job as a preceptor, in the same time as before. The structure and format of the cards are less important than the simple fact of having any card at all, although space for case-specific comments is essential. If you are not using them you should be. Talk to your colleagues, and make it so!

References

1. McGraw R, Verma S. The trainee in difficulty. CJEM 2001;3(3):205-8.
2. Rhoton MF. A new method to evaluate clinical performance and critical incidents in anesthesia: quantification of daily comments by teachers. Med Ed 1989;23:280-9.
3. Rhoton MF, Barnes A, Flashburg M, Ronai A, Springman S. Influence of anesthesiology residents' noncognitive skills on the occurrence of critical incidents and the residents' overall clinical performances. Acad Med 1991;66:359-61.