Emergency medicine: A practice prone to error?

ED Administration

Pat Croskerry, MD;* Douglas Sinclair, MD†

*Department of Emergency Medicine, Dartmouth General Hospital Site, Capital District Health Authority, Dartmouth, NS, and Department of Emergency Medicine, Dalhousie University, Halifax, NS †Director of Emergency Services, Department of Emergency Medicine, Queen Elizabeth II Health Sciences Centre, Halifax, NS

CJEM 2001;3(4):271-276

Abstract

The last decade has witnessed a rapidly growing public and academic interest in medical error, an interest that has culminated in the emergence of the science of error prevention in health care. The impact of this new science will be felt in all areas of medicine but perhaps especially in emergency medicine (EM). The emergency department's unique operating characteristics make it a natural laboratory for the study of error. These characteristics, combined with the complex and myriad activities of EM, predict vulnerability to a multitude of errors. Overcrowding and other resource limitations impair continuous quality improvement, and many errors result from high decision density, excessive cognitive load and flawed thinking in the decision-making process. A large proportion of these errors have serious outcomes but an even higher proportion are preventable.

The historical practice of blaming individuals for errors needs to be replaced by root-cause analysis that identifies process and systemic weaknesses. Quantitative and qualitative methods are needed to detect, describe and classify error at all levels in the system. Research is needed into the processes that underlie EM error. Educational initiatives should be developed at all levels, for everyone from undergraduate trainees to practicing emergency physicians. Changes in societal attitudes will be an important component of the new culture of patient safety.

A nationwide reporting system is proposed to disseminate error information expediently. Canadian EM providers are in a pivotal position to provide leadership to the Canadian health care system in this important area.

Résumé

Au cours de la dernière décennie, la question de l'erreur médicale a suscité un intérêt grandissant parmi le grand public et le milieu universitaire. Cet intérêt s'est traduit par l'émergence de la science de la prévention de l'erreur dans les soins de santé. L'impact de cette nouvelle science sera ressenti dans tous les domaines de la médecine, mais peut-être plus particulièrement en médecine d'urgence (MU). Les caractéristiques de fonctionnement uniques du département d'urgence en font un laboratoire naturel pour l'étude de l'erreur. Ces caractéristiques, combinées aux activités complexes et diversifiées de la MU, rendent le département d'urgence particulièrement vulnérable à une multitude d'erreurs. L'encombrement et autres limitations des ressources entravent l'amélioration continue de la qualité et de nombreuses erreurs sont le résultat d'un volume décisionnaire important, d'une charge cognitive excessive et de raisonnements imparfaits dans le processus de prise de décision. Une grande proportion de ces erreurs entraîne des résultats graves, mais une proportion encore plus grande est évitable.

L'attitude traditionnelle selon laquelle les individus sont blâmée pour les erreurs commises doit être remplacée par une analyse des causes premières de ces erreurs qui identifie les faiblesses au niveau des processus et du système. Des méthodes quantitatives et qualitatives sont nécessaires pour déceler, décrire et classifier les erreurs à tous les niveaux du système. On doit effectuer des recherches sur les processus qui sont à la base des erreurs en MU. Des initiatives éducatives doivent être mises sur pied à tous les niveaux, autant pour les étudiants du niveau post-universitaire que pour les médecins d'urgence en pratique active. Un changement sociétal d'attitude constituera une composante importante de cette nouvelle philosophie axée sur la sécurité des patients.

Un système d'information à l'échelle nationale est proposé visant à disséminer de manière opportune l'information au sujet des erreurs. Les dispensateurs de soins dans les urgences canadiennes occupent une position privilégiée pour agir à titre de chefs de file au sein du système de santé canadien dans ce domaine important.

Introduction

Over the past 15 years there has been a gradual unmasking of the nature and scope of error in medicine. What began as a trickle of reports in the 1940s and 1950s increased almost exponentially toward the turn of the century.1 In recent years, the topic has received in-depth treatment in major works originating both inside2-5 and outside6-11 the medical profession. The scale of the problem is now being appreciated. Conservative estimates suggest that, in US hospitals, medical error accounts for 1 million injuries and 100 000 deaths per annum.12 The total cost of medical error in the US has been estimated at up to US$50 billion.12

Almost 40 years ago, US and Canadian reports suggested that 20% to 24% of hospital inpatients suffered iatrogenic injuries.13,14 Subsequently, larger studies have consistently reported AE rates from 3% to 12%,15-17 ranging as high as 16%.18 The variation is presumably due to differences in study methodology, health care systems, local practice patterns and other variables.

The term "iatrogenic error" has been used historically to describe medical error, whereas the newer and more appropriate term, "comiogenic error,"8 refers to errors arising at any point in the patient care continuum, involving physicians, nurses, technical staff, administrative personnel or others. AE is generally defined as an unintended injury caused by medical management that results in prolonged hospital stay, temporary or permanent disability or death. The term "negligence" has a variety of usages and implications; in the context of medical malpractice it derives from the theory of tort law. A preferable term when discussing error theory is "preventable error," which will be used here. Clearly, not all AEs are preventable, not all are due to error, not all errors lead to AEs, and relatively few are detected as AEs.

The medical profession has always tacitly acknowledged medical error,13,14 but its prevalence was not widely appreciated until publication of the landmark Harvard Medical Practice Study.16 This retrospective, randomized chart review covered more than 30 000 discharges from 51 New York State acute care hospitals in 1984. Adverse events occurred in 3.7% of the cases. More than half were due to errors and were theoretically preventable. Approximately 25% were attributed to negligence, defined as "care that fell below the standard expected of physicians in their community." Most of the resulting disabilities resolved within 6 months, but 2.6% were permanent and 13.6% were fatal. Extrapolated to a national level, these data suggest that, at that time, 98 000 deaths a year could be attributed to preventable medical error in the United States alone (on the basis of the 1984 population of 236 million).

Three further studies have corroborated these findings. In a smaller 1992 study, AEs were recorded in 2.9% of patients, with 6.6% of these leading to death.17 In 1995, Australian researchers used methods similar to those of the Harvard study, documenting permanent disability in 13.7% of patients and death in 4.9%.18 Again, half of all AEs were considered preventable. Extrapolating their findings, the authors estimated that AEs caused 180 000 deaths a year in a country of only 15 million people -- a rate considerably higher than that for the United States. Using similar methodology, a recent preliminary study of 2 acute-care hospitals in Greater London, UK, found an AE rate of almost 12%; one-third of these caused moderate disability or death, and about one-half were considered preventable.19

Growing concern about patient safety led the American Medical Association to establish the National Patient Safety Foundation in 1997. The next year, the Institute of Medicine initiated the Quality of Health Care in America project. The first publication arising from that project, To Err is Human: Building a Safer Health System,12 outlined the nature and extent of the problem and was a first step toward dismantling the culture of blame associated with error. Further momentum was gained when the British Medical Journal devoted its issue of 21 March 2000 to medical error and patient safety.

If the data discussed above can be extrapolated to the Canadian population, the annual number of deaths due to preventable medical error would be in the order of 5000 to 10 000 -- the equivalent of an airline crash every week. At a British conference in 2000, the UK's chief medical officer called for international collaboration to address the issue of medical error.20 In Canada, however, there has been little acknowledgment of the extent of the problem.

Error in emergency medicine

Apart from a few articles describing error in the diagnosis and management of psychiatric patients,21-25 the first report dealing with emergency department (ED) error did not appear until 1999.26 A year later, the Society for Academic Emergency Medicine (SAEM) sponsored a 1-day Error in Emergency Medicine conference to develop a consensus definition of EM error, to quantify error and identify its root causes, to set research and educational agendas and to establish SAEM policy and advocacy roles. An SAEM task force was established, and the conference proceedings were published in a special issue of Academic Emergency Medicine in November 2000.27

Most studies13,14,16-19 describe outcomes of hospitalized patients; therefore, the extent of error in the ED is largely unknown. EM error can take many forms, ranging from simple lapses, such as failing to send a radiography requisition, to more complex errors such as administering thrombolysis to a patient whose electrocardiographic changes are due to aortic dissection.

The 3 major studies described above16-18 found that a small proportion of all AEs (from 1.5% to 3%) occurred in the ED -- surprising given the operating characteristics of a typical ED (Table 1). But EDs had the highest proportion (70% to 82%) of preventable errors. In all 3 studies, preventable ED errors were most commonly diagnostic errors, and these often led to permanent disability or death. The authors suggested that inadequate physician training, high acuity and high volume (with limited time for individual patients) contributed to the substantial rate of preventable ED error.16 These data indicate that cognitive errors associated with clinical decision-making are critically important in the ED.

These studies probably underestimated the rate of ED error, because they examined only the records of hospitalized patients. Diagnostic and therapeutic errors undoubtedly occurred among patients discharged from the ED without being admitted, and these would only come to light if the patient returned to the same ED and if there was a systematic feedback mechanism to identify the problem.28 Furthermore, when such errors are discovered, inappropriate defence mechanisms such as secrecy, denial, projection and blaming often inhibit learning from the event.29,30

Several other factors may contribute to the higher rate of preventable error in the ED, especially the operating characteristics of the ED (Table 1). Ergonomists have determined that several of these factors are generic, producing error in other than medical settings.31,32

First, the patients are usually unknown to the physicians and nurses, and the patient information available to the ED staff does not match in continuity and completeness the history that would be available to the patient's family doctor. This problem is compounded by the relatively short time available for patient assessment and by the overall imperative to think and act quickly.

Second, decision density (the number of decisions that the physician must make during a shift) and cognitive load (the background information that the physician must bring to bear on those decisions) appear to underlie many cognitive errors. Common ED problems such as weakness, dizziness, and chest or abdominal complaints have a wide differential diagnosis and carry a high degree of diagnostic uncertainty. The combination of high decision density and diagnostic uncertainty leads to high error prevalence.1 The large number of physical, laboratory, radiographic and electrocardiographic examinations performed in the ED, and the need to accurately interpret their findings also increases decision density and cognitive load. Physicians can reduce cognitive load through the use of algorithms, clinical guidelines and decision rules. These and other innovative techniques, such as colour-coding resuscitation equipment,33 may have broad applications, particularly in pediatric care (Luten R, Wears R, Broselow J, Croskerry P, Joseph M, Frush K. Managing the unique size related issues of pediatric resuscitation: reducing cognitive load with resuscitation aids. [Manuscript submitted for publication]).

Third, the level of experience of physicians and nurses is intrinsically linked to preventability of error. The past 20 years have seen significant improvement in this regard. Fewer practitioners are itinerant, and many more now enter the discipline through formal training programs, committed to careers in EM. Nevertheless, it remains difficult to prepare for the wide range of clinical problems seen in the ED, which are often atypical or ambiguous and not infrequently have a catastrophic outcome. Under these conditions, experience counts. According to decision-making theorists, expertise in any domain typically requires about 10 years' experience;34 thus, the quality of decision-making by physicians and nurses depends on their EM experience.

Table 1. Unique operating characteristics of the emergency department predisposing to medical error
High levels of diagnostic uncertainty
High decision density
High cognitive load
High levels of activity
Inexperience of some physicians and nurses
Interruptions and distractions
Uneven and abbreviated care
Narrow time windows
Shift work
Shift changes
Compromised teamwork
Poor feedback

Clinical decision-making is compromised by interruptions35 and distractions. In Canadian EDs, available resources are commonly overwhelmed, which leads to conditions of overcrowding and prolonged waiting times. Despite the best efforts of those working under these conditions, it is inevitable that care will sometimes be compromised. This is well-known to physicians and nurses and is periodically publicized by high-profile cases in the media.36,37 Teamwork, a feature of optimal ED performance that may profoundly affect decision-making, is influenced by specific aspects of the trade-off between resource availability and efforts to achieve continuous quality improvement.38 Thus, as resources become limited the quality of both decision-making and the care provided by the team declines.

Another problem is the lack of feedback emergency physicians receive, from within the ED and from other specialties, medical records departments and the coroner's office. Without timely and reliable feedback, acquisition and maintenance of cognitive, procedural and affective skills can be compromised.28

All of these potential sources of error are compounded by shift work. Changeover from one physician or nursing shift to another disrupts care and increases the chance of error. Circadian rhythm disturbances and fatigue associated with night work lead to cognitive errors and impaired performance.39

Many medical errors result from flaws in thinking that affect clinical decision-making.1,40-45 Physicians and nurses,46 especially those working in the ED,47 are frequently unaware of how they evaluate the often haphazardly gathered evidence at their disposal. Considerable effort needs to be directed at understanding how emergency physicians solve problems and, more important, how they might avoid the many cognitive pitfalls that characterize EM.1,41,43

Error analysis offers a unique opportunity for emergency physicians to examine themselves and the system in which they work. When error is discovered, they must avoid the trap of assigning blame elsewhere, a reflex that can further compound the error. Objective root-cause analysis often illuminates the process or system flaws that underlie preventable AEs.

Detection, identification and measurement of error

The nature and extent of ED error is poorly defined. Current reporting mechanisms (e.g., incident reports) fail to capture up to 96% of errors.48,49 Lack of feedback prevents detection of and learning from errors; therefore it is critical to develop improved feedback mechanisms. Furthermore, there is no taxonomic system for classifying error, no consensus of what constitutes significant error and no reasonable estimate of the base rate of EM error. Several studies are under way to assess these important issues, including a pilot study in Australian EDs as part of the Australian Incident Monitoring Study.50

Morbidity and mortality (M&M) rounds are one avenue for open discussion of error in clinical case management; however, they may be subject to a number of biases.1 Case management may appear in retrospect better or worse than it actually was, sometimes because the conditions under which the original decisions were made are not reproducible and sometimes because other variables that exerted an influence at the time have been forgotten. Esoteric cases, which may be colourful but contribute little to clinical learning, are overrepresented in M&M rounds. These rounds should focus on cases that represent typical management problems or adverse outcomes, including ambient conditions at the time. Cases should be reviewed promptly so that clinicians can recall their decision-making process. Given that many serious EM errors are associated with misdiagnosis, root-cause analysis at M&M rounds may identify specific cognitive errors, the awareness of which may lead to strategies for their prevention in the future.1

The SAEM conference addressed many of these issues. In addition to developing definitions for common terms, it recommended nonpunitive systems for identifying and reporting potential AEs, as well as patient safety boards to monitor and review error reports within institutions, and provided recommendations for error prevention.51

Education

Recognition of the science of error prevention in health care52 has provided the imperative for educational programs. A recent proposal for an educational curriculum for error prevention in EM53 is based on the following principles: that basic training in error theory and management be multidisciplinary and that the culture of patient safety be inculcated in all sectors of society; that a core curriculum be developed to cover a wide spectrum of topics in basic error theory; that innovative teaching techniques be promoted, especially those using narrative accounts, clinical case analysis and high-fidelity simulation techniques; and that emergency physicians and nurses teach the program, with input from an expert interdisciplinary faculty.

Educational initiatives have already begun. Several years ago, Michigan State University implemented a multidisciplinary program to teach the science of medical error.54 Two years ago, a didactic medical error course (ACTAM: Applied Cognitive Training in Acute-Care Medicine) was introduced into the medical undergraduate program at Dalhousie University in Halifax. The ACTAM course, taught by an emergency physician, places considerable emphasis on cognitive errors in clinical practice. A companion case-based teaching manual includes a comprehensive glossary of multidisciplinary terms used in error theory.

Given that the ED has been characterized as a natural laboratory for the study of error,3 emergency residents enjoy a unique advantage. Clinical teaching positions will be needed for this new science, and residency-trained emergency physicians may be ideal candidates. Fellowships in EM error research have already been established in several EDs in the United States.

A critical aspect of the new culture of patient safety is the need to change societal attitudes toward medical error. Current error theory judiciously shifts the focus from individual blame to a better understanding of system and process factors. Just as many clinicians must become better acquainted with error theory, the public will also need to acquire a more realistic understanding of the fallibility of health care providers and the system in which they practise. Considerable effort will be required to overturn traditional attitudes toward medical error.

Sharing error information

The aviation industry, where much of error theory has already been applied and where considerable expertise in dealing with error has evolved, uses an excellent system of no-blame error reporting. Under the Aviation Safety Reporting System,48,55 errors are not treated as sentinel events, but as an inherent property of any system that involves decision-making. Given the universality and repetitive nature of error, workers in any field should have the opportunity to learn from the mistakes of others. Thus, medical staff should be apprised of significant errors. This can be done through hospital quality assurance committees, but an EM error section on the Canadian Association of Emergency Physicians (CAEP) Web site would be another way for emergency physicians and nurses to share such information openly. Setting up such a section would require quality assurance guidelines, and the information provided would have to be protected under the Freedom of Information Act, to ensure confidentiality and protection from discovery. A review panel could vet submissions to ensure that these conditions are met.

Conclusions

If research from other countries can be extrapolated to Canada, the prevalence of error and the rate of preventable AEs in our emergency departments is probably high, perhaps exceeding that in any other medical setting. The analysis and prevention of medical error will require a major investment of resources by the EM community. The Canadian Association of Emergency Physicians can take a leadership role by supporting error research, by facilitating discussions at its annual scientific meeting, by publicizing this important problem in the Canadian Journal of Emergency Medicine, by establishing a reporting site on the CAEP Web site, and through proactive advocacy, clarifying the central role of ED overcrowding as a root cause of medical error. One of our greatest challenges will be to change societal attitudes by promoting a clear understanding that errors are, in large part, secondary to system and process problems.

References

  1. Croskerry P. The cognitive imperative: thinking about how we think. Acad Emerg Med 2000;7:1223-31.
  2. Riegelman RK. Minimizing medical mistakes: the art of medical decision making. Boston: Little, Brown and Co; 1991.
  3. Bogner MS, editor. Human error in medicine. Hillsdale (NJ): Lawrence Erlbaum Associates; 1994.
  4. Youngson RM, Schott I. Medical blunders. New York: New York University Press; 1996.
  5. Spath PL, editor. Error reduction in health care. San Francisco: Jossey-Bass Publishers; 1999.
  6. Dowie J, Elstein A, editors. Professional judgement. Cambridge: Cambridge University Press; 1988.
  7. Kahneman D, Slovic P, Tversky A, editors. Judgement under uncertainty: heuristics and biases. New York: Cambridge University Press; 1982.
  8. Sharpe VA, Faden AI. Medical harm. Cambridge: Cambridge University Press; 1998.
  9. Bosk CL. Forgive and remember: managing medical failure. Chicago: University of Chicago Press; 1979.
  10. Paget MA. The unity of mistakes. Philadelphia: Temple University Press; 1988.
  11. Klein GA, Orasanu J, Calderwood, R, Zsambok CE. Decision making in action: models and methods. Norwood (NJ): Ablex Publishing Corp.; 1995.
  12. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health care system [report of the Institute of Medicine]. Washington: National Academy Press; 1999.
  13. Shimmel EM. The hazards of hospitalization. Ann Intern Med 1964;60:100-1.
  14. Ogilvie IG, Ruedy J. Adverse reactions during hospitalization. CMAJ 1967;97:1445-50.
  15. Mills DH, Boyden JS, Rubsamen DS, editors. California Medical Association medical insurance feasibility study. San Francisco: Sutter; 1977.
  16. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study 1. N Engl J Med 1991;324:370-6.
  17. Gawande AA, Thomas EJ, Zinner MJ, Brennan TA. The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery 1999;126:66-75.
  18. Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD. The quality in Australian health care study. Med J Aust 1995;163:458-71.
  19. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001;322:517-9.
  20. Donaldson L. How to take things forward [paper presentation]. In: Reducing Error in Health Care and Improving Patient Safety [conference of British Medical Association and British Medical Journal]; 2000 Mar 21; London.
  21. Herridge CF. Physical disorders in psychiatric illness: a study of 209 consecutive admissions. Lancet 1960;2:949-51.
  22. Carlson RJ, Nayar N, Suh M. Physical disorders among emergency psychiatric patients. Can J Psychiatry 1981;26:65-7.
  23. Hoffman RS. Diagnostic errors in the evaluation of behavioral disorders. JAMA 1982;248:964-7.
  24. Tintinalli JE, Peac Jun 7.
  25. Henneman PL, Mendoza R, Lewis RJ. Prospective evaluation of emergency department clearance. Ann Emerg Med 1994;24:672-7.
  26. Wears RL, Leape LL. Human error in emergency medicine. Ann Emerg Med 1999;34:370-2.
  27. Vincent C, Simon R, Sutcliffe K, Adams JG, Biros MH, Wears RL. Errors conference: executive summary. Acad Emerg Med 2000;7:1180-2.
  28. Croskerry P. The feedback sanction. Acad Emerg Med 2000;7:1232-8.
  29. Wu AW, Folkman S, McPhee SJ, Lo B. Do house officers learn from their mistakes? JAMA 1991;265:2089-94.
  30. Wu AW. Medical error: the second victim. BMJ 2000;237:726-7.
  31. Williams JC. A data-based method for assessing and reducing human error to improve operational performance. In: Hagen W, editor. 1988 IEEE 4th Conference on Human Factors and Power Plants. New York: Institute for Electrical and Electronic Engineers; 1988. p. 200-31.
  32. Reason JT. Managing the risks of organizational accidents. Aldershot (UK): Ashgate; 1997.
  33. Broselow J. Broselow pediatric resuscitation tape. Self-produced.
  34. Klein G. Sources of power: How people make decisions. Cambridge (MA): The MIT Press; 1998.
  35. Chisholm CD, Collison EK, Nelson DR, Cordell WH. Emergency department workplace interruptions: Are emergency physicians "interrupt-driven" and "multitasking"? Acad Emerg Med 2000;7:1239-43.
  36. Gillespie K. Paramedics to defy hospital redirects: death of asthmatic teen sparks new order to bring most seriously ill patients to nearest emergency room. Toronto Star 2000 Jan 16.
  37. Boyleand T, Keung N. Clement probing deaths from lack of hospital care: crowded ERs link 3 cases, coroner says. Toronto Star 2001 Jun 7.
  38. Risser DT, Rice MM, Salisbury ML, Simon R, Jay GD, Berns SD, Medteams Consortium. The potential for improved teamwork to reduce medical errors in the emergency department. Ann Emerg Med 1999;34:373-83.
  39. Tepas DI, Monk TH. Work schedules. In: Salvendy G, editor. Handbook of Human Factors. New York: Wiley; 1987. p. 819-43.
  40. Kassirer JP, Kopelman RI. Learning clinical reasoning. Baltimore: Williams and Wilkins; 1991.
  41. Croskerry P. Avoiding pitfalls in the emergency room. Can J Contin Med Educ 1996;Apr:1-10.
  42. Bartlett EE. Physicians' cognitive errors and their liability consequences. J Healthc Risk Manage 1998;Fall:62-9.
  43. Kovacs G, Croskerry P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med 1999;6:947-52.
  44. Elstein AS. Heuristics and biases: selected errors in clinical reasoning. Acad Med 1999;74:791-4.
  45. Redelmeier DA, Ferris LE, Tu JV, Hux JE, Schull MJ. Problems for clinical judgement: introducing cognitive psychology as one more basic science. CMAJ 2001;164:358-60.
  46. Epstein RM. Mindful practice. JAMA 1999;282:833-9.
  47. Kushniruk AW, Patel VL. Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inf 1998;51:83-90.
  48. Bogner MS. Human error in medicine: a frontier for change. In: Human error in medicine. Hillsdale (NJ): Lawrence Erlbaum Associates; 1994. p. 373-83.
  49. Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ 2000;320:759-63.
  50. Vinen J. Incident monitoring in emergency departments: an Australian model. Acad Emerg Med 2000;7:1290-7.
  51. Brennan TA. The Institute of Medicine report on medical error: Could it do harm? N Engl J Med 2000;342:1123-5.
  52. Croskerry P, Wears RL, Binder LS. Setting the educational agenda and curriculum for error prevention in emergency medicine. Acad Emerg Med 2000;7:1194-200.
  53. Gosbee JW. Human factors engineering is the basis for a practical error-in-medicine curriculum. In: Johnson C, editor. Proceedings of the First Workshop on Human Error and Clinical Systems. Glasgow Accident Analysis Group Technical Report G99-1. Glasgow: Glasgow Accident Analysis Group; 1999.
  54. Reynard WD, Billings CE, Cheaney ES, Hardy R. The development of the NASA aviation safety reporting system. NASA Publ 1114. Washington: National Aeronautical and Space Administration; 1986.