Important returns on investment: an evaluation of a national research grants competition in emergency medicine

EM Advances

CJEM 2010;12(1):33-38

Abstract

Résumé

Introduction

Emergency medicine (EM) is a relatively new specialty that has focused on developing expertise in clinical practice and medical education.1 Recently, research in EM has grown, and productivity, especially from Canadian researchers, has been impressive.2 Experts in EM recently authored a report on the future of emergency care in US emergency departments for the Institute of Medicine.3 This report suggests that research opportunities created by EM researchers could serve as an interdisciplinary bridge in the area of clinical and translational research. Despite the importance of research, funding for EM projects has always been limited. Moreover, support for researchers in EM has also been limited,4 as there are few sources of dedicated funding in the field.

Several steps have been taken to address the gap between the need for research and the funding required to support its growth. More EM trainees have obtained training in postgraduate research and evidence based medicine. Furthermore, the Canadian Journal of Emergency Medicine (CJEM) was established to encourage Canadian EM researchers to publish scholarly work. Also, dedicated funding sources for EM research have been established.

Grants programs from the US based Emergency Medicine Foundation and the Society of Academic Emergency Medicine (SAEM), provide dedicated EM research funds.5 The Canadian Association of Emergency Physicians (CAEP) has created national research funding through an endowment fund and other research initiatives. As a charitable nonprofit organization representing EM physicians across Canada, CAEP plays an important role in EM research. Finally, in the 1990s, Ontario developed the Emergency Health Services Research Advisory Committee to exclusively consider emergency based research and research fellowships.

In 1995, CAEP developed the CAEP Research Grants Competition from funds generously provided by Hoffmann-La Roche Ltd. Annually, Can$25 000 is distributed through the competition, with a maximum of $5000 for each successful application. We sought to determine the status of projects funded by the CAEP Research Grants Competition, examine the outcomes of these projects and determine the opinions of the investigators regarding the usefulness of the funding.

Methods

Design and survey methods

We conducted an email based survey of recipients of the CAEP Research Grant. We distributed an English language electronic or paper based cover letter and a brief survey to each of the 62 CAEP Research Grant recipients and/or principal investigators between 1996 and 2005. We designated grants as accepted, returned (grant awarded, but funds returned) or refused (other funding became available). We sent up to 3 reminders to investigators if the survey was not completed and returned.

We obtained administrative approval from CAEP before beginning the study. The University of Alberta Research Ethics Board approved the protocol for our study. By completing the survey and returning it, researchers implied consent for inclusion in the study.

Survey instrument

Our survey consisted of 2 main parts: a series of questions concerning the project itself (e.g., publication status, presentations and additional funding) and a component on the opinions of the research investigators. Using a 1-7 Likert scale (1 = strongly disagree and 7 = strongly agree), all respondents answered a series of questions regarding the utility of the grants.

We tailored surveys to the individual researchers by searching for publications and presentations of the projects online using MEDLINE, EMBASE and The Cochrane Library. In addition, we hand searched abstracts in an attempt to identify those primary authors presenting at annual scientific meetings (journal): CAEP (CJEM), SAEM (Academic Emergency Medicine) and American College of Emergency Physicians (Annals of Emergency Medicine). We then added to the survey any details obtained regarding publications or presentations, and we asked the researcher to verify these and add to the list as necessary. In an effort to reduce the respondent burden and increase compliance with the request, we also removed certain questions from the survey if they were deemed unnecessary or unclear using consensus methods.

Data collection

We encouraged the researchers to complete the survey and return it for processing either via email as an attachment or via fax. We requested that researchers provide contact information in the event that any additional information was required, and we informed them about confidentiality and consent.

Impact factor (IF) scoring is a quality rating system for biomedical journals. We recorded the IF of articles for the 2008 year for each journal. In cases where a project contributed to a multicentred series of publications, we used the median IF employed for one publication per project. If a project produced multiple publications, but was not part of a multicentred study, we used all publication IFs.

Statistical analysis

We present descriptive data as numbers with their respective proportions and as medians, with interquartile range (IQR), as all data were not normally distributed. We performed analyses using Stata Statistical Software: Release 10.0 (Stata Corporation).

Results

Survey response rate

We received a total of 58 (97%) responses; 51 (88%) projects were completed at the time of contact (Fig. 1).

Study data

Of the 60 grants awarded, the median award value was $4700 (IQR $3250-$5000). Because of a funding irregularity, there were no grants awarded in 1998; however, 13 grants were awarded the following year. A median of 6 grants were awarded annually. Grants were awarded most commonly to residents (21 [36%]), followed by senior (16 [28%]) and junior (13 [22%]) emergency staff. Researchers in Ontario (26 [45%]) and Quebec (11 [19%]) received the majority of the grants. A summary of the studies and their funding is provided in Table 1.

Scholarly productivity

From the projects of the 58 respondents, 99 presentations, or approximately 2 presentations per grant, were reported. There were 23 local, 39 national and 37 international presentations made.

Of the completed projects, 37 were published and 2 were in press. Of respondents with complete yet unpublished projects, 1 (2%) had attempted publication and 2 were revising their manuscript. Of the remaining 9 who had not published at the time of the survey, 4 respondents reported they had no time to write the manuscript and 3 were currently working on a submission (2 didn't give a specific reason for this finding). There were 7 projects that were incomplete at the time of contact. Of these, 6 were ongoing and 1 (14.3%) was terminated.

Impact factors

From 39 projects with at least 1 publication, 42 manuscripts were confirmed. Overall, 9 manuscripts were published in journals that did not yet have IFs (e.g., CJEM). Of the remaining 33 manuscripts, the median IF was 3.755 (IQR 2.460-4.122).

Other funding

Of the 58 respondents, 19 (33%) were able to obtain additional funding. Sources included 3 (16%) government grants, 3 (16%) charitable organizations, 1 (5%) pharmaceutical company and 12 (63%) other sources. Overall, median additional project funding was $5116 (IQR $4010-$17 500).

Opinions

Most respondents felt that CAEP funding was important in accomplishing the project and that holding the grant assisted somewhat in obtaining additional funding (median 3.5, IQR 1-5) (Table 2). Most respondents believed EM researchers require designated research funds to stimulate research in their field and most reported they would highly value another grant through the CAEP competition (both medians 7, IQR 7-7).

Discussion

Since the inception of the CAEP Research Grants Competition, the funding award has been static while the demand has increased (Dr. Eddy Lang, CAEP Research Chair, Montréal, Que.: personal communication, 2009). This CAEP research initiative has been supported exclusively by Hoffmann-La Roche; how ever, this partnership has recently stopped. Despite the involvement of other industry partners in the CAEP annual meeting and other CAEP sponsored activities, interest in research funding has been difficult to generate. In addition, the CAEP Research Endowment Fund has not yet achieved the magnitude required to support an annual research competition (Valoree McKay, CAEP CEO, Ottawa, Ont.: personal communication, 2009). Targeted research funding for Canadian EM research remains infrequent. As a professional organization, before attempting to secure additional research resources, it is important for funding organizations to evaluate their progress and justify further funding.

This evaluation of the first decade of the CAEP Research Grants Competition suggests that the targeted approach sponsored by CAEP is an efficient and well regarded use of limited research funds. Of the 62 award winners, only 2 declined the grant; in both cases, the researcher had received additional funding from other sources. Of the grant recipients, the large majority of projects were completed, an average of 2 abstracts were presented at scientific meetings, and most were published in high ranking EM journals (median IF 3.755). In addition, the recipients were both grateful for the funding and highlighted its importance in project completion.

It is important to compare this evaluation to similar EM grant programs. In a recent SAEM Grants Committee survey, grant recipients remained in academic medicine, were producing approximately 2 manuscripts per year, and were not considering departure from academia.6 Although the SAEM grants are substantially larger than those provided by CAEP, it is encouraging that the grants in some manner contributed to retention of the award recipients in academic medicine. Other research has demonstrated an association between program location and faculty size and the number of publications for a specific institution.4 However, funding must be considered an important facilitator for productivity.

Further primary research is required to characterize efforts of emergency physicians to obtain funding for research. There also needs to be more research on what factors influence publication in EM. For example, many more presentations were recorded than publications and some research suggests that publication biases (e.g., negative studies, small studies) exist in the medical literature.7 An evaluation of randomized controlled trial abstracts presented at SAEM conferences over 10 years found less than 60% were published up to 5 years after presentation, although the traditional publication bias was not identified.8 The study illustrated how publication is delayed or nonexistent, even for studies that are considered to be of high methodological quality. Finally, award winners still in training (e.g., residents or fellows) require more evaluation than this study can generate, since at least 1 institution has shown less frequent publications for resident projects where research was mandated.9 Understanding the factors that influence publication will permit the new generation of educators and researchers to benefit from the lessons learned.

The Institute of Medicine report recommended the development of strategic plans, including research, on a local level in conjunction with support from national EM organizations, allied health care, specialty organizations and consumer groups.10 Although reports like these represent important policy statements, until efforts to support research are developed, innovative approaches to EM research funding may also be required. For example, many societies fund their own research activities through endowment funds. The CAEP endowment fund languishes behind other similar societies and requires a jump start in order to provide meaningful funding opportunities to researchers. In addition, institutions may especially need to support junior investigators in EM. This CAEP evaluation has shown that even small amounts of funding are sufficient to produce a return on investment and others have shown how research contributes to improving health care delivery.11

Limitations

This survey has several potential limitations. First, some grants during one year supported a multicentred study at several sites. Although these studies were considered completed, the recipients were often not the primary authors of the final manuscripts. Nonetheless, this collaboration produced many manuscripts, CAEP researchers contributed to the success of the project and CAEP researchers were involved (i.e., authors or acknowledged) in each manuscript. We do not feel this issue influences the results.

Second, the IF is a controversial quality grading of journals, and the majority of EM journals have low impact factors (the highest IF belonged to Annals of Emergency Medicine at 3.755). In fact, CJEM is too new to receive an IF (Penelope Gray Allan, CJEM Managing Editor, Vancouver, BC: personal communication, 2009). Consequently, the IF assessment may not accurately reflect the topic's importance to emergency physicians. Third, this evaluation did not investigate the role this funding may have played in the future retention and productivity of the award recipients.

Finally, no comparison data are available with which to compare these results. Although other grants specifically targeting EM research exist, we could not identify any similar documentation. Many of these agencies provide grants of larger magnitude and would be expected to have higher scholarly success rates.

Conclusion

Notwithstanding these limitations, the nearly complete response rate and detailed information provides a valid evaluation of the CAEP Research Grants Competition. Even small research grants have been successful in promoting scholarship among Canadian EM researchers. Productivity, measured by presentations and publications, was high in this cohort. Researchers report that these small grants are vitally important for project completion. Additional effort is needed to enhance funding of EM research in Canada and this research suggests such efforts would be expected to be similarly productive and justified.