The winners and losers of the 2013 National Health and Medical Research (NHMRC) Project Grants were announced in October. A record low success rate of just 16.9% (down from last year’s 20.5%) meant the champagne stayed in the fridge for most.
Project grants are the major source of funding for new ideas in health and medical research. Many scientists rely upon them for their job or the jobs of their staff.
Failure is a major blow which is made harder by being given almost no feedback. Applicants are told their scores, but are given no idea where they went wrong.
Australian scientists are becoming frustrated by the system, with a survey revealing 95% of scientists agree that changes to the application process are needed, 90% agree that changes to the peer review system are needed and 99% agree that they would like more feedback.
Now for something completely different
Problems with funding peer review are nothing new. An article in Science back in 1981 discussed the “wastefulness of a system” where scientists spend too much time writing applications at the expense of doing actual research.
The suggested solution was that written applications be scrapped and that research dollars could be allocated to departments using the formula:
Dollars per department =
A × (number of Masters degrees + 3 × number of PhD degrees) +
B × (number of published papers) +
C × (dollar support from other agencies) +
D × (dollar support from industry).
A, B, C and D are multipliers that control the importance of each research output.
The issue is there would be a huge row about what these multipliers should be. Departments with lots of students would argue for “A” to be large, whereas departments that worked closely with industry would argue for “D” to be large. The formula would be good for some, but bad for others who would resist its introduction.
A formula that mimics current funding
It might be possible to keep everyone happy if a formula could be designed that mimicked current funding. The formula would be trained to predict past winners based on historical data for research performance. Its accuracy would then be prospectively tested by its ability to predict the winners in a future round.
This big data problem is ideally suited to a Kaggle competition in which multiple formulae would compete to give the closest match to the current system. Selecting the closest matching formula would mean that the historic knowledge of the funding system would be preserved in formula form.
The formula would be stratified according to research field, because the definition of research quality varies greatly between academic fields. The formula would also be stratified according to experience to ensure that early career scientists were not disadvantaged.
A formula has many benefits
The biggest advantage of allocating funding using a formula would be the enormous amounts of scientists’ time that would no longer be wasted on lengthy funding applications. It would also save peer reviewers time, and cut funding agencies’ administrative costs.
A formula would remove the subjective human element from funding, which would remove the randomness in funding decisions. A formula is also blind to gender, age and geography, and it solves the conflict of interest problem, which is especially relevant in Australia’s small research community.
It would be a transparent process and the monthly list of winners could be published online. Research money could be distributed at any time, with payments on a monthly basis rather than the current boom and bust which inhibits career development (promising scientists have quit because of their fragile career tenure).
A formula would starve uncompetitive scientists (who would hopefully seek other activities) and nurture competitive scientists to thrive.
Objections to a formula
There may be concerns about scientists gaming the formula, but any formula is likely to be based on outcomes such as getting high quality publications that are highly cited, so it should reward good research behaviour.
Gaming is present in current funding systems, including submitting the same application to multiple agencies and submitting applications where the work has mostly been completed.
A formula may be objected to on the grounds that it would stifle innovation. But many current funding schemes reward conformity rather than risky research. A written application ties a scientist to their plans for three or more years. Funding high performing scientists without requiring a specific research plan should encourage more innovation.
Moving with the times
Funding agencies were essential from the 1970s to 1990s when the need to distribute research funding became great, but the information required to decide on who to fund was not easily available. They played a valuable agency role between governments and scientists.
Today the information needed to make funding decisions is likely freely available on the internet. Funding agencies have become bureaucratic, and have externalised large and avoidable costs onto the research community they are supposed to serve. A formula would be a radical change to funding, but a welcome one.