Peer Review & Funding Cuts

Gary Stormo stormo at beagle.Colorado.EDU
Fri Jun 2 17:09:53 EST 1995


Would you be so kind as to give a brief summary of proposals to replace peer 
review?  Unfortunately I do not have the time, at least right now, to track
down the references you provided.  I am very familiar with the peer review
process, both as reviewer and reviewee, and I know that it is not perfect.
However, I am unaware of alternatives, possibly through ignorance, that
would appear to be better.

With regard to funding, peer review is really designed to assign priorities
to competing applications.  Although it has changed somehwat, the process
at NIH used to be first deciding if an appication was to be approved or not.
Disapproval meant, in simplistic terms, that the application was not worth
funding even if sufficient funds existed (i.e. all approved applications
were funded).  This happened to only a very small percentage of applications
and usually indicated some fatal flaw in the application.  All of the 
other grants were approved, menaing that if sufficient funds existed they
should receive them.  Of course there have never been enough funds to
fund all of the approved applications; several years ago the percentage 
that were funded was often in the 25-30% range, and lately its more often
been near, or below, the 15% range.  So given that there are a lot more
good ideas, in the form of approvalable applications, than there are funds
to support them, how does one, of the government in this case, go about
deciding which are to receive funds and which are not?  I think most
people would agree that the advice of experts should be obtained, and 
that's the basic idea behind peer review.  As I said, its not perfect and
some ideas that later prove to have been worthwhile were passed over,
and other ideas were funded that turned out to be complete wastes of money.
But those kinds of errors are to expected given that the reviewers are
merely experts and not omniscient.  Other kinds of problems with peer
review, such as conflicts of interest, are attempted to be avoided with
great care, at least at NIH.  Probably some cases slip through.  Perhaps
a more pervasive problem is the tendency of reviewers to look more favorably
at applications that look like a sure bet than at ones that seem more likely
to fail, but have large benefits if they succeed.  But this is not really
a fault of the peer review system per se, that is one would still like
the advice of experts, but more a problem in how the government should
invest its limited resources.  In fact, NIH has instigated a special
category of grants, called R21s, that are pilot projects or feasibility
studies for applications that are high risk/high payoff.  That is they
may well not succeed, but if they did they could prove to be very
important.  The idea is to identify applications in that category and
to give them enough funding to be able to show whether the idea has
real merit.  If so then it will be able to compete with the regular
grants, and if not it will at lesat have been tried without allocating
a large amount of resources.

Of course, most of what I just went through is just details, but the essence
of peer review is to decide how to spend public money based on the advice
of experts, i.e. the peers of the applications who are supposed experts in
their field or they wouldn't even be applying for funds.  If you can
come up with a better alternative to allocating limited resources I
would love to hear about it.

-- 
Gary Stormo       |   
MCD Biology       |   Keep in mind that to the advertising industry, 
Univ. of Colorado |   every day is April Fool's Day.                 
Boulder, CO 80309 |



More information about the Bioforum mailing list