PEER REVIEW: FIXES POSSIBLE
berezin at MCMAIL.CIS.MCMASTER.CA
Mon Jun 17 13:12:11 EST 1996
On Mon, 17 Jun 1996, Phil Koopman wrote:
> Bert Gold <bgold at itsa.ucsf.edu> wrote:
> >PAPER in FASEB Journal (1993) volume 7 pages 619-621
> >On giraffes and peer review
> >D. R. FORSDYKE
> >Department of Biochemistry, Queen's University, Kingston,
> >Ontario, Canada. K7L3N6
> Peer review could also happen because the people in charge of the
> money aren't able to make the decisions themselves. For example, with
> increased specialization it is hard to be knowledgeable enough to make
> decisions across a large field of competing requests. In my mind this
> is a significant driving force that has to be dealt with (bicameral
> review might work -- I don't know much about it).
In academia, at least, the above uncertainties can be
alleviated by having a sliding scale for the researchers
based on track record instead of putting a heavy weigth on
proposals (you can't reliably assess proposed [ undone ]
EXCEPT of those cases when the proposals are clearly
incompetent - but this is NOT the case with most
ZERO-funded proposals at NIH/NSF/NSERC).
Questionable proposals (and/or mixed track record) and
those getting conflicting peer review assessments should
be funded at SOME (often quite low) level, but NOT at
(ZERO) NIL level as the system does now.
> I recently came from an industrial lab where the funding allocation
> changed from "benign" dictatorship to a group-grope model. IMHO
> productivity plunged as people spent most of their time chasing a pool
> of money that diminished with each downsizing (sound familiar?).
> There are three problems with any such system:
> - How do you find and fund the important radical new idea (especially
> in the face having to sift through the chaff of poorly researched new
> ideas that are simply recastings of old tired ideas). The best ideas
> tend to upset the entrenched Establishment.
If is unlikely that the grant applicant has no track
record whatsoever. So, here you are talking of someone
who is apparently quite junior faculty and came up
with 'radiacaly new idea'. Well, give him/her a chance
to prove it: fund him/her at 5 to 10 % of the average
level and let him/her look for the avenues of
cooperation. But if you give no money whatsoever
cooperation is (almost) a non-starter option.
This is why sharp cut-off system is so detrimental
and even token awards INSTEAD OF NIL will be of
great relieve to many.
> - How do you break the feedback of a dead-end research topic that has
> gained momentum (and thus is supported by researchers all praising the
> others' work)?
If I inderstand you correctly, here you are talking
about a well established, well funded lab, which has
proper connections, long CV, etc but went to the
stagnation stage. Nonetheless, because all the above,
the (high) funding is rolling. How to curb this
Well, again an assessment questioning which will
center on the question "What Prof. XYZ has REALLY
discovered/developed over the last 10 years ?
(NOT "how many papers s/he has published') may help
to re-orient the granting process towards more
Obviously, this will be resisted by all Fat Cats,
who often have very fat CVs and generous funding,
but in fact have discoverd very little (often,
nothing at all) being essentially successful
copycats of other's prime ideas (often their
postdocs and/or grad. students).
This is where a visible grassroot effort is needed
for some noticable reversals to happen.
> - *ANY* system is capable of being "gamed", and will be so in the face
> of inadequate resources. How do you minimize the resources spent
> gaming the system? IMHO a system that tends to bring dirty laundry
> out into the open is the best way to go, although no system is
Yes, of course, we all know that any system can be
gamed. But some systems are more gamable that others.
The present (grant selection, winners-looser, secretive
peer review) system is HIGHLY gamable. Th system
based on the funding _researchers_ (rather than
proposals) is LESS gamable by its very nature than
the present (completely secretive and hence unaccountable)
> -- Phil
> Phil Koopman -- koopman at cs.cmu.edu -- http://www.cs.cmu.edu/~koopman
More information about the Bioforum