PEER REVIEW: FIXES POSSIBLE

Alexander Berezin berezin at MCMAIL.CIS.MCMASTER.CA
Mon Jun 17 13:12:11 EST 1996



On Mon, 17 Jun 1996, Phil Koopman wrote:

> 
> Bert Gold <bgold at itsa.ucsf.edu> wrote:
> >PAPER in FASEB Journal (1993) volume 7  pages 619-621
> >On giraffes and peer review
> >D. R. FORSDYKE
> >
> >Department of Biochemistry, Queen's University, Kingston,
> >Ontario, Canada. K7L3N6
> 
> Peer review could also happen because the people in charge of the
> money aren't able to make the decisions themselves.  For example, with
> increased specialization it is hard to be knowledgeable enough to make
> decisions across a large field of competing requests.  In my mind this
> is a significant driving force that has to be dealt with (bicameral
> review might work -- I don't know much about it).

In academia, at least, the above uncertainties can be
alleviated by having a sliding scale for the researchers 
based on track record instead of putting a heavy weigth on
proposals (you can't reliably assess proposed [ undone ]
EXCEPT of those cases when the proposals are clearly 
incompetent - but this is NOT the case with most 
ZERO-funded proposals at NIH/NSF/NSERC).

Questionable proposals (and/or mixed track record) and 
those getting conflicting peer review assessments should 
be funded at SOME (often quite low) level, but NOT at 
(ZERO) NIL level as the system does now.
 
> 
> I recently came from an industrial lab where the funding allocation
> changed from "benign" dictatorship to a group-grope model.  IMHO
> productivity plunged as people spent most of their time chasing a pool
> of money that diminished with each downsizing (sound familiar?).
> 
> There are three problems with any such system:
> 
> - How do you find and fund the important radical new idea (especially
> in the face having to sift through the chaff of poorly researched new
> ideas that are simply recastings of old tired ideas).  The best ideas
> tend to upset the entrenched Establishment.

If is unlikely that the grant applicant has no track
record whatsoever. So, here you are talking of someone
who is apparently quite junior faculty and came up
with 'radiacaly new idea'. Well, give him/her a chance
to prove it: fund him/her at 5 to 10 % of the average
level and let him/her look for the avenues of 
cooperation. But if you give no money whatsoever
cooperation is (almost) a non-starter option.

This is why sharp cut-off system is so detrimental
and even token awards INSTEAD OF NIL will be of
great relieve to many.  
 
> - How do you break the feedback of a dead-end research topic that has
> gained momentum (and thus is supported by researchers all praising the
> others' work)?

If I inderstand you correctly, here you are talking
about a well established, well funded lab, which has
proper connections, long CV, etc but went to the
stagnation stage. Nonetheless, because all the above,
the (high) funding is rolling. How to curb this
effect ? 

Well, again an assessment questioning which will 
center on the question "What Prof. XYZ has REALLY 
discovered/developed over the last 10 years ?
(NOT "how many papers s/he has published') may help 
to re-orient the granting process towards more 
ballanced one. 

Obviously, this will be resisted by all Fat Cats,
who often have very fat CVs and generous funding,
but in fact have discoverd very little (often,
nothing at all) being essentially successful 
copycats of other's prime ideas (often their
postdocs and/or grad. students).

This is where a visible grassroot effort is needed
for some noticable reversals to happen.
 
> - *ANY* system is capable of being "gamed", and will be so in the face
> of inadequate resources.  How do you minimize the resources spent
> gaming the system?  IMHO a system that tends to bring dirty laundry
> out into the open is the best way to go, although no system is
> perfect.

Yes, of course, we all know that any system can be
gamed. But some systems are more gamable that others. 
The present (grant selection, winners-looser, secretive
peer review) system is HIGHLY gamable. Th system
based on the funding  _researchers_ (rather than
proposals) is LESS gamable by its very nature than
the present (completely secretive and hence unaccountable)
system.

> -- Phil
> 
> 
> Phil Koopman -- koopman at cs.cmu.edu -- http://www.cs.cmu.edu/~koopman
> 
> 



More information about the Bioforum mailing list