Peer Review: 2nd Reply to Harriman

Alexander Berezin berezin at MCMAIL.CIS.MCMASTER.CA
Fri Nov 3 19:45:12 EST 1995

On 3 Nov 1995, Gregory R. Harriman wrote:

> In article <Pine.3.89.9511031409.A9059-0100000 at mcmail.CIS.McMaster.CA>,
> Alexander Berezin <berezin at mcmail.CIS.McMaster.CA> wrote:
> > 
> > (HARRIMAN): 
> > 
> > > You say that anyone who wants to do an experiment 
> > > should be funded.  
> > 
> > (BEREZIN):
> > No, not a single bit. 
> I guess I have trouble understanding English.  But what you wrote in a
> previous post was: "anyone who wants to do an experiment should be
> funded".

Assuming, of course, he/she is:

(1) qualified to do the proposed type of research

(2) do have reasonable conditions to do this, including
being employed at teaching position in university/college, 
etc, having access to the equipment, etc

(3) having track record to warrant the funding.

You may say that the above conditions are tough. Yes,
they are. But paradoxically under the present system
(so called "proposal competition"):

1) many people who DO satisfy all the above 
conditions are nonetheless unfunded

2) many who do NOT satisfy [ mostly point 3 ] are
nonetheless funded
[ how to check for point 3 see below ]

> > (BEREZIN):
> > What I am saying is that ONLY
> > people who have clearly demonstarted that they are able 
> > to do research AND produce impactful results (not just 
> > papers) should be funded (within reasonable resources, 
> > of course, which in practice means sliding scale) be 
> > funded to conduct further studies.
> Where I am having difficulty is in understanding how you propose to ASSESS
> whether researchers have "clearly demonstarted (sic) that they are able to
> do research".  It seems obvious that someone (a peer perhaps?) would have
> to determine this.  How does this differ in a meaningful way from peer
> review?

Very much differ. Peer review (in a regular meaning) deals with 
a single item (manuscript or a grant proposal). Under the present
system the item can be rejected ALTOGETHER (paper denied publication, 
grant application not funded).

When you assess LIFE-TIME record this much less likely to happen.
The only time it CAN happen is when the assessors (call them peer 
reviewers or not) are ALL IN UNISON saying:

  "Soandso in a totally useless researcher, discovered
   nothing, all his/her work worth nothing at all".

For the rest the task is not that difficult as it may seem.
After all, what is asked is "What Soandso REALLY discovered ?"
(as opposed to publishing whatwever many hunderds of papers).

First, you have the statement from the applicant 
"What did I discover" as a (required) part of his/her CV.

Second, you have other evidences which the applicant can
present (Science Citation Index data, citing reprints
to show the impact of the work, etc).

The difference from peer review that the committee
assessing track record does NOT need to be a very narrow
specialized in the applicant's area (hence arm-length
rule applies and very low, if any, possibility of
conflict of interest). 

> > (BEREZIN):
> > I am against anonymity, 
> > not against peer review altigether, as I explained
> > earlier).
> Here, I think we come closer to agreement.  There are certainly problems
> with the current method of peer review.  This could and should be improved
> upon.  I don't pretend to have all the answers (unlike some) but perhaps
> providing anonymity to the researcher who submits a manuscript or grant
> for review might be a step in the right direction.

The proposal you are talking about (anonymous submission of 
manuscripts and/or grant proposals) is often discussed, however 
there are fundamenat flaws in it:

(1) the author almost invariably can be identified
through the bibliography

(2) the MAIN point that the anonymity of the reviewer
essentially realeases him from ANY responsibility for
the process is not addressed. 

(double-blind reviewing has even more flaws and 
despite been attemplted many times never really
kicked in - althogether this is highly unlikely
route to follow)

I don't think that the flaws of the present system
can be addressed in any serious way for as long as
the anonymity of the REVIEWERS remains in force.
This is why it is so important to get some legal
definition against APR (e.g. Henke's case, now in 
Federal Appeal court - how it will be resolved we 
don't know yet).

> > (BEREZIN):
> > To assess CV is a much more reliable proceedure than
> > to assess "proposals" (futurology - i.e. the work yet
> > to be done). The evaluation of the impact of ALREADY
> > DONE work is much less error prone than the 
> > evaluation of chances that the PROMISED work will be 
> > done (we all can make a lot of promises). If system 
> > will eventually return to the formula "fund RESERCHERS, 
> > not proposals", proposal evaluation (and grant writing) 
> > will not be neccessary altogether. (this is what the
> > paper shuffling funding bureaucracy resists by obvious 
> > reasons).
> I do agree that track record should count for something.  The question is
> how and who determines what your track record is?  And how would this
> method of evaluation avoid the same problems and shortcomings of the
> current system?  I doubt I am the only one who has heard comments made by
> other researchers that the track record of the investigator (ie. the
> quantity and quality of the CV) is what's important in a grant
> application, not the quality of the scientific proposal. 

It is virtually impossible that someone with a weak track
record came up with really strong proposals. The only possible
exceptions are some "junior geniuses" who can come with indeed 
something promising. (you ask about such case in your next passage). 
Well, I can suggest a relatively simple way to deal with such
young Einsteins. Give him/her SMALL bona fide grant for, say 
2 or 3 years, and see how s/he handles the promiss. 
Yes, for younger applicant the whole scheme appears more
close to traditional peer review scheme (and so it is).
Nonetheless, even in this case peer review is de-facto 
downplayed and some kind of a risk is taken.
But this is where the risk is, I believe, well justified 
(esp. taking into account a small funding level).    

> If this attitude
> doesn't facilitate abuse and perpetuation of an old boy's network, then I
> don't know what does.  How can a new or struggling investigator (even if
> they have a great idea) hope to be successful with this kind of an
> environment

(see above comment - AB)
> Greg Harriman

More information about the Bioforum mailing list