Peer Review: Reply to Harriman
berezin at MCMAIL.CIS.MCMASTER.CA
Fri Nov 3 15:34:32 EST 1995
Dear Gregory (Harriman):
You make very good points and your questions
express true concerns. I am addressing most
of them by next set of comments (see below).
On 3 Nov 1995, Gregory R. Harriman wrote:
> In article <Pine.3.89.9511030911.A13466-0100000 at mcmail.CIS.McMaster.CA>,
> berezin at MCMAIL.CIS.MCMASTER.CA (Alexander Berezin) wrote:
> * (much deleted)
> ** -"anyone who wants to do an experiment should be funded":
> * (BEREZIN):
> ** -precisely, for as long as he/she can reliably demonstate
> ** -his/her qualifications (track record assessment - I am
> ** -NOT against this and this is NOT a peer review).
> * (CLARK):
> ** So would you be happier if we called APR "research product assessment"?
> ** I fail to see any substantive difference between peer review of grants
> ** and peer review of manuscripts.
> * BEREZIN:
> * I am not talking here about the difference between PR for
> * grants and manuscripts. I am talking about the "track record
> * assessment" (to determine the funding level) which is NOT
> * peer review by the following reasons:
> * In APR (both of grants and manuscripts) the reviewers
> * (peers: by definition your competitors) have access to
> * the priveledged (yet unpublished) information.
> * Much of recent criticism of APR was related to the
> * unlawful use of this priveledge information (in short,
> * intellectual theft). (this is not to say that all APRers
> * are thiefs, but some unfortunately are).
> * In assessing track record (actually, you CV), only
> * ALREADY PUBLISHED work is considerd, so all of it is
> * a public domain information the reviewers have no
> * advantages in comparison with all the rest.
> I am having a real problem accepting this line of reasoning. I'm
> afraid it sounds very circular if not specious.
> You say that anyone who wants to do an experiment
> should be funded.
No, not a single bit. What I am saying is that ONLY
people who have clearly demonstarted that they are able
to do research AND produce impactful results (not just
papers) should be funded (within reasonable resources,
of course, which in practice means sliding scale) be
funded to conduct further studies. This is why we have
all our PhDs defended, posdocs passed, papers published
(yes, PEER REVIEED papers too, I am against anonymity,
not against peer review altigether, as I explained
earlier). In short, the best criteria for the good
probability of success is the previous record.
To assess CV is a much more reliable proceedure than
to assess "proposals" (futurology - i.e. the work yet
to be done). The evaluation of the impact of ALREADY
DONE work is much less error prone than the
evaluation of chances that the PROMISED work will be
done (we all can make a lot of promises). If system
will eventually return to the formula "fund RESERCHERS,
not proposals", proposal evaluation (and grant writing)
will not be neccessary altogether. (this is what the
paper shuffling funding bureaucracy resists by obvious
> You indicate that their track
> record should be used to determine their funding level. Next you say that
> their track record should be based on their publications (CV). But in
> earlier posts you have argued that scientists should be allowed to publish
> anything they want without prior peer review (so called option B).
It is clear that if you publish several papres under
the option (B) [ sever criticsm notwithstanding ] and
if later development will support that your critics
(rather than you) were right, than such a record will
be manifestely AGAINST you. Therefor, in practical terms
I don't think people will often use option (B) in case
of severe critism. Nonetheless, they (the authors)
should have the RIGHT to use this option, should they
chose to risk.
> If one follows the logic of this, the result will be: anyone doing
> research of any kind or quality will always be able to publish their
> papers. Consequently, every scientist will have an extensive publication
> record and therefore will be qualified to receive as much funding as they
> want. Sounds wonderful! There are unfortunately a few problems. First,
> there is not an infinite amount of money for research. Second, how do you
> distinguish between research which is high quality and valid versus that
> which is garbage?
No, this is a (very common) misreading of what Forsdyke,
Osmond, Roy and many other (incl.humbly myself) mean by
"track record evluation" (TRE, for brief). TRE is NOT paper
counting. It is the assessment of the IMPACT of the work.
It should be conducted commulatively on a number of items,
such as citations, expert evaluations by CLOSE peers AND
by 'at-arm-length', i.e. DISTANT peers, etc.
Important that the TRE should be done in relationship
with prevous funding. So, the main question to be asked
TRE panel is something like this:
"Prof. Soandso for the last 10 years got $X,XXX,XXX total
funding. S/he (his/her group) has achieved such and such
results, as documented by the following data:....".
Dollar yield of the work (say, cummulative impact factor
DIVIDED by total dollars) is such-and-such. On the
basis on this [ and taking into account the overal
budet situation) the coniniing funding level is recommended
for Soandso for the next 3 (or whatever finding cycle)
years is $.... per year."
With this scheme you may end up with prolific (graphomanic)
and (so far) generously funded profs/groups end up much
worse than some small scale researchers. So, thick CV
not nesessary a guarantee of fat funding continuation.
And, of course, to address you concerns about "garbage",
yes, if ALL the parameters turn against you, well
bad luck - you are out of your personal funding spiral
and look for somebody else (who is funde) to join
temporarily if you determined for the next bid.
But I don't think that cases of total unfunding if the
above scheme is used will be frequent.
> Gregory Harriman
More information about the Bioforum