Peer Review: A light-hearted reply 8-)
berezin at MCMAIL.CIS.MCMASTER.CA
Sat Nov 4 12:20:30 EST 1995
Some comments to Dr. Kristofferson's post. Alex Berezin
On 4 Nov 1995, David Kristofferson wrote:
> Unfortunately I've been too busy running BIOSCI/bionet lately to enjoy
> it, but this discussion is too intriguing to pass up 8-). I apologize
> if the issues below have already been addressed; if so, please just
> say so. I've been through my share of grant/contract battles over the
> years too unfortunately.
> Dr. Berezin's proposals are very interesting, but I wonder how he
> would react to the following problems:
> 1) The peer review process focusses on a single concrete proposal.
> Despite the fact that the information to review is relatively finite,
> how many times have you seen reviewers who do nothing prior to jumping
> on an airplane to, e.g., Washington, D.C., and then attempt to read
> eveything the day before.... Given the fact that most reviewers are
> already extremely busy and are not paid for their work, do you think
> that it would be less work to evaluate an entire track record than a
> single proposal??!!??
Yes, it may appear rather contr-logical at first glance,
but to evaluate the entire track record (life-time work)
of a researcher is EASIER, LESS ERROR PRONE, and takes
LESS time than the evaluation of a single grant
proposal (or even a single manuscript). For the latter
(propsals and manuscripts) (and assuming you want to
do your job properly) you are confined by pretty narrow
contextual margins in terms of beeing fully updated on
the topic, know ALL published literature in it and on
top of this be reasonably confident in the rightness of
In track record this is not the case. You deal with
secondary inputs (citations, etc). Even if the
assessment will be biased altogether, in a sliding
scale scenario (fund researchers, not proposals) this
will result in relatively minor funding anomalies
instead of "all-or-nothing" decisions in the present
> It seems that one runs the danger of allowing
> people to sluff off into "paper counting" with the track record
> method. Reviewers undoubtedly already do take the proposer's track
> record into account when looking at a proposal, but if that is what
> they are supposed to focus on in its entirety, the conscientious
> reviewers will be burdened with a far larger assessment task than
> passing judgement on a single proposal (the "At Hahvahd we read
> papers, not count them" syndrome), while the less conscientious ones
> will even be able to watch the in-flight movie to DC 8-) since it is
> much easier to BS about a large track record than actually address a
> concrete issue on the table. Counting the number of times papers are
> referenced would also have made heros of, e.g., the cold fusion group.
Good that you brought Pons and Fleschmann as an example.
Both people had a very strong track record prior to the
cold fusion saga. Furtheremore, the experiments they
proposed (heavy water in palladium jar) are not very
expensive. My solution in their case will be: keep funding
them on someewhat lower end of the sliding scale, but
not off the funding altogether. This would be a normal
cautious response. Unfortunately, the current system
is not prepared to deal with episodes like this in
an educated way.
> 2) In my mind the people who are really getting the shaft these days
> are young scientists who have a far less-established track record.
> Not only do postdocs get paid less than secretaries at many companies
> and have minimal benefits to boot (god help them if they decide to be
> selfish, marry and have a family ...), staking their future on a
> system that rewards the past without any consider of the quality of
> future thought makes the life of young investigators even more
> perilous. If you saw the Oct. 6th issue of Science on the Future of
> the Ph.D., many people are already questioning their sanity after
> having chosen science as a profession. Adding further burdens to them
> might seriously impact the future of science. If I wanted to be
> facetious 8-), I could even see a new kind of fraud develop, i.e.,
> posthumous proposals winning funding or professors emeriti funding
> bungalows in Tahiti with NIH money ...
Yes, I know all this and have all PhD stuff from Oct.6
copied and read. The probelm that the present NSERC/MRC-like
system (competition, grantsmanship) encourages well above
needed funding of some selected groups which in turn,
rerruit more and more grad.students without any realy
concern of what happen with these students later.
Unfortumately, we have here the Somalia effect - all these
people are already here, you feed them even more and
more (grantsmanship), they produece even more offsprings, etc.
But contrary to a general overpopulation crises,
we in science, can at least do something about it
(if we want). Reward system should shift to DE-emphasise
grad.students and number of G-students per prof should be
limited by, say, 1 PhD student at a time (no funding
should be provided beyond this level). Even this is
much higher than the replacement level, which a letter
in the last Science issue (Oct.13 ?) estimates
as 1 PhD student per prof per TWENTY (!) years.
It might appear too harsh. But if we want to feed
the family this is how far we have to go.
> Having said this, I don't mean to sound like I love the current
> system. The current system is a big reason why I switched careers
> Have fun with this! I probably won't have time to write extensive
> replies, but I'm happy to be able to provide all of you the
> opportunity to do so!!
> David Kristofferson, Ph.D., M.B.A.
> BIOSCI/bionet Manager
> Sr. Director, Scientific Market
> Strategy, Knight-Ridder
> Information, Inc.
> former Director of Govt. Business
> Development, IntelliGenetics, Inc.
> former Manager of GenBank National
> Nucleic Acid Sequence Database
> former Manager of the BIONET National
> Computer Resource for Molecular
> Extensive publication record in the
> area of Microtubule Dynamics
> PLEASE SEND MONEY TO THE FOLLOWING
> ADDRESS 8-):
> biosci-help at net.bio.net
More information about the Bioforum