AN HONEST APPRAISAL OF PEER REVIEW

Herman Rubin hrubin at b.stat.purdue.edu
Tue Jun 18 13:23:55 EST 1996


In article <31c567fd.5270657 at usenet.interramp.com>,
Phil Koopman <koopman at cs.cmu.edu> wrote:

>Bert Gold <bgold at itsa.ucsf.edu> wrote:
>>PAPER in FASEB Journal (1993) volume 7  pages 619-621
>>On giraffes and peer review
>>D. R. FORSDYKE

>>Department of Biochemistry, Queen's University, Kingston,
>>Ontario, Canada. K7L3N6

>Peer review could also happen because the people in charge of the
>money aren't able to make the decisions themselves.  For example, with
>increased specialization it is hard to be knowledgeable enough to make
>decisions across a large field of competing requests.  In my mind this
>is a significant driving force that has to be dealt with (bicameral
>review might work -- I don't know much about it).

As you stated it, the problem is unsolvable.  That is one reason why
we need to go back to the situation we had when it pretty much did 
not arise, especially for pure research.  

The first mistake is that one can estimate the quality of real research,
as distinguished from development, by reading research proposals.  Much
of the research which leads to the significant discoveries is quite 
different from what the researcher set out to do, and the good researchers
cannot possibly say what they will be working on, except in very broad
generality.  Peer review cannot do even a fair job on this.

>I recently came from an industrial lab where the funding allocation
>changed from "benign" dictatorship to a group-grope model.  IMHO
>productivity plunged as people spent most of their time chasing a pool
>of money that diminished with each downsizing (sound familiar?).

Is this really different from the way the now largest source of funding,
the federal contracts/grants, are done?  But this was always at the
peer review level, with its attendant problems.  Leo Szilard wrote
in a science fiction story about a scenario in which the grant system,
with its proposals and peer review, was used to stifle research.  Make
it hard enough, and it will cost much more to administer than the value
of the research done.

>There are three problems with any such system:

>- How do you find and fund the important radical new idea (especially
>in the face having to sift through the chaff of poorly researched new
>ideas that are simply recastings of old tired ideas).  The best ideas
>tend to upset the entrenched Establishment.

It is often, though not always, the case that it will be seen after
a reasonable amount of development.  About the best you can do is to
encourage the 'crazy" ideas.  

I do not remember the physicists involved, but the story goes that one
physicist told another that he thought his ideas were wrong.  As I 
recall it, the statement was, "Your ideas are crazy, to be sure.  But
they are not crazy enough to be right."  Not all sufficiently crazy
ideas will be right, either.  But this IS the situation.

>- How do you break the feedback of a dead-end research topic that has
>gained momentum (and thus is supported by researchers all praising the
>others' work)?

About all you can do is not to support it.  And this might not work.

>- *ANY* system is capable of being "gamed", and will be so in the face
>of inadequate resources.  How do you minimize the resources spent
>gaming the system?  IMHO a system that tends to bring dirty laundry
>out into the open is the best way to go, although no system is
>perfect.

Minimize the gaming aspect.  We need to go back to the situation which
was effectively destroyed by grants, that of the research university.
This was the institution which would not consider hiring anyone except 
with research in mind, either a researcher, one who researchers find
useful to have around, or one who knows how to help researchers.  Then
one has to support the research out of the institutions resources; this
used to be the case, but the availability of federal funds has more than
dried up this source of funding.  Instead of universities funding research,
they collect more than the research funds from granting agencies.

We need to start the slow road back to this; no university now could take
over more than a small portion of the research in the sciences done there.
There needs to be a recognition that we need research universities, which
are to be run on the assumption that research support will be made available
to its faculty, and that this will be almost the sole criterion for faculty
advancement.  Hire good people and let them loose.

This does not mean that the government will not be able to get research help
from the universities, but it must not run the ship.  A very few now manage
to get grants on their reputations, but unfortunately many, if not most, of
the best results come from junior people.  Now, SOME senior people can get
the funding to hire junior people on speculation and fund their research,
but not too many, and this still restricts the bright young men and women.
-- 
Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
hrubin at stat.purdue.edu	 Phone: (317)494-6054	FAX: (317)494-0558



More information about the Bioforum mailing list