Peer Review: Reply to Kathy

U27111 at uicvm.uic.edu U27111 at uicvm.uic.edu
Fri Dec 15 13:34:07 EST 1995



Sorry for delay in replying... life's been a bit hectic lately.  A
toxic waste recycling plant 3 blocks from my house had a spill and
things got a bit crazy for a while.  All is ok now and back to
normal.

...I know, it's the old toxic cloud excuse - sorry.  :-)

On Tue, 12 Dec 1995 13:13:57 -0500 (EST) Alexander Berezin wrote:

>BEREZIN:
>No, this is not quite my argument. The "lack" of something
>(e.g. in this case "good" peer review) can't be a driving
>(prime) reason for the (over)production of (whatever) garbage.
>Of course, peer review AS SUCH not a driving force. The latter
>is a VALIDATION VALUE which present-day scientific ethos puts
>on a "peer reviewed paper".

Ok... that makes more sense and I will totally agree with.

>Transition to the researcher-centerd mode of publication will
>allevieate this pressure. I would rather expect that the
>abolishing of peer review will lead to the (sharp) DECREASE in
>quantity of papers (no longer a bonus is assigned for ### of
>papers), and (overall) INCREASE of their quality, as each
>reasearcher will be much more careful in what they decide to
>publish.(publishing a lot of gargabe will quicky erode your
>reputation).

I don't know... one would think these days publishing garbage would
erode your reputation as well?

But the problem is, in many of these fields, it can takes years to
determine something is truly garbage.  And in the meantime; time,
energy, and effort is wasted following that path.

I don't know... I see your point, but I really don't see how it
would decrease the quantity. ...unless it goes hand in hand with a
new system for obtaining grants.  For unfortunately, publishing
goes hand in hand with obtaining monies.

It's all connected and one would have to reform the entire system.

>BEREZIN:
>In traditional view Peer Review is essentially (supposed to be) a
>Quality Police in Science. Yes, it is pretty functional (like
>police) in identifying incompetent work, but pretty much
>useless (like police) anywhere beyond this.

>I am not saying that police is not necessary. Of course it is. But
>it ONLY social function is to curb the crime - it does not have
>any socially constructive of "creative" role.  (who wants a
>society where the police is given a main "creative" role ?).

But I think that is exactly what some peer-review does?  When there
are giant political issues involved... what gets accepted or
unaccepted sometimes depends upon more of that journal's
'political' position rather than a scientific one.

One example I have knowledge of is again in the Origins of AIDS
debate.  Here, Elswood and Stricker developed a polio vaccine
origins theory and submitted it to the Journal of Virology (a very
prestigious journal BTW).  Initially the paper was accepted, then
once Tom Curtis wrote a lay article in the Rolling Stones on
this... it started a heated debate.  People didn't like the idea
that AIDS could've possibly came from a man-made accident.  And the
article was put on hold for an entire year.

At the end of that year (and after the Wistar Report came out)...
they allowed the authors to edit their paper into a letter to the
editor type of publication.  And then in the first time in the
history of that journal... the editors added a commentary at the
end of their small article, citing stuff from the Wistar Report
(like the case of the Manchester Sailor... which we all *now* know
was a fraud)... all to disprove the possibility of such a theory.

So see... they determine what is politically correct instead of
allowing the rest of the scientific community to read the issues
and decide for themselves.  Hell, the Wistar Report was never
published...period, just released at a press conference... let
alone never peer-reviewed.

DISCLAIMER:  just because I have followed this issue (as well as
contributed to it)... it doesn't mean I believe this theory is
correct.  I have always believed (and still do) that there was
enough here to look into the matter and investigate it... and not
just sweep it under a rug like it was.

>Same is with peer review. "More" of it will not help - as it is is
>already pretty efficient is sorting out obvious incompetence.

???  I don't know about that.  I've read some pretty obvious
incompetent, incomplete worked published.  My favorite is when you
find something you want to duplicate somewhat... and you try
following the materials and methods section and later you come to
find out some important fact was missing (this is as opposed to
finding you get different results).

And I know from personally experience, that when a procedure is
being applied for a patent... the author deliberately left
important steps out so that nobody could follow her work until the
patent went through.  I have said for a few years now that any work
being presented which has a patent involved should state as such
right in the abstract... so that people don't waste their time
trying to duplicate it!

>The problem is that most garbage published in science is
>a (reasonably) "competent" garbage and peer review is helpless to
>stop it but rather promoting ist by the reasons indicated above.


That's true.  Very true.

>> -    It's either not reviewed well

>> -    or the authors lied so well it fooled the reviewers (but in
>>      may cases that's not very hard to do?)

>BEREZIN:
>You re-state my thesis - that peer review is largely a fool's
>game. That's what I am saying all along.

But how are we then to determine what is true?

>BEREZIN:
>In the present system: why peer reviewer (unpaid for the most
>part) should bother about all the above at first place ?

???  I don't know.

>Even if we agree to postpone dismantling of peer review
>altogether, lets take a good half way at least:

>1) get peer rviewres paid for their work
>
>2) get fiscal control on their work (have random checkes on how
>they are doing by the third parties, and "fire" them if necessary)

yes... money always talks.

>3) have a provision that their reviews MAY be published
>with names attached. That does not mean that this should
>be practiced often, but the clause which GUARANTEES
>anonymity to the reviewer should be removed. Instead,
>journal editors should say that  they reserve the right
>to publish your review [ singed ] along with the paper.

Could certainly cut down on some of the politics involved.  ???

>BEREZIN:
>Peer review (like any politically loaded term) can mean a lot of
>things. What people start talking now is that interactive peer
>review (open commenting on paper) will transform it to the
>value-added process and hence will make it creative process. Then
>(but ONLY then) the above analogy with police is no longer valid
>and SUCH peer review is what I will what for.

I see you've read Judson's paper:

"Structural transformations of the sciences and the end of peer
review" (The Second International Congress on Peer Review in
Biomedical Publication) by Horace Freeland Judson v272 JAMA, July
13, 1994 p.92(3).

Article Highlights:

     "The first and most familiar of the transformations we
     can think of as internal to research and publishing; it
     comprises the declining standards and the growing, built-
     in tendency toward corruption of the peer-review and
     refereeing processes."

[Note: the author makes the distinction between peer review for
grants and published articles by referring to 'peer review' for the
grant process and 'refereeing' for journal submissions]

The other two transformations is considered external to peer
review.

II.  The transition from exponential growth of the sciences to a
     steady state.

III. The appearance and development of the electronic publishing
     and electronic collaboration.
-------

I.   Considerations of the first transformation - that peer review
     and refereeing are inherently threatened by corruption:

1.   The root of the frequency of plagiarism is the fact that the
     persons most qualified to judge a specific grant proposal or
     a submitted paper are that scientist's closest competitor.
     Besides the theft of graphs, tables and entire paragraphs; the
     theft of intellectual ideas is probably the most common.

2.   Fatigue:  "The systems are wearing out with time, breaking
     down under pressure."

     "As the number of scientists has increased so vastly in
     the last 50 years, as specialties have multiplied and
     journals so promiscuously proliferated, the familiar
     consequences has been increased competition for funding.
     On the grants side, even as the demands on the process
     have grown, the qualifications of participants, and above
     all their dedication and enthusiasm, their morale, have
     sadly waned.  What was a high and interesting duty has
     become a wearisome chore."

     "Worse, as over the years an ever-smaller proportion of
     grants get funded and as the applications themselves, in
     the top quartile, are more difficult to put in any
     reliable rank order, politics become overt in the review
     process.  Rivalries between scientists, laboratories, and
     schools of thought emerge as palpable factors.  What
     began as a means of keeping external pressures at arm's
     length has turned, to some extent at least, into a
     cockpit in which the internal politics of the sciences
     are fought out."

II.  Considerations of the second transformation - to a steady
     state:  where scientists and techs will be trained at a rate
     sufficient to replace those which die, retire or quite... all
     while funds grow no faster than the inflation of costs for
     personal, materials and facilities.

1.   The first sign of this transition is a shortage of
     funding and an intensification of competition.  Along
     with an ever increasing pressure by the government for
     directed or targeted research (ie. "national needs").

2.   The internationalization of research

3.   Greater and more complex linkages between university and
     industry:

     "Increasingly, potential profit drives the direction of
     research"

The end result of this transition... both industry and academic
research will increasingly be judged by the evaluation of outcomes.

     "For those in science, evaluation of outcomes will mean
     evaluation of their work by nonscientists, evaluation at
     end points rather than prospectively or in mid-process,
     evaluation according to new criteria over which
     scientists will have far less control."

III. Considerations of the third transformation - electronic
     publishing and collaboration:

1.   "it  offers the only possible remedy to the problems of
     sorting out from the vast scientific literature all the
     articles, but only the articles, that are directly relevant to
     the individual's work."

2.   "it will allow one to record, retain, and keep accessible
     one's own responses-notes, commentary linkages, inspired
     ideas- to those articles."

3.   It can help to overcome the problem of lag time between
     submission of an article to publication.

4.   There will no longer be a need to 'condense' or 'simplify' an
     article.

5.   The use of bulletin boards:

     "The communities-they are being called "collaborators"-
     that are coming to use Internet in this and related ways
     are, in aggregate, potentially numerous, international,
     and highly active.  Yet the number of scientists in any
     one such subset may number a dozen, two score, 200.  The
     members of such a group, competing and simultaneously
     collaborating are each and collectively one another's
     peers, doing in effect their own reviewing, not blinded
     or anonymously but open and in a manner that concatenates
     publication and responses."

"Eventually-but sooner than you can easily imagine-we will see an
evolution toward a form of publication that will be a continuing
open dialogue and collaboration among contributing scientists,
editors, expert commentators, and readers."
***************************

>(KATHY):
>> To cut down on the politics... we need to create a more ethical
>> environment (where ethics is put back on the top of the list!).
>> To do that, we need to standardized the field... certified
>> researchers in accredited laboratories.

>BEREZIN:
>Perhaps, though it's likely debatable. There are stiong
>PROs and CONs on the issue of sectification od scientists.
>Amer. Phys.Soc. been milling the idea of "chartered physicist"
>for some time. No definite action, as far as I know.

I don't know... I think it's similar to your idea of paying people
to be reviewers.  It's how the money is controlled which will make
a difference.

For instead of grants depending upon quantity of publications and
who you know (rather than what you know)... if we made it so that
only accredited labs with certified techs were even allowed to
apply - grants would then depend upon 'what you know' and how well
your lab is ran.  Ie. less garbage and slop would get funded and
there would be more monies available for those who do good work.

>BEREZIN:
>That's OK. Of course, no system can eliminate sloppiness,
>neither it should. Life without cheaters would be quite
>boring. I would be easy on the above, that't not major.

I don't know... I tend to think that is one of the *major* problems
with AIDS research (and somewhat in cancer)... too much slop and
garbage.  I mean, we are talking about people's lives depending
upon how efficient this work is done.  ???

I'll take boring any day over watching more people die.

-Kathy



More information about the Bioforum mailing list