Peer Review and Quality

Keith Robison robison at lipid.harvard.edu
Thu Jan 18 08:37:30 EST 1996


Alexander Berezin (berezin at MCMAIL.CIS.MCMASTER.CA) wrote:

: (KATHY):

: > Don't get me wrong... what I am talking about here is how 
: > much slop which does get through peer-review, accepted into 
: > the community .....

: BEREZIN:

: Nice point. (how much crap comes through peer review).
: But let's make logical steps in attempt to see how 
: the whole process likely works:

: (1) if you publish crap which is non-peer reviewed 'every 
: one instantly knows that this a crap'.

: (2) but if same crap gets published through PR (most gets
: through, don't worry), especially in 'prestigious' journal, 
: it's no longer a crap ... at least who dares to question ?

UTTER HOGWASH!!!!  You've apparently been doing so little
science that you've forgotten how it works.  If you publish
in a prominent journal and say something silly, then other
people will _line up_ with their take-downs.  If you actually
_read_ Science, Nature, or Cell you'd notice that -- 
sections titled "Scientific Correspondence", "Technical Comment",
and "Matters Arising".  If you bothered to read the
"Minireviews" and "News & Views", you'd notice that these are
not always glowing summaries, but frequently raise concerns
about newly published work or older work that has been 
called into question.

: (3) Hence the rat race to get publsihed through PR 
: INCREASES (not decreases) a drive for more and more 
: crap to print (about 90 % of all published).
                       ^^^^
Even more evidence to suggest that 90% of all statistics
are invented on the spot.  

: (4) It's not that most people deliberately want to publish 
: crap, but since the mechanism is that EVERYTHING which gets 
: published through PR automatically gets some kind of
: a validation sticker (somewhat similar to a money laundring 
: process), the pressure to publish ANY-THING (and eventually 
: almost anything slips trough PR), inevitably results in a
: very large proporton of garbage.  The paradox of PR is 
: that, in average, it leads to the results EXACTLY OPPOSITE
: to what is the alleged itention of PR (to be quality
: control cleansing mechanism).  

: In short, PEER REVIEW SYSTEM IS THE MAIN GARBAGE GENERATOR 
: IN SCIENE, not the other way around.   

: (5) PR is essentially risk insurance. Paper in a PR journal
: essentially re-deposits the risk from the author on-to the 
: system (special bonus - system is anonymous, so almost NO 
: attributed personal responsibility for the risk). 

Again, hogwash.  Ever notice that when a paper is published
there is a list of names over it?  They're called authors.
They are claiming they wrote the paper.  They are stating
it publically.  They are taking responsibility.  This has
been abused in the past, but the general trend has been
to be much more careful in this regard (look what happened
to D. Baltimore; name on one questionable paper led to
losing a plum job).  If you put your name on a paper,
you are hitching your wagon to that paper's star -- should
it be a bad paper, and revealed as such, your reputation
will suffer accordingly. 

: (6) In the peer-less environment (no peer review, or at 
: least no in-depth PR, just relevance check and common sense), 

Pardon, but what does Tom Paine's writings have to do with
science?  Seriously, how would you define common sense?
Isn't common sense our reflex rejection of that which
does not conform to our previous experience, the exact
phenomenon you claim to be trying to eliminate?

: the publication will be at the SOLE risk of the author, hence
: people will be thinking twice (or many more times) before
: publishing.

Most people do.  And if they didn't think enough times, then
often the reviewers force them to think again.  I know -- from
experience.  I also note that the Acknowledgements in many
papers thank the anonymous reviewers.  The real world is
much more complex than your caricature of it.

: (7) as a result, de-peer reviewisation of science will 
: REDUCE (not increase) the amount of sloppy papers and very 
: likely proportion of garbage will sharply FALL (not all 
: people will be happy with the last aspect, though).             

Experience with non-peer reviewed scientific media suggests
how blinded you have become by zealotry.  The public DNA
sequence databanks are not peer-reviewed; they shouldn't be,
because they are a data repository.  These databanks are 
notorious for their highly uneven quality and lack of
detail/accuracy.  The Usenet is not peer-reviewed; the
result is a very high noise-to-signal ratio.  PNAS is
pseudo peer-reviewed, and it shows; lots of weak papers
contributed by society members.


: PRACTICAL SUGGESTION FOR JOURNAL EDITORS:

: As experiment, start running non-peerreviewd
: sewctions of the journals. Papers published
: there will be labled clearly "NOT PEER REVIWED".
: See how many authors will opt for this
: category as opposed to standard peer-review (APR)
: process. Perhaps, to run a controlled experiment,
: (for the beginning) allocate 10 % to 20 % of
: all space for NPR section. Impose length limitations,
: if you deem necessary, but accept EVERYTHING 
: relevant to your area on a line-up basis (first
: come, first published - the length of line will
: likely be eventually deternined by the waiting
: period). Naturally, those submitting to NPR
: can NOT submit the same to PR process.

And of course, you will quickly have a long line for the NPR
section.  Why?  Because almost everyone wants to be out quickly,
and there would be the perception that avoiding review 
expedites publishing.  Perhaps careful study of the
time from submit-to-publish would give people reasonable
predictions of the speed tradeoff (in an ideal "market",
of course, the ratio of PR to NPR would be close to whatever
gives both the same lag time).

PRACTICAL SUGGESTION FOR APR OPPONENTS:  Start your own journal.
Set up an E-journal.  Set up a site which is a central repository
or listing of NPR E-papers.  Become a journal editor and institute
NPR.  Publish your own papers by the mechanisms listed above.
_Demonstrate_ the feasibility and side-effects of your proposals,
rather than just making basis-free assertions.  You'd be amazed
how many people you'd convince if you actually could back it
up with experience.  

Keith Robison
Harvard University
Department of Cellular and Developmental Biology
Department of Genetics / HHMI

robison at mito.harvard.edu 











More information about the Bioforum mailing list