Peer Review: Reply to Robison
berezin at MCMAIL.CIS.MCMASTER.CA
Thu Jan 18 10:37:59 EST 1996
Dear Dr. Robison:
Some readers may likely attribute the fact that
your response below (no deletions this time) contains
quite a number of personal remarks about my (alleged)
abilities, etc indicates that your argumentation
IN ESSENSE is somewhat circular.
I provide few more comments to your point
(not all points are discussed - have not much time
On 18 Jan 1996, Keith Robison wrote:
> Alexander Berezin (berezin at MCMAIL.CIS.MCMASTER.CA) wrote:
> : (KATHY):
> : > Don't get me wrong... what I am talking about here is how
> : > much slop which does get through peer-review, accepted into
> : > the community .....
> : BEREZIN:
> : Nice point. (how much crap comes through peer review).
> : But let's make logical steps in attempt to see how
> : the whole process likely works:
> : (1) if you publish crap which is non-peer reviewed 'every
> : one instantly knows that this a crap'.
> : (2) but if same crap gets published through PR (most gets
> : through, don't worry), especially in 'prestigious' journal,
> : it's no longer a crap ... at least who dares to question ?
> UTTER HOGWASH!!!! You've apparently been doing so little
> science that you've forgotten how it works. If you publish
> in a prominent journal and say something silly, then other
> people will _line up_ with their take-downs.
Hogwash or not, but you may bother to look through
citations studies which say that about HALF of all
published papers (peer reviewed, of course) are NEVER
cited even a single time (and half of the remaining
are cited just once - often as self-citation).
Are this papers silly or not is not critically
important - most imopratnt is that they are useless
for all practical purposes (again, some exceptions
of course may happen, but these are rarities).
> If you actually
> _read_ Science, Nature, or Cell you'd notice that --
> sections titled "Scientific Correspondence", "Technical Comment",
> and "Matters Arising". If you bothered to read the
> "Minireviews" and "News & Views", you'd notice that these are
> not always glowing summaries, but frequently raise concerns
> about newly published work or older work that has been
> called into question.
> : (3) Hence the rat race to get publsihed through PR
> : INCREASES (not decreases) a drive for more and more
> : crap to print (about 90 % of all published).
> Even more evidence to suggest that 90% of all statistics
> are invented on the spot.
> : (4) It's not that most people deliberately want to publish
> : crap, but since the mechanism is that EVERYTHING which gets
> : published through PR automatically gets some kind of
> : a validation sticker (somewhat similar to a money laundring
> : process), the pressure to publish ANY-THING (and eventually
> : almost anything slips trough PR), inevitably results in a
> : very large proporton of garbage. The paradox of PR is
> : that, in average, it leads to the results EXACTLY OPPOSITE
> : to what is the alleged itention of PR (to be quality
> : control cleansing mechanism).
> : In short, PEER REVIEW SYSTEM IS THE MAIN GARBAGE GENERATOR
> : IN SCIENE, not the other way around.
> : (5) PR is essentially risk insurance. Paper in a PR journal
> : essentially re-deposits the risk from the author on-to the
> : system (special bonus - system is anonymous, so almost NO
> : attributed personal responsibility for the risk).
> Again, hogwash. Ever notice that when a paper is published
> there is a list of names over it? They're called authors.
> They are claiming they wrote the paper. They are stating
> it publically. They are taking responsibility. This has
> been abused in the past, but the general trend has been
> to be much more careful in this regard (look what happened
> to D. Baltimore; name on one questionable paper led to
> losing a plum job). If you put your name on a paper,
> you are hitching your wagon to that paper's star -- should
> it be a bad paper, and revealed as such, your reputation
> will suffer accordingly.
You are confusing relatively small fraction of papers
which went to some stage of misconduct allegations with
the solid majority of correct (ethically clean) papers
which nonetheless useless in a short AND long run.
I have not seen a single case when a 'reputation'
was spoiled by publishing numerous 'correct' but
useless papers. If you play safe you are safe.
> : (6) In the peer-less environment (no peer review, or at
> : least no in-depth PR, just relevance check and common sense),
> Pardon, but what does Tom Paine's writings have to do with
> science? Seriously, how would you define common sense?
> Isn't common sense our reflex rejection of that which
> does not conform to our previous experience, the exact
> phenomenon you claim to be trying to eliminate?
I am ashamed not to know who is Tom Paine and what
his writing have to do with science. What I am saying
about common sense is that I don't expect Cell to
publish paper on astrophusics. As simple as this.
There are many good books on common sense and how to
> : the publication will be at the SOLE risk of the author, hence
> : people will be thinking twice (or many more times) before
> : publishing.
> Most people do. And if they didn't think enough times, then
> often the reviewers force them to think again. I know -- from
> experience. I also note that the Acknowledgements in many
> papers thank the anonymous reviewers. The real world is
> much more complex than your caricature of it.
The progress in the world depend on the ability
of people to caricature it. That's what I am
trying to do - and that't what I found occasionally
> : (7) as a result, de-peer reviewisation of science will
> : REDUCE (not increase) the amount of sloppy papers and very
> : likely proportion of garbage will sharply FALL (not all
> : people will be happy with the last aspect, though).
> Experience with non-peer reviewed scientific media suggests
> how blinded you have become by zealotry. The public DNA
> sequence databanks are not peer-reviewed; they shouldn't be,
> because they are a data repository. These databanks are
> notorious for their highly uneven quality and lack of
> detail/accuracy. The Usenet is not peer-reviewed; the
> result is a very high noise-to-signal ratio. PNAS is
> pseudo peer-reviewed, and it shows; lots of weak papers
> contributed by society members.
> : PRACTICAL SUGGESTION FOR JOURNAL EDITORS:
> : As experiment, start running non-peerreviewd
> : sewctions of the journals. Papers published
> : there will be labled clearly "NOT PEER REVIWED".
> : See how many authors will opt for this
> : category as opposed to standard peer-review (APR)
> : process. Perhaps, to run a controlled experiment,
> : (for the beginning) allocate 10 % to 20 % of
> : all space for NPR section. Impose length limitations,
> : if you deem necessary, but accept EVERYTHING
> : relevant to your area on a line-up basis (first
> : come, first published - the length of line will
> : likely be eventually deternined by the waiting
> : period). Naturally, those submitting to NPR
> : can NOT submit the same to PR process.
> And of course, you will quickly have a long line for the NPR
> section. Why? Because almost everyone wants to be out quickly,
> and there would be the perception that avoiding review
> expedites publishing. Perhaps careful study of the
> time from submit-to-publish would give people reasonable
> predictions of the speed tradeoff (in an ideal "market",
> of course, the ratio of PR to NPR would be close to whatever
> gives both the same lag time).
> PRACTICAL SUGGESTION FOR APR OPPONENTS: Start your own journal.
> Set up an E-journal. Set up a site which is a central repository
> or listing of NPR E-papers. Become a journal editor and institute
> NPR. Publish your own papers by the mechanisms listed above.
Why do you think that I want/need to do any of the above ?
My reasons for the criticsm of the APR system (and PR in
general) may be quite different from what you apparently
believe they are.
> _Demonstrate_ the feasibility and side-effects of your proposals,
> rather than just making basis-free assertions. You'd be amazed
> how many people you'd convince if you actually could back it
> up with experience.
Again, not my problem. Have no need to volunteer for
the above assignment.
> Keith Robison
> Harvard University
> Department of Cellular and Developmental Biology
> Department of Genetics / HHMI
> robison at mito.harvard.edu
More information about the Bioforum