Peer Review: Reply to Kathy

Alexander Berezin berezin at MCMAIL.CIS.MCMASTER.CA
Tue Dec 12 13:33:17 EST 1995

On Tue, 12 Dec 1995 U27111 at UICVM.CC.UIC.EDU wrote:

> Alexander A. Berezin, PhD wrote:
> >> ckrepel at (Candace Krepel) wrote on 8 Dec 1995
> >> 08:41:52 -0600:
> >
> >> >As I am sure you know, the scientific method requires that the
> >> >investigator control for all factors other than the one being
> >> >tested.
> >(KATHY):
> >> And I'm sure you know just how much garbage does get published
> >> under the guise that the scientific method was followed!
> >Garbage in science (all disciplines) gets published in
> >astronomical quantities BECAUSE of peer review and
> >resulting from it paper counting mentality (it is
> >"prestigeous" to publish "peer reviewed" paper in
> >"prestigeous" journal - no matter the content or "who
> >reads it ?").
> I disagree...
> I think a lot of garbage gets published because:
> 1.   The lack of peer review... how many of the articles in Medline
>      (for example) are peer-reviewed in the first place?  I seemed
>      to recall reading somewhere that only about half (or a little
>      more than half) of them are... ???

No, this is not quite my argument. The "lack" of something 
(e.g. in this case "good" peer review) can't be a driving
(prime) reason for the (over)production of (whatever) garbage.
Of course, peer review AS SUCH not a driving force. The latter
is a VALIDATION VALUE which present-day scientific ethos puts
on a "peer reviewed paper". Transition to the researcher-centerd
mode of publication will allevieate this pressure. I would
rather expect that the abolishing of peer review will lead to
the (sharp) DECREASE in quantity of papers (no longer a
bonus is assigned for ### of papers), and (overall) INCREASE
of their quality, as each reasearcher will be much more 
careful in what they decide to publish.(publishing a lot
of gargabe will quicky erode your reputation).     

> 2.   Or because when there is peer review...

In traditional view Peer Review is essentially
(supposed to be) a Quality Police in Science.
Yes, it is pretty functional (like police) in
identifying incompetent work, but pretty much
useless (like police) anywhere beyond this.

I am not saying that police is not necessary.
Of course it is. But it ONLY social function
is to curb the crime - it does not have any
socially constructive of "creative" role.
(who wants a society where the police is given 
a main "creative" role ?).

Same is with peer review. "More" of it will not
help - as it is is already pretty efficient
is sorting out obvious incompetence. The problem
is that most garbage published in science is
a (reasonably) "competent" garbage and peer review
is helpless to stop it but rather promoting ist
by the reasons indicated above.     

> -    It's either not reviewed well
> -    or the authors lied so well it fooled the reviewers (but in
>      may cases that's not very hard to do?)

You re-state my thesis - that peer review is 
largely a fool's game. That's what I am saying
all along. 

> -    or the reviewers themselves are biased and only care to allow
>      data/materials to get published which only agrees with their
>      own ideas or models of what is supposed to be going on (thus,
>      not really caring how credible it may really be... just that
>      it is more ammunition to support their own claims or political
>      objectives);
> -    or, if someone is already 'established',... does the reviewers
>      actually bother to question the validity/credibility of this
>      'latest' paper presented for peer-review? ie., resting on
>      one's laurels.

In the present system: why peer reviewer (unpaid
for the most part) should bother about all the 
above at first place ?

Even if we agree to postpone dismantling of peer review 
altogether, lets take a good half way at least:

1) get peer rviewres paid for their work

2) get fiscal control on their work (have random
checkes on how they are doing by the third parties,
and "fire" them if necessary)

3) have a provision that their reviews MAY be published
with names attached. That does not mean that this should
be practiced often, but the clause which GUARANTEES
anonymity to the reviewer should be removed. Instead,
journal editors should say that  they reserve the right 
to publish your review [ singed ] along with the paper.  

For those who are competent and honest there is
no fear in the above changes. 

> -    And finally, what of conflict of interest problems?  How do we
>      know the reviewers for a paper on the effectiveness of a new
>      drug\treatment don't have personal interest at stake (stock in
>      the company, receives grants from that company, or in the case
>      of that one journal editor - he received monies from the
>      company involved in a lawsuit for being an expert witness in
>      the court proceedings for them... he rejected two articles
>      which supported the claims made by those who initiated the
>      lawsuit while published an article he wrote himself which
>      supported the company's position on the matter)
> >"The more the better" is the prime principle in modern science
> >(after it became an industry), and I've heard things like
> >"... he must be a dead wood because he publishers ONLY two
> >papers per year ..." (Copernicus had one in a whole life).
> Quantity over quality is a problem due to the competition for grant
> monies... ie. the 'publish or perish' syndrome.  And *that* is the
> reason for the present paper counting mentality we see.

BEREZIN: Agree. This is what I said above. 

> I don't think this is necessarily a direct problem with peer-review
> in it self?
> It does result in too may papers/too little time problems which
> probably effects the caliber of the peer-view process in general.
> But I really think it's really more of a combination of problems...
> with the major one being politics!
> >Only the scrapping of anonymous peer review system and its
> >replacemet by a system of open commenting to all published
> >(electronically and/or paper-bound) papers will relieve
> >science from publish-perish trap and release the creative
> >energy of scientists towards solving REAL problems instead
> >spending their lives to fight pseudo-problems of prestige,
> >grantsmanship, "competition" and other crap.

> Now here I disagree... instead of scrapping the system entirely...
> I think we just need to make the process better.  It could be
> improved by creating a more strict conflict of interest definition
> and enforcing it (possibly by legal measures?); by making the
> authors of the papers presented anonymous to those reviewing it;
> and by cutting down on the amount of politics involved.

Peer review (like any politically loaded term) can mean
a lot of things. What people start talking now is that 
interactive peer review (open commenting on paper) will
transform it to the value-added process and hence will
make it creative process. Then (but ONLY then) the above
analogy with police is no longer valid and SUCH peer review
is what I will what for. 
But we have tremendous barriers in its implementing
(despite that technically it is rather easy to do now).  

> To cut down on the politics... we need to create a more ethical
> environment (where ethics is put back on the top of the list!).  To
> do that, we need to standardized the field... certified researchers
> in accredited laboratories. 

Perhaps, though it's likely debatable. There are stiong
PROs and CONs on the issue of sectification od scientists.
Amer. Phys.Soc. been milling the idea of "chartered physicist"
for some time. No definite action, as far as I know. 

> People may still lie about their work
> presented (but no matter what system you create, some will always
> try to find a way around it)... but it would definitely cut down on
> the slop being done in research which gets presented as 'good
> work'.  For a sloppy lab would have a very hard time getting
> accredited in the first place... let alone maintaining one.

That's OK. Of course, no system can eliminate sloppiness,
neither it should. Life without cheaters would be quite
boring. I would be easy on the above, that't not major.

> As for 'open' comments system you purpose... I personally believe
> people, in general, are just too apathetic to care either way.  And
> it would most likely result with those with the loudest voices as
> the only ones being heard in the end.  Not much different then we
> have today actually?

Will all PROs and CONs on complete openness I am on a PRO
side. I don't mind even if all salaries of all people will
be published in public directories (no "unlisting" options).

> -Kathy

More information about the Bioforum mailing list