Citations and Quality

Alexander Berezin berezin at MCMAIL.CIS.MCMASTER.CA
Sun Jan 21 11:26:25 EST 1996

On Sun, 21 Jan 1996, Simon M. Brocklehurst wrote:

[ discussion on : 'is 90 % rate of all PR papers
   been garbage/redundant, etc is a plausible
   estimate or exaggeration ?' ] 
>   Your way of assessing the quality of a piece of work (based
> on citation searches alone) is not sensible.

You take the above out of context. I never said that 
citation rate is the ONLY contributor to this assessment
( ca. 90 % redundancy rate; 'redundancy' - polite term
for garbage). There are some papers which never been cited 
for many years and then produced explosion of attention and 
citations (so called premature discoveries, read P. Merton). 
There are also papers which have very high citations due to 
the methodology they report (or summarize) but these
are not necessarilly the original papers. See recent letter 
on citations by Yaalon on p. 760 of Nature, 21/28 dec. 1995.  

> Surely you must realize that it is important to consider 
> things like: originality; elegance of experimental design; 
> clarity and completeness of the explanation of the work 
> described; contribution it makes to the field; seriousness 
> of any  flaws in the work.

Yes, I consider all the above factors, and nonetheless
maintainn that the estimate 90 % is still quite robust. 
Even the legacy of great scientists can be adequately 
presented in 10 to 30 papers in many cases (for Einstein 
probally in about 10). So, the publication lists of 
some 300-400 lifetime papers are inflated by about the
factor of 10 (and of course, not only for 'great'
scientists) . This is the result of PEER-REVIEW DRIVEN 
publish-perish syndrom, reward system in science, and
need to publish a LOT to get funding (it is virually
impossible to get grant on a basis of 1 paper, not 
matter how 'good' it can be).

And remenber: NON-junk rate of ca. 10 % is still
quite high. I, personally, having published ca. 80 papers 
can just humbly aspire that some 8-10 of them are not be 
garbage/junk. Wish you the same with your numbers, whatever 
(proportionally) they may be. And, of course, we should
not take 90/10 % interface as a sharp boundary, more
likely is is a kind of sigmoid curve.

>    The number of times a paper is cited is not trivially related to 
> its quality. Having said that, I am surprised by your statistics 
> that indicate half of all papers are never cited, and the other 
> half are cited just once.

I am sorry, this was a trivial error. The correct
  about half of all PR paprer never cited at at all
  about HALF OF THE REMAINING HALF is cited just 
  once, et cetera

Usually, citation analysis people (Garfield's institute
in Philadelphia, etc) fit this in Lotka-Bratford distibution,
number of papers having N or citations falls off an
inverse power of N (power in between 1.5 and 2). This
approximately fits the known 80/20 rule: 20 % of all
papers attract 80 % of all citations.

But again, citations are not all and the MAIJOR
couse for paper inflating are the reasons listed

 > An unscientific "citation pole" for a few colleagues does not 
> support this claim.

It actually works on 'my' side. Some papers are highly 
cited on the Round-Robin principle (you cite me, I cite
you) and as a result you have formation of highly
cited mutual clusters. They appear (because of this
effect) as an 'important science'. The trick here is 
that high activity on certain area of research tells 
very little if the WHOLE area is important for the 
outside. Many start to suspect that (well, almost)
all molecular biology may be such a bluff - at least
from what I see/hear/read nowdays, I don't buy your
main message - no, you don't know how 'it all works'. 
Or at least, what many people will agree on nowdays
is that MB is wastely oversold, perhaps by the
factor of 10. Sorry.

In physics examples of recent bluffs when some 
areas quickly and sensationally exploded to just
as quickly decay into oblivion are many. This is not 
to say this is that bad (after all, science goes but 
try and err), but publish-perish hysteria is greatly 
damaging to the process due to the conformistic effect. 

In exploding area people are coerced to copy-catting 
and publish around the dominant party line, rather 
than taking risks to explore alternatives, which
are most likely be pogromed by 'peer reviewers'.

Alexander A. Berezin, PhD
Department of Engineering Physics
McMaster University, Hamilton,
Ontario, Canada, L8S 4L7
tel. (905) 525-9140 ext. 24546

More information about the Bioforum mailing list