Citations and Quality
berezin at MCMAIL.CIS.MCMASTER.CA
Sun Jan 21 11:26:25 EST 1996
On Sun, 21 Jan 1996, Simon M. Brocklehurst wrote:
[ discussion on : 'is 90 % rate of all PR papers
been garbage/redundant, etc is a plausible
estimate or exaggeration ?' ]
> Your way of assessing the quality of a piece of work (based
> on citation searches alone) is not sensible.
You take the above out of context. I never said that
citation rate is the ONLY contributor to this assessment
( ca. 90 % redundancy rate; 'redundancy' - polite term
for garbage). There are some papers which never been cited
for many years and then produced explosion of attention and
citations (so called premature discoveries, read P. Merton).
There are also papers which have very high citations due to
the methodology they report (or summarize) but these
are not necessarilly the original papers. See recent letter
on citations by Yaalon on p. 760 of Nature, 21/28 dec. 1995.
> Surely you must realize that it is important to consider
> things like: originality; elegance of experimental design;
> clarity and completeness of the explanation of the work
> described; contribution it makes to the field; seriousness
> of any flaws in the work.
Yes, I consider all the above factors, and nonetheless
maintainn that the estimate 90 % is still quite robust.
Even the legacy of great scientists can be adequately
presented in 10 to 30 papers in many cases (for Einstein
probally in about 10). So, the publication lists of
some 300-400 lifetime papers are inflated by about the
factor of 10 (and of course, not only for 'great'
scientists) . This is the result of PEER-REVIEW DRIVEN
publish-perish syndrom, reward system in science, and
need to publish a LOT to get funding (it is virually
impossible to get grant on a basis of 1 paper, not
matter how 'good' it can be).
And remenber: NON-junk rate of ca. 10 % is still
quite high. I, personally, having published ca. 80 papers
can just humbly aspire that some 8-10 of them are not be
garbage/junk. Wish you the same with your numbers, whatever
(proportionally) they may be. And, of course, we should
not take 90/10 % interface as a sharp boundary, more
likely is is a kind of sigmoid curve.
> The number of times a paper is cited is not trivially related to
> its quality. Having said that, I am surprised by your statistics
> that indicate half of all papers are never cited, and the other
> half are cited just once.
I am sorry, this was a trivial error. The correct
about half of all PR paprer never cited at at all
about HALF OF THE REMAINING HALF is cited just
once, et cetera
Usually, citation analysis people (Garfield's institute
in Philadelphia, etc) fit this in Lotka-Bratford distibution,
number of papers having N or citations falls off an
inverse power of N (power in between 1.5 and 2). This
approximately fits the known 80/20 rule: 20 % of all
papers attract 80 % of all citations.
But again, citations are not all and the MAIJOR
couse for paper inflating are the reasons listed
> An unscientific "citation pole" for a few colleagues does not
> support this claim.
It actually works on 'my' side. Some papers are highly
cited on the Round-Robin principle (you cite me, I cite
you) and as a result you have formation of highly
cited mutual clusters. They appear (because of this
effect) as an 'important science'. The trick here is
that high activity on certain area of research tells
very little if the WHOLE area is important for the
outside. Many start to suspect that (well, almost)
all molecular biology may be such a bluff - at least
from what I see/hear/read nowdays, I don't buy your
main message - no, you don't know how 'it all works'.
Or at least, what many people will agree on nowdays
is that MB is wastely oversold, perhaps by the
factor of 10. Sorry.
In physics examples of recent bluffs when some
areas quickly and sensationally exploded to just
as quickly decay into oblivion are many. This is not
to say this is that bad (after all, science goes but
try and err), but publish-perish hysteria is greatly
damaging to the process due to the conformistic effect.
In exploding area people are coerced to copy-catting
and publish around the dominant party line, rather
than taking risks to explore alternatives, which
are most likely be pogromed by 'peer reviewers'.
Alexander A. Berezin, PhD
Department of Engineering Physics
McMaster University, Hamilton,
Ontario, Canada, L8S 4L7
tel. (905) 525-9140 ext. 24546
e-mail: BEREZIN at MCMASTER.CA
More information about the Bioforum