In my opinion,
THE problem, not only in immunological research, but in all bio-medical
research, is NOT more funds (nice though that would be), but a BETTER distrib-
ution of the large amount of funds which are available. The attached preprint,
which will be appearing in the May issue of FASEB.Journal, perhaps will make
this point clearer.
Sincerely, Don Forsdyke Discussion Leader
__________________________________________________________
ON GIRAFFES AND PEER REVIEW
D. R. FORSDYKE
Department of Biochemistry, Queen's University, Kingston,
Ontario, Canada. K7L3N6
SUMMARY
For several decades grant applications in the biomedical sciences
have been assessed by peer review. However, the design of the
peer review system was based on past precedent rather than on
recognition that a novel approach was needed. Flaws in system
design have been exposed by funding cut-backs. As a result the
research community is being torn apart.
Introduction
VACANT: one ecological nitch. WANTED: an animal that can run like
a horse, but can also nibble the most juicy leaves at the tops of
trees. If you had to design such a beast from scratch, you would
probably end up drawing a horse-like quadruped with a long neck.
You would figure that the animal should be able to hear predators
and alarm calls and you would equip it with well-hooded ears.
Since it would receive alarm calls, it should also be able to
send them. So you would equip it with a larynx. You would then
pencil in a nerve running from the brain to the larynx, a
distance of perhaps 20 cm. When checking your design against the
real world, you would find a great similarity to the giraffe.
However, the nerve to the larynx is actually several meters in
length! From the brain, it runs down the neck to the chest where
it loops round a major blood vessel and then returns up the neck
to the larynx.
Design by Revolution
The reason for this strange peregrination is quite well
understood. In the course of evolution, tissues began moving
around taking their nerve and blood supplies with them. Some
tissues migrated forward to form structures in the neck; adjacent
tissue migrated into the chest. When this happened the "wires got
crossed". A nerve got caught round a blood vessel. To solve the
problem either the blood vessel had to loop up into the neck and
then back to the chest, or the nerve had to loop down to the
chest and then back to the neck. The giraffe has not gone the way
of the dinosaurs because the length of its laryngeal nerve was
not critical for its survival. But millions of equally outrageous
evolutionary design flaws have resulted in early extinction for
the species concerned. Design by evolution is often very
inefficient. Design by evolution is always constrained by the
past. Sometimes, in human affairs, past intellectual baggage
hinders our ability to forge novel approaches. Problems which
require solution by revolution, rather than by evolution, are not
seen as such. The bold line drawn from the brain to the larynx of
your prototypic giraffe would be an example of "design by
revolution".
Present at the Creation
The origins of the modern peer review system are murky1,2. It
seems that no one ever sat down and tried to design the system
from scratch. Rather, it evolved in a piece-meal fashion. Peer
review has been with us for several decades. Yet, as currently
practiced, it threatens the renaissance in the biological
sciences that began with Darwin and Mendel and gained fresh
impetus with the discovery of the structure of our genetic
material in the 1950s1. Although historians may one day tell us
which committees and which individuals were responsible for
introducing the various aspects of the peer review process3, it
is doubtful whether we will ever know and fully understand the
factors, conscious or unconscious, which guided their
deliberations. I here offer an explanation of how the peer review
system arose in the hope that any insight provided may hasten
reform.
The system as we know it today was clearly discernable in the
late 40's when the benefits to be derived from a large public
investment in biomedical research became readily apparent.
Briefly defined, the task was to devise a system for allocating
public funds so as to harness optimally the energy, enthusiasm
and expertise of a nation's biomedical workforce to the goal of
attaining solutions to problems such as cancer, heart disease,
etc..
The design of the system appears to have been evolutionary; it
was based conceptually on other systems with which the designers
were familiar and with which they approved. Prominent among these
would have been the education system. We may assume that the
designers had all been through the education system and that the
system had been kind to them. One feature of the education system
is that a limited resource, such as access to university, is
rationed out based on one's ability to pass examinations. The
designers were all very good at examinations. A teacher had
taught them the dates of the Battle of Hastings and of the
American War of Independence. Subsequently there was a test. The
test was marked by the teacher who knew the correct dates. Then,
there was a ranking of the students based on the marks they had
received. A comforting feature of the test was that, when
repeated with different sets of questions, the previous ranking
was closely approximated. Thus it was perceived as objective and
just. Personal attributes needed to fare well in the examination
system, such as the possession of a good memory and the ability
to work hard in an organized manner, are attributes required for
many complex tasks in modern society. The examination system
worked well in allocating rewards to those who could best benefit
from the further educational opportunities needed to prepare them
for such complex tasks.
In gaining the approval of the education system, the designers
had come to accept a variety of its premises, which included: (i)
that if you want to select people with some attribute you make
them take a test, (ii) that all attributes are testable and (iii)
that tests are accurate predictors.
So, in the late 1940's, there were a number of biomedical
researchers who, by surmounting various academic obstacles, had
won positions at universities and research institutes. It was
very natural to think of asking them to write a "test" (grant
application) stating what they wanted to do and why they wanted
to do it. They had all been very good at writing tests, so did
not demur. Then there was a stumbling block. Where was the
teacher who, knowing the right answers, would mark the papers?
Thus, peer review was born. The researchers would mark each
other's papers. The loss of the authority figure (teacher) gave
the process a democratic air, which may have made it easier to
sell to the politicians. Another selling point was the notion
that the researchers would be competing with each other. Perhaps
the "spur of competition" would drive the biomedical research
system as effectively as it appeared to drive the capitalist
economic system4. Thus the designers would have drawn heavily on
analogies, not only with the educational system, but also with
the political and economic systems.
Cutbacks Reveal Flaws
And so the process began. The grant applications were written
and duly marked. Funds were awarded to those who scored highly.
For many years, as long as adequate funds chased the pool of
talent there were few complaints from the research community.
Progress was hailed by system administrators as a sign that all
was well. The fact that a train is moving ahead at 20 miles/hour
sounds great if you do not know that trains are capable of much
greater speeds. Since the same peer review system, with minor
modifications, was adopted throughout the western world, there
were no adequate controls to allow one to determine whether the
system was better than any alternative.
Then in the early 1970s came the crunch. For the first time (at
least in North America), there were insufficient funds to sustain
all the talented researchers5-7. The administrators, muttering
among themselves about the invigorating effects of heightened
competition, responded by elevating the cut-off point below which
funds would not be given. Suddenly, a new selective gate had been
imposed. Being able at research was no longer a guarantee of
getting through. A new breed of scientist began to emerge,...the
grantsmen,... people whose skills lay not so much in doing good
science, but in tuning into the perceptions of the peer group. (I
am generalising here. Fortunately a few precious individuals, we
all know who they are, escape such facile classification.)
The new selective gate also influenced the choice of the peers
who would act as gate-keepers for the rest. There had always been
a tendency to choose the "best", as defined by being successful
at doing research (and hence getting funded), to act as peer-
reviewers. The grantsmen, by definition, were now the best and
these came to dominate the peer review process. So grantsmen were
being judged by grantsmen and their expertise lay, not in being
creative scientists, but in being able to tune-in to the
perceptions of other grantsmen.
In response to mounting unrest, in the mid 1970s the US
National Institutes of Health launched a national enquiry into
the peer review system under the chairmanship of Ruth
Kirschstein. Much was said by all interested constituencies. Of
course the grantsmen were delighted with the system. We are
excellent; the system judges us as excellent; therefore the
system must be excellent. In time a multivol
