HOW NIH HURTS HIV RESEARCH
bgold at itsa.ucsf.edu
Sun Jun 16 17:19:27 EST 1996
Lancet (1989) Dec. 9th. 1:281-283.
A SYSTEMS ANALYST ASKS ABOUT AIDS
DONALD R. FORSDYKE
Department of Biochemistry, Queen's University, Kingston
Ontario, Canada K7L 3N6
What might a systems analyst (SA) with HIV infection
want to know from the director of a medical research funding
SA - Thank you for agreeing to see me. I'm here because I'm
seropositive for the AIDS virus. I want to do something about it.
D - Well, our organization doesn't canvas for funds directly, but if you
are able to make a donation, that could help.
SA - I think I might be able to make a more distinctive contribution. My
original training was as a design engineer. For the past 20 years I've
been a systems analyst advising organizations, mainly in the private
sector, how to make their operations more efficient.
D - If you could apply your skills to increase the efficiency of
fund-raising for AIDS research...
SA - Only if I can be sure that it is really a shortage of funds that is
D - Well, there are many good ideas out there. We haven't enough
funds to try them all out. So we have to be selective. If we had
sufficient funds we could support more ideas and we might have an
AIDS cure very soon!
SA - My doctor tells me that there is an unpredictable latent period
before the onset of symptoms. I may only live a year, but the
chances are that I will live five or ten years or even longer.
D - Quite correct. Medical researchers have already come up with at
least one drug, AZT, which can prolong the life of AIDS patients.
SA - So I can take a long-term approach in analysing AIDS from the
D - What information do you need from me?
SA - Well, tell me how the medical research system works. New
initiatives in business and industry need, first and foremost, bright
and well informed people. In the right environment they will come
up with ideas. Then funds have to be committed to test
the ideas. People, ideas, and funds. Presumably these are also the
key components in medical research problem-solving?
D - Certainly. Having obtained advanced degrees in the biomedical
sciences, future independent researchers have to compete for one of
the scarce positions in universities or research institutes. Individuals
who successfully surmount all the hurdles must be both highly
motivated and very bright.
SA - OK. Let's assume that the particular qualities selected for by the
appointment processes are the qualities needed for creative medical
research, and that the institutions to which the researchers are
appointed have all the necessary facilities. What happens next?
D - The researchers must apply for research funds to one or more of the
funding bodies, such as that which I head. If you like, think of the
researcher as a business entrepreneur who has an idea and my
institution as an investment company that can help get the project
moving. A financier in an investment company cannot fund
everyone who applies. A successful financier has to be very shrewd
in deciding among the entrepreneurs who apply.
SA - So the research funding system is "capitalist" in philosophy to the
extent that researchers must compete with each other for a limited
quantity of funds. Even though the researchers have had to compete
with their peers to gain their positions, they must compete yet again
for the funds to test their ideas?
D - Yes. The spur of competition is probably a major factor motivating
researchers. The vigour of the western capitalist economies,
compared with that of the socialist-block countries, surely supports
SA - Hold on. Let's back up a bit. First, please tell me more about the
funding organizations. These get their funds from the public either
through taxes or as direct donations. Now, I'm interested in
accountability. If a finance company makes unwise decisions it loses
money and may become a target for a takeover. The spur of
competition, as you say, keeps the financiers on their toes just as
much as those who apply to the financiers. What are the penalties
to a research funding organization if it fails? Indeed, how is failure
or success monitored at the organizational level?
D - The spectacular advance in biomedical research over the past few
decades speaks for itself. With more funds the advance might have
been even more spectacular. We do have periodic internal reviews
of our operations and, of course, we are always seeking
input and advice on how we might improve. But there simply are not
enough funds for all the researchers. The organizations do not
compete with each other. We try to coordinate our efforts to avoid
SA - So the capitalist model does not really apply to the funding
D - We funding organizations are essentially monopolies. We enjoy this
situation because, if you like, it is a sellers' market. We "sell" our
funds to those researchers who, in our judgement, come up with the
best research proposals.
SA - The idea of researchers competing for funds has an obvious appeal
to someone with my background. But businesses and industries in
the capitalist countries work primarily for themselves and their
shareholders. Most medical researchers, as I understand it, are not
trying to become financially rich. They are trying to obtain new
knowledge which they donate freely to the nation and the world.
Just as the funding bodies are trying, as you say, to coordinate their
efforts, shouldn't the researchers be doing the same?
D - Of course, we encourage researchers to collaborate and
communicate with each other. For example, we look very favourably
upon researchers with skills in different areas
who come together and apply for funding as a group.
SA - But you still have competition, be it between individual researchers
or small groups of researchers. Clearly, as in business and industry,
you cannot have free and open communication between groups in
competition with each other. I've been wading through "Natural
Obsessions: the Search for the Oncogene" by Natalie Angier
(Houghton Mifflin, 1988). Much of it is quite above my head I'm
afraid. Please bear with me while I read from the introduction by
Lewis Thomas: "If there is any single influence that will take the
life out of research, it will be secrecy and enforced confidentiality.
The network of science. . . works only because the people involved
in research are telling each other everything that they know. . .". If
what Thomas is saying is correct, there must be a trade-off between
the spur of competition and, if you like, the spur of
communication. Perhaps this is too simple an analysis. Tell me how
the system works in practice.
D - Well, we ask researchers to submit written proposals. We allow
only 20 pages. The proposals must contain a review of the published
work, a hypothesis, and the experiments designed to test the
hypothesis. They must spell out the implications of the new
knowledge they expect to obtain and provide a detailed budget.
SA - Most successful financiers I have met place a considerable
emphasis on track record. This is relatively objective. An
entrepreneur who has come up with successful ideas in the past will
usually get support for ideas which may, at face value, not seem
D - The applicants are indeed asked to describe their past performance.
However, an applicant would penalise himself if he used up too many
pages describing past work and did not give sufficient information to
permit evaluation of the proposed work.
SA - So here we have another difference with industry and business.
The organizations funding medical research emphasise the evaluation
of future "promise" of what might be done, rather than of past
performance. Now how is this evaluation carried out? A financier
might, in confidence, consult with one or more industry analysts.
These would be people with specialist knowledge in the area of a
proposal, who themselves have a track record for giving good advice
and for not leaking ideas to potential competitors. They are paid
handsomely for their advice. If they fail, they are consulted less
frequently in future. There is a dollar penalty.
D - The medical research system is quite different. We have a system
of peer review. Copies of each application are sent to three or more
researchers with expert knowledge in the area of the application.
These reviewers are placed on their honour not to disclose the
contents of the application to others and to evaluate the proposal
objectively even though they may be advocating support of research
in competition with their own. A peer reviewer does not know who
the other reviewers are. If his reviews are consistently out of line
with those of other reviewers then he may not be consulted in
SA - If it became known that a financier were sending business
proposals to competitors for review, he would soon find a decline in
the number of proposals submitted. His business would suffer. The
system you describe would seem to work only if operated by saints.
Yet hardly a week goes by without some medical research scandal
- fraud or plagiarism -being aired in the newspapers.
D - The system is not without drawbacks. There are so many
applications to review. A class of professional reviewers does not
exist. It is paradoxical that, while the best persons to review an
application are those engaged in the same research, these same
people have the most to gain from the privileged information they are
given access to. Somehow the system works.
SA - But does it work as well as it could? A system where a competitor
has only to sit back and wait for the latest crop of bright ideas to
arrive on his desk seems wide open to abuse. There seem scarcely
any penalties for inadequate advice. For my analysis to be complete
I will need to know more about the methods both of selecting
reviewers and of monitoring the quality of their advice. However, to
save time let's say that applications have been reviewed by the
methods you describe. What happens next?
D - The applicants are given a numerical rating so that they can be
rank-ordered. Of course no system of this sort is perfect: the skills
of the reviewers are severely tried as they attempt to evaluate the
relative merits of different projects. But the rank-ordering allows us
to assign funds in a logical way. Those ranking highest get all the
funds they need to complete the work in reasonable time. Funds are
then allotted similarly to applicants with successively lower rankings
until the funds run out. Then there is a cut-off. Those below the
cut-off point get no funds. This means that many very good
applicants do not get funded.
SA - And since they are not funded, presumably they will not be able to
do the work and show whether the rating was wrong. Sharp cut-off
points in evaluation-determined allocation systems tend to turn the
evaluations into self-fulfulling prophesies. The funded succeed
because they are funded. The unfunded fail because they are
D - Do you have an alternative suggestion?
SA - The first thing an engineer wants to know when asked to design a
new system is what the system is required to and WITH WHAT
LEVEL OF ACCURACY. If the system is error-prone, as you
acknowledge the research funding system is, then this has to be
taken into account in system design. From what you tell me, the
most certain fact you have is that the person at the very top of the
rating scale is likely to be better than the person at the very bottom
of the scale. To give the person at the top everything he
or she needs and the person at the bottom nothing seems appropriate
in a competitive system. But as you move progressively down from
the top of the scale and up from the bottom of the scale, your
confidence that the rating system has properly discriminated between
the competitors must be much less. In that circumstance a
design engineer would probably come up with a sliding scale of fund
allocation, rather than a sharp-cut off point.
D - How would the sliding scale operate?
SA - Well, first a decision would be made as to how many projects were
of sufficient merit to justify support. This might eliminate the very
lowest rated projects. Then the sliding scale of funding would be
applied to the approved projects. Only those at the very top of the
funding scale would get all the funds they needed to complete the
work in a reasonable time. Those just below the top would get, say,
90% of what they needed, and so on down to the approved projects
of lowest merit which might receive only 10%.
D - But what if a project just below the top were directed at a critical
aspect of the AIDS problem? A cut-back to 90% funding would
surely slow the rate of progress towards a cure. Would you want
SA - THAT IS PRECISELY THE POINT. A design engineer would be
trying to optimise the rate of progress in the face of UNCERTAINTY
in the rating system. Maybe the project awarded only 10% funding
will be found, WITH HINDSIGHT, to have made an important
contribution to knowledge leading to an AIDS cure. The 10% of
funding will at least allow the project to move ahead, albeit very
slowly. In the absence of funding the proposed experiments might
never be performed. The research team might be disbanded and its
laboratory space allocated to others. The damage might be
D - That is an unescapable fact of a competitive system. If you like,
fund-withdrawal is a punishment. The cut-off point is a guillotine.
Fail to score above the cut-off point and it's "off with his head". The
perception of the possibility of a loss of funds should be a spur.
SA - But the punishment should fit the crime. Is it appropriate that an
applicant rated just below the cut-off point receive the same capital
sentence as an applicant at the bottom of the rating scale? And is
an applicant just above the cut-off point, knowing that his
research life hangs on a thread, more or less likely to collaborate and
communicate with others? A sliding scale would retain some
element of competition, but would make that competition fairer.
With a performance-evaluation approach, past performance would be
assessed against the funds that had been received. A person
who performed better than expected, having been given only 10%
funding, might find himself/herself getting 20% funding in the next
competition. In that way, over a period of years, individuals might
move smoothly up and down the scale until they found a level
appropriate to their abilities.
D - Your suggestion doesn't take into account the political realities.
What you are proposing is that we adapt to, that we accept, the
present low level of total system funding. A research team cut-off
from funds is visible and often vocal. In various ways it protests to
government and the general public. Take this away and you would
see total system funding shrink even more.
SA - The sliding scale would not dampen protest, it would probably
increase it. Individuals with 90% funding, who might have received
100% under the present "guillotine" system, will join the ranks of the
disenchanged. Most important of all, no longer under the shadow of
the guillotine, researchers will feel more free to follow Lewis
Thomas's imperatives and collaborate.
D- Individuals with 90% funding would probably direct their
disenchantment not at the public and the politicians but at the
funding organisations for having adopted a sliding scale in the first
SA- Yes, that would probably happen. But that is irrelevant to whether
or not a sliding scale would produce a more efficient distribution of
research funds. It's very easy for those who win in a competitive
system to accept the syllogism, "I am excellent, the system
recognises that I am excellent, therefore the system must be
excellent". One cannot expect pressure for reform to come from the
D - You came here to learn about the funding system. Your remarks
indicate that you haven't been convinced by what I've told you?
SA - Systems for organising human beings must be based on the
assumption that the decision-makers will not be saints. The system
you've described seems to be "capitalist" in spirit but contains none
of the constraints that make capitalist economic systems to vigorous,
powerful, and yes - Wall Street scandals notwithstanding -honest.
More information about the Bioforum