HOW NIH HURTS HIV RESEARCH

[Bert Gold]

Lancet (1989) Dec. 9th. 1:281-283.

A SYSTEMS ANALYST ASKS ABOUT AIDS
RESEARCH FUNDING


DONALD R. FORSDYKE


Department of Biochemistry, Queen's University, Kingston
Ontario, Canada  K7L 3N6





What might a systems analyst (SA) with HIV infection
want to know from the director of a medical research funding
organization (D)?



SA -   Thank you for agreeing to see me.  I'm here because I'm
       seropositive for the AIDS virus.  I want to do something about it.

D -    Well, our organization doesn't canvas for funds directly, but if you
       are able to make a donation, that could help.

SA -   I think I might be able to make a more distinctive contribution.  My
       original training was as a design engineer.  For the past 20 years I've
       been a systems analyst advising organizations, mainly in the private
       sector, how to make their operations more efficient.

D -    If you could apply your skills to increase the efficiency of
       fund-raising for AIDS research...

SA -   Only if I can be sure that it is really a shortage of funds that is
       limiting progress.

D -    Well, there are many good ideas out there.  We haven't enough
       funds to try them all out.  So we have to be selective.  If we had
       sufficient funds we could support more ideas and we might have an
       AIDS cure very soon!

SA -   My doctor tells me that there is an unpredictable latent period
       before the onset of symptoms.  I may only live a year, but the
       chances are that I will live five or ten years or even longer.

D -    Quite correct.  Medical researchers have already come up with at
       least one drug, AZT, which can prolong the life of AIDS patients.

SA -   So I can take a long-term approach in analysing AIDS from the
       systems viewpoint.

D -    What information do you need from me?

SA -   Well, tell me how the medical research system works.  New
       initiatives in business and industry need, first and foremost, bright
       and well informed people.  In the right environment they will come
       up with ideas.  Then funds have to be committed to test
       the ideas.  People, ideas, and funds.  Presumably these are also the
       key components in medical research problem-solving?

D -    Certainly.  Having obtained advanced degrees in the biomedical
       sciences, future independent researchers have to compete for one of
       the scarce positions in universities or research institutes.  Individuals
       who successfully surmount all the hurdles must be both highly
       motivated and very bright.

SA -   OK.  Let's assume that the particular qualities selected for by the
       appointment processes are the qualities needed for creative medical
       research, and that the institutions to which the researchers are
       appointed have all the necessary facilities. What happens next?

D -    The researchers must apply for research funds to one or more of the
       funding bodies, such as that which I head.  If you like, think of the
       researcher as a business entrepreneur who has an idea and my
       institution as an investment company that can help get the project
       moving.  A financier in an investment company cannot fund
       everyone who applies.  A successful financier has to be very shrewd
       in deciding among the entrepreneurs who apply.

SA -   So the research funding system is "capitalist" in philosophy to the
       extent that researchers must compete with each other for a limited
       quantity of funds.  Even though the researchers have had to compete
       with their peers to gain their positions, they must compete yet again
       for the funds to test their ideas?

D -    Yes.  The spur of competition is probably a major factor motivating
       researchers.  The vigour of the western capitalist economies,
       compared with that of the socialist-block countries, surely supports
       that?

SA -   Hold on.  Let's back up a bit.  First, please tell me more about the
       funding organizations.  These get their funds from the public either
       through taxes or as direct donations.  Now, I'm interested in
       accountability.  If a finance company makes unwise decisions it loses
       money and may become a target for a takeover.  The spur of
       competition, as you say, keeps the financiers on their toes just as
       much as those who apply to the financiers.  What are the penalties
       to a research funding organization if it fails?  Indeed, how is failure
       or success monitored at the organizational level?

D -    The spectacular advance in biomedical research over the past few
       decades speaks for itself.  With more funds the advance might have
       been even more spectacular.  We do have periodic internal reviews
       of our operations and, of course, we are always seeking
       input and advice on how we might improve.  But there simply are not
       enough funds for all the researchers.  The organizations do not
       compete with each other.  We try to coordinate our efforts to avoid
       overlap.

SA -   So the capitalist model does not really apply to the funding
       organizations themselves?

D -    We funding organizations are essentially monopolies.  We enjoy this
       situation because, if you like, it is a sellers' market.  We "sell" our
       funds to those researchers who, in our judgement, come up with the
       best research proposals.

SA -   The idea of researchers competing for funds has an obvious appeal
       to someone with my background.  But businesses and industries in
       the capitalist countries work primarily for themselves and their
       shareholders.  Most medical researchers, as I understand it, are not
       trying to become financially rich.  They are trying to obtain new
       knowledge which they donate freely to the nation and the world.
       Just as the funding bodies are trying, as you say, to coordinate their
       efforts, shouldn't the researchers be doing the same?

D -    Of course, we encourage researchers to collaborate and
       communicate with each other. For example, we look very favourably
       upon researchers with skills in different areas
       who come together and apply for funding as a group.

SA -   But you still have competition, be it between individual researchers
       or small groups of researchers.  Clearly, as in business and industry,
       you cannot have free and open communication between groups in
       competition with each other.  I've been wading through "Natural
       Obsessions: the Search for the Oncogene" by Natalie Angier
       (Houghton Mifflin, 1988). Much of it is quite above my head I'm
       afraid.  Please bear with me while I read from the introduction by
       Lewis Thomas:  "If there is any single influence that will take the
       life out of research, it will be secrecy and enforced confidentiality.
       The network of science. . . works only because the people involved
       in research are telling each other everything that they know. . .".  If
       what Thomas is saying is correct, there must be a trade-off between
       the spur of competition and, if you like, the spur of
       communication.  Perhaps this is too simple an analysis.  Tell me how
       the system works in practice.

D -    Well, we ask researchers to submit written proposals.  We allow
       only 20 pages.  The proposals must contain a review of the published
       work, a hypothesis, and the experiments designed to test the
       hypothesis.  They must spell out the implications of the new
       knowledge they expect to obtain and provide a detailed budget.

SA -   Most successful financiers I have met place a considerable
       emphasis on track record. This is relatively objective.  An
       entrepreneur who has come up with successful ideas in the past will
       usually get support for ideas which may, at face value, not seem
       very promising.

D -    The applicants are indeed asked to describe their past performance.
       However, an applicant would penalise himself if he used up too many
       pages describing past work and did not give sufficient information to
       permit evaluation of the proposed work.

SA -   So here we have another difference with industry and business.
       The organizations funding medical research emphasise the evaluation
       of future "promise" of what might be done, rather than of past
       performance.  Now how is this evaluation carried out?  A financier
       might, in confidence, consult with one or more industry analysts.
       These would be people with specialist knowledge in the area of a
       proposal, who themselves have a track record for giving good advice
       and for not leaking ideas to potential competitors.  They are paid
       handsomely for their advice.  If they fail, they are consulted less
       frequently in future.  There is a dollar penalty.

D -    The medical research system is quite different.  We have a system
       of peer review. Copies of each application are sent to three or more
       researchers with expert knowledge in the area of the application.
       These reviewers are placed on their honour not to disclose the
       contents of the application to others and to evaluate the proposal
       objectively even though they may be advocating support of research
       in competition with their own.  A peer reviewer does not know who
       the other reviewers are.  If his reviews are consistently out of line
       with those of other reviewers then he may not be consulted in
       future.

SA -   If it became known that a financier were sending business
       proposals to competitors for review, he would soon find a decline in
       the number of proposals submitted.  His business would suffer.  The
       system you describe would seem to work only if operated by saints.
       Yet hardly a week goes by without some medical research scandal
       - fraud or plagiarism -being aired in the newspapers.

D -    The system is not without drawbacks.  There are so many
       applications to review.  A class of professional reviewers does not
       exist.  It is paradoxical that, while the best persons to review an
       application are those engaged in the same research, these same
       people have the most to gain from the privileged information they are
       given access to. Somehow the system works.

SA -   But does it work as well as it could?  A system where a competitor
       has only to sit back and wait for the latest crop of bright ideas to
       arrive on his desk seems wide open to abuse.  There seem scarcely
       any penalties for inadequate advice.  For my analysis to be complete
       I will need to know more about the methods both of selecting
       reviewers and of monitoring the quality of their advice.  However, to
       save time let's say that applications have been reviewed by the
       methods you describe.  What happens next?

D -    The applicants are given a numerical rating so that they can be
       rank-ordered.  Of course no system of this sort is perfect: the skills
       of the reviewers are severely tried as they attempt to evaluate the
       relative merits of different projects.  But the rank-ordering allows us
       to assign funds in a logical way.  Those ranking highest get all the
       funds they need to complete the work in reasonable time.  Funds are
       then allotted similarly to applicants with successively lower rankings
       until the funds run out.  Then there is a cut-off.  Those below the
       cut-off point get no funds.  This means that many very good
       applicants do not get funded.

SA -   And since they are not funded, presumably they will not be able to
       do the work and show whether the rating was wrong.  Sharp cut-off
       points in evaluation-determined allocation systems tend to turn the
       evaluations into self-fulfulling prophesies.  The funded succeed
       because they are funded.  The unfunded fail because they are
       unfunded.

D -    Do you have an alternative suggestion?

SA -   The first thing an engineer wants to know when asked to design a
       new system is what the system is required to and WITH WHAT
       LEVEL OF ACCURACY.  If the system is error-prone, as you
       acknowledge the research funding system is, then this has to be
       taken into account in system design.  From what you tell me, the
       most certain fact you have is that the person at the very top of the
       rating scale is likely to be better than the person at the very bottom
       of the scale.  To give the person at the top everything he
       or she needs and the person at the bottom nothing seems appropriate
       in a competitive system.  But as you move progressively down from
       the top of the scale and up from the bottom of the scale, your
       confidence that the rating system has properly discriminated between
       the competitors must be much less.  In that circumstance a
       design engineer would probably come up with a sliding scale of fund
       allocation, rather than a sharp-cut off point.

D -    How would the sliding scale operate?

SA -   Well, first a decision would be made as to how many projects were
       of sufficient merit to justify support.  This might eliminate the very
       lowest rated projects.  Then the sliding scale of funding would be
       applied to the approved projects.  Only those at the very top of the
       funding scale would get all the funds they needed to complete the
       work in a reasonable time.  Those just below the top would get, say,
       90% of what they needed, and so on down to the approved projects
       of lowest merit which might receive only 10%.

D -    But what if a project just below the top were directed at a critical
       aspect of the AIDS problem?  A cut-back to 90% funding would
       surely slow the rate of progress towards a cure.  Would you want
       that?

SA -   THAT IS PRECISELY THE POINT.  A design engineer would be
       trying to optimise the rate of progress in the face of UNCERTAINTY
       in the rating system.  Maybe the project awarded only 10% funding
       will be found, WITH HINDSIGHT, to have made an important
       contribution to knowledge leading to an AIDS cure.  The 10% of
       funding will at least allow the project to move ahead, albeit very
       slowly.  In the absence of funding the proposed experiments might
       never be performed.  The research team might be disbanded and its
       laboratory space allocated to others.  The damage might be
       irreversible.

D -    That is an unescapable fact of a competitive system.  If you like,
       fund-withdrawal is a punishment.  The cut-off point is a guillotine.
       Fail to score above the cut-off point and it's "off with his head".  The
       perception of the possibility of a loss of funds should be a spur.

SA -   But the punishment should fit the crime.  Is it appropriate that an
       applicant rated just below the cut-off point receive the same capital
       sentence as an applicant at the bottom of the rating scale?  And is
       an applicant just above the cut-off point, knowing that his
       research life hangs on a thread, more or less likely to collaborate and
       communicate with others?  A sliding scale would retain some
       element of competition, but would make that competition fairer.
       With a performance-evaluation approach, past performance would be
       assessed against the funds that had been received.  A person
       who performed better than expected, having been given only 10%
       funding, might find himself/herself getting 20% funding in the next
       competition.  In that way, over a period of years, individuals might
       move smoothly up and down the scale until they found a level
       appropriate to their abilities.

D -    Your suggestion doesn't take into account the political realities.
       What you are proposing is that we adapt to, that we accept, the
       present low level of total system funding.  A research team cut-off
       from funds is visible and often vocal.  In various ways it protests to
       government and the general public.  Take this away and you would
       see total system funding shrink even more.

SA -   The sliding scale would not dampen protest, it would probably
       increase it.  Individuals with 90% funding, who might have received
       100% under the present "guillotine" system, will join the ranks of the
       disenchanged.  Most important of all, no longer under the shadow of
       the guillotine, researchers will feel more free to follow Lewis
       Thomas's imperatives and collaborate.

D-     Individuals with 90% funding would probably direct their
       disenchantment not at the public and the politicians but at the
       funding organisations for having adopted a sliding scale in the first
       place.

SA-    Yes, that would probably happen.  But that is irrelevant to whether
       or not a sliding scale would produce a more efficient distribution of
       research funds.  It's very easy for those who win in a competitive
       system to accept the syllogism, "I am excellent, the system
       recognises that I am excellent, therefore the system must be
       excellent".  One cannot expect pressure for reform to come from the
       top.

D -    You came here to learn about the funding system.  Your remarks
       indicate that you haven't been convinced by what I've told you?

SA -   Systems for organising human beings must be based on the
       assumption that the decision-makers will not be saints.  The system
       you've described seems to be "capitalist" in spirit but contains none
       of the constraints that make capitalist economic systems to vigorous,
       powerful, and yes - Wall Street scandals notwithstanding -honest.