View from the Trenches
berezin at MCMAIL.CIS.MCMASTER.CA
Sat Aug 5 12:44:40 EST 1995
On 4 Aug 1995, William Tivol wrote:
> Dear Alexander,
> > The above may be true for some highly specific projects with
> > well spelled goals. The work you decribe above sounds more
> > like R&D company development, rather than open-ended research.
> True, but appropriate for the small grant applied for.
> > Much (I believe, MOST) university research does not fit the
> > above.
> Possibly a more typical example is a grant (not yet
> applied for) to develop electron crystallography on a high
> voltage electron microscope. Like yours, the core is a set
> of interrelated ideas; however, they *can* be properly form-
> ulated at this time. There has been enough done to know how
> to go about getting and analysing the data. That is not to
> say that surprizes will not arise during the course of the
We don't have any real disagreement on the above.
Basic funding to all research active (not welfare)
will render most of these problems non-existing
(unless you plan VERY expensive experiment(s))
> > The whole point that it is much more often than
> > not that there is NO WAY to say beforehands if the
> > idea is "good" or not. Was the idea of Pons and
> > Fleishmann to search for nuclear reactions in
> > deuterated palladium electrodes "good" or "bad" ?
> An easy one. A few calculations (done subsequently
> by others) show that there is no reason to expect the ex-
> perimental set-up of Pons & Fleishmann to produce nuclear
> Any competant study section would call this a
> bad idea *unless* there was convincing evidence from pre-
> liminary experiments that the conventional wisdom was in-
This is the whole problem. What means "any competent
section" ? Were professors refusing to look thru Galileo's
'devile's tube' a "competent section" ? (MAJORITY
(all the "establishemnt") refused). I can make
a VERY long list, but here are just two items:
1611: "there are no spots on the Sun - "WE KNOW THIS"
1989" "there are no nuclear reactions at low (chemically
induced) energies - "WE KNOW THIS"
Were do you see the difference ?
> If P & F had applied to NSF for a grant for cold
> fusion with no calculations showing the feasibility of pro-
> ducing nuclear reactions at room temperature and no results
> suggesting that they would be successful, they would not
> get a grant for cold fusion,
Neither anybody will get grant to study
high-temperature superconductivity in
barium-copper oxides PRIOR to (larglely
serendepedous) finding of Bendros and Muller
> again *unless* the study sec-
> tion decided that they knew P & F were so competant that
> the grant should be awarded.
Perhaps, but more likely if they are part
of establishement, rather that actual competence.
"It is much more important whom you know, than what you
know" - in my 30 years in science I have yet to see
a single violation of this principle.
> My point is that the bad
> idea should overrule the decision based on who P & F are,
> or who they know.
Again, you are stucked with "bad idea" undefinability.
> > Despite all the b/s thrown on them during 1989-1995
> > we still don't know the answer. What we have are
> > at best opinions, highly biased for the most part.
> I believe the results from Caltech do provide the
> answer. Those experiments are more than opinions.
You believe. I don't. (actually, it is good
that you said "I _believe_" - it shows were
we ALL are). Why not agree that the differences
in opinions, beliefs and expectations in science
is a perfectly normal matter (perhaps, even
Again, basic grants regardless of what "we"
(committee) may think, will help here a lot.
> > This is why the
> > "track record evaluation" as described in
> > more detailes by Donald Forsdyke asks referees
> > (composed not only from seniors, but with a
> > good share of juniors) to state:
> > "What such and such ACTUALLY accomplished in
> > science". And you may end up with somebody
> > with 20 papers coming out better than 300+
> > graphomaniac.
> I should read Donald Forsdyke's scheme for evaluation.
> It is more difficult to evaluate track records this way, but
> isn't doing things the correct way often more difficult? Try-
> ing to evaluate how much of the work the PI did and how much
> should be credited to (say) a postdoc is not going to be easy.
> If a PI is good at gathering and directing good people, so the
> lab is productive, is it a good idea to fund the lab just on
> that basis? I'd guess "yes".
Good point to argue about. What I see around quite
often is this:
"Big boss - big group". Typical grantsmanship
latifundia with 10 (or more) cheap slaves. Often
VERY TALANTED slaves (esp. PDFs). Natuarally,
good products. Even if boss himself is a (largely)
copycat, people tend to say, " ok, ok, but he
is a good organizer..., let the empire go..."
Because, the loop here is [ mostly ] closed
(more postdocs - more results - more funding -
- more postdocs --- ad infinum ...), the
whole who-is-who-ing gets highlty contentious.
Some cases are clear, but some (perhaps, the
most) are practically unanalysable at ANY
> > > If you are arguing
> > > for a granting system like the McArthur Foundation grants, which
> > > are unrestricted money, I agree that there is a place for them, and
> > > perhaps all research funding should be in the form of unrestricted
> > > grants plus equipment grants, etc., so that competant researchers
> > > can be assured of funding and they can then generate any ideas
> > > they want.
> > I have never said such nonsense. My point is how to
> > the get the best mileage from the (very) lean budgets.
> > Sorry, but the true science is NEVER done by
> > (materially) happy scientists.
> "Unrestricted" need not mean "big". Small grants to provide
> supplies for research on any topic the recipient chooses, with sup-
> plementary funds for expensive equipment was what I have in mind.
> I don't think it's nonsense.
Than I misuderstood you. Of course, it will be a heaven if
the system can start with "small grants to provide supplies
for research on any topic the recipient chooses"
(even if limited to QUALIFIED [ by track record ] recipients,
still [ almost ] a paradise)
> > >
> > The above is not convincing example for me. High energy
> > physics (contary to most non-nuclear physics) is for a
> > long time done by very large groups of people [ papers
> > with several HUNDERD co-authors are not that rare).
> The Segre-Chamberlain group consisted of about ten people.
> That was about the right number for the 50's to 60's. My own re-
> search at LBL was in few-nucleon experiments, with about six people.
> This provided enough for two or three people for each 12 hour shift,
> needed for the 24 hour-a-day operation of the cyclotron. Other
> branches of physics probably lead to different group sizes, but I'm
> not familiar with them.
Good all days !
> >Well, of course,
> > you have Leon Lederman, Carlo Rubia, etc but even they
> > did not manage to save Supercollider [ I personally,
> > believe - for good ].
> The only bad part of the supercollider is the cost. The buil-
Not the "only". More importantly, its highly
politiciesed nature. I have reservations that
SSC (contrary to NASA) was a good idea even
at FUNDAMENTAL level (I wrote about this
in "Physics World", December, 1993 p. 19).
But take it easy. On this (SSC) yours and
mine are just opinions. Let's keep them
> ding of the supercollider is, in fact, amply justified on scientific
> grounds, and it would already be built if it could be done cheaply.
> As it is, I agree that that much money can be used more effectively.
> > As Don Forsdyke wrote in one of his papers:
> > "to put your best ideas in the propoasls is
> > to committ professional suicide. And this
> > is not because referees can pirate your ideas
> > (although this may happen), but because best
> > ideas [ really innovative ] are almost certainly
> > will not be understood by peers"
> Yes, I've seen this.
> > (his recent paper in "Accountability in
> > Research").
> > Those who are norally successful in getting
> > funded write proposals EXCLUSIVELY as students
> > often write their term essays:
> > "what 'they' [ reviwers ] want to see in my
> > proposals which will likely satisfy them the most".
> > (appologies before STUDENTS for the above
> > comparision - actually, term essays are normally
> > a much cleaner deal, while grantsmanship is
> > almost invariably asslicking).
> Here we agree on the problem.
> > Again, to say that "proposal has merit"
> > is a vicious circle strategy. If we so
> > much disagree on the (value of) research
> > ALREADY DONE AND PUBLISHED, how we can
> > say anything reliable about the merit of
> > work YET TO BE DONE. This is simply
> > a nonsense.
> I think you're wrong here. To use your example of the
> supercollider, its scientific merits can be reasonably evaluated.
Highly disputable. (in view of the costs and
questionability of results. again, it is
a matter of opinion - is Higgs boson worth
$ 11 B ? - ask Bill Gates, he is better on
> A study section can determine both the merit (will it work) and
> the value (what good is it when it works) of a proposal, compare
> these to the cost, and come to a reasonable decision. BTW, if
> P & F wanted a small grant to explore the possibility of cold fu-
> sion, were able to provide calculations of D-atom density in Pd
> and the expected D-D reaction rates, and stated that the idea,
> while perhaps improbable, held out the possibility of great
> reward, I'd be inclined to fund the work.
Me too. But not the present NSF/NSERC/NIH, etc.
> It is no more difficult to evaluate the merits of re-
> search already done and published, than it is to evaluate the
> track record of a researcher based on what is already done &
> published--the latter is, after all, only a subset of the former.
Yes, this is what I am saying. They are ALMOST
identical (track record additionally may
include not-published activities, which
sometime [ but by no means always ] can be
credited) - point is rel, minor anyway.
> > In our system (Canadian NSERC) it is largely
> > irrelevant if you were or not assessed properly.
> Not so.
Better, I ask those who been kicked out
if they agree on the above.
> > As I have explained ours in a SINGLE GRANT system
> > working on a SELECTIVITY principle. Every year
> > it leave about 2,500 professors COMPLETLY unfunded.
> Although the numbers of unfunded professors and the
> nature of the all-or-nothing process leave a lot to be desired,
> If you were over-assessed, you will be funded, and if under-as-
> sessed, you will not.
The case-base which I have does not support it.
In NSERC system people are kicked out more or
> IMHO, the accuracy of assessment is very
If you go to a robut ranking (or track record
and hence basic grant), this is NOT very
important. "accuracy" here is FUNDAMENTALLY
has high uncertaintly. I would consider use
of fuzzy logic in this case.
> It is also true that the system you describe has
> fundamental flaws unrelated to assessment.
> > > Of course,
> > > if researchers are not funded, their ideas will not be implemented
> > > (at least by them).
> > FORTUNATELY, NOT SO. People doing mostly theoretical,
> > or computer work can get by paying for most critical
> > expenses (hardware, newtworking, publishing, etc)
> > from their personal salaries [ not tax-deductable,
> > though ]. I do know a case of a biochemist who
> > re-mortgaged his hourse to continue on-going research,
> Not necessarily true here. Often, promotion and tenure decisions
> are based on successful grant funding. Our institution recently put in
> a requirement that all PI's must have grant funding for their research,
> otherwise their PI status is removed and their research is directed by a
> supervisor, or they can be removed from research altogether. This holds
> true for theoretical work as well. Even re-mortgaging one's house (I as-
> sume not horse :-)) will not be sufficient unless one were to work on one's
> own time using one's own equipment.
No principle disagreement on the above.
Of course, science is money.
> > There are no really unsolvable problems in the
> > implementing all the above (e.g. young scientists, etc)
> > within the sliding scheme(s), track record review, etc.
> > Even quantifyable schemes were suggested (e.g. Rostom Roy),
> > to minimize subjectivity/bias factors.
> Yes, but these schemes are not used. Instead, it's who-you-know.
> If the track record review was to be done correctly--that is, the highest-
> rated would be the most productive
Be careful with the "most productive". Of what:
papers ? "discoveries" ? ...
However, ranking of "impact factor" is not that
>--then funding those with the best track
> records would make good sense. However, this always seems to decend into
> the political.
> > > My
> > > contention is that the detailed plans will still work, regardless of who
> > > proposes them.
> > I am sorry to disagree - my experience that it is very
> > much depends WHO handles the idea. Again, the above
> > example from high energy phisics may not be the
> > best here (due to the diminished human factor in very
> > large groups) but for the rest (of science) I certainly
> > would object your "regardless".
> The nub of science is that the outcome of an
> experiment is independent of the observer.
The sheer bibliography on the above statement will
take me a few days to collect. The issue of
reproducibility and (in)dependency of the observer
is getting hotter and hotter. Again, I see no
problem with variety of opinions on this (actually,
happy to see it) - but it has little to do with
how to alocate base funding [ at least SHOULD
have little to do ]
> I know some people who would not be able to implement an
> idea, but they would not be able to write up the detailed
> plans either.
Idea(s) for which you are able to write a "detailed
plan" how to implement them will mostly not worth
too much. After all, science is [ supposed to be ]
a "journey into the unknown"
> My contention is that anyone who can set out an
> experimental protocol which is convincing to a study
> section should be allowed the opportunity to pursue it.
Normally, you go to a (new) dentist on the
basis of recommendations (i.e. how he did it
the past). To put an emphasis on assessment of
experimental protocol is a sheer nonsense of
NIH+ process : leave it up to a researcher,
and if he fails : too bad, track record
(and subsequent funding) will suffer. But
at least YOU took the risk, and YOU pay for it.
> > In general terms, it is a rather philosophical
> > discussion: ("are DIS-covering the laws of nature
> > or CREATE them" ?).
> I'm on the side of the existance of objective
> reality; thus, nature is what it is, and we are discovering
> what already exists.
If all people could agree on a single
formula for the above, the philosophy would
be extinct centuries ago.
Thanks God (or "Universal Consciousness",
or "objective reality", or whatever)
we would NEVER have this debate closed
(same as "meaning of life" or other
[ Again, wonderful to discuss all this
with you, but has and SHOULD NOT have
any bearing on funding issue ]
> > >
> > Yes, of course. But in many cases it takes not
> > too much. Plus there is a lot of cooking books
> > around.
> If I can set out a protocol, and if I use a cookbook (or send out
> the samples to have gels run) won't I get results which can be duplicated
> by anyone else following the same procedures? Isn't that what makes for
> good science? If the ideas have both merit and value, who cares which
> person did the work? If I am confident that any lab can reproduce the
> same results, how does it matter that the researcher has a Nobel Prize?
For me the above is much more convoluted.
But again, matter of opinion (or rather
a peronal taste). Little relevancy to funding.
> > Yes, these are the pipe dreams we all seem to share.
> > Fine. But there is absolutely no way to move anywhere
> > from "competition to cooperation" (with all
> > qualifications on these words) unless the system
> > recognizes some kind of basic funding for ALL active
> > researches,
> Or, at least, many.
Yes, I of course, agree with some qualifications
for what means "ALL". No police wants to eliminate
crime completely as then they will be out of job.
> >at least those who are in the univeristy
> > system
> Salaries and supply $ used to be provided
> by universities, and it would be a good idea if this were done again.
> If necessary, give block grants to each university to accomplish this.
Great ! Let's make it to the first page of
> > anyone having 3 peer reviewd papers
> > for the last 5 years is entitled on
> > basic grant of $ 3,000 per year (US)
> > or $ 4,000 (Can).
> Not too tough for PI's, especially if account is taken of other
> measures of productivity (for example, no papers written because of a
> published text, not pre-peer-reviewed, but generally accepted as good).
> I also assume that straight-forward arrangements can be made for cases
> where productivity is not in doubt, but the numbers aren't there.
No problem. All the above can sorted out.
> > Highly primitive; LOOKS like
> > welfare (but not, it is NOT welfare),
> > but at least give us a point to start.
> It is payment for services rendered; i.e. money to do research
> based on research done over the past 5 years. Neither primitive nor
See it this way, if you prefer (pay for past service).
Ready to take, for as long as a scheme for basic funding
will function as suggested. Prostitutes are paid in
advance, most others after the services. Where we see
ourselves is a matter of personal preference, and I
am we will find a mixture of views here.
> Bill Tivol
More information about the Bioforum