[Journal-notes] OA Testimonial from BBS Author

Stevan Harnad harnad at ecs.soton.ac.uk
Sat Sep 17 12:41:40 EST 2005

    [ MODERATOR'S NOTE: Posted for Seth Roberts, with permission.

    Some background information about BBS and CUP (Behavioral and Brain
    Sciences, published by Cambridge University Press): 
    http://www.ecs.soton.ac.uk/~harnad/Temp/bbs.valedict.html ]

---------- Forwarded message ----------
Date: Sat, 17 Sep 2005 09:52:40 -0700
From: Seth Roberts <roberts AT berkeley.edu>
To: Stevan Harnad <harnad AT ecs.soton.ac.uk>
Subject: Re: Unofficial BBS Disposition Letter

Dear Stevan,

I'm afraid I am having quite a bit of trouble figuring out how to post
it. So I hereby give you permission to post it.  Thanks.


>On Sat, 17 Sep 2005, Seth Roberts wrote:
>> Dear Dr. Harnad:
>> Here is one concrete effect of your belief in open access: After my
>> paper (below) appeared in BBS, 


>> my friend Andrew Gelman blogged about it;
>> and readers of his blog could read my paper because BBS had allowed
>> me to put it in a University-of-California repository. It was one of
>> the first papers in that repository, in fact. One of Andrew's readers
>> was Alex Tabarrok, who wrote very favorably about it in his blog,
>> Marginal Revolution. This got the attention of Stephen Dubner, who with
>> Steven Levitt writes a column called Freakonomics in the New York Times
>> Magazine. An article based on my BBS paper (see www.freakonomics.com for
>> details) appeared last Sunday but even before then I had been contacted
>> by Dubner and Levitt's agent -- now my agent, too -- who suggested I not
>> only write a diet book (which will probably be called The Shangri-La Diet)
>> but also a second book about self-experimentation (working title The
>> Science of One). There is great interest among publishers; and although my
>> colleagues have been very skeptical of my self-experimentation, the
>> rest of the thinking world, at least as reflected in comments on the
>> freakonomics website, is enthusiastic.
>> My BBS paper has been downloaded over 6000 times, 75% of them in the last week.
>> I have little doubt that both books will eventually exist and reach
>> large audiences.
>> It happened a lot sooner due to open access.
>> Seth Roberts
>> At 09:43 AM 8/31/2001, Stevan Harnad wrote:
>> >Dear Dr. Roberts:
>> >
>> >Here is an unofficial version of the reports and disposition.
>> >BBS will shortly send you and the referees an official one.
>> >
>> >Sincerely,
>> >
>> >Stevan Harnad
>> >Editor, BBS
>> >
>> >--------------
>> >
>> >Appended below are the 5 BBS referee reports on your manuscript:
>> >"Self-Experimentation as a Source of New Ideas..."
>> >
>> >The reports indicate that your manuscript is potentially acceptable if
>> >you can successfully revise it in accordance with the referees'
>> >recommendations (which I will summarize below). BBS policy under
>> >these conditions is that the revised manuscript must be re-refereed and
>> >must be accompanied by a detailed, itemized statement of how and where
>> >in the revised draft each referee's specific points AS CAPITALIZED AND
>> >NUMBERED IN THE REFEREE REPORTS have been accommodated.
>> >
>> >To focus your revision, here is a summary of the CAPITALIZED points in
>> >the reports that particularly call for attention, according to the
>> >numbered categories in the composite ratings that follow the last
>> >referee report below.
>> >
>> >
>> >The topic is potentially significant enough for BBS Commentary but
>> >the presentation has to be made stronger, tighter and more rigorous
>> >if it is to elicit useful Commentary.
>> >
>> >
>> >There are many problems with the presentation as it now stands. It is
>> >not sufficiently well-organized or focussed. It wanders, repeats
>> >itself, contains unnecessary material and omits necessary material.
>> >Referees indicate that many points are unclear, needing more detail and
>> >fuller explanation. The figures and tables also need a good deal of
>> >work to make them self-contained, clear and informative.
>> >
>> >
>> >The referees draw a good deal of existing work to your attention that
>> >you need to take into account. More attention also needs to be given to
>> >the historical precursors of your approach.
>> >
>> >[4] EVIDENCE:
>> >
>> >More statistical details and testing are needed for your single-subject
>> >data. The referees also raise a number of important methodological
>> >points that will require clarification, amplification, and perhaps some
>> >further empirical work.
>> >
>> >[5] REASONING:
>> >
>> >The referees raise a number of prima facie questions and objections
>> >regarding some of your inferences and conclusions. Not all of these
>> >need to be decisively replied to in the target article, but please do
>> >address the more important ones. Also, all outright errors pointed out
>> >by the referees should be corrected, rather than being allowed to be
>> >pointed out again in the Commentary. Similarly, points that require
>> >further explanation or justification should be strengthened here rather
>> >than left to be raised anew by the commentators.
>> >
>> >[6] THEORY:
>> >
>> >You are asked by the referees to amplify and elaborate a number of
>> >conceptual, theoretical and methodological points in order to provide
>> >more rigor as well as substance for eliciting constructive commentary.
>> >
>> >[7] LENGTH:
>> >
>> >Although the referees' critiques and recommendations will require
>> >adding to the paper, they do also stress that the paper currently has
>> >extraneous material and repetition, and that the present material does
>> >not justify the length. So please strike a balance between trimming and
>> >tightening the present material, and adding the requisite further
>> >material to strengthen it in response to {3] - [6].
>> >
>> >For the details, please see the 5 thoughtful referee reports.
>> >
>> >I hope you will accept the challenge to revise your paper. Most
>> >ultimately accepted BBS papers first undergo major revision. This is
>> >necessary not only to ensure the quality of BBS target articles,
>> >but also to protect authors from running the gauntlet of open peer
>> >commentary before being adequately forearmed. The "mini-treatment"
>> >consisting of the BBS referee reports tends to provide a fair
>> >sample of what a paper is likely to encounter in Commentary; and so
>> >experience has dictated that to elicit commentary that is constructive
>> >and useful to the author as well as to the field, a paper must fully
>> >accommodate these prima facie criticisms in advance.
>> >
>> >I would also like to recommend that, before resubmitting to BBS, you
>> >take advantage of a new intermediate medium for "test-piloting"
>> >material that is being prepared for BBS. Psycoloquy is BBS's electronic
>> >counterpart: a refereed electronic journal sponsored by the American
>> >psychological Association that specialises in shorter target articles
>> >for open peer commentary in much more rapid, global and interactive
>> >form (dubbed "scholarly skywriting") than the print medium permits.
>> >Target articles accepted by Psycoloquy are immediately edited, archived
>> >on the Web, and circulated around the world to the journal's
>> >readership, who may then submit comments (which likewise appear as soon
>> >as they have been refereed and accepted). Psycoloquy Commentary can be
>> >very valuable in revising a BBS target article. More and more BBS
>> >target articles follow successful Psycoloquy treatments (Koehler, Wright,
>> >Pulvermueller, Fitch & Denenberg, Glenberg, etc.).
>> >
>> >Please inform us of how you intend to proceed, and according to what
>> >timetable.
>> >
>> >Sincerely,
>> >
>> >
>> >Stevan Harnad
>> >Editor, BBS
>> >
>> >
>> >========================================================================
>> >Seth Roberts    <roberts at socrates.berkeley.edu>
>> >
>> >"Self-Experimentation as a Source of New Ideas:
>> >Ten Examples About Sleep, Mood, Health, and Weight"
>> >========================================================================
>> >
>> >Referee #1 ANON
>> >
>> >I read the abstract and glanced at the article.  I did not find it
>> >comfortable to read the article in full.  I do not think it is suitable
>> >for regular review because of EXCESSIVE LENGTH [2], DISORGANIZATION OF
>> >EXPERIMENTS [4,5].
>> >
>> >
>> >========================================================================
>> >Referee #2
>> >
>> >The paper proposes that using oneself as the subject of experimental
>> >investigation is a good means of generating new ideas for further
>> >investigation.  This argument is supported by descriptions of ten
>> >studies (five concerning sleep, activity and mood, and five concerning
>> >dietary interventions), which the author has conducted on himself over
>> >a period of many years.  The results from these two sets of studies are
>> >explained using a theory of stone-age living and a theory of
>> >taste-calorie associations.
>> >
>> >The paper is one of the most stimulating that I have ever reviewed and
>> >as such proves its point concerning the method as a means of idea
>> >generation.  The dedication and investment that the author has put into
>> >the studies is also quite remarkable.  I found many of the results
>> >intriguing and I concur with the notion that the behavioral sciences
>> >would benefit from more understanding and methods concerning idea
>> >generation.  In terms of suitability for BBS, the paper is certainly
>> >likely to provoke interesting comment from various quarters (although
>> >this is in part due to its OVER- BROAD AIMS [1,2,5] -- see below).
>> >However, this 'labour of love' is in need of a MAJOR OVERHAUL TO GIVE
>> >IT FOCUS, COHERENCE AND BREVITY [2,5,7].  The LENGTH [7] of the paper
>> >is currently twice that recommended for BBS I believe.  The author
>> >argues that 'innovation benefits from diverse goals' but unfortunately
>> >there is probably material for 3 papers here: self- experimentation for
>> >generating ideas, and the two sets of studies.
>> >
>> >1) If the author really wishes to make this a paper concerning the use
>> >of self-experimentation then MANY OF THE DETAILS WITHIN THE EXAMPLE
>> >STUDIES ARE UNNECESSARY [4].  The author would also need to FURTHER
>> >FOR FURTHER EXPERIMENTS [4,6].  Alternatively the author could use
>> >either of the two sets of studies as the focus (although some of the
>> >second set appear to have been published before) and make the
>> >methodology secondary.  At present there is a danger that some very
>> >interesting preliminary findings (e.g., morning faces and sugar-water)
>> >
>> >2) One problem I had in reviewing this paper is that it is DIFFICULT TO
>> >of the examples DO NOT HAVE THE EXPERIMENTAL RIGOUR [4,5] that would
>> >normally be expected but presumably the author would argue that their
>> >purpose is simply to generate ideas for further testing.  Perhaps the
>> >[4,6].  More generally, it raises the question of WHETHER CURRENT
>> >
>> >3) Nevertheless, there appears to be a real danger that SOME OF THE
>> >SAME TIME [4,5].  For example, in 1996 the author conducted self-
>> >experiments on faces, water and legumes, and in 1997 conducted
>> >self-experiments on faces, standing, walking and pasta.  A diary
>> >showing the timing of the 10 studies might help allay these fears.
>> >There is also a related concern that these RESULTS MIGHT BE
>> >IDIOSYNCRATIC [4,5].  For example, the author had persistent sleep
>> >problems and was constantly juggling with his routine.  What effect
>> >might this have had on the results?
>> >
>> >Turning next to the details of the paper.
>> >
>> >4) Introduction on missing methods.  I'm not convinced that concepts
>> >concerning idea generation are quite as missing as the author
>> >suggests.  For example, SIMONTON'S WORK [3] on creative productivity
>> >(including his variation-selection model) and WEICK'S WORK [5] on
>> >theory construction as disciplined imagination come to mind.  The
>> >notion that expert tools for idea generation will resemble those for
>> >idea testing also seems MORE LIKELY TO BE A CHOICE THAN A NECESSITY
>> >[5].
>> >
>> >5) Introduction on self experimentation.  The author should CLARIFY IN
>> >[2,5,6].  Although it is claimed that one of the main advantages of
>> >self-experimentation is that it is easy to do, it struck me that MANY
>> >INVESTMENT [5].  The claim that discovery is always unexpected may be
>> >true by definition but this does not mean that some expectations were
>> >not involved and confirmed in the example studies.
>> >
>> >6) Stone-age living.  I am uncomfortable with the notion that our
>> >brains and bodies worked well in the Stone Age (indeed mortality rates
>> >might suggest otherwise).  NATURAL SELECTION does not optimise
>> >components of living systems, it ACTS ON SYSTEMS AS A WHOLE [5] to
>> >produce better adapted systems.  Hence MANY ASPECTS OF HUMAN PSYCHOLOGY
>> >[5].  This type of theory tends to provide POST HOC EXPLANATIONS [5]
>> >i.e., the description of Stone Age living CAN BE ADJUSTED TO FIT THE
>> >RESULTS [4,5].
>> >
>> >7)  Example 1, breakfast.  It is suggested that the lights could have
>> >produced a 90 minute oscillator, but alternatively they MAY HAVE
>> >and activity cycle).
>> >
>> >8)  Example 2, faces.  A problem in this example and elsewhere is that
>> >INTEGRATED WITH THE FINDINGS [2,4,6].  Missing from the related results
>> >is recent research that has found a CIRCADIAN RHYTHM IN HAPPY MOOD
>> >(e.g., BOIVIN ET AL., 1997) [3,4].  However, this rhythm is usually
>> >only apparent under certain conditions such as irregular sleep-wake
>> >cycles and depression (e.g., TOTTERDELL, 1995) [3].  At other times,
>> >the rhythm is masked by other factors.  Related to this, circadian
>> >rhythms usually only explain a small amount of the variance in mood.
>> >This suggests that the circadian effect may not have a strong influence
>> >on everyday living (now or in the Stone Age).
>> >
>> >The author has related his results on faces to those concerning
>> >depression.  However, the effects he found on overall mood were based
>> >THE TRIGGER EVENT [4,5].  Exposure to faces in the morning and evening
>> >might also cancel each other out.  Is it not likely that the related
>> >results concerning depression were caused by irregular sleep-wake
>> >cycles (e.g., staying up late to watch TV) rather than by the faces on
>> >TV.  Is there evidence to support the claim that modern living enforces
>> >greater variation in time of arising?  Don't most people start
>> >school/work about the same time and therefore arise about the same
>> >time?  Commuter trains/buses might also involve much exposure to early
>> >morning faces.  It also wasn't clear to me that morning exposure to
>> >faces prevents insomnia -- the author's awakenings persisted, for
>> >example.
>> >
>> >9) Example 3, standing.  Models of alertness (e.g. FOLKARD AND
>> >AKERSTEDT [3]) suggest that alertness is determined by the combined
>> >influence of an endogenous circadian component C, an exogenous sleep
>> >need component S which is recuperated during sleep but builds up with
>> >time awake, and a wake-up component W.  Given that time standing was
>> >correlated with time awake and that it was associated with reduced
>> >sleep duration, could its effect on feeling rested actually be due to
>> >changes in S (as an artefact or otherwise) and even C?  Perhaps the
>> >
>> >10)  Example 4,  morning light.  The description suggests that the
>> >experiment was conducted near a window, so does the No- Light Condition
>> >really mean NO LIGHT OR DOES IT MEAN NO-LAMPS [4,5] (but with
>> >sunlight).  The author is quite right in pointing out that most
>> >research has concentrated on phase rather than amplitude effects.
>> >
>> >11)  Example 5, colds.  There are some interesting observations here.
>> >
>> >12)  Pavlovian weight control.  Although the principles outlined in the
>> >second set of studies seemed plausible to me, I am not sufficiently
>> >expert on dietary factors to judge the details of the example studies.
>> >AMOUNT EATEN [4,5] in example 6 concerning water or in example 9
>> >concerning sushi.  How can the explanation of the Ramirez study (in
>> >example 10) that the rats gained weight because of increased water
>> >EXAMPLE 6 [5,6] ?  The ideas that water erases flavor memories and that
>> >sweetness doesn't raise the set point also seem HIGHLY SPECULATIVE
>> >[4,5] (but not necessarily wrong).
>> >
>> >13)  General discussion.  The AUTHOR'S CATEGORIES FOR THE EXAMPLES SEEM
>> >SYSTEM BE DEVISED? [4,6]  There also seems to be some IRRELEVANT
>> >MATERIAL [2,7], for example the discussion of JEP.  The author calls
>> >for methods that can test many possible causes at one time, but what
>> >about the problem that there will be a HIGH NUMBER OF FALSE POSITIVES
>> >[5]?  The author has observed that the self- experiments all involve
>> >self-regulatory processes.  Could it therefore be the case that this
>> >method is useful because it is suitable for addressing the sorts of
>> >phenomenon (e.g., temporal processes) that most experiments cannot or
>> >do not address.  Similarly perhaps it is good for generating
>> >applications because it addresses everyday practical matters.
>> >
>> >14)  There are QUITE A FEW MINOR ERRORS IN THE TEXT [2] (e.g., figure
>> >12 duplication in caption, Table 4 missing words).  The use of hyphens
>> >is also unusual.
>> >
>> >
>> >In summary, I liked much of the contents of the paper and would like to
>> >see many of the ideas and results published.  However, I think the
>> >paper is in need of an extensive overhaul.  In terms of a commentary,
>> >points 8 and 9 might form the basis for a commentary.
>> >
>> >
>> >========================================================================
>> >Referee #3
>> >
>> >This reviewer very strongly supports publication of this paper,
>> >A PROPER DISCUSSION [2,4,5,6]. The author could thus set an example for
>> >many others to follow in sleep and much broader research with
>> >relatively simple tools, complementing the current use of expensive
>> >devices during a night in the sleep lab and continuing with the use of
>> >simple tools to collect data amenable to the resolution of infradian,
>> >such as about half-weekly or weekly variation (1-5). The main point
>> >that self-experimentation can give new ideas, COULD BE DOCUMENTED WITH
>> >BY SEQUENTIAL TESTING, BY CUMULATIVE SUMS [4,5] (6) or otherwise. It
>> >would be a pity if others had to take data off graphs that may involve
>> >considerable error in so doing.
>> >
>> >Credit is given to the few instances where the author mentions
>> >statistics, albeit not applied to the rhythmic functions he
>> >investigates. A general comment is that the author should TEST WHATEVER
>> >PARAMETER ESTIMATIONS [4,5], such as those on page 19. The author also
>> >deals with periodic functions and any INFRADIAN RHYTHMS, notably weekly
>> >ones, characterizing, for instance, sleep and wakefulness SHOULD BE
>> >RESOLVED [4,5].  He could assess circaseptan parameters on different
>> >regimens by TESTS OF THE ZERO AMPLITUDE ASSUMPTION [4,5], and if that
>> >can be rejected, by SUBSEQUENT PARAMETER ESTIMATIONS [4] (7). Changes
>> >in parameters are revealed by PARAMETER COMPARISONS [4] (8). The
>> >ADDRESSED IN EACH CASE [5], whether or not it can be answered
>> >definitively.
>> >
>> >The author deserves great credit for studying himself in the medical
>> >tradition of the physician Santorio, who recorded his body weight for
>> >30 years, in keeping with many others in chronobiology, where
>> >self-study, self-measurement and self-rating has been the rule, even
>> >when it involves drawing blood every 90 minutes, day and night for 2
>> >days (9) or much longer. Some individuals have measured ten or more
>> >body functions around the clock for years, more than one even for
>> >decades (10, 11). The more rigorous documentation of the manipulations
>> >of body weight could benefit from his consulting publications that
>> >showed the stage-dependence of the effect of meals, which he has
>> >apparently not examined (12, 13).
>> >
>> >The author may wish to discard his statement that medical uses of
>> >self-experimentation, on p.  3,  2, "have also been quite brief,
>> >typically also for a few days"; it is AT VARIANCE WITH SELF-STUDIES,
>> >[4,3,5] (9), but continued in several cases for months or covering over
>> >3 decades, as noted. On p. 8 and on p. 16, about social
>> >synchronization, the author may wish to CONSULT REFERENCE 14 [3]. The
>> >so that statistical test results complement eyeballing. In summary, I
>> >congratulate the author for following the medical tradition of another
>> >era. I am glad to waive anonymity and am ready to serve him if he
>> >should be interested.
>> >
>> >1. Beau J., Carlier M., Duyme M., Capron C., Perez-Diaz F. Procedure to
>> >extract a weekly pattern of performance of human reaction time.
>> >Perceptual and Motor Skills 88: 469-483, 1999.
>> >
>> >2. Hildebrandt G., Bandt-Reges I. Chronobiologie in der Naturheilkunde:
>> >Grundlagen der Circaseptanperiodik. Haug, Heidelberg, 1992, 102 pp.
>> >
>> >3. D'rer L. Rhythm and proliferation with special reference to the
>> >six-day rhythm of blood leukocyte count. Neoplasma 7: 117-134, 1960.
>> >
>> >4. Halberg F., Engeli M., Hamburger C., Hillman D. Spectral resolution
>> >of low-frequency, small-amplitude rhythms in excreted 17-ketosteroid;
>> >probable androgen induced circaseptan desychronization. Acta
>> >endocrinol. (Kbh.) Suppl. 103, 5-54, 1965.
>> >
>> >5. Halberg F. The week in phylogeny and ontogeny: opportunities for
>> >oncology. In vivo 9:  269-278, 1995.
>> >
>> >6. Corn'lissen G., Halberg F., Hawkins D., Otsuka K., Henke W.
>> >Individual assessment of antihypertensive response by self-starting
>> >cumulative sums. J. Medical Engineering & Technology 21: 111-120,
>> >1997.
>> >
>> >7. Corn'lissen G., Halberg F. Chronomedicine. In: Encyclopedia of
>> >Biostatistics, Armitage P., Colton T. (editors-in-chief), v. 1, John
>> >Wiley & Sons Ltd., Chichester, UK, 1998, pp. 642-649.
>> >
>> >8. Bingham C., Arbogast B., Corn'lissen Guillaume G., Lee J.K., Halberg
>> >F. Inferential statistical methods for estimating and comparing cosinor
>> >parameters. Chronobiologia 9:  397-439, 1982.
>> >
>> >9. Halberg F., Visscher M.B., Flink E.B., Berge K., Bock F. Diurnal
>> >rhythmic changes in blood eosinophil levels in health and in certain
>> >diseases. Journal-Lancet (Minneapolis) 1951; 71: 312-319.
>> >
>> >10. Halberg F, Corn'lissen G, Otsuka K, Watanabe Y, Katinas GS, Burioka
>> >N, Delyukov A, Gorgo Y, Zhao ZY, Weydahl A, Sothern RB, Siegelova J,
>> >Fiser B, Dusek J, Syutkina EV, Perfetto F, Tarquini R, Singh RB, Rhees
>> >B, Lofstrom D, Lofstrom P, Johnson PWC, Schwartzkopff O, International
>> >BIOCOS Study Group. Cross-spectrally coherent ~10.5- and 21-year
>> >biological and physical cycles, magnetic storms and myocardial
>> >infarctions.  Neuroendocrinol Lett 2000; 21: 233-258.
>> >
>> >11. Corn'lissen G., Halberg F., Schwartzkopff O., Delmore P., Katinas
>> >G., Hunter D., Tarquini B., Tarquini R., Perfetto F., Watanabe Y.,
>> >Otsuka K. Chronomes, time structures, for chronobioengineering for "a
>> >full life". Biomedical Instrumentation & Technology 1999; 33: 152-187.
>> >
>> >12. Halberg F., Haus E., Corn'lissen G. From biologic rhythms to
>> >chronomes relevant for nutrition. In: Not Eating Enough: Overcoming
>> >Underconsumption of Military Operational Rations, Marriott B.M. (ed.),
>> >National Academy Press, Washington D.C., 1995, pp. 361- 372.
>> >
>> >13. Halberg F. From aniatrotoxicosis and aniatrosepsis toward
>> >chronotherapy: Introductory remarks to the 1974 Capri Symposium on
>> >timing and toxicity: the necessity for relating treatment to bodily
>> >rhythms. In: Chronobiological Aspects of Endocrinology, J. Aschoff, F.
>> >Ceresa, F. Halberg eds., F.K. Schattauer Verlag, Stuttgart, 1974, pp.
>> >1-34.
>> >
>> >14. Apfelbaum M., Reinberg A., Nillus P., Halberg F. Rythmes circadiens
>> >de l'alternance veille-sommeil pendant l'isolement souterrain de sept
>> >jeunes femmes. Presse m'd. 77: 879-882,1969.
>> >
>> >As to ratings it is clearly A with the above descriptive, not
>> >prescriptive, commentary
>> >
>> >
>> >========================================================================
>> >Referee #4 ANON
>> >
>> >I think that "Self-experimentation as a source of ideas" is an
>> >interesting and important paper.  It is particularly appropriate for
>> >BBS because it is likely to arouse intense commentary - both positive
>> >and negative - from a wide range of disciplines.  Psychology, and
>> >surely other scientific areas implicated in these arguments, is often
>> >reluctant to spread its wings and search out alternative ways of
>> >looking at research topics - and, according to Roberts' argument, for
>> >ways of identifying research topics.  In this paper Roberts has drawn
>> >upon insights, research, confirmation and challenge from a remarkably
>> >wide range of disciplines and sub-disciplines.  Bringing together such
>> >a wide range of research findings is also likely to make the paper
>> >interesting and provocative to numerous individuals.
>> >
>> >I like the organization followed in each of the sections: The
>> >self-experimentation Example; the presentation of Related Results,
>> >drawing from a wide range of research areas; finally, the Discussion,
>> >which pulls these findings and the research together.  I don't think
>> >this can be improved.
>> >
>> >Possible problems with the paper:
>> >
>> >The paper is TOO LONG [2,7] for my tastes.  The length has the
>> >advantages of making sub-elements of the paper quite clear.  Methods
>> >that have not been fully spelled out in earlier Roberts' papers are
>> >described in enough detail that replication would be possible (at least
>> >to a first pass).  However, replication seems only a secondary goal of
>> >the paper.  Roberts' primary goal is to persuade that
>> >self-experimentation is as an overlooked or undervalued method - and
>> >the details with which the paper is laden may be an obstacle to
>> >achieving that goal.
>> >
>> >A related difficulty is that in his eagerness to lay out the wealth of
>> >ideas that self-experimentation has generated (certainly a goal
>> >consistent with the paper's purpose) he DESCRIBES A LESS STRONG
>> >EXPERIMENT OR TWO [4,5].  He also includes a wealth of TABLES THAT DO
>> >NOT ADD [2,4] fundamentally to his core issue.  Such complexity does
>> >not weaken the overall argument, but it adds to the length, makes it
>> >more difficult to follow and does not strengthen the major arguments.
>> >
>> >Roberts, one would guess, knows from experience that critics will
>> >attack the issue of "expectations" as having a significant impact on
>> >outcomes.  Such arguments occur with all self-experimentation studies.
>> >He deals with this issue throughout the paper protesting perhaps too
>> >much.  I think he should DEAL WITH THE ISSUE ONCE AND MOVE ON [2,7] (if
>> >this is possible).
>> >
>> >Commentary
>> >
>> >Roberts is clearly NOT THE ONLY ONE [3] who has engaged in
>> >self-experimentation in areas of health, wealth and happiness.  The
>> >present reviewer, and undoubtedly thousands of others, makes more or
>> >less coherent records (some lasting for many decades) of modifications
>> >in diets on weight, of imposition of drugs on happiness, weight and
>> >other measures.  What distinguished Roberts from this large company is
>> >the care with which records his data (three scales!!), his shrewd
>> >ability to select the critical variable and the critical intervention,
>> >and his ability to impose upon himself experimental conditions few
>> >would find it easy or even possible to comply with.  I am led, however,
>> >to a vision of a research method in which hundreds or thousands of
>> >individuals amassed their data on a common topic (more on this below).
>> >
>> >Although Roberts' arguments are quite comprehensive (sometimes almost
>> >overwhelmingly so), he does omit (or bury) one theme -the issue of
>> >INDIVIDUAL DIFFERENCES [5].  To take only one example: fascinated by
>> >his mention of fructose water as a weight loss mechanism, I implemented
>> >a modest level of fructose in my drinking water.  Although my beginning
>> >concentrations were considerably lower than even his last and lowest
>> >level (I was concerned about blood sugar levels) I encountered a very
>> >different reaction - pains in joints that are ordinarily all pain
>> >free.  Using a standard design of fructose, no-fructose, etc., in a
>> >matter of two weeks it was clearly demonstrable that I have a strong
>> >sensitivity to fructose (at least in the powered form I used).  This
>> >relationship (like a number of Roberts appeared to be causal) and like
>> >many of his, was quite surprising.  Moreover, like Roberts I am able to
>> >speak authoritatively about pain in joints because over a 20-year
>> >period I have monitored the health of individual joints on each hand
>> >(codes for each joint, a variety of indicators of pain - at least three
>> >measures on each joint, etc.).  Distinctive patterns of pain accompany
>> >certain additives in food.  Eliminating the offending additive
>> >eliminates the difficulty.  Adding the offending element back into the
>> >diet immediately (within hours) reproduces the difficulty.  I
>> >assiduously avoid any food that produces a measurable reaction.
>> >
>> >In summary, there are probably enormous individual differences in
>> >reaction to foods, as well as to other elements in Roberts' fascinating
>> >programs.  This is not an argument against his approach.  In fact, an
>> >approach such as his, with an informed citizenry contributing data, may
>> >be the only way for us to ever recognize the full range of differences
>> >among individuals on health and psychological dimensions.  Birdsource
>> >(www.birdsource.com <http://www.birdsource.com>) permits individuals
>> >throughout the country to make contributions to observations on the
>> >locations of birds.  Enormously more is known about the migration of
>> >species than can be learned from other more formal scientific sources.
>> >I can imagine and much approve of a new contribution to health issues
>> >provided by an expanded application of Roberts' method.
>> >
>> >========================================================================
>> >Referee #5
>> >
>> >A. General comments and evaluation
>> >
>> >This paper describes a body of ingenious, systematic, and risky
>> >research conducted over more than a decade.  It is focused on the use
>> >of self-experimentation (SE) as a source of new ideas.  However, it
>> >also demonstrates the use of SE as a means of testing ideas that would
>> >be difficult and expensive to test using conventional methods, because
>> >the testing involves treatments that radically affect life-style, that
>> >must be applied for long periods, and that involve measurements that in
>> >some cases would be difficult without a live-in sleep-in laboratory
>> >and/or major modifications of a home and/or work environment.
>> >
>> >The domains of research are interesting and important: as the title
>> >says, they include sleep quality, mood, level of health, and control of
>> >weight.  It is true that the research involves just one subject,
>> >experimenting on himself, raising issues of generality and bias, and no
>> >doubt raising alarms in some readers because of their belief that SE is
>> >taboo as a research method.  But if further work, suggested by these
>> >findings, showed that they were valid and that they applied to even a
>> >small fraction of the general population, the payoff in human health
>> >and happiness would be very great.  And it is not at all clear that
>> >such systematic work on these questions would have been attempted,
>> >using conventional methods.  Indeed, the fact that so many new,
>> >interesting, surprising, and potentially important and useful findings
>> >could have emerged from this work shows that SE may promote research
>> >that is otherwise not done, even though in principal it could be.  And
>> >one major new finding with these features -- and one that was
>> >surprising to the investigator, is the remarkable productivity of SE in
>> >answering questions in these domains.
>> >
>> >The main question raised in the context of these findings is how to go
>> >about generating interesting and promising new ideas in human
>> >behavioral and biological science, a very important question about
>> >which we know little.  The commentaries that would be stimulated by
>> >this article are likely to further illuminate this issue, especially if
>> >commentators are explicitly invited to do so, as well as to address the
>> >specific material presented in the paper.
>> >
>> >Another way to regard the paper is as a case study in the history of
>> >ideas within science, something with which few scientists ever provide
>> >us.  In this case, the paper will be of interest to many readers for
>> >professional reasons, because of the substantive issues addressed and
>> >the research method, but should also be of interest to almost all
>> >readers for personal reasons.
>> >
>> >Overall, I find this paper fascinating, provocative, profoundly
>> >original, imaginative, highly controversial, and likely to change some
>> >readers' beliefs about how to make progress in behavioral and
>> >biological science.  It reflects the many years of research and
>> >thinking that led up to it, in its refinement of ideas and
>> >consideration of their implications.  In short, this paper is excellent
>> >and, except for its length, is perfect for BBS.  Furthermore, while I
>> >desirable (to help readers cope with the large amount of material the
>> >paper contains), the large number (ten) of diverse examples is an
>> >important contributor to the points made by the paper, so that any
>> >salient length reduction would weaken it considerably.  I also believe
>> >and recommend that BBS publish the paper after such revisions are
>> >made.
>> >
>> >---------------------------------------------------------------------
>> >B. Comments on major issues.
>> >
>> >B1. The emphasis in the title and introduction is the use of SE
>> >(self-experimentation) as a means of generating ideas.  (See the cell
>> >with the question-mark in Table 1.)  I have four comments here:
>> >
>> >First, it would be helpful to PROVIDE EMPHASIS, IN EACH EXAMPLE, ON THE
>> >done, e.g., by typographical emphasis, or (better) a NUMBERED
>> >SUB-SECTION [2].  If unexpected observations, facilitated by
>> >self-experimentation, are important (as indicated in the final section)
>> >then perhaps they, also, should be given emphasis in a similar way, or
>> >be included in the same sub-section.
>> >
>> >Second, some of the examples, especially those concerned with weight
>> >regulation, are presented in a way that emphasizes the testing of ideas
>> >rather than their generation.  Yet the introduction separates
>> >generation and testing of ideas ("two ends of a continuum"), and the
>> >title emphasizes the utility of SE for idea generation.  At the end of
>> >the paper (Section 5.3) several other desirable features of SE are
>> >discussed, including the ease of testing ideas.  The paper should take
>> >[4,5]?  Is the paper concerned primarily with idea generation or with
>> >both that and idea testing.  If not the latter, then the PRESENCE OF
>> >weight regulation the main ideas are embodied in the theory described
>> >in the introduction to Section 4, and even the new ideas added to the
>> >"standard" set-point theory appear not to have originated from SE.
>> >
>> >Third, if possible I would like to see MORE ANALYSIS AND DISCUSSION OF
>> >[4,6].  This could be compared, for example, to reading the literature
>> >and taking note of features of the data reported that may deviate
>> >systematically from what the author or the reader expects, or to
>> >noticing unexpected features in one's own data from a standard
>> >experiment.  Other methods that might be compared with SE include the
>> >incorporation into standard experiments of supplementary observations
>> >not called for by the main goals of the experiment, or the
>> >incorporation of extra experimental manipulations as factors crossed
>> >with the ones of main interest (which, as R. A. Fisher showed, costs
>> >little).
>> >
>> >Fourth, if an important feature of SE is the ability to easily test
>> >ideas (as distinct from generating them), a point made in Section 5,
>> >THIS FEATURE [2,4,6].
>> >
>> >----------------------
>> >
>> >B2. My impression is that the two main reasons people give when they
>> >argue against SE are the possible role of expectancies, and the SMALL
>> >SAMPLE SIZE [4,5] (one).  The first issue is considered in the paper,
>> >but the second gets little attention.  THIS ISSUE OF SAMPLE SIZE AND
>> >SOMEWHERE [4,5,6], and at least mentioned in the introduction, to make
>> >it clear that the author has considered it.  (Otherwise readers may be
>> >deterred from reading further.) It may also be helpful to mention early
>> >the idea that there is a TABOO AGAINST THE SE METHOD, and to at least
>> >start discussing WHAT THE BASIS (AND THE HISTORY) MIGHT BE [2,3].  My
>> >impression is that the taboo may be against using only data from SE;
>> >see comment below on the study of sensory processes.
>> >
>> >----------------------
>> >
>> >B3. With respect to the role of expectations, I think that the argument
>> >that they are unimportant, based on findings that were initially
>> >unexpected, is a good one (and one of the reasons why examples not be
>> >omitted to shorten the paper).  But once an observation has been made,
>> >this argument disappears; the impressive consistency in some of the
>> >findings, when they have been repeated, could have been produced by
>> >expectations of consistency.  This ISSUE SHOULD BE ACKNOWLEDGED [5].
>> >
>> >It is also possible that the influence of expectations MAY VARY FROM
>> >EXPERIMENTER TO EXPERIMENTER [5], in SE, so that even if Roberts'
>> >findings from SE are found to generalize to other subjects, another
>> >investigator's findings from his or her SE, influenced more by
>> >expectations, might not.
>> >
>> >Another sense in which the success of SE might not generalize is to
>> >other kinds of questions or measures.  The unexpected findings about
>> >mood impress me more than unexpected findings about weight gain,
>> >because I believe that mood ratings are more likely to be influenced by
>> >expectations that is weight gain.  The long tradition of SE in the
>> >study of sensory processes probably reflects the belief, held until the
>> >advent of signal-detection theory, that decision processes were
>> >relatively unimportant in sensory experiments, and that the results of
>> >such experiments depend on physiological states of affairs that are
>> >relatively uniform among most subjects.
>> >
>> >----------------------
>> >
>> >B4. In Section 2.1 some of the earlier thinking about how discoveries
>> >are made, and mentions Hyman (1964) and Root-Bernstein (1989).  But
>> >there is ACTUAL RESEARCH ON THIS QUESTION, reviewed, for example, in
>> >KLAHR & SIMON (1999) [3], Psychological Bulletin, 125, 524-543.  It
>> >would be helpful if Roberts could CONNECT HIS IDEAS AND FINDINGS TO
>> >THOSE [3,6], and do so succinctly so as not to enlarge the paper
>> >further.
>> >
>> >----------------------
>> >
>> >B5. Evolutionary arguments play important roles at several points in
>> >the manuscript.  There seems to be an implicit assumption that human
>> >characteristics evolve independently, and hence that every
>> >characteristic must be adaptive -- must itself serve some useful
>> >[5,6].
>> >
>> >---------------------------------------------------------------------
>> >
>> >C. Comments on expository issues
>> >
>> >INDICATE ORGANIZATION AT OUTSET (e.g., tell reader that each example
>> >will include related results and discussion); important to make the
>> >organization of this long paper more transparent.  If possible, a table
>> >of contents would help greatly.  E.g., say anticipate the "related
>> >results" sections.
>> >
>> >
>> >
>> >Reconsider numbering and naming of sections and sub-sections.
>> >
>> >Consider numbering examples with the main sections they are from -
>> >e.g., example 3.3 instead of example 2.
>> >
>> >Continue decimal numbering in all sub-sections, to help reader to know
>> >where he/she is in the text, to help in searching for particular parts
>> >of examples, and to make cross-referencing within the paper (and in the
>> >commentaries) easier.
>> >
>> >SUBDIVIDE the sections that are especially long, with suitable
>> >headings.
>> >
>> >Reconsider all subheads to help reader keep oriented.
>> >
>> >Some statements are made with TOO MUCH CERTAINTY. [5]
>> >
>> >"DIFFERENT FROM" [2] rather than "different than" in most cases (as in
>> >"separate from", "distinct from", "apart from"; one thing differs from
>> >another.)
>> >
>> >DATES and other biographical details.  Dates are provided from time to
>> >time, e.g., the first probably appears in the caption of Figure 1.  It
>> >is not clear what the reader should do with these.  Are they important
>> >enough for him to remember?  Are they important enough to be included?
>> >I suggest that these be given in a SEPARATE TABLE, POSSIBLY IN AN
>> >CAPTIONS. [2,7]
>> >
>> >
>> >Data points on a number of the plots (in the copy I am using) are not
>> >closed shapes.  E.g. many of the points in Fig. 23 consist of two sides
>> >of a triangle.  In all these cases, CLOSED SHAPES WOULD MAKE THE PLOTS
>> >
>> >"Processing" of food is sometimes used in a way that makes it unclear.
>> >Aren't all foods "processed"?  (Is the term well enough defined so that
>> >spraying, picking, shipping, and affixing a code label to a piece of
>> >fruit are not instances of processing?)
>> >
>> >---------------------------------------------------------------------
>> >
>> >D. Comments about particular points in the text
>> >
>> >These are keyed to page numbers in a pdf version of the manuscript
>> >dated 3/5/2001.  With the page number is indicated quarter of the page,
>> >as a, b, c, or d.  Also, for stylistic simplicity, some of these
>> >comments are addressed to the author ("you", "your").
>> >
>> >2b. a large amount -> considerably
>> >
>> >2c. big difference between first and second series (theory in second)
>> >
>> >4b. "all possible ideas".  Is this what actually happens?   Doesn't one
>> >often start with a question, or an observation (including, in
>> >psychology, an introspection)
>> >
>> >5a. Should say that in SE the investigator and the subject are one.  It
>> >is therefore restricted to experiments where the subjects are humans
>> >and (presumably) the outcome can't be fatal.  Also, SE differs from
>> >other methods in being subject to a special kind of bias.  Deal with
>> >this problem here, or anticipate dealing with it later.
>> >
>> >5d. "differed even more" than what?
>> >
>> >5d. I believe there are other famous instances of SE in the psychology
>> >of perception.  E.g. HELMHOLTZ, HERING, MATIN [3] (curare).  (The first two
>> >checked their findings with other subjects, but Matin did not, because
>> >of the severity of the treatment.)
>> >
>> >6a, 6c. I think the importance of Kristofferson's experiments are
>> >underestimated, and that they did reveal new cause-effect linkages.
>> >See comment later.
>> >
>> >6c. "discoveries .. cannot be due to expectations".  Seems glib.
>> >
>> >6d. "testing possible solutions".  But this paper is supposedly about
>> >generating the ideas for solutions.
>> >
>> >7b. input/output distinction should be made more clear.
>> >
>> >8c. What about the stimulus suggested a 90-minute period?
>> >
>> >9b. "baked goods".  This will be interpreted as including bread. Ok?
>> >
>> >11d. "no breakfast".  This paragraph describes the birth of the first
>> >idea, and should be given emphasis (see above).
>> >
>> >12ab. This FIGURE COULD BE IMPROVED [2,4].  The scale of the x-axis
>> >could be days (into the experiment, starting with 1); Given that the
>> >dates are unimportant, particular ones could then be equated to day
>> >number in the caption; this would reduce clutter, Regions for different
>> >treatments should be delimited more clearly.  The same measure (time to
>> >fall asleep after awakening) is used to define "early awakening" and to
>> >measure its magnitude.  These two features are hard to separate from
>> >the plot.  One possible improvement would be to plot them separately,
>> >the first as a smoothed proportion or cumulative number, and the second
>> >as a smoothed duration versus time in treatment.  Perhaps, given Figure
>> >3, Figure 2 is not even needed.
>> >
>> >12d. *positive* correlation.
>> >
>> >13a-d. Make caption conform to plot labels.  If Fig. 3 is based just on
>> >Fig. 2 data, say so in caption.
>> >
>> >14c. due to *experimenter* expectations.
>> >
>> >14d. There should be more analysis of and emphasis on *HOW* SE
>> >
>> >15a. Conjectures about what our ancestors did or didn't eat should be
>> >stated more tentatively.  This one should be accompanied by an
>> >explanation.
>> >
>> >15b. This paragraph describes the birth of the first idea associated
>> >with example 2, and should be emphasized, along, perhaps, of comments
>> >
>> >15c. "Events that affect the phase of a rhythm usually affect its
>> >amplitude": is this a conjecture?  If not, can you provide a
>> >reference?  Perhaps it doesn't matter; in that case you could say "I
>> >believed that events ... amplitude".
>> >
>> >16a. Re Szalai's survey: As written, this suggests that this survey
>> >information helped to generate the idea.  If this is autobiography you
>> >might say when you learned about the survey.
>> >
>> >16a. than in persons -> than persons
>> >
>> >16a. watching (upper -> watching at midnight (upper
>> >
>> >17a. difference was -> difference seemed (autobiographical)
>> >
>> >19a. This paragraph describes the birth of the second idea associated
>> >with example 2. See comment re 15b and first idea.
>> >
>> >20ab. There are no filled points.
>> >
>> >22a-d. To demonstrate a delayed effect of the treatment, don't you need
>> >data between the treatment time (6am) and the following 6pm?  And to
>> >show that the effect ends 24 hours after it starts, don't you need data
>> >after the second 6pm?  The absence of any effect during the first 12
>> >hours or after the next 24 hours seems quite important, so more should
>> >be done to persuade the reader that it is true.
>> >
>> >23a. It would be good to say, explicitly, that exposure to faces raised
>> >mood with a one-day lag *and for just one day* (or something like
>> >this).
>> >
>> >23a-d. Caption should be adjusted: This figure is concerned with TV
>> >distance as well as TV size.  Visual angle subtended by the face should
>> >be indicated in the legend.  To determine the roles of distance versus
>> >visual angle, a (projected) 3-D plot could be shown, with visual size
>> >as distance as independent variables.  Alternatively, it may be helpful
>> >to perform a multiple regression of mood versus those two factors.
>> >
>> >24b. Conversational distance should be specified.  It would also be
>> >useful to know the approximate fraction of the screen covered by the
>> >face.  It is not at all clear whether you are claiming that there is a
>> >distance effect in addition to a visual-angle effect.  Why is 20/1
>> >better than 32/1.5?  This suggests that there is a separate distance
>> >effect.
>> >
>> >27b-c. If possible, make the points for Vancouver and Seattle distinct
>> >from the others, and easy to group together.  E.g., they might be + and
>> >x.  Explain filled points in the caption as well as in the text.
>> >
>> >28d. Did the increase occur after the second interview?
>> >
>> >29c. Re bipolar disorder: Isn't it important whether the equal
>> >intervals were daily - i.e. whether the shift occurs within 24 hours?
>> >
>> >29c. Figure 6 doesn't show that there's no effect during the initial 12
>> >hours.
>> >
>> >29d. If the new ideas were highly plausible, wouldn't someone have
>> >thought of them without SE?  More generally, mustn't there be a limit
>> >on the plausibility of new ideas based on unexpected findings?
>> >
>> >31a-c. Without support from research, several of these statements are
>> >expressed too strongly.  As conjectures they seem reasonable and
>> >interesting, but stated as they are, they may alienate readers. (E.g.,
>> >"productive group work requires . .", "happy people complain less . .
>> >")
>> >
>> >31c. if you starts at 70; -> if you start at 70,
>> >
>> >31c. "another reason for a rhythm": benefit rather than explanation?
>> >
>> >31d. "this data bears" -> these data bear
>> >
>> >32b. "breakfast and go" -> breakfast and do go
>> >
>> >32c. "no other explanation": You haven't explicitly provided an
>> >explanation.  So one has to assume that it is no faces in the am, and
>> >faces in the evening.  Do you really want to assert this as an
>> >"explanation" of depression?
>> >
>> >33a. Birth of idea in Example 3.  See above.
>> >
>> >33a. "Stone-Age humans": Evidence or other justification?
>> >
>> >33c-d, 34a. Make clear that there were three levels of the treatment,
>> >and relate the separation between the second and third phase explicitly
>> >to Table 3. Make clear where the percentages (29%, 9%) come from.
>> >Given the data, it is surprising that the test results are so
>> >different.
>> >
>> >34a. after standing that much *during a day* I awoke *the next morning*
>> >feeling . .
>> >
>> >34b. "no psychology experiment".  Is it a psychology experiment or a
>> >physiology experiment?
>> >
>> >34d. median of *observations on* 20 days.
>> >
>> >36a. Anticipate this suggestion (and others?) about ways of making SE
>> >double blind, in initial discussions of the method and what the paper
>> >will do.
>> >
>> >37d. Caption: "time to of bed" -> time to bed.
>> >
>> >38a. First sentence mentions 5 panels; next sentence applies to only
>> >one.
>> >
>> >38b. ("The lack of a correlation . . ") Are there statistically better
>> >ways to make these inferences?  E.g., partial correlation?  Isn't there
>> >a problem if the other factors that do have effects might themselves be
>> >correlated, or that, together, they might explain as much variance as
>> >the prior duration of standing?
>> >
>> >39b. This argument would suggest that in someone who suffers from
>> >osteoporosis, the condition does not apply to, e.g., the arms.  Is that
>> >the case?
>> >
>> >39d. "made sleep deeper":  Given its use here, the idea of *depth of
>> >sleep* requires a definition, or at least a reference.
>> >
>> >39d.  The results cannot -> Some of the results cannot
>> >
>> >40a. "predicted ... more poorly".  See comment above about partial
>> >correlation.  (How much additional variance is accounted for in a
>> >multiple regression when standing duration is added to the other
>> >factors?)
>> >
>> >40b. "did not make me feel more tired": any data other than subjective
>> >impressions?
>> >
>> >40b. Re: "third, and most decisive": If the idea of sleep depth is
>> >meaningful, then unless you say that you awaken when there is no S
>> >remaining, which you cannot say) why assume that the duration/depth
>> >tradeoff leaves S invariant?
>> >
>> >43b. "Stone-age people .. not stand all day"; "usually not in short
>> >supply": Evidence or other justification?  On page 33 you suggest a
>> >lower bound on the amount they stood, here an upper bound.  Better to
>> >put these aspects of your conjecture about how much they stood
>> >together, and then later tell the reader what he needs to believe for
>> >your various arguments.  Also, given these bounds, the health/sleep of
>> >modern humans whose jobs are associated with standing becomes
>> >especially important.
>> >
>> >44cd. In my version the plotted points are not closed curves, making
>> >the graph harder to read.  Even though the y-label says so, the caption
>> >should also make clear that the rested ratings are on the subsequent
>> >
>> >45c. "ratings are easier to understand":  But are they as valid?  do
>> >they show relations more clearly?
>> >
>> >46d. "on sleep-wanted ratings *on the next day*" (caption).
>> >
>> >47a. Summarize the rested ratings, to avoid skepticism.
>> >
>> >47a. "no clear effect": see comment on Fig. 17.
>> >
>> >48d. title of plot:  "parameters *on the following night*".
>> >
>> >49a. "did not notice": puzzling: why not?
>> >
>> >49a. "on the time of day" -> on the time the ratings were made.  Also,
>> >this is written as if the time of the light is being varied.
>> >
>> >51a. "step function": mentioned earlier; refer back?  more numbered
>> >sub-sections would help in such references.  Might help to anticipate
>> >this part of this example in example 1.
>> >
>> >52c. These tests are not independent; better to do one test on points 1
>> >- 7.
>> >
>> >52cd. If the points are referred to by number, then number them on the
>> >plot.  See earlier comment about dates on plots.  As above, for points
>> >indicated by squares, the squares are not closed, having only three
>> >sides.
>> >
>> >53a. bright light and non-seasonal depression: not clear whether it has
>> >been shown not to help, or hasn't been tried.
>> >
>> >54a. Wehr, Wirz-Justice mentioned in previous paragraph; is redundancy
>> >needed?
>> >
>> >54b. "spring of 1997": example of a biographical detail that is hard
>> >for the reader to deal with (see above).  If the date is important
>> >because of what preceded it, then this should be stated.  If the
>> >chronology is important in general, then consider providing a time
>> >line, perhaps labeling the critical points to be mentioned in the text
>> >(T1, T2, etc.).
>> >
>> >55d. See comment about dates on plots.
>> >
>> >56c. "may have been fighting off an infection.  *If so,* it is an
>> >indication . ."
>> >
>> >56d. "sleep may have begun as a response to infection".  Begun in which
>> >species?  Say something to increase plausibility?
>> >
>> >57a. "surely due to" seems too strong.
>> >
>> >57b. Bigger ratio of winter colds/summer colds in more northern
>> >regions?
>> >
>> >58a. No need to mention Nieman (1994) in two successive sentences:  use
>> >"He".
>> >
>> >58c. Show dates (symbolically) on figure 20.
>> >
>> >58d. Re morning light in the workplace.  Doesn't the light have to be
>> >quite early, which limits applicability?  (Does late light hurt the way
>> >late faces are supposed to?)
>> >
>> >58d. "with some dissent": Worth indicating basis?
>> >
>> >58d. "helped develop a theory": Presenting the theory first weakens
>> >this argument, which is also the main point of the paper as it is now
>> >formulated.  It is important to ANTICIPATE THIS, IN A SECTION THAT
>> >HERE [2].  Important to anticipate that you will say how components of the
>> >theory came from SE.  And it is important to say this, as the sequence
>> >of SE tests are describes.  The main emphasis now in these descriptions
>> >is how the SE sequence tested and confirmed the theory, not how it gave
>> >rise to elements of the theory.  The best solution (if it actually
>> >happened this way!) would be to use the standard theory and develop its
>> >extensions and elaborations in pieces, along with the SE sequence.
>> >Now, even the two assumptions added to the standard theory are not
>> >described as coming from SE.  All of this is, of course, related to B1
>> >above; an alternative is to recast the paper to emphasize two features
>> >of SE.  But if that is done, the introduction would have to be
>> >radically altered.
>> >
>> >59a. Should you not also say that for a given degree of association,
>> >more calories raise the set point more?
>> >
>> >59d. Worth pointing out that the broken lines are linear with time
>> >(does this need justification?  I.e., is it known to be true? Is it a
>> >part of the theory? A new part?), while the solid lines are
>> >decelerating, per added assumption (b).
>> >
>> >59d. (a) and (b) were also used above for the added two assumptions.
>> >Use different letters.  Indicate the status of these new two points.
>> >The new (a) is, of course, simply a restatement of part of assumption
>> >(b).  If the new (b) is another assumption in the theory, say so, and
>> >indicate its status relative to the standard theory.
>> >
>> >60a. If this is a fictitious numerical example, it would be better to
>> >say something like "if at a high weight it falls 1kg/day . . .".  but
>> >if you point out that the solid curves are decelerating, that makes the
>> >example unnecessary.
>> >
>> >60d. "by flavors" -> by a flavor.
>> >
>> >60d. "overlaps" -> temporally overlaps.
>> >
>> >61a. "weight that rats" -> weight than rats
>> >
>> >61c. (plentiful vs scarce food): (1) Doesn't this depend on the idea
>> >that all foods are equally reduced in availability when there is a
>> >shortage?  Any basis for this?  (2) Won't learning occur during the
>> >shortage?
>> >
>> >62a. Describe the effects of additional food processing earlier, before
>> >your conclusion about association strength.
>> >
>> >62a. Important to emphasize the role of Fig. 1 data, possibly
>> >presenting it earlier in Section 4.1.
>> >
>> >62a. "Examples 6-10 developed the theory":  Testing should be
>> >distinguished from elaborating it.  "Developing" it in the sense of
>> >making it "plausible, meaningful, and detailed" should be related to
>> >the primary thesis about SE and the generation of new ideas.
>> >
>> >62b. "change to the theory" -> change in the theory.
>> >
>> >64d. "This explanation implies": The principle on which this
>> >implication is based is not clear.  How does the implication follow?
>> >
>> >64d. Why not in "Related Results"?
>> >
>> >66b. "closely correlated" -> correlated negatively
>> >
>> >67abc. To unify the organization of the paper, shouldn't much of the
>> >"Discussion" in this case be in "Related Results"?
>> >
>> >68d. Indicate the two weeks clearly on the plot.
>> >
>> >69b. "reduced the processing in my diet"  (unclear; not standard usage)
>> >-> reduced the amount of (highly) processed food . . (Could be the
>> >degree of processing, or the presence of any processing.  Aren't all
>> >foods processed in some sense?
>> >
>> >71c. "had I started with a typical" -> had I preceded the sushi diet
>> >with a typical.
>> >
>> >71c. "how much weight I would have lost": This reasoning depends on one
>> >or more assumptions that may not be true.  They should be brought out.
>> >
>> >72b. "23 versus 65" -> 23, versus 65
>> >
>> >75a. "but it helped discover it": How did it help?  And what does this
>> >say in relation to the main thesis about idea generation?
>> >
>> >75d. "bad for you" -> bad for sleep.
>> >
>> >77c. "That an effect (e.g." -> That a treatment (e.g..
>> >
>> >77c. "does not imply that others": Unclear; depending on how one
>> >interprets "cured" it may seem like a contradiction.
>> >
>> >77d. Last sentence before Table 4 is unclear.  First, what is
>> >"determined" - not "sleep"; perhaps the amount of time spent sleeping.
>> >Your problem was a particular one, i.e., early awakening.  Why not use
>> >this as what is "determined"?  Perhaps what you mean to say is that
>> >early awakening in others may result from causes other than those
>> >associated with your early awakening, hence may be ameliorated by other
>> >treatments.
>> >
>> >77d. Table 4: Either expand, to make the support more evident, or give
>> >specific references to appropriate sub-sections in the text.
>> >
>> >78a. "constantly monitor".  One of the important contributions to
>> >theses instances of SE is the systematic and careful measurement and
>> >recording that they exemplify.  Exactly what do we "constantly
>> >monitor"?  In which examples was an effect evident without systematic
>> >and careful measurement?  It may also be important that even if you
>> >made important discoveries without careful measurement, the
>> >self-training associated with careful measurement may have made you
>> >especially skilled at more informal monitoring.
>> >
>> >80a. (notice better mood): It would be helpful to catalog such
>> >instances versus others, and whether they occurred before or after you
>> >had systematically measured that variable (here mood).
>> >
>> >80c. "easy to test treatments never tested before": should add. "simply
>> >because it made all tests easy"
>> >
>> >80d. "using conventional methods":  Are conventional methods instances
>> >of random search?
>> >
>> >80d. "effect of processing": again, unclear. -> effect of eating highly
>> >processed food.
>> >
>> >81c. "include only one new cause-effect relationship":  This seems much
>> >too strong, partly because "cause-effect relationship" is not well
>> >defined.  For example, is the invariance of a measure over levels of a
>> >potentially controlling factor a cause-effect relationship?  Or is this
>> >classed as a cause-noneffect relationship?  I've checked only in the
>> >case of the Kristofferson papers.  In the 1976a paper mentioned, he
>> >showed that under a certain procedure (a cause?) and extensive practice
>> >(another cause?) the variability of duration estimates is astonishingly
>> >invariant over a range of durations of about 150 to 550 msec.
>> >(Duration is a potentially controlling factor, and one that shows
>> >itself in other parts of the range.) In the 1976b paper mentioned, he
>> >showed that after extensive practice (cause?) the increase of variance
>> >with mean duration (another cause?) becomes similar to a step
>> >function.  You seem to be saying that our goal is to determine which
>> >factors influence a given measure, and that the details of how they
>> >influence that measure (the "shape" or "form" of the effect as a
>> >function of the cause), or of exactly when they do or don't influence
>> >it, or of factors that "ought to" but don't influence it, are of less
>> >importance or interest.  It seems to me that this goes in the direction
>> >of trivializing quantitative relationships relative to qualitative
>> >ones.  This may be a reasonable view in some more primitive fields, but
>> >may also be a view that keeps them primitive.  Another issue is whether
>> >the discovery of cause-effect relationships is the only goal, or the
>> >only measure of effectiveness of a particular method of investigation.
>> >Turning to cosmology, what would be the status of the discovery of the
>> >background microwave radiation, with its spectrum and isotropy?  This
>> >discovery had a huge impact, but it is hard to describe it as a
>> >discovery of a cause-effect relationship.  (The cause-effect
>> >relationship was purely theoretical.)
>> >
>> >81d.  My guess is that a great deal of research in experimental
>> >psychology with humans starts with SE - especially if one includes
>> >cases where the observations are introspections.  What is special in
>> >the instances of SE described here is the great magnitude of the
>> >effort, the deliberate manipulation of conditions, and the systematic
>> >and numerous measurements.  Thus, this is "formal" SE, not simply SE.
>> >
>> >82a.  "regulatory processes are . . more difficult to study".
>> >Certainly they seem to take longer.  (On the other hand, can't
>> >light-adaptation be described as a regulatory process?) Are they more
>> >difficult in other ways?
>> >
>> >82b. "single intriguing observations".  These should be emphasized when
>> >they are initially presented, and included in a sub-section on "birth
>> >of the idea", which could then be referred to here.
>> >
>> >83a. "experimental psychology . . applications have yet to appear".
>> >This seems like a hazardous statement.  HOW DOES ONE COUNT? [4,5]
>> >Consider the case of the applications of psychoacoustic research to
>> >speech communication by telephone, or to the design of hearing aids.
>> >Or of applications of experimental psychology to the design of the
>> >telephone key-pad, or the choice of typography, or the design of
>> >computer displays.  Or of psychometric research to testing.  Or of
>> >animal (and human) research to the development of behavior-relevant
>> >drugs.  Even given this short list, we are all exploiting applications
>> >of experimental psychology every day of our lives.  Why did the number
>> >of PhDs in experimental psychology rise to about 200 in the Bell Labs
>> >Development Area in the late 1970s?
>> >
>> >83a. "10-20 person years".  This is normally taken to mean full-time.
>> >Given evidence of your other work, it is probably a vast
>> >over-estimate.
>> >
>> >83a.  "new tool".  As you say, there are other instances, and, as noted
>> >above, there are still others.
>> >
>> >83b.  Clearly if these findings are found to generalize, there are
>> >applications, probably of great importance.  BUT IT IS NOT AT ALL CLEAR
>> >THAT IF THEY OCCUR IT WILL BE BECAUSE OF SE [4,5].  After all, most of
>> >the SE work described in the present paper was motivated by the goal of
>> >solving particular practical problems, unlike a great deal of the
>> >research that experimental psychologists conduct.  If there are fewer
>> >applications that there "should be", it could equally well be a
>> >consequence of numerous other factors, including the snobbishness of
>> >some (many?) academic experimental psychologists about applied
>> >research, the kinds of pressure placed (and not placed) on
>> >investigators by government funding agencies.  In the environment even
>> >of the Research Area of Bell Labs, where technology transfer was more
>> >salient than in academia, a discovery by experimental psychologists
>> >became, during one year, the highest priority idea for patent
>> >application in an organization that contained many engineers with
>> >numerous patents.
>> >
>> >84a. These comments seem premature.  Almost nothing is said in this
>> >paper about whether there ARE any successful practical applications.
>> >Given the studies described here, the problem of generalization over
>> >subjects, alone, could be a major stumbling block.
>> >
>> >84d. Samples are often larger than 10, especially in JEP:HLM.  Some of
>> >the studies in JEP:HPP could surely be described as "high-tech".
>> >
>> >85b. "low-tech".  Many of the experiments that I have done would never
>> >have been attempted without computers.
>> >
>> >85bc. These statements are HIGHLY QUESTIONABLE, AND WILL ALIENATE
>> >READERS [5,1].  I believe that JEP experiments often involve subtle
>> >manipulations, delicate measurements, and sophisticated designs.  It
>> >may be easy to do research in experimental psychology, but it usually
>> >isn't easy to do good research, in my view.
>> >
>> >86d.  The stated importance of evaluating a function seems inconsistent
>> >with the earlier emphasis on qualitative cause-effect relationships,
>> >and the devaluing of Kristofferson's findings.
>> >
>> >87ab. This starts with literally spatial (i.e. geographic) data, and
>> >ends with the conversion of non-spatial data into a spatial display.
>> >(It isn't clear from the quotes which meaning Oliver had in mind. )
>> >
>> >87b. It isn't clear that SE needs to involve a large number of
>> >measures.  (Consider the work mentioned above in vision.) Also, in
>> >research in some domains (or for other experimenters and subjects),
>> >expectations may have large effects, possibly precluding SE.
>> >
>> >88b. "published SE .. is rare".  Many sensory studies include the main
>> >author as one of the two or three subjects, so it is not so rare in
>> >some domains.  The fact that there are few publications of SE studies
>> >need not be an indicator that it is seldom done.  My own observations
>> >(and practices) suggest that there is a great deal of of non-SE "pilot"
>> >data that is also unpublished, probably for a number of reasons:
>> >procedures different, evolving, and/or more informal, concern about
>> >learning or practice effects and the inability to balance over them
>> >when a procedure is evolving, and probably simply the complexity of the
>> >description of design and procedure under those conditions.  These
>> >considerations surely also apply (with others, such as the possible
>> >influence of expectations) to SE itself.  To use Medawar's terms, SE
>> >and pilot data are probably thought of in the "context of discovery";
>> >what gets published is in the "context of verification".   That doesn't
>> >mean that SE and pilot experiments are not used.
>> >
>> >
>> >========================================================================
>> >========================================================================
>> >
>> >ACC 2 3 4
>> >MIN 5
>> >MAJ
>> >NOT
>> >
>> >ACC 3 4
>> >MIN 5
>> >MAJ 2
>> >NOT
>> >
>> >ACC 3 4 5
>> >MIN
>> >MAJ 2
>> >NOT
>> >
>> >[4]  EVIDENCE:
>> >ACC 4 5
>> >MIN
>> >MAJ 2 3 
>> >NOT
>> >
>> >[5]  REASONING:
>> >ACC 3 4
>> >MIN 5
>> >MAJ 2
>> >NOT
>> >
>> >[6]  THEORY:
>> >ACC 3 4 5
>> >MIN
>> >MAJ 2
>> >NOT
>> >
>> >[7}  LENGTH:
>> >ACC 3 5
>> >MIN 4
>> >MAJ 2
>> >NOT
>> >
>> >ACC
>> >MIN 3 4 5
>> >MAJ 2 3
>> >NOT 1
>> >ELS

More information about the Jrnlnote mailing list