# Some myths concerning statistical hypothesis testing

John H. johnh at faraway.xxx
Sun Nov 17 01:13:02 EST 2002

```As a mathematical outsider I find it quite surprising that some of the
fundamentals of statistical analysis, utilised so widely in the sciences,
remains in hot dispute by many people who obviously are not intellectual
outsiders. It just shows how even in such a 'pure' field as mathematics
there remains plenty of room for debate.

John H.
"Herman Rubin" <hrubin at odds.stat.purdue.edu> wrote in message
news:ar6t0j\$4ano at odds.stat.purdue.edu...
> In article <pan.2002.11.14.16.35.59.925178.12323 at molden_dot_net.invalid>,
> Sturla Molden  <sturla at molden_dot_net.invalid> wrote:
> >On Thu, 14 Nov 2002 14:49:36 +0100, Herman Rubin wrote:
>
> >>>the p-value is p(data or more extreme data | H0)
>
> >> And what relation does this have with inference about the truth of H0?
>
> >Very little, but still the majority of research workers insist
> >on using this useless statistic.
>
> >> Even more so, what does this have to do with the appropriate action to
> >> take?
>
> >Pearson and Neuman argued that the appropriate action to select
> >is the one that will fixate the proportion of false positives to
> >a certain level, if H0 is true and the same experiment is repeated
> >many many times. The p-value can be used as a statistic that instructs
> >us on how to act given a Pearson-Neuman type descision rule: If the
> >so-called "level of significance" is set to 0.05, then obtaining
> >p<0.05 would instruct us to reject H0.
>
>
> It is Neyman, not Neuman.  However, they did not introduce
> this idea; it goes back to the 18th century.  What they did
> was to show that the best test against a simple alternative
> is the likelihood ratio test, with arbitrariness only if
> there are ties, and some further results along that line.
>
> In fact, Neyman pointed out that it is false to argue from
> the p-value to the probability that the null hypothesis is
> wrong, which it seems almost everyone had been acting on
> for more than a century.  Further developments on the
> general problem had to wait for utility theory and game
> theory to get to the point that the mathematics was clear.
> --
> This address is for information only.  I do not claim that these views
> are those of the Statistics Department or of Purdue University.
> Herman Rubin, Deptartment of Statistics, Purdue University
> hrubin at stat.purdue.edu         Phone: (765)494-6054   FAX: (765)494-0558

```