# Some myths concerning statistical hypothesis testing

Glen M. Sizemore gmsizemore2 at yahoo.com
Fri Nov 8 09:41:52 EST 2002

```> GS: You really should learn to be patient. The first is not an
> assumption, it is a fact. A p-value expresses a conditional probability.
> That is, a p-value expresses the probability of obtaining the
> observation in question GIVEN THAT THE NULL HYPOTHESIS IS TRUE.

SM: No it does not. This would be the likelihood of the null hypothesis.
The p-value is the probability of getting at least as impressive
data by fluke. The p-valueis not the probability of getting equally
impressive data by fluke.

GS: Sorry, but the p-value IS the conditional probability of observing the
data given that the null hypothesis (where the null hypothesis is that both
groups are equal, i.e., from the same population) is true. I am willing to
consider that you are correct, but if you are, thousands of statisticians
will be enormously surprised. The problem with conventional statistics is
the way they are used. BTW, I don't think you meant to say "The p-value is
the probability of getting at least as impressive
data by fluke. The p-valueis not the probability of getting equally
impressive data by fluke." Did you?

SM: I.e. a p-value DOES NOT express "the probability of obtaining the
observation in question GIVEN THAT THE NULL HYPOTHESIS IS TRUE."

GS: It does if the null hypothesis is true. Problem is, this is exactly what
you don't know.

SM: This means that the p-value is not the "statistcial likelihood" of
the null hypothesis. A fundamental statistical theorem is the likelihood
principle that can be interpreted as "evidence is proportional to
likelihood".
As the p-value is not proportional to likelihood, it cannot be proportional
to evidence. And this is where classical statistics fail.

GS: Again, any statistician will tell you what I have said, but I am willing
to consider that they are wrong.

SM: A number of problems raise from this. A well-known problem is that of
stopping rules in clinical trials: A severe ethical dilemma is introduced
simply by using a flawed measure of evidence.

GS: I don't really follow but I'll keep trying.

> GS: Wrong. Remember that a p-value represents the probability that one
> will observe certain data given that the null hypothesis is true. If one
> asserts that the p-value is really the probability that the null
> hypothesis is true given the data (which is the same thing as saying it
> represents the probability that the observed data are "due to chance")
> is to "reverse the conditionality."

SM: Wrong again.

GS: I don't think so.

SM: Bayes theorem is used to reverse conditionality in probability
theory (and this is what "Bayesian" tests does). However as the p-value
is not equal to the likelihood (as you claimed it to be), it cannot be
used to reverse the conditionality by inserting it into Bayes formula.

Cordially,

Glen

"Sturla Molden" <sturla at molden_dot_net.invalid> wrote in message
news:pan.2002.11.08.13.18.06.391156.1184 at molden_dot_net.invalid...
> On Thu, 07 Nov 2002 00:22:18 +0100, Glen M. Sizemore wrote:
>
>
> > GS: You really should learn to be patient. The first is not an
> > assumption, it is a fact. A p-value expresses a conditional probability.
> > That is, a p-value expresses the probability of obtaining the
> > observation in question GIVEN THAT THE NULL HYPOTHESIS IS TRUE.
>
> No it does not. This would be the likelihood of the null hypothesis.
> The p-value is the probability of getting at least as impressive
> data by fluke. The p-valueis not the probability of getting equally
> impressive data by fluke.
>
> I.e. a p-value DOES NOT express "the probability of obtaining the
> observation in question GIVEN THAT THE NULL HYPOTHESIS IS TRUE."
>
> This means that the p-value is not the "statistcial likelihood" of
> the null hypothesis. A fundamental statistical theorem is the likelihood
> principle that can be interpreted as "evidence is proportional to
likelihood".
> As the p-value is not proportional to likelihood, it cannot be
proportional
> to evidence. And this is where classical statistics fail.
>
> A number of problems raise from this. A well-known problem is that of
> stopping rules in clinical trials: A severe ethical dilemma is introduced
> simply by using a flawed measure of evidence.
>
>
>
> > GS: Wrong. Remember that a p-value represents the probability that one
> > will observe certain data given that the null hypothesis is true. If one
> > asserts that the p-value is really the probability that the null
> > hypothesis is true given the data (which is the same thing as saying it
> > represents the probability that the observed data are "due to chance")
> > is to "reverse the conditionality."
>
> Wrong again. Bayes theorem is used to reverse conditionality in
probability
> theory (and this is what "Bayesian" tests does). However as the p-value
> is not equal to the likelihood (as you claimed it to be), it cannot be
> used to reverse the conditionality by inserting it into Bayes formula.
>
>
>
> Sturla Molden

```