Some myths concerning statistical hypothesis testing

[Sturla Molden]

On Fri, 08 Nov 2002 20:04:27 +0100, Glen M. Sizemore wrote:

> Sorry, but the p-value IS the conditional probability of observing
> the data given that the null hypothesis 

The p-value is not the conditional probability of getting the observed 
data under the null hypothesis. Let me provide a simple proof for this:

Assume that you want to test if a dice is fair. Then you have: 

   H0:  Prob of getting 6 is 1/6  
   HA:  Prob of getting 6 is not 1/6
      
Now you throw the dice 10 times and observe a 6 on five of the trials.
Then the conditional probability of getting the observed result under 
given that the nullhypotheis is true is:

   Prob(data | H0) = Likelihood of H0      
                   = Prob(x = 5) with x ~ binom(10, 1/6)  
                   = 0.0130

However, the p-value is:

   p-value = Prob(x = 5) + Prob(x = 6) + Prob(x = 7) + Prob(x = 8)
             + Prob(x = 9) + Prob(x = 9) + Prob(x = 10) with x ~ binom(10, 1/6)  
           = 0.0155

Thus in this case 

   p-value is not equal to Prob(data | H0),

which proves you wrong. 

You can present this to any statistican you choose for verification.



Sturla Molden

(crossposted to sci.stat.math)