[Bio-information-theory] Re: A question about surprisals

[Gordon D. Pusch]

"William Benish" <william.benish at gmail.com> writes:

> Can anyone help me support the following statement with a published reference?
> 
> 	"Fundamental information theory concepts—entropy, mutual information,
> relative entropy, and channel capacity—are functions of a more
> primitive concept, the surprisal (S). "
> 
> I am aware of use of the term surprisal by Tribus, but I have not
> found an information theory textbook that defines entropy as the
> expected value of the "surprisal".

The "surprisal" of observing category "i" is 

  S({i}) = lg[1/P({i})] = -lg[P({i})].

where P({i}) is the probability of observing category "i;"
it may be interpreted as the average number of "Yes/No" questions
(i.e., number of "bits") required to ascertain that category "i"
has been observed.

By definition of "Expectation Value," the "expected surprisal" is

  <S> = \Sum_i P({i}) * lg[1/P({i}})]
      = - \Sum_i P({i}) * lg[P({i}})]

The above is exactly Shannon's entropy.


-- Gordon D. Pusch   

perl -e '$_ = "gdpusch\@NO.xnet.SPAM.com\n"; s/NO\.//; s/SPAM\.//; print;'