"William Benish" <william.benish at gmail.com> writes:
> Can anyone help me support the following statement with a published reference?
>> "Fundamental information theory concepts—entropy, mutual information,
> relative entropy, and channel capacity—are functions of a more
> primitive concept, the surprisal (S). "
>> I am aware of use of the term surprisal by Tribus, but I have not
> found an information theory textbook that defines entropy as the
> expected value of the "surprisal".
The "surprisal" of observing category "i" is
S({i}) = lg[1/P({i})] = -lg[P({i})].
where P({i}) is the probability of observing category "i;"
it may be interpreted as the average number of "Yes/No" questions
(i.e., number of "bits") required to ascertain that category "i"
has been observed.
By definition of "Expectation Value," the "expected surprisal" is
<S> = \Sum_i P({i}) * lg[1/P({i}})]
= - \Sum_i P({i}) * lg[P({i}})]
The above is exactly Shannon's entropy.
-- Gordon D. Pusch
perl -e '$_ = "gdpusch\@NO.xnet.SPAM.com\n"; s/NO\.//; s/SPAM\.//; print;'
