greggt at strauss.udel.edu (Thomas R. Gregg) writes:
> <jwwilliams at gems.vcu.edu> wrote:
>>As per the discussion: Paul Werbos recently presented a model showing that it
>>takes 20 neurons to process one bit of information. (I don't know who
>>developed it or if its been submitted for critical review). One bit, for
>>instance, being one color. If we look at one landscape imagine the bits of
>>information that must be processes instantaneously,
>
> This ratio of "20 neurons to 1 bit" is misleading. A bit is a one or a
> zero. Neurons fire APs, lasting about 1 ms. These APs could possibly be
> considered a "bit" of one. Let us say that the average neuron can fire at
> 200 Hz. Then it is capable of processing 200 bits per second. If there
> are a billion neurons in the cerebral cortex, this is comparable to a
> 32-bit microprocessor with a 6250 MHz speed. The new Pentiums have 90 MHz
> speed.
First: one bit is not one color. One bit equals one out of two
equally likely possibilities. The amount of information in one color
is log_2(N), where N is the number of possible colors (assuming all
colors are equally likely). The only way one color is one bit is if
there are only two possible colors (a black-and-white universe!). See
any introductory book on information theory for more explanation.
Second: it is wrong to equate an action potential with one bit of
information. The amount of information (about some event) conveyed by
one AP depends on the code being employed, which in most cases is not
known with any certainty. The range of possibilities is enormous.
Look for recent papers by Bill Bialek and colleagues for some very
nice work on these issues; there are also contributions by other
people (including me) in each of the recent NIPS (Neural Information
Processing Systems) proceedings volumes.
Third: there are about ten billion neurons in the cerebral cortex (not
one billion), but it is most likely that each processes around 10 bits
per second (rather than 200). (It would take too long to explain
where this estimate comes from.) This yields a "speed" of 3125 MHz,
which is a lot faster than a Pentium but comparable to the fastest
existing supercomputers.
In any case, I think I can explain roughly how numbers along the lines
of "20 neurons to 1 bit" are arrived at. Suppose we are looking at a
group of neurons involved in motor control, which have the job of
outputting one control signal every tenth of a second, and suppose the
signal needs to be a floating-point value with 6 bits of accuracy.
Let us ask how many neurons are needed for this task, assuming that i)
information is conveyed only by the total number of spikes the cells
fire during one tenth of a second, and ii) the firing of each neuron
is approximately Poisson during one tenth of a second, with a mean
rate in the neighborhood of 20 Hz (which is not unreasonable).
The main fact we need to know to solve this problem is that, if
firing is Poisson, then the average error in the output signal will be
sqrt(2/N), where N is the total number of spikes fired. To have an
accuracy of 6 bits, we need to have sqrt(2/N) = 1/2^6; this yields
N ~= 2000 spikes, and with cells firing at 20 Hz this requires 100
cells. Therefore, with these assumptions, 100 cells are required to
get 6 bits of accuracy, for a net yield of about 17 bits per cell (per
tenth of a second).
Of course this number depends very strongly on the accuracy required,
the time available, and the values of the other numerical parameters,
but I think it makes clear how numbers in this range can arise. The
critical assumptions are that information is transmitted by net firing
rates over some time interval, and that firing is approximately
Poisson over short time intervals.
-- Bill
