machine brains

[David Longley]

In article <MPG.ed4ed5b4d7b67a398968a at news3.wt.net>
           solntsev at wt.net "Alexander Solntsev" writes:

> I want to make a couple of points.
> 
> First on the subject of neural nets: one should not try to extend the 
> capabilities of such computing device to its use as a model of a brain 
> simply on the basis both apply similar terminology. For it is exactly that 
> which some researchers (and others) appear to be doing. A neural net is an 
> interesting computing device that is both inartistically parallel and 
> distributed. Its operation is determined by multiple interconnected nodes, 
> not unlike that of a brain. Yet it is not the only device that exhibits 
> such properties. Actually, any system of multiple cooperating 
> interconnected nodes could look like a brain. We should avoid making 
> parallels between computing devices and brain unless we can reasonably show 
> such devices as explaining more than one aspect of brains operation.
> 
> Second on the subject of linearity: most of us have became trapped in the 
> Von Neumann world of sequential computation with only neural nets seen as 
> the way out. Well, thank goodness, there are additional inherently parallel 
> and distributed computing devices that could save the day. While there are 
> several such devices, I will name the Dataflow computers specifically for I 
> find them most usefull. Actually, a neural net could be considered a 
> special case of a Dataflow computer. What is a Dataflow computer? It is a 
> device composed on multiple cooperating interconnected nodes that represent 
> a dataflow graph of the computation they perform. Each node can have 
> multiple inputs and outputs connecting it to other nodes. Each node 
> represents a computing function based on those inputs, and that function is 
> executed when these inputs become available. All nodes exchange information 
> using messages that could represent complex data structures.
> 
> Just a couple of points (for now as this message is getting too long :-))
> 
> Alex

I'm  not  going  to apologise for length as to discuss  any  of  these 
issues  usefully requires books usually. The following is  an  extract 
from "Fragments" on the subject. I would tidy up my usage of inductive 
vs.  deductive a little today, but the major theme remains. Whilst  we 
might be able to model some of the adaptive processes which biological 
systems  seem to instantiate, *THIS IS NOT AI BUT PSYCHOLOGY. We  know 
that  such systems are good models of the heuristics which seem to  be 
part of simple and complex animals' repertoire, and my thesis is  that 
we  have,  largely  through collective social  efforts,  discovered  a 
technology (the extensional stance) which goes far beyond what any  of 
these  adaptive  systems  are capable of. That stance has  led  to  an 
extensive  technology  based on science, and it is THAT which  we  are 
really talking about when we bandy the work "Artificial  Intelligence" 
about. That's why just about every "AI" development so rapidly becomes 
just  one more aspect of our engineering - it *IS* just an example  of 
engineering (albeit often hyped through anthropomorphic metaphor). 

In  my view, the obstacle to this being more widely understood is  the 
muddle which currently pervades so much of contemporary psychology and 
cognitive  science. Whilst it may need some further work to  make  the 
point as clear as it should be, the seeds of this idea are outlined in 
"Fragments  of  Behaviour: The  Extensional  Stance"  (1993;1994;1997) 
which provides the theoretical background for how the philosophy of AI 
might be usefully applied in the field of "Corrections".

                                o o o

    'Connectionist  networks  are well  suited  to  everyday 
    common sense reasoning. Their ability to  simultaneously 
    satisfy  soft  constraints allows them  to  select  from 
    conflicting   information   in   finding   a   plausible 
    interpretation  of a situation. However, these  networks 
    are  poor at reasoning using the standard  semantics  of 
    classical logic, based on truth in all possible models.'

    M. Derthick (1990)
    Mundane Reasoning by Settling on a Plausible Model
    Artificial Intelligence 46,1990,107-157

Connectionist  systems, it is claimed, do not represent  knowledge  as 
production rules, ie as well-formed-formulae represented in the syntax 
of  the  predicate calculus (using conditionals, modus  ponens,  modus 
tollens  and  the  quantifiers), but  as  connection  weights  between 
activated predicates in a parallel distributed network:

    'Lawful  behavior  and judgments may be  produced  by  a 
    mechanism  in which there is no explicit  representation 
    of  the  rule. Instead, we suggest that  the  mechanisms 
    that   process   language   and   make   judgments    of 
    grammaticality are constructed in such a way that  their 
    performance  is characterizable by rules, but  that  the 
    rules  themselves  are  not  written  in  explicit  form 
    anywhere in the mechanism.'

    D E Rumelhart and D McClelland (1986)
    Parallel Distributed Processing Ch. 18

Such    systems   are   function-approximation   systems,   and    are 
mathematically  a development of Kolmogorov's Mapping  Neural  Network 
Existence  Theorem  (1957). Such networks consist of three  layers  of 
processing  elements. Those of the bottom layer simply distribute  the 
input  vector (a pattern of 1s and 0s) to the processing  elements  of 
the  second  layer. The processing elements of this middle  or  hidden 
layer implement a *'transfer function'* (more on this below). The  top 
layer are output units. 

An  important  feature  of Kolmogorov's Theorem, is  that  it  is  not 
constructive. That is, it is not algorithmic or 'effective'. Since the 
proof  of  the  theorem is not constructive, we do  not  know  how  to 
determine  the key quantities of the transfer functions.  The  theorem 
simply tells us that such a three layer mapping network must exist. As 
Hecht-Nielsen (1990) remarks:

    'Unfortunately,  there  does not appear to be  too  much 
    hope  that  a method of finding the  Kolmogorov  network 
    will  be developed soon. Thus, the value of this  result 
    is  its  intellectual assurance that  continuous  vector 
    mappings   of  a  vector  variable  on  the  unit   cube 
    (actually,  the theorem can be extended to apply to  any 
    COMPACT, ie, closed and bounded, set) can be implemented 
    EXACTLY with a three-layer neural network.'

    R. Hecht-Nielsen (1990)
    Kolmogorov's Theorem
    Neurocomputing

That is, we may well be able to find weight-matrices which capture  or 
embody certain functions, but we may not be able to say  'effectively' 
what  the  precise equations are which  algorithmically  compute  such 
functions.  This is often summarised by statements to the effect  that 
neural  networks  can model or fit solutions to sample  problems,  and 
generalise  to  new cases, but they can not provide a rule as  to  how 
they  make such classifications or inferences. Their ability to do  so 
is  distributed  across the weightings of the whole weight  matrix  of 
connections  between the three layers of the network. The above is  to 
be  contrasted  with the fitting of linear discriminant  functions  to 
partition  or  classify  an  N dimensional space  (N  being  a  direct 
function   of   the  number  of  classes  or   predicates).   Fisher's 
discriminant  analysis  (and  the  closely  related  linear   multiple 
regression   technology)   arrive   at   the   discriminant   function 
coefficients through the Gaussian method of Least Mean Squares, each b 
value  and the constant being arrived at deductively via the  solution 
of   simultaneous   equations.   Function   approximation,   or    the 
determination  of  hidden  layer weights or connections  is  based  on 
recursive feedback, elsewhere within behaviour science, this is  known 
as  'reinforcement',  the differential strengthening or  weakening  of 
connections  depending  on feedback or knowledge of  results.  Kohonen 
(1988)   commenting   on  "Connectionist  Models"   in   contrast   to 
conventional, extensionalist relational databases, writes:

    'Let  me make it completely clear that one of  the  most 
    central functions coveted by the "connectionist"  models 
    is the ability to solve *simplicitly defined  relational 
    structures*.  The latter, as explained in  Sect.  1.4.5, 
    are  defined  by  *partial relations*,  from  which  the 
    structures are determined in a very much similar way  as 
    solutions to systems of algebraic equations are  formed; 
    all  the  values  in the  universe  of  variables  which 
    satisfy  the  conditions  expressed  as  the   equations 
    comprise, by definition, the possible solutions. In  the 
    relational    structures,   the    knowledge    (partial 
    statements,   partial   relations)  stored   in   memory 
    constitutes  the universe of variables, from  which  the 
    solutions  must be sought; and the conditions  expressed 
    by  (eventually incomplete) relations, ie, the  "control 
    structure" [9.20] correspond to the equations.

    Contrary  to  the conventional database  machines  which 
    also  have  been  designed  to  handle  such  relational 
    structures, the "connectionist" models are said to  take 
    the relations, or actually their strengths into  account 
    statistically. In so doing, however they only apply  the 
    Euclidean  metric, or the least square loss function  to 
    optimize   the  solution.  This  is  not  a  very   good 
    assumption for natural data.'

    T. Kohonen (1988)
    Ch. 9 Notes on Neural Computing
    In Self-Organisation and Associative Memory
     
Throughout  the  1970s  Nisbett  and colleagues  studied  the  use  of 
probabilistic   heuristics  in  real  world  human  problem   solving, 
primarily in the context of Attribution Theory (H. Kelley 1967, 1972). 
Such  inductive  as opposed to deductive heuristics  of  inference  do 
indeed  seem  to be influenced by training (Nisbett and  Krantz  1983, 
Nisbett et. al 1987). Statistical heuristics are naturally applied  in 
everyday  reasoning  if  subjects  are trained in  the  Law  of  Large 
Numbers. This is not surprising, since application of such  heuristics 
is an example of response generalisation - which is how  psychologists 
have  traditionally  studied the vicissitudes of  inductive  inference 
within  Learning  Theory.  As Wagner (1981) has pointed  out,  we  are 
perfectly  at liberty to use the language of Attribution Theory as  an 
alternative,  this  exchangeability  of  reference  system  being   an 
instance of Quinean Ontological Relativity, where what matters is  not 
so  much  the  names in argument positions,  or  even  the  predicates 
themselves,  but  the  *relations*  (themselves  at  least   two-place 
predicates) which emerge from such systems.

Under  most natural circumstances, inductive inference  is  irrational 
(cf.  Popper 1936, Kahneman et al. 1982, Dawes, Faust and Meehl  1989, 
Sutherland   1992).  This  is  because  it  is  generally   based   on 
unrepresentative   sampling   (drawing  on  the   'availability'   and 
'representativeness'  heuristics), and this is so simply because  that 
is  how data in a structured culture often naturally presents  itself. 
Research has therefore demonstrated that human inference is  seriously 
at  odds with formal deductive logical reasoning, and the  algorithmic 
implementation  of  those inferential processes by  computers  (Church 
1936, Post 1936, Turing 1936). One of the main points of this paper is 
that  we  generally  turn  to  the  formal  deductive  technology   of 
mathematico-logical method (science) to compensate for the  heuristics 
and  biases which typically characterise natural inductive  inference. 
Where possible, we turn to *relational databases and 4GLs*  (recursive 
function  theory and mathematical logic) to provide  descriptive,  and 
deductively valid pictures of individuals and collectives.

This  large, and unexpected body of empirical evidence from  decision-
theory, cognitive experimental social psychology and Learning  Theory, 
began accumulating in the mid to late 1970s (cf. Kahneman, Tversky and 
Slovic 1982, Putnam 1986, Stich 1990), and began to cast serious doubt 
on  the  viability  of  the  'computational  theory'  of  mind  (Fodor 
1975,1980)  which was basic to functionalism (Putnam 1986).  That  is, 
the  substantial body of empirical evidence which  accumulated  within 
Cognitive  Psychology itself suggested that, contrary to the  doctrine 
of  functionalism,  there exists a system  of  independent,  objective 
knowledge,  and reasoning against which we can judge human, and  other 
animal cognitive processing. However, it gradually became  appreciated 
that  the  digital computer is not a good model of  human  information 
processing, at least not unless this is conceived in terms of  'neural 
computing'  (also  known as 'connectionism' or  'Parallel  Distributed 
Processing). The application of formal rules of logic and  mathematics 
to  the  analysis of behaviour solely within the  language  of  formal 
logic  is the professional business of Applied  Behaviour  Scientists. 
Outside  of the practice of those professional skills,  the  scientist 
himself is as prone to the irrationality of intensional heuristics  as 
are laymen (Wason 1966). Within the domain of formal logic applied  to 
the  analysis of behaviour, the work undertaken by applied  scientists 
is impersonal. The scientists' professional views are dictated by  the 
laws   of   logic  and  mathematics  rather  than   personal   opinion 
(heuristics).

The alternative, intensional heuristics, which are the mark of natural 
human  judgement  (hence  our rich folk  psychological  vocabulary  of 
metaphor)  have  to  be  contrasted  with  extensional  analysis   and 
judgement  using technology based on the deductive algorithms  of  the 
First Order Predicate Calculus (Relational Database Technology).  This 
is  not  only  coextensive with the 'scope and  language  of  science' 
(Quine  1954) but is also, to the best of our knowledge from  research 
in  Cognitive  Psychology,  an effective compensatory  system  to  the 
biases of natural intensional, inductive heuristics (Agnoli and Krantz 
1989). Whilst a considerable amount of evidence suggests that training 
in formal logic and statistics is not in itself sufficient to suppress 
usage  of  intensional  heuristics  in any  enduring  sense,  ie  that 
generalisation  to  extra-training  contexts  is  limited,  there   is 
evidence  that judgement can be rendered more rational by training  in 
the  use of extensional technology. The demonstration by Kahneman  and 
Tversky  1983, that subjects generally fail to apply  the  extensional 
conjunction rule in probability that conjunctions are always equal  or 
less  probable  than  its elements, and that  this  too  is  generally 
resistant  to counter-training, is another example, this  time  within 
probability theory (a deductive system) of the failure of  extensional 
rules  in  applied  contexts. Careful use of I.T.  and  principles  of 
deductive  inference  (e.g. semantic tableaux,  Herbrand  models,  and 
Resolution  methods)  promise, within the limits  imposed  by  Godel's 
Theorem,  to  keep us on track if we restrict our  technology  to  the 
extensional.

The reawakening of interest in connectionism in the early to mid 1980s 
can  indeed  be  seen  as a vindication of  the  basic  principles  of 
behaviourism. What is psychological may well be impenetrable, for  any 
serious scientific purposes, not because it is in any way a  different 
kind  of 'stuff', but because structurally it amounts to no more  than 
an n-dimensional weight space, idiosyncratic and context specific,  to 
each and every one of us. 

    'Uncertain situations may be thought of as  disjunctions 
    of  possible  states: either one state will  obtain,  or 
    another....

    Shortcomings in reasoning have typically been attributed 
    to   quantitative   limitations  of  human   beings   as 
    processors of information. "Hard problems" are typically 
    characterized  by reference to the "amount of  knowledge 
    required," the "memory load," or the "size of the search 
    space"....Such limitations, however, are not  sufficient 
    to account for all that is difficult about thinking.  In 
    contrast  to many complicated tasks that people  perform 
    with  relative ease, the problems investigated  in  this 
    paper  are  computationally  very  simple,  involving  a 
    single  disjunction  of  two well  defined  states.  The 
    present   studies  highlight  the  discrepancy   between 
    logical  complexity  on the one hand  and  psychological 
    difficulty  on  the  other. In contrast  to  the  "frame 
    problem"  for example, which is trivial for  people  but 
    exceedingly  difficult  for  AI, the  task  of  thinking 
    through disjunctions is trivial for AI (which  routinely 
    implements "tree search" and "path finding"  algorithms) 
    but  very  difficult for people. The failure  to  reason 
    consequentially may constitute a fundamental  difference 
    between natural and artificial intelligence.'

    E. Shafir and A. Tversky (1992)
    Thinking through Uncertainty: Nonconsequantial Reasoning 
    and Choice
    Cognitive Psychology 24,449-474