.okidata9
.ipr//
.nopage break
.page length 66
.lpi6
.above header 3
.below header 3
.above footer 3
.below footer 3
.page break
.page number 1
.pindent 15
.pitch12
.head//
.foot/                                                            #/
.ipr/T/




          Competing Network Models and Problem-Solving


     (Poster Presentation at the First Annual Meeting of the
   International Neural Network Society, September 6-10, 1988.)

                       Diane J. Blackwood, 
            Department  of  Biomedical  Engineering,  
                University of Texas at Arlington

                       Wesley R. Elsberry, 
                 Department of Computer Science,
                University of Texas at Arlington

                               and

                           Sam Leven, 
                   Neural Systems and Science,
                       45 San Jacinto Way
                     San Francisco, CA 94127


                            ABSTRACT

     Three of the most-often discussed neural networks models are
analyzed  and  differentiated.  The Hopfield, PDP, and ART models
ask different questions, it is asserted --  and  offer  different
answers  for  analyzing and construing complex environments.  The
three may not be competitors but, rather, complements.  In  fact,
they may replicate different neural processes (Leven, 1987b).  We
seek  to  demonstrate the value of each model -- in a single case
study.

     The  model  offered  by  Hopfield  (e.g., 1982) represents a
fast-converging computable technique for analyzing highly limited
classes of inputs.  The PDP  model  (Rumelhart,  et  al.,  1986)
offers the prospect of adoption of varied schemas, at the cost of
a  larger,  more complex system.  The ART model (e.g., Carpenter,
et al., 1987a) allows the greatest adaptability,  including  the
capacity  to  vary  vigilance  levels  and  emulate  many  neural
functions -- with the costs of much greater complexity and strain
on system resources.

     We  present a single system, including analysis of different
aspects of a problem by Hopfield, PDP, and ART  networks,  as  an
example  of  the potential for including many capabilities within
the same environment.
.start page


     While  self-criticism in the neural network community is not
unusual (eg., Rumelhart, et al., 1986, Ch 1; Grossberg, 1987a),
we  may  find  rapprochement  among  "competing  paradigms"  more
effective  than  the  occasional  nastiness  we  encounter.  Some
problems, especially in complex controls on robotics, may be best
addressed by a cooperative approach.

     In  fact, the three paradigms most often considered mutually
exclusive  (Hopfield,  PDP,  and  ART)  may  actually   represent
different  neural  processes  (Leven,  1987a).  In any case, they
clearly contemplate  separate  issues  --  and  may  be  best  in
approaching distinct problems.

     Hopfield's  model  (Hopfield, 1982; Hopfield and Tank, 1986)
represents a fast-converging computable technique  for  analyzing
stereotyped or highly limited classes of inputs.  Achieved minima
have   the   virtue  of  remaining  highly  stable  (representing
permanent learning).  This virtue has the accompanying  cost,  of
course,  of  minimizing adaptability --recognizing new aspects of
data is not seriously contemplated for a  stable  implementation.
The  model has a notable tolerance for data sets containing great
amounts of simple noise; however, it tends to shrink from  "multi
flavored" problems, which require category or schema formation in
an extensive environment.

     The model of the Parallel Distributed Processing (PDP) group
(Rumelhart,  et  al.,  1986)  contemplates  "schema  formation",
seeking to apply standard  cognitive  psychological  insights  to
pattern  recognition and category formation processes.  They have
sought to take minimal anatomies and build, following the work of
Schank and Abelson (1977), basic semantic structures.

     The   PDP   school   has   achieved   notable  successes  in
representing language (Sejnowski,  1986)  and  other  areas  with
stable  knowledge  domains.   Where  "dynamic  schemata" (Schank,
1982)  are  generic  to  a  problem  --  where  existing   memory
structures  must  be  modified  --  the strength of the simulated
annealing  algorithm  becomes  a  weakness.   Changing   existing
knowledge  structures (by modification or replacement in the same
state space) is well-nigh impossible (Yoon, et al., 1988).

     This weakness of the PDP, its stubbornness in resisting data
that  should  produce  restructured schemata, is also a strength.
In certain environments, stable representations  of  higher-order
structures  (rules)  coupled  with  the  capacity  to learn or be
trained "up-front" may offer system  designers  desired  control.
Some systems should not be ENDLESSLY adaptive.
.start page

     Stephen  Grossberg  and his school (1987b & c, Carpenter et
al.  1987a & b) have suggested that the Adaptive Resonance  (ART)
model  best  represents  higher-order neural functions.  Equipped
with representations for motivational processes and  interactions
between routines ("avalanches") and higher order structures (eg.,
motivational  dipoles and associated READ architectures), a full-
blown ART system  can  model  highly  adaptive  motor  tasks  and
emulate  higher-order  behaviors (Levine, 1986; Leven, 1987a & b;
and Ricart, 1988).

     ART  has  the  capacity  to  RECONSTRUE categories, based on
continuing mismatches between  data  and  existing  higher  order
constructs and motivating environmental feedback.  It also allows
"masking  fields"  to eliminate from consideration whole segments
of data which the  system  anticipates  to  be  inappropriate  or
unnecessarily unsettling.

     Under  some  circumstances,  when using dipole structures to
eliminate whole sets of competing representations (or rules), for
example, ART can be faster -- and  more  effective  --  than  the
alternatives   we  have  presented.   However,  training  an  ART
environment to  perform  highly  routinized  behaviors  in  which
context   has   limited   relevance   has  been  considered  more
inefficient than using, say, the Hopfield model.  Ordinarily, the
powerful structures an ART  modeler  employs  slow  the  learning
process   with   error-checking   routines   which  value  fault-
intolerance  over  speed.   Yet,  sometimes,  in  highly   stable
environments, designers may be uncomfortable with an ART system's
capacity to "re-learn" essential skills they must employ.

     Additionally,  the rapid trainability and stability of a PDP
environment may prove superior to  ART,  for  many  of  the  same
reasons.   Some  higher-order  rules  (schemata)  may  be system-
critical.  In these cases, PROGRAMMERS SHOULD DESIGN  SYSTEMS  --
NOT  THE  SYSTEMS  DESIGNING THEMSELVES.  Hence, some systems may
require less-intrusive network engines  (like  PDP)  --especially
when these engines also provide greater speed.

     Thus,  the  three  models  for  neural network design may be
COMPLEMENTARY in function: Hopfield offering speed and stability,
PDP providing up-front learning and stable rule  structures,  and
ART  employing  context-  and  environment-sensitive capabilities
(see Figure 1).  We demonstrate, below, that  modelers  ought  to
consider  these  qualities in developing extensive systems -- and
utilize the many effective tools at our disposal.

.start page

EXAMPLE PROBLEM

     BEETHOVEN  is  a  "music composition" system (see Figure 2).
It provides a  three  part  neural  network  model.   The  system
emulates  fundamental compositional rules to generate and perform
a musical sequence.

     BACH  is  a  Hopfield  net  provides  a  sequence  of notes,
emulating musical melodic performance.  A  single  voice  selects
notes  from  within a single octave.  Biases are provided -- as a
composer has the innate tendency to choose certain intervals  and
to  reject  notes  that  tend  to violate common rules of harmony
(eg., Aldwell and Schachter, 1978).

     This network of notes is output, in sequence, to a PDP back-
propagation  network  named  SALIERI,  which has learned a set of
standard, somewhat  higher-order  harmonic  rules.   The  network
judges  the effectiveness of the sequence, note by note, based on
the intervals involved and the  absolute  note  values  (eg.,  #7
should precede #8 -- and, almost always, at the end of a phrase).
These  schemata, then, reject inappropriate sequences AND INHIBIT
SOME INAPPROPRIATE NEXT NOTES.  This "look-ahead"  capability  is
unusual  in  a PDP environment, yet is fitting for the inhibitory
role the network is playing and for the  stability  of  the  rule
structure being employed.

     The  output  from  PDP  flows,  directly, to an ART network,
BEETHOVEN.   Employing  a   model   of   motivation   (based   on
construction  of  category  valuation  and  a  healthy boredom at
repetition), BEETHOVEN rejects "unaesthetic" sequences.   As  the
number  of  phrases  performed  increases, the ART model develops
intense biases, which it imposes on BACH and SALIERI.

     One  additional  component  of the environment is LOBES, the
Context Manager.  LOBES, loosely emulative of human frontal lobes
(see Levine, 1986), maintains  information  about  the  processes
being  performed,  mediates inter-model interaction, and provides
for the final external output (sounding the speaker).

     The  model,  then,  utilizes  the best capabilities of three
distinctly  different  paradigms.   Hopfield  performs  efficient
routine  processes, as would a "reptilian brain" (MacLean, 1970).
PDP serves as an insistent schoolmarm,  observing  and  enforcing
higher-level rules, like a "neo-mammalian brain."  ART provides a
sense  of  fitness, an aesthetic fitting for models of the limbic
system (or "mammalian brain").

     Integration  of  many  memory  and processing functions in a
three part model may be similar to human brain  function  (Leven,
1987b).   Regardless  of  its  biological versimilitude, however,
such an approach seems to offer  unique  combinations  of  speed,
stability, and flexibility.

.start page
.nopage break
.page length 88
.lpi8
.page break
.ipr//
.pindent 12
.ipr//
.foot/                                                                                                 #/

REFERENCES


Aldwell, E & C Schachter.  1978.  Harmony and voice leading.  Harcourt, Brace & Jovanovich, New York.

Carpenter,  G.A  &  S  Grossberg.   1987a.   A  massively parallel architecture for a self-organizing neural
    pattern recognition machine.  Computer Vision, Graphics, and Image Processing 37:54-115.

Carpenter,  G.A  &  S  Grossberg.  1987b.  ART 2: self-organization of stable category recognition codes for
    analog input patterns.  Applied Optics 26(23):4919-4930.

Grossberg, S.  1987a Competitive Learning: From interactive activation to adaptive resonance.
    Cognitive Science 11:23-63.

Grossberg, S., ed.  1987b & c.  The Adaptive Brain.  Vol.  I and II.  Elsevier/North-Holland, Amsterdam.

Hartley,  R and H Szu.  1987.  A comparison of the computational power of neural network models.  IEEE Proc.
    ICNN III:15-22.

Hopfield,  J.J.  1982.  Neural networks and physical systems with emergent collective computational abilities.
    Proc.  Natl.  Acad.  Sci.  USA 79:2554-2558.

Hopfield,  J.J  and  D.W  Tank.   1985.   "Neural" computation of decisions in optimization problems.  Biol.
    Cybern.  52:141-152.

Hopfield, J.J and D.W Tank.  1986.  Computing with neural circuits: A model.  Science 233:625-633.

Leven,  S.   1987a.   Choice  and  neural  process.   Unpublished  Ph.D.  Dissertation, University of Texas at
    Arlington.

Leven, S.  1987b.  S.A.M.: A triune extension to the ART model.  Symposium on Neural Networks, North
    Texas State University.  (Poster presentation)

Leven,  S.   1988.   Memory, helplessness, and the dynamics of hope.  Presented at the Metroplex Institute for
    Neural Dynamics' Workshop on Motivation, Emotion, and Goal Direction in Neural Networks.

Levine,  D.S.   1986.   A  neural  network theory of frontal lobe function.  In: The Proceedings of the Eighth
    Annual Conference of the Cognitive Science Society.  Erlbaum.

MacLean, P.  1970.  The triune brain, emotion, and scientific bias.  In: F Schmitt, ed.  The
    Neurosciences: Second Study Program.  Rockefeller University Press.

Ricart, R.  1988.  Backward conditioning: A neural network model which exhibits both excitatory and inhibitory
    conditioning.   Presented at the Metroplex Institute for Neural Dynamics' Workshop on Motivation, Emotion,
    and Goal Direction in Neural Networks.

Rumelhart, D & J McClelland.  1986.  Parallel Distributed Processing.  MIT Press.

Schank, R.  1982.  Dynamic memory.  Cambridge University Press.

Schank, R.C & R.P Abelson.  1977.  Scripts, Plans, Goals, and Understanding.  Erlbaum, Hillsdale, NJ.

Sejnowski,  T.J  1986.  Open  questions  about  computation in cerebral cortex.  In: J.L.  McClelland & D.E.
    Rumelhart, eds.  Parallel Distributed Processing Volume 2.  MIT Press.

Simpson,  R.   1988.   A review of artificial neural systems II: Paradigms, applications, and implementations.
    Prepublication copy of paper submitted to CRC Critical Reviews in Artificial Intelligence.

Tank,  D.W  & J.J Hopfield.  1986.  Simple "neural" optimization networks: An A/D converter, signal decision
    circuit, and a linear programming circuit.  IEEE Transaction on Circuits and Systems CAS-33(5):533-541

Yoon, Y, L.L Peterson, & P.R Bergstrasser.  1988.  A dermatology expert system using connectionist
    network.  Unpublished poster presentation, IEEE ICNN.

.start page
.nopage break
.page length 88
.ipr//
.lpi8
.page break
.ipr//
.pindent 12
.foot/                                                                                                   #/




          Convergence  Convergence  Stability  Feedback   Category    Mixed Data     Category   Computational
             Speed     Likelihood      Of     Capability  Formation    (Complex   Reconstruction  Simplicity
                                     Network                         Environment)
            -------      -------     -------    -------    -------     -------       -------       -------

Hopfield       +            +           +          -          -           -             -             +

PDP            0            0          +/0         +          +           0             -             0

ART            -            -          0/-         +         ++           +             +             -

Where '+' indicates a relative advantage, '0' indicates no special advantage or disadvantage, 
  and '-' indicates a relative disadvantage.

Figure 1.  Comparative analysis of features of the Hopfield, PDP, and ART artificial neural network models


.ipr//
.pitch12
.ipr/T/
.pindent 10





                 +-----------------+
                 |                 |  (Match, Other Info)
                 |    Beethoven    |---------------------+
                 |                 |                     |
                 |     (ART 1)     |<----------------+   |
                 |                 |    (Context)    |   |
                 +-----------------+                 |   |
                          ^                          |   |
                          |                          |   |
                          |(Approval)                |   |
                          |                          |   |
                          |                          |   V
                 +-----------------+             +-----------------+
                 |                 |             |                 |
                 |     Salieri     | (Approval)  |      Lobes      | 
                 |                 +------------>|    (Context     |
                 |      (PDP)      |<------------|   Management)   |
                 |                 |  (Silence!) |                 |
                 +-----------------+             +-----------------+
                          ^                         ^   |     |
                          |   (Candidate note)      |   |     |
                          +-------------------------+   |     |
                          |                             |     |
                          |                             |     |
                 +-----------------+                    |     |
                 |                 |                    |     |
                 |      Bach       |  (Generate Note!)  |     |
                 |                 |<-------------------+     |
                 |   (Hopfield)    |                          |
                 |                 |                          |
                 +-----------------+               (New Note) |
                                                              |
                                                              V
                                             +-------------------+
                                             |                   |
                                             |                   |
                                             |      Speaker      |
                                             |                   |
                                             |                   |
                                             +-------------------+


Figure 2.  Structure of sample system utilizing Hopfield, PDP, and ART models.
.start page
.nopage break

