190 lines
16 KiB
TeX
190 lines
16 KiB
TeX
\documentclass[12pt]{article}
|
|
\usepackage[utf8]{inputenc}
|
|
\usepackage[T1]{fontenc}
|
|
\usepackage{geometry}
|
|
\geometry{margin=1in}
|
|
\title{Competing Network Models and Problem-Solving}
|
|
\author{Converted from legacy plain-text source}
|
|
\date{}
|
|
\begin{document}
|
|
\maketitle
|
|
\begin{flushleft}
|
|
% This is a conservative automated conversion from the legacy text file.
|
|
% Manual cleanup will still be necessary for figures, references, footnotes, and formatting.
|
|
|
|
% raw formatter directive: .okidata9
|
|
% raw formatter directive: .ipr//
|
|
% raw formatter directive: .nopage break
|
|
% raw formatter directive: .page length 66
|
|
% raw formatter directive: .lpi6
|
|
% raw formatter directive: .above header 3
|
|
% raw formatter directive: .below header 3
|
|
% raw formatter directive: .above footer 3
|
|
% raw formatter directive: .below footer 3
|
|
% raw formatter directive: .page break
|
|
% raw formatter directive: .page number 1
|
|
% raw formatter directive: .pindent 15
|
|
% raw formatter directive: .pitch12
|
|
% raw formatter directive: .head//
|
|
% raw formatter directive: .foot/ \#/
|
|
% raw formatter directive: .ipr/T/
|
|
Competing Network Models and Problem-Solving
|
|
|
|
(Poster Presentation at the First Annual Meeting of the International Neural Network Society, September 6-10, 1988.)
|
|
|
|
Diane J. Blackwood, Department of Biomedical Engineering, University of Texas at Arlington
|
|
|
|
Wesley R. Elsberry, Department of Computer Science, University of Texas at Arlington
|
|
|
|
and
|
|
|
|
Sam Leven, Neural Systems and Science, 45 San Jacinto Way San Francisco, CA 94127
|
|
|
|
\section*{ABSTRACT}
|
|
Three of the most-often discussed neural networks models are analyzed and differentiated. The Hopfield, PDP, and ART models ask different questions, it is asserted -- and offer different answers for analyzing and construing complex environments. The three may not be competitors but, rather, complements. In fact, they may replicate different neural processes (Leven, 1987b). We seek to demonstrate the value of each model -- in a single case study.
|
|
|
|
The model offered by Hopfield (e.g., 1982) represents a fast-converging computable technique for analyzing highly limited classes of inputs. The PDP model (Rumelhart, et' al., 1986) offers the prospect of adoption of varied schemas, at the cost of a larger, more complex system. The ART model (e.g., Carpenter, et' al., 1987a) allows the greatest adaptability, including the capacity to vary vigilance levels and emulate many neural functions -- with the costs of much greater complexity and strain on system resources.
|
|
|
|
We present a single system, including analysis of different aspects of a problem by Hopfield, PDP, and ART networks, as an example of the potential for including many capabilities within the same environment.
|
|
|
|
% raw formatter directive: .start page
|
|
While self-criticism in the neural network community is not unusual (eg., Rumelhart, et' al., 1986, Ch' 1; Grossberg, 1987a), we may find rapprochement among "competing paradigms" more effective than the occasional nastiness we encounter. Some problems, especially in complex controls on robotics, may be best addressed by a cooperative approach.
|
|
|
|
In fact, the three paradigms most often considered mutually exclusive (Hopfield, PDP, and ART) may actually represent different neural processes (Leven, 1987a). In any case, they clearly contemplate separate issues -- and may be best in approaching distinct problems.
|
|
|
|
Hopfield's model (Hopfield, 1982; Hopfield and Tank, 1986) represents a fast-converging computable technique for analyzing stereotyped or highly limited classes of inputs. Achieved minima have the virtue of remaining highly stable (representing permanent learning). This virtue has the accompanying cost, of course, of minimizing adaptability --recognizing new aspects of data is not seriously contemplated for a stable implementation. The model has a notable tolerance for data sets containing great amounts of simple noise; however, it tends to shrink from "multi flavored" problems, which require category or schema formation in an extensive environment.
|
|
|
|
The model of the Parallel Distributed Processing (PDP) group (Rumelhart, et' al., 1986) contemplates "schema formation", seeking to apply standard cognitive psychological insights to pattern recognition and category formation processes. They have sought to take minimal anatomies and build, following the work of Schank and Abelson (1977), basic semantic structures.
|
|
|
|
The PDP school has achieved notable successes in representing language (Sejnowski, 1986) and other areas with stable knowledge domains. Where "dynamic schemata" (Schank, 1982) are generic to a problem -- where existing memory structures must be modified -- the strength of the simulated annealing algorithm becomes a weakness. Changing existing knowledge structures (by modification or replacement in the same state space) is well-nigh impossible (Yoon, et' al., 1988).
|
|
|
|
This weakness of the PDP, its stubbornness in resisting data that should produce restructured schemata, is also a strength. In certain environments, stable representations of higher-order structures (rules) coupled with the capacity to learn or be trained "up-front" may offer system designers desired control. Some systems should not be ENDLESSLY adaptive.
|
|
|
|
% raw formatter directive: .start page
|
|
Stephen Grossberg and his school (1987b \& c, Carpenter et' al. 1987a \& b) have suggested that the Adaptive Resonance (ART) model best represents higher-order neural functions. Equipped with representations for motivational processes and interactions between routines ("avalanches") and higher order structures (eg., motivational dipoles and associated READ architectures), a full- blown ART system can model highly adaptive motor tasks and emulate higher-order behaviors (Levine, 1986; Leven, 1987a \& b; and Ricart, 1988).
|
|
|
|
ART has the capacity to RECONSTRUE categories, based on continuing mismatches between data and existing higher order constructs and motivating environmental feedback. It also allows "masking fields" to eliminate from consideration whole segments of data which the system anticipates to be inappropriate or unnecessarily unsettling.
|
|
|
|
Under some circumstances, when using dipole structures to eliminate whole sets of competing representations (or rules), for example, ART can be faster -- and more effective -- than the alternatives we have presented. However, training an ART environment to perform highly routinized behaviors in which context has limited relevance has been considered more inefficient than using, say, the Hopfield model. Ordinarily, the powerful structures an ART modeler employs slow the learning process with error-checking routines which value fault- intolerance over speed. Yet, sometimes, in highly stable environments, designers may be uncomfortable with an ART system's capacity to "re-learn" essential skills they must employ.
|
|
|
|
Additionally, the rapid trainability and stability of a PDP environment may prove superior to ART, for many of the same reasons. Some higher-order rules (schemata) may be system- critical. In these cases, PROGRAMMERS SHOULD DESIGN SYSTEMS -- NOT THE SYSTEMS DESIGNING THEMSELVES. Hence, some systems may require less-intrusive network engines (like PDP) --especially when these engines also provide greater speed.
|
|
|
|
Thus, the three models for neural network design may be COMPLEMENTARY in function: Hopfield offering speed and stability, PDP providing up-front learning and stable rule structures, and ART employing context- and environment-sensitive capabilities (see Figure 1). We demonstrate, below, that modelers ought to consider these qualities in developing extensive systems -- and utilize the many effective tools at our disposal.
|
|
|
|
% raw formatter directive: .start page
|
|
\section*{EXAMPLE PROBLEM}
|
|
BEETHOVEN is a "music composition" system (see Figure 2). It provides a three part neural network model. The system emulates fundamental compositional rules to generate and perform a musical sequence.
|
|
|
|
BACH is a Hopfield net provides a sequence of notes, emulating musical melodic performance. A single voice selects notes from within a single octave. Biases are provided -- as a composer has the innate tendency to choose certain intervals and to reject notes that tend to violate common rules of harmony (eg., Aldwell and Schachter, 1978).
|
|
|
|
This network of notes is output, in sequence, to a PDP back- propagation network named SALIERI, which has learned a set of standard, somewhat higher-order harmonic rules. The network judges the effectiveness of the sequence, note by note, based on the intervals involved and the absolute note values (eg., \#7 should precede \#8 -- and, almost always, at the end of a phrase). These schemata, then, reject inappropriate sequences AND INHIBIT SOME INAPPROPRIATE NEXT NOTES. This "look-ahead" capability is unusual in a PDP environment, yet is fitting for the inhibitory role the network is playing and for the stability of the rule structure being employed.
|
|
|
|
The output from PDP flows, directly, to an ART network, BEETHOVEN. Employing a model of motivation (based on construction of category valuation and a healthy boredom at repetition), BEETHOVEN rejects "unaesthetic" sequences. As the number of phrases performed increases, the ART model develops intense biases, which it imposes on BACH and SALIERI.
|
|
|
|
One additional component of the environment is LOBES, the Context Manager. LOBES, loosely emulative of human frontal lobes (see Levine, 1986), maintains information about the processes being performed, mediates inter-model interaction, and provides for the final external output (sounding the speaker).
|
|
|
|
The model, then, utilizes the best capabilities of three distinctly different paradigms. Hopfield performs efficient routine processes, as would a "reptilian brain" (MacLean, 1970). PDP serves as an insistent schoolmarm, observing and enforcing higher-level rules, like a "neo-mammalian brain." ART provides a sense of fitness, an aesthetic fitting for models of the limbic system (or "mammalian brain").
|
|
|
|
Integration of many memory and processing functions in a three part model may be similar to human brain function (Leven, 1987b). Regardless of its biological versimilitude, however, such an approach seems to offer unique combinations of speed, stability, and flexibility.
|
|
|
|
% raw formatter directive: .start page
|
|
% raw formatter directive: .nopage break
|
|
% raw formatter directive: .page length 88
|
|
% raw formatter directive: .lpi8
|
|
% raw formatter directive: .page break
|
|
% raw formatter directive: .ipr//
|
|
% raw formatter directive: .pindent 12
|
|
% raw formatter directive: .ipr/
|
|
/
|
|
|
|
% raw formatter directive: .foot/
|
|
\#/
|
|
|
|
\section*{REFERENCES}
|
|
Aldwell, E' \& C' Schachter. 1978. Harmony and voice leading. Harcourt, Brace \& Jovanovich, New York.
|
|
|
|
Carpenter, G.A' \& S' Grossberg. 1987a. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing 37:54-115.
|
|
|
|
Carpenter, G.A' \& S' Grossberg. 1987b. ART 2: self-organization of stable category recognition codes for analog input patterns. Applied Optics 26(23):4919-4930.
|
|
|
|
Grossberg, S. 1987a' Competitive Learning: From interactive activation to adaptive resonance. Cognitive Science 11:23-63.
|
|
|
|
Grossberg, S., ed. 1987b \& c. The Adaptive Brain. Vol. I and II. Elsevier/North-Holland, Amsterdam.
|
|
|
|
Hartley, R' and H' Szu. 1987. A comparison of the computational power of neural network models. IEEE Proc.
|
|
|
|
\section*{ICNN III:15-22.}
|
|
Hopfield, J.J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79:2554-2558.
|
|
|
|
Hopfield, J.J' and D.W' Tank. 1985. "Neural" computation of decisions in optimization problems. Biol. Cybern. 52:141-152.
|
|
|
|
Hopfield, J.J' and D.W' Tank. 1986. Computing with neural circuits: A model. Science 233:625-633.
|
|
|
|
Leven, S. 1987a. Choice and neural process. Unpublished Ph.D. Dissertation, University of Texas at Arlington.
|
|
|
|
Leven, S. 1987b. S.A.M.: A triune extension to the ART model. Symposium on Neural Networks, North Texas State University. (Poster presentation)
|
|
|
|
Leven, S. 1988. Memory, helplessness, and the dynamics of hope. Presented at the Metroplex Institute for Neural Dynamics' Workshop on Motivation, Emotion, and Goal Direction in Neural Networks.
|
|
|
|
Levine, D.S. 1986. A neural network theory of frontal lobe function. In: The Proceedings of the Eighth Annual Conference of the Cognitive Science Society. Erlbaum.
|
|
|
|
MacLean, P. 1970. The triune brain, emotion, and scientific bias. In: F' Schmitt, ed. The Neurosciences: Second Study Program. Rockefeller University Press.
|
|
|
|
Ricart, R. 1988. Backward conditioning: A neural network model which exhibits both excitatory and inhibitory conditioning. Presented at the Metroplex Institute for Neural Dynamics' Workshop on Motivation, Emotion, and Goal Direction in Neural Networks.
|
|
|
|
Rumelhart, D' \& J' McClelland. 1986. Parallel Distributed Processing. MIT Press.
|
|
|
|
Schank, R. 1982. Dynamic memory. Cambridge University Press.
|
|
|
|
Schank, R.C' \& R.P' Abelson. 1977. Scripts, Plans, Goals, and Understanding. Erlbaum, Hillsdale, NJ.
|
|
|
|
Sejnowski, T.J' 1986. Open questions about computation in cerebral cortex. In: J.L. McClelland \& D.E. Rumelhart, eds. Parallel Distributed Processing Volume 2. MIT Press.
|
|
|
|
Simpson, R. 1988. A review of artificial neural systems II: Paradigms, applications, and implementations. Prepublication copy of paper submitted to CRC Critical Reviews in Artificial Intelligence.
|
|
|
|
Tank, D.W' \& J.J' Hopfield. 1986. Simple "neural" optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit. IEEE Transaction on Circuits and Systems CAS-33(5):533-541
|
|
|
|
Yoon, Y', L.L' Peterson, \& P.R' Bergstrasser. 1988. A dermatology expert system using connectionist network. Unpublished poster presentation, IEEE ICNN.
|
|
|
|
% raw formatter directive: .start page
|
|
% raw formatter directive: .nopage break
|
|
% raw formatter directive: .page length 88
|
|
% raw formatter directive: .ipr//
|
|
% raw formatter directive: .lpi8
|
|
% raw formatter directive: .page break
|
|
% raw formatter directive: .ipr/
|
|
/
|
|
|
|
% raw formatter directive: .pindent 12
|
|
% raw formatter directive: .foot/ \#/
|
|
Convergence Convergence Stability Feedback Category Mixed Data Category Computational Speed Likelihood Of Capability Formation (Complex Reconstruction Simplicity Network Environment) ------- ------- ------- ------- ------- ------- ------- -------
|
|
|
|
Hopfield + + + - - - - +
|
|
|
|
\section*{PDP 0 0 +/0 + + 0 - 0}
|
|
\section*{ART - - 0/- + ++ + + -}
|
|
Where '+' indicates a relative advantage, '0' indicates no special advantage or disadvantage, and '-' indicates a relative disadvantage.
|
|
|
|
Figure 1. Comparative analysis of features of the Hopfield, PDP, and ART artificial neural network models
|
|
|
|
% raw formatter directive: .ipr//
|
|
% raw formatter directive: .pitch12
|
|
% raw formatter directive: .ipr/T/
|
|
% raw formatter directive: .pindent 10
|
|
+-----------------+ | | (Match, Other Info) | Beethoven |---------------------+ | | |
|
|
|
|
\section*{| (ART 1) |<----------------+ |}
|
|
| | (Context) | | +-----------------+ | | \textasciicircum{} | | | | | |(Approval) | | | | |
|
|
|
|
\section*{| | V}
|
|
+-----------------+ +-----------------+ | | | | | Salieri | (Approval) | Lobes | | +------------>| (Context | | (PDP) |<------------| Management) | | | (Silence!) | | +-----------------+ +-----------------+ \textasciicircum{} \textasciicircum{} | | | (Candidate note) | | | +-------------------------+ | | | | | | | | +-----------------+ | | | | | | | Bach | (Generate Note!) | | | |<-------------------+ | | (Hopfield) | | | | | +-----------------+ (New Note) | |
|
|
|
|
+-------------------+ | | | | | Speaker | | | | | +-------------------+
|
|
|
|
Figure 2. Structure of sample system utilizing Hopfield, PDP, and ART models.
|
|
|
|
% raw formatter directive: .start page
|
|
% raw formatter directive: .nopage break
|
|
\end{flushleft}
|
|
\end{document}
|