welsberr
/
TriuneCadence
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
TriuneCadence/THES/INT_ANN.TXT

15445 lines
312 KiB
Plaintext
Raw Blame History

INTEGRATION AND HYBRIDIZATION IN NEURAL NETWORK MODELLING
The members of the Committee approve the masters thesis of
Wesley Royce Elsberry
Karan Briggs
_____________________________________
Supervising Professor
Lynn Peterson
_____________________________________
Daniel Levine
_____________________________________
Copyright by Wesley Royce Elsberry 1989
All rights reserved
INTEGRATION AND HYBRIDIZATION IN NEURAL NETWORK MODELLING
by
WESLEY ROYCE ELSBERRY
Presented to the Faculty of the Graduate School of
The University of Texas at Arlington in Partial Fulfillment
of the Requirements
for the Degree of
MASTER OF SCIENCE IN COMPUTER SCIENCE
THE UNIVERSITY OF TEXAS AT ARLINGTON
August 1989
ACKNOWLEDGEMENTS
I wish to thank the many people who have made my graduate program a
rewarding, enlightening, and interesting experience. My especial thanks
to Bob Weems, Ken Youngers, Vijay Raj, Steve Hufnagel, and Farhad Kamangar
for their exemplary instruction. The advice and encouragement of Bill
Buckles was critically important to this refugee from the life sciences.
The members of my graduate committee, Karan Briggs, Lynn Peterson, and
Daniel Levine, have provided me with technical resources, instruction,
referrals, and general good advice in plenty. Sam Leven provided the
example problem description, the classical sequence data, and much of his
own time and expertise to aid me in developing both the simulation program
and my understanding of the field. I am indebted to Dr. Levine for his
great interest in teaching the principles of cognitive modelling to as
wide an audience as possible, for without his express encouragement I
would not have become acquainted with the field, and also for his
considerable personal assistance in developing this thesis. The
enthusiasm of Harold Szu helped to motivate me to undertake a deeper
inquiry into neural network modelling. Finally, without the continual
support of my spouse, Diane Blackwood, this thesis and the classroom work
which formed the basis for it would not have been possible.
July 28, 1989
iv
ABSTRACT
INTEGRATION AND HYBRIDIZATION IN NEURAL NETWORK MODELLING
Publication No.
_______
Wesley Royce Elsberry, M.S.
The University of Texas at Arlington, 1989
Supervising Professor: Karan Briggs
Artificial neural network models derived from different biological
behaviors or functions can be used in an integrative fashion to create an
extensive problem-solving environment. An example problem of limited
melodic composition is approached by the use of Hopfield-Tank, back-
propagation, and Adaptive Resonance Theory networks serving as plausible
next note generator, musical sequence critic, and novelty detector,
respectively. Biological bases for integrative function are discussed,
and the experimental role of synthetic systems such as the example
integrated network is explored.
v
TABLE OF CONTENTS
ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
LIST OF ILLUSTRATIONS . . . . . . . . . . . . . . . . . . . . . . . . viii
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Chapter
1. DEFINING THE ROLE AND NATURE OF ARTIFICIAL NEURAL NETWORK
MODELLING . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. INTEGRATION IN NEURAL NETWORK MODELLING . . . . . . . . . . . 33
3. EXAMPLE PROBLEM . . . . . . . . . . . . . . . . . . . . . . . 43
4. RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5. DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Appendix
1. SAMPLE MELODY OUTPUT OF THE VARIOUS NOTE GENERATOR PROGRAMS . 66
2. PROGRAM SOURCE LISTING: INTEGRATED ANN NOTE GENERATOR . . . . 79
3. PROGRAM SOURCE LISTING: BACK-PROPAGATION UNIT . . . . . . . . 115
4. PROGRAM SOURCE LISTING: LIST STRUCTURES UNIT . . . . . . . . . 153
5. PROGRAM SOURCE LISTING: MISCELLANEOUS PROCEDURES UNIT . . . . 160
6. PROGRAM SOURCE LISTING: GLOBAL TYPE AND VARIABLES UNIT . . . . 172
7. PROGRAM SOURCE LISTING: CLASSICAL INSTRUCTOR UNIT . . . . . . 177
8. PROGRAM SOURCE LISTING: ANSI SCREEN CONTROL UNIT . . . . . . . 181
9. PROGRAM SOURCE LISTING: MUSICAL SEQUENCE EVALUATOR PROGRAM . . 190
10. PROGRAM SOURCE LISTING: RANDOM NOTE GENERATION PROGRAM . . . . 193
11. PROGRAM SOURCE LISTING: NOTE SEQUENCE PLAYING PROGRAM . . . . 195
vi
Appendix Page
12. PROGRAM SOURCE LISTING: RULE-BASED NOTE SEQUENCE GENERATION
PROGRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
13. PROGRAM SOURCE LISTING: OFFLINE BACK-PROPAGATION NETWORK
TRAINING PROGRAM . . . . . . . . . . . . . . . . . . . . . . . 202
14. DATA FILE LISTING: HOPFIELD-TANK NETWORK WEIGHT DATA FILE . . 208
15. DATA FILE LISTING: CLASSICAL SEQUENCES DATA FILE . . . . . . . 214
16. DATA FILE LISTING: BACK-PROPAGATION NETWORK DATA FILE . . . . 216
17. PROGRAM SOURCE LISTING: TRANSLATOR FROM PROGRAM NOTE FILES TO
MUSIC TRANSCRIPTION SYSTEM SONG FORMAT . . . . . . . . . . . . 218
BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
vii
LIST OF ILLUSTRATIONS
Figure Page
1. McCulloch-Pitts Network . . . . . . . . . . . . . . . . . . . . . . 8
2. Standard Hopfield-Tank Network . . . . . . . . . . . . . . . . . . 20
3. Back-propagation Network . . . . . . . . . . . . . . . . . . . . . 25
4. ART 1 Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . 29
5. ART 1 Detailed Diagram . . . . . . . . . . . . . . . . . . . . . . 32
6. Modified Hopfield-Tank Network . . . . . . . . . . . . . . . . . . 48
7. Note Generator Block Diagram . . . . . . . . . . . . . . . . . . . 53
viii
LIST OF TABLES
Table Page
1. System Properties Summary . . . . . . . . . . . . . . . . . . . . . 42
2. Classical Components of Output Sequences . . . . . . . . . . . . . 57
ix
CHAPTER 1
DEFINING THE ROLE AND NATURE OF ARTIFICIAL
NEURAL NETWORK MODELLING
Cultural Bias and Its Application to Cognitive Inquiry
Plato believed that there existed a pure world of ideas, whose
perfect forms were poorly mimicked by copies in the reality apparent to
our senses. This concept of the dominant, lofty nature of ideas and the
thoughts that manipulated those ideas would do more to inhibit inquiry
into material processes than virtually any other single cause in the
succeeding two millennia. The legacy of this philosophical outlook still
permeates our culture, coloring the basic assumptions and viewpoints of
researchers even after extended scientific training.
The firmness of general belief in the separate existence of ideas
contributed to the Lamarckian hypothesis of the inheritance of acquired
characters. While there were critics of Jean Baptiste Lamarck's work from
its introduction, these were by no means an overwhelming group by numbers.
The rediscovery of Gregor Mendel's work in 1905 provided clear and
convincing evidence against Lamarck's hypothesis, yet even Watson and
Crick's elucidation and characterization of DNA failed to dispel the last
vestiges of belief in the inheritance of acquired characters. The ready
and continued acceptance of the Lamarckian hypothesis into the mid-
twentieth century gives an indication of the continued influence of
Plato's world of ideas, even when confronted with contradicting evidence
1
2
and convincing counterarguments.
Biology, however, was by no means the only scientific discipline to
be touched by the cultural bias of Platonic ideals. Physics experienced
great upheaval and internal dissent as the deterministic view of Newton
and Laplace gave way to the quirkiness and Heisenbergian uncertainty of
quantum mechanics (Hawking 1988).
In thinking about thinking, researchers have tended to denigrate
approaches dependent upon investigation of physical processes. The high
regard given ideas and thought processes has nearly always provided
Western observers with a sense of revulsion at even considering that such
sacred items could be solely the product of soft, squishy brain tissues
and their component parts, neurons. This could be considered somewhat
akin to Roquentin's nausea upon consideration of radical contingency
(Ferguson 1987).
Our culture does not encourage modes of inquiry that tend to displace
the traditional view of the mind as the only known organ with which the
perfect world of ideas can be perceived. It especially does not encourage
the denial of the Platonic ideal world, yet it has been quite some time
since any reputable scientist has explicitly advanced support for that
concept.
Cognitive science, however, must deal explicitly and forthrightly
with the subject of ideas and thoughts. In a field where cultural bias
has, can, and will directly effect the discussion of the subject at hand,
it pays to recognize the existence and probable magnitude of that bias. I
have briefly delineated the persistence and magnitude of our Platonic bias
in biology and physics; I will assert that the bias is greater in
3
cognitive science, where it may be more directly challenged.
Artificial Intelligence: Definitions
Artificial intelligence describes at once a sub-discipline of
cognitive science and the goal of that sub-discipline. The goal is the
description and production of an artificial system or systems that can be
described as intelligent in operation, function, or effect, in both global
and local contexts. As with most complex fields, there are several ways
in which to approach direct inquiry into the topic. In what has been
termed the "top-down modelling school" of artificial intelligence, the
emphasis has been upon systems of formal logic and other explicit symbol
manipulation techniques. As may have been apparent from the previous
description, there is a "bottom-up modelling school" of artificial
intelligence. This school of inquiry seeks to examine the basic processes
that are known to produce intelligent action in humans, those basic
processes of neural function and coordination.
By extension, the bottom-up modelling school is concerned with neural
function in general. This is considered necessary because of the great
complexity of biological neural systems. While much research has been
done and continues to be done, there exists no basic understanding of the
detailed structure and operation of biological neural systems. There is a
wealth of data, but a paucity of organizing principles. The bottom-up
modelling school attempts to provide possible organizing principles,
testing these through a process of modelling and incremental design
improvements. This area of research has become known as artificial neural
4
network modelling.
The definitions given above differ somewhat from what may be
considered standard in the artificial intelligence community. In their
Turing Award speech, Newell and Simon state:
The notion of physical symbol system had taken essentially
its present form by the middle of the 1950's, and one can date
from that time the growth of artificial intelligence as a coherent
subfield of computer science. The twenty years of work since then
has seen a continuous accumulation of empirical evidence of two
main varieties. The first addresses itself to the sufficiency of
physical symbol systems for producing intelligence, attempting to
construct and test specific systems that have such a capability.
The second kind of evidence addresses itself to the necessity of
having a physical symbol system wherever intelligence is
exhibited. It starts with Man, the intelligent system best known
to us, and attempts to discover whether his cognitive activity can
be explained as the working of a physical symbol system. . . .
The first is generally called artificial intelligence, the second,
research in cognitive psychology. (Newell and Simon 1976)
This formulation of definitions seems at once too narrow and not properly
descriptive. It is too narrow in that it limits severely the range of
activities which may be considered artificial intelligence research.
While Newell and Simon talk of physical symbol systems, there is the
strong implication that they refer only to computational methods of
explicit symbol manipulation, as in the use of languages such as LISP and
Prolog. The definition of the second part is too narrow, in that it
postulates no overlap between the fields of artificial intelligence and
cognitive psychology, and also provides no linkage for concepts of
operation derived from empirical studies of biological intelligent systems
to be incorporated into an artificial intelligence framework. As stated
later by Newell and Simon,
The symbol system hypothesis implies that the symbolic behavior of
man arises because he has the characteristics of a physical symbol
system. Hence, the results of efforts to model human behavior
5
with symbol systems become an important part of the evidence for
the hypothesis, and research in artificial intelligence goes on in
close collaboration with research in information processing
psychology, as it is usually called.
This seems to indicate that the previously given definitions were not
truly descriptive of the relationship between artificial intelligence and
other components of cognitive science. By including artificial
intelligence as a sub-discipline of cognitive science, it becomes clear
that artificial intelligence is not disjoint from research into
considerations of what constitutes intelligence.
An alternative formulation for defining artificial intelligence in
more general terms is given by Charniak and McDermott (1985), where they
state that artificial intelligence "is the study of mental faculties
through the use of computational models." This form of definition allows
for the top-down and bottom-up approaches to artificial intelligence to be
given co-equal rank as complementary research disciplines.
Artificial Neural Network Modelling
Artificial neural network modelling, the bottom-up school of
artificial intelligence, derives from the use of biological nervous
systems as suitable exemplars for an approach to coordination, cognition,
and control in artificial systems. This approach, in one form or another,
has been with us for a long time.
In the 1940's, McCulloch and Pitts demonstrated the possibility of
casting boolean logic systems into networks of thresholded logic units.
This certainly fits the Newell-Simon definition of a physical symbol
6
system. McCulloch and Pitts' seminal paper and subsequent work introduced
a new concept for consideration: rather than driving the understanding of
biological cognitive processes through increasing sophistication of
technology, perhaps technology can be made more capable and sophisticated
by elucidating mechanisms of function from biological processes involved
in cognition (Levine 1983, 1990).
This differs qualitatively from the viewpoint common to many
observers of human intelligence. There has been a tendency for people to
compare the operation of thought processes with the current leading edge
of technology. Freud compared the mind to a steam engine, Twain would
speak of "mill-works" in relation to speech production, and various
persons in this century have blithely and confidently settled upon the
computer as a fully analogous system. A circular system can be noted
here, though, as the image of the mind as being like some mechanism helped
contribute to such endeavors as Pascal's calculator, Babbage's Analytical
Engine, and the Hollerith punched card controlled census tabulator. These
efforts, in turn, inspired the modern computer. (The influence of the two
World Wars upon the motivation for the development of the computer must
also be acknowledged.) However, as the Metroplex Study Group on
Computational Neuroscience (1988) points out,
Computational study of brain structures began in the 1940's
when digital computers were emerging. The first computer
architectures were developed in a collaboration between the
mathematicians John von Neuman and Norbert Wiener working closely
with physiologists like Warren McCulloch. Early computer
scientists were captivated by the analogy between the all-or-none
action potentials of the neuron and binary switches representing
bits in computers. Models of biological computation thus had a
seminal role in the design of modern computer architecture.
Thus, while comparisons of the workings of the mind to technology helped
7
to motivate research into further technological advances, the insight into
the successful design for a significant new technology, computers, came
instead from paying attention to the actual mechanisms of brain function.
The computer has come to be pressed into service as a "new" metaphor
for thought processes in biological systems. While the use of
technological metaphors for cognition is persistent, having produced
quaint turns of speech and aphorisms, it has not been significantly
conducive to a direct understanding of the actual cognitive processes
being described. The turnabout analogy produced by McCulloch and Pitts
has provided far more benefits to cognitive science and computing through
its establishment of neural modelling. Binary switching technology failed
to provide significant insight into biological neural activity and
function. Leon Harmon (1970) noted that "[a]nother kind of difficulty in
coming to an understanding of nervous systems is that we may be
conditioned into thinking about them in ways that are more constrained
that we like to admit or are sufficiently aware of." The cultural
identity which we share through environment can be a handicap in research
into the nature and mechanisms of our intelligence.
The model networks of McCulloch and Pitts, an example of which is
shown in Figure 1, were seen to provide useful mechanisms and insights for
computing machinery. These networks consist of threshold logic units
receiving excitatory and inhibitory inputs of integer value. A simple sum
of these values is compared with a threshold level for the unit, and the
unit is considered to "fire" if its inputs equal or exceed the threshold
value. A unit's activity is then carried forward to further connected
units in the system. McCulloch and Pitts (1943) proved that any boolean
8
Figure 1, Mc-P
9
function could be created by the appropriate combination of such units.
The McCulloch-Pitts formalism is quite similar to principles of
design using TTL logic, as diagrams can attest. It should be noted that
McCulloch and Pitts described their formalism in 1943, some sixteen years
before the invention of the integrated circuit. While significant
problems reduce the overall utility of McCulloch-Pitts networks, modified
forms continue to be advanced (Szu 1989).
Given the basis of a neural computational framework, additional
features extracted from biological research were gradually added to
proposed models. The idea that inputs to neurons had graded values
modified by the efficiency of synaptic connections led to several
important advances. Hebb proposed a synaptic modification rule to allow a
form of learning in randomly connected networks (Hebb 1949, Levine 1983).
His formulation was that the efficiency of a synapse which connects two
neurons increases if both of the neurons have high activities at the same
time. This relatively simple rule has a number of drawbacks, which have
since been extensively cataloged and discussed. However, it cannot be
denied that Hebb's Law, as it has been called, was an important advance
for modelling.
The idea of stating an organizational principle for network design
also represented an important advance for artificial neural network
modelling. Rosenblatt's Perceptron architectures demonstrate this well.
Rosenblatt postulated several network models, among these was a three-
layer network. The layer which received external inputs was the sensory
layer, and the layer which gave an output (either as a raw signal or
interpreted as a motor output) was termed the response layer. An
10
associative layer of neurons provided the articulation between sensory and
response layers (Levine 1983). This general method of organization helped
move artificial neural network modelling away from specifically designed
networks and randomly organized network models.
The drawbacks associated with the special design of network models
remain a concern for ANN modellers today. Such systems are fragile or
brittle, meaning that a fault in any part of the network could cause a
general failure of function. Also, if an error in design occurred, the
resulting network would have no method of overcoming the design fault.
This particular pitfall is known by the assumption necessary for correct
function to be achieved: "programmer omniscience." Since programmer
omniscience cannot be guaranteed, systems predicated on this principle are
inherently unreliable (Hecht-Nielsen 1986).
Designing with generality as an important principle of function
results in artificial neural networks that are described as self-
organizing. A self-organizing network will conform its function to the
particular problem or context through a process of adaptation or learning.
Current Models Narrowly Focussed
The tendency in developing an artificial neural network model is to
constrain its function to a well-defined domain. This design principle
enables better control over evaluation of the functionality of the design.
Data from neurological or behavioral studies dealing with the problem
domain can then often be directly applied to either training or evaluating
the model.
11
There exist many general functions that are commonly encountered in
biological neural networks which can be considered to have important
implications for computational study. Among these are included
associative memory, classification, pattern recognition, and function
mapping. ANN models proposed to implement these functions include Bi-
directional Associative Memory, Adaptive Resonance Theory
(classification), Brain-State-in-a-Box (classification), Neocognitron
(invariant pattern recognition), and Back-Propagation (function mapping)
(Simpson 1988).
Unfortunately, the underlying reason for existence of each model is
overlooked in comparing different models. The human tendency to wish to
attribute a single scalar quantity denoting how "good" an ANN model is
will often cause people to overlook the fact that certain networks should
not be compared as being equivalent. Lippmann provides a good basic
overview of six different ANN models, yet falls prey to this pitfall.
Each of the models is interpreted as a classification network, yet only
the Hamming and ART networks are designed to function as classifiers
(Lippmann 1987).
Multi-architecture Integrative Systems
The use of multiple complementary architectures in designing
application systems has not often been explored. While this approach to
system design is commonly touted in ANN simulation aid advertisements, it
is less frequently featured in the literature. In an advertisement for
the ANZA Neurocomputer, HNC Corporation states, "In these networks the
12
interconnect geometry is already determined and the form of the transfer
equations is fixed. However, the number of processing elements (neurons),
their initial state and weight values, learning rates, and time constants
are all user selectable, thus allowing one to customize a particular
network paradigm (or combination of paradigms) to suit a particular
application." There are some reasons for the relative lack of multi-
paradigm (multi-architecture) research results in publication.
The design principles stated earlier still hold, that it is easiest
to design an architecture with a narrow definition of function. There is
the tendency for such architectures to incorporate simplifying assumptions
which aid in simulation or real-world implementation ("casting in
silicon," as the phrase goes). The ease of verifying correct operation is
increased for architectures which have a narrowly defined application, as
relatively clean data or simple theoretical findings are likely to be
available against which the architecture may be evaluated.
The relative complexity of modelling for even constrained contexts
provides another reason for concentration upon single paradigm systems.
The number of parameters which can effect a model's performance ranges
from zero for linear systems, such as Widrow's ADALINE (Widrow 1987), to
tens of scalar parameters, as may be encountered in Carpenter and
Grossberg's Adaptive Resonance Theory architecture (Carpenter and
Grossberg 1987a, 1987b; Simpson 1988). When dealing with nonlinear
dynamical systems for which no closed-form solution exists, slight changes
of system parameters can result in large scale changes in behavior.
Carpenter and Grossberg point out relative constraints upon the parameters
used in the ART 1 architecture, giving guidelines for values which should
13
yield stable operation of the network. Finding suitable parameters for
operation of a particular architecture can be a frustrating and time-
intensive experience.
As an example, an architecture called the "on-center, off-surround"
network (OCOS) is based upon a relatively simple equation that appears in
slightly modified form in many competitive networks. One form of this
equation (Grossberg 1973) is:
dx/dt = -A Xi [first term]
+ (B-Xi) (Ii+f(Xi)) [second term]
- (Xi+C) (Sum from k = 1 to n of ((k<>i) (Jk+f(Xk))) [third term]
(Eq. 1)
The first term represents decay of activity over time; the second
term represents increase in activity due to excitatory input values, Ii,
and recurrent self-excitation; the third term represents decrease in
activity due to inhibitory input values, Jk and competition from other
nodes in the network. For each term, there is an associated parameter. A
hidden parameter of the architecture is a factor by which to multiply the
result of the difference equation. This is usually set to be much less
than one, especially when using the simple forward Euler method of system
updating. However, there is a performance trade-off involved with setting
this factor too small: the system will converge to a stable result, but
will take a long time to converge. If the factor is set to a large value
the network will "blow-up," which is a state of wild fluctuation in
activation values for the nodes in the network. Such a system cannot
converge. The solution is to find a value for this factor which will
ensure convergence without an inordinate amount of time taken in reaching
14
convergence. The explicit parameters given in the equation must be
matched to the size of the other parameters to maintain set relationships.
In order to find parameter values, I have used a network simulation which
runs through permutations of sets of discrete parameter values. By
examining the resulting output which indicated whether the network run
achieved convergence and the number of time steps necessary to achieve
convergence, I was able to settle upon a suitable choice of parameters for
regular use and was able to decipher a coarse set of relationships between
the parameters.
In consideration of these complexities in modelling and simulation,
synthesis of complex systems from simpler subunits has some significant
obstacles to overcome. There are more degrees of freedom for system
operation, leading to complexities in articulation and coordination of
subunits. This may increase the necessary number of system parameters,
increasing complexity in a combinatorial fashion. Usually, there will
either be less data available for evaluation of the complete system
behavior, or the data that is available will be uncertain. The general
principles to apply to a synthetic endeavor in artificial neural network
modelling remain to be elucidated.
In addition, synthesis has traditionally been given short shrift in
Western culture (Paris 1989). The application of synthesis in philosophy
has largely been confined to the works of Wittgenstein and Marx. Marx, of
course, developed a philosophy whose application was inimical to most
Western economic structures. This has caused both his work and his
approach to be considered with contempt.
On the positive side, however, synthesis can lead to significant
15
insights as different assumptions and features can be applied to
functional design. The process of integration and hybridization in
artificial neural network modelling may be expected to lead to new
insights into structure and function of benefit to artificially
intelligent systems, given the past history of benefits derived from
consideration of concepts borrowed from disparate disciplines. In this
case, the complexity of knowledge or topic of a contributing discipline
can provide a measure of the possible beneficial overlap: as complexity
increases, the set of concepts which may profitably be applied to the new
context also increases. As this rule would predict, biological nervous
systems provide a very large pool of concepts ready for reconstruction in
the artificial intelligence framework.
Biological Neural System Complexity
The complexity which drives that expectation can be readily
appreciated, as in the human nervous system the number of neurons is in
the billions, and each neuron will have connection to between tens and
tens of thousands of other neurons. This physio-spatial complexity pales
when compared to the complexity that can result when one attempts to
define a state for the neuron. The state can be considered to be a
combination of electrical activity, ionic balance, hormonal levels, input
activities, and many other factors. Since for several of these attributes
there exist separate contributing factors, the number of components
contributing to neuron state can easily exceed twenty, and there may be
hundreds of such components. Add to this that these components often have
16
analog values, separate time scales of action, and capacity for permanent
change in the neuron's response, and the possible state function for a
single neuron displays a suitably bewildering complexity.
Since this level of complexity is not conducive to direct
examination, the process of problem decomposition and solution is applied.
The implication for artificial neural network modelling is that certain
gross features are modelled. Typically, a model assumes that the major
interesting component of neural function is the electrical activity of the
contributing neurons or nodes. A further assumption in general currency
is that the electrical activity of an ANN may be modeled with the elements
or nodes responding in the manner expected of neural populations rather
than individually thresholded neurons (Levine 1983). Some models take
into account generalized neurotransmitter effects, with Grossberg's gated
dipole model providing a good example (Grossberg 1972). These simplifying
assumptions still leave a high level of complexity to be dealt with in
developing ANN designs and applications.
Relevance of Biological Models
Biology provides the exemplars for self-organizing adaptive systems
which have proven useful in artificial neural network modelling thus far.
While strict adherence to biological accuracy in modelling may well be
counter-productive to advances in modelling techniques, rejecting
biological principles may then lead to difficulties in later integrative
work.
The biological framework of neurophysiological function provides a
17
basic structure which makes for a common ground of interaction in models.
For example, the range of activation and output of neurons is rather
strictly proscribed, while the range of activation of an analogous unit in
neural network models is limited only by the capability of the processor
in specifying a floating point exponent. This would make interfacing two
models using different ranges of activation less straightforward.
The inherent complexity of biological systems can support two
different arguments concerning the future of modelling. As Hawking (1988)
says, describing a complex system all at once is terrifically difficult.
It is much simpler to model several different components of the overall
system. Bringing these partial models together to form a complete model
can become problematic, as in Hawking's example of the search for the
Grand Unified Theory in physics. In one sense, then, the effort to
integrate models of limited function may be premature. However, it may be
that along with the incremental advances made in modelling small subunits
of biological neural function, we should also attempt incremental
integration of well-understood low level models. This would help prevent
the kind of situation which exists in physics, with two or more highly
disjoint models prevailing and no unification or integration yet in sight.
Since biological systems are inherently more complex than the physical
systems upon which they are based, it becomes important to keep an eye on
the eventual need for integration and resolution of subsidiary models.
Models which utilize features from two or more extant ANN
architectures such as Hecht-Nielsen's Counterpropagation Network (Hecht-
Nielsen 1987) demonstrate the useful qualities which synthesis of models
can bring to functionality. The counterpropagation network is derived
18
from Kohonen's self-organizing map model and Grossberg's competitive
learning networks. Hecht-Nielsen notes that this network is designed for
function mapping and analyzes its performance as compared to the back-
propagation architecture. While the general functionality of counter-
propagation networks (CPN) remains lower than that of back-propagation
networks, there exists a subclass of mapping functions for which the CPN
will train faster, and there also exists a closed-form solution for the
error of the CPN. As Hecht-Nielsen notes, "Finally, CPN illustrates a key
point about neurocomputing system design. Namely, that many of the
existing network paradigms can be viewed as building block components that
can be assembled into new configurations that offer different information
processing capabilities."
Artificial Neural Network Models: Three Architectures
For the purposes of exploring multi-architecture system design, I
selected three artificial neural network architectures for incorporation
into the overall system. These were the Hopfield-Tank network, back-
propagation, and Adaptive Resonance Theory 1 models. These networks
perform three different functions: Hopfield-Tank is used for optimization
or constraint satisfaction, back-propagation is a general-purpose mapping
network, and Adaptive Resonance Theory 1 is a classifier network.
19
Hopfield-Tank Networks
In 1982, John Hopfield wrote an article, "Neural networks and
physical systems with emergent collective computational abilities," which
was published in Proceedings of the National Academy of Sciences,
describing a model of neural computation which was readily implementable
in current solid-state technology. This article and the model which it
described has been widely credited with a resurgence of interest in ANN
modelling.
The network architecture presented by Hopfield and Tank (1985),
hereafter known as HTN for "Hopfield-Tank Network[s]", is a single-layer
fully interconnected network model (Figure 2). There is no learning rule
for this network, although various researchers have proposed modified HTN
architectures that do incorporate adaptive learning. Weights between
nodes in an HTN are fixed and symmetric, and connections between a node
and itself are zero. The advantages of these criteria for network design
are that the system dynamics can be shown to perform an "energy
minimization" in reaching a stable state. When the weights are determined
according to system constraints, the system can be characterized by a
Liapunov function, providing a measure for system energy.
HTN's have been applied to various constraint satisfaction and
optimization problems. Hopfield and Tank attracted much attention by
demonstrating the utility of an HTN in generating good solutions to the
"Traveling Salesman Problem" (or TSP), a non-polynomial time complete
problem. The TSP can be described as choosing a minimum length path among
20
Figure 2
21
a set of cities such that each city is visited once, and the salesman
returns to the city of origin. The closed path length constitutes the
measure for any particular solution. As the number of cities increases,
the number of possible valid tours increases combinatorially. However, it
can be shown that an HTN will produce good tours in constant time. An HTN
used for TSP computation does not necessarily converge upon the optimum
solution, but will reject non-optimum solutions and give relatively "good"
solutions. Unfortunately, it can also be shown that the HTN trades off
constant time operation for O(n2) space considerations.
Figure 2 shows the HTN architecture for solution of the TSP.
Although the nodes in the network are treated as elements in a vector, the
visual representation which makes the most sense to the designer and other
interested humans is a matrix of nodes. By imposing this structure upon
our view of the network, we can associate each row with a city in the
tour, and each column with the position of a city in the final tour (City
n is in Position m in the tour). The constraints of importance are that
for a valid tour, each city appears once, and each position in the tour
has only one city. Thus, for the state of the network at equilibrium, we
should find high activities in n nodes, where n is equal to the number of
cities or positions. The interconnections between nodes in a row or
column are inhibitory, causing highly active nodes to reduce the activity
of other nodes in the same row or column. The interconnections from a
node to its neighbors in adjacent columns are proportionally more
inhibitory as distance between cities represented by those nodes
increases.
22
Hopfield-Tank Equation
The equation defining the network's activity over time (Hopfield and
Tank 1985) is
Ci (dui/dt) = (Sum from j = 1 to N of (TijVj - ui/Ri + Ii)) (Eq. 2)
where i and j designate neurons in the network,
N is the total number of neurons in the network,
T is a connection weight between neurons,
V is the output value function for a neuron,
u is the activity of a neuron,
I is the external input for a neuron,
C is the capacitance of a neuron,
and R is the resistance of a neuron.
Back-propagation Networks
Back-propagation is a short version of "correction by the backward
propagation of errors." The learning rule used in back-propagation
networks (BPNs) is termed the generalized delta rule (Simpson 1988;
Rumelhart, Hinton, and Williams 1986). Basically, a BPN is a multi-layer
(with at least three layers) network whose nodes use a sigmoid output
function. The BPN will map a set of input activities to another set of
output activities, given training upon a set of example input/expected
output vector pairs. The BPN will generally discriminate and adapt to
23
non-linear relationships in the training data. For example, a BPN can
learn the exclusive-or relation, which a single-layer perceptron cannot.
Nodes in the BPN are often called "units."
The basic premise of the back-propagation algorithm has been
independently derived several times. Werbos (1974) is generally credited
with the first publication of the learning rule, which he called dynamic
feedback; Parker (1985) gave the rule the name, "learning logic;" and the
popularity of the BPN architecture is primarily attributable to Rumelhart,
Hinton, and Williams (1986), as related by Simpson (1988).
All units in the BPN operate in basically the same manner. There are
some slight differences dependent upon whether a unit resides in the
input, hidden, or output layer. Generally, however, a unit generates an
output signal as follows:
oj = f(netj) (Eq. 3)
where j is a unit in the BPN,
net represents the input to a unit,
f is a sigmoid function,
and o is the output of a unit.
netj = (Sum over i of (oiwij)) (Eq. 4)
where i is a unit in a preceding layer of the BPN,
and w is a connection weight linking two units.
24
f(netj) = 1 / (1 + e(-(netj + thetaj))) (Eq. 5)
where theta is the bias weight for the unit.
An input unit has a net input which is simply the provided external input,
as there are no preceding layers.
Figure 3 demonstrates a three layer BP network. At the bottom are
five input units, which receive their activation from an external source.
In the middle are sixteen hidden units, which receive their input
according to Equation 4. The hidden units' output is sent on over another
set of weights to the single output node.
The error between the external training signal and the output node's
output activity provides the basis for correcting the behavior of the
network. This raw measure of error is used to find the delta for the
output node.
deltak = (tk - ok) f'(netk) (Eq. 6)
where k is an output unit,
t is the expected output,
f' is the derivative of the output function.
25
Figure 3
26
Fortunately, the derivative of the sigmoid function is symbolically easy
to specify:
f' = f (1 - f) (Eq. 7)
The network then must distribute this error measure backward through the
network. For each of the hidden units, then, deltas are found as follows:
deltaj = (Sum over k of (deltakwjk)) f'(netj) (Eq. 8)
where k is a unit in the succeeding layer of the BPN.
Each of the weights in the network is changed according to:
(Change in wij) = L * oideltaj (Eq. 9)
where L is a constant representing the learning rate for the BPN.
Theta values for nodes are treated in the same manner, thus
(Change in thetaj) = L * f(thetaj)deltaj (Eq. 10)
So, for the example BPN in Figure 3, an input vector causes output of
the input units to be distributed to the hidden units, modified by the
intervening weights. Similarly, the hidden units send on their output
through weights to the output unit, which provides the response of the
network to the particular input vector. This is called "feed-forward"
27
processing.
Once the network output is known, it can be compared to the expected
output, and the delta value for the output unit is determined. This
begins the "back-propagation" process. The delta values for the hidden
units can now be derived as in Equation 8. The amount of change for each
of the weights between the hidden and output layers can now be found
according to Equation 9. Weight changes between input and hidden layers
proceed similarly. For each node in the hidden and output layers, the
value of theta is also changed. This completes the normal back-
propagation phase.
In the example problem, we have made a slight change to the normal
back-propagation process, and have allowed theta for the input units to
also be adaptively changed.
In practice, one would normally construct a network with the correct
numbers of input and output units, make some guess as to the number of
hidden units needed, and assign random values to the weights. The network
would then be trained upon the available data vector pairs until the error
becomes suitably low, or the implementor decides to make a change in
network design.
Possible applications for BPNs include encrypting, data compression,
non-linear pattern matching and feature detection. Existing BP
applications include translation of text inputs into phoneme outputs,
acoustic signal classification, character recognition, speech analysis,
motor learning, image processing, knowledge representation, combinatorial
optimization, natural language, forecasting and prediction, and multi-
target tracking. BP has been implemented or theorized in electronic,
28
VLSI, and optical formats (Simpson 1988).
Adaptive Resonance Theory 1
Adaptive Resonance Theory 1 (ART 1) is a model introduced by Gail
Carpenter and Stephen Grossberg (Carpenter and Grossberg 1987a). There
are two ART models of note, ART 1 and ART 2, and many modified
architectures which are premised upon one or the other of the ART models.
Basically, an ART architecture is a two-layer network which provides
unsupervised learning of categories of inputs (Figure 4). The F1 layer is
composed of "feature nodes," which accept external inputs. In ART 1,
these inputs are binary patterns, while ART 2 incorporates preprocessing
to accept analog inputs to the F1 layer. The F2 layer is composed of
"category nodes," which compete to respond to valid F1 activations. There
are control structures built into the architecture to prevent F2
activation without input being received at the F1 layer. There are other
control structures to prevent "resonance" from occurring when the
prototype pattern determined by the most active F2 node does not
correspond to the pattern of F1 activation.
Short term and long term memory are represented in ART architectures
by node activations and inter-node weights, respectively. A "bottom-up
activation" refers to the pattern of activation received by F2 nodes
through the weighted links from F1 nodes. Similarly, a "top-down
activation" is the pattern of activation received by F1 nodes through
weighted links from F2 category nodes. Long term memory is changed only
through resonance between the F1 nodes and a selected F2 node. A more
29
Figure 4
30
detailed diagram of the ART 1 interconnections is shown in Figure 5.
Resonance is a state in which long term memory traces between the F1
and F2 layers are modified to more closely represent the input activation
in the category node's top down weights. An F2 node which wins the
competition among F2 nodes has its top-down activation tested against the
bottom-up activation of the F1 feature nodes. If these match within a
level of tolerance, called the vigilance level, a resonant state is
entered and long term memory is changed. If not, the current winning F2
category node is made ineligible for further consideration against the
input, and F2 competition is restarted among remaining eligible F2 nodes.
For an example pattern presented to the ART network given in Figure
4, the presence of input turns on the gain control nodes and activates the
F1 layer. The activation of the F1 layer, or bottom-up activation, is fed
across a set of long term memory weights to generate a new pattern of
activity at the F2 layer. The F2 layer responds with a top down
activation which is filtered by the top down weights linking each of the
F2 nodes to the F1 layer. Competition among the F2 nodes results in a
single winner for application to the current feature input. A comparison
of the bottom up activation with the top down activation yields a set
theoretic measure of the match between the presented pattern and the
category represented by the F2 node. If I represents the input pattern,
V(J) represents the F2 category node J's archetypal pattern, and p
represents the vigilance parameter, then the cardinality of the set
intersection between pattern I and the category pattern V(J) must be
greater than or equal to p times the cardinality of I, or reset occurs.
If reset occurs, the F2 node J becomes ineligible for matching to the
31
current input pattern, and the process of bottom up activation, top down
activation, competition, and match testing continues until some category
node is found to match the input sufficiently well, or until all category
nodes have been matched against the input and found to be too different.
Figure 5 displays a more detailed ART 1 network, with the various weights,
nodes, and connections visible.
Obviously, it is possible that none of the eligible F2 nodes will
match within the vigilance level's acceptable tolerance. When an ART
network is first trained, there are no eligible F2 nodes, rather there are
a number of uncommitted F2 nodes. An F2 node will be selected and enter a
resonant state, providing the first category. If a subsequent input does
not match this category within the vigilance level, the single F2 category
is rendered ineligible and the F2 layer is reset. This brings the network
to a state analogous to that of having no eligible category nodes
available, and a second category node is selected and resonated with the
F1 layer. At any point where no further eligible F2 nodes exist, but an
uncommitted F2 node remains, then a new category node is formed from the
formerly uncommitted node. If no uncommitted category nodes remain, then
the input has been found not to match the available categories.
32
Figure 5
CHAPTER 2
INTEGRATION IN NEURAL NETWORK MODELLING
Integration in neural network modelling is taken here to mean the
combination of different neural network architectures in a coordinated
system. This differs from the casual usage sometimes found in the
literature where the term has been applied either to systems composed of
multiple units of the same base architecture, or else to trivial
modifications of a known architecture. Integration applies properly to
cases of multiple-architecture systems and there are some instances of
systems for which the term genuinely applies but has not been used.
As an example, Matsuoka, Hamada, and Nakatsu (1989) have proposed an
architecture for phoneme recognition that subdivides the hidden and output
layers of a back-propagation network in order to enhance the network's
ability to recognize phonemes and also to substantially reduce the
training time necessary for the network. However, Matsuoka terms this
architecture the Integrated Neural Network (INN). While there is a
substantial improvement in training time, there is no fundamental
difference between an INN and a back-propagation network: they differ only
in the connectivity of the weights between the hidden and output layers.
The reduction in internal complexity of the INN can explain the decreased
training time discovered.
A slightly different form of integration is pursued by Foo and Szu
(1989). A "divide and conquer" approach to problem solving employs the
same architecture, a modified Hopfield-Tank network, to handle smaller
33
34
subproblems and then brings together the resulting subproblem solutions
into an overall solution. This requires some coordination to effect the
overall solution, bringing into play elements of the broader integrative
issues I have noted, but is not properly an integrated system as I have
defined it.
A better example of an integrated artificial neural network system
appears in Cruz et al. (1989). Cruz uses a MADALINE architecture for
image preprocessing and a back-propagation network for removing image
distortion. The MADALINE architecture is a Multiple ADALINE system,
developed by Widrow (Widrow, Pierce, and Angell 1961). The ADALINE, short
for Adaptive Linear Neuron, is a neuronal model using the Least Mean
Square learning rule developed by Widrow and Hoff. Widrow and Winter
(1988) have updated the learning rule used for multiple layer, multiple
ADALINE networks. The specific example given by Widrow and Winter
presents a MADALINE network for invariant pattern recognition. The
MADALINE architecture most closely resembles the Perceptron architecture
given by Rosenblatt. In creating his integrated system, Cruz applied the
"divide and conquer" concept not for the purpose of reducing simulation
time, but in consideration of space requirements needed for a system which
could handle the 256x256 pixel images used.
Integration and Convergence
A perennial problem for the artificial neural network modeller is the
issue of convergence in finite time. It is nice to know that the
architecture selected for a function will converge to a solution before
35
the heat death of the universe. It is similarly a concern that a system
composed of subsidiary architectures will converge. This can be
problematic, since general convergence theorems have not been found for
several of the most popular architectures (Widrow 1987).
Back-propagation provides a particularly pointed example, since it is
by far the most popularly used network, if number of papers concerning
applications is taken as the criterion. The generalized delta learning
rule of back-propagation has long been appreciated to be generally useful,
yet significant progress in firmly establishing this usefulness in the
form of theorems concerning convergence has been lacking. Sontag and
Sussman (1989) provide a theorem demonstrating for back-propagation an
analogous result to the perceptron learning rule: if a separation solution
exists, the generalized delta rule's gradient descent will find the
solution in finite time. While much has been made of a theorem by
Kolmogorov (Farhat 1986), it must be conceded that Kolmogorov's theorem is
an existence theorem: there is some network based upon the back-
propagation architecture which will perform a mapping from n to m, but the
theorem gives no clues as to what that specific network looks like.
There is some promising news concerning convergence which is of
interest for building integrative ANN systems. Hirsch (1989) gives
several theorems which hold that if component subnets of a neural network
converge, then the network will converge. While Hirsch's theorems assume
that the convergence properties of the subnetworks can be described by
Liapunov energy functions, he notes, "It is more difficult to obtain
convergence for cascades of systems that are merely assumed to be
convergent, but without benefit of Liapunov functions or global asymptotic
36
stability. One way of doing this is to place strong restrictions on the
rates of convergence." Hirsch defines a cascade as a layered network
where the output of one layer serves as the input of the next. Many
integrative designs can be cast into this framework.
Incremental Synthesis
The process of synthesis leading to integrative artificial neural
network modelling is important to the development of insights into topics
of critical application, such as sensor fusion. By confronting directly
the need for coherent internal use of available resources and
capabilities, we are more likely to generate an understanding of fusion
principles.
The synthetic approach to modelling provides a supportive environment
for creating extensive systems. Just as the topic of artificial neural
network modelling benefits from the interdisciplinary nature of its
supporting sciences, so a synthetically derived artificial neural network
system benefits from the range of problem solution approaches and features
inherent in the underlying network architectures. By creating and
maintaining a system of network architectures applicable to subfunctions
in the problem solution, subfunction solution by a particular system
component can benefit from the co-option of features normally found in
different components of the artificial neural network system.
37
Networks Under Consideration, System Properties
Each of the three network architectures used in the example problem
of melodic composition has its own set of features and drawbacks which
play an important role in system design.
Hopfield-Tank Networks
As mentioned earlier, the Hopfield-Tank architecture is generally
used in achieving a specific function. By this I mean that each Hopfield-
Tank network is designed for a particular purpose, and can provide no
functionality for other unrelated purposes. So the use of a Hopfield-Tank
network should generally be reserved for functions which do not change
over time. However, I will immediately cite a counter-example, due to its
elegance of integrative design.
An ingenious mechanism for extending the utility of the Hopfield-Tank
architecture is pursued by Tsutsumi (1989), where one back-propagation net
remaps Hopfield-Tank network inputs and another back-propagation network
remaps Hopfield-Tank network outputs. The problem given is one of
avoiding robotic arm deadlocking. The movement space is constrained, and
therefore applicable to Hopfield-Tank network solution. However, the
adaptation of the internal space representation to the real world arm
movement must be adaptive. The Hopfield-Tank network does not provide
learning rules, so the back-propagation networks provide the adaptation to
real world feedback. In this manner, some significant benefits accrue to
the use of the integrated system: since the Hopfield-Tank network is
38
static with respect to the encoded weights, it provides a good repository
for the robot's joint-arm space; since the back-propagation networks are
adaptive, the system can configure itself to respond to a changing
environment.
Since the Hopfield-Tank network was conceived of in the context of
implementation in silicon, the possibility for reducing a Hopfield-Tank
network instance to a hardware component can be important for real-world
applications. This step would bring the benefit often touted for
Hopfield-Tank networks, speed. Speed is rarely noted in practice, since
most practice involves simulation. The simplicity of the Hopfield-Tank
architecture can be a strong point for system design and integration,
however, even in simulated systems.
A drawback which may eliminate the Hopfield-Tank network from
consideration for a particular function is that the weights for the
network must be derived from the constraints to be implemented and from
any data functions necessary for solution but not available in the input
to the network. This requires an understanding of the system to be
solved, which may not be available. Some architecture with learning rule
would then be more suitable for application.
The output of a Hopfield-Tank network must be deciphered from the
final pattern of activation of the net at equilibrium (cf. discussion in
Chapter 1). In the case of the Travelling Salesman Problem, position of
active nodes provides an encoding of placement of cities in the tour.
This information might be rendered more compact in another format,
depending upon the input type expected for further networks in the system.
39
Back-propagation Network System Considerations
The back-propagation architecture provides a mapping from the input
vector to the output vector, and can be trained by example. The system
properties of back-propagation networks include stability of learning
given a fixed universe. By implication, the learning is not stable if
some perturbation in the problem set changes the mapping function. The
back-propagation network would then learn the new mapping over time, and
the old mapping would be lost.
Back-propagation networks have a moderately complex structure. The
properties gained from this increase in complexity over the Hopfield-Tank
architecture include the capacity for learning, the ability to extract
features from input data and generate internal representations for those
features, and the possibility of complex input to output transforms in
accordance with learned associations.
Back-propagation networks can accept binary or analog inputs, so the
inputs can represent conditional probabilities as well as more strictly
constrained values. The outputs are analog values which can be
interpreted as binary through the use of thresholding functions. This
allows a wide variety of input and output possibilities to achieve overall
system function. However, the choice of representation of inputs can be
critical for speedy and reliable training to occur. For example, use of
analog values should be avoided when there exists a natural partition of
the range of the input into distinct states. Our example of the melodic
note generator will illustrate this concern.
40
ART 1 System Properties
Adaptive Resonance Theory 1 architectures provide unsupervised
learning of "clusters" or classifications of input vectors. An internal
representation of classification archetypes is generated. This
architecture ensures that new inputs will tend not to perturb
classifications of previous inputs. This compromise in the stability-
plasticity tradeoff (Carpenter and Grossberg 1987a, b) can be modified for
special purposes, as the melodic note generator program will demonstrate.
The internal operation of the ART 1 network can provide certain features
of especial interest to system design. Specifically, one by-product of
the classification algorithm is the detection of novelty, which will be
shown to have functional significance beyond that of the original design
of the ART 1 network.
Some properties of ART 1 require particular attention from the
designer. For example, there is a matching parameter (vigilance) which
controls how much deviation from a category prototype is acceptable.
There are no guidelines for the selection of the vigilance parameter, and
it is left to the designer to select and assign a "proper" value. Some
guidelines do exist for certain of the other learning parameters in the
ART 1 architecture, such as the learning coefficients for each of the top-
down and bottom-up memory equations (Carpenter and Grossberg 1987a).
The ART 1 architecture provides no standard output. The designer
must access internal values of the ART 1 network to provide useful
information to the remainder of the system which includes that network.
ART 1 is a highly complex architecture with many parameters to be
41
selected and set by the designer. Useful modifications to be made to the
architecture would include creating adaptive functions to replace some of
the static and arbitrary parameters of the network, such as the gain and
reset parameters (cf. Figure 5).
42
Table 1. System Properties Summary
Hopfield-Tank network properties:
- Data initialization process only place for changing adaptively
(not "learning" at all)
- Fast convergence (on system, not necessarily on simulation)
- Inflexible structure (individual design necessary)
- Simple structure
Back-propagation Network Properties:
- Stable learning given fixed universe
- Change to design implies relearning necessary
(costs for self-adaptation include forgetting what has been learned)
- Adaptive weights changed as consequence of training
(supervised learning)
- Medium complexity of structure
- Input to output transform can be computationally complex
- Output nodes may deliver digital, bipolar, or analog values
ART Network Properties:
- Stable self-organizing learning
- Change in design produces unknown effect on existing "knowledge"
- Non-adaptive parameters for gain and reset possible drawbacks, should be
replaced by adaptive functions
System considerations
Considerations of interfacing outputs of one model to inputs of another:
Inputs:
HTN : analog or discrete value, one per node
BPN : analog or discrete value, one per input node
ART 1: discrete value, one per input node
ART 2: analog value, one per input node
Outputs:
HTN : discrete value, one per node
BPN : analog or discrete value, one per output node
ART 1 : none specified
ART 2 : none specified
CHAPTER 3
EXAMPLE PROBLEM
In developing a simulation to test out processes of integration,
several factors have to be considered. The simulation should have enough
scope to provide good subsidiary roles for the component functions, it
should be small enough to be implementable in a reasonable period of time,
and it should produce output of a form which is readily apprehensible and
analyzable. The first criterion is easily fulfilled; the second and third
are rather more difficult to fulfill.
As a starting point for exploring the possibilities of an integrated
approach to artificial neural network modelling, the problem of producing
a melodic line in music composition was selected. The complexity of music
composition in general provides ample considerations for the application
of component networks. Unfortunately, the complexity of musical
composition offers no hope for the relatively simple design and
implementation of an artificial neural network system which addresses all
the salient points. Therefore, simplifying assumptions are made to ease
the requirements for the ANN system. The output, to be interpreted as a
sequence of musical notes, can present some problems with evaluation, due
to the qualitative context of musical evaluation in general. However, it
is possible to treat the note sequence as a set of concatenated symbols,
and apply some of the information theory concepts of Shannon (1948) to
conduct an analysis.
43
44
Simplifying Assumptions
The great complexity of musical composition in general is constrained
to yield a problem of suitable scope for the example integrated ANN
system. A limited scale covering one octave is assumed, in the key of C.
There are no accidentals, and there is no explicit timing of notes. A
single voice is assumed, and there are no harmonics generated. A limited
and fixed set of classical composition rules forms the basis for the
constraint and comparison of the system.
Problem Approach
The use of several ANN architectures in creating an integrated system
whose function fulfills the requirements of the musical composition
problem is assumed. A preliminary set of hypotheses as to how a composer
develops a line of melody was advanced for defining the subfunctions of
the composition system, or note generator. As Teuvo Kohonen (1989) notes
in discussing his own ANN system for musical composition,
It is not possible to survey here the development of ideas in
computer music. One of the traditional approaches, however, may
be mentioned. It is based on Markov processes. Each note (pitch,
duration) is thereby regarded as a stochastic state in a
succession of states. The probability Pr = Pr(Si | Si-1, Si-2,
...) for state Si, on the condition that the previous states are
Si-1, Si-2, ..., is recorded from given examples. Usually three
predecessor states are enough. New music is generated by starting
with a key sequence to which, on the basis of Pr and, say, the
three last notes, the most likely successor state is appended.
The augmented sequence is used as a new key sequence, and so the
process generates melodic successions ad infinitum. Auxiliary
operations or rules are necessary to make typical forms
(structures) of music out of pieces of melodic passages.
45
Leaving the problem of constructing musical forms for possible later
consideration, the approach given by Kohonen matches well the procedure of
composition undertaken here. A candidate note and note sequence are
proposed, a critique according to classical rules is made concerning the
last note in the candidate sequence. The approval of the critic should
mostly cause the acceptance of the candidate note, but the rules should be
broken often enough that there is not an absolute conformity to the rules
postulated.
Now we must confront the problem of how to best combine models in a
unified structure that accomplishes the example function of melodic
composition. The solution involves identifying the strengths and
weaknesses of the component models, and which functions each may
accomplish. Then when the subproblem mapping has been accomplished, it
must be determined how to integrate the subfunction module outputs to
other modules to accomplish the overall task.
Example Problem Network System
The identifiable problem subfunctions are candidate note generation,
sequence critique, and novelty detection. The candidate note generator
subfunction should propose notes in general, but not complete, conformance
to probabilities of the next note filling the requirement of being part of
a classical sequence. The sequence critique subfunction should evaluate
the proposed next note in strict conformance to the set of classical
sequences provided. Since the evaluation given by the sequence critique
subfunction is an incomplete criterion for composition given expectations
46
of novelty, some means of detecting novel sequences must exist for the use
of the coordinating system. The coordinating system then has the
information necessary to "break the rules" when needed to avoid long,
boring sequences of strictly mechanical melody. This can also be seen as
a requirement for stable and continued operation, as there exist sequences
which have no classical next note possibilities.
Candidate Note Generation
The candidate note generator should produce plausible next notes
given a historical partial sequence. Since the rules for identification
of plausible next notes are fixed and known, there is no need for learning
in this stage of the network. The candidate note generator should also
occasionally provide notes which do not necessarily conform to the
expectation of a classical next note.
The Hopfield-Tank network (HTN) was selected as the candidate note
generator since it provides the above features. HTNs are noted for their
utility in constraint satisfaction and other optimization problems, which
fits the requirement that only a single next note is to be proposed at a
given time and that it should basically follow the probabilities of a
classical next note. A well known attribute of the HTN architecture is
the inclusion of spurious local minima which do not represent "valid"
solution states. By purposefully utilizing this feature, we can convert
what normally constitutes a drawback into a asset for the subfunction.
The spurious local minima will give the occasional proposal of non-
classical next notes. The constant and known nature of expected sequences
47
determines the formation of the HTN weights, in conjunction with the known
constraints for proper HTN function. The use of "noisy" input values can
produce the semi-random distribution of possible notes that is needed for
variability.
Figure 6 shows the modified HTN architecture, called Bach, used in
our simulation. The rows represent note values and the columns represent
sequence placement. The constraints imposed on this network include the
need to present a single winning note in each place in the sequence
pattern, and the need to prevent endless repetition of the same candidate
note. By introducing relatively strong inhibitory links within rows and
columns, we can satisfy the constraint requirements. We achieve
preferential selection of classical next notes by reducing those
inhibitory links somewhat for connections which follow classical sequence
patterns.
Sequence Critique
The sequence critic should provide an evaluation of the conformance
of the proposed next note to classical melodic sequence rules. If the
sequence rules are assumed to be provided by example, then some learning
function is required to allow the critic to become adept and reliable.
The subfunction receives a note sequence as input, and produces an output
which may be interpreted as a boolean statement of the classical nature of
the candidate note.
The back-propagation network (BPN) was selected as the sequence
48
Figure 6
49
critic. The BPN, as discussed in Chapter 2, can learn any input/output
transformation given to it (within some constraints upon the availability
of sufficient hidden nodes to form a stable internal representation).
The form of the BPN used, called Salieri, was in the end a network
accepting a binary representation of the input sequence, using twenty
hidden units for internal representation, and producing an output value
which was interpreted as a yes-or-no output. So the net structure used
forty input units, twenty hidden units, and one output unit. This is a
medium sized BPN.
Novelty Detection
The novelty detection function must have components to enable the
recollection of past sequences for discrimination of novel sequences.
However, the space requirements should not be overwhelming. A
classification system would provide good data compression while allowing
the nearly complete context information needed for novelty detection.
The ART 1 architecture was selected for the role of the novelty
detector, for it provides novelty detection as part of a classification
framework. The ART 1 architecture also has other features of interest to
further research on integrated and extensive systems.
The ART 1 network used here, called Beethoven, is modified from the
Carpenter-Grossberg architecture. Some of the explicit features of the
Carpenter-Grossberg architecture are handled as implicit assumptions in
the procedural simulation. The "2/3rds Rule," for example, is not
invoked, since the only time that Beethoven is active, the network meets
50
the constraints of the rule. The separate rules for top-down and bottom-
up weight modifications are replaced by a single rule for both, as is done
in the ART 2 architecture (Carpenter and Grossberg 1987b).
Coordination
In any integrative system, some means of coordinating subfunctions
becomes necessary. Some interpretation and processing of input or output
terms may be accomplished by the coordinating system. The ultimate
decision for whether or not a candidate note is accepted falls to the
coordinating system.
The coordinating system is called Lobes, since the features and
activities of the frontal lobes (Levine 1986, Levine and Prueitt 1989)
provided the inspiration for its operation. Lobes generates the context
management and state dependent actions which drive the integrated system
to completion of the intended function, melodic composition.
Lobes also contains an internal boredom function, which tends to
increase over time. There is a boredom threshold, which causes a change
in behavior in Lobes if it is exceeded.
Operation
In operation, the integrated ANN note generator uses its components
in a sequential manner. Lobes generates a call to Bach for a candidate
note, which when returned is sent to Salieri for critique. As a mechanism
for preventing wastage of Beethoven category nodes, the entire sequence
51
which Bach settles upon is used for further processing by Salieri and
Beethoven. Since at the beginning of composition there is no sequence
history, Bach's inputs are determined randomly over the entire sequence.
As notes are added to the sequence history, Bach inputs are determined
with the addition of some noise, except for the candidate note column
which receives only random noise as input.
Salieri receives the Bach sequence values and makes a determination
of classical conformance, which it passes back to Lobes. Lobes then sends
on the entire context, sequence plus critique, to Beethoven. At first,
Beethoven is virtually certain to encode new categories for each input it
receives. As time goes by, the likelihood that Beethoven will encode a
new category decreases, until all category nodes are utilized. So it is
more likely that the first few notes generated will conform to classical
sequence examples. Sometimes, however, a note which has no possible
classical successors will be generated early in the simulation. In this
case, it is likely that Beethoven's category encoding will proceed at a
much faster than average pace.
The indication of novelty generated by Beethoven is used by Lobes to
modify the system response to the internal state, which is determined by
Salieri's critique of the next note, Beethoven's detection of novelty, and
the boredom threshold in Lobes. If Lobes is not bored and Salieri
approves of the candidate note, the note is accepted. If, however, Lobes
has reached its boredom threshold and Salieri approves of a note, it will
reject the candidate note and request another one from Bach. Likewise, if
Salieri disapproves of the note but Lobes is bored, Salieri may be
overridden and Lobes may accept the note. Indications of novelty from
52
Beethoven can satisfy Lobes' drive to be "excited," or not-bored. This
will tend to make Lobes more conservatively classical as Beethoven
continues to detect novelty. Figure 7 shows the operation of the
coordinated system.
53
Figure 7
CHAPTER 4
RESULTS
Performance
The integrated ANN note generator system produces 152 notes in about
three hours when run on an 80386 PC compatible at 16MHz clock speed. It
takes approximately fifteen hours to produce the same number of notes on
an 8088 based machine with an 8087 numeric coprocessor.
Example and Analysis of Output
Appendix A contains sample output from a run of the note generator.
With a problem such as musical composition, assigning an objective measure
denoting the "worth" of the output is not possible. However, it is
possible to compare the output of our note generator system with random
sequences of note values. By use of a binomial performance measure, it is
possible to define how much the sample output differs from a random
sequence.
Random sequences have their own mystique and interest, but subjective
evaluations of random melodic forms by the untrained ear tend toward the
negative. The output of our note generator network was intended to
basically follow the guidelines of an example set of classical sequences.
The system included mechanisms for breaking out of a strict adherence to
the guide set of sequences. Thus, it would be expected that the output
54
55
have somewhat more resemblance to random sequences than the "rules" would
state. Since the classical rules of composition, while not our sole
criteria of fitness, are the only quantifiable part of our criteria, we
compare our output with random note sequences on the basis of these rules.
Table 2 summarizes the characteristics of output sequences generated
under several different conditions. A random note generator, a mostly
classical note generator, and our integrated ANN note generator each
produced outputs and were evaluated using the critic developed for
training Salieri. The "Successes" column indicates the number of times
the next note in the sequence could be considered to be classical.
The random note generator simply output a sequence of random numbers
for a set sequence length. The Classical Instructor sequence generator
operated by determining the available pool of possible classical next
notes, then randomly selecting one of those notes. In some cases, no next
note fit the criteria of being "classical," and a random note was
generated. The rest of the sequence generators were variations upon our
ANN note generator. The use of different back-propagation nets in the
critic role gave different results in the output. Trained and untrained
back-propagation nets with analog inputs were used. No significant
difference in output could be distinguished between the trained and
untrained versions, but the result was far closer to random performance
than classical. The inability of the Salieri net with analog inputs to
converge to a reliable and accurate performance measure explains the
similarity of result with the untrained version of the same type. On the
other hand, a Salieri composed of a trained back-propagation network using
a binary representation for inputs was able to converge to a stable and
56
fairly accurate state. Hence the performance of the trained Salieri with
binary inputs was considerably closer to classical than was the
performance of the random sequence generator. In two cases, a rule-based
critique system was substituted for the back-propagation network.
57
Table 2. Classical components of output sequences.
Sequence generator Sequence Successes % Z versus Z versus
Length random classical
Random 10,000 890 0.089 0.00 -102.54
System w/
untrained Salieri
(analog inputs) 152 18 0.118 1.26 -23.84
System w/
trained Salieri
(analog inputs) 152 19 0.125 1.54 -23.64
System w/
trained Salieri
(binary inputs) 152 36 0.237 6.28 -20.07
System w/
rule-based critique
(Salieri supervisor)
Run 1 150 35 0.233 6.10 -20.06
Run 2 150 47 0.313 9.42 -17.50
Classical Instructor 8,150 6898 0.846 102.54 0.00
(rule-based critique
w/out ANN system)
CHAPTER 5
DISCUSSION
The integrated note generation system's performance suggests that it
met operational expectations. It produced notes according to a mixture of
somewhat conflicting criteria. This kind of operation in the midst of
uncertainty characterizes many human decision-making processes, and may be
assumed to play a role in human music composition as well. Our framework
allows for further experimentation with hypotheses concerning the
fundamental processes involved in higher-order constraint satisfaction
systems in an extensive environment (cf. Pao 1989).
The integrated approach has demonstrated several advantages and
disadvantages in development and operation. The disadvantages include the
complexity of handling several different network architectures at once,
which can contribute to programmer confusion (the downfall of programmer
omniscience). The necessity for dealing directly with "model mismatches,"
where one subnetwork may produce a different representation as output than
the next subnetwork requires as input, can cause system design time to be
protracted. Failure to recognize that a problem exists in articulation of
networks can result in behavior that diverges wildly from expected norms.
On the other hand, any complex problem may present similar
difficulties regardless of the choice of solution approach. By using
subnetworks of known characteristics, one may be able to achieve a
solution with fewer uncertainties than a totally top-down approach would
yield. With a range of different capabilities available from subsidiary
58
59
networks, the likelihood of encountering an insoluble subproblem is
reduced. Synthetic integrated systems also lead to a combinatorial
explosion in the richness of possible system behaviors, which again is
reminiscent of the increasingly interesting behaviors noted as more
complex biological organisms are considered.
Simulation Concerns
The integrated ANN system from the example problem relied on several
procedural programming shortcuts. For example, the implementation of
"boredom" in Lobes was simply a counter which would be compared with a
threshold value. This does not have any basis in biological neural
systems, yet rather neatly simulates the behavior of a simple neuronal
model for the same task. As another example, the decision in the ART 1
network as to which category an input belongs to is currently made on the
basis of an arbitrary, winner-take-all rule. The same effect could
probably be achieved by means of on-center-off-surround neural
interactions that are more biologically realistic. Casting the system
functions into an entirely biological framework would yield a better, more
capable system for future work. However, the system stands as a first
effort toward this goal.
60
Integration and Artificial Intelligence
There have been several efforts toward integrative system design in
the top-down modelling school. In the HEARSAY system (Reddy et. al.
1973), the blackboard model was developed. The blackboard model
presupposes the combination of a set of possible problem-solving systems
which have available the current system state. The system state is said
to reside upon "the blackboard" as a visualization of the process of
problem-solving. Each of the several applicable subsidiary problem-
solving systems attempts to derive an incremental step toward the global
solution, and competes with other such problem-solvers to control the
blackboard, and thus be able to change the system state. This is a
significant development, and one which is paralleled by concepts in
several artificial neural network models. In ART models, for example, the
concept of competition among various classification prototypes bears a
strong resemblance to the blackboard model. If the F1 layer's activation
is considered analogous to the system state, the F2 nodes each are
analogous to problem-solvers in the blackboard model.
There have been some efforts toward explicitly bringing together top-
down and bottom-up models in hybrid systems. Amano et al. (1989) present
an example of a phoneme recognition system combining an expert system with
a perceptron network. The expert system provides feature extraction from
speech data, which is input to the perceptron. The perceptron allows
decision-making under uncertainty, whose output is interpreted using fuzzy
logic rules. This avoids drawbacks associated with "template matching"
phoneme recognition schemes. Rabelo and Alptekin (1989) have integrated a
61
neural network with an expert system into an intelligent scheduling system
for manufacturing applications. Their system has the ability to learn
from experience and generate schedules within real-time constraints.
Neurobiological Evidence of Multifunctionality
Multi-state functionality of memory is supported by work of Nottebohm
(1989). Nottebohm's work involves the development and remembrance of
complex songs in songbirds. Through a series of studies, Nottebohm
demonstrates that hormonal changes can cause the forgetting of songs in a
bird's repertoire, or allow the formation and remembrance of new songs.
The hormonal changes in question center around testosterone, and typically
the mating season is the time when levels of testosterone allow the
formation of songs, presumably conferring a reproductive advantage for
male songbirds. Nottebohm demonstrates that the changes are only
hormonally dependent by the simple strategy of artificially inducing song
creation by the application of testosterone to songbirds of both sexes at
various times of the year. The withholding of regular doses of
testosterone was also shown to cause the forgetting of known songs in the
same birds.
The implications of Nottebohm's work include support for state-
dependent memory. Since a specific memory function can be modulated by a
specific hormone, this implies that other memory systems may also have
recall dependent upon some hormone or other chemically mediated process.
Given that recall and learning can be so modulated, the necessity for
taking state dependencies (levels of hormones and other neuroactive
62
substances) into account for system function can be appreciated.
State-dependent memory is also supported by the work of many other
investigators, such as Bower (1981). In Bower's experiments, happy or sad
moods were induced in subjects by hypnotic suggestion, in order to
investigate the influence of emotions on memory and thinking. This
influence was profound; for example, people recalled a greater percentage
of those experiences which had taken place when they were in the same mood
as they were in during recall. Also, when the feeling tone of a story
agreed with a reader's hypnotically generated emotion, the reader found
the events and characters in the story more memorable and easier to
identify with.
State dependence of memory or other neural function can give rise to
quite useful modelling constructs. The ability to recast problem
solutions given functional states becomes biologically justifiable. The
logical power of conditional activation of entire subnetworks becomes
available through the modelling, however coarse, of these state
dependencies. The almost direct implementation of expert system analogues
which can be analyzed in a completely biological and ANN context is made
possible.
Implications include the higher order integration of functionally
changed sub-units over time. In humans, well known state dependencies
include the fight or flight response to norepinephrine production and the
diving response typical of mammals entering cold water. Without the
appropriate integrative control, neither fight or flight nor the diving
response would produce the desired, or selected, effect. The coordination
of separate functional neural "circuits" is clearly present; the exact
63
mechanism remains to be elucidated but there have been some promising
beginnings in neural network models. For example, in the neural model of
attention described by Grossberg (1975), there is competition between
nodes representing activations of different drives (hunger, thirst, sex,
etc.). The winner of this competition is not determined solely by which
drive is highest, but also by the availability of compatible cues in the
environment.
State-dependent memory implies the existence of functional changes
over time in cortical structures. Since we now have evidence of multi-
modal neural circuitry, at least some consideration should be given to the
implied necessary integration. The problem of understanding a system
which is dynamic not only in processing of input but also in functional
neural subsystems is both daunting and exhilarating. It is daunting,
because the complexities of modelling such systems exceeds our current
capacity for ready assimilation and understanding of the underlying
concepts and mechanisms (which have not yet been elucidated), and
exhilarating because there appears to be no end to the variety of
expression of these systems in the natural world, and thus no apparent end
to the problem-solving challenge awaiting the researcher.
The function of speech processing in humans, for example, requires
the acquisition of external signals, the separation of those signals into
semantic and affective content, the recognition of mode in affective
content, the parsing of semantic content, and the integration of semantic
and affective content to determine meaning. This list of subfunctions is
not complete, which gives an indication of the extent to which integration
remains a regular and important activity in biological systems.
64
The Triune Brain Theory
Integrative theories of neural/cognitive function have a long
history. One of the best known is the triune brain theory of Paul MacLean
(1970). MacLean's research into behavioral studies of different brain
areas led him to propose that the human brain is divided into three
developmentally derived regions of separate function (see Levine 1990 for
discussion). The earliest, and presumably the most primitive region, is
termed the reptilian brain, and is composed of the the brainstem and basal
ganglia. The reptilian brain is responsible, in this theory, for the
preprogrammed, innate behaviors. The paleomammalian brain, composed of
the limbic system, modifies the expressed pattern of reptilian brain
responses and is the source, in this theory, of the basic emotions (love,
hate, fear, arousal, etc.). The neomammalian brain, composed of the newer
parts of the cerebral cortex, provides further modifications of the
expression of the two older brain areas, and gives us our rational
capacity, seen in the ability for planning and verbal expression.
While MacLean's theory is oversimplified, it does provide a useful
set of distinctions between various cognitive subfunctions, all of which
are involved in complex behaviors. In fact, if one stretches the
imagination, one can draw analogies between the reptilian brain and our
Hopfield-Tank network; the paleomammalian brain and our ART 1 network, and
the neomammalian brain and our back-propagation network.
The integrated ANN note generator had its origins in a collaborative
effort to develop an extensive ANN system suitable for exploring multi-
65
modal cognitive hypotheses (Blackwood, Elsberry, and Leven 1988). That
project, in turn, was derived from insights provided by Leven (1987b).
Leven's SAM model was depicted in a manner which led to a discussion of
the possibility of replacing the components of MacLean's triune brain
model with current ANN architectures. The difficulty of describing a
suitably restricted problem for adequate application of the limited
current architectures was resolved with the simple melodic composition
problem outlined previously. Points of difference from the original
MacLean theory can be attributed to certain changes in model context (as
modified by Leven's separation of memory into three components:
motoric/instinctive [reptilian], sensory/affective [paleomammalian], and
associative/semantic [neomammalian] (Leven 1987a)) and to the mismatch
between architectures derived not for their similarity to these basic
cognitive forms, but to satisfy more immediate criteria such as being
implementable in current electronic devices. The desired system based on
our loose analogy to MacLean's theory has been demonstrated to be
operational and ready for incremental refinement.
We hope that in future work such analogies can be made more precise.
The development of both neural network theory and neuroscientific data
should allow the critical research to continue into these theories of
integrative cognitive function.
APPENDIX 1
SAMPLE MELODY OUTPUT OF THE VARIOUS NOTE GENERATOR PROGRAMS
66
67
Integrated ANN Note Generator Sample Output
68
b61T output, page 1
69
b61t output, page 2
70
b61t output, page 3
71
Random Note Generator Sample Output
72
random output, page 1
73
random output, page 2
74
random output, page 3
75
Classical Note Generator Sample Output
76
classical output, page 1
77
classical output, page 2
78
classical output, page 3
APPENDIX 2
PROGRAM SOURCE LISTING: INTEGRATED ANN NOTE GENERATOR
79
80
PROGRAM Note_Generator (INPUT,OUTPUT);
{
This program demonstrates a small application which uses three
major neural network models to advantage in a cooperative,
integrated manner. The problem is that of generating musical
notes in a well-formed, but not repetitious manner. The scale of
notes is limited, and only one note is generated per time interval.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
DOS, {Turbo Pascal MS-DOS functions}
CRT, {Turbo Pascal IBM-PC screen and keyboard functions}
ANN, {Artificial Neural Network functions}
Struct, {Linked-list structure functions}
Misc1, {Miscellaneous functions}
ANSI_Z, {ANSI screen control functions}
BP_unit, {Back-propagation network functions}
Globals, {Global types and constants}
ClasInst; {Classical Instructor function}
VAR
inf : TEXT; {Input file handle}
outf : TEXT; {Output file handle}
main_ii, {Loop variable}
main_jj : INTEGER; {Loop variable}
main_done : BOOLEAN; {Loop variable}
main_time : Time_rec_; {Record for time info}
Time1, Time2 : Time_rec_;
Testr : REAL;
Inchar : char;
note_rec : note_record_;
ii : INTEGER;
snet : BP_net_;
81
{For Beethoven (ART 1)}
F1 : F1_layer_ptr_;
F2 : F2_layer_ptr_;
{End VAR declarations}
{---------------------------------------------------------}
PROCEDURE Wait;
{}
BEGIN {}
{ Writeln('Press a key...');}
{ READ(inchar);}
END; {}
{----------------------------------------------------------}
PROCEDURE report_notes (VAR cmn : Common_area_);
TYPE
outstr_ = STRING[10];
VAR
ii : INTEGER;
raoutf : FILE OF CHAR;
raoutfname : STRING;
och : CHAR;
BEGIN
{open note file}
{skip to end}
{write note}
{close}
raoutfname := FSEARCH('BEETHOVN.MUS',GETENV('PATH'));
IF raoutfname <> '' THEN BEGIN
Assign(raoutf,raoutfname);
Reset(raoutf);
END
ELSE BEGIN
Assign(raoutf,'BEETHOVN.MUS');
Rewrite(raoutf);
END;
Seek (raoutf, FileSize(raoutf));
{go to end of file}
IF note_rec.c = 2 THEN BEGIN
och := ascii_cr;
Write(raoutf,och);
och := ascii_lf;
82
Write(raoutf,och);
och := ascii_asterisk;
Write(raoutf,och);
och := ascii_equal;
Write(raoutf,och);
och := ascii_asterisk;
Write(raoutf,och);
och := ascii_cr;
Write(raoutf,och);
och := ascii_lf;
Write(raoutf,och);
END;
och := Chr(note_rec.n[note_rec.c-1]+48);
Write(raoutf,och);
och := ascii_cr;
Write(raoutf,och);
och := ascii_lf;
Write(raoutf,och);
IF note_rec.c = 153 THEN BEGIN
och := ascii_cr;
Write(raoutf,och);
och := ascii_lf;
Write(raoutf,och);
och := ascii_asterisk;
Write(raoutf,och);
och := ascii_equal;
Write(raoutf,och);
och := ascii_asterisk;
Write(raoutf,och);
och := ascii_cr;
Write(raoutf,och);
och := ascii_lf;
Write(raoutf,och);
END;
Close(raoutf);
ANSI_CUP(2,0);
Write('Notes generated: ');
ANSI_CUP(2,25);
Write((note_rec.c-1):3);
END;
PROCEDURE record_a_note (VAR cmn : Common_area_);
CONST
Init : BOOLEAN = FALSE;
c : INTEGER = 1;
TYPE
outstr_ = STRING[10];
83
VAR
ii : INTEGER;
raoutf : FILE OF CHAR;
raoutfname : STRING;
och : CHAR;
BEGIN
IF NOT init THEN BEGIN
FillChar(note_rec.n,SizeOf(note_rec.n),#0);
note_rec.c := 1;
Init := TRUE;
END;
FOR ii := 1 TO V_len_out-1 DO BEGIN
{}
cmn.notes[ii] := cmn.notes[ii+1];
END; {}
cmn.notes[V_len_out] := 0;
note_rec.n[note_rec.c] := cmn.notes[v_len_out-1];
INC(note_rec.c);
report_notes(cmn);
END;
PROCEDURE play_a_note(VAR cn : BYTE);
BEGIN
CASE cn OF
1 : Sound(n_c_mid);
2 : Sound(n_d);
3 : Sound(n_e);
4 : Sound(n_f);
5 : Sound(n_g);
6 : Sound(n_a);
7 : Sound(n_b);
8 : Sound(n_c_hi);
ELSE
NoSound;
END;
Delay(180);
NoSound;
Delay(55);
END;
PROCEDURE play_notes (nr : note_record_);
VAR
ii : INTEGER;
BEGIN
ii := 1;
84
FOR ii := 1 TO nr.c DO play_a_note ( nr.n[ii]);
END;
PROCEDURE {Change_global_factors}
user_keys;
CONST
Initialized : BOOLEAN = FALSE;
VAR
inch : CHAR;
instr : STRING;
tempr : REAL;
err : INTEGER;
PROCEDURE display_global_factors;
BEGIN
ANSI_CUP(13,12);
Write('*res: ',HTN_co_res:5:4);
ANSI_CUP(14,12);
Write('*cap: ',HTN_co_cap:5:4);
ANSI_CUP(15,12);
Write(' *wt: ',HTN_co_wt:5:4);
ANSI_CUP(16,12);
Write('*inp: ',HTN_co_inp:5:4);
ANSI_CUP(17,12);
Write('epsi: ',epsilon:5:4);
ANSI_CUP(18,12);
Write('iter: ',HTN_co_iter:5:4);
ANSI_CUP(9,55);
Write('*Vigilance: ',ART_co_vigilance:5:4);
ANSI_CUP(23,0);
END;
BEGIN
IF NOT Initialized THEN BEGIN
ANSI_CUP(23,0);
Write
('Type "C" to change factors, "P" to play notes so far.');
display_global_factors;
ANSI_CUP(23,0);
Initialized := TRUE;
END; {IF NOT Initialized}
IF check_kbd_status THEN BEGIN
IF dir_console_IO(inch) THEN BEGIN
inch := UpCase(inch);
85
IF inch = 'C' THEN BEGIN
ANSI_CUP(21,0);
Write(
'Change: 1)epsi 2)*res 3)*cap 4)*wt 5)*inp 6)iter 7)*vigilance'
);
REPEAT
WHILE (NOT dir_console_IO(inch)) DO ;
UNTIL (inch IN ['1','2','3','4','5','6','7']);
REPEAT
ANSI_CUP(21,0);
ANSI_EEOL;
ANSI_CUP(21,0);
Write('Input value: ');
Readln(instr);
Val(instr,tempr,ii);
UNTIL (ii = 0);
CASE inch OF
'1' : epsilon := tempr;
'2' : HTN_co_res := tempr;
'3' : HTN_co_cap := tempr;
'4' : HTN_co_wt := tempr;
'5' : HTN_co_inp := tempr;
'6' : HTN_co_iter := tempr;
'7' : ART_co_vigilance := tempr;
END;
ANSI_CUP(22,0);
ANSI_EEOL;
display_global_factors;
ANSI_CUP(23,0);
END
ELSE IF inch = 'P' THEN BEGIN
play_notes(note_rec);
END; {Else if inch}
END;
END;
END;
{----------------------------------------------------------}
PROCEDURE Bach(VAR cmn : Common_Area_);
{Generates a new note from past sequence and frequency information.
Uses a Hopfield-Tank network to accomplish this task. }
{INPUTS:
Sequence of notes, 4 notes long
OUTPUT:
Single note, valued from 1 to 8
}
{Hopfield-Tank network. Given data and input values,
processes for output.}
86
PROCEDURE HTN(VAR cma : Common_Area_);
CONST
Initialized : BOOLEAN = FALSE;
TYPE
W_A_ptr_ = ^Weight_Array_;
weight_array_ = ARRAY[1..64,1..64] OF REAL;
file_string_ = STRING[127];
neuron_ = RECORD
a : REAL; {activation value}
r : REAL; {resistance}
c : REAL; {capacitance}
o : REAL; {output}
i : REAL; {input}
END;
neuron_array_ = ARRAY[1..v_len_in,1..v_len_out] OF neuron_;
note_array_ = ARRAY[1..5] OF INTEGER;
CONST
WA : W_A_ptr_ = NIL;
VAR
inf : FILE OF weight_array_;
Time_step : INTEGER;
ns : ARRAY[0..1] OF neuron_array_;
ii, jj, kk : INTEGER;
nbase, nindex : INTEGER;
FUNCTION Neuron_Output(act,cap :REAL):REAL;
BEGIN
neuron_output := 0.5 *(1 + tanh(act/cap));
END;
FUNCTION max_cell_in_column(col : INTEGER):INTEGER;
VAR
ii, jj : INTEGER;
Hi : REAL;
nsptr : INTEGER;
BEGIN {max_cell_in_column}
Hi := 0.0;
nsptr := time_step MOD 2;
FOR ii := 1 TO v_len_in DO BEGIN
IF (ns[nsptr,ii,col].o > Hi) THEN BEGIN
Hi := ns[nsptr,ii,col].o;
jj := ii;
END;
END;
87
max_cell_in_column := jj;
END; {max_cell_in_column}
FUNCTION done(epsi : REAL):BOOLEAN;
VAR
finish : BOOLEAN;
ii, jj : INTEGER;
t : REAL;
BEGIN
finish := FALSE;
ii := 1;
WHILE (NOT finish) AND (ii <= V_len_in) DO BEGIN
FOR jj := 1 TO 5 DO BEGIN
IF ABS(ns[0,ii,jj].o - ns[1,ii,jj].o) > epsi THEN
finish := TRUE;
END;
ii := ii + 1;
END;
done := NOT finish;
END;
FUNCTION Convert_to_weight_coord(note,posit : INTEGER):INTEGER;
BEGIN {Convert_to_weight_coord}
Convert_to_weight_coord := (v_len_in*(posit-1)+note);
END; {Convert_to_weight_coord}
FUNCTION delta_neuron_activation(a,r,i : REAL;
note,posit : INTEGER):REAL;
VAR
ii, jj : INTEGER;
tempr1, tempr2 : REAL;
sum : REAL;
di1,di2 : INTEGER;
current : INTEGER;
BEGIN {delta_neuron_activation}
current := time_step MOD 2;
sum := 0.0;
di1 := Convert_to_weight_coord(note,posit);
FOR ii := 1 TO v_len_in DO
FOR jj := 1 TO v_len_out DO BEGIN
di2 := Convert_to_weight_coord(ii,jj);
sum := sum
+ (WA^[di1,di2] * HTN_co_wt)
* ns[current,ii,jj].o;
END;
88
delta_neuron_activation :=
(-(ns[current,note,posit].a
/ (ns[current,note,posit].r * HTN_co_res))
+ (ns[current,note,posit].i * HTN_co_inp) + sum)
/ (ns[current,note,posit].c * HTN_co_cap);
END; {delta_neuron_activation}
FUNCTION iterate_htn(VAR nts : notes_):INTEGER;
VAR
II, JJ, next_time : INTEGER;
PROCEDURE display_neuron_activation;
CONST
column = 0;
row = 10;
VAR
ii, jj : INTEGER;
ts : INTEGER;
active : INTEGER;
BEGIN {display_neuron_activation}
ts := time_step;
FOR ii := 1 TO v_len_in DO BEGIN
{FOR ii}
ANSI_CUP(row+ii,0);
FOR jj := 1 TO v_len_out DO BEGIN
{FOR jj}
active := Round(ns[ts,ii,jj].o * 10);
IF active > 10 THEN active := 10;
IF active < 1 THEN active := 1;
Write(Copy(graphic_string,active,1),' ');
END; {FOR jj}
END; {FOR ii}
ANSI_CUP(23,0);
END; {display_neuron_activation}
PROCEDURE update_neuron_output;
VAR
ii, jj : INTEGER;
BEGIN
FOR ii := 1 TO v_len_in DO BEGIN
{FOR ii}
FOR jj := 1 TO v_len_out DO BEGIN
{FOR jj}
ns[time_step,ii,jj].o :=
89
neuron_output(ns[time_step,ii,jj].a,
ns[time_step,ii,jj].c);
END; {FOR jj}
END; {FOR ii}
END;
BEGIN {iterate_HTn}
time_step := 0;
{initialize the neuron arrays}
FOR ii := 1 TO 8 DO
FOR jj := 1 TO 5 DO BEGIN
ns[0,ii,jj].a := 0.5;
ns[0,ii,jj].o := 0.0;
ns[1,ii,jj].o := 0.0;
IF (nts[jj] = 0) THEN BEGIN
{randomize input}
ns[0,ii,jj].i := gaussian_noise(0.5,0.25);
ns[1,ii,jj].i := ns[0,ii,jj].i;
END
ELSE BEGIN
IF (nts[jj] = ii) THEN BEGIN
ns[0,ii,jj].i := 0.67
+ gaussian_noise(0.0,0.1);
ns[1,ii,jj].i := ns[0,ii,jj].i;
END
ELSE BEGIN
ns[0,ii,jj].i := 0.33
+ gaussian_noise(0.0,0.1);
ns[1,ii,jj].i := ns[0,ii,jj].i;
END;
END;
END;
{prevent premature end}
ns[1,1,1].o := 20;
update_neuron_output;
WHILE (NOT done(epsilon)) DO BEGIN
user_keys;
time_step := time_step MOD 2;
next_time := (time_step + 1) MOD 2;
{determine output term for neurons}
update_neuron_output;
FOR ii := 1 TO v_len_in DO BEGIN
{FOR ii}
FOR jj := 1 TO v_len_out DO BEGIN
{FOR jj}
ns[next_time,ii,jj].a := ns[time_step,ii,jj].a
+ HTN_co_iter
* delta_neuron_activation(ns[time_step,
90
ii,jj].a, ns[time_step,ii,jj].r,
ns[time_step,ii,jj].i,ii,jj);
END; {FOR jj}
END; {FOR ii}
display_neuron_activation;
time_step := time_step + 1;
END; {WHILE}
{**Change of note: Finding the highest activity in each column and
setting it for further processing will change the behavior of the
net and program. The change is being made to improve performance of
the ART1 network.**}
FOR ii := 1 TO v_len_out DO BEGIN
nts[ii] := max_cell_in_column(ii);
END;
iterate_htn := nts[v_len_out];
END; {Iterate_HTn}
BEGIN {HTn}
{ Initialized := False;}
IF (NOT Initialized) THEN{do initialize}
BEGIN {get weight matrix}
New(WA);
Assign(inf,'htn.dat');
Reset(inf);
Read(inf,WA^);
Close(inf);
Initialized := TRUE;
ANSI_CUP(8,0);
Write('HTN:');
ANSI_CUP(9,0);
Write('Node activity');
ANSI_CUP(11,12);
Write('R: ',global_resistance:5:3);
ANSI_CUP(12,12);
Write('C: ',global_capacitance:5:3);
ANSI_CUP(23,0);
END;
FOR ii := 1 TO 8 DO
FOR jj := 1 TO 5 DO BEGIN
ns[0,ii,jj].r := global_resistance;
ns[0,ii,jj].c := global_capacitance;
ns[1,ii,jj].r := global_resistance;
ns[1,ii,jj].c := global_capacitance;
END;
FOR ii := 1 TO v_len_out DO
{clear notes}
cma.notes[ii] := 0;
91
nbase := note_rec.c - v_len_out;
FOR nindex := 1 TO (v_len_out - 1) DO BEGIN
IF ((nbase + nindex) > 0) THEN
cma.notes[nindex] := note_rec.n[(nbase+nindex)];
END; {For}
cma.Candidate_note := iterate_htn(cma.notes);
cma.notes[v_len_out] := cma.candidate_note;
END; {HTn}
BEGIN {Bach}
Dump_Common(cmn);
HTn(cmn);
END; {Bach}
{----------------------------------------------------------}
PROCEDURE Salieri(VAR cmn : Common_Area_);
{Compares past information and proposed note generated by Bach with
rules of classical composition. A PDP network is used to do this.}
PROCEDURE Back_propagation(VAR cmn : Common_Area_);
{A PDP style back propagation network.}
CONST
Initialized : BOOLEAN = FALSE;
count : WORD = 1;
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, kk : INTEGER;
Done : BOOLEAN;
tp1 : DVE_ptr_;
error_m, tne, sum : ARRAY[1..3] OF REAL;
ss : STRING;
binsum : ARRAY[1..3] OF INTEGER;
FUNCTION max (r1, r2 :REAL):REAL;
BEGIN
IF r1 >= r2 THEN max := r1
ELSE max := r2;
END;
PROCEDURE Set_input_vector_from_notes (vp : DVE_ptr_;
92
n : notes_);
VAR
ii : INTEGER;
vpt : DVE_ptr_;
vn : ARRAY[1..40] OF INTEGER;
BEGIN
FillChar (vn,SizeOf(vn),#0);
{Blank the current vector}
FOR ii := 1 TO 5 DO BEGIN
{Notes subscript}
IF n[ii] > 0 THEN vn [((ii-1)*8)+n[ii]] := 1;
END; {For notes subscript}
vpt := vp;
FOR ii := 1 TO snet.n_input DO BEGIN
vnp_(vpt^.dptr)^.v := vn[ii];
vpt := vpt^.right;
END; {FOR ii}
END;
BEGIN
IF NOT Initialized THEN BEGIN
snet.data_fname := 's61.dat';
ANSI_CUP(15,40);
Setup_BP_net (snet,snet.data_fname);
Write(snet.data_fname);
ANSI_CUP(16,40);
Set_BP_net_weights_from_file(snet,snet.wt_fname);
Write(snet.wt_fname);
Initialized := TRUE;
END;
{now set up current input vector}
Set_input_vector_from_notes(snet.vi,cmn.notes);
{get the supervisor's critique for the current sequence}
vnp_(snet.vts^.dptr)^.v := Classical_instructor (cmn.notes);
{Feed-forward, back-propagate, and make changes}
BP_train_and_change (snet);
{determine response of the net to the current sequence}
cmn.Is_classical := (vnp_(snet.vos^.dptr)^.v > 0.50);
93
{keep weight changes that have been made now and then}
IF (count MOD 49) = 0 THEN
Dump_BP_net_weights(snet,snet.wt_fname);
INC(count);
END;
BEGIN {Salieri}
Dump_common(cmn);
Back_propagation(cmn);
END; {Salieri}
{----------------------------------------------------------}
PROCEDURE Beethoven(VAR cmn : Common_Area_);
{}
PROCEDURE ART1(VAR cma : Common_Area_);
{Binary Adaptive Resonance Theory model}
{
Carpenter and Grossberg's ART1 paradigm
Copyright 1988, W.R. Elsberry & D.J. Blackwood
Written in Turbo Pascal 5.5, September 1988
}
{
Terminology:
STM = Short Term Memory
LTM = Long Term Memory
TD = Top-down
BU = Bottom-up
F1 layer = a vector of nodes which receive input from three sources:
External input, a binary vector
Gain control, an internal processing input
F2 layer, activation of category nodes through LTM traces
F2 layer = a set of nodes which have LTM memory traces associated
with them that relate to the BU activations to particular nodes in
the F2 level.
Inputs: F1 activation through BU LTM traces, Gain Control,
System Reset
}
{
Parameter constraints from Carpenter & Grossberg, 1987
94
A >= 0
mu1, mu2 >= 0 (Simpson)
C >= 0
epsilon1, epsilon2 >= 0 (Simpson)
MAX(1,D) < B < 1 + D
MAX(1,gamma1) < sigma1 < (1 + gamma1) (Simpson)
0 < e << 1
K = O(1)
L > 1
0 < p <= 1
0 < Vigilance <= 1 (Simpson)
0 < Z_IJ(0) < (L / (L-1+M))
0 < Wup(0) < (L / (L - 1 + Max_F1_nodes)) (Simpson)
1 >= Z_JI(0) > Z_BAR == ((B-1)/D)
1 >= Wdn(0) > ? == ((sigma2-1)/gamma2) (Simpson)
0 <= I,f,g,h <= 1
}
CONST
Initialized : BOOLEAN = FALSE;
Initial_Wup = 0.1;
Initial_Wdn = 0.9;
Number_committed_F2 : INTEGER = 0;
Vigilance : REAL = 0.9; {Determines what level of mismatch will
cause reset}
Time_slice : REAL = 0.1; {Factor to multiply deltas by}
{The following are part of the F1 STM recall equation}
mu1 : REAL = 1; {Positive constant controlling BU input & TD
feedback}
sigma1 : REAL = 1.4; {Positive constant regulating gain control}
epsilon1 : REAL = 1; {Positive constant regulating gain control}
gamma1 : REAL = 1; {Positive constant regulating TD and BU
feedback}
{The following are part of the F2 STM recall equation}
mu2 : REAL = 1; {Positive constant controlling BU input & TD
feedback}
sigma2 : REAL = 1.4; {Positive constant regulating gain control}
epsilon2 : REAL = 1; {Positive constant regulating gain control}
95
gamma2 : REAL = 1; {Positive constant regulating BU input}
{The following are part of the Bottom-Up LTM equation}
alpha1 : REAL = 1; {Positive constant for learning rate}
beta1 : REAL = 1; {Positive constant for passive decay}
{The following are part of the Top-Down LTM equation}
alpha2 : REAL = 1; {Positive constant for learning rate}
beta2 : REAL = 1; {Positive constant for passive decay}
VAR
i, j : INTEGER;
F2_winner : INTEGER; {Index of winning F2 node}
Input_on : BOOLEAN; {Is input currently being received?}
Resonance : BOOLEAN;
{The following are part of the match operation equation}
Vector_I : Vector_; {Binary input vector}
Vector_X : Vector_; {Binary expected vector}
PROCEDURE Build_Expected_Vector;
{}
CONST
low = 0.0;
high = 1.0;
{thresh = 0.5;}
VAR
ii, jj : INTEGER;
thresh : REAL;
BEGIN {Build_Expected_Vector}
thresh := 0.0;
WITH F2^[F2_winner] DO BEGIN
{with}
FOR ii := 1 TO Max_F1_nodes DO BEGIN
{}
thresh := thresh + Wdn[ii];
END; {}
thresh := thresh / Vector_length;
FOR ii := 1 TO Max_F1_nodes DO BEGIN
Vector_X[ii] := BYTE(Round (Threshold(low, high, thresh,
Wdn[ii]) ));
END;
END; {with}
END; {Build_Expected_Vector}
96
PROCEDURE Build_Input_Vector;
{}
VAR
ii, jj : INTEGER;
FUNCTION One_if_NZ(This, That : INTEGER) :INTEGER;
{}
VAR
ii : INTEGER;
BEGIN {}
ii := This AND That;
IF (ii <> 0) THEN {}
BEGIN
ii := 1;
END;
One_if_NZ := ii;
END; {}
BEGIN {Build_Input_Vector}
FillChar (vector_i,SizeOf(vector_i),#0);
{Blank the current vector}
FOR ii := 1 TO 5 DO BEGIN
{Notes subscript}
IF cma.notes[ii] > 0 THEN
vector_i [((ii-1)*8)+cma.notes[ii]] := 1;
END; {For notes subscript}
IF (cma.Is_classical) THEN
{}
BEGIN
Vector_I[Max_F1_nodes] := 1;
END
ELSE {}
BEGIN
Vector_I[Max_F1_nodes] := 0;
END;
END; {Build_Input_Vector}
PROCEDURE Display_vectors;
{}
VAR
vii : INTEGER;
BEGIN {}
97
ANSI_CUP(8,28);
Write('ART1: # Committed F2: ',Number_committed_F2,
' Vigilance: ',vigilance:5:4);
ANSI_CUP(9,37);
Write('F2 Winner : ',F2_winner:3);
ANSI_CUP(10,28);
Write('Expected Vector vs. Input Vector');
ANSI_CUP(12,28);
Write('IV: ');
FOR vii := 1 TO vector_length DO BEGIN
{}
IF ((vii MOD 8) = 1) AND (vii <> 1) THEN Write(' ');
Write(Vector_I[vii]);
END; {}
ANSI_CUP(11,28);
Build_Expected_Vector;
Write('EV: ');
FOR vii := 1 TO vector_length DO BEGIN
{}
IF ((vii MOD 8) = 1) AND (vii <> 1) THEN Write(' ');
Write(Vector_X[vii]);
END; {}
END; {}
PROCEDURE Clear_ART1;
{}
VAR
ii, jj : INTEGER;
BEGIN {Clear_ART1}
FOR ii := 1 TO Max_F2_nodes DO BEGIN
{Clear F2 node parameters}
WITH F2^[ii] DO BEGIN
Eligible := TRUE;
Curr_B := 0;
Last_B := 0;
END;
END; {Clear F2 node parameters}
FOR ii := 1 TO Max_F1_nodes DO BEGIN
{Clear F1 node parameters}
WITH F1^[ii] DO BEGIN
Curr_A := 0;
Last_A := 0;
END;
END; {Clear F1 node parameters}
F2_winner := 0;
END; {Clear_ART1}
PROCEDURE Initialize_ART1;
98
{}
VAR
ii, jj, kk : INTEGER;
BEGIN {Initialize_ART1}
Number_committed_F2 := 0;
New(F2);
New(F1);
FOR kk := 1 TO Max_F2_nodes DO BEGIN
{FOR}
WITH F2^[kk] DO BEGIN
{WITH}
Committed := FALSE;
FOR ii := 1 TO Vector_Length DO BEGIN
{For}
Wup[ii] := Initial_Wup;
Wdn[ii] := Initial_Wdn;
Last_B := 0.0;
Curr_B := 0.0;
END; {For}
END; {WITH}
END; {FOR}
FOR kk := 1 TO Max_F1_nodes DO BEGIN
{}
WITH F1^[kk] DO BEGIN
{}
Last_A := 0.0;
Curr_A := 0.0;
END; {}
END; {}
END; {Initialize_ART1}
FUNCTION Delta_STM_F1_node(nde : INTEGER):REAL;
{
Simpson (1988) Eq. 19
a_dot[nde] = - a[nde]
+ (1 - mu1 * a[nde]) *
(gamma1 * F2[f2_winner].wdn[nde] + Input[nde])
- (sigma1 + epsilon1 * a(nde) * (1 if there is a winner)
(0 otherwise)
t1 = - a[nde]
t2 = + (1 - mu1 * a[nde])
t3 = (gamma1 * F2[f2_winner].wdn[nde] + Input[nde])
t4 = (sigma1 + epsilon1 * a(nde)
t5 = (1 if there is a winner)
(0 otherwise)
99
so,
Delta_STM_F1_node := t1 + t2*t3 - t4*t5;
}
VAR
ii, jj : INTEGER;
t1, t2, t3, t4, t5 : REAL;
BEGIN {Delta_STM_F1_node}
t1 := 0;
t2 := 0;
t3 := 0;
t4 := 0;
t5 := 0;
WITH F1^[nde] DO BEGIN
t1 := - Last_A;
t2 := (1 - mu1*Last_A);
IF (F2_winner <> 0) THEN
{}
BEGIN {Make TD term}
t3 := F2^[F2_winner].Wdn[nde];
END;
t3 := (gamma1*t3 + Vector_I[nde]);
t4 := (sigma1 + epsilon1*Last_A);
IF (F2_winner > 0) THEN
{}
BEGIN
t5 := 1;
END
ELSE {}
BEGIN
t5 := 0;
END;
END;
Delta_STM_F1_node := t1 + t2*t3 - t4*t5;
END; {Delta_STM_F1_node}
FUNCTION Delta_STM_F2_node(nde : INTEGER):REAL;
{
Simpson (1988) Eq. 20
b_dot[nde] = - b[nde]
+ (1 - mu2 * b[nde])b*
(gamma2 * [sum over i of S(a[i] * F2[i].wup[nde] ]
+ f(b[nde])
- (sigma2 + epsilon2 * b(nde) *
[sum over k<>j of S(b[k]) ]
100
where,
t1 = - b[nde]
t2 = (1 - mu2 * b[nde])
t3 = (gamma2 * [sum over i of S(a[i] * F2[i].wup[nde] ] + f(b[nde])
t4 = (sigma2 + epsilon2 * b(nde)
t5 = [sum over k<>j of S(b[k]) ]
Delta_STM_F2_node := t1 + t2*t3 - t4*t5;
}
CONST
range = 1;
slope_mod = 1;
shift = 0;
VAR
ii, jj : INTEGER;
t1, t2, t3, t4, t5 : REAL;
BEGIN {Delta_STM_F2_node}
t1 := 0;
t2 := 0;
t3 := 0;
t4 := 0;
t5 := 0;
WITH F2^[nde] DO BEGIN
t1 := - Last_B;
t2 := (1 - mu2*Last_B);
FOR ii := 1 TO Max_F1_nodes DO BEGIN
{Make TD term}
t3 := t3 + sigmoid(1,1,0,F1^[ii].Curr_A)*Wup[ii];
END; {Make TD term}
IF (nde = F2_winner) THEN{}
BEGIN
jj := 1;
END
ELSE {}
BEGIN
jj := 0;
END;
t3 := (gamma2*t3 + jj);
t4 := (sigma2 + epsilon2*Last_B);
FOR ii := 1 TO number_Committed_F2 DO BEGIN
{for}
IF (Eligible) THEN BEGIN
IF (ii <> nde) THEN
t5 := t5 + Sigmoid(range, slope_mod, shift,
101
F2^[ii].Last_B);
END;
END; {for}
END;
Delta_STM_F2_node := t1 + t2*t3 - t4*t5;
END; {Delta_STM_F2_node}
FUNCTION Delta_LTM_Bottom_Up(F2_nde, F1_nde : INTEGER):REAL;
{
Simpson (1988) Eq. 16
wup_dot(ij) = alpha1 * f(b[j]) * (-beta1 * wup[ij] + S(a[i]) )
This corrects an error in the text!
}
CONST
range = 1;
slope_mod = 1;
shift = 0;
VAR
ii, jj : INTEGER;
t1, t2, t3,t4, t5 : REAL;
BEGIN {Delta_LTM_Bottom_Up}
IF (F2_winner = F2_nde) THEN{}
BEGIN
WITH F2^[F2_nde] DO BEGIN{}
Delta_LTM_Bottom_Up := alpha1 * ( -beta1 * Wup[F1_nde]
+ Sigmoid (range, slope_mod, shift,
F1^[F1_nde].curr_A) );
END;
END
ELSE BEGIN
Delta_LTM_Bottom_Up := 0;
END;
END; {Delta_LTM_Bottom_Up}
FUNCTION Delta_LTM_Top_Down(F2_nde, F1_nde : INTEGER):REAL;
{
Simpson (1988) Eq. 17
wdn_dot(ji) = alpha2 * f(b[j]) * (-beta2 * wdn[ji] + S(a[i]) )
This also corrects an error in the text!
}
CONST
range = 1;
slope_mod = 1;
102
shift = 0;
VAR
ii, jj : INTEGER;
t1, t2, t3,t4, t5 : REAL;
BEGIN {Delta_LTM_Top_Down}
IF (F2_winner = F2_nde) THEN{}
BEGIN
WITH F2^[F2_nde] DO BEGIN{}
Delta_LTM_Top_Down :=
alpha2 * ( -beta2 * Wdn[F1_nde] + Sigmoid(
range, slope_mod, shift, F1^[F1_nde].curr_A) );
END; {}
END
ELSE BEGIN
Delta_LTM_Top_Down := 0;
END;
END; {Delta_LTM_Top_Down}
FUNCTION Raw_match:INTEGER;
{
Result of bitwise AND of Vector_I and Vector_X
}
VAR
ii, jj : INTEGER;
BEGIN {Raw_match}
jj := 0;
FOR ii := 1 TO Vector_length DO BEGIN
{}
IF (Vector_I[ii] = 1) AND (Vector_X[ii] = 1) THEN
{}
BEGIN
jj := jj + 1;
END
ELSE {}
BEGIN
END;
END; {}
Raw_match := jj;
END; {Raw_match}
FUNCTION Ones_in_Vector_I:REAL;
{}
VAR
ii, jj : INTEGER;
BEGIN {Ones_in_Vector_I}
103
jj := 0;
FOR ii := 1 TO Vector_Length DO BEGIN
{}
jj := jj + Vector_I[ii];
END; {}
Ones_in_Vector_I := jj;
END; {Ones_in_Vector_I}
FUNCTION Match:BOOLEAN;
{
Return TRUE if Match between I and X exceeds vigilance
}
VAR
RM, OVI : REAL;
BEGIN {Match}
RM := Raw_Match;
OVI := Ones_in_Vector_I;
IF (OVI = 0) THEN {}
BEGIN
IF (RM > 0) THEN {}
BEGIN
Match := TRUE;
END
ELSE {}
BEGIN
Match := FALSE;
END;
END
ELSE {}
BEGIN
Match := ((RM/OVI) >= (Vigilance * ART_co_vigilance));
END;
END; {Match}
FUNCTION Saturated:BOOLEAN;
{}
BEGIN {Saturated}
Saturated := (Number_committed_F2 >= Max_F2_nodes);
END; {Saturated}
FUNCTION Find_F2_winner : INTEGER;
{}
VAR
ii, jj : INTEGER;
max_value : REAL;
Max_position : INTEGER;
104
Min_num : INTEGER;
BEGIN {Find_F2_winner}
{Find the maximum activation}
max_value := -1.0E38;
MAX_POSITION := 1;
IF (Number_Committed_F2 < Max_F2_nodes) THEN
{}
BEGIN
Min_num := Number_Committed_F2;
END
ELSE {}
BEGIN
Min_num := Max_F2_nodes;
END;
FOR jj := 1 TO Min_Num DO BEGIN
{FOR}
WITH F2^[jj] DO BEGIN {WITH}
IF (Eligible) AND (Committed) THEN BEGIN
{Eligible AND Committed}
IF (Curr_B > max_value) THEN BEGIN
{Current > Max}
max_value := Curr_B;
max_position := jj;
END; {Current > Max}
END; {Eligible AND Committed}
END; {WITH}
END; {FOR}
IF (Number_committed_F2 > 0) THEN
{}
BEGIN
Find_F2_winner := max_position;
ANSI_CUP(8,28);
Write
('ART1: # Committed F2: ',Number_committed_F2,
' Vigilance: ',vigilance:5:4);
ANSI_CUP(9,37);
Write('F2 Winner : ',max_position:3);
ANSI_CUP(23,0);
wait;
END
ELSE {}
BEGIN
Find_F2_winner := 0;
ANSI_CUP(9,37);
Write('F2 Winner : ',0:3);
ANSI_CUP(23,0);
END;
END; {Find_F2_winner}
105
FUNCTION Changed_STM_F1(epsilon : REAL) : BOOLEAN;
{}
VAR
ii, jj : INTEGER;
Temp : BOOLEAN;
rr : REAL;
BEGIN {Changed_STM_F1}
Temp := FALSE;
FOR II := 1 TO Vector_Length DO BEGIN
{For}
WITH F1^[ii] DO BEGIN {With}
rr := ABS(Curr_A - Last_A);
IF (rr > epsilon) THEN{Changed}
BEGIN
Temp := TRUE;
END;
END; {With}
END; {For}
Changed_STM_F1 := Temp;
END; {Changed_STM_F1}
FUNCTION Changed_STM_F2 (epsilon : REAL): BOOLEAN;
{}
VAR
ii, jj : INTEGER;
Temp : BOOLEAN;
rr : REAL;
BEGIN {Changed_STM_F2}
Temp := FALSE;
IF (F2_winner > 0) AND (F2_winner <= Max_F2_nodes) THEN BEGIN
WITH F2^[F2_winner] DO BEGIN
{With}
rr := ABS(Curr_B - Last_B);
IF (rr > epsilon) THEN{Changed}
BEGIN
Temp := TRUE;
END;
END; {With}
END
ELSE BEGIN
Temp := FALSE;
END;
Changed_STM_F2 := Temp;
END; {Changed_STM_F2}
PROCEDURE Do_resonate;
106
{}
CONST
Iter = 10;
E = 0.05;
VAR
ii, jj, End_loop : INTEGER;
BEGIN {Do_resonate}
End_loop := 0;
{While change in STM do alternate BU and TD STM and BU and TD LTM}
REPEAT {}
{BU STM}
FOR ii := 1 TO Vector_Length DO BEGIN
{For F1 STM}
WITH F1^[ii] DO BEGIN {With F1}
Last_A := Curr_A;
Curr_A := Last_A + time_slice * Delta_STM_F1_node(ii);
END; {With F1}
END; {For F1 STM}
{BU LTM}
FOR ii := 1 TO Vector_Length DO BEGIN
{For BU LTM}
WITH F2^[F2_winner] DO BEGIN
{With F2_winner}
Wup[ii] := Wup[ii] + time_slice
* Delta_LTM_Bottom_up(F2_winner,ii);
END; {With F2_winner}
END; {For BU LTM}
{ Display_vectors;}
{TD STM}
WITH F2^[F2_winner] DO BEGIN
{}
Last_B := Curr_B;
Curr_B := Last_B + time_slice
* Delta_STM_F2_node(F2_winner);
END; {}
{TD LTM}
FOR ii := 1 TO Vector_Length DO BEGIN
{For TD LTM}
WITH F2^[F2_winner] DO BEGIN
{With F2_winner}
Wdn[ii] := Wdn[ii] + time_slice
* Delta_LTM_Top_Down(F2_winner,ii);
END; {With F2_winner}
107
END; {For TD LTM}
{ Display_vectors;}
End_Loop := End_Loop + 1;
UNTIL ((NOT Changed_STM_F1(e)) AND (NOT Changed_STM_F2(e)))
OR (End_loop > Iter); {}
Display_vectors;
Resonance := TRUE;
END; {Do_resonate}
FUNCTION Exists_eligible : BOOLEAN;
{}
VAR
ii, jj : INTEGER;
Temp : BOOLEAN;
BEGIN {Exists_eligible}
Temp := FALSE;
FOR jj := 1 TO Number_committed_F2 DO BEGIN
{FOR}
WITH F2^[jj] DO BEGIN {WITH}
IF (Eligible) AND (Committed) THEN BEGIN
{Eligible AND Committed}
Temp := TRUE;
END; {Eligible AND Committed}
END; {WITH}
END; {FOR}
Exists_eligible := Temp;
END; {Exists_eligible}
PROCEDURE Encode_New_F2;
{}
VAR
ii, jj : INTEGER;
BEGIN {}
IF number_committed_F2 < Max_F2_nodes THEN BEGIN
{Find next uncommitted F2 node}
F2_winner := Number_committed_F2 + 1;
{Resonate uncommitted F2 node with F1}
Do_resonate;
F2^[F2_winner].Eligible := TRUE;
F2^[F2_winner].Committed := TRUE;
Number_committed_F2 := Number_committed_F2 + 1;
END; {IF }
END; {}
108
PROCEDURE Find_F1_activation;
VAR
i, j : INTEGER;
BEGIN
ANSI_CUP(14,37);
Write('Find F1 Activation':30);
ANSI_CUP(23,0);
FOR j := 1 TO 3 DO
FOR i := 1 TO Vector_Length DO BEGIN
{FOR}
WITH F1^[i] DO BEGIN {WITH}
Last_A := Curr_A;
Curr_A := Last_A + time_slice * Delta_STM_F1_Node(i);
END; {WITH}
END; {FOR}
END;
PROCEDURE Find_F2_activation;
VAR
i, J : INTEGER;
BEGIN
ANSI_CUP(14,37);
Write('Find F2 Activation':30);
ANSI_CUP(23,0);
FOR j := 1 TO Number_committed_F2 DO BEGIN
{FOR}
WITH F2^[j] DO BEGIN {WITH}
IF (Eligible) AND (Committed) THEN
{}
BEGIN
Last_B := Curr_B;
Curr_B := Last_B + time_slice * Delta_STM_F2_node(j);
END;
END; {WITH}
END; {FOR}
END;
BEGIN {ART1}
ANSI_CUP(8,28);
Write('ART1:');
ANSI_CUP(14,28);
Write('Process:');
ANSI_CUP(23,0);
Build_Input_vector;
IF (NOT Initialized) THEN BEGIN {do initialize}
109
ANSI_CUP(14,37);
Write('Initialize':30);
ANSI_CUP(23,0);
Initialize_ART1;
Initialized := TRUE;
END;
{Clear_ART}
ANSI_CUP(14,37);
Write('Clear':30);
ANSI_CUP(23,0);
Clear_ART1;
cma.Delta_Vigilance := FALSE;
cma.New_Category := FALSE;
REPEAT
user_keys;
Resonance := FALSE;
{Find current F1 activation}
{F1_i node activation * Wij}
Find_F1_activation;
{Send F1 activation to F2}
Find_F2_activation;
{If no committed F2 nodes, then
proceed to new encoding}
IF (NOT exists_eligible) OR (number_committed_F2 = 0) THEN
{}
BEGIN
ANSI_CUP(14,37);
Write('No comm. F2, encoding':30);
ANSI_CUP(23,0);
Encode_New_F2;
Resonance := TRUE;
cma.New_Category := TRUE;
END
ELSE BEGIN
{F2 competition}
{Determine maximum of
eligible F2 nodes}
ANSI_CUP(14,37);
Write('F2 Competition':30);
ANSI_CUP(23,0);
F2_winner := Find_F2_winner;
{F2 winner sends TD image back to F1}
{Activation of F2 winner * Wji}
Display_vectors; {Makes a call to Build_Expected_Vector}
{Compare Input vector to F2 TD vector}
IF (Match) THEN BEGIN {Resonate}
ANSI_CUP(14,37);
Write('Matched, now resonate':30);
110
ANSI_CUP(23,0);
Resonance := TRUE;
Do_resonate;
END {Resonate}
ELSE BEGIN {Mismatch}
ANSI_CUP(14,37);
Write('Mismatch':30);
ANSI_CUP(23,0);
{Make the F2 node ineligible}
F2^[F2_winner].Eligible := FALSE;
{Have we saturated?}
{Yes, lower vigilance and restart}
{ * While this is not part of the
Carpenter-Grossberg ART 1 architecture,
this modification we felt necessary for
the small number of category nodes which
we are using in the model. * }
IF (Saturated) AND (NOT exists_eligible) THEN
{If saturation reached decrease vigilance and
restart}
BEGIN {Saturation}
ANSI_CUP(14,37);
Write('Saturated, vigilance--':30);
ANSI_CUP(23,0); {vigilance is decreased}
Vigilance := Vigilance * 0.99;
ANSI_CUP(8,57);
Write('Vigilance: ',vigilance:5:4);
ANSI_CUP(23,0);
cma.Delta_Vigilance := TRUE;
{call clear_ART1}
Clear_ART1;
Find_F1_activation;
Find_F2_activation;{now find closest match and resonate}
F2_winner := Find_F2_winner;
Resonance := TRUE;
Do_resonate;
END {Saturation}
ELSE BEGIN {Not saturated or exists_eligible}
{Are there eligible F2 nodes?}
IF (Exists_eligible) THEN
{Yes, go on with current process}
BEGIN
ANSI_CUP(14,37);
Write('Search Eligible F2':30);
ANSI_CUP(23,0); {Just continue}
END
ELSE BEGIN
{No, form a new encoding if not saturated}
111
IF NOT saturated THEN BEGIN
ANSI_CUP(14,37);
Write('Encode new category':30);
ANSI_CUP(23,0);
Encode_New_F2;
Resonance := TRUE;
cma.New_Category := TRUE;
END; {IF NOT saturated}
END;
END; {Not saturated}
END; {Mismatch}
END; {ELSE}
UNTIL (Resonance);
{Prep info to pass back}
END; {ART1}
{----------------------------------------------------}
BEGIN {Beethoven}
Dump_common(cmn);
ART1(cmn);
END; {Beethoven}
{----------------------------------------------------------}
PROCEDURE Lobes;
{
Keeps track of played notes, maintaining sequence information.
Uses data from Beethoven to determine when to override Salieri.
}
CONST
Max_notes_in_composition = 152;
Object_threshold = 3;
Frustration_threshold = 10;
VAR
Common : Common_Area_;
Number_notes : INTEGER; {Note counter}
Objects : BOOLEAN;
Note_Played : BOOLEAN;
Generate_Candidate : BOOLEAN;
Need_Critique : BOOLEAN;
Need_Compose : BOOLEAN;
Since_Novelty, Frustration : INTEGER;
ii : INTEGER;
BEGIN {Lobes}
112
Randomize;
note_rec.c := 0;
Since_Novelty := 0;
Frustration := 0;
Common.notes[1] := 0;
Common.notes[2] := 0;
Common.notes[3] := 0;
Common.notes[4] := 0;
Common.notes[5] := 0;
Common.delta_vigilance := FALSE;
Common.new_category := FALSE;
Common.candidate_note := 0;
Common.is_classical := FALSE;
ANSI_CUP(14,37);
Write('Begin Simulation':30);
ANSI_CUP(23,0);
FOR Number_notes := 1 TO Max_notes_in_composition DO BEGIN
{}
user_keys;
Note_played := FALSE;
Generate_Candidate := TRUE;
Need_Critique := TRUE;
Need_Compose := TRUE;
REPEAT
IF Generate_candidate THEN
{Generate a candidate note, HTn}
BEGIN
Bach(Common);
ANSI_CUP(6,0);
Write('Candidate Note:');
ANSI_CUP(6,35);
Write(common.candidate_note);
ANSI_CUP(23,0);
END;
wait;
IF Need_Critique THEN BEGIN
{Find if it is a candidate sequence, PDP}
Salieri(Common);
ANSI_CUP(5,0);
Write('Candidate sequence classical?:');
ANSI_CUP(5,31);
Write(common.Is_Classical);
ANSI_CUP(23,0);
END;
wait;
IF Need_Compose THEN {Pass through ART and }
BEGIN
113
Beethoven(Common);
END;
wait; {IF Delta_vigilance or New_category,
then zero the count}
{Else increment the count}
IF (Common.Delta_vigilance OR Common.New_Category) THEN
{}
BEGIN
Since_Novelty := 0;
END
ELSE {}
BEGIN
Since_novelty := Since_Novelty + 1;
END;
IF (Common.Delta_vigilance) THEN BEGIN
INC(Frustration);
END;
{If count >= Object_threshold), then Objects
is true, reset count}
{Else Objects is false}
Objects := (Since_Novelty >= Object_threshold);
{OR (Frustration > Frustration_Threshold);}
IF Objects THEN since_novelty := 0;
IF (objects AND common.is_classical) OR ((NOT objects)
AND (NOT common.is_classical)) THEN BEGIN
generate_candidate := TRUE;
need_critique := TRUE;
need_compose := TRUE;
common.notes[v_len_out] := 0;
INC(Frustration);
END
ELSE BEGIN
record_a_note(Common);
note_played := TRUE;
Frustration := 0;
END;
UNTIL (Note_played); {A note has been played}
END; {}
END; {Lobes}
{----------------------------------------------------------}
BEGIN {Main}
WRITELN('Copyright 1989 by Wesley R. Elsberry');
114
DELAY(2000);
ANSI_CLRSCR;
Lobes;
END. {Main}
APPENDIX 3
PROGRAM SOURCE LISTING: BACK-PROPAGATION UNIT
115
116
UNIT BP_unit;
{
This unit implements the necessary functions for modelling back-
propagation artificial neural network architectures.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
USES
DOS, Struct, ANN;
CONST
mach_inf = 1E37;
exp_max = 80.0;
TAB = ^I;
Debug : BOOLEAN = FALSE;
TYPE
REAL = SINGLE;
file_string_ = STRING;
node_type_ = (Input,hidden,Output);
weight_ptr_ = ^weight_;
weight_ = RECORD
w, dw : REAL;
END;
vector_node_ptr_ = ^vector_node_;
117
vector_node_ = RECORD
V : REAL;
END;
sptr_ = ^STRING;
data_rec_ptr_ = ^data_rec_;
data_rec_ = RECORD
s : sptr_;
n : data_rec_ptr_;
END;
weight_node_ptr_ = ^weight_node_;
weight_node_ = RECORD
connect : BOOLEAN;
w, dw, ldw : REAL;
END;
BP_net_ptr_ = ^BP_net_;
BP_node_ptr_ = ^BP_node_;
BP_net_ = RECORD
vs, ve : DVE_ptr_; {node vector start and vector end}
ws : DVE_ptr_; {weight array}
learning_rate : REAL;
alpha : REAL; {factor for momentum term}
vi : DVE_ptr_; {input vector}
vos, voe : DVE_ptr_; {output vector}
vts, vte : DVE_ptr_; {training vector}
n_input, n_hidden, n_output : WORD;
maxerr : REAL;
errtol : REAL;
data_fname : file_string_;
data_f : TEXT;
training_iterations : INTEGER;
out_fname : file_string_;
out_f : TEXT;
wt_fname : file_string_;
wt_f : TEXT;
END;
BP_node_ = RECORD
nt : node_type_; {Input, hidden, or output}
loc : WORD;
ni : REAL; {net input value}
delta : REAL; {delta value for node}
base : REAL;
range : REAL;
118
theta : REAL;
dtheta, ldtheta : REAL;
fw, bw : DVE_ptr_; {points to entries in weight_matrix}
END;
PROCEDURE Dump_BP_net_weights
(VAR BPN : BP_net_; VAR Fname : STRING);
{Save weights and node bias unit values to a file}
PROCEDURE Set_BP_net_weights_from_file
(VAR BPN : BP_net_; VAR Fname : STRING);
{Restore weights and node bias unit values from a file}
PROCEDURE BP_set_net_connects_from_file
(VAR BPN : BP_net_; VAR Fname : STRING);
{Sets network connectivity values from a file}
PROCEDURE Setup_BP_net
(VAR BPN : BP_net_;VAR Fname : STRING);
{
Get data values from a text file to set up basic BP constants, sizes,
and other necessary information, or query user if filename is not valid.
}
PROCEDURE Set_Input_vector_from_file
(VAR BPN : BP_net_);
{Get data values from a text file to fill input vector.}
PROCEDURE Set_Training_vector_from_file
(VAR BPN : BP_net_);
{Get data values from a text file to fill training vector.}
PROCEDURE BP_Feed_forward
(VAR BPN : BP_net_);
{Present values to network and propagate values forward, set the output
vector.}
119
PROCEDURE BP_train_presentation
(VAR BPN : BP_net_);
{
Present values to network, propagate forward, set output, compare output
to training, back-propagate, collect statistics but do not change
weights.
}
PROCEDURE BP_train_and_change
(VAR BPN : BP_net_);
{Present values to network, propagate forward, set output, compare output
to training, back-propagate, collect statistics, change weights, and reset
statistic variables.}
PROCEDURE BP_change
(VAR BPN : BP_net_);
{Change weights using current statistics and reset statistics.}
PROCEDURE BP_dump_net
(VAR BPN : BP_net_);
{Dump net parameters, node activities, and weights for inspection.}
FUNCTION BP_net_error
(VAR BPN : BP_net_):REAL;
{Returns the largest error from the output nodes}
PROCEDURE Display_weights
(BPN : BP_net_);
{Display of the current weight values for the network}
{---------------------------------------------------------------------}
IMPLEMENTATION
{---------------------------------------------------------------------}
{Private, internal functions}
120
FUNCTION max (r1, r2 : REAL):REAL;
BEGIN
IF r1 >= r2 THEN max := r1
ELSE max := r2;
END;
{---------------------------------------------------------------------}
PROCEDURE Dump_BP_net_weights (VAR BPN : BP_net_;
VAR Fname : STRING);
{Save weights and node bias unit values to a file}
{
Preface vector length with !V
Preface weight vectors with !W
Preface bias unit vector with !T
}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, m, n : WORD;
inch : CHAR;
done : BOOLEAN;
tp1 : DVE_ptr_;
ss : STRING;
BEGIN
n := BPN.n_input + BPN.n_hidden + BPN.n_output;
(*
ss := FSEARCH(Fname,GETENV('PATH'));
IF LENGTH(ss) = 0 THEN BEGIN
WRITELN('**ERROR** File does not exist');
EXIT;
END; *)
BPN.wt_fname := Fname;
Assign(BPN.wt_f,Fname);
Rewrite(BPN.wt_f);
done := FALSE;
{Write vector length}
Writeln(BPN.wt_f,'!V ',n:1);
121
FOR jj := 1 TO n DO BEGIN
Write(BPN.wt_f,'!W ');
FOR ii := 1 TO n DO BEGIN
tp1 := Find_element_matrix(ii,jj,BPN.ws);
IF wnp_(tp1^.dptr)^.connect THEN
Write(BPN.wt_f,wnp_(tp1^.dptr)^.w:4:4,' ')
ELSE Write(BPN.wt_f,0.0:4:4,' ');
END;
Writeln(BPN.wt_f);
END;
Write(BPN.wt_f,'!T ');
FOR ii := 1 TO n DO BEGIN
tp1 := Find_element_DVE(ii,BPN.vs);
Write(BPN.wt_f,bpnp_(tp1^.dptr)^.theta:4:4,' ');
END;
Writeln(BPN.wt_f);
Writeln(BPN.wt_f,'!Z ');
Close(BPN.wt_f);
END; {Dump_BP_net_weights}
PROCEDURE Set_BP_net_weights_from_file (VAR BPN : BP_net_;
VAR Fname : STRING);
{Restore weights and node bias unit values from a file}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, m, n : WORD;
rr : REAL;
tp1, wp1, wp2 : DVE_ptr_;
inch : CHAR;
done : BOOLEAN;
ss : STRING;
BEGIN {}
n := BPN.n_input + BPN.n_hidden + BPN.n_output;
ss := FSEARCH (Fname,GETENV('PATH'));
IF (Length(ss) = 0) THEN BEGIN
Writeln ('**ERROR** File does not exist');
EXIT;
END;
122
Assign (BPN.wt_f,ss);
Reset (BPN.wt_f);
done := FALSE;
{Find vector length, compare to net vector length}
REPEAT
REPEAT {find command}
Read (BPN.wt_f,inch);
UNTIL (inch = '!') OR Eof(BPN.wt_f);
{}
Read (BPN.wt_f,inch);
UNTIL (UpCase (inch) = 'V') OR Eof (BPN.wt_f);
IF Eof (BPN.wt_f) THEN BEGIN
EXIT;
END;
Read (BPN.wt_f,inch);
Read (BPN.wt_f,m);
IF (m <> n) THEN BEGIN {Vector lengths don't match, quit}
EXIT;
END;
wp1 := BPN.ws;
REPEAT {get net params}
REPEAT {find command}
Read (BPN.wt_f,inch);
UNTIL (inch = '!'); {}
Read (BPN.wt_f,inch);
CASE UpCase (inch) OF
'T' : BEGIN {get bias values}
Read (BPN.data_f, inch);
FOR ii := 1 TO n DO BEGIN
tp1 := Find_element_DVE (ii,BPN.vs);
Read (BPN.wt_f,rr);
bpnp_(tp1^.dptr)^.theta := rr;
END;
END; {}
'W' : BEGIN {get weights}
IF wp1 <> NIL THEN BEGIN
Read (BPN.data_f,inch);
FOR ii := 1 TO n DO BEGIN
wp2 := Find_element_DVE (ii,wp1);
Read (BPN.wt_f,rr);
wnp_(wp2^.dptr)^.w := rr;
END; {For ii}
wp1 := wp1^.down;
END;
123
END; {}
'Z' : DONE := TRUE;
ELSE
BEGIN
DONE := TRUE;
END;
END;
UNTIL (done OR Eof (BPN.wt_f));
{}
END; {set_BP_net_weights_from_file}
PROCEDURE BP_set_net_connects_from_file (VAR BPN : BP_net_;
VAR Fname : STRING);
{Sets network connectivity values from a file}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, kk, m, n : WORD;
tp1, wp1, wp2 : DVE_ptr_;
inch : CHAR;
done : BOOLEAN;
ss : STRING;
cfile : TEXT;
BEGIN {}
n := BPN.n_input + BPN.n_hidden + BPN.n_output;
ss := FSEARCH (Fname,GETENV('PATH'));
IF (Length(ss) = 0) THEN BEGIN
Writeln ('**ERROR** File does not exist');
EXIT;
END;
Assign (cfile,ss);
Reset (cfile);
done := FALSE;
{Find vector length, compare to net vector length}
REPEAT
REPEAT {find command}
Read (cfile,inch);
UNTIL (inch = '!') OR Eof(cfile);
{}
124
Read (cfile,inch);
UNTIL (UpCase (inch) = 'V') OR Eof (cfile);
IF Eof (cfile) THEN BEGIN
EXIT;
END;
Read (cfile,inch);
Read (cfile,m);
IF (m <> n) THEN BEGIN {Vector lengths don't match, quit}
EXIT;
END;
wp1 := BPN.ws;
REPEAT {get net params}
REPEAT {find command}
Read (cfile,inch);
UNTIL (inch = '!'); {}
Read (cfile,inch);
CASE UpCase (inch) OF
'C' : BEGIN {get weights}
IF wp1 <> NIL THEN BEGIN
Read (cfile,inch);
FOR ii := 1 TO n DO BEGIN
wp2 := Find_element_DVE (ii,wp1);
Read (cfile,kk);
wnp_(wp2^.dptr)^.connect := (kk = 1);
END; {For ii}
wp1 := wp1^.down;
END;
END; {}
'Z' : DONE := TRUE;
ELSE
BEGIN
DONE := TRUE;
END;
END;
UNTIL (done OR Eof (cfile));
{}
END; {BP_set_net_connects_from_file}
PROCEDURE Dump_node (net : BP_net_;
node : BP_node_ptr_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
125
BEGIN
IF (debug) THEN {}
BEGIN
Writeln ('DUMP_NODE');
Writeln (NET.OUT_F,'DUMP_NODE');
END;
IF (node <> NIL) THEN BEGIN
CASE node^.nt OF
Input : BEGIN
Write (net.out_f,'INPUT');
Write ('INPUT');
END;
hidden : BEGIN
Write (net.out_f,'HIDDEN');
Write ('HIDDEN');
END;
Output : BEGIN
Write (net.out_f,'OUTPUT');
Write ('OUTPUT');
END;
END;
Write (net.out_f,tab,'LOC : ',NODE^.LOC);
Write ('LOC : ',NODE^.LOC);
Writeln (net.out_f,tab,'ADDR : ',Seg(NODE),':',Ofs(NODE));
Writeln (tab,'ADDR : ',Seg(NODE),':',Ofs(NODE));
Write (net.out_f,'NI : ',node^.ni:8);
Write ('NI : ',node^.ni:8);
Writeln (net.out_f,tab,'DELTA : ',NODE^.DELTA:8);
Writeln (tab,'DELTA : ',NODE^.DELTA:8);
Write (net.out_f,'BASE : ',NODE^.BASE:8);
Write ('BASE : ',NODE^.BASE:8);
Writeln (net.out_f,tab,'RANGE : ',NODE^.RANGE:8);
Writeln (tab,'RANGE : ',NODE^.RANGE:8);
Write (net.out_f,'THETA : ',NODE^.THETA:8);
Write ('THETA : ',NODE^.THETA:8);
Writeln (net.out_f,tab,'DTHETA : ',NODE^.DTHETA:8,tab,
'LDTHETA : ',
NODE^.LDTHETA:8);
Writeln (tab,'DTHETA : ',NODE^.DTHETA:8,tab,
'LDTHETA : ',NODE^.LDTHETA:8);
Writeln (net.out_f,'FW : ',Seg(NODE^.FW),':', Ofs(NODE^.FW),
tab,'BW : ', Seg(NODE^.BW),':',Ofs(NODE^.BW));
Writeln ('FW : ',Seg(NODE^.FW),':', Ofs(NODE^.FW),tab,'BW : ',
Seg(NODE^.BW),':',Ofs(NODE^.BW));
Writeln (net.out_f);
Writeln;
END;
IF (debug) THEN BEGIN
Writeln ('END DUMP_NODE');
Writeln (NET.OUT_F,'END DUMP_NODE');
END;
126
END;
PROCEDURE Display_weights (BPN : BP_net_);
{Display of the current weight values for the network}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
twpd, twpa : DVE_ptr_;
wptr : DVE_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('DISPLAY_WEIGHTS');
Writeln (BPN.OUT_F,'DISPLAY_WEIGHTS');
END;
twpd := BPN.ws;
twpa := BPN.ws;
WHILE (twpd <> NIL) DO BEGIN{}
WHILE (twpa <> NIL) DO BEGIN
{}
wptr := twpa^.dptr;
IF (wnp_(wptr)^.connect) THEN BEGIN
Write (BPN.out_f,wnp_(wptr)^.w:5:1,' ');
Write (wnp_(wptr)^.w:5:1,' ');
END
ELSE {}
BEGIN
Write (BPN.out_f,' --- ');
Write (' --- ');
END;
twpa := twpa^.right;
END; {}
Writeln (BPN.out_f);
Writeln ;
twpd := twpd^.down;
twpa := twpd;
END; {}
Writeln (BPN.out_f,'End of weights');
Writeln ('End of weights');
Writeln;
Writeln;
IF (debug) THEN {}
BEGIN
Writeln ('END DISPLAY_WEIGHTS');
Writeln (BPN.OUT_F,'END DISPLAY_WEIGHTS');
END;
127
Flush (BPN.out_f);
END; {}
PROCEDURE Display_Vector (vp : DVE_ptr_;
N : INTEGER;
NET : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii : INTEGER;
vptr : vector_node_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('DISPLAY_VECTOR');
Writeln (NET.OUT_F,'DISPLAY_VECTOR');
END;
FOR II := 1 TO N DO BEGIN {}
vptr := vp^.dptr;
Write (net.out_f,vptr^.V:8,' ');
Write (vptr^.V:8,' ');
vp := vp^.right;
END;
Writeln (net.out_f);
Writeln;
IF (debug) THEN {}
BEGIN
Writeln ('END DISPLAY_VECTOR');
Writeln (NET.OUT_F,'END DISPLAY_VECTOR');
END;
END; {}
FUNCTION BP_net_error (VAR BPN : BP_net_):REAL;
{Returns the largest error from the output nodes}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
rr : REAL;
tp, vp1, vp2 : DVE_ptr_;
ii, n : INTEGER;
128
BEGIN
rr := 0;
n := BPN.n_input + BPN.n_hidden + BPN.n_output;
vp1 := BPN.vos;
vp2 := BPN.vts;
FOR ii := 1 TO BPN.n_output DO BEGIN
IF (vp1^.dptr <> NIL) AND (vp2^.dptr <> NIL) THEN BEGIN
rr := max (ABS(rr),
ABS(vnp_(vp2^.dptr)^.v - vnp_(vp1^.dptr)^.v));
IF vp1^.right <> NIL THEN vp1 := vp1^.right;
IF vp2^.right <> NIL THEN vp2 := vp2^.right;
END
ELSE BEGIN
END;
END;
BP_net_error := ABS(rr);
END;
PROCEDURE Allocate_IO_vectors (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, N : WORD;
BEGIN
IF debug THEN BEGIN
Writeln ('ALLOCATE_IO_VECTORS');
Writeln (NET.OUT_F,'ALLOCATE_IO_VECTORS');
END;
N := net.n_input + net.n_hidden + net.n_output;
net.vi := Create_DVE_vector (net.n_input, SizeOf(vector_node_));
net.vos := Create_DVE_vector (net.n_output, SizeOf(vector_node_));
net.vts := Create_DVE_vector (net.n_output, SizeOf(vector_node_));
net.voe := Find_element_DVE (net.n_output, net.vos);
net.vte := Find_element_DVE (net.n_output, net.vts);
IF (debug) THEN {}
BEGIN
Writeln ('END ALLOCATE_IO_VECTORS');
Writeln (NET.OUT_F,'END ALLOCATE_IO_VECTORS');
END;
END;
PROCEDURE Allocate_node_vector (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
129
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, cnt, N : INTEGER;
Temp : BP_node_ptr_;
nptr : DVE_ptr_;
low, high : REAL;
ns : BP_node_;
BEGIN
cnt := 1;
N := net.n_input + net.n_hidden + net.n_output;
IF debug THEN BEGIN
Writeln ('ALLOCATE_NODE_VECTOR');
Writeln (NET.OUT_F,'ALLOCATE_NODE_VECTOR');
END;
net.vs := Create_DVE_vector (net.n_input, SizeOf (BP_node_));
net.ve := Find_element_DVE (net.n_input, net.vs);
nptr := net.vs;
FOR ii := 1 TO net.n_input DO BEGIN
temp := nptr^.dptr;
Temp^.loc := cnt;
cnt := cnt + 1;
Temp^.nt := Input; {set node type}
Temp^.ni := 0; {init net input}
Temp^.fw := NIL; {no weight yet}
Temp^.bw := NIL; {no weight yet}
Temp^.delta := 0;
Temp^.base := 0;
Temp^.range := 1;
Temp^.theta := 0;
Temp^.dtheta := 0;
Temp^.ldtheta := 0;
nptr := nptr^.right;
END; {FOR}
net.ve^.right := Create_DVE_vector (net.n_hidden,
SizeOf(BP_node_));
net.ve^.right^.left := net.ve;
nptr := net.ve^.right;
net.ve := Find_element_DVE (net.n_input + net.n_hidden, net.vs);
FOR II := 1 TO net.n_hidden DO BEGIN
temp := nptr^.dptr;
Temp^.loc := cnt;
cnt := cnt + 1;
Temp^.nt := hidden; {set node type}
Temp^.ni := 0; {init net input}
Temp^.fw := NIL; {no weight yet}
Temp^.bw := NIL; {no weight yet}
Temp^.delta := 0;
130
Temp^.base := 0;
Temp^.range := 1;
Temp^.theta := gaussian_noise (0,0.25);
Temp^.dtheta := 0;
Temp^.ldtheta := 0;
nptr := nptr^.right;
END;
net.ve^.right := Create_DVE_vector (net.n_output,
SizeOf(BP_node_));
net.ve^.right^.left := net.ve;
nptr := net.ve^.right;
net.ve := Find_element_DVE (net.n_input + net.n_hidden
+ net.n_output,net.vs);
FOR II := 1 TO net.n_output DO BEGIN
temp := nptr^.dptr;
Temp^.loc := cnt;
cnt := cnt + 1;
Temp^.nt := Output; {set node type}
Temp^.ni := 0; {init net input}
Temp^.fw := NIL; {no weight yet}
Temp^.bw := NIL; {no weight yet}
Temp^.delta := 0;
Temp^.base := 0 {low};
Temp^.range := 1 {high - low};
Temp^.theta := gaussian_noise (0,0.25);
Temp^.dtheta := 0;
Temp^.ldtheta := 0;
nptr := nptr^.right;
END;
IF (debug) THEN BEGIN
Writeln ('END ALLOCATE_NODE_VECTOR');
Writeln (NET.OUT_F,'END ALLOCATE_NODE_VECTOR');
END;
END;
PROCEDURE Allocate_weight_matrix (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, N : WORD;
temp : DVE_ptr_;
tl, tc, tls, tcs : weight_node_ptr_;
Vt : Vector_node_ptr_;
cnt : INTEGER;
131
BEGIN
IF debug THEN BEGIN
Writeln ('ALLOCATE_WEIGHT_MATRIX');
Writeln (NET.OUT_F,'ALLOCATE_WEIGHT_MATRIX');
END;
cnt := 1;
N := net.n_input + net.n_hidden + net.n_output;
net.ws := create_matrix (n,n,SizeOf(weight_node_));
FOR ii := 1 TO n DO BEGIN
FOR jj := 1 TO n DO BEGIN
temp := Find_element_matrix(ii, jj, net.ws);
IF temp <> NIL THEN BEGIN
wnp_(temp^.dptr)^.connect := FALSE;
wnp_(temp^.dptr)^.w := 2 * Random - 1;
{random weights, -1 < w < 1 }
wnp_(temp^.dptr)^.dw := 0;
wnp_(temp^.dptr)^.ldw := 0;
END; {IF temp <> NIL}
END;
END;
IF (debug) THEN {}
BEGIN
Writeln ('END ALLOCATE_WEIGHT_MATRIX');
Writeln (NET.OUT_F,'END ALLOCATE_WEIGHT_MATRIX');
END;
END;
PROCEDURE Link_weights_to_nodes (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
Start, TempD, TempA : DVE_ptr_;
Vt : DVE_ptr_;
ii, jj, N : INTEGER;
BEGIN
{Link to node vector}
IF debug THEN BEGIN
Writeln ('LINK_WEIGHTS_TO_NODES');
Writeln (NET.OUT_F,'LINK_WEIGHTS_TO_NODES');
END;
N := net.n_input + net.n_hidden + net.n_output;
Start := net.ws;
TempD := Start;
TempA := Start;
Vt := net.vs;
132
FOR ii := 1 TO N DO BEGIN
bpnp_(Vt^.dptr)^.fw := TempD;
TempD := TempD^.down;
bpnp_(Vt^.dptr)^.bw := TempA;
TempA := TempA^.right;
Vt := Vt^.right;
END;
IF (debug) THEN {}
BEGIN
Writeln ('END LINK_WEIGHTS_TO_NODES');
Writeln (NET.OUT_F,'END LINK_WEIGHTS_TO_NODES');
END;
END; {Link_weights_to_nodes}
PROCEDURE Display_node_type (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
tp : DVE_ptr_;
N, ii : INTEGER;
BEGIN {}
IF debug THEN BEGIN
Writeln ('DISPLAY_NODE_TYPE');
Writeln (NET.OUT_F,'DISPLAY_NODE_TYPE');
END;
tp := net.vs;
N := net.n_input + net.n_hidden + net.n_output;
FOR ii := 1 TO n DO BEGIN {}
CASE bpnp_(tp^.dptr)^.nt OF
Input : BEGIN
Write (net.out_f,'I');
Write ('I');
END;
hidden : BEGIN
Write (net.out_f,'H');
Write ('H');
END;
Output : BEGIN
Write (net.out_f,'O');
Write ('O');
END;
END;
TP := tp^.right;
END; {}
Writeln (net.out_f);
133
Writeln ;
IF (debug) THEN {}
BEGIN
Writeln ('END DISPLAY_NODE_TYPE');
Writeln (NET.OUT_F,'END DISPLAY_NODE_TYPE');
END;
END; {}
PROCEDURE Display_node_deltas (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
tp : DVE_ptr_;
N, ii : INTEGER;
BEGIN {}
IF debug THEN BEGIN
Writeln ('DISPLAY_NODE_DELTAS');
Writeln (NET.OUT_F,'DISPLAY_NODE_DELTAS');
END;
tp := net.vs;
N := net.n_input + net.n_hidden + net.n_output;
FOR ii := 1 TO n DO BEGIN {}
TP := tp^.right;
END; {}
IF (debug) THEN {}
BEGIN
Writeln ('END DISPLAY_NODE_DELTAS');
Writeln (NET.OUT_F,'END DISPLAY_NODE_DELTAS');
END;
END; {}
PROCEDURE set_connectivity (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
wpt : DVE_ptr_;
tvd, tva : DVE_ptr_;
ii, jj, n : INTEGER;
BEGIN
IF debug THEN BEGIN
134
Writeln ('SET_CONNECTIVITY');
Writeln (NET.OUT_F,'SET_CONNECTIVITY');
END;
n := net.n_input + net.n_hidden + net.n_output;
tvd := net.vs; {beginning of node vector}
tva := net.vs;
wpt := bpnp_(tvd^.dptr)^.fw;
{display_node_type (net);}
FOR ii := 1 TO n DO BEGIN
FOR jj := 1 TO n DO BEGIN
CASE bpnp_(tvd^.dptr)^.nt OF
Input : BEGIN
wnp_(wpt^.dptr)^.connect := FALSE;
END;
hidden : BEGIN
IF (bpnp_(tva^.dptr)^.nt = Input) THEN BEGIN
wnp_(wpt^.dptr)^.connect := TRUE;
END
ELSE BEGIN
wnp_(wpt^.dptr)^.connect := FALSE;
END;
END;
Output : BEGIN
IF (bpnp_(tva^.dptr)^.nt = hidden) THEN BEGIN
wnp_(wpt^.dptr)^.connect := TRUE;
END
ELSE BEGIN
wnp_(wpt^.dptr)^.connect := FALSE;
END;
END;
END; {case}
wpt := wpt^.right;
tva := tva^.right;
END;
tvd := tvd^.right;
tva := net.vs;
wpt := bpnp_(tvd^.dptr)^.fw;
END;
IF (debug) THEN {}
BEGIN
Writeln ('END SET_CONNECTIVITY');
Writeln (NET.OUT_F,'END SET_CONNECTIVITY');
END;
END;
PROCEDURE Display_output (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
135
vnp_ = vector_node_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('DISPLAY_OUTPUT');
Writeln (NET.OUT_F,'DISPLAY_OUTPUT');
END;
display_vector (net.vos,net.n_output,net);
IF (debug) THEN {}
BEGIN
Writeln ('END DISPLAY_OUTPUT');
Writeln (NET.OUT_F,'END DISPLAY_OUTPUT');
END;
END; {}
FUNCTION BP_error_measure (Output_vector_ptr : DVE_ptr_;
Training_vector_ptr : DVE_ptr_;
net : BP_net_): REAL;
BEGIN
END; {BP_error_measure}
{---------------------------------------------------------------------}
PROCEDURE BP_set_net_defaults (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
BEGIN {}
WITH net DO BEGIN {}
vs := NIL;
ve := NIL;
ws := NIL;
vi := NIL;
vos := NIL;
voe := NIL;
vts := NIL;
vte := NIL;
maxerr := 0.2;
errtol := 0.1;
learning_rate := 0.5;
alpha := 0.9; {factor for momentum term}
n_input := 1;
n_hidden := 1;
n_output := 1;
136
{ data_fname := 'BP.DAT';}
training_iterations := 1;
out_fname := 'BP.OUT';
wt_fname := '';
END; {}
END; {}
PROCEDURE BP_get_net_params_from_user (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
BEGIN {}
Write ('File to output run data to : ');
Readln (net.out_fname);
Assign (net.out_f,net.out_fname);
Rewrite (net.out_f);
{learning rate}
Write ('Learning rate? : ');
Readln (net.learning_rate);
Writeln (NET.OUT_F,'Learning rate : ',net.learning_rate:5:3);
Writeln ('Learning rate : ',net.learning_rate:5:3);
{momentum factor}
Write ('Momentum factor? : ');
Readln (net.alpha);
Writeln (NET.OUT_F,'Momentum factor : ',net.alpha:5:3);
Writeln ('Momentum factor : ',net.alpha:5:3);
{# of input nodes?} {BP_INPUT_NODES_NUM}
Write ('Number of input nodes? : ');
Readln (net.n_input);
Writeln (NET.OUT_F,'# of input nodes : ',net.n_input:3);
Writeln ('# of input nodes : ',net.n_input:3);
{# of hidden nodes in vector}
Write ('Number of hidden nodes? : ');
Readln (net.n_hidden);
Writeln (NET.OUT_F,'# of hidden nodes : ',net.n_hidden:3);
Writeln ('# of hidden nodes : ',net.n_hidden:3);
{# of output nodes}
Write ('Number of output nodes? : ');
Readln (net.n_output);
Writeln (NET.OUT_F,'# of output nodes : ',net.n_output:3);
Writeln ('# of output nodes : ',net.n_output:3);
{error tolerance}
Write ('Error tolerance? : ');
Readln (net.errtol);
Writeln (NET.OUT_F,'Error tolerance : ',net.errtol:5:3);
Writeln ('Error tolerance : ',net.errtol:5:3);
Writeln;
137
Write ('Name of data file : ');
Readln (net.data_fname);
Writeln (NET.OUT_F,'Data file : ',net.data_fname:15);
Writeln ('Data file : ',net.data_fname:15);
Write ('Number of training runs? : ');
Readln (net.training_iterations);
Writeln (NET.OUT_F,'# of iterations : ',
net.training_iterations:3);
Writeln ('# of iterations : ',net.training_iterations:3);
Writeln;
END; {}
PROCEDURE BP_set_net_params_from_file (VAR net : BP_net_;
VAR Fname : STRING);
{
Use the following format for data entries in this section:
!<option letter><space character><data item><CR>
where the !<option letter> combinations are as follows:
!L set learning_rate
!A set alpha factor for momentum term
!I set n_input
!H set n_hidden
!O set n_output
!T set training_iterations
!E set error tolerance
!D set data_fname
!R set out_fname
!W set wt_fname
!Z mark end of parameter data
}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
inch : CHAR;
done : BOOLEAN;
BEGIN {}
Fname := FSEARCH(Fname,GETENV('PATH'));
IF Length(Fname) = 0 THEN BEGIN
Writeln('**ERROR** File does not exist');
EXIT;
END;
Assign(net.data_f,Fname);
138
Reset(net.data_f);
done := FALSE;
REPEAT {get net params}
REPEAT {find command}
Read (net.data_f,inch);
UNTIL (inch = '!'); {}
Read (net.data_f,inch);
CASE UpCase(inch) OF
'L' : BEGIN {get learning rate}
Read (net.data_f, inch);
Read (net.data_f, net.learning_rate);
END; {}
'A' : BEGIN {get alpha}
Read (net.data_f, inch);
Read (net.data_f, net.alpha);
END; {}
'I' : BEGIN {get # inputs}
Read (net.data_f, inch);
Read (net.data_f, net.n_input);
END; {}
'E' : BEGIN {get error tolerance}
Read (net.data_f, inch);
Read (net.data_f, net.errtol);
END; {}
'H' : BEGIN {get # hidden units}
Read (net.data_f, inch);
Read (net.data_f, net.n_hidden);
END; {}
'O' : BEGIN {get # output units}
Read (net.data_f, inch);
Read (net.data_f, net.n_output);
END; {}
'T' : BEGIN {get # of training iterations}
Read (net.data_f, inch);
Read (net.data_f, net.training_iterations);
END; {}
'D' : BEGIN {get datafile name}
Read (net.data_f, inch);
Readln (net.data_f, net.data_fname);
END; {}
'R' : BEGIN {get outfile name}
Read (net.data_f, inch);
Readln (net.data_f, net.out_fname);
END; {}
'W' : BEGIN {get weight file name}
Read (net.data_f, inch);
Readln (net.data_f, net.wt_fname);
END; {}
'Z' : DONE := TRUE;
ELSE
139
BEGIN
DONE := TRUE;
END;
END;
UNTIL (done OR Eof (net.data_f));
{}
Close (net.data_f);
END; {BP_set_net_params_from_file}
{$V-}
PROCEDURE Setup_BP_net (VAR BPN : BP_net_;
VAR Fname : STRING);
{Get data values from a text file to set up basic BP constants, sizes,
and other necessary information, or query user if filename is not
valid.}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
s : STRING;
BEGIN {Setup_BP_net}
BP_set_net_defaults (BPN);
s := FSEARCH(Fname,GETENV('PATH'));
IF (s = '') THEN {}
BEGIN
BP_get_net_params_from_user (BPN);
END
ELSE {}
BEGIN
BP_set_net_params_from_file (BPN,s);
Assign(BPN.out_f,BPN.out_fname);
Rewrite(BPN.out_f);
END;
s := FSEARCH(BPN.data_fname,GETENV('PATH'));
IF (s = '') THEN {}
BEGIN
Assign(BPN.data_f,BPN.data_fname);
Rewrite(BPN.data_f);
Writeln(BPN.data_f);
Close(BPN.data_f);
Reset(BPN.data_f);
END
ELSE {}
BEGIN
140
Assign(BPN.data_f,s);
Reset(BPN.data_f);
END;
Randomize;
Allocate_IO_vectors (BPN);
Allocate_node_vector (BPN);
Allocate_weight_matrix (BPN);
Link_weights_to_nodes (BPN);
set_connectivity (BPN);
{ display_weights (BPN);}
END; {Setup_BP_net}
{$V+}
PROCEDURE set_input_vector_from_file (VAR BPN : BP_net_);
{Get data values from a text file to fill input vector.}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii : INTEGER;
inch : CHAR;
vp : DVE_ptr_;
nptr : DVE_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('SET_INPUT_FROM_FILE');
Writeln (BPN.OUT_F,'SET_INPUT_FROM_FILE');
END;
IF Eof(BPN.data_f) THEN BEGIN
Close(BPN.data_f);
Reset(BPN.data_f);
END;
{find beginning of input line}
REPEAT {}
Read (BPN.data_f,inch);
IF (inch = '!') THEN {skip over net param commands}
BEGIN
141
Read (BPN.data_f,inch);
Read (BPN.data_f,inch);
END;
UNTIL (UpCase(inch) = 'I') OR (Eof(BPN.data_f));
{}
vp := BPN.vi;
IF NOT Eof(BPN.data_f) AND (vnp_(vp^.dptr) <> NIL) THEN BEGIN
FOR ii := 1 TO BPN.n_input DO BEGIN
{}
Read (BPN.data_f,vnp_(vp^.dptr)^.v);
vp := vp^.right;
END;
END;
IF (debug) THEN {}
BEGIN
Writeln ('END SET_INPUT_FROM_FILE');
Writeln (BPN.OUT_F,'END SET_INPUT_FROM_FILE');
END;
END; {}
PROCEDURE set_training_vector_from_file (VAR BPN : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii : INTEGER;
inch : CHAR;
vp : DVE_ptr_;
nptr : DVE_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('SET_TRAINING_V_FROM_FILE');
Writeln (BPN.OUT_F,'SET_TRAINING_V_FROM_FILE');
END;
IF Eof(BPN.data_f) THEN BEGIN
Close(BPN.data_f);
Reset(BPN.data_f);
END;
{find beginning of input line}
REPEAT {}
Read (BPN.data_f,inch);
IF (inch = '!') THEN {skip over net param commands}
BEGIN
Read (BPN.data_f,inch);
142
Read (BPN.data_f,inch);
END;
UNTIL (UpCase(inch) = 'T') OR (Eof(BPN.data_f));
{}
vp := BPN.vts;
IF NOT Eof (BPN.data_f) THEN
FOR ii := 1 TO BPN.n_output DO BEGIN
{}
Read (BPN.data_f,vnp_(vp^.dptr)^.v);
vp := vp^.right;
END;
IF (debug) THEN {}
BEGIN
Writeln ('END SET_TRAINING_V_FROM_FILE');
Writeln (BPN.OUT_F,'END SET_TRAINING_V_FROM_FILE');
END;
END; {}
PROCEDURE Back_propagate (VAR net : BP_net_);
{}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, n : INTEGER;
rr, ss : REAL;
tr, ts, tt, tu, tv : REAL;
npa, npd : DVE_ptr_; {node ptr across, down}
cw : DVE_ptr_; {current weight}
out, trn : DVE_ptr_;
inch : CHAR;
BEGIN {}
IF debug THEN BEGIN
Writeln ('BACK_PROPAGATE');
Writeln (NET.OUT_F,'BACK_PROPAGATE');
END;
n := net.n_input + net.n_hidden + net.n_output;
{calculate deltas for the nodes}
npa := net.ve; {node ptr across}
npd := net.vs; {node ptr down}
cw := bpnp_(net.ve^.dptr)^.bw;
{current weight ptr}
trn := net.vte; {training vector ptr}
out := net.voe; {output vector ptr}
FOR ii := 1 TO N DO BEGIN {}
143
cw := bpnp_(npa^.dptr)^.bw;
CASE bpnp_(npa^.dptr)^.nt OF
{node pointer across type}
Input, hidden : BEGIN {need to assign delta}
{set rr to current node output}
rr := sigmoid (bpnp_(npa^.dptr)^.range, 1,
bpnp_(npa^.dptr)^.base, (bpnp_(npa^.dptr)^.ni
+ bpnp_(npa^.dptr)^.theta));
{find error term from forward connections}
ss := 0;
npd := net.vs;
cw := bpnp_(npa^.dptr)^.bw;
FOR jj := 1 TO N DO BEGIN
{}
IF (wnp_(cw^.dptr)^.connect) THEN BEGIN
ss := ss + (bpnp_(npd^.dptr)^.delta
* wnp_(cw^.dptr)^.w);
END;
IF (jj < n) THEN BEGIN
npd := npd^.right;
cw := cw^.down;
END;
END; {}
IF DEBUG THEN
Write ('Delta node ',bpnp_(npa^.dptr)^.loc,' = ',
rr:6,' * (1 - ',rr:6,') * ',ss:6);
bpnp_(npa^.dptr)^.delta := rr * (1 - rr) * ss;
END;
Output : BEGIN {}
rr := sigmoid (bpnp_(npa^.dptr)^.range,1,
bpnp_(npa^.dptr)^.base,
(bpnp_(npa^.dptr)^.ni + bpnp_(npa^.dptr)^.theta));
IF DEBUG THEN
Write ('Delta node ',bpnp_(npa^.dptr)^.loc,' = (',
vnp_(trn^.dptr)^.v:6,' - ',vnp_(out^.dptr)^.v:6,
') * ', rr:6,' * (1 - ',rr:6,') = ');
IF (trn <> NIL) AND (out <> NIL) THEN BEGIN
bpnp_(npa^.dptr)^.delta := (vnp_(trn^.dptr)^.v
- vnp_(out^.dptr)^.v) * rr * (1 - rr);
rr := ABS (vnp_(trn^.dptr)^.v - vnp_(out^.dptr)^.v);
IF (net.maxerr < rr) THEN BEGIN
net.maxerr := rr;
END;
END
ELSE BEGIN
Writeln ('NIL pointer to train or output');
Halt;
END;
IF DEBUG THEN Writeln (bpnp_(NPA^.dptr)^.DELTA:6);
trn := trn^.left;
out := out^.left;
144
END; {CASE output term}
END; {CASE}
npa := npa^.left;
cw := bpnp_(npa^.dptr)^.bw;
END; {FOR ii}
{now calculate weight changes for weights and update}
npa := net.ve; {node ptr across}
npd := net.vs; {node ptr down}
cw := bpnp_(npa^.dptr)^.bw; {current weight ptr}
trn := net.vte; {training vector ptr}
out := net.voe; {output vector ptr}
FOR ii := 1 TO N DO BEGIN {}
npd := net.vs;
cw := bpnp_(npa^.dptr)^.bw;
FOR jj := 1 TO N DO BEGIN{}
IF (wnp_(cw^.dptr)^.connect) THEN
{}
BEGIN
rr := sigmoid(bpnp_(npa^.dptr)^.range,1,
bpnp_(npa^.dptr)^.base,
(bpnp_(npa^.dptr)^.ni + bpnp_(npa^.dptr)^.theta));
wnp_(cw^.dptr)^.dw := wnp_(cw^.dptr)^.dw
+ (net.learning_rate *
(bpnp_(npd^.dptr)^.delta * rr));
{error * activation}
END;
npd := npd^.right;
cw := cw^.down;
END; {}
IF DEBUG THEN
Writeln (bpnp_(npa^.dptr)^.delta:7,' ',bpnp_(npa^.dptr)
^.theta:7);
{From Simpson, II, p 73}
bpnp_(npa^.dptr)^.dtheta := bpnp_(npa^.dptr)^.dtheta + net.
learning_rate * bpnp_(npa^.dptr)^.delta;
{bpnp_(npa^.dptr)^.theta :=
bpnp_(npa^.dptr)^.theta +
bpnp_(npa^.dptr)^.dtheta;}
npa := npa^.left;
cw := bpnp_(npa^.dptr)^.bw;
END; {}
IF (debug) THEN {}
BEGIN
Writeln ('END BACK_PROPAGATE');
Writeln (NET.OUT_F,'END BACK_PROPAGATE');
END;
END; {}
PROCEDURE BP_Feed_forward (VAR BPN : BP_net_);
145
{Present values to network and propagate values forward, set the output
vector.}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj, n : INTEGER;
rr, ss : REAL;
dp1, dp2, dp3 : DVE_ptr_;
npa, npd : DVE_ptr_;
np1 : BP_node_ptr_; {node ptr across, down}
cw : DVE_ptr_; {current weight}
vin, vout : DVE_ptr_;
BEGIN {}
IF debug THEN BEGIN
Writeln ('FEED_FORWARD');
Writeln (BPN.OUT_F,'FEED_FORWARD');
END;
{npd is the pointer to the node to change}
{npa points to the node that the current weight may modify the
output of}
n := BPN.n_input + BPN.n_hidden + BPN.n_output;
IF BPN.vs <> NIL THEN BEGIN
npa := BPN.vs;
npd := BPN.vs;
END
ELSE BEGIN
Writeln ('ERROR -- NIL pointer encountered #1');
Halt;
END;
IF (BP_node_ptr_(npd^.dptr)^.fw <> NIL) AND (BPN.VI <> NIL)
AND (BPN.vos <> NIL) THEN BEGIN
cw := BP_node_ptr_(npd^.dptr)^.fw;
vin := BPN.vi;
vout := BPN.vos;
END
ELSE BEGIN
Writeln ('ERROR -- NIL pointer encountered #2');
Halt;
END;
FOR ii := 1 TO N DO BEGIN {}
BP_node_ptr_(npd^.dptr)^.ni := 0;
IF (BP_node_ptr_(npd^.dptr)^.nt = Input) THEN
{}
BEGIN
BP_node_ptr_(npd^.dptr)^.ni := VNP_(vin^.dptr)^.v;
146
IF debug THEN Writeln ('I_node ',
BP_node_ptr_(npd^.dptr)^.loc,
' = ',VNP_(vin^.dptr)^.v:4);
vin := vin^.right;
END; {IF input}
npa := BPN.vs;
cw := BP_node_ptr_(npd^.dptr)^.fw;
FOR jj := 1 TO N DO BEGIN{}
IF (WNP_(cw^.dptr)^.connect) THEN
{}
BEGIN
IF (BP_node_ptr_(npa^.dptr)^.nt = Input) THEN
{}
BEGIN
IF DEBUG THEN
Write ('Node ',BP_node_ptr_(npd^.dptr)^.loc,
' = ', BP_node_ptr_(npd^.dptr)^.ni:4,' + ',
BP_node_ptr_(npa^.dptr)^.ni:4,' * ',
WNP_(cw^.dptr)^.w:4,' = ');
rr := BP_node_ptr_(npa^.dptr)^.ni
* WNP_(cw^.dptr)^.w;
BP_node_ptr_(npd^.dptr)^.ni :=
BP_node_ptr_(npd^.dptr)^.ni + rr;
IF DEBUG THEN Writeln (rr);
END
ELSE BEGIN
rr := (sigmoid(BP_node_ptr_(npa^.dptr)^.range,1,
BP_node_ptr_(npa^.dptr)^.base,
(BP_node_ptr_(npa^.dptr)^.ni
+ BP_node_ptr_(npa^.dptr)^.theta))
* WNP_(cw^.dptr)^.w);
IF DEBUG THEN
Writeln ('Node ',BP_node_ptr_(npd^.dptr)^.loc,
' = ', BP_node_ptr_(npd^.dptr)^.ni:4,' + ',rr:4);
BP_node_ptr_(npd^.dptr)^.ni :=
BP_node_ptr_(npd^.dptr)^.ni + rr;
END; {IF}
END; {IF cinnected}
IF (cw^.right <> NIL) AND (jj < n) THEN BEGIN
cw := cw^.right;
npa := npa^.right;
END
ELSE IF (cw^.right = NIL) AND (jj < n) THEN BEGIN
Writeln ('ERROR -- NIL pointer encountered #3 jj=',jj);
Halt;
END;
END; {FOR jj = 1 to N }
IF DEBUG THEN
Writeln ('Node net in : ',BP_node_ptr_(npd^.dptr)^.ni:6);
IF (BP_node_ptr_(npd^.dptr)^.nt = Output) THEN BEGIN
VNP_(vout^.dptr)^.v :=
147
sigmoid(BP_node_ptr_(npd^.dptr)^.range,1,
BP_node_ptr_(npd^.dptr)^.base,
(BP_node_ptr_(npd^.dptr)^.ni
+ BP_node_ptr_(npd^.dptr)^.theta));
vout := vout^.right;
END;
IF (npd^.right <> NIL)
AND (BP_node_ptr_(npd^.dptr)^.fw <> NIL)
AND (ii < N) THEN BEGIN
npd := npd^.right;
cw := BP_node_ptr_(npd^.dptr)^.fw;
END
ELSE IF ((npd^.right = NIL)
OR (BP_node_ptr_(npd^.dptr)^.fw = NIL))
AND (ii < n) THEN BEGIN
Writeln ('ERROR -- NIL pointer encountered #4');
IF (npd^.right = NIL) THEN Write ('NPD^.RIGHT is NIL ');
IF (BP_node_ptr_(npd^.dptr)^.fw = NIL) THEN Write (
'NPD^.FW is NIL ');
Writeln;
Halt;
END;
END; {}
IF (debug) THEN {}
BEGIN
Writeln ('END FEED_FORWARD');
Writeln (BPN.OUT_F,'END FEED_FORWARD');
END;
END; {}
PROCEDURE BP_train_presentation (VAR BPN : BP_net_);
{Present values to network, propagate forward, set output, compare
output to training, back-propagate, collect statistics but do not
change weights.}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
BEGIN {BP_train_presentation}
BP_feed_forward(BPN);
back_propagate(BPN);
END; {BP_train_presentation}
PROCEDURE BP_change (VAR BPN : BP_net_);
{Change weights using current statistics and reset statistics.}
{CONST}
TYPE
148
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii, jj : WORD;
node_total : WORD;
dp1, dp2, wp1, wp2 : DVE_ptr_;
bpnp1, bpnp2 :BP_node_ptr_;
wnp1, wnp2 : weight_node_ptr_;
BEGIN {BP_change}
node_total := BPN.n_input + BPN.n_hidden + BPN.n_output;
{Change thetas} {set pointer to first node}
dp1 := BPN.vs;
FOR ii := 1 TO node_total DO BEGIN
{set pointer to node}
bpnp1 := dp1^.dptr;
IF (bpnp1 <> NIL) THEN BEGIN
{set new theta value}
bpnp1^.ldtheta := bpnp1^.dtheta + BPN.alpha
* bpnp1^.ldtheta;
bpnp1^.theta := bpnp1^.theta + bpnp1^.ldtheta;
END
ELSE BEGIN
IF bpnp1 = NIL THEN Writeln (
'**Error** Nil ptr encountered: BP_unit.BP_change');
END;
IF bpnp1 <> NIL THEN BEGIN
{ bpnp1^.ldtheta := bpnp1^.dtheta;}
bpnp1^.dtheta := 0;
END; {set pointer to next node}
dp1 := dp1^.right;
END; {for ii}
{For each weight, do change, reset stats}
dp1 := BPN.ws;
FOR ii := 1 TO node_total DO BEGIN
dp2 := dp1;
FOR jj := 1 TO node_total DO BEGIN
IF (dp2 <> NIL) THEN BEGIN
wp1 := dp2^.dptr;
wnp_(wp1)^.ldw := wnp_(wp1)^.dw
+ BPN.alpha * wnp_(wp1)^.ldw;
wnp_(wp1)^.w := wnp_(wp1)^.w + wnp_(wp1)^.ldw;
wnp_(wp1)^.dw := 0;
dp2 := dp2^.right;
END;
END; {for jj}
dp1 := dp1^.down;
END; {for ii}
149
END; {BP_change}
PROCEDURE BP_train_and_change (VAR BPN : BP_net_);
{
Present values to network, propagate forward, set output, compare output
to training, back-propagate, collect statistics, change weights, and
reset statistic variables.
}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
BEGIN {BP_train_and_change}
BP_train_presentation(BPN);
BP_change(BPN);
END; {BP_train_and_change}
PROCEDURE BP_dump_net (VAR BPN : BP_net_);
{Dump net parameters, node activities, and weights for inspection.}
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
BEGIN {}
END; {}
PROCEDURE BP_driver (VAR net : BP_net_);
TYPE
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
VAR
ii : INTEGER;
nptr : DVE_ptr_;
BEGIN {}
{first, check for parameter}
IF (ParamCount > 0) THEN {}
BEGIN
net.data_fname := ParamStr(1);
Writeln ;
Writeln ('BACK PROPAGATION SIMULATION');
150
Writeln ;
DEBUG := FALSE;
{open data file}
Assign (net.data_f,net.data_fname);
Reset (net.data_f);
BP_set_net_params_from_file (net,net.data_fname);
{close data file}
Close (net.data_f);
END
ELSE {}
BEGIN
Writeln ;
Writeln ('BACK PROPAGATION SIMULATION');
Writeln ;
BP_get_net_params_from_user (net);
Write ('DEBUG ON OR OFF? (1 or 0) : ');
Readln (ii);
CASE ii OF
0 : debug := FALSE;
1 : debug := TRUE;
ELSE
debug := FALSE;
END;
END;
Assign (net.out_f,net.out_fname);
Rewrite (net.out_f);
Writeln (net.out_f,'DEBUG : ',debug);
Writeln ('DEBUG : ',debug);
{allocate input node vector}
{BP_INPUT_NODES_VECTOR}
{allocate hidden node vector}
{allocate output node vector}
{get scale for output node}
{allocate weight matrix}
{seed with random values}
{# to times to repeat data}
Randomize;
Allocate_IO_vectors (net);
Allocate_node_vector (net);
Allocate_weight_matrix (net);
Link_weights_to_nodes (net);
set_connectivity (net);
151
display_weights (net);
net.maxerr := net.errtol + 1.0;
ii := 1;
WHILE (net.maxerr > net.errtol)
AND (ii <= net.training_iterations) DO BEGIN
IF ((ii MOD 50) = 1) THEN BEGIN
Writeln (net.out_f,'At iteration ',ii);
Writeln ('At iteration ',ii);
END;
Assign (net.data_f,net.data_fname);
Reset (net.data_f);
net.maxerr := 0.0;
WHILE (NOT Eof (net.data_f)) DO BEGIN
{}
IF debug THEN Writeln ('In data loop ');
set_input_vector_from_file (net);
IF (NOT Eof(NET.DATA_F)) THEN BEGIN
IF ((ii MOD 50) = 1) THEN BEGIN
Writeln (NET.OUT_F);
Writeln ;
Write (net.out_f,'Input : ');
Write ('Input : ');
display_vector (net.vi,net.n_input,net);
END;
set_training_vector_from_file (net);
BP_feed_forward (net);
IF ((ii MOD 50) = 1) THEN BEGIN
Write (net.out_f,'Output : ');
Write ('Output : ');
display_vector (net.vos,net.n_output,net);
Write (net.out_f,'Expected : ');
Write ('Expected : ');
display_vector (net.vts,net.n_output,net);
END;
back_propagate (net);
END;
END; {}
Close (net.data_f);
ii := ii + 1;
END; {while ii and out of tolerance}
Writeln (net.out_f,'Iterations = ',ii);
Writeln ;
Writeln (net.out_f,'Weights after training : ');
Writeln ('Weights after training : ');
display_weights (net);
Writeln (NET.OUT_F);
Writeln ;
Writeln (NET.OUT_F,'Final values associated with nodes :');
152
Writeln ('Final values associated with nodes :');
nptr := net.vs;
FOR II := 1 TO (net.n_input + net.n_hidden + net.n_output) DO
BEGIN
dump_node (net,bpnp_(nptr^.dptr));
nptr := nptr^.right;
Writeln (net.out_f);
Writeln ;
END;
Flush (NET.OUT_F);
Close (NET.OUT_F);
END; {}
BEGIN {BP_unit}
END. {BP_unit}
APPENDIX 4
PROGRAM SOURCE LISTING: LIST STRUCTURES UNIT
153
154
UNIT struct;
{
This unit provides a general linked-list seet of functions.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
{PUBLIC DECLARATIONS}
TYPE
dve_ptr_ = ^dve_;
dve_ = RECORD {terminology: DVE is diploid vector
element}
{Linkage can be both forward and
backward}
up, down, left, right : dve_ptr_;
{Forward and backward links}
dptr : POINTER; {Untyped pointer}
END;
hve_ptr_ = ^hve_;
hve_ = RECORD {terminology: HVE is haploid vector
element}
{Linkage is forward only}
down, right : hve_ptr_; {Forward links}
dptr : POINTER; {Untyped pointer}
END;
FUNCTION Create_DVE_vector (num_elem : WORD; size : WORD): POINTER;
{This function takes a size for a pointer based structure which is based
155
on the DVE_ type declaration and returns a pointer to a doubly linked
list which has NUM_ELEM number of elements. The calling program should
assign this pointer's value to a pointer of the type which SIZE
describes. The head element's LEFT pointer is NIL, RIGHT points to the
next element or is NIL, UP and DOWN are NIL. Elements beside the head
element have non-NIL LEFT pointers. Calling routines should check for
NIL returned pointers, as this indicates that not enough memory was
available to allocate the vector.}
FUNCTION copy_DVE_vector
(aptr : POINTER; num_elem : WORD; size : WORD): POINTER;
{ This function takes a pointer to a DVE vector and creates a copy, then
passes back a pointer to the copy vector.}
FUNCTION Create_HVE_vector
(num_elem : WORD; size : WORD; rt : BOOLEAN): POINTER;
{This function creates a linked list of HVE_ type, with space allocated
based on size of a HVE_ based structure definition. If RT is TRUE, then
the function allocates a vector using the RIGHT linkages, and DOWN
pointers are set to NIL. Otherwise, the obverse is true. A NIL pointer
returned as a result indicates that not enough memory was available for
allocation.}
FUNCTION Create_matrix (n_across, n_down : WORD; size : WORD): POINTER;
{This function creates a doubly linked matrix of DVE vector elements,
using the provided size.}
FUNCTION Find_element_DVE (num : WORD; pntr : POINTER): POINTER;
FUNCTION Find_element_HVE (num : WORD; pntr : POINTER): POINTER;
FUNCTION Find_element_matrix (n1, n2 : WORD; pntr : POINTER): POINTER;
IMPLEMENTATION
{PRIVATE DECLARATIONS}
{IMPLEMENTATIONS OF PROCEDURES AND FUNCTIONS}
FUNCTION Create_DVE_vector (num_elem : WORD;
size : WORD): POINTER;
{This function takes a size for a pointer based structure which is based
on the DVE_ type declaration and returns a pointer to a doubly linked
list which has NUM_ELEM number of elements. The calling program should
assign this pointer's value to a pointer of the type which SIZE
describes. The head element's LEFT pointer is NIL, RIGHT points to the
next element or is NIL, UP and DOWN are NIL. Elements beside the head
element have non-NIL LEFT pointers. Calling routines should check for
NIL returned pointers, as this indicates that not enough memory was
available to allocate the vector.}
156
VAR
ii : INTEGER;
TempStart, Temp : dve_ptr_;
BEGIN
GetMem (TempStart,SizeOf(TempStart^));
Temp := TempStart;
Temp^.left := NIL;
Temp^.up := NIL;
Temp^.down := NIL;
GetMem(Temp^.dptr,size);;
FOR II := 2 TO num_elem DO BEGIN
GetMem (Temp^.right,SizeOf(TempStart^));
Temp^.right^.left := Temp;
Temp := Temp^.right;
Temp^.up := NIL;
Temp^.down := NIL;
GetMem(Temp^.dptr,size);
END; {for}
Temp^.right := NIL;
Create_DVE_vector := TempStart;
END;
FUNCTION copy_DVE_vector (aptr : POINTER;
num_elem : WORD;
size : WORD): POINTER;
{ This function takes a pointer to a DVE vector and creates a copy, then
passes back a pointer to the copy vector.}
VAR
tptr, bptr, t2ptr : dve_ptr_;
ii, jj : WORD;
BEGIN
bptr := create_DVE_vector (num_elem,size);
t2ptr := bptr;
Tptr := aptr;
ii := 1;
WHILE (Tptr <> NIL) AND (ii < num_elem) DO BEGIN
Move (bptr^.dptr^,tptr^.dptr^,size);
Tptr := Tptr^.right;
bptr := bptr^.right;
INC(ii);
END; {while}
copy_DVE_vector := t2ptr;
END;
FUNCTION Create_HVE_vector (num_elem : WORD;
size : WORD;
rt : BOOLEAN): POINTER;
{This function creates a linked list of HVE_ type, with space allocated
157
based on size of a HVE_ based structure definition and on the size of the
user structure pointed to within each vector element. If RT is TRUE,
then the function allocates a vector using the RIGHT linkages, and DOWN
pointers are set to NIL. Otherwise, the obverse is true. A NIL pointer
returned as a result indicates that not enough memory was available for
allocation.}
VAR
ii : INTEGER;
TempStart, Temp : HVE_ptr_;
BEGIN
IF rt THEN BEGIN
GetMem (TempStart,SizeOf(TempStart^));
Temp := TempStart;
Temp^.down := NIL;
GetMem(Temp^.dptr,size);;
FOR II := 2 TO num_elem DO BEGIN
GetMem (Temp^.right,SizeOf(TempStart^));
Temp := Temp^.right;
Temp^.down := NIL;
GetMem(Temp^.dptr,size);
END; {for}
Temp^.right := NIL;
END {if}
ELSE BEGIN
GetMem (TempStart,SizeOf(TempStart^));
Temp := TempStart;
Temp^.right := NIL;
GetMem(Temp^.dptr,size);;
FOR II := 2 TO num_elem DO BEGIN
GetMem (Temp^.down,SizeOf(TempStart^));
Temp := Temp^.down;
Temp^.right := NIL;
GetMem(Temp^.dptr,size);
END; {for}
Temp^.down := NIL;
END; {else}
Create_HVE_vector := TempStart;
END;
FUNCTION Create_matrix (n_across, n_down : WORD;
size : WORD): POINTER;
{This function creates a doubly linked matrix of DVE vector elements,
using the provided size for allocation of user space.}
VAR
ii, jj : WORD;
StartPtr, T1Ptr, T2Ptr : DVE_ptr_;
PROCEDURE Link_DVE_vectors (v1, v2 : DVE_ptr_);
158
BEGIN
WHILE (v1 <> NIL) AND (v2 <> NIL) DO BEGIN
v1^.down := v2;
v2^.up := v1; {Next elements}
v1 := v1^.right;
v2 := v2^.right;
END; {while}
END; {Link_DVE_vectors}
BEGIN
StartPtr := Create_DVE_vector (n_across, size);
T1Ptr := StartPtr;
FOR jj := 2 TO n_down DO BEGIN
T2Ptr := Create_DVE_vector (n_across, size);
Link_DVE_vectors (T1Ptr, T2Ptr);
T1Ptr := T2Ptr;
END; {for}
Create_matrix := StartPtr;
END; {Create_matrix}
FUNCTION Find_element_DVE (num : WORD;
pntr : POINTER): POINTER;
VAR
ii : WORD;
T1 : DVE_ptr_;
BEGIN
T1 := pntr;
ii := 1;
WHILE (T1 <> NIL) AND (ii < num) DO BEGIN
T1 := T1^.right;
INC(ii);
END; {while}
Find_element_DVE := T1;
END; {}
FUNCTION Find_element_HVE (num : WORD;
pntr : POINTER): POINTER;
VAR
ii : WORD;
T1 : HVE_ptr_;
BEGIN
T1 := pntr;
ii := 1;
WHILE (T1 <> NIL) AND (ii < num) DO BEGIN
T1 := T1^.right;
INC(ii);
159
END; {while}
Find_element_HVE := T1;
END; {}
FUNCTION Find_element_matrix (n1, n2 : WORD;
pntr : POINTER): POINTER;
VAR
ii, jj : WORD;
T1 : DVE_ptr_;
BEGIN
T1 := pntr;
ii := 1;
jj := 1;
WHILE (T1 <> NIL) AND (jj < n2) DO BEGIN
T1 := T1^.down;
jj := jj + 1;
END; {while}
WHILE (T1 <> NIL) AND (ii < n1) DO BEGIN
T1 := T1^.right;
ii := ii + 1;
END; {while}
Find_element_matrix := T1;
END; {}
BEGIN {Initialization}
END.
APPENDIX 5
PROGRAM SOURCE LISTING: MISCELLANEOUS PROCEDURES UNIT
160
161
UNIT misc1;
{
This unit provides a number of functions of general utility.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
USES DOS;
CONST
ASCII_NUL = #0;
ASCII_SOH = #1;
ASCII_STX = #2;
ASCII_ETX = #3;
ASCII_EOT = #4;
ASCII_ENQ = #5;
ASCII_ACK = #6;
ASCII_BEL = #7;
ASCII_BS = #8;
ASCII_HT = #9;
ASCII_LF = #10;
ASCII_VT = #11;
ASCII_FF = #12;
ASCII_CR = #13;
ASCII_SO = #14;
ASCII_SI = #15;
ASCII_DLE = #16;
ASCII_DC1 = #17;
ASCII_XON = #17;
ASCII_DC2 = #18;
ASCII_DC3 = #19;
162
ASCII_XOFF = #19;
ASCII_DC4 = #20;
ASCII_NAK = #21;
ASCII_SYN = #22;
ASCII_ETB = #23;
ASCII_CAN = #24;
ASCII_EM = #25;
ASCII_SUB = #26;
ASCII_EOF = #26;
ASCII_ESC = #27;
ASCII_FS = #28;
ASCII_GS = #29;
ASCII_RS = #30;
ASCII_US = #31;
ASCII_SP = #32;
ASCII_EXCL = #33;
ASCII_DQUOTE = #34;
ASCII_POUND = #35;
ASCII_DOLLAR = #36;
ASCII_PERCENT = #37;
ASCII_AMPERSAND = #38;
ASCII_SQUOTE = #39;
ASCII_OPAREN = #40;
ASCII_CPAREN = #41;
ASCII_ASTERISK = #42;
ASCII_PLUS = #43;
ASCII_COMMA = #44;
ASCII_DASH = #45;
ASCII_PERIOD = #46;
ASCII_SLASH = #47;
ASCII_ZERO = #48;
ASCII_ONE = #49;
ASCII_TWO = #50;
ASCII_THREE = #51;
ASCII_FOUR = #52;
ASCII_FIVE = #53;
ASCII_SIX = #54;
ASCII_SEVEN = #55;
ASCII_EIGHT = #56;
ASCII_NINE = #57;
ASCII_COLON = #58;
ASCII_SEMICOLON = #59;
ASCII_LESSTHAN = #60;
ASCII_EQUAL = #61;
ASCII_GREATERTHAN = #62;
ASCII_QMARK = #63;
ASCII_AT = #64;
ASCII_OBRACKET = #91;
ASCII_BACKSLASH = #92;
ASCII_CBRACKET = #93;
ASCII_CARAT = #94;
163
ASCII_UNDERLINE = #95;
ASCII_BACKQUOTE = #96;
ASCII_OBRACE = #123;
ASCII_VLINE = #124;
ASCII_CBRACE = #125;
ASCII_TILDE = #126;
ASCII_DEL = #127;
TYPE
Time_rec_ = RECORD
h,m,s,f : INTEGER;
END;
PROCEDURE Time(VAR TR : Time_rec_);
{Gets system time from MS-DOS}
PROCEDURE Elapsed_time(VAR TR1, TR2 : Time_rec_);
{Computes the difference between TR1 and TR2, returns result in TR1.
TR1's previous value is destroyed.}
FUNCTION Convert_time_to_real(VAR CTR : Time_rec_):REAL;
{}
PROCEDURE Convert_real_to_time(VAR RT : REAL; VAR CTR : Time_rec_);
{}
PROCEDURE Trim(VAR alex : STRING;tchar : CHAR);
{ This procedure trims a string variable of type STRING beginning
PROCEDURE StrUp(VAR strng : STRING);
{ This procedure maps the characters of a string of type STRING to
uppercase}
FUNCTION IsUpper(x : CHAR):BOOLEAN;
{Returns true if x is an uppercase letter}
FUNCTION IsLower(x : CHAR):BOOLEAN;
{Returns true if x is a lowercase letter}
PROCEDURE Error(msg : STRING);
{ writes error message out to screen}
FUNCTION Gaussian(x,mu,sigma : REAL):REAL;
{returns the gaussian density function of x, where mu is the}
FUNCTION Normal_Prob(x,mu,sigma : REAL):REAL;
{uses a polynomial approximation to estimate
the area under the normal curve}
164
FUNCTION Power(num,expon : REAL):REAL;
{returns num^expon}
FUNCTION Slope(sumx,sumy,sumxy,sumx2,n :REAL):REAL;
{returns linear regression determined slope of line}
FUNCTION Intercept(sumx,sumy,n,m : REAL):REAL;
{returns linear regression determined intercept of line}
FUNCTION CorrCo(m,sigmax,sigmay : REAL):REAL;
{returns correlation coefficient of x and y}
FUNCTION SD(sum,sum_sqrd,n : REAL):REAL;
{returns standard deviation given the sum of values, the sum of
the squares of values, and the number of values}
FUNCTION Map_Real(mapval, domain_min, domain_max,
range_min, range_max : REAL): REAL;
{ this functions maps the value passed to it into a new range }
FUNCTION Map_Int(mapval, domain_min, domain_max,
range_min, range_max : INTEGER): INTEGER;
{ this functions maps the value passed to it into a new range }
{ must have MAP_REAL as above in program }
FUNCTION Map_Int_From_Real(mapval, domain_min, domain_max : REAL;
range_min, range_max : INTEGER): INTEGER;
{ this functions maps the value passed to it into a new range of type
integer}
FUNCTION dir_console_IO (VAR ch :CHAR) : BOOLEAN;
{Returns TRUE if a character has been captured at the keyboard, FALSE
otherwise. If a character has been captured, CH contains it.}
FUNCTION check_kbd_status : BOOLEAN;
{Returns TRUE if a key has been pressed, FALSE otherwise}
FUNCTION max_single(s1,s2 : SINGLE):SINGLE;
{Returns the greater of two SINGLE type values}
FUNCTION min_single(s1,s2 : SINGLE):SINGLE;
{Returns the lesser of two SINGLE type values}
IMPLEMENTATION
PROCEDURE Time(VAR TR : Time_rec_);
{Gets system time from MS-DOS}
165
CONST
lllama = 0;
VAR
regs : registers;
BEGIN {Time}
WITH regs DO BEGIN
ax:=$2c00;
MSDos(regs);
TR.h := Hi(cx);
TR.m := Lo(cx);
TR.s := Hi(dx);
TR.f := Lo(dx);
END;
END; {Time}
FUNCTION Convert_time_to_real(VAR CTR : Time_rec_):REAL;
{}
VAR
Tempr : REAL;
BEGIN {Convert_time_to_real}
WITH CTR DO Tempr := f + (s*100.0) + (m*6000.0) + (h*360000.0);
Convert_time_to_real := Tempr;
END; {Convert_time_to_real}
PROCEDURE Convert_real_to_time(VAR RT : REAL;
VAR CTR : Time_rec_);
{}
VAR
TempI : INTEGER;
Tempr1, Tempr2 : REAL;
BEGIN {Convert_real_to_time}
WITH CTR DO BEGIN
Tempr2 := RT;
Tempr1 := INT(Tempr2 / 360000.0);
h := Trunc(Tempr1);
Tempr2 := Tempr2 - (Tempr1 * 360000.0);
Tempr1 := INT(Tempr2 /6000.0);
m := Trunc(Tempr1);
Tempr2 := Tempr2 - (Tempr1 * 6000.0);
Tempr1 := INT(Tempr2 / 100);
s := Trunc(Tempr1);
Tempr2 := Tempr2 - (Tempr1 * 100);
Tempr1 := INT(Tempr2);
f := Trunc(Tempr1);
END;
166
END; {Convert_real_to_time}
PROCEDURE Elapsed_time(VAR TR1, TR2 : Time_rec_);
{Computes the difference between TR1 and TR2, returns result in TR1.
TR1's previous value is destroyed.}
VAR
Dif : TIme_rec_;
T1 , T2 : REAL;
BEGIN {Elapsed_time}
Write('Time difference ',TR2.h:2,ascii_Colon,TR2.m:2,ascii_Colon,
TR2.s:2,ascii_Colon,TR2.f:2, ' - ',TR1.h:2,ascii_Colon,TR1.m:
2,ascii_Colon,TR1.s:2,ascii_Colon,TR1.f:2);
T1 := Convert_time_to_real(TR1);
T2 := Convert_time_to_real(TR2);
IF (T2 < T1) THEN {}
BEGIN
T2 := T2 + 8640000.0;
END
ELSE {}
BEGIN
END;
T1 := T2 - T1;
Convert_real_to_time(T1,TR1);
Writeln(' = ',TR1.h:2,ascii_Colon,TR1.m:2,ascii_Colon,TR1.s:2,
ascii_Colon,TR1.f:2);
END; {Elapsed_time}
{$V-}
PROCEDURE TRIM(VAR alex : STRING;
tchar : CHAR);
{ This procedure trims a string variable of type STRING beginning
with the first occurrence of the character TCHAR}
VAR
ii,jj :INTEGER;
BEGIN
ii := Pos(tchar,alex);
IF ii <> 0 THEN alex := Copy(alex,1,ii-1);
END;
{$V+}
{$V-}
PROCEDURE STRUP(VAR strng : STRING);
{ This procedure maps the characters of a string of type STRING to
167
uppercase}
VAR
ii : INTEGER;
BEGIN
FOR ii := 1 TO Length(strng) DO strng[ii] := UpCase(strng[ii]);
END;
{$V+}
FUNCTION ISUPPER(x : CHAR):BOOLEAN;
{Returns true if x is an uppercase letter}
BEGIN
IF (x IN ['A'..'Z']) THEN isupper := TRUE
ELSE isupper := FALSE;
END;
FUNCTION ISLOWER(x : CHAR):BOOLEAN;
{Returns true if x is a lowercase letter}
BEGIN
IF (x IN ['a'..'z']) THEN islower := TRUE
ELSE islower := FALSE;
END;
{$V-}
PROCEDURE ERROR(msg : STRING);
{ writes error message out to screen}
CONST
bell = ^G;
BEGIN
Write(bell,msg);
END;
{$V+}
FUNCTION GAUSSIAN(x,mu,sigma : REAL):REAL;
{returns the gaussian density function of x, where mu is the
mean and sigma is the standard deviation}
BEGIN
gaussian := (1/(sigma*Sqrt(2*Pi)))*Exp(-Sqr(x-mu)/(2*Sqr(sigma)));
END;
FUNCTION NORMAL_PROB(x,mu,sigma : REAL):REAL;
{uses a polynomial approximation to estimate
168
the area under the normal curve}
CONST
b1 = 0.319381530;
b2 = -0.356563782;
b3 = 1.781477937;
b4 = -1.821255978;
b5 = 1.330274429;
p = 0.2316419;
epsi = 7.5E-09;
VAR
t, t2, t3, t4, t5, q, z : REAL;
BEGIN
z := gaussian(x,mu,sigma) * ((x-mu)/sigma);
t := 1/(1+p*x);
t2 := t*t;
t3 := t2*t;
t4 := t3*t;
t5 := t4*t;
q := z * (b1*t + b2*t2 + b3*t3 + b4*t4 + b5*t5) + epsi;
normal_prob := 1-q;
END;
FUNCTION POWER(num,expon : REAL):REAL;
{returns num^expon}
CONST
Machine_infinity = 1E37;
VAR
temp : REAL;
BEGIN
temp := expon*Ln(num);
IF temp >= Ln(machine_infinity) THEN power := machine_infinity
ELSE power := Exp(temp);
END;
FUNCTION SLOPE(sumx,sumy,sumxy,sumx2,n :REAL):REAL;
{returns linear regression determined slope of line}
BEGIN
slope := (sumxy-(sumx*sumy/n))/ (sumx2-(Sqr(sumx)/n));
END;
FUNCTION INTERCEPT(sumx,sumy,n,m : REAL):REAL;
{returns linear regression determined intercept of line}
BEGIN
169
intercept := ((sumy-(m*sumx))/n);
END;
FUNCTION CORRCO(m,sigmax,sigmay : REAL):REAL;
{returns correlation coefficient of x and y}
BEGIN
corrco := m*sigmax/sigmay;
END;
FUNCTION SD(sum,sum_sqrd,n : REAL):REAL;
{returns standard deviation given the sum of values, the sum of
the squares of values, and the number of values}
BEGIN
sd := Sqrt((sum_sqrd-(Sqr(sum)/n))/(n-1));
END;
FUNCTION MAP_REAL(mapval, domain_min, domain_max, range_min, range_max :
REAL): REAL;
{ this functions maps the value passed to it into a new range }
BEGIN
map_real := (((mapval - domain_min)/(domain_max - domain_min))
* (range_max - range_min)) + range_min;
END;
FUNCTION MAP_INT(mapval, domain_min, domain_max, range_min, range_max :
INTEGER): INTEGER;
{ this functions maps the value passed to it into a new range }
{ must have MAP_REAL as above in program }
VAR
mv, dn, dx, rn, rx : REAL;
BEGIN
mv := mapval;
dn := domain_min;
dx := domain_max;
rn := range_min;
rx := range_max;
map_int := Round(map_real(mv,dn,dx,rn,rx));
END;
FUNCTION MAP_INT_FROM_REAL(mapval, domain_min, domain_max : REAL;
range_min, range_max : INTEGER): INTEGER;
{ this functions maps the value passed to it into a new range of type
integer}
BEGIN
map_int_from_real := Round(map_real(mapval,domain_min,domain_max,
170
range_min,range_max));
END;
FUNCTION dir_console_IO (VAR ch :CHAR) : BOOLEAN;
VAR
regs : registers; {From the DOS unit}
BEGIN
regs.AH := $06;
regs.DL := $FF;
MSDos(regs);
IF ((regs.flags AND FZERO) = 0) THEN BEGIN
ch := Chr(regs.AL);
dir_console_IO := TRUE;
END
ELSE BEGIN
dir_console_IO := FALSE;
END;
END;
FUNCTION check_kbd_status : BOOLEAN;
VAR
regs : registers; {From the DOS unit}
BEGIN
regs.AH := $0B;
MSDos(regs);
IF (Ord(regs.AL) = $FF) THEN BEGIN
check_kbd_status := TRUE;
END
ELSE BEGIN
check_kbd_status := FALSE;
END;
END;
FUNCTION max_single(s1,s2 : SINGLE):SINGLE;
{Returns the greater of two SINGLE type values}
BEGIN
IF s1 >= s2 THEN max_single := s1
ELSE max_single := s2;
END;
FUNCTION min_single(s1,s2 : SINGLE):SINGLE;
{Returns the lesser of two SINGLE type values}
171
BEGIN
IF s1 < s2 THEN min_single := s1
ELSE min_single := s2;
END;
BEGIN {initialize}
END. {INITIALIZE}
APPENDIX 6
PROGRAM SOURCE LISTING: GLOBAL TYPE AND VARIABLES UNIT
172
173
UNIT Globals;
{
This unit provides a variety of constants and types used in the
integrated ANN note generator and related programs.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
USES
Misc1, DOS, ANSI_Z;
{PUBLIC DECLARATIONS}
CONST
Pi = 3.141592653589793;
Exp_Max = 80.0;
Colon = ':';
graphic_string = '0123456789';
{For Bach}
v_len_in = 8;
v_len_out = 5;
epsilon : REAL = 0.005;
HTN_co_res : REAL = 3.5;
HTN_co_cap : REAL = 10.0;
HTN_co_wt : REAL = 1.0;
HTN_co_inp : REAL = 1.0;
HTN_co_iter : REAL = 1.0;
ART_co_vigilance : REAL = 1.0;
174
global_resistance : REAL = 1.0;
global_capacitance : REAL = 1.0;
{For Play_note}
N_C_mid = 264;
N_D = 297;
N_E = 330;
N_F =352;
N_G = 396;
N_A = 440;
N_B = 495;
N_C_hi = 528;
{For Beethoven (ART 1)}
Max_F2_nodes = 25;
Max_F1_nodes = 41;
Vector_length = Max_F1_nodes;
(* Max_F1_nodes = 16;
Vector_length = 16; c. 6/18/89 *)
TYPE
REAL = SINGLE;
{For Beethoven (ART 1)}
LTM_weights_ = ARRAY[1..Max_F1_nodes] OF REAL;
F1_node_ = RECORD
Curr_A : REAL; {Value of node now}
Last_A : REAL; {Value from last time step}
END;
F1_layer_ptr_ = ^F1_layer_;
F1_layer_ = ARRAY[1..Max_F1_nodes] OF F1_node_;
F2_node_ = RECORD
Curr_B : REAL; {Value of node now}
Last_B : REAL; {Value from last time step}
Wup : LTM_weights_; {BU LTM weights}
Wdn : LTM_weights_; {TD LTM weights}
WIN : INTEGER; {0 if not winner, 1 if winner}
Eligible : BOOLEAN; {TRUE if not rejected}
Committed : BOOLEAN; {TRUE if represents a category}
END;
F2_layer_ptr_ = ^F2_layer_;
F2_layer_ = ARRAY[1..Max_F2_nodes] OF F2_node_;
{General}
175
file_string_ = STRING[127];
Time_rec_ = RECORD
h,m,s,f : INTEGER;
END;
Note_ = (Note_C_Lo,Note_D,Note_E,Note_F,Note_G,Note_A,Note_B,
Note_C_Hi);
Vector_ = ARRAY[1..Max_F1_nodes] OF BYTE;
Notes_ = ARRAY[1..V_LEN_OUT] OF INTEGER;
Common_Area_ = RECORD
Notes : Notes_;
Delta_Vigilance : BOOLEAN;
New_category : BOOLEAN;
Is_Classical : BOOLEAN;
Candidate_Note : INTEGER;
END;
note_record_ = RECORD
n : ARRAY[1..500] OF BYTE;
c : INTEGER;
END;
PROCEDURE Dump_Common(cmn : Common_Area_);
IMPLEMENTATION
{PRIVATE DECLARATIONS}
{IMPLEMENTATIONS OF PROCEDURES AND FUNCTIONS}
PROCEDURE Dump_Common(cmn : Common_Area_);
{}
VAR
ii : INTEGER;
BEGIN {}
{ WRITELN('Dump of Common Area -----------');}
ANSI_CUP(1,0);
Write('Notes in sequence:');
ANSI_CUP(1,27);
FOR ii := 1 TO V_Len_Out DO BEGIN
{}
Write(cmn.Notes[ii],' ');
END; {}
{ WRITELN;}
ANSI_CUP(3,0);
Write('Change in vigilance:');
ANSI_CUP(3,31);
Write(cmn.Delta_Vigilance:5);
ANSI_CUP(4,0);
176
Write('New Category formed: ');
ANSI_CUP(4,31);
Write(cmn.New_category:5);
ANSI_CUP(5,0);
Write('Candidate sequence classical?:');
ANSI_CUP(5,31);
Write(cmn.Is_Classical:5);
ANSI_CUP(6,0);
Write('Candidate note:');
ANSI_CUP(6,35);
Write(cmn.Candidate_Note);
ANSI_CUP(7,0);
Write('------------------------------------');
ANSI_CUP(23,0);
END; {}
{----------------------------------------------------------}
BEGIN {INITIALIZATION}
END.
APPENDIX 7
PROGRAM SOURCE LISTING: CLASSICAL INSTRUCTOR UNIT
177
178
UNIT ClasInst; {Classical_Instructor}
{
This unit provides a critique of note sequences based on a data file of
example sequences, 'SEQUENCE.DAT'.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
USES DOS, misc1, ANSI_Z, globals;
{General}
FUNCTION Classical_instructor(Seq : Notes_):INTEGER;
{This function does a look-up of a sequence of notes and returns a 1 if
it is a listed sequence, a 0 if it is not recognized as being a classical
sequence. This function is used in the training of Salieri, the PDP
network.}
IMPLEMENTATION
CONST
Max_Seq = 100;
TYPE
N_str_ = STRING[v_len_in];
N_Str_Ary_Ptr_ = ^N_Str_Ary_;
N_Str_Ary_ = ARRAY[1..Max_Seq] OF N_Str_;
CONST
Seq_Table : N_Str_ary_ptr_ = NIL;
Number_of_seqs : INTEGER = 0;
VAR
179
Target, Instring : N_Str_;
Inf : TEXT;
Inchar : CHAR;
ii, jj : INTEGER;
Found : BOOLEAN;
nchar : CHAR;
FUNCTION Classical_instructor(Seq : Notes_):INTEGER;
{This function does a look-up of a sequence of notes and returns a 1 if
it is a listed sequence, a 0 if it is not recognized as being a classical
sequence. This function is used in the training of Salieri, the PDP
network.}
{Determine if a sequence of notes can be considered to be classical in
form. Does this by executing a look-up matching against the last n notes
in the passed-in sequence. Returns 0 if not found, 1 if found.
Requires the following TYPE definition:
Notes_ = ARRAY[1..v_len_in] OF INTEGER;
}
BEGIN {Classical_instructor}
{Convert notes to string representation}
Target := ''; {Clear string}
FOR ii := 1 TO v_len_out DO BEGIN
nchar := Chr(seq[ii]+48);
Target := Target + nchar;
END;
ANSI_CUP(0,39);
Write('Classical_Instructor: Target: ',target);
ANSI_CUP(23,0);
{Run through possible sequences, mark if found}
ii := 1;
Found := FALSE;
REPEAT {}
jj := Length(Seq_Table^[ii]);
Found := (Copy(Target,v_len_out-jj+1,jj) = Seq_Table^[ii]);
ii := ii + 1;
UNTIL (ii>Number_of_seqs) OR (Found);
{}
IF (Found) THEN {Return 1}
BEGIN
Classical_instructor := 1;
END
ELSE {Return 0}
BEGIN
Classical_instructor := 0;
180
END;
END; {Classical_instructor}
BEGIN
ii := 1;
NEW(Seq_Table); {Allocate space for table}
ASSIGN(Inf,'SEQUENCE.DAT'); {Set up to read data}
RESET(Inf);
WHILE (NOT EOF(Inf)) AND (ii <= Max_Seq) DO
{Get data, put in table}
BEGIN
READLN(Inf,Seq_Table^[ii]);
ii := ii + 1;
END;
CLOSE(Inf);
Number_of_seqs := ii - 1;
END.
APPENDIX 8
PROGRAM SOURCE LISTING: ANSI SCREEN CONTROL UNIT
181
182
UNIT ANSI_Z;
{
This unit provides certain ANSI and VT-52 screen control functions.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
INTERFACE
USES DOS, MISC1;
TYPE
ANSI_MODE_ = (NULL_,ANSI_,VT52_,IBM_PC_);
CONST
ANSI_MODE : ANSI_MODE_ = ANSI_;
PROCEDURE ANSI_CLRSCR;
{Clear the screen using ANSI control}
PROCEDURE ANSI_CUU(VAR ii : INTEGER);
{Cursor up}
PROCEDURE ANSI_CUD(VAR ii : INTEGER);
{Cursor down}
PROCEDURE ANSI_CUF(VAR ii : INTEGER);
{Cursor forward or right}
PROCEDURE ANSI_CUB(VAR ii : INTEGER);
{Cursor backward or left}
PROCEDURE ANSI_EEOL;
{Erase to End Of Line (VT52)}
183
PROCEDURE ANSI_CUH;
{Cursor home}
PROCEDURE ANSI_CUP(line, col : INTEGER);
{Cursor position}
IMPLEMENTATION
TYPE
Position_ = RECORD
l : BYTE;
c : BYTE;
END;
CONST
C_pos : Position_ = (l : 0;
c : 0);
VAR
inch : CHAR;
{ ANSI Control sequences
ESC [ Pn ; Pn R -> Cursor Position Report (CPR)
ESC [ Pn D -> Cursor Backward (CUB)
ESC [ Pn B -> Cursor Down (CUD)
ESC [ Pn C -> Cursor Forward (CUF)
ESC [ Pn ; Pn H -> Cursor Position (CUP)
ESC [ Pn A -> Cursor Up (CUU)
ESC [ Pn c -> Device Attributes (DA)
ESC # 8 -> Screen Alignment Display (DECALN)
ESC Z -> Identify Terminal (DECID)
ESC = -> Keypad Application Mode (DECKPAM)
ESC > -> Keypad Numeric Mode (DECKPNM)
ESC 8 -> Restore Cursor (DECRC)
ESC [ <sol> ; <par> ; <nbits> ; <xspeed> ; <rspeed> ; <clkmul> ; <flags> x
-> Report Terminal Parameters (DECREPTPARM)
ESC [ <sol> x -> Request Terminal Parameters (DECREQTPARM)
ESC 7 -> Save Cursor (DECSC)
ESC [ Pn ; Pn r -> Set Top and Bottom Margins (DECSTBM)
ESC [ Ps n -> Device Status Report (DSR)
ESC [ Ps J -> Erase in Display (ED)
ESC [ Ps K -> Erase in Line (EL)
ESC H -> Horizontal Tabulation Set (HTS)
ESC [ Pn ; Pn f -> Horizontal and Vertical Position (HVP)
ESC D -> Index (IND)
ESC E -> Next Line (NEL)
ESC M -> Reverse Index (RI)
184
ESC c -> Reset to Initial State (RIS)
ESC [ Ps ; Ps ; ... ; Ps l -> Reset Mode (RM)
ESC ( A | B | 0 | 1 | 2 -> Select Character Set (SCS)
ESC [ Ps ; ... ; Ps m -> Select Graphic Rendition (SGR)
ESC Ps ; ... Ps h -> Select Mode (SM)
ESC [ Ps g -> Tabulation Clear (TBC)
VT52 Mode control sequences
ESC A -> Cursor Up
ESC B -> Cursor Down
ESC C -> Cursor Right
ESC D -> Cursor Left
ESC F -> Enter Graphics Mode
ESC G -> Exit Graphics Mode
ESC H -> Cursor to Home
ESC I -> Reverse Line Feed
ESC J -> Erase To End Of Screen
ESC K -> Erase to End Of Line
ESC Y line column -> Direct Cursor Address
ESC Z -> Identify
ESC = -> Enter Alternate Keypad Mode
ESC > -> Exit Alternate Keypad Mode
ESC < -> Enter ANSI Mode
}
{
CASE ANSI_MODE OF
ANSI_ : BEGIN
END;
VT52_ : BEGIN
END;
IBM_PC_ : BEGIN
END;
END;
}
{$V-}
FUNCTION INT_TO_STR (ii : INTEGER):STRING;
VAR
tstr : STRING;
BEGIN
Str(ii,tstr);
int_to_str := tstr;
END;
{$V-}
185
PROCEDURE ANSI_CLRSCR ;
{Clear the screen using ANSI control}
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket,ascii_two,'J');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'H',ascii_esc,'J',ascii_esc,'H');
END;
{ IBM_PC_ : BEGIN
CRT.CLRSCR;
END; }
END;
c_pos.l := 0;
c_pos.c := 0;
END; {}
PROCEDURE ANSI_CUU(VAR ii : INTEGER);
{Cursor up}
BEGIN {}
IF ii <= 1 THEN ii := 1;
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket);
IF (ii > 1) THEN Write(Output,int_to_str(ii));
Write(Output,'A');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'A');
END;
{ IBM_PC_ : BEGIN
c_pos.l := CRT.WHEREY;
c_pos.c := CRT.WHEREX;
IF c_pos.l-ii >= 0 THEN
c_pos.l := c_pos.l - ii
ELSE
c_pos.l := 0;
CRT.GOTOXY(c_pos.c,c_pos.l);
END; }
END;
END; {}
PROCEDURE ANSI_CUD(VAR ii : INTEGER);
{Cursor down}
186
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket);
IF (ii > 1) THEN Write(Output,int_to_str(ii));
Write(Output,'B');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'B');
END;
{ IBM_PC_ : BEGIN
c_pos.l := CRT.WHEREY;
c_pos.c := CRT.WHEREX;
IF c_pos.l+ii <= 24 THEN
c_pos.l := c_pos.l + ii
ELSE
c_pos.l := 24;
CRT.GOTOXY(c_pos.c,c_pos.l);
END; }
END;
END; {}
PROCEDURE ANSI_CUF(VAR ii : INTEGER);
{Cursor forward or right}
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket);
IF (ii > 1) THEN Write(Output,int_to_str(ii));
Write(Output,'C');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'C');
END;
{ IBM_PC_ : BEGIN
c_pos.l := CRT.WHEREY;
c_pos.c := CRT.WHEREX;
IF c_pos.c+ii <= 79 THEN
c_pos.c := c_pos.c + ii
ELSE
c_pos.c := 79;
CRT.GOTOXY(c_pos.c,c_pos.l);
END;}
END;
END; {}
PROCEDURE ANSI_CUB(VAR ii : INTEGER);
{Cursor backward or left}
187
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket);
IF (ii > 1) THEN Write(Output,int_to_str(ii));
Write(Output,'D');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'D');
END;
{ IBM_PC_ : BEGIN
c_pos.l := CRT.WHEREY;
c_pos.c := CRT.WHEREX;
IF ((c_pos.c-ii) >= 0) THEN
c_pos.c := c_pos.c - ii
ELSE
c_pos.c := 0;
CRT.GOTOXY(c_pos.c,c_pos.l);
END; }
END;
END; {}
PROCEDURE ANSI_EEOL;
{Erase to End Of Line (VT52)}
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket,ascii_zero,'K');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'K');
END;
{ IBM_PC_ : BEGIN
CRT.CLREOL;
END; }
END;
END; {}
PROCEDURE ANSI_EEOS;
{Erase to End Of Screen (VT52)}
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'J');
END;
{ IBM_PC_ : BEGIN
END; }
188
END;
END; {}
PROCEDURE ANSI_CUH;
{Cursor home}
BEGIN {}
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket,'0;0','H');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'H');
END;
{ IBM_PC_ : BEGIN
CRT.GOTOXY(BYTE(1),BYTE(1));
END; }
END;
c_pos.l := 0;
c_pos.c := 0;
END; {}
PROCEDURE ANSI_CUP(line, col : INTEGER);
{Cursor position}
BEGIN {}
line := line MOD 256;
col := col MOD 256;
CASE ANSI_MODE OF
ANSI_ : BEGIN
Write(Output,ascii_esc,ascii_obracket,int_to_str(line),';',
int_to_str(col),'f');
END;
VT52_ : BEGIN
Write(Output,ascii_esc,'Y',Chr(Ord(BYTE(line+32))),Chr(Ord(
BYTE(col+32))));
END;
{ IBM_PC_ : BEGIN
CRT.GOTOXY(BYTE(col+1),BYTE(line+1));
END; }
END;
c_pos.l := line;
c_pos.c := col;
END; {}
{
IF (ANSI_MODE = ANSI) THEN BEGIN
END
ELSE BEGIN {VT52}
END;
189
}
BEGIN {Initialization}
ASSIGN (INPUT,'');
RESET(INPUT);
ASSIGN(OUTPUT,'');
REWRITE(OUTPUT);
WRITELN(OUTPUT);
REPEAT
WRITE('Is this machine''s video 1) ANSI or 2) VT-52 compatible ?');
READLN(inch);
inch := UPCASE(inch);
UNTIL (inch IN ['1','2','3']);
CASE inch OF
'1' : ANSI_MODE := ANSI_;
'2' : ANSI_MODE := VT52_;
{ '3' : ANSI_MODE := IBM_PC_; }
END;
ANSI_CLRSCR;
END.
APPENDIX 9
PROGRAM SOURCE LISTING: MUSICAL SEQUENCE EVALUATOR PROGRAM
190
191
PROGRAM test_composition (Input,Output);
{
This program compares the conformance of an input note sequence to the set
of example sequences in 'SEQUENCE.DAT'.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
DOS, misc1, ANSI_Z, globals, clasinst;
{General}
VAR
ii, jj, kk : INTEGER;
notes : notes_;
sinch : CHAR;
scon,infname : STRING;
inf : TEXT;
cnt,success : INTEGER;
BEGIN
Success := 0;
Cnt := 0;
FOR ii := 1 TO 5 DO BEGIN
notes[ii] := 0;
END;
{Get filename to test}
Write('Name of file to process: ');
Readln(infname);
{Open for input}
Assign(inf,infname);
192
Reset(inf);
REPEAT
Readln(inf,notes[5]);
IF (classical_instructor(notes) = 1) THEN BEGIN
INC(success);
END;
INC(cnt);
FOR ii := 1 TO 4 DO BEGIN
notes[ii] := notes[ii+1];
END;
UNTIL (Eof(inf));
Close(inf);
Writeln('Count = ',cnt:5,' Successes = ',success:5);
END.
APPENDIX 10
PROGRAM SOURCE LISTING: RANDOM NOTE GENERATION PROGRAM
193
194
PROGRAM random_compositon (Input,Output);
{
This program outputs random notes.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
VAR
ii, jj : INTEGER;
Outf : TEXT;
BEGIN
Assign(outf,'rmus.mus');
Rewrite(outf);
Randomize;
FOR ii := 1 TO 152 DO BEGIN
Writeln(outf,(Random(8)+1):1);
END;
Close(outf);
END.
APPENDIX 11
PROGRAM SOURCE LISTING: NOTE SEQUENCE PLAYING PROGRAM
195
196
PROGRAM play_composition(Input,Output);
{
This program plays notess output by the integrated ANN note generator, the
random composition program, and the classical composition program.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
Dos, CRT, misc1;
CONST
Pi = 3.141592653589793;
Exp_Max = 80.0;
Colon = ':';
graphic_string = '0123456789';
{For Play_note}
N_C_mid = 264;
N_D = 297;
N_E = 330;
N_F =352;
N_G = 396;
N_A = 440;
N_B = 495;
N_C_hi = 528;
TYPE
REAL = SINGLE;
{General}
Note_ = (Note_C_Lo,Note_D,Note_E,Note_F,Note_G,Note_A,Note_B,
Note_C_Hi);
197
VAR
inf : TEXT; {Input file handle}
instr : STRING;
cnote : INTEGER;
inch : CHAR;
{----------------------------------------------------------}
PROCEDURE play_a_note(cn : INTEGER);
VAR
ii : INTEGER;
BEGIN
CASE cn OF
1 : Sound(n_c_mid);
2 : Sound(n_d);
3 : Sound(n_e);
4 : Sound(n_f);
5 : Sound(n_g);
6 : Sound(n_a);
7 : Sound(n_b);
8 : Sound(n_c_hi);
ELSE
NoSound;
END;
Delay(180);
NoSound;
Delay(55);
END;
BEGIN {Main}
{get filename}
REPEAT
Write ('File to play? ');
Readln (instr);
instr := FSearch(instr,GetEnv('PATH'));
UNTIL (Length(instr) <> 0);
Assign (inf,instr);
Reset(inf);
WHILE NOT Eof(inf) DO BEGIN
Readln(inf,cnote);
play_a_note(cnote);
IF dir_console_IO(inch) THEN
IF UpCase(inch) = 'Q' THEN BEGIN
NoSound;
EXIT;
END;
198
END;
NoSound;
END. {Main}
APPENDIX 12
PROGRAM SOURCE LISTING: RULE-BASED NOTE SEQUENCE GENERATION PROGRAM
199
200
PROGRAM classical_composition (Input,Output);
{
This program composes note sequences in a manner designed to conform as
far as possible to a set of example sequences.
}
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
DOS, misc1, ANSI_Z, globals, clasinst;
{General}
VAR
ii, jj, kk : INTEGER;
notes : notes_;
sinch : CHAR;
scon, outfname : STRING;
outf : TEXT;
cnt,success : INTEGER;
ana : ARRAY[1..8] OF INTEGER;
ana_cnt : INTEGER;
PROCEDURE fill_ana (notes : notes_);
VAR
ii, tr : INTEGER;
BEGIN
ana_cnt := 0;
FOR ii := 1 TO 8 DO BEGIN
notes[5] := ii;
ana[ii] := classical_instructor(notes);
201
END;
FOR ii := 1 TO 8 DO BEGIN
IF ana[ii] = 1 THEN BEGIN
INC(ana_cnt);
ana[ana_cnt] := ii;
END;
END;
END;
BEGIN
Randomize;
FOR ii := 1 TO 5 DO BEGIN
notes[ii] := 0;
END;
{Get filename to test}
Write('Name of file to process: ');
Readln(outfname);
{Open for input}
Assign(outf,outfname);
Rewrite(outf);
FOR ii := 1 TO 10000 DO BEGIN
fill_ana(notes);
IF ana_cnt <> 0 THEN BEGIN
notes[5] := ana[(Random(ana_cnt) + 1)];
END
ELSE BEGIN
notes[5] := Random(8) + 1;
END;
FOR jj := 1 TO 4 DO BEGIN
notes[jj] := notes[jj+1];
END;
Writeln(outf,notes[5]);
END;
Close(outf);
END.
APPENDIX 13
PROGRAM SOURCE LISTING: OFFLINE BACK-PROPAGATION NETWORK TRAINING PROGRAM
202
203
PROGRAM Salieri_network_training_program (Input,Output);
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
DOS, struct, BP_unit, misc1, ANSI_Z, globals, clasinst;
{General}
TYPE
REAL = SINGLE;
bpnp_ = BP_node_ptr_;
wnp_ = weight_node_ptr_;
vnp_ = vector_node_ptr_;
seq_pop_rec_ = RECORD
n : notes_;
t : INTEGER;
e : REAL;
END;
seq_pop_ = ARRAY[0..99] OF seq_pop_rec_;
seq_pop_command_ = (init,replace);
VAR
snet, s31net, s46net : BP_net_;
ii, jj, kk : INTEGER;
Done : BOOLEAN;
cmn : common_area_;
notes : notes_;
tp1, tp2, tp3 : DVE_ptr_;
error_m, tne, sum : ARRAY[1..3] OF REAL;
ss : STRING;
binsum : ARRAY[1..3] OF INTEGER;
204
fpos, fneg : INTEGER;
tr : REAL;
sinch : CHAR;
scon : STRING;
sp : seq_pop_;
PROCEDURE maintain_seq_pop (VAR sp1 : seq_pop_;
spot : INTEGER;
cmd : seq_pop_command_);
VAR
ii, jj : INTEGER;
BEGIN
CASE cmd OF
init : BEGIN
FOR ii := 0 TO 99 DO BEGIN
FOR jj := 1 TO 5 DO BEGIN
sp1[ii].n[jj] := 0;
END;
sp1[ii].t := 0;
sp1[ii].e := 0.0;
END; {FOR ii}
END; {init}
replace : BEGIN
REPEAT
FOR jj := 1 TO 3 DO BEGIN
IF (jj = 1) THEN BEGIN
sp1[spot].n[jj] := Random(9);
END
ELSE BEGIN
IF (sp1[spot].n[jj-1] = 0) THEN BEGIN
sp1[spot].n[jj] := Random(9);
END
ELSE BEGIN
sp1[spot].n[jj] := Random(8) + 1;
END;
END;
END;
FOR jj := 4 TO v_len_out DO BEGIN
sp1[spot].n[jj] := Random(8) + 1;
END; {FOR jj}
sp1[spot].t := Classical_instructor(sp1[spot].n);
UNTIL (Odd(spot)) OR (sp1[spot].t = 1);
END; {replace}
ELSE
BEGIN
205
END;
END; {Case CMD}
END;
PROCEDURE Set_input_vector_from_notes (vp : DVE_ptr_;
n : notes_);
VAR
ii : INTEGER;
vpt : DVE_ptr_;
vn : ARRAY[1..40] OF INTEGER;
BEGIN
FillChar (vn,SizeOf(vn),#0);
{Blank the current vector}
FOR ii := 1 TO 5 DO BEGIN{Notes subscript}
IF n[ii] > 0 THEN vn [((ii-1)*8)+n[ii]] := 1;
END; {For notes subscript}
vpt := vp;
FOR ii := 1 TO 40 DO BEGIN
vnp_(vpt^.dptr)^.v := vn[ii];
vpt := vpt^.right;
END; {FOR ii}
END;
BEGIN
Done := FALSE;
s46net.data_fname := 's61.dat';
ANSI_CUP(13,0);
Writeln(MemAvail:8);
Writeln(s46net.data_fname);
Setup_BP_net (s46net,s46net.data_fname);
Writeln;
Writeln(s46net.wt_fname);
Set_BP_net_weights_from_file(s46net,s46net.wt_fname);
ANSI_CLRSCR;
Writeln(MemAvail:8);
maintain_seq_pop(sp,0,init);
FOR ii := 1 TO 100 DO BEGIN
maintain_seq_pop(sp,ii-1,replace);
END;
206
REPEAT
IF dir_console_IO (sinch) THEN BEGIN
IF (UpCase(sinch) = 'Q') THEN BEGIN
Close (s46net.out_f);
EXIT;
END;
END;
FOR ii := 1 TO 3 DO BEGIN
error_m[ii] := 0;
sum[ii] := 0;
binsum[ii] := 0;
fpos := 0;
fneg := 0;
END;
FOR ii := 1 TO 100 DO BEGIN
IF dir_console_IO (sinch) THEN BEGIN
IF (UpCase(sinch) = 'Q') THEN BEGIN
Close (s46net.out_f);
EXIT;
END;
END;
Set_input_vector_from_notes (s46net.vi,sp[ii-1].n);
vnp_(s46net.vts^.dptr)^.v := sp[ii-1].t;
BP_train_and_change (s46net);
tne[3] := ABS(BP_net_error(s46net));
sp[ii-1].e := tne[3];
notes := sp[ii-1].n;
FOR kk := 3 TO 3 DO BEGIN
error_m[kk] := max_single(ABS(error_m[kk]),tne[kk]);
END;
IF ((tne[3] > 0.50) AND (vnp_(s46net.vts^.dptr)^.v = 1.0)) OR
((tne[3] >= 0.50) AND (vnp_(s46net.vts^.dptr)^.v = 0.0))
THEN BEGIN
INC(binsum[3]);
IF ((tne[3] > 0.50) AND (vnp_(s46net.vts^.dptr)^.v = 1.0))
THEN INC(fneg);
IF ((tne[3] >= 0.50) AND (vnp_(s46net.vts^.dptr)^.v = 0.0)
207
) THEN INC(fpos);
Write (s46net.out_f,'I ');
FOR kk := 1 TO 5 DO BEGIN
Write (s46net.out_f,(notes[kk]/1.0):1:1,' ');
END;
Writeln (s46net.out_f);
Writeln (s46net.out_f,'T ',vnp_(s46net.vts^.dptr)^.v:1:1);
END;
ANSI_CUP(20,0);
Write(ii:4,' Max Current Ave. Binary ');
FOR kk := 1 TO 5 DO Write(notes[kk]:1);
Write(' ',vnp_(s46net.vts^.dptr)^.v:2:1);
ANSI_CUP(24,17);
FOR kk := 1 TO 5 DO Write(notes[kk]:1);
FOR kk := 3 TO 3 DO BEGIN
ANSI_CUP(20+kk,0);
sum[kk] := sum[kk] + tne[kk];
Write(kk:4,' ',error_m[kk]:5:3,' ',tne[kk]:5:3,
' ',(sum[kk]/ii):5:3,' ',binsum[kk]:3);
END;
Write(' FPOS: ',fpos:3,' FNEG: ',fneg:3);
IF (sp[ii-1].e < Random) THEN BEGIN
maintain_seq_pop(sp,ii-1,replace);
END;
END; {FOR ii}
FOR kk := 3 TO 3 DO BEGIN
ANSI_CUP(14+kk,0);
Write(kk:4,' ',error_m[kk]:5:3,' ',(sum[kk]/100):5:3,' ',
binsum[kk]:3);
END;
Write(' FPOS: ',fpos:3,' FNEG: ',fneg:3);
Done := (error_m[3] <= s46net.errtol);
Dump_BP_net_weights(s46net,s46net.wt_fname);
UNTIL (Done);
Close (s46net.out_f);
END.
APPENDIX 14
DATA FILE LISTING: HOPFIELD-TANK NETWORK WEIGHT DATA FILE
208
209
PROGRAM HTN_data_build(Input,Output);
{
Copyright 1989 by Wesley R. Elsberry. All rights reserved.
Commercial use of this software is prohibited without written consent of
the author.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
CRT, misc1, DOS;
CONST
row_inhibition : REAL = -0.08;
col_inhibition : REAL = -0.08;
seq_add : REAL = 0.0;
TYPE
REAL = SINGLE;
file_string_ = STRING[127];
data_array_ = ARRAY[1..64,1..64] OF REAL;
VAR
inf, outf : TEXT;
outdatf : FILE OF data_array_;
data : data_array_;
ii, jj, kk, ll, mm, nn : INTEGER;
inch : CHAR;
line, value : file_string_;
di1, di2 : INTEGER;
note1, note2, posit1, posit2 : INTEGER;
error : INTEGER;
min, max, range : REAL;
tii, tjj : INTEGER;
tr, ts : REAL;
sums : ARRAY[1..5,1..8] OF REAL;
sumssum : REAL;
210
PROCEDURE init_sums ;
VAR
ii, jj : INTEGER;
BEGIN
FOR ii := 1 TO 5 DO
FOR jj := 1 TO 8 DO sums[ii,jj] := 0;
END;
FUNCTION maximum(r1,r2:REAL):REAL;
BEGIN
IF r1 >= r2 THEN maximum := r1
ELSE maximum := r2;
END;
FUNCTION signum(x : REAL):REAL;
BEGIN
IF (x >= 0.0) THEN BEGIN
signum := 1;
END
ELSE BEGIN
signum := -1;
END;
END;
PROCEDURE show_node_sums;
VAR
ii, jj : INTEGER;
BEGIN
init_sums;
sumssum := 0;
FOR ii := 1 TO 5 DO
FOR jj := 1 TO 8 DO BEGIN
FOR kk := 1 TO 5 DO
FOR ll := 1 TO 8 DO BEGIN
sums[ii,jj] := sums[ii,jj] + data[(8*(ii-1)+jj),
(8*(kk-1)+ll)];
END;
END;
FOR jj := 1 TO 8 DO BEGIN
FOR ii := 1 TO 5 DO BEGIN
WRITE(sums[ii,jj]:6:3,' ');
211
sumssum := sumssum + sums[ii,jj];
END;
WRITELN;
END;
WRITELN (sumssum);
WRITELN;
END;
PROCEDURE set_row_and_column_inhibition;
VAR
ii, jj, kk, ll : INTEGER;
BEGIN
FOR note1 := 1 TO 8 DO
FOR posit1 := 1 TO 5 DO BEGIN
di1 := (8*(posit1-1)+note1);
FOR ii := 1 TO 8 DO{increase column inhibition}
BEGIN
IF (note1 <> ii) THEN BEGIN
di2 := (8*(posit1-1)+ii);
data[di1,di2] := data[di1,di2] + col_inhibition;
data[di2,di1] := data[di1,di2];
END;
END;
FOR jj := 1 TO 5 DO BEGIN
IF (posit1 <> jj) THEN BEGIN
di2 := (8*(jj-1)+note1);
data[di1,di2] := data[di1,di2] + row_inhibition;
data[di2,di1] := data[di1,di2];
END
ELSE BEGIN
END;
END;
END;
END;
PROCEDURE clear_diagonal;
VAR
ii, jj, kk, ll : INTEGER;
BEGIN
FOR ii := 1 TO 40 DO data[ii,ii] := 0.0;
END;
BEGIN
212
col_inhibition := ((8.0+(8.0-5.0))/5.0) * row_inhibition;
seq_add := -(row_inhibition/7.0);
init_sums;
NoSound;
FOR ii := 1 TO 40 DO
FOR jj := 1 TO 40 DO data[ii,jj] := 0.0;
Assign(inf,'sequence.dat');
Reset(inf);
Assign(outdatf,'htn.dat');
ReWRITE(outdatf);
WHILE NOT Eof(inf) DO BEGIN {get a line}
Readln(inf,line);
WRITELN(line); {increment connection values in the
data array}
FOR ii := 1 TO (Length(line)-1) DO BEGIN
Val(Copy(line,ii,1),note1,error);
Val(Copy(line,ii+1,1),note2,error);
WRITELN(note1,',',note2);
FOR posit1 := 1 TO 4 DO BEGIN
di1 := (8*(posit1-1))+note1;
di2 := (8*(posit1))+note2;
data[di1,di2] := data[di1,di2] + seq_add;
data[di2,di1] := data[di1,di2];
{symmetric weights!}
END;
END;
IF Length(line) >= 3 THEN
FOR ii := 1 TO (Length(line)-2) DO BEGIN
Val(Copy(line,ii,1),note1,error);
Val(Copy(line,ii+2,1),note2,error);
WRITELN(note1,',',note2);
FOR posit1 := 1 TO 3 DO BEGIN
di1 := (8*(posit1-1))+note1;
di2 := (8*(posit1+1))+note2;
data[di1,di2] := data[di1,di2] + seq_add;
data[di2,di1] := data[di1,di2];
END;
END;
END;
show_node_sums;
set_row_and_column_inhibition;
show_node_sums;
clear_diagonal;
213
show_node_sums;
WRITE(outdatf,data);
Close(inf);
Close(outdatf);
END.
APPENDIX 15
DATA FILE LISTING: CLASSICAL SEQUENCES DATA FILE
214
215
154
145
15
14
51
41
146
245
135
358
78
751
251
258
APPENDIX 16
DATA FILE LISTING: BACK-PROPAGATION NETWORK DATA FILE
216
217
Salieri net set up
!L 0.5
!A 0.5
!I 40
!H 20
!O 1
!T 1
!E 0.1
!D s61.dat
!R s61.out
!W s61.wt
!Z
APPENDIX 17
PROGRAM SOURCE LISTING: TRANSLATOR FROM PROGRAM NOTE FILES TO
MUSIC TRANSCRIPTION SYSTEM SONG FORMAT
218
219
PROGRAM trannote(Input,Output);
{
This program reads a file written by note_generator and converts it
to a form usable by the Music Transcription System (MTS).
}
{
Copyright 1989 by Wesley R. Elsberry & Diane J. Blackwood.
All rights reserved.
Commercial use of this software is prohibited without written consent of
the authors.
For information, bug reports, and updates contact
Wesley R. Elsberry
528 Chambers Creek Drive South
Everman, Texas 76140
Telephone: (817) 551-7018
}
USES
Dos, CRT, ANN, struct;
CONST
{For file_note}
fnote: ARRAY[1..8] OF INTEGER = (12,14,16,17,19,21,23,24);
VAR
cnote : INTEGER;
hunt : STRING;
inf : TEXT; {Input file handle}
instr : STRING;
line_ct : INTEGER;
line : STRING; {String to hold line from temp file
before writing to final file}
outf : TEXT; {Output file handle}
panel : INTEGER;
{----------------------------------------------------------}
PROCEDURE filenote(cn: INTEGER;
VAR ln_ct: INTEGER);
VAR
220
i : INTEGER;
BEGIN
IF ( (ln_ct = 11) OR (SeekEof(inf)) ) THEN Write (outf,
'0 1 4 40 0 ')
ELSE Write (outf,'1 1 4 40 0 ');
Write (outf,fnote[cn]);
Writeln (outf,' -1 -1 ');
IF (ln_ct = 1) THEN inc(panel);
IF (ln_ct = 11) THEN BEGIN
ln_ct := 0;
Writeln (outf);
Writeln (outf);
END;
END;
BEGIN {Main}
{initialize variables}
line_ct := 1;
panel := 0;
{get filename of input file}
REPEAT
Write ('File to translate? ');
Readln (instr);
instr := FSearch(instr,GetEnv('PATH'));
UNTIL (Length(instr) <> 0);
Assign (inf,instr);
Reset(inf);
{get filename of output file}
REPEAT
Write ('File to store translated music? ');
Readln (instr);
hunt := FSearch(instr,GetEnv('PATH'));
UNTIL (Length(hunt) = 0);
Assign (outf,'temp.sng');
Rewrite(outf);
{write header to mts file}
Writeln(outf,'30');
Writeln(outf,'10');
Writeln(outf,'4 4 0 3 3 7 6 4 2 2 2 1 1 1 1 2 ');
Writeln(outf,'0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ');
{process each note for use by mts}
WHILE NOT Eof(inf) DO BEGIN
Readln(inf,cnote);
filenote(cnote,line_ct);
line_ct := line_ct+1;
221
END;
{close the files in use}
Close(outf);
Close(inf);
{Copy temp file to final file with the value for panel corrected}
Assign (inf,'temp.sng');
Reset (inf);
Assign (outf,instr);
Rewrite (outf);
{read first two header lines and use panel to correct the number of
panels}
Readln(inf,line);
Writeln(outf,line);
Readln(inf,line);
Writeln(outf,panel);
{read the rest of the temp file and write to the final file}
WHILE NOT Eof(inf) DO BEGIN
Readln(inf,line);
Writeln(outf,line);
END;
Writeln (outf);
Writeln (outf);
{Close the files in use}
Close(outf);
Close(inf);
END. {Main}
BIBLIOGRAPHY
Amano, A., Aritsuka, T., Hataoka, N., and Ichikawa, A. 1989. On the use
of neural networks and fuzzy logic in speech recognition. Proceedings
of the IEEE/INNS International Joint Conference on Neural Networks
(IJCNN-89) Vol. I, 301-305.
Blackwood, D. J., Elsberry, W. R., and Leven, S. 1988. Competitive
network models and problem solving. Poster Presentation at the First
Annual Meeting of the International Neural Network Society.
Bower, G. 1981. Mood and memory. American Psychologist 36, 129-148.
Carpenter, G., and Grossberg, S. 1987a. A massively parallel
architecture for a self-organizing neural pattern recognition machine.
Computer Vision, Graphics, and Image Processing 37, 54-115.
Carpenter, G., and Grossberg, S. 1987b. ART 2: self-organization of
stable category recognition codes for analog input patterns. Applied
Optics 26, 4919-4930.
Charniak, E., and McDermott, D. 1985. Introduction to Artificial
Intelligence. Addison-Wesley. Reading, Massachusetts. 701 pp.
Cruz, V., Cristobal, G., Michaux, T., and Barquin, S. 1989. Invarient
image recognition using a neural network model. Proceedings of the
IEEE/INNS International Joint Conference on Neural Networks (IJCNN-89)
Vol. II, 17-22.
Farhat, N. 1986. Neural net models and optical computing: an overview.
In `Hybrid and Optical Computing,' Szu, Harold, Ed. SPIE, Bellingham,
Washington, Vol. 634, pp. 277-306.
Ferguson, J. 1987. Personal Communication.
Foo,Y. P. S., and Szu, H. 1989. Solving large-scale optimization
problems by divide and conquer neural networks. Proceedings of the
IEEE/INNS International Joint Conference on Neural Networks (IJCNN-89)
Vol. I, 507-512.
Grossberg, S. 1972. A neural theory of punishment and avoidance. II.
Quantitative theory. Mathematical Biosciences 15, 253-285.
Grossberg, S. 1973. Contour enhancement, short term memory, and
constancies in reverberating neural networks. Studies in Applied
Mathematics 52, 213-257.
222
223
Grossberg, S. 1975. A neural model of attention, reinforcement, and
discrimination learning. International Review of Neurobiology, 18,
263-327.
Harmon, L. D. 1970. Neural subsystems: an interpretive summary. In The
Neurosciences Second Study Program, F. O. Scmitt, Ed. Rockefeller
University Press, New York, pp. 486-494.
Hawking, S. W. 1988. A Brief History of Time: From the Big Bang to Black
Holes. Bantam Books, New York.
Hebb, D. 1949. The Organization of Behavior. Wiley, New York.
Hecht-Nielsen, R. 1986. Performance limits of optical, electro-optical,
and electronic neurocomputers. In `Hybrid and Optical Computing,' H.
Szu, Ed. SPIE, Bellingham, Washington, Vol. 634, 277-306.
Hecht-Nielsen, R. 1987. Counterprpagation Networks. Proceedings of the
IEEE International Joint Conference on Neural Networks (ICNN-87) Vol.
II, 19-32.
Hewitt, C. 1985. The challenge of open systems. Byte 10, 4, 223-242.
Hirsch, M. 1989. Convergence in Cascades of Neural Networks.
Proceedings of the IEEE/INNS International Joint Conference on Neural
Networks (IJCNN-89) Vol. I, 207-208.
Hopfield, J. J. 1982. Neural networks and physical systems with emergent
collective computational abilities. Proceedings of the National
Academy of Sciences 79, 2554-2558.
Hopfield, J. J., and Tank, D. 1985. "Neural" computation of decisions in
optimization problems. Biological Cybernetics 52, 141-152.
Kohonen, T. 1989. A self-learning Musical Grammar, or "Associative
memory of the second kind". Proceedings of the IEEE/INNS
International Joint Conference on Neural Networks (IJCNN-89) Vol. I,
1-6.
Leven, S. 1987a. Choice and Neural process: A dissertation. Chapter 5:
Neural process and form -- mathematics and meaning. University of
Texas at Arlington.
Leven, S. 1987b. S.A.M.: a triune extension to the ART model. 'North
Texas State University Symposium on Neural Networks' Poster
Presentation.
Levine, D. 1983. Neural population modeling and psychology: a review.
Mathematical Biosciences 66, 1-86.
224
Levine, D. 1986. A neural network theory of frontal lobe function. In
`Proceedings of the Eighth Annual Conference of the Cognitive Science
Society,' Erlbaum, Hillsdale, New Jersey, pp. 716-727.
Levine, D. 1990. Integration, disintegration and the frontal lobes. To
appear in `Motivation, Emotion, and Goal Direction in Neural
Networks,' D. Levine, and S. Leven, Eds. Erlbaum, Hillsdale, New
Jersey.
Levine, D., and Prueitt, P. 1989. Modeling some effects of frontal lobe
damage: novelty and perseveration. Neural Networks 2, 103-116.
Lippmann, R. P. 1987. An introduction to computing with neural nets.
IEEE ASSP Magazine, April 1987, 4-22.
MacCulloch, W. S., and Pitts, W. 1943. A logical calculus of the ideas
immanent in nervous activity. Bull. Math. Biophys. 5, 115-133.
MacLean, P. D. 1970. The triune brain, emotion, and scientific bias. In
`The Neurosciences Second Study Program,' F. O. Scmitt, Ed.
Rockefeller University Press, New York, pp. 486-494.
Matsuoka, T. Hamada, H., and Nakatsu, R. 1989. Syllable recognition
using integrated neural networks. Proceedings of the IEEE/INNS
International Joint Conference on Neural Networks (IJCNN-89) Vol. I,
251-258.
Metroplex Study Group on Computational Neuroscience. 1988. Computational
neuroscience: an opportunity for technology leadership for the
Metroplex. Report to the North Texas Commission Regional Technology
Program.
Newell, A., and Simon, H. A. 1976. Computer science as empirical
inquiry: symbols and search. Communications of the ACM 19, 3, 113-
126.
Nottebohm, F. 1989. From bird song to neurogenesis. Scientific American
February, 1989, 74-79.
Pao,Y.-H. 1989. Adaptive Pattern Recognition and Neural Networks.
Addison-Wesley, Reading, Massachusetts.
Paris, M. 1989. Personal Communication.
Parker, D. B. 1985. Learning-logic (TR-47). Massachusetts Institute of
Technology, Center for Computational Research in Economics and
Management Science, Cambridge, Massachusetts.
Rabelo, L. C., and Alptekin, S. 1989. Using hybrid neural networks and
expert systems. Proceedings of the IEEE/INNS International Joint
Conference on Neural Networks (IJCNN-89) Vol. II, 608.
225
Reddy, D. R., Erman, L. D., Fennell, R. D., and Neely, R. B. 1973. The
Hearsay speech understanding system: an example of the recognition
process. Proceedings of the International Conference on Artificial
Intelligence, 185-194.
Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning internal
representations by back propagation. In `Parallel Distributed
Processing,' D. Rumelhart, J. McClelland, and the PDP Research Group,
Eds. MIT Press, Cambridge, Massachusetts, Vol. 1, 365-422.
Shannon, C. 1948. A mathematical theory of communication. Bell System
Technical Journal 27, 379-423.
Simpson, P. 1988. A review of artificial neural systems, Parts 1 and 2.
Submitted to CRC Critical Reviews in Artificial Intelligence.
Sontag, E. D., and Sussman, H. J. 1989. Back-propagation <20>separates when
perceptrons do. Proc. IEEE/INNS Int'l Joint Conf. on Neural
Networks (IJCNN-89) Vol. I, 639-642.
Szu, H. 1989. Reconfigurable neural nets by energy convergence learning
principle based on extended McCullch-Pitts neurons and synapses.
Proceedings of the IEEE/INNS International Joint Conference on Neural
Networks (IJCNN-89) Vol. I, 485-496.
Tsutsumi, K. 1989. A multi-layered neural network composed of backprop.
and Hopfield nets and internal space representation. Proceedings of
the IEEE/INNS International Joint Conference on Neural Networks
(IJCNN-89) Vol. I, 507-512.
Werbos, P. J. 1974. Beyond regression: new tools for prediction and
analysis in the behavioral sciences. Unpublished Ph. D. dissertation,
Harvard University.
Widrow, B. 1987. ADALINE and MADALINE - 1963. Proceedings of the IEEE
International Joint Conference on Neural Networks (ICNN-87) Vol. I,
143-158.
Widrow, B., Pierce, W. H., and Angell, J. B. 1961. Birth, life, and
death in microelectronic systems. IRE Transactions on Military
Electronics 4, 191-201.
Widrow, B. and Winter, R. 1988. Neural nets for adaptive filtering and
adaptive pattern recognition. IEEE Computer 21, 3, 25-39.
Reference in New Issue View Git Blame Copy Permalink