CS 294-4: Connectionist and Neural Computation
Lecture 8 - September 18, 1997
In previous lectures we have seen Elman Nets that deal with
temporal context within the framework of feedforward nets.
Today we discussed the Temporal Flow Model which admits links with variable
delays and lateral and top-down (recurrent) connections. Such a recurrent
network model provides a natural computational framework for dealing with
context and time varying signals. A related model
that uses variable delay links to deal with temporal context is the Time
Delay Neural Network model.
- Recurrent Nets pose two problems
- Training requires backpropagation over time. However, an efficient solution
to this problem is described in (Watrous et al., 1989).
- Since each token extends over time, a continuous time-varying target
function must be specified to cover the duration of the token
(in practice, any reasonable target function such as a ramp, or a
sigmoid suffices)
- Some References
- "Complete Gradient Optimization of a recurrent network applied to
/b/,/d/,/g/ discrimination", Watrous, R., Ladendorf, B., and Kuhn, G.M. (1989).
Journal of Acoustic Society of America 87, 1301-1309.
(describes an efficient solution to the problem of computing gradients
in recurrent networks).
- Learning Phonetic Features Using Connectionist Networks. Watrous, R. and
Shastri, L. (1986) Tech. Report MS-CIS-86-78. University of Pennsylvania.
(describes the Temporal Flow Model).
- "Phoneme Recognition using time-delay neural networks", Waibel, A.,
Hanazawa, T. Hinton, G., Shikano, K. and Lang, K. (1988)
Tech. Report TR-1-0006. ATR Telephony Research Laboratories.
(describes the Time Delay Neural Net Model).
Biologically motivated learning rules
- Hebb's rule
- Hebbian and anti-hebbian learning
- Probabilistic interpretation of Hebb's rule
- BCM rule
- Competitive Learning
- Topological Maps
Long-term Potentiation (LTP): a biological synaptic modification rule
Homosynaptic and associative LTP
Long-term Depression (LTD); homosynaptic and heterosynaptic LTD
Hebb's paper in "Neurocomputing" is the original reference for Hebb's rule.
The BCM algorithm is described in the Bienenstock, Cooper and Munro
paper in "Neurocomputing" which also contains papers by Kohonen,
Malsburg, and Grossberg on the related topics of competitive learning
and topological maps. The Rojas book is also a good reference for
competitive learning and topological maps.
Some viewgraphs from Lecture 8
Lokendra Shastri