CS 294-4: Connectionist and Neural Computation

Lecture 3 - September 2, 1997

Nerve cells (neurons) are extremely complex entities. Their behavior and their interaction with other nerve cells are studied at a cellular and molecular level by neurobiologists. We reviewed some of this material in brief in Lecture 2. For a detailed discussion refer to Principles of Neural Science, 3rd edition, Kandel, E.R., Schwartz, J.H. and Jessell, T.M.

In this lecture we will look at several computational abstractions of neurons and neuronal interactions. One of the first such abstraction was proposed in 1943 by McCulloch and Pitts in their paper "A logical calculas of the ideas immanent in nervous activity," which appeared in Bulletin of Mathematical Biophysics 9: (reprinted in Neurocomputing (eds) J.A. Anderson and E. Rosenfeld (on reserve)). McCulloch and Pitts modeled neurons as simple Binary Threshold Units (or Threshold Logic Units) with fixed thresholds and uniform weights and showed that networks of such units could realize any finite expression of propositional logic.

In 1958 Rosenblatt proposed a family of computational abstractions of neurons called perceptrons ("The perceptron: a probabilistic model for information storage and organization in the brain" Psychological Review 65: (reprinted in the Anderson and Rosenfeld book on reserve). Perceptrons were more complex than the McCulloch and Pitts neurons, and most importantly, they were capable of learning. Subsequently, it was shown that simple perceptrons can be trained to discriminate linearly separable classes. This result, known as the "perceptron convergence theorem", appears in a 1962 paper by H.D. Block "The perceptron: a model of brain functioning. I", Reviews of Modern Physics 34: (reprinted in the Anderson and Rosenfeld book).

In 1982 Feldman and Ballard proposed a computationally sophisticated model of a neuron that incorporated several interesting ideas. Their model characterized a neuron in terms of a potential (analogous to its membrane potential), an output value (analogous to its firing rate), and a state together with associated functions that specified how the potential, output, and state of a unit at time t+1 were derived from the input, potential, and state of the unit at time t. The Feldman and Ballard model also incorporated other ideas such as units with multiple input sites and conjunctive connections (see "Connectionist models and their properties" Cognitive Science, 6: (reprinted in the Anderson and Rosenfeld book).

The year 1982 also saw the publication of a paper by Hopfield where he described emergent associative memory properties of symmetrically connected Binary Threshold Units (see "Neural networks and physical systems with emergent collective computational abilities" Proceedings of the National Academy of Sciences 79: (reprinted in Anderson and Rosenfeld). This model was extended by Hinton and Sejnowski who proposed the use of stochastic Binary Threshold Units to realize the "Boltzmann Machine" (see "A learning algorithm for Boltzmann machines" Cognitive Science 9 (reprinted in the Anderson and Rosenfeld book).

We will also discuss:

  • sigmoid units made famous by their use in gradient descent learning algortihms such as backpropagation (see D.E. Rumelhart, G.E. Hinton and R.J. Williams "Learning internal representations by error propagation", in Parallel Distributed Processing, vol. 1},
  • 3/2 binder units used in many structured connectionist models (see "Semantic Nets, Neural Nets, and Routines" L. Shastri and J.A. Feldman in Advances in Cognitive Science, 1 (Ed) N.E. Sharkey, Horwood. 1986).
  • rho-btu units introduced by V. Ajjanagadde and L. Shastri in their model of dynamic binding using temporal synchrony. These units encode information not in their firing rate but rather in their time of firing relative to other units. (see "Rules and Variables in Neural Nets" Neural Computation, 3(1) 1991.)

    Lokendra Shastri