April 29, 1998

This class covers the Shruti system, temporal binding and the binding problem.

The binding problem can be illustrated with the visual system. The visual system has some 32 different areas which each specialize in picking out various features of the visual world. This makes sense as an architecture for a visual system, but how is it that we always see the world as a coherent image? We don't see it as separate features. Essentially, the brain does a great deal of parallel processing but somehow that information gets bound together to make a coherent perception. How does this happen. There is no good solution for this as of yet.

This kind of binding will have to be electrical because the bindings occur very quickly; there are no permanent connections created.

The variable binding problem is a particular case of the dynamic binding problem. For example, you probably have learned a general X-schema for push, but the actual action of pushing varies greatly depending on what you're pushing. The problem, technically, is how one can take the X-schemas and bring them together with connectionist ideas and create a neurally, psychologically plausible solution to the dynamic binding problem. There are two ways to look at the problem. One is to see what could be done in a connectionist system, without changing the connections, to have the system respond to rapidly changing bindings. Another way of looking at this is to see how a logical statement, such as "All men are mortal" could be represented in a way that it can be used to make logical inferences. If we know that all men are mortal and Socrates is a man, then the inference is that Socrates is mortal and so is anyone else who is a man (or woman). So 'mortal' has to be quickly bound to any person.

The systems discussed in this lectures are ways to go from the computational level to the structured connectionist level. This lecture will also show how X-schemas can be done at the connectionist level. It will involve the same mechanisms used in Shruti.

*******Aa Bb Network slide*******

In an earlier lecture, there was a discussion of dynamic binding of a simple sort. In the simple network designed to make connections between A and a or B and a, etc. There are excitatory connections between the letters and the center combination nodes and inhibitory connections between the center combination nodes themselves. If A and b are activated simultaneously, then the Ab node will become active so that if in the (near) future if only A is activated b will become activated. This is a sort of dynamic binding, a temporary electrical link, but it won't serve the general purpose of dynamic bindings. There is no way for the bindings to propagate.

Recruitment learning, discussed in the 3-16 lecture, is another kind of binding in which triangle nodes are recruited. Triangle nodes are mechanisms in which if any two of its connections are active, the third connection becomes active. The tooth diagram from a the 3-16 lecture illustrates the triangle node circuitry. It is proposed as a plausible base for learning concepts.

An old slide of a model for choosing a wine for dinner was put up to illustrate how triangle nodes can be used. The triangle nodes for this slide included a node with 'ham', 'has-taste', and 'salty' and one with 'ham', 'has-color', and 'pink'. These are like feature structures for 'ham' in which the taste of ham is salty and the color of ham is pink. Again, the point was made that triangle nodes can be used for deduction or for recognizing. In the recognizing mode, if the model is given that a food has a taste which is salty and a color which is pink, then spreading activation will activate ham. This is an indication of how triangle nodes might be used.

The "blue-green color shape slide" from 3-16 was put up again to illustrate how triangle nodes might be learned. The idea is to recruit triangle nodes. This is a cartoon of a model for learning that some Frisbee (your sister's) is blue and round. There is an assumption that there are some nodes which represent features, such as 'has-color' and values, such as 'blue'. You know that things have color and that blue is a color, but you don't know beforehand that the Frisbee is blue. There are several nodes which are also connected to these features and to other nodes. The idea is to recruit the nodes which have just the right connections, so that one of the concept nodes (in the center box) will be connected to a triangle node with 'has- color', 'blue', on one side and a triangle node with 'has shape' and 'round' on the other. All the nodes in the slide start as uncommitted nodes. When 'blue' and 'has color' are simultaneously activated, the nodes connected to them receive activation. Presumably, one of these nodes will have the best connections and so receive the most activation. If the brain is in learning mode, then the node with the most activation will have its connections strengthened and other connections will be weakened. The same process happens for the nodes between 'has shape' and 'round'. When the shape and color triangle nodes are recruited and activated, they will send activation to a central concept node and recruit it in the same way. When all the nodes are recruited, you have a conceptual structure which shows that the Frisbee has the color blue and the shape round. Each node which is recruited may have extra connections, but these connections are gradually weakened as the important connections are strengthened. For each node, between 10 and 100 neurons are postulated. These representations have to be localized to the 10-100 unit level.

In the verb learning model, the system has to learn which word applies to very different actions, so the connectionist version of model merging is re-recruitment. The first node recruited to represent an action may not turn out to be the best node after some more actions have been experienced. So the first node has to be ditched and a better one recruited. Or the first concept node can be fixed and an additional node for the new features can be added. People seem to have a difficult time with at least the first process of recruiting an entirely new node.

Again, the triangle node recruitment is a kind of dynamic binding, but it can't do inferences. We still need some way to represent bindings of variables as it's done in logic or parsing.The implementation problem is to take the neural connection hardware and change the way it works rapidly. So you want to be able to say that if George gave Mary the book, then Mary now owns the book, from general knowledge about giving and owning. This also has to be done in a neurologically plausible way. One of the proposals for doing this is the Shruti model developed by Lokendra Shastri and his students at Penn before he came to ICSI. The idea is that the temporal structure of cell firing is used to encode dynamic structures. This has been worked on by a number of people including Walter Freeman at Berkeley and Hebb.

We know that neurons in the brain fire about 100 times/sec. Are the synchronized? Some people believe that if cells fire in sync they features the cells represent are perceived as coherent. For instance, there are various parts of the visual system, and in line detection one cell can detect only a small part of a line which you may be looking at. Measurements can be taken of different cells which are separate from each other but have the same angle. So they are perceiving separate sections of the line. But measurements of the firing rate have shown that such cells fire more or less in sync which may be why the line is perceived as a coherent whole rather than as pieces. These synchronous oscillations occur in the 35-80 Hz range. The period of such oscillations is 15-30 seconds. Which means that the number of different temporal bindings possible is limited. It turns out that the limit is 7 plus or minus 2, which is a famous psychological result on short term memory. We can handle 7 plus or minus 2 items in the short term over several modalities. The synchronization can occur with a few cycles. It can last for a few hundred milliseconds. There can be about 3 msec of slop. It can occur among local cortical cells, among distant cells in the same cortical area, among cells from different cortical areas and among cells across the hemispheres. However, the measurements which show that neurons actually do fire in sync are not always reproduced in research, so synchronization is still controversial, but some biological evidence for it does exist.

*******give/own slide*********

The slide above shows how temporal dynamic binding can propagate through a system to yield inferences based on general knowledge. The bottom of the slide gives the logical rules for world knowledge about giving, owning, buying and selling. The diagram represents the general knowledge in a structured connectionist way. The idea is that you are going to have clusters of connectionist stuff which represent the general knowledge. These clusters have certain properties and implication is activation. If there is an instance of buying, then the 'buying' cluster is activated and activation spreads from it to the 'owning' cluster. Thus, an instance of buying leads to an instance of owning. This is very general knowledge. To make this knowledge useful we need a way to know who owns what. That is where the dynamic binding comes in. The diagram above represents neural firing phases as different colors. So all nodes which are the same color can be considered to be firing in the same phase. The slide represents an input which was "John gave the book to Mary". So the single node John and giver are blue, Mary and recipient are red and book and object are green. Now this activation can propagate through the system, according to the arrows. Ultimately, in the can-sell clusters, we have potential seller in red, so it is bound to Mary and can-sell object is green, so it is bound to the book. So we know that Mary can sell the book. This is an inference made as a result of propagating the activation and thus the dynamic bindings through the system.

*****neurons in phase*******

Again, the colors represent different phases of the firing patterns of the neurons. In other words, the neurons which are firing in sync are firing in the same phase, as the slide above shows. In this slide, we see that John and giver are firing in the same phase, so they are synchronized or dynamically bound. Also, Mary and recipient are is the same phase, as are book and given-object. The patterns of firing repeat and are lined up through time. In the system, only things from given-object below will be firing at first because that's what's activated by the input. After a few cycles, activation will spread to the other focal clusters and then owner and owned-object will start to fire in the same phase as Mary and book. Then, a little later, activation spreads further and potential-seller and can-sell-object start to fire in phase with Mary and book as well. The slide above shows the state of the system after activation has spread all the way through it.

The particular logic used in this system doesn't include all world knowledge we have, such as if you own a book, you can give it or sell it. And that John would have to own the book before he could give it to Mary or that it's possible to lend books and have a physical owner separate from a legal owner. This information is not built into the system above, but it could be, so that inference could be made based on this information as well.

*******cluster node slide*********

The cluster nodes (ovals) have inside them two outputs, plus or minus, which are mutually inhibitory and an enabler or enquiry node, which indicates that some information is being looked for. If there is something the system wants to know, it will send activation to the enquiry node which will cause the roles (giver, recip, etc) to be bound dynamically and then propagate the information through the system.

An on-line demonstration of the Shruti model was given. You can see the demonstration for yourself by going to the NTL web page and clicking on Shruti. The right side of the screen shows the architecture of a system which makes inferences about dates and holidays, so that the system can figure out whether the post office is open on the date in question (which is President's day). The links between the nodes are what embody the logic of the system. There is a link from plus federal holiday and plus post office to minus open. This means that the system incorporates the general knowledge that post offices are closed on federal holidays. On the left side are elements of the cluster are nodes which will show how the variables are bound and what output (plus or minus) the cluster nodes are giving at each time step. Each step of the demonstration is explained in words at the bottom of the screen. The buttons for moving the demonstration from step to step are at the far bottom of the screen.

As you run the demonstration, notice that if there is a square under a particular node, then it is activated; the four dots mean no activation. Each time step has two phases because the enabler cluster node has two variables to be bound, X is bound to the building (the Post Office) and Y is bound to the day. The X phase is shown first in the demonstration, showing all the activation and bindings connected with X. The Y phase is shown second, giving all the activation and bindings connected with Y. Keep and mind that the X and Y together are considered one time step. Stepping through the system allows you to see you the activation and bindings are propagated through the system. Notice that at some point the system determines that the day is a weekday and it is built into the system that if the day is a weekday, the Post Office is open. This is a shorter path than the one which determines that the day is a federal holiday, so it sends evidence early back to the original 'query' node that the post office is open. After a few more time steps, the system can determine that the day is a holiday and that the post office is closed. This will override the earlier evidence and the system will finally conclude that the post office is closed.

*******connectionist push schema********

Shruti can also be used to do X-schemas at a connectionist level. 'John gave the book to Mary' is similar to 'John pushed the book to the corner' in that you still have to bind participants and roles and propagate those bindings to make inferences. The bindings are encoded using temporal synchrony in the same way as in the logic case. The slide schema above shows how the push X-schema can be done in the Shruti system. In the X-schemas, each hexagon, translates to one of the oval focal clusters. Control flow in X-schemas is activity flow in this diagram. The focal clusters, as in the previous system, have plus, minus and enquiry nodes and binding information. Activation spreads from the enabler cluster to the enquiry (question mark) node in the Locate-Object Cluster and from there to a further, perhaps more complicated X-schema for visually locating the object. If the Locate Object(LO) X-schema succeeds, it sends activation back to the plus node on the LO cluster. If it fails it sends activation back to the minus node of the cluster, which immediately sends activation to the minus node of the enabler cluster and returns a minus (or fail) for the whole push enterprise. This will happen if any of the outside X-schemas fail. If the Locate Object X-schema succeeds and sends activation to the plus node of the LO cluster, then activation will spread from there to the enquiry node of the EM cluster which will in turn activate another complicated X-schema for estimating the mass of the object. If that X-schema succeeds it returns activation to the plus node of the EM cluster and it returns a value for the mass of the object, which is propagated through the system and passed on to other X-schemas such as Apply Force. The EM cluster actives the Ready cluster which activates the enquiry node of both the OH and RO clusters which activate the Shaping and Reach X-schemas respectively. If both of these are successful, the activation will continue up to the AF cluster which will activate the Apply Force X-schema and if that one is successful then activation will flow from the plus node on the AF cluster to the plus node on the Enable cluster at the bottom, signaling success for the push.

Notice that besides connections between plus, minus and enquiry nodes, there are also connections between the dots, which indicates that there are variable bindings (done with temporal phases as before) which are also getting passed to X-schemas. So the Estimate Mass X-schema needs to know what the object is that it has to estimate the mass for. In the beginning, the actual object is bound to the object role in the Enable cluster and that binding is passed to the LO cluster and the LO X-schema where it is bound with a location and sent to the EM cluster which sends the binding on to the Estimate Mass X-schema, etc. So the X-schema knows that the object corresponds to whatever is firing in the third phase, for instance. And it knows to act on whatever object is firing in that phase.

The argument is that this is at least a plausible version of how the X-schemas can be reduced to the connectionist level. Various Shruti papers describe how it could be reduced to the neural level.