April 12, 14, 1999

One of the main claims of the NTL enterprise is that "many basic concepts are directly embodied in the senses, emotions, social cognition, etc." Regier's and Bailey's systems are illustrations of this concept. In addition, for these models "structured connectionist models provide appropriate realizations" neurally. Narayanan's system is based on another claim that "all other concepts derive their meaning through metaphorical and other mappings." This is a connectionist style model which can carry out the interpretation of stories in an abstract domain based on explicit mapping to a concrete domain.

In cognitive linguistics, a target domain (or a target of a metaphorical map) is generally abstract, such as economics or politics. A source domain (the source of a metaphorical map) is generally embodied and it makes use of the same mechanisms used in VerbLearn. It is important that the same mechanisms used for understanding actions and other embodied domains can also be used also for understanding abstract domains, such as economics.

The system is "an implemented computational model that uses active, dynamic knowledge of motion and manipulation and the Event-structure metaphoric map to interpret expressions about abstract actions and plans found in simple newspaper stories." The system doesn't take raw text as its input. The newspaper stories are "pre-parsed" into f-structs containing roles in semantic frames and their values. These are filled in by hand. The program takes the input information and fills in additional facts which are implicit in the story from explicit story information, metaphorical knowledge, and world knowledge. News stories about abstract domains such as politics and economics get a significant portion of their semantics from specific schemas of temporal and spatial domains and other embodied domains like health.

*********System slide***********

The slide above is a diagram of the system. The bottom of the diagram represents the source, the middle represents the metaphor maps and the top represents the target. The f-structs are represented with tokens(dots) rather than feature-value pairs, but the mechanisms are similar to VerbLearn. It is computationally similar to VerbLearn, but cognitively quite different. In VerbLearn X-schemas were used, but the learning was done based on features. For this model, the execution of the X-schemas will be important. What happens in the model is that an X-schema executes and fills in slots in the f-struct. The timing of slot filling becomes important, which is why the f-structs are represented with tokens. This allows us to see change over time and make inferences based on the sequence of events rather than just the final outcome. In SN's system the X-schemas do not actually cause the agent to walk. Instead the agent uses the ability to walk to simulate or imagine walking. This gives the agent the inferential power needed to take these source domain schemas and bring them to bear on economics. The claim is that when you read a story about economics, you project yourself into the story and simulate the actions described and apply the results of the actions metaphorically to the story. This is a major theoretical claim.

Some examples of the kind of text which the model is designed to work with include:

France fell into recession. Pulled out by Germany.
Economy moving at the pace of a Clinton jog.
World-Bank prescribed Structural Adjustment Program bleeding developing countries.
Government taking bold new steps. Loosens strangle hold on business, slashes tariffs, and removes roadblocks to international trade.
Japan continues its long, painful slide into recession.
US economy on the verge of falling back into recession after lurching forward on an anemic recovery.
Protectionist policy is the mistaken therapy of prescribing palliatives to the economy in response to painful change and perception of injury.

The major example used for the discussion of this model is from the New York Times, August 1995:

"In 1991, India set out on a path of liberalization. Government started to loosen its strangle-hold on business; removed obstacles to international trade. Now the government is stumbling in implementing the liberalization plan."

In this model, the knowledge which we have about abstract domains is used. For instance, we know that if unemployment is low, inflation is likely to rise. This is knowledge specific to economics, which the system is able to use in drawing inferences. However, many inferences we make about sentences, such as the ones above, rely not on your specific knowledge of the economics domain but your knowledge of spatial and temporal domains.

*****I/O as f-structs slide******

The input to the system is an f-struct which has certain roles from the scene and values filled in. The values may be graded because probabilities will be very important to the operation of the model. The slots are the kinds of slots expected for a narrative: event, domain, actor. The aspect slot is very important because there are different inferences drawn from different aspects. "India was liberalizing" is very different from "India is liberalizing" or "India stopped liberalizing." So a theory of aspect is crucial to this model and will be discussed later on.

In the output of the system, there have been slots and fillers added. The context comes out as an ongoing plan which is having difficulty. The aspectual status will come out as interrupted. The most likely outcome is predicted to be failure. The next sentence in the story could cause you to revise these values. If the next sentence says that the government recovers, for instance, you will revise your prediction of failure. The task for the program is to have the feature values in the right state so that new information has the correct effect on beliefs. If you get no new information, the final state is .7 probability of failure. Notice that the inferences which provide the output feature values cannot be drawn with only the event structure metaphor or only knowledge of economics. We need both to understand what "stumble" could mean within the field of economics and in the context of the story. So there are two inference structures. One is in the source domain where X-schema execution is used to draw inferences about what it means to "stumble" for instance. The other is in the target domain where belief nets are used to model our knowledge about how government policies may affect inflation or how inflation may affect interest rates. Both kinds of inference are necessary to understand statements about economics, such as the examples below.

Now moving to the technical details, there are two new computational-level mechanisms which are used for this system. One of these is belief networks, which are based on Bayesian reasoning. The other is maps which relate one thing to another.

In the system, the knowledge at the source and target levels are represented with different computational mechanisms. The target domain knowledge is represented with belief nets, linked nodes representing probabilistic knowledge and conditional independencies. Belief nets do not represent activities. The belief nets are temporally extended. There are separate belief nets for several time slices because there are some inferences for which current beliefs depend on past beliefs or past beliefs are modified by current beliefs. The source domain knowledge is represented by X-schemas, which are active; these are the same mechanisms used to drive muscles, etc. The source domain and target domain are connected by metaphorical maps which could be thought of neurally as triangle nodes, so when they are activated and either source or target domain is activated, then all three become activated together: source, metaphor and target.

The Event Structure Metaphor in your readings is an illustration of a metaphor map in which, if the metaphor is active and if there is some knowledge in the source domain that you are at your destination, then the "achieved goal" node in the target domain gets activated. This shows how some of the inferences arrived at in the source domain from X-schema execution get mapped back to the target domain to be used in understanding the story.

These are some linguistics expression which might occur in a news story and their intuitive mapping to the event structure metaphor. Many of them you have probably not seen before in a news story about economics, but you can understand them easily. Many of them are aspectual; many have to do with goals.

******features projected across domains (see readings)******

This slide shows how some expressions from the event structure metaphor map to the economics domain. One very general mapping is actions are self-propelled motions. This is basic to the event structure metaphor and applies to more domains than just economics. These are mappings from the domains of states to the domains of actions and events with special cases for economics. For instance, general goals are understood in terms of destinations, and there is a special case in which economic goals (like lowering inflation) are understood in terms of a destination. Some of these mappings are specific mappings to the domain of economics. For instance, there is a health metaphor for economics itself, but not for actions or goals in general.

*******X-schema for walk (see readings)*******

The source domain is represented using X-schemas and their accompanying f-structs. These X-schemas will actually execute and that execution is crucial to the model. The example X-schema for walk uses the same notations as previously seen X-schemas. If you are okay and the path is okay, then there are two tokens which enable the walk X-schema. If you have enough energy, then you can start walking. Then there is an ongoing iterative walking process which itself includes an embedded X-schema (represented by the hexagon) which could be described in more detail. You test to see if you are at the goal. If you are, then you inhibit start, and you can finish and be done. This is a version of the walking X-schema.

Several things are happening at once: (1) X-schemas are inhibited from controlling actual movements, but it is the same mechanism which you could use to move. This kind of simulation is how you understand what movement is when you are not moving. (2) In older metaphor theory, we talk about projecting the source onto the target, but in this system, the metaphorical inferences are done in the source domain, not in the target. It's the results of those inferences which are projected back to the target domain.

In X-schemas, control is modeled with tokens. Tokens (the black dots) enable transitions which deposit further tokens enabling further transitions, etc. X-schemas are also hierarchical; a transition in one schema may be another X-schema, so when there are enough tokens in the right place in the higher schema, the lower-level embedded X-schema will execute.

******Source domain illustration (see readings)*******

The slide above diagrams the source domain used for our example. There are four X-schemas (Walk, Fall, Get Up, Stabilize) and a main source domain feature structure which has links to the various X-schemas. When an X-schema executes it will send tokens to f-struct parameters or take them away. Tokens in f-struct slots can cause other X-schemas to execute. These four X-schemas and f-struct can be used to model stumbling. When the general system is trying to understand a story in which stumble is used, the source domain part of the model will simulated stumbling. To do this, the walk X-schema is in the on-going state and there is a token in the "in control"slot, which indicates that the agent is in control of the walk action. There is also a token in the bump slot, which means the agent encounters a bump. This causes the Walk X-schema to interrupt and either Fall or Stabilize can be activated. If Fall is activated, a token gets sent to Get Up, which will execute if there is enough "energy". When reading a story, the simulation may stop in this state. For instance, for "the Indian government is stumbling . . ." the aspect indicates that the stumbling is in process. Its final result is unknown, so the simulation is in the state shown in the slide above. If no other information is available, then the inferences based on this state are the ones passed back to the target domain. So if the likelihood of falling given that the agent is stumbling is .7, then that probability will be passed metaphorically to the target domain and the agent will assume that the likelihood of the Indian government's failure is .7. However, if there is more information, the next sentence will be interpreted based on this state of the system. So if the next sentence is "the Indian government will likely recover," then the simulation will activate Stabilize and inferences can be drawn from this new state. Such inferences may indicate that the likelihood of falling is now .2. This probability is then passed back to the target domain and the likelihood of failure falls accordingly.

In the brain there is general high-level motor control. There is in fact a coordinated hierarchy of schemas in the brain which drive low-level muscle synergies and higher-level coordinated actions. These schemas are distributed because they apply in areas all over the brain. They are parameterized in order to help control the execution and they are active. Finally, the have to be able to model concurrent actions and interrupts. These are requirements of motor control which X-schemas are able to model.

********A controller schema *********

The generalized controller X-schema above captures important generalizations about actions. The key thing is the process. There is a notion of being ready, canceling, starting, being ongoing, iterating or interrupting, stopping and being done. The way it has been described is all about motor control and children, well before they learn language, have the concepts of starting actions, continuing, stopping actions, etc. The key idea is that you can do the same thing at a conceptual level for modeling or describing events and also for linguistic aspect. The slide below shows how the controller schema can be used for aspect. (The next lecture will cover that story.) Linguistic lexical items will code for particular states of this general controller schema. For instance "stumble" indicates that the suspend node of the Walk schema is active. This in turn means that the process node would have had to be active before the suspend. There is no other way to activate the suspend node without having the process node active. This is exactly what "stumble" means, that someone was walking and that walk was suspended for a particular reason.

The general idea is that some of thought and some of language has this character that all actions or events have the same control semantics which can be modeled with the controller X-schema and that language will highlight or focus on part of this schema. For instance, "set to go" focuses on the Ready state of the controller schema. "Set out" point to the Start. "Be in" refers to the Process state and "exit" highlights the Finish.

So the idea is that the metaphors will enable you to take a term like "stumble" in the economics domain, map it to the concrete source domain, know that "stumble" is interrupted walk, from which you may infer in the source domain that you didn't reach your destination. If the metaphor that goals are destinations, is active there will be an inference back to the target domain that you didn't achieve your goal. That inference may lead to other inferences in the economics domain. For instance, the country's currency may be devalued. The metaphor itself is active because the lexical items from both source ("stumble," "set out," etc.) and target ("liberalization plan," etc.) domain trigger it. The coactivation of the economics frame and the traveling frame activates the event structure metaphor of which Goals are Destinations is a part.

In this model, target domain knowledge is represented as "known" information, but not embodied information the same way that source domain information is. The particular kind of AI formalism and the particular computations used in the belief net is not an embodied representation. There are two kinds of knowledge in the target domain. Factual knowledge is facts about the domain which you know. For instance, you know that the US is a market economy. Correlational knowledge is about how things in the target domain affect each other. For instance, you know that fast economic growth can lead to high inflation. Probability theory is used for this kind of representation because if you are going to work with graded concepts, you need a set of rules for combining those concepts. The rules need to be consistent and need to be proven to yield the correct results without making wild predictions or contradictions. Probability theory has been proven to be a consistent set of rules, so it is a good and convenient option for this kind of knowledge representation. Probability theory does not always model human behavior because humans sometimes do not take the time to do the calculations involved, so they make choices which are not based purely on probabilistic reasoning.

In the target domain, there are also constraints and inherent structure in the target domain. The key requirement of the target domain is to "combine background knowledge of economics and inherent structure and constraints of the target domain with inferential products of metaphoric projections from multiple source domains." The model is able to handle different metaphoric projections, so it can handle mixed metaphors as well as single metaphors. Belief nets can combine this kind of information. Belief nets also have the ability to keep track of states. So there is a network of beliefs and if a new observation comes in, the system can adjust all of the beliefs in the network appropriately given the new information.

*******Belief networks for abstract domain theory*******

The slide above gives some examples of belief net notation. The circles refer to probability distributions. For instance, if one node is inflation it may have values such as high, medium and low and each has a probability associated with it. The probabilities will add up to one. Neurally, this can be thought of as a competitive network in which the activation of the three competing nodes sums to one. The belief nets allow us to do reasoning forward, backward and intercausally. This is completely non-neural. Neurons would have to have explicit forward and backward connections. For instance, if there is a link that says high growth causes inflation and if you know that there is high growth, then there is a certain probability that you will have inflation and the network can calculate this. Likewise, if you know that you have high growth, there is a certain probability that it was caused by inflation and the network can calculate this probability too. So you can do forward and backward reasoning. If there are two possible causes for some event, then the belief net can give the relative likelihoods for each cause. For instance, if you have inflation and it could be caused by high growth or by low unemployment, the belief net will calculate the probability of each possible cause. In the SN model, information comes to the belief net from the story, metaphorical connections, and general knowledge. The belief net uses its connections to take all that information and give the most probable state of beliefs.

*******What is a Bayesian network?******

One of the key features of a belief network is that the parts of a network which are not connected are conditionally independent. For instance, the slide above says that there is no connection between lung cancer and headaches. And that there is no relationship between asbestos and headaches that can't be explained by bronchitis. Independence allows the computations required for calculating the various probabilities to be much shorter as the slide shows. The belief nets used in the model are temporally extended, so there is a belief net for each time slice and there are connections between the belief nets of different time slices. In the system, the beliefs at time t are based on all the information available from the story, the target domain and the metaphorical links and the beliefs at time t-1.

Bayes Net Tutorial

In sum the action of the model goes as follows: the story fills in some parameters, then the appropriate X-schema executes and as a result fills in other parameters. Those parameters are passed back to the target domain via the metaphor maps. The belief net is adjusted accordingly. New information causes the system to continue and the effects of that new information and the resulting inferences are recorded in the belief net. So the process continues as long as new information keeps coming in. This is models how we understand a story as we are reading it.