We introduce the notion of symmetric multimodality for dialogue systems in which all input modes (eg. speech, gesture, facial expression) are also available for output, and vice versa. A dialogue system with symmetric multimodality must not only understand and represent the user's multimodal input, but also its own multimodal output. We present the SmartKom system, that provides full symmetric multimodality in a mixed-initiative dialogue system with an embodied conversational agent. SmartKom represents a new generation of multimodal dialogue systems, that deal not only with simple modality integration and synchronization, but cover the full spectrum of dialogue phenomena that are associated with symmetric multimodality (including crossmodal references, one-anaphora, and backchannelling). The application of the SmartKom technology is especially motivated in non-desktop scenarios, such as smart rooms, kiosks, or mobile environments. SmartKom features the situated understanding of possibly imprecise, ambiguous or incomplete multimodal input and the generation of coordinated, cohesive, and coherent multimodal presentations. We show that SmartKom's plug-an-play architecture supports multiple recognizers for a single modality, eg. the user's speech signal can be processed by three unimodal recognizers in parallel (speech recognition, emotional prosody, boundary prosody). We detail SmartKom's three-tiered representation of multimodal discourse, consisting of a domain layer, a discourse layer, and a modality layer. Finally, we give a brief overview of our new SmartWeb project, the follow-up to SmartKom, that deals with the foundations for the mobile multimodal access to semantic web services via next generation UMTS cell phones.

Download paper from URL: http://www.dfki.de/~wahlster/Publications/KI-2003-SmartKom-Wahlster_Springer.pdf

Speaker Bio:

Wolfgang Wahlster is the Director and CEO of the German Research Center for Artificial Intelligence (DFKI GmbH) and a Professor of Computer Science at Saarland University, Saarbruecken. He received his diploma and doctoral degree (1981) in Computer Science from the University of Hamburg. He has published more than 160 technical papers and 7 books on language technology and intelligent user interfaces. His current research includes multimodal and perceptive user interfaces, user modeling, embodied conversational agents, smart navigation systems, semantic web services, and resource-adaptive cognitive technologies. He is a AAAI Fellow, a ECCAI Fellow, and a recipient of the Fritz Winter Award (1991), and an IST Prize (1995) and the Beckurts prize (2000) for his research on cooperative user interfaces In 2001, the President of the Federal Republic of Germany, Dr. Johannes Rau, presented the German Future Prize to Prof. Wahlster for his work on language technology and intelligent user interfaces. Prof. Wahlster was the first computer scientist to receive Germany's highest scientific prize that is awarded each year for outstanding innovations in technology, engineering, or the natural sciences. He is a member of the German Academy of Sciences and the Royal Nobel Prize Academy in Sweden. Prof. Wahlster is on the Executive Board of ICSI.