Multi-modal interaction in the age of information appliances

Abstract

The coming millenium will be characterized by the availability of multiple information appliances that make ubiquitous information access an accepted fact of life. The ability to access and transform information via a multiplicity of appliances, each designed to suit the user's specific usage environment, requires the exploitation of all available input and output modalities to maximize the band-width of man-machine communication. There will be an increasingly strong demand for devices that present the same set of functionalities when accessing and manipulating the information, independently of the access device. The resulting uniform interface must be inherently multi-modal and dialog driven. During this evolution, conventional phones will remain a major type of access device. Also, the clients will offer more and more complex functions on ever more miniaturized devices. Eventually, we need to use these devices while performing other activities, and therefore often in hands-free or eyes-free mode. This paper addresses the challenges of coordinated, synchronized multimodal user interaction that is inherent in designing user interfaces that work across these multiplicity of information appliances. Amongst the key issues to be addressed are the user's ability to interact in parallel with the same information via a multiplicity of appliances and user interfaces, and the need to present a unified, synchronized view of information across the various appliances that the user deploys to interact with information. We achieve such synchronized interactions and views by adopting the well-known Model, View, Controller (MVC) design paradigm and adapting it to conversational interactions. The resulting Conversational MVC (CMVC) is to be considered as the key underlying principle of any conversational multi-modal application.