SLS RESEARCH

Multilingual Interaction

For over a decade, SLS researchers have been developing multilingual spoken dialogue systems. The goal of this work is to enable people to interact with machines in their preferred language. Much of the efforts have been devoted towards developing multilingual interfaces; systems that are capable of supporting conversational interaction in more than one language. Several European and Asian languages have been explored, including French, Italian, Spanish, Japanese, and Mandarin. Currently, most efforts are focused on Mandarin-based systems, especially with regards to language learning and translation. We have also begun to explore Arabic-based processing for Arabic-to-English and English-to-Arabic translation.

The video below shows a bilingual conversational system that provides weather information in either English or Japanese. Note how it follows the conversation in either language.

Much of the research in the speech and language community to date has focused on a relatively small subset of resource rich languages, such as English. It remains an open research problem to port spoken language technology to languages with less well developed resources, and without the intensive engineering efforts that have been associated with existing languages. For a fun discussion on this topic, listen to a podcast from the SETI Institute on "Speaking Klingon".

Further Reading

M. Nakano, T. Minami, S. Seneff, T. J. Hazen, D. Scott Cyphers, J. Glass, J. Polifroni, V. Zue, "Mokusei: A Telephone-based Japanese Conversational System in the Weather Domain," Proc. Eurospeech, Aalborg, Denmark, September 2001. (PDF)

T. J. Hazen, I. Lee Hetherington and A. Park, "FST-Based Recognition Techniques for Multi-Lingual and Multi-Domain Spontaneous Speech," Proc. Eurospeech 2001, Aalborg, Denmark, September 2001. (PDF),

V. Zue, et al., "From JUPITER to MOKUSEI: Multilingual Conversational Systems in the Weather Domain," Proc. Workshop on Multilingual Speech Communications (MSC2000), Kyoto, Japan, October 2000. (PDF)

J. Glass, et al., "Multilingual Spoken-Language Understanding in the MIT Voyager System," Speech Communication, 17(1), March 1995. (PDF)