Shigeo Morishima, Shin Ogata, Satoshi Nakamura

Research output: Contribution to conferencePaperpeer-review

1 Citation (Scopus)


A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip-synchronized speaking face image translation. We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic multimodal translation can be achieved.

Original languageEnglish
Number of pages6
Publication statusPublished - 2001
Externally publishedYes
Event2001 International Conference on Auditory-Visual Speech Processing, AVSP 2001 - Aalborg, Denmark
Duration: 2001 Sept 72001 Sept 9


Conference2001 International Conference on Auditory-Visual Speech Processing, AVSP 2001

ASJC Scopus subject areas

  • Language and Linguistics
  • Speech and Hearing
  • Otorhinolaryngology


Dive into the research topics of 'MULTIMODAL TRANSLATION'. Together they form a unique fingerprint.

Cite this