"Eigenlips" for Robust Speech Recognition

Title"Eigenlips" for Robust Speech Recognition
Publication TypeTechnical Report
Year of Publication1994
AuthorsBregler, C., & Konig Y.
Other Numbers872

In this study we improve the performance of a hybrid connectionist speech recognition system by incorporating visual information about the corresponding lip movements. Specifically, we investigate the benefits of adding visual features in the presence of additive noise and crosstalk (cocktail party effect). Our study extends previous experiments by using a new visual front end, and an alternative architecture for combining the visual and acoustic information. Furthermore, we have extended our recognizer to a multi-speaker, connected letters recognizer. Our results show a significant improvement for the combined architecture (acoustic and visual information) over just the acoustic system in the presence of additive noise and crosstalk.

Bibliographic Notes

ICSI Technical Report TR-94-002

Abbreviated Authors

C. Bregler and Y. Konig

ICSI Publication Type

Technical Report