MR-ATT: Automatic Transcription & Translation in Mixed Reality
Link to Full Report (PDF) Contributions: Development of a mixed reality application for real-time transcription and translation using the Magic Leap 2. Integration of OpenAI’s Whisper model with a Flask backend for accurate multilingual transcription and translation. Quantitative and qualitative evaluation through a comprehensive user study. Authors: Alejandro Cuadron Lafuente, Elisa Martinez Abad, Ruben Schenk, Sophya Tsubin Institution: ETH Zurich, Institute for Visual Computing Overview The project aimed to bridge language barriers in mixed reality (MR) by developing a real-time transcription and translation application for the Magic Leap 2. The application leverages OpenAI’s Whisper model to accurately transcribe and translate spoken language in MR environments, enhancing communication across multilingual users. ...