Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents a modular extended reality (XR) platform integrating six AI services: speech recognition (Whisper), multilingual translation (NLLB, EuroLLM), speech synthesis (AWS Polly), emotion classification (RoBERTa), dialogue summarization (Flan-T5), and sign language rendering (MediaPipe). The platform maps International Sign gestures to 3D avatar animations in VR, using a custom corpus of IS gesture recordings. Benchmarking reveals AWS Polly offers the lowest latency for speech synthesis, and EuroLLM 1.7B outperforms NLLB in translation BLEU score, demonstrating the feasibility of cross-modal AI orchestration for accessible multilingual education.
Combining speech, translation, and sign language AI models inside virtual reality creates a surprisingly effective and accessible multilingual learning environment.
This work introduces a modular platform that brings together six AI services, automatic speech recognition via OpenAI Whisper, multilingual translation through Meta NLLB, speech synthesis using AWS Polly, emotion classification with RoBERTa, dialogue summarisation via flan t5 base samsum, and International Sign (IS) rendering through Google MediaPipe. A corpus of IS gesture recordings was processed to derive hand landmark coordinates, which were subsequently mapped onto three dimensional avatar animations inside a virtual reality (VR) environment. Validation comprised technical benchmarking of each AI component, including comparative assessments of speech synthesis providers and multilingual translation models (NLLB 200 and EuroLLM 1.7B variants). Technical evaluations confirmed the suitability of the platform for real time XR deployment. Speech synthesis benchmarking established that AWS Polly delivers the lowest latency at a competitive price point. The EuroLLM 1.7B Instruct variant attained a higher BLEU score, surpassing NLLB. These findings establish the viability of orchestrating cross modal AI services within XR settings for accessible, multilingual language instruction. The modular design permits independent scaling and adaptation to varied educational contexts, providing a foundation for equitable learning solutions aligned with European Union digital accessibility goals.