Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
The authors release MAESTRO, an open multimodal auditory attention decoding (AAD) dataset that synchronously captures EEG, eye gaze, pupillometry, head IMU data, and egocentric video during four-speaker competing speech tasks across variable noise levels. Standard AAD pipelines rely predominantly on unimodal neural recordings, which neglect the behavioral and visual orienting cues humans naturally leverage in dense acoustic environments. Benchmarking on a four-speaker task reveals that fusing behavioral dynamics with neural signals substantially outperforms unimodal EEG-only baselines under challenging signal-to-noise ratios.
Decoding who you are listening to in a crowded room fails on brainwaves alone—grounding EEG with egocentric video, gaze, and head kinematics drastically boosts multi-speaker attention tracking.
Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .