Search papers, labs, and topics across Lattice.
This paper introduces MindAU, a novel framework that utilizes EEG signals to control fine-grained facial action unit (AU) edits, addressing the challenge of mapping noisy neural data to precise facial expressions. By employing techniques such as temporal masked reconstruction and Dual-Stream Manifold Alignment, the authors successfully align EEG features with facial expression semantics, enabling high-fidelity, identity-preserving edits. Extensive experiments validate the effectiveness of MindAU, highlighting its potential as a tool for assistive technologies aimed at individuals with facial neuromuscular disorders.
EEG signals can now guide precise facial expression edits, opening new avenues for assistive technologies in facial neuromuscular disorders.
Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.
