Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for synthesizing sign language videos using a loss-guided multi-expert Generative Adversarial Network (GAN), which enhances communication for individuals with hearing impairments. By employing three specialized discriminators鈥攇lobal, hand, and head鈥攖he framework enables implicit feature specialization and stabilizes training dynamics through a United Loss consensus mechanism. The resulting models achieve competitive PSNR scores of 29.8 and 30.7 on a custom dataset, demonstrating the potential for deployment on consumer-grade hardware.
Achieving over 30 PSNR in sign language video synthesis with a GAN framework that balances stability and detail could revolutionize communication for the hearing impaired.
This preliminary technical report presents a framework for sign language video synthesis using a loss-guided multi-expert Generative Adversarial Network (GAN) to enhance communication for individuals with hearing impairments. Three specialized discriminators -- global, hand, and head -- each guide a corresponding expert branch in the generator toward a distinct visual region, enabling implicit feature specialization without explicit diversity losses. To stabilize this multi-discriminator system, whose early-phase training otherwise exhibits chaotic dynamics, we introduce a United Loss consensus mechanism that regularizes each discriminator toward the ensemble average at a 10% weight. Each branch further adopts a dual-pathway convolutional-transformer design with learnable AdaptiveFeatureFusion, balancing the stability of convolutions against the detail of windowed self-attention. The generator is trained using an alternating three-mode schedule (discriminator, holistic generation, branch-specialized generation). On a custom 156GB dataset with a filtered test set that removes easy and repetitive samples, our 0.2B-parameter variant achieves 29.8 PSNR and the 1.3B-parameter variant achieves 30.7 PSNR, with inference VRAM footprints of 1.5 GB and 8 GB respectively, enabling deployment on consumer-grade hardware. Full ablation studies remain ongoing due to the 2-3 month training cycle on a single GPU. The system was showcased at the 2025 Hong Kong Frontier Technology Summit.
