Search papers, labs, and topics across Lattice.
To overcome the structural hand and facial articulation failures of generic video diffusion models, the authors develop SignRefine, which adapts a pretrained video diffusion transformer using 2D keypoint conditioning and spatially grounded local refinement adapters. This approach addresses the chronic shortage of native sign language training signals alongside NVSign, a new large-scale native sign dataset with diverse signers and environments. The resulting system improves hand pose precision by up to 30% over the strongest baseline and achieves over 80% preference among native sign language users for visual quality and comprehensibility.
Video diffusion models notoriously mangle hands and facial micro-expressions, but spatially grounded local adapters conditioned on 2D keypoints can recover the fine articulation required for intelligible sign language, outperforming existing baselines by 30% in pose precision.
Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.
