Search papers, labs, and topics across Lattice.
This paper introduces PIDiffSign, a physics-informed diffusion model that generates 3D sign language motions from spoken language while ensuring both semantic fidelity and biomechanical plausibility. By integrating anatomical constraints into the model architecture and training objectives, it effectively addresses issues such as bone length drift and joint angle violations that plague existing methods. Experiments reveal significant enhancements in pose accuracy and joint-angle correctness compared to traditional diffusion baselines, underscoring the model's capability to produce more realistic and semantically aligned sign language motions.
Physics-informed diffusion not only enhances the realism of sign language generation but also ensures that the motions adhere to anatomical constraints, bridging the gap between semantics and biomechanics.
Sign language production, which generates continuous 3D skeletal motion from spoken language input, must simultaneously satisfy two constraints: semantic fidelity, so that a deaf viewer can recognize the intended sequence of glosses, and biomechanical plausibility, so that the generated skeleton respects anatomical constraints. Existing approaches optimize semantic reconstruction through coordinate-based objectives that treat the skeleton as an unstructured vector, thus allowing for bone length drift, joint angle violations, and temporarily locked fingers. We introduce PIDiffSign, a physics-informed diffusion model for gloss-to-pose translation that incorporates anatomical constraints into both the architecture and training objective. The model uses a Transformer encoder-decoder, where the decoder is conditioned on the diffusion time step through adaptive zero-initialized layer normalization and cross-attends to gloss representations. A differentiable geometry module enforces bone length consistency and biologically valid joint angles throughout generation. Training combines anthropomorphic, kinematic, angular, and finger-joint constraints with a contrastive gloss-pose alignment loss and classifier-free guidance for semantically conditioned sampling. Experiments on the PHOENIX14T and CSL-Daily benchmarks show consistent improvements over a strong diffusion baseline in pose accuracy, joint-angle correctness, distributional realism, and back-translation quality. These results demonstrate that physics-informed diffusion improves both motion realism and semantic fidelity for sign language generation.
