Search papers, labs, and topics across Lattice.
This paper trains a diffusion model for 3D avatar sign language motion generation conditioned on text inputs, exploring the impact of phonological attribute conditioning using ASL-LEX 2.0 annotations. They establish a strong MDM-style diffusion baseline with SMPL-X representation that outperforms SignAvatar on gloss discriminability. Through systematic analysis of text encoders (CLIP vs. T5), conditioning modes, and attribute notation formats, they find that translating symbolic ASL-LEX notations to natural language is crucial for effective CLIP-based attribute conditioning.
Simply translating symbolic sign language notations into natural language unlocks significantly better motion generation when conditioning on phonological attributes with CLIP.
Generating natural, correct, and visually smooth 3D avatar sign language motion conditioned on the text inputs continues to be very challenging. In this work, we train a generative model of 3D body motion and explore the role of phonological attribute conditioning for sign language motion generation, using ASL-LEX 2.0 annotations such as hand shape, hand location and movement. We first establish a strong diffusion baseline using an Human Motion MDM-style diffusion model with SMPL-X representation, which outperforms SignAvatar, a state-of-the-art CVAE method, on gloss discriminability metrics. We then systematically study the role of text conditioning using different text encoders (CLIP vs. T5), conditioning modes (gloss-only vs. gloss+phonological attributes), and attribute notation format (symbolic vs. natural language). Our analysis reveals that translating symbolic ASL-LEX notations to natural language is a necessary condition for effective CLIP-based attribute conditioning, while T5 is largely unaffected by this translation. Furthermore, our best-performing variant (CLIP with mapped attributes) outperforms SignAvatar across all metrics. These findings highlight input representation as a critical factor for text-encoder-based attribute conditioning, and motivate structured conditioning approaches where gloss and phonological attributes are encoded through independent pathways.
