Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores using image-to-video generative AI models to create synthetic deictic gesture datasets, addressing the data scarcity issue in gesture recognition research. They introduce a prompt-based video generation pipeline that leverages a small set of human-recorded gestures to produce a larger, more diverse synthetic dataset. Experiments show that models trained on a mix of real and synthetic data outperform those trained only on real data, indicating that the generated gestures are visually realistic and introduce beneficial variability.
Synthetic data generated by image-to-video models can significantly improve the performance of downstream gesture recognition tasks, offering a promising solution to the persistent data scarcity problem.
Gesture recognition research, unlike NLP, continues to face acute data scarcity, with progress constrained by the need for costly human recordings or image processing approaches that cannot generate authentic variability in the gestures themselves. Recent advancements in image-to-video foundation models have enabled the generation of photorealistic, semantically rich videos guided by natural language. These capabilities open up new possibilities for creating effort-free synthetic data, raising the critical question of whether video Generative AI models can augment and complement traditional human-generated gesture data. In this paper, we introduce and analyze prompt-based video generation to construct a realistic deictic gestures dataset and rigorously evaluate its effectiveness for downstream tasks. We propose a data generation pipeline that produces deictic gestures from a small number of reference samples collected from human participants, providing an accessible approach that can be leveraged both within and beyond the machine learning community. Our results demonstrate that the synthetic gestures not only align closely with real ones in terms of visual fidelity but also introduce meaningful variability and novelty that enrich the original data, further supported by superior performance of various deep models using a mixed dataset. These findings highlight that image-to-video techniques, even in their early stages, offer a powerful zero-shot approach to gesture synthesis with clear benefits for downstream tasks.