Search papers, labs, and topics across Lattice.
This paper introduces an object-centric video understanding framework that separates action recognition from object identification to enhance the generation of robotic manipulation commands. By employing Temporal Shift Modules for efficient spatio-temporal action classification and a novel Object Selection algorithm, the method accurately identifies task-relevant objects, leading to improved command precision. The framework achieves significant performance improvements on the modified Something-Something V2 dataset, surpassing state-of-the-art baselines in both action classification accuracy and language generation metrics.
Achieving 86.79% action classification accuracy, this framework outperforms existing methods by over 80% in generating precise robotic commands from video demonstrations.
Translating video demonstrations into executable robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel \textbf{Object Selection} algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by Vision-Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2 dataset, our method achieves 86.79\% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2\% and 143.9\%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9\% and 171.7\% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.
