Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a hierarchical spatiotemporal action tokenizer (HiST-AT) for improved in-context imitation learning in robotics. HiST-AT uses a two-level vector quantization approach, first clustering actions into fine-grained subclusters and then mapping these to higher-level clusters, incorporating both spatial and temporal information. Experiments on simulated and real-world robotic manipulation tasks demonstrate that HiST-AT achieves state-of-the-art performance compared to non-hierarchical and spatial-only approaches.
Hierarchical quantization of robot actions, incorporating both spatial and temporal information, unlocks new state-of-the-art performance in in-context imitation learning.
We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, the lower level assigns input actions to fine-grained subclusters, while the higher level further maps fine-grained subclusters to clusters. Our hierarchical approach outperforms the non-hierarchical counterpart, while mainly exploiting spatial information by reconstructing input actions. Furthermore, we extend our approach by utilizing both spatial and temporal cues, forming a hierarchical spatiotemporal action tokenizer, namely HiST-AT. Specifically, our hierarchical spatiotemporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps. Finally, extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.