Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
KITE is introduced as a training-free front-end for VLMs that converts robot execution videos into compact, interpretable tokenized evidence. It distills trajectories into keyframes paired with schematic BEV representations, serialized with robot/scene tokens into a unified prompt for failure analysis. Experiments on RoboFAC show KITE with Qwen2.5-VL significantly outperforms the vanilla model in simulation failure detection, identification, and localization, while remaining competitive with a tuned baseline.
Unlock robot failure analysis with KITE, a training-free front-end that distills long robot videos into interpretable tokens for off-the-shelf VLMs, boosting performance on failure detection and localization.
We present KITE, a training-free, keyframe-anchored, layout-grounded front-end that converts long robot-execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird's-eye-view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: https://m80hz.github.io/kite/