Search papers, labs, and topics across Lattice.
The paper introduces Rank-Then-Act (RTA), a novel framework for learning control policies from expert video demonstrations without relying on environment rewards. By employing a Vision-Language Model (VLM) as a progress-based ordinal scorer and utilizing a correlation-based reward function derived from Spearman rank correlation, RTA effectively recovers temporal ordering from visual semantics. Evaluations across various control benchmarks reveal that RTA not only matches but often surpasses existing video-based reward learning methods, highlighting its potential for scalable policy learning without explicit reward design.
Correlation-structured supervision from video-derived ordinal signals can replace traditional reward mechanisms, achieving robust policy learning across diverse tasks.
We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-Language Model (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.
