Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
Addressing the failure of purely in-policy memory mechanisms on long-horizon robotic manipulation, Ledger establishes an architectural split between short-term perceptual memory and long-term object state. The system augments a single fine-tuned $\pi_{0.5}$ policy with an internal frame-sampling buffer alongside an external, readable spatio-temporal object ledger maintained by SAM3 and a VLM. Evaluated on the RoboMME benchmark, this hybrid approach achieves a 64.3% average success rate—substantially outperforming the previous best baseline of 45.9%—with dramatic gains on object permanence (86.7% vs. 56.2%) without requiring a task-level router.
Forcing all memory into policy weights fundamentally bottlenecks robot manipulation: offloading persistent object identity and containment to an external, LLM-queried ledger jumps benchmark success from 45.9% to 64.3%.
Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned $π_{0.5}$ policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.
ETH