Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To solve non-Markovian manipulation tasks requiring spatial recall, event counting, and interval timing, ReCAT combines an instruction-conditioned observation encoder, a recurrent state-space memory using Mamba-2 and causal attention, and a flow-matching Transformer decoder. Controlled evaluations reveal that sequence model update dynamics specialize by cognitive function, with additive updates driving peak performance in counting and timing, while delta-rule updates govern spatial recall. Across benchmark evaluations, ReCAT achieves 95.3% success on LIBERO and boosts real-robot memory-dependent manipulation success from 8.3% to 66.7% over competitive short-history baselines.
Standard robot policies collapse on non-Markovian tasks requiring temporal tracking and recall, but pairing Mamba-2 structured memory with flow matching boosts real-world success from 8.3% to 66.7% while revealing that counting and spatial memory demand fundamentally different recurrent update rules.
Memory-dependent manipulation requires robots to make decisions using information that is no longer available to their current sensors, such as recalling an earlier visual cue, tracking task progress, counting repeated events, or estimating elapsed time. We present ReCAT, a language-conditioned policy with structured recurrent memory. An instruction-conditioned encoder forms features from the current observation. A recurrent memory integrates the observation stream through Mamba-2 layers and one causal attention layer. A flow-matching Transformer decoder reads the current and the historical representation through separate cross-attention in every block. ReCAT reaches 95.3\% average success on LIBERO and 62.4\% on RMBench, with the best or tied-best result on six of nine tasks. On three real-robot tasks probing spatial recall, event counting, and interval timing, the best ReCAT variant reaches 66.7\% average success, against 8.3\% for the strongest short-history baseline. Controlled comparisons within ReCAT show that the observation encoder and every-block memory conditioning are needed for this performance. They also show that update rules developed for efficient sequence modeling behave differently as robot memory: additive updates have the highest observed success on counting and timing, and delta-rule updates on spatial recall. Project website is at https://intuitive-robots.github.io/ReCAT
NVIDIA