Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes test-time training (TTT) with KV binding, demonstrating that its behavior contradicts the common interpretation of online meta-learning and memorization. The authors prove that a broad class of TTT architectures can be reformulated as a learned linear attention operator. This reformulation allows for architectural simplifications, fully parallel implementations, and a unified view of diverse TTT variants, ultimately reframing TTT as learned linear attention with enhanced representational capacity.
Test-time training with KV binding isn't memorization, it's secretly a learned linear attention mechanism, unlocking architectural simplifications and parallelization.
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity.
NVIDIA