Search papers, labs, and topics across Lattice.
We investigate the optimization challenges of sequential test-time training (TTT) in nonlinear recurrent memory networks, finding that state dependence does not consistently yield performance gains over fixed-base baselines. To address this, we introduce DeltaTTT, a layerwise optimization approach that replaces joint inner-loop updates with local prediction targets and state-dependent delta rules, enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT demonstrate that DeltaTTT improves performance on language modeling and retrieval tasks compared to their recurrent counterparts.
Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.
Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.
