Search papers, labs, and topics across Lattice.
This paper conducts a comparative analysis of softmax attention and four recent linear attention architectures鈥擠eltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2鈥攆ocusing on their expressivity, memory management, and training efficiency. By employing a recurrent-memory notation, the authors highlight the trade-offs in implementation complexity and performance across these architectures, revealing that Kimi Delta Attention with Muon achieves the lowest final validation loss among the tested models. Additionally, the introduction of Cross-Layer Value Routing (CLVR) demonstrates a modest improvement in validation loss for DeltaNet and Gated DeltaNet, suggesting potential enhancements in memory utilization strategies.
Kimi Delta Attention with Muon outperforms other architectures in validation loss, while introducing Cross-Layer Value Routing reveals new avenues for optimizing memory management in linear attention models.
Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.
