Search papers, labs, and topics across Lattice.
This paper introduces DeltaLog, a novel decoding scheme that optimizes recurrent state management in linear attention models by maintaining a dense base state and a log of compact updates. By minimizing the need for full-state write-backs and replacing them with lightweight append operations, DeltaLog significantly reduces memory traffic and enhances computational efficiency. The implementation of DeltaLog across various models, including GDN, KDA, and RWKV6, results in up to 1.86x acceleration in recurrent-state updates and 1.05-1.20x end-to-end serving speedups compared to traditional dense recurrent baselines.
DeltaLog slashes recurrent-state write traffic by up to 7.83x while boosting decoding speed, revolutionizing how linear attention models manage memory.
Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to $1.86\times$, reduces profiled recurrent-state write traffic by up to $7.83\times$, and achieves $1.05$--$1.20\times$ end-to-end serving speedups over dense recurrent baselines.
