Search papers, labs, and topics across Lattice.
ReplaySSM addresses the state I/O bottleneck in State-Space Models (SSMs) by representing the recurrent state as a periodically updated checkpoint and a buffer of recent inputs, rather than eagerly managing a fixed-size state. This design decouples output computation from state updates, significantly reducing state memory traffic and enabling efficient speculative decoding rollbacks. Experimental results show ReplaySSM accelerates standard decoding by up to $1.47\times$ and achieves $1.8\times$ the throughput of vLLM's speculative decoding on large models, while also improving output quality through separate state precisions.
State-space models can achieve $1.8\times$ speculative decoding throughput and $1.47\times$ faster standard decoding by caching SSM inputs instead of their full state.
State-space models (SSMs) maintain a fixed-size recurrent state, but reading and writing it at every decoding step can bottleneck hybrid models. This eager state management causes large state I/O with low arithmetic intensity, leaves speculative decoding no cheap rollback, and feeds state quantization error into subsequent recurrent updates. We introduce ReplaySSM, which unrolls the recurrence to represent the state as a checkpoint and a buffer of recent SSM inputs. This separates output computation from checkpoint updates. By updating the checkpoint periodically, ReplaySSM avoids state writes on most steps and roughly halves the dominant state memory traffic. Keeping recent inputs also makes speculative rollback a buffer-pointer operation and enables output-only verification without materializing intermediate states. The same separation allows a low-precision checkpoint to serve output computation on most steps while a higher-precision checkpoint serves recurrent updates, so rounding error in the low-precision copy never enters later updates. We implement ReplaySSM in vLLM for Mamba-2 and Gated DeltaNet. At serving batch size 256, it accelerates standard decoding by $1.20$--$1.47\times$ over a tuned vLLM baseline across four hybrid models, and delivers about $1.8\times$ the throughput of vLLM's speculative decoding on the two 120B-class models. Separate state precisions also reduce output KL divergence and downstream accuracy losses compared with a shared low-precision checkpoint.
