Search papers, labs, and topics across Lattice.
Cassandra, a training-free self-speculative decoding framework, accelerates LLM inference on edge devices by constructing a high-performance draft model via fine-grained data selection and optimized pruning/mantissa truncation of weights and KV cache. This allows for rapid candidate token generation and parallel verification, achieving higher efficiency than layer skipping or structured KV compression techniques. Hardware acceleration further reduces format conversion overhead, resulting in up to 2.41x speedup over BF16 baselines and 1.81x more tokens generated compared to Eagle-3 on Llama 3 8B using an RTX 4090.
LLMs can run over 2x faster on your phone without any extra training, thanks to a clever pruning and quantization strategy that doesn't sacrifice accuracy.
Speculative decoding has emerged as a promising lossless approach for accelerating Large Language Models (LLMs). As reasoning LLMs increasingly suffer from decode-stage overhead and approximation-based methods degrade accuracy, lossless speculative decoding has become essential for efficient inference. However, existing methods still struggle to deliver strong low-batch performance without additional training, limiting practical deployment on consumer devices. To address this challenge, we propose Cassandra, an algorithm-hardware co-designed self-speculative decoding framework optimized for low-batch scenarios. Cassandra constructs a high-performance, training-free draft model through fine-grained data selection. Using optimized pruning and mantissa truncation, it identifies the most salient values in both model weights and the Key-Value (KV) cache, enabling rapid candidate token generation before full-precision parallel verification. Unlike prior self-speculative decoding methods based on layer skipping or structured KV compression, Cassandra achieves significantly higher efficiency. To further reduce the overhead of format conversion between Cassandra representations and standard floating-point formats, we also introduce a lightweight encoder-decoder hardware module designed for seamless integration with commercial GPUs and NPUs. Experimental results show that Cassandra achieves up to 2.41x speedup over the BF16 baseline without additional training. Furthermore, on Llama 3 8B running on an NVIDIA GeForce RTX 4090, Cassandra generates 1.81x more tokens under the same memory budget compared to Eagle-3, a state-of-the-art speculative decoding method.
