Search papers, labs, and topics across Lattice.
This paper explores adaptive compression techniques for retrieval-augmented generation (RAG) on edge devices, addressing the significant overhead caused by increased context length during language model generation. By analyzing the trade-offs between compression rates and inference quality on the NVIDIA Jetson AGX Thor, the authors demonstrate that intermediate compression can lead to substantial energy savings鈥攗p to 53.2% for GPU energy and 48.2% for SoC energy鈥攚ithout compromising output quality. The findings advocate for dynamic runtime policies that adjust compression based on real-time workload telemetry, enhancing the efficiency of edge-based RAG systems.
Intermediate context compression can cut GPU energy usage by over 50% while maintaining quality, challenging static compression strategies in edge RAG applications.
Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.
