Search papers, labs, and topics across Lattice.
This paper introduces ReCache, a framework designed to enhance the efficiency of key-value (KV) cache reuse in tool-augmented language models by independently caching resource representations and minimizing computational overhead. By employing resource-wise attention, ReCache eliminates cross-resource interactions and optimizes KV blocks through structural and semantic pruning, achieving a significant reduction in memory usage and inference time. Evaluations demonstrate that ReCache maintains high performance while providing a 3.655脳 speedup in time-to-first-token and reducing KV-tensor memory allocation by 92.43%, highlighting its effectiveness in lowering inference costs for agentic models.
ReCache achieves a staggering 92.43% reduction in KV-tensor memory while maintaining nearly identical performance in tool-augmented language models.
Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce \textbf{ReCache}, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3\% versus 82.4\% Inv-F1) while providing a 3.655$\times$ time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43\% and accelerates attention by 1.423$\times$. These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.
