Search papers, labs, and topics across Lattice.
The paper introduces SkyMemory, a key-value cache (KVC) protocol designed for Low Earth Orbit (LEO) satellite constellations to accelerate LLM inference by increasing cache hits and reducing latency. They demonstrate the benefits of SkyMemory through simulations and a proof-of-concept implementation on a 95-node constellation testbed using Intel NUCs and an NVIDIA Jetson Nano for LLM hosting. The results show improved inference speed and cache hit rates for both terrestrially hosted and satellite-based LLMs.
LLM inference gets a speed boost from space, thanks to a novel caching strategy that leverages LEO satellite constellations.
We expand the scope of cache memory to include LEO constellations, which are highly distributed systems with thousands of satellites connected with free-space optics inter-satellite links (ISL) with numerous nodes only one hop from any point on earth. We show how to increase the number of cache hits and improve the speed of inference for the important use case of LLMs. These benefits apply not only to LLMs, both terrestrially hosted and on satellites, but also generalize to any cache distributed over multiple locations that needs to be accessed in a timely manner. We show the benefit of our key value cache (KVC) protocol in simulations and present a proof-of-concept implementation of the protocol for KVCs on a testbed comprising 5 Intel NUC Linux mini PCs hosting a 95 node, 19x5 constellation, with an NVIDIA Jetson Nano 8GB GPU hosting the LLM.
