Search papers, labs, and topics across Lattice.
This paper introduces TensorCast, a distributed tensor management layer that addresses the limitations of existing LLM infrastructure by decoupling tensor state management from computation logic. By implementing Tensor-as-a-Service (TaaS), TensorCast enables programmable lifecycle primitives and first-class tensor abstractions, facilitating the reuse and composition of tensor management strategies across various workloads. The evaluation demonstrates that TensorCast not only matches the performance of specialized systems but also enhances efficiency, achieving a 93.2% improvement in median TTFT for concurrent multi-turn agent workloads through a programmable policy.
TensorCast reveals that decoupling tensor management from computation can boost performance by over 90% in multi-turn interactions, challenging the status quo of LLM infrastructure.
Modern LLM infrastructure increasingly manages tensors not only as computation data, but also as persistent states shared across distributed components. Existing systems optimize individual tensor management tasks, such as model weight loading, KV cache management, and checkpoint synchronization, by deeply integrating task-specific mechanisms with execution engines, networks, or storage backends. However, this specialization creates isolated silos that hinder the reuse and composition of tensor management strategies across evolving LLM workloads. In this paper, we identify tensor lifecycle management as a missing abstraction layer in LLM infrastructure and propose Tensor-as-a-Service (TaaS), which decouples tensor state management from computation logic. We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions, programmable lifecycle primitives, and a runtime that separates tensor management policies from execution mechanisms. This enables developers to write tensor management programs using TensorCast APIs while transparently leveraging distributed execution and data movement. We integrate TensorCast with vLLM and SGLang and evaluate it across diverse tensor lifecycle workloads, including model weight materialization, weight synchronization, KV cache management, and programmable request routing. Our results show that TensorCast achieves competitive performance with specialized tensor management systems while enabling new cross-component optimization policies. A programmable policy implemented with TensorCast improves median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads.
