Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Kalypso, a relational LLM serving system that enhances the efficiency of semantic query processing by making LLMs aware of query plans. By enabling pipelined execution of semantic operators and reusing KV-cache states, Kalypso significantly reduces query completion times, achieving speedups of up to 4.57x compared to traditional request-centric systems. The adaptive, memory-aware scheduling algorithm employed in Kalypso effectively manages GPU memory pressure while optimizing operator execution, showcasing a novel approach to LLM serving.
Query-aware LLM serving can achieve up to 4.57x faster completion times by intelligently reusing memory states across semantic operators.
Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan, leaving substantial performance opportunities unused. This paper introduces relational LLM serving, an abstraction that makes LLM serving aware of semantic query structure while preserving query semantics and output accuracy. The key opportunity is pipelined execution across semantic operators: when intermediate tuples flow directly from one operator to the next, their KV-cache state can be reused instead of recomputed. We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm. Kalypso addresses a new online scheduling problem in which pipelined operator execution is coupled with GPU memory pressure management to reuse KV-cache state in the serving engine before eviction. Its scheduler continuously adjusts memory allocations to balance upstream parallelism, downstream progress, and GPU utilization. Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads, demonstrating that query-aware LLM serving can substantially improve the efficiency of semantic query execution.