Search papers, labs, and topics across Lattice.
This paper introduces a compositional online learning framework designed to optimize semantic data processing systems by effectively integrating online-learning components at the boundary of large language model (LLM) calls. Given that LLM compute constitutes 80-90% of query costs, the proposed method allows for the execution of learning updates within the latency of LLM calls, thereby transforming traditional adaptive query processing constraints. A case study in Cortex AISQL demonstrates that the framework can achieve up to an 8x reduction in costs for a conjunction-filter workload by leveraging memoization, filter-ordering, and cascade-routing learners.
By embedding online learning directly into LLM call latencies, this framework can cut query costs by nearly 8x, revolutionizing how we optimize semantic data processing.
An LLM call in a semantic data processing system is expensive enough to dominate query cost, yet slow enough to hide a CPU-side learner's update behind its round-trip. In production, LLM compute accounts for $80-90\%$ of query cost, and each call costs $10^5-10^7\times$ a relational predicate. The latency window inverts a design constraint of classical adaptive query processing, where online learners had to stay lightweight to avoid dominating the predicates they optimize. At LLM latency, per-call gradient steps and per-batch threshold solves fit inside the round-trip. We develop compositional online learning at the LLM call boundary: a framework for combining online-learning components in semantic data processing systems. Each component makes execution-time decisions and refines its learned artifacts online. The design space spans two axes, decision granularity and learner update cadence, and the components share a single learning pattern that hides each trainer step inside the next LLM round-trip. A production case study in Cortex AISQL composes three components: a memoization layer, an online per-call filter-ordering learner, and an online per-batch cascade-routing learner. A conditional cost decomposition assigns each learning component to a distinct factor of per-row LLM cost. Under independence, the two learning components compose multiplicatively to an $11.4\times$ upper bound on a representative conjunction-filter workload. Self-selection at the cascade boundary, sample-budget shrinkage, and selectivity-estimation drift reduce it to a realistic figure near $8\times$.
