Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SemSpot, a semantics-aware service model that optimizes the use of spot capacity in LLM inference platforms by allowing agent applications to dynamically select from short-lived offers based on task state and completion rules. By analyzing 1,535 cases across six agent benchmarks, the authors identify four recurring workflow structures that highlight the model's potential to enhance cost efficiency, service time, and fallback behavior in cloud inference workloads. The findings suggest that integrating SemSpot with specialized Model-as-a-Service (MaaS) support can significantly improve the management of inference state and semantic segments during long requests.
SemSpot enables LLM agents to dynamically optimize cloud inference costs and performance by leveraging short-lived offers tailored to task semantics.
LLM agents increasingly drive long-running cloud inference workloads in which model calls differ in urgency, redundancy, completion semantics, and replay cost. Model-as-a-Service (MaaS) platforms expose several service models for trading cost against latency, availability, and capacity commitment. These models operate primarily at request, job, or endpoint scopes and provide limited support for combining transient platform supply with the evolving semantics of an agent task. We present SemSpot, a semantics-aware service model that allows agent applications to leverage the spot capacity of LLM inference platforms. At the request level, SemSpot lets a provider publish short-lived offers over successful price, completion probability, and failure-notification deadline; the agent runtime selects among these offers using the current task state and completion rule. An audit of 1,535 cases from six agent benchmarks identifies four recurring workflow structures and shows how this service model may produce different cost, service-time, and fallback behavior. With specialized MaaS support, token-level SemSpot further preserves provider inference state and runtime-verified semantic segments inside a long request. We develop the service model, economic boundary, and the cross-layer research agenda required to realize SemSpot.