Search papers, labs, and topics across Lattice.
This paper investigates replacing Large Language Models (LLMs) with Jev decision models for low-latency edge service orchestration by integrating Jev's decision-oriented API to extract bounded intent fields and apply orchestration logic. The study compares Jev against self-hosted and hosted LLMs on verified requests and a live admission path, demonstrating significant latency reductions and improved request completion rates. Jev achieves 22.7-64.5% lower median decision latency than the fastest LLM, with performance largely invariant to input size, contract width, or catalog size, making it suitable for latency-bound admission on bounded contracts.
Edge service orchestration can achieve up to 64.5% lower decision latency than LLMs by using Jev decision models, even at high loads where LLMs fail.
Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion. The integration extracts four to eight bounded intent fields and applies a shared validator, admission policy, and scheduler, accounting for decision waiting throughout the request timeline. We compare Jev, two self-hosted decision models, and three hosted large language models (LLMs) on 8,280 verified requests and on a live admission path with modeled execution and a real optical character recognition service. Across 33 test conditions, Jev reduces median decision latency by 22.7-64.5% relative to the fastest LLM. This latency barely moves with input size, contract width, or catalog size. On four-field contracts, Jev's API fees per correct decision are 59.7-80.9% lower at a cost of a few exact-match points, while wide contracts mark the limit of the substitution. Receiving the service catalog with each request, Jev names unseen services as accurately as known ones. On the live admission path, Jev keeps 0.91-0.95 of requests exact and on time at loads where the LLMs fall below 0.1. Since caching repeated descriptions gives the interpreters nearly the same latency, Jev's gain lies in fresh decisions. These results support decision-model substitution for latency-bound admission on bounded contracts.
