Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of Large Language Models (LLMs) in managing backend fault isolation for HAProxy, addressing the limitations of static load balancers that fail to handle degraded servers. By benchmarking 15 models across various architectures and sizes, the research identifies a capability threshold of approximately 3 billion parameters, above which LLMs achieve an 88% reduction in client-perceived 5xx errors compared to static policies. However, the findings also reveal trade-offs, such as increased tail latency and token costs, necessitating a careful balance between model complexity and operational efficiency.
LLMs can dramatically reduce backend error rates in load balancing, but only if they exceed a critical parameter threshold鈥攂elow that, they may perform worse than static policies.
Static load balancers cannot mitigate a backend that is degraded rather than down: round-robin and least-connections keep routing traffic to a server returning HTTP 500s until an operator intervenes. We ask whether a Large Language Model can replace the static routing policy itself, reading HAProxy and Prometheus telemetry every 10 seconds and isolating faulty servers through guardrailed calls to the HAProxy Data Plane API. On a reproducible benchmark with a persistent structural fault built into roughly one-third of a heterogeneous fleet, we sweep 15 open-weight models across five families (0.35B to 35B total parameters; dense, mixture-of-experts, and efficient-sparse architectures), reasoning modes, fleet scales of 3 to 9 backends, and two routing algorithms, totaling 240 runs. We find a capability threshold near 3B active parameters. Below it, LLM policies are typically unreliable and sometimes worse than no policy; above it, every model, regardless of architecture, saturates near an 88% reduction in client-perceived 5xx errors over the static baseline. The threshold is approximate: Gemma 4 E2B clears it with 2B active parameters, while the dense 3B Granite 4.0 Micro does not. The availability gain has costs. Draining concentrates load onto surviving servers, inflating tail latency 2.6 to 2.8 times, and enabling reasoning multiplies token spend roughly tenfold, overrunning the control interval and degrading effectiveness. The efficient operating point is a supra-threshold model in its cheapest non-reasoning mode, wrapped inside deterministic guardrails.
