Search papers, labs, and topics across Lattice.
This paper introduces Atropos, a method for improving the cost-benefit trade-off of LLM-based agents using self-consistency by predicting inference success and hotswapping to a more capable LLM when failure is predicted. Atropos uses a Graph Convolutional Network (GCN) to analyze the graph structure of merged inference paths and predict whether an ongoing inference will succeed. Experiments on three LLM-based agents demonstrate that Atropos achieves 74.35% of the performance of closed LLMs at only 23.9% of the cost by accurately predicting early termination and successfully hotswapping up to 27.57% of failing inferences.
Get 74% of GPT-4 performance for just 24% of the cost by predicting when a cheaper, faster model is about to fail and seamlessly "hotswapping" the task to a more powerful one.
Open-weight Small Language Models(SLMs) can provide faster local inference at lower financial cost, but may not achieve the same performance level as commercial Large Language Models (LLMs) that are orders of magnitudes larger. Consequently, many of the latest applications of LLMs, such as software engineering agents, tend to be evaluated on larger models only, leaving the issue of improving the cost-benefit trade-off of such applications neglected. This paper proposes Atropos, a predictive early-termination analysis and hotswap technique that aims to improve the cost-benefit trade-off for LLM-based agents that use self-consistency. The core component of ATROPOS is a predictive model based on structural properties of LLM inferences: after merging multiple agentic inference paths into a graph representation, ATROPOS uses Graph Convolutional Network (GCN) to predict whether an ongoing inference will eventually succeed or not. If an agentic task instance running on the source LLM is predicted to fail, ATROPOS subsequently performs hotswapping, i.e., migrating the on-going inference context onto the more capable target LLM: this is feasible because LLM contexts are stateless. An empirical evaluation of ATROPOS using three recent LLM-based agents shows that ATROPOS can predict early termination of eventually failing inferences with the accuracy of 0.85 at the midpoint of the inference. Hotswapping LLMs for such inferences can convert up to 27.57% of them to be successful. Consequently, ATROPOS achieves 74.35% of the performance of closed LLMs with as low as only 23.9% of the cost.
