Search papers, labs, and topics across Lattice.
The paper introduces Pyramid MoA, a hierarchical Mixture-of-Agents architecture designed to optimize the cost-accuracy trade-off in LLM inference by dynamically routing queries to larger models only when smaller models exhibit low confidence or disagreement. This routing mechanism leverages semantic agreement and confidence calibration among an ensemble of smaller models to identify "hard" problems requiring escalation. Experiments on GSM8K demonstrate that Pyramid MoA achieves 93.0% accuracy, closely approaching the performance of a Llama-3-70B baseline (98.0%), while reducing compute costs by 61% with minimal latency overhead.
Slash LLM inference costs by 61% without sacrificing accuracy by dynamically escalating queries to larger models only when smaller models express uncertainty.
Large Language Models (LLMs) face a persistent trade-off between inference cost and reasoning capability. While"Oracle"models (e.g., Llama-3-70B) achieve state-of-the-art accuracy, they are prohibitively expensive for high-volume deployment. Smaller models (e.g., 8B parameters) are cost-effective but struggle with complex tasks. In this work, we propose"Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary. By leveraging semantic agreement and confidence calibration among an ensemble of small models, our Router identifies"hard"problems with high precision. On the GSM8K benchmark, our system achieves 93.0% accuracy, effectively matching the Oracle baseline (98.0%) while reducing compute costs by 61%. We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget.
