Search papers, labs, and topics across Lattice.
This paper introduces a Bayesian framework for confidently migrating production LLM-based systems to new models, addressing the challenge of model obsolescence. The framework calibrates automated evaluation metrics using limited human judgments to enable accurate model comparison across key dimensions like correctness, refusal behavior, and style. Applied to a large-scale question-answering system, the framework successfully identified suitable replacement models, demonstrating its practical utility for enterprises managing LLM portfolios.
Stop sweating LLM migrations: this Bayesian framework lets you confidently swap models in production, even with limited human evals.
We present a framework for migrating production Large Language Model (LLM) based systems when the underlying model reaches end-of-life or requires replacement. The key contribution is a Bayesian statistical approach that calibrates automated evaluation metrics against human judgments, enabling confident model comparison even with limited manual evaluation data. We demonstrate this framework on a commercial question-answering system serving 5.3M monthly interactions across six global regions; evaluating correctness, refusal behavior, and stylistic adherence to successfully identify suitable replacement models. The framework is broadly applicable to any enterprise deploying LLM-based products, providing a principled, reproducible methodology for model migration that balances quality assurance with evaluation efficiency. This is a capability increasingly essential as the LLM ecosystem continues to evolve rapidly and organizations manage portfolios of AI-powered services across multiple models, regions, and use cases.
