Search papers, labs, and topics across Lattice.
This paper introduces a probabilistic programming language designed for LLM-based application flows, addressing the challenge of uncertainty in LLM outputs. By allowing developers to quantify and propagate uncertainty without altering the underlying flow logic, it enhances the reliability of applications that utilize multiple LLM calls. The experimental study and case study involving a theorem proving agent for the Rocq theorem prover illustrate the practical benefits of this approach in improving accuracy and trustworthiness in LLM applications.
Uncertainty in LLM outputs can now be quantified and managed seamlessly, transforming how developers build reliable AI applications.
Building reliable applications that leverage large language models (LLMs) remains a significant challenge. While LLMs offer impressive capabilities across diverse tasks, their outputs often lack accuracy and provide no clear measure of confidence. This uncertainty compounds in flows of multiple calls to LLMs and other tools, making it difficult for developers and end-users to trust the results. This paper introduces a probabilistic language for programming LLM-based flows. It enables developers to quantify and propagate uncertainty throughout the application's flow, and experiment with different inference scaling techniques without adding a single line of code beyond the flow's logic. We present an experimental study to demonstrate this capability, and a case study building a theorem proving agent for the Rocq theorem prover.
