Search papers, labs, and topics across Lattice.
The paper introduces Hybrid Utility Minimum Bayes Risk (HUMBR), a framework for mitigating hallucination in LLMs within high-stakes enterprise workflows. HUMBR combines semantic embedding similarity and lexical precision to identify consensus among LLM outputs without relying on ground truth references, along with derived error bounds. Empirical evaluations on public benchmarks and Meta production data demonstrate that HUMBR significantly outperforms Universal Self-Consistency, achieving a preference rate of 81% over human-crafted ground truth and virtually eliminating critical recall failures.
Hallucinating LLMs in enterprise workflows can be tamed by a new Hybrid Utility Minimum Bayes Risk (HUMBR) framework that synthesizes semantic and lexical signals to achieve consensus without ground truth.
Although LLMs drive automation, it is critical to ensure immense consideration for high-stakes enterprise workflows such as those involving legal matters, risk management, and privacy compliance. For Meta, and other organizations like ours, a single hallucinated clause in such high stakes workflows risks material consequences. We show that by framing hallucination mitigation as a Minimum Bayes Risk (MBR) problem, we can dramatically reduce this risk. Specifically, we introduce a Hybrid Utility MBR (HUMBR) framework that synthesizes semantic embedding similarity with lexical precision to identify consensus without ground-truth references, for which we derive rigorous error bounds. We complement this theoretical analysis with a comprehensive empirical evaluation on widely-used public benchmark suites (TruthfulQA and LegalBench) and also real world data from Meta production deployment. The results from our empirical study show that MBR significantly outperforms standard Universal Self-Consistency. Notably, 81% of the pipeline's suggestions were preferred over human-crafted ground truth, and critical recall failures were virtually eliminated.
Meta AI