Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a semantic map that bridges the gap between language model probabilities and meaningful state probabilities necessary for professional decision-making. By employing semiparametric inference, the authors establish a framework that allows for the calibration of language-derived probabilities against a reference model, ensuring that the resulting posterior estimates are reliable and stable under paraphrasing. The findings demonstrate that language-derived probabilities not only outperform traditional numerical probabilities but also maintain valid uncertainty coverage and adapt appropriately to changes in evidence, highlighting the importance of semantic stability in professional contexts.
Language-derived probabilities can significantly outperform traditional numerical confidence measures, offering a more reliable foundation for professional decision-making.
Language models produce probabilities over words, but professional decisions require uncertainty over meaningful states such as diagnoses, hypotheses or operational conditions. A model's printed numerical confidence does not establish reliability. We introduce a semantic map: a prespecified, testable bridge from probabilities over verbal responses to probabilities over declared states, formulated as semiparametric inference for a finite-valued latent state. A reference model defines the target posterior, the language model supplies an unrestricted conditional distribution over verbal responses, and held-out calibration connects them. We derive posterior-error bounds and conditions for existence, uniqueness, stability and sequential Bayesian updating. Crucially, language probabilities depend on the prompt's lexical form, whereas the target posterior is unchanged by information-equivalent rewording. We test the method on professional market text compiled from Federal Reserve economic and financial series and on controlled simulations with exact posteriors. Across two fitted language models, language-derived probabilities outperform printed numerical probabilities, recover held-out posteriors with valid uncertainty coverage, remain largely stable under paraphrasing, and respond appropriately to altered evidence. The broader implication is that prompt engineering optimizes a wording-dependent response, whereas scientific and professional use requires validated stability of application-relevant meaning. The semantic map turns this general concern into a testable statistical problem and, when its acceptance conditions hold, yields an auditable posterior estimate. The same principle offers a template for auditing classifications, recommendations and other fluent responses that may conceal semantic instability.