Search papers, labs, and topics across Lattice.
This paper investigates the critical "narration gap" in the interaction between language models and formal solvers, which can lead to the loss of soundness in the answers provided to users. By modeling the LLM-solver loop as a verified decision procedure, the authors evaluate five open-source models under prompt injection attacks, revealing that while certificate gating can maintain soundness, adversaries can still manipulate the output across different phrasings. The findings highlight that despite efforts to mitigate risks through hardened prompts, robustness does not extend to the final user-facing answers, raising concerns about the reliability of hybrid reasoning systems in safety-critical applications.
The interaction between language models and formal solvers can compromise the soundness of answers, even when using certificate gating to ensure correctness.
Formal tools such as SAT and SMT solvers are increasingly embedded in language model reasoning pipelines when a safety or security critical question can be formulated in logic. Unlike chain of thought whose steps are sampled from the model distribution without formal guarantee, a solver produces a sound and independently verifiable answer. However, the soundness guarantee can be lost in the interaction between the solver and the model. The hybrid pipeline has three components: formalizing the question, deciding it, and narrating the result. Prior work has studied the formalization and decision, but not narration, which is the step that turns a formal tool's output into the user answer. To fill the narration gap, we first model the LLM-solver loop as a verified decision procedure. We further evaluate five open-sourced models under prompt injection, and we find certificate gating makes the solver verdict sound, while an adversary can invert a verified conclusion across phrasings and channels. We study the mitigation through hardened prompt that reduces injection significantly but cannot eliminate it and still suffers under adaptive attack. Combining the formal analysis and empirical studies, we show in the LLM-solver loop, robustness does not reach to the answer that the user finally reads.
