Search papers, labs, and topics across Lattice.
This paper investigates how memory consolidation in LLM agents transforms tentative statements into confident assertions that the agents subsequently treat as verified facts. The authors reveal that this manufactured confidence can lead agents to act on incorrect information without the need for adversarial manipulation, as the phrasing of the information鈥攔ather than its source鈥攄etermines the level of trust. Their findings highlight that maintaining hedged language in memory storage could prevent the propagation of false confidence, while a single authoritative memory source can restore accurate decision-making.
Confidence in LLMs can be manufactured from hearsay, leading agents to treat unverified claims as established facts, with serious implications for reliability.
LLM agents carry conclusions across steps and sessions in compressed memory, and memory products (e.g., mem0, LangMem) rewrite conversation into stored "facts" that later steps trust. We show this rewriting manufactures confidence: across our constructed agent settings, a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact, granting every above-clearance request it faces. No attacker is needed: a role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection. We then isolate what the agent responds to. It is not the source: attributed, unattributed, and even forged "system of record" claims all grant alike. It is the confidence of the phrasing. A hedge is discounted, a flat assertion is obeyed, and this holds with no special keyword. Not all hedges are equal, though: the evidential register is the least-discounted, with "reportedly" obeyed like a flat assertion on most models. The obvious fixes fail. A passive "unverified" tag is ignored, and an active "do not trust this" instruction escalates even correct memory, so it is safe only by refusing to decide. The real fix lives in the store: keep the tentative phrasing rather than upgrade it. But that is hygiene, not a defense against an attacker who can simply write a confident lie. The deployable lesson is narrower and constructive: a single load-bearing memory is the hazard, and one redundant source restores correct decisions. We release the harness and demonstrations.
