Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces "entity-collision," a novel protocol for benchmarking agent memory retrievers that isolates the impact of embedding models by controlling for lexical leakage and stratifying queries by discriminator tag. This protocol ensures that any performance lift over a BM25 baseline is directly attributable to the embedder's ability to capture semantic similarity beyond simple token overlap. Experiments using this protocol reveal that larger embedding models do not always outperform smaller ones, with MiniLM-384 showing superior performance on lexical queries compared to BGE-large, and that adaptive vector-weight routing on LoCoMo fails to recover existing oracle headroom.
Bigger embedding models don't always win in agent memory retrieval: MiniLM-384 beats BGE-large on lexical queries, suggesting that encoder capacity isn't the only bottleneck.
End-to-end agent-memory benchmarks report a single hit@k per retriever, confounding lexical leakage (uncontrolled query/gold/distractor entity overlap) with tag-mixing (preferences, services, tools averaged together). We propose entity-collision, a system-agnostic protocol that pins the BM25 floor by construction -- every distractor shares the answer's entity tokens -- and stratifies queries by discriminator tag, so any lift over BM25 is attributable to the embedder. Applied to an open-source agent-memory testbed across 5 tags x 3 embedders x 5 collision degrees with paired-bootstrap 95% CIs, the protocol reveals a two-axis pattern: a 256-d hash trigram helps only on closed-vocabulary lexical tags at deep collision; MiniLM-384 dominates both axes; and a 2.7x-parameter BGE-large does not uniformly improve on MiniLM -- it wins on intent-style queries but loses on lexical ones. Encoder capacity alone is not the binding constraint. The synthetic intent-tag null replicates on LongMemEval (n=500) as a single-session-preference recall cliff. Adaptive vector-weight routing on LoCoMo is a measured null: 11.7pp of oracle headroom exists, but no signal we tested recovers it. All 26 result tables and 37 reproduce scripts are version-controlled and verified by a public registry; the protocol is exercised on a deterministically governed memory testbed (event-sourced decision log, DAG-state-machine schema lifecycle) so every reported CI is reproducible byte-for-byte from the ingest stream.