Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a new benchmark, Selective QA over Conflicting Multi-Source Personal Memory, to evaluate how well personal AI agents resolve conflicting or incomplete information from multiple sources. The benchmark contains 34,560 instances with controlled source distortions and deterministic ground truth, covering 18 question templates and 8 reasoning types. Experiments show that while trained fusion resolvers outperform prompt-only LLMs in accuracy (80.3% vs 70.0%), LLMs achieve higher coverage with abstention (95.4% vs 78.3%), highlighting different strengths across reasoning types.
Personal AI agents still struggle to reconcile conflicting information from multiple sources, with even the best LLMs only achieving 70% accuracy on a new benchmark designed to isolate this challenge.
Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method's conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse.