Search papers, labs, and topics across Lattice.
This study introduces the concept provenance framework to analyze LLM-generated search queries, identifying the presence of answer-side document knowledge that violates the information-access boundary for pre-search users. By applying this framework to a large dataset of 77,004 queries across various models and conditions, the authors reveal that 7.40% of concepts are candidate answer-side intrusions, with significant variance explained by topic. The findings indicate that while no prompt condition can fully eliminate intrusion, a post-generation selection process can achieve a remarkable 99% reduction in such intrusions.
7.40% of LLM-generated query concepts are found to be answer-side intrusions, challenging the validity of current information retrieval evaluations.
LLM-generated search queries are widely used to augment IR evaluation, yet they may contain concepts that presuppose answer-side document knowledge, violating the information-access boundary of pre-search users. Existing validation metrics, including overlap, diversity, and effectiveness, cannot distinguish rare human-tail variation from candidate answer-side intrusion. We introduce concept provenance, a framework that assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones, operationalizing a boundary that retrieval metrics alone cannot detect. Applying concept provenance to 77,004 queries across 100 UQV100 topics, 8 LLMs, and 5 prompt conditions with two extraction pipelines, we obtain a cross-pipeline token-HCIR Spearman rho of 1.0 over five condition means. Candidate answer-side concepts constitute 7.40 percent of non-generic concepts and appear in 97 of 100 topics, with topic explaining approximately 67 percent of variance. Human validation yields 68.2 percent relaxed precision, revealing two mechanisms: knowledge intrusion at 45.5 percent and deployment intrusion at 45.0 percent. Diagnostic probes show disproportionate localized retrieval effects, with deletion effect size d = -0.47 compared with d = -0.34 for random deletion, but these concepts explain less than 2 percent of aggregate evaluation variance. Concept provenance therefore serves as a boundary-compliance diagnostic rather than an evaluation-shift predictor. Under the tested conditions, no prompt condition eliminates intrusion; post-generation concept-provenance selection achieves 99 percent elimination.
