Search papers, labs, and topics across Lattice.
This study introduces the concept provenance framework to analyze LLM-generated search queries, identifying the presence of concepts that imply prior knowledge of answer-side documents, which can mislead pre-search users. By applying this framework to a large dataset of 77,004 queries across various topics and LLMs, the authors reveal that candidate answer-side concepts constitute 7.40% of non-generic concepts and significantly impact retrieval effectiveness. The findings highlight that while no prompt condition can fully eliminate knowledge intrusion, a post-generation selection process can achieve 99% elimination, underscoring the importance of boundary compliance in information retrieval systems.
Candidate answer-side concepts infiltrate 7.40% of LLM-generated queries, challenging the integrity of information retrieval evaluations.
LLM-generated search queries are widely used to augment IR evaluation, yet they may contain concepts that presuppose answer-side document knowledge, violating the information-access boundary of pre-search users. Existing validation metrics, including overlap, diversity, and effectiveness, cannot distinguish rare human-tail variation from candidate answer-side intrusion. We introduce concept provenance, a framework that assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones, operationalizing a boundary that retrieval metrics alone cannot detect. Applying concept provenance to 77,004 queries across 100 UQV100 topics, 8 LLMs, and 5 prompt conditions with two extraction pipelines, we obtain a cross-pipeline token-HCIR Spearman rho of 1.0 over five condition means. Candidate answer-side concepts constitute 7.40 percent of non-generic concepts and appear in 97 of 100 topics, with topic explaining approximately 67 percent of variance. Human validation yields 68.2 percent relaxed precision, revealing two mechanisms: knowledge intrusion at 45.5 percent and deployment intrusion at 45.0 percent. Diagnostic probes show disproportionate localized retrieval effects, with deletion effect size d = -0.47 compared with d = -0.34 for random deletion, but these concepts explain less than 2 percent of aggregate evaluation variance. Concept provenance therefore serves as a boundary-compliance diagnostic rather than an evaluation-shift predictor. Under the tested conditions, no prompt condition eliminates intrusion; post-generation concept-provenance selection achieves 99 percent elimination.
