Search papers, labs, and topics across Lattice.
This paper addresses the growing misalignment between traditional Information Retrieval (IR) systems, designed for human queries, and the emerging landscape of agentic search. To bridge this gap, the authors introduce a methodology for collecting data from agentic retrieval-augmented systems and release the Agentic Search Queryset (ASQ) dataset. ASQ comprises reasoning-induced queries, retrieved documents, and agent "thoughts" across HotpotQA, Researchy Questions, and MS MARCO, for three agents and two retrieval pipelines, along with a toolkit for extending the dataset.
IR systems optimized for humans may fail to satisfy AI agents, necessitating new datasets like ASQ that capture the nuances of agentic search behavior.
With automated systems increasingly issuing search queries alongside humans, Information Retrieval (IR) faces a major shift. Yet IR remains human-centred, with systems, evaluation metrics, user models, and datasets designed around human queries and behaviours. Consequently, IR operates under assumptions that no longer hold in practice, with changes to workload volumes, predictability, and querying behaviours. This misalignment affects system performance and optimisation: caching may lose effectiveness, query pre-processing may add overhead without improving results, and standard metrics may mismeasure satisfaction. Without adaptation, retrieval models risk satisfying neither humans, nor the emerging user segment of agents. However, datasets capturing agent search behaviour are lacking, which is a critical gap given IR's historical reliance on data-driven evaluation and optimisation. We develop a methodology for collecting all the data produced and consumed by agentic retrieval-augmented systems when answering queries, and we release the Agentic Search Queryset (ASQ) dataset. ASQ contains reasoning-induced queries, retrieved documents, and thoughts for queries in HotpotQA, Researchy Questions, and MS MARCO, for 3 diverse agents and 2 retrieval pipelines. The accompanying toolkit enables ASQ to be extended to new agents, retrievers, and datasets.
