Search papers, labs, and topics across Lattice.
This paper introduces the concept of Conversational Reference Grounding (CoRG), which addresses the challenge of resolving indirect references in collaborative conversations by leveraging external tools and contextual information. The authors create the RepoRef benchmark, consisting of 400 developer-chat segments that require agents to utilize multiple tools to identify the intended references, demonstrating the complexity of integrating lexical, semantic, and temporal cues. Results indicate that even the most advanced agents achieve only a 67.0% success rate, highlighting the ongoing difficulties in reference resolution within multi-tool environments.
Current agents struggle with indirect references in conversations, achieving only a 67% success rate in resolving them, revealing significant gaps in their contextual understanding.
Collaborative conversations frequently contain references whose targets are indirect rather than named: resolving "this looks like the fix discussed yesterday" requires combining conversational context with evidence from the surrounding workspace which is accessible through APIs or user interfaces. We formalize this problem as Conversational Reference Grounding (CoRG): using a given set of tools to resolve a reference in conversation to the unique external item intended by the speaker. CoRG is challenging because it combines lexical, semantic, and temporal cues distributed across the conversation and the external workspace. Agents must translate these heterogeneous signals into effective tool use: formulating strategies, discovering plausible candidates, inspecting their metadata and content, and ruling out close alternatives. We study CoRG through RepoRef, a benchmark of 400 developer-chat segments grounded in GitHub issues, pull requests, and commits across 92 repositories. Unlike single-shot retrieval tasks, RepoRef often requires multi-step tool use. Our results show that CoRG remains challenging for current agents, even the best agent reaches only 67.0% success rate, leaving one third of references unresolved. These findings position CoRG as a concrete benchmark for studying how agents search, inspect, and verify information in realistic multi-tool environments.
