Search papers, labs, and topics across Lattice.
This paper introduces XREPOTEST, a comprehensive multilingual benchmark for evaluating unit test generation by large language models (LLMs) across five programming languages: Rust, Go, Julia, PHP, and Ruby. By employing a containerized execution framework and various context augmentation strategies, the study reveals significant discrepancies between LLM performance in isolated and repository-level settings, highlighting the importance of realistic constraints. Key findings indicate that while richer context can enhance test generation, it may also compromise test reliability, underscoring the need for more robust evaluation metrics like Invocation Rate (IR).
LLMs show a stark performance drop in unit test generation when evaluated in realistic repository contexts, revealing a gap that could hinder practical deployment.
Large language models (LLMs) have shown promise for automated unit test generation, but existing evaluations largely rely on standalone settings and a narrow set of programming languages, overestimating real-world readiness. We introduce XREPOTEST, a multilingual repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby. XREPOTEST evaluates tests under realistic repository constraints using a containerized execution framework and multiple context augmentation strategies, including file-level, LSP-based, and retrieval-based context. Beyond standard metrics such as test pass rate and coverage, we propose Invocation Rate (IR) to assess whether generated tests meaningfully exercise the intended functionality. Experiments with 14 state-of-the-art LLMs, including Claude 4.5, GPT-5.2, DeepSeek V4-Pro, and Qwen families, reveal a substantial gap between standalone and repository-level performance, as well as trade-offs between richer context and test reliability. Overall, XREPOTEST provides a challenging and informative benchmark to advance scalable and robust unit test generation in realistic software environments. The dataset and code are publicly available at: https://github.com/solis-team/XRepoTest
