Search papers, labs, and topics across Lattice.
This paper introduces XREPOTEST, a comprehensive multilingual benchmark for evaluating unit test generation by large language models (LLMs) across five programming languages: Rust, Go, Julia, PHP, and Ruby. By employing a containerized execution framework and various context augmentation strategies, the study reveals significant discrepancies between LLM performance in standalone versus repository-level settings, highlighting the importance of realistic constraints in test generation. The novel metric Invocation Rate (IR) is proposed to better assess the effectiveness of generated tests, demonstrating a critical need for improved methodologies in automated testing.
LLMs show a stark performance drop in realistic repository-level unit test generation, revealing hidden challenges in their deployment across diverse programming languages.
Large language models (LLMs) have shown promise for automated unit test generation, but existing evaluations largely rely on standalone settings and a narrow set of programming languages, overestimating real-world readiness. We introduce XREPOTEST, a multilingual repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby. XREPOTEST evaluates tests under realistic repository constraints using a containerized execution framework and multiple context augmentation strategies, including file-level, LSP-based, and retrieval-based context. Beyond standard metrics such as test pass rate and coverage, we propose Invocation Rate (IR) to assess whether generated tests meaningfully exercise the intended functionality. Experiments with 14 state-of-the-art LLMs, including Claude 4.5, GPT-5.2, DeepSeek V4-Pro, and Qwen families, reveal a substantial gap between standalone and repository-level performance, as well as trade-offs between richer context and test reliability. Overall, XREPOTEST provides a challenging and informative benchmark to advance scalable and robust unit test generation in realistic software environments. The dataset and code are publicly available at: https://github.com/solis-team/XRepoTest
