Search papers, labs, and topics across Lattice.
This paper introduces OdinEval, a reproducible benchmark designed for evaluating LLM-based program repair specifically within the Odin programming language, which has been largely overlooked in existing benchmarks. The benchmark comprises 168 instances derived from documented defects in public Odin repositories, with a rigorous admission process ensuring that each test fails on the base revision and passes after applying the gold fix. Evaluation results show that Kimi-K3 achieves a Resolved score of 66.7%, while Qwen3.8-Max excels with a Repro score of 96.4%, highlighting the potential of LLMs in program repair tasks for less common programming languages.
OdinEval reveals that even in niche programming languages, LLMs can achieve impressive repair accuracy, with top models scoring over 66% in resolving defects.
Repository-level repair benchmarks still center on a few mainstream languages, leaving systems languages such as Odin largely untested. We present OdinEval, a reproducible benchmark built from documented defects in public Odin repositories. Each instance binds an issue to base and fix commits, a gold patch, an issue-specific regression test, a historical toolchain, and execution records. Admission requires the test to fail on the base revision and pass after the gold fix. When no usable developer test exists, a black-box test is reviewed independently by three instances of the same model, executed in both historical states, and revised from recorded feedback under a versioned Test Writing Skill. We evaluate six language models on 168 filtered instances under one shared protocol. Kimi-K3 records the highest Resolved score at 66.7%, while Qwen3.8-Max has the highest Repro score at 96.4%. The release includes frozen data, source archives, containers, validators, model patches, and audit manifests.
