Search papers, labs, and topics across Lattice.
Tsinghua AIAdobe ResearchBJTUHKUInfinigence AIPKUShanghai Jiaotong UniversityTokenRhythm TechnologiesJun 10, 2026arXiv:2606.12344This paper introduces Claw-SWE-Bench, a benchmark designed to evaluate OpenClaw-style agent harnesses on coding tasks, addressing the limitations of existing evaluation methods that fail to account for the specific requirements of coding environments. The benchmark includes 350 instances across multiple programming languages and incorporates a structured protocol to ensure fair comparisons among heterogeneous agent harnesses. Key findings reveal that the design of the adapter significantly impacts performance, with the full adapter achieving a Pass@1 score of 73.4%, compared to just 19.1% with a minimal adapter, highlighting the importance of harness design in coding task effectiveness.
Adapter design can make or break coding performance in OpenClaw-style agents, with a full adapter boosting success rates by over 50 percentage points.
General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. We also release Claw-SWE-Bench Lite for faster validation, which is an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns. On the full benchmark, OpenClaw with a minimal direct-diff adapter scores only 19.1% Pass@1, whereas the full adapter reaches 73.4% with the same GLM 5.1 backbone, showing that adapter design is essential for enabling OpenClaw-style harnesses to perform coding tasks effectively. Across an OpenClaw times nine-model sweep and a five-claw times two-model sweep, model choice changes Pass@1 by 29.4 pp and harness choice by 27.4 pp under fixed models; systems with similar accuracy can differ substantially in total API cost. Claw-SWE-Bench therefore treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation, providing both a full benchmark and a low-cost reference set for reproducible comparison. The data is available at https://github.com/opensquilla/claw-swe-bench and https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.
