Search papers, labs, and topics across Lattice.
This study investigates how code language models, specifically Qwen2.5 and Qwen3.6, utilize public tests in test-driven code generation across various benchmarks. By comparing the effects of different types of prompts鈥攏atural language, relevant tests, and synthetic tests鈥攖he authors find that while visible tests can enhance performance in some cases, their overall impact is inconsistent and often minimal. Notably, the research reveals that increased representation shifts do not necessarily correlate with improved accuracy, highlighting a complex relationship between test usage and model behavior.
Code models may leverage tests more for semantic guidance than as executable specifications, with surprising implications for their performance consistency.
Public tests are widely used to guide large language model code generation, but whether models treat them as executable specifications or merely as extra prompt context remains unclear. We study test-driven code generation on HumanEval+, MBPP+, and recent LiveCodeBench tasks using Qwen2.5-Coder-7B and Qwen3.6-27B. We compare natural-language-only prompts with relevant visible tests, shuffled outputs, irrelevant tests, assertion-only tests, and stronger-model-generated synthetic tests. Evaluation combines hidden or private test pass rates with task-level behavior flips, linear probes, and layer-wise hidden-state shifts. Visible tests substantially improve Qwen2.5 performance on MBPP+ but have little or unstable effect on HumanEval+ and LiveCodeBench. For Qwen3.6 on LiveCodeBench, the natural-language-only pass rate rises from 13.1% to 39.4%, yet relevant original tests add only 2.9 percentage points (p = .458), and synthetic high5 tests add 1.7 points over their matched baseline (p = .701). More tests produce larger representation shifts, but accuracy gains quickly saturate or disappear; assertion-only and irrelevant-test prompts can strongly alter hidden states without improving correctness. After controlling for prompt length, task-level shifts are associated with both beneficial and harmful flips, with a substantially stronger association for harms. These results show that tests influence code models through both semantic guidance and prompt-context perturbation, and that representational change alone does not demonstrate effective test utilization.
