Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates whether hints in code generation merely guide models to solutions they can already find or provide essential missing information. Using HumanEval+ and MBPP+, the authors show that while relevant hints can significantly rescue failing code samples, many of these solutions are also achievable through unhinted sampling, indicating that hints do not necessarily enhance the model's underlying capabilities. Mechanistic analyses reveal that while hints influence model activation, they do not lead to a net gain in accuracy or establish robust task-general transfer of skills.
Relevant hints can rescue failing code generation attempts, but most solutions are already accessible through standard sampling methods, challenging assumptions about their instructional value.
When a hint turns a failing generated program into a passing one, does it provide missing information or merely steer the model toward a solution it could already produce? We test these hypotheses on HumanEval+ and MBPP+ using executable evaluation. For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures; an unrelated hint rescues 19, while eight unhinted samples solve 46 and recover 31 of the 36 relevant-hint rescues. Phi-3.5-mini shows the same pattern: relevant hints rescue 42 of 101 failures, an unrelated hint rescues 17, and unhinted sampling solves 57, including 36 of the 42 relevant-hint rescues. Because the hint conditions use different attempt budgets, these comparisons do not isolate a purely semantic effect. Mechanistic tests on Qwen identify a stable activation direction shared by relevant and unrelated hints. Persistently adding this direction yields 14 rescues and 18 regressions, with no detectable net accuracy gain; learned low-rank interventions have a positive but imprecise estimated effect. Full textual specifications solve 22 of 24 context-defined problems, versus 5-11 for tested virtual-KV prefixes. Post-generation hidden-state probes transfer across benchmarks, with pooled AUROC 0.806 and 0.780, but their top-one selection advantage over token confidence is statistically unresolved. Overall, relevant hints can rescue failures, but most rescued solutions are already reachable through ordinary sampling, and the internal interventions tested here do not establish task-general capability transfer.