Search papers, labs, and topics across Lattice.
Investigating why self-generated natural language impacts LLM code generation, this work evaluates comment frequency, intent, and cross-model comment injection across LiveCodeBench. Controlled transplant experiments reveal that while comment surface features do not predict pass@1, comments extracted from passing solutions boost recipient model performance by 17.2%, whereas failed-solution comments offer no gain and irrelevant comments degrade accuracy by 20.8%. Crucially, extensive prompt engineering recovers at most 24% of this benefit, indicating that comments assist models by injecting elusive algorithmic content rather than functioning as generic reasoning scaffolding.
In-code comments only help LLMs when they smuggle in correct downstream solution content that the model could not otherwise elicit, debunking the idea that self-commentary acts as general chain-of-thought scaffolding.
Large Language Models (LLMs) often generate natural-language comments while writing code, and these comments become part of the context used to generate the code that follows. However, it remains unclear which properties of comments affect code-generation performance. We study this question through observational analyses and controlled interventions. On LiveCodeBench, neither comment frequency nor broad comment intent reliably predicts pass@1. We then prefill weaker recipient models with comment blocks written by stronger source models, allowing us to separate comment surface form from the solution content they convey. Comments from source solutions that pass the tests raise recipient pass@1 by 17.2% on average. In contrast, comments describing failed solutions provide no reliable gain, while comments written for a different problem reduce pass@1 by 20.8%. Finally, across a wide range of models and prompt variants, most recipient models show no significant recovery of the external-comment gain, and the best case recovers only 24%. These results show that comments help code generation not merely because they are comments, but because they can provide correct solution content that prompting cannot reliably elicit.
