Search papers, labs, and topics across Lattice.
This paper critically examines the limitations of existing training datasets for generating code review comments, revealing that many instances suffer from misalignment between code changes and comments, leading to ineffective learning signals. The authors propose a taxonomy of misalignment that includes semantic ambiguity, lack of actionability, and context dependence, highlighting the need for a more nuanced understanding of training data quality. Their findings suggest that merely cleaning datasets is insufficient; instead, they advocate for enhanced contextual inputs and evaluation methods that prioritize actionable feedback in code reviews.
Misaligned training pairs in code review datasets can undermine the effectiveness of LLMs, revealing that dataset cleaning alone won't solve the problem of generating actionable feedback.
Generating code review comments has become a prominent research direction in automated code review, commonly formulated as a text generation task over diff-comment pairs. Despite advances in learning-based approaches, generated review comments are often generic, weakly grounded, or non-actionable. Recent studies have also shown that review comment datasets contain noisy or unsuitable training instances, motivating LLM-based dataset cleaning approaches. In this paper, we argue that problematic training instances are not homogeneous and that some limitations stem from deeper issues in the task formulation itself. Through an empirical inspection of a widely used review comment dataset, we identify misaligned training pairs: instances where the relationship between the code change and the review comment does not provide a reliable learning signal for generating actionable review feedback from localized inputs. We derive a taxonomy of misalignment capturing three recurring sources: semantic ambiguity, lack of actionability, and context dependence. We further explore whether incorporating this taxonomy into LLM-based filtering improves the identification of problematic training instances, observing that detecting misaligned training pairs remains challenging. Based on these observations, we argue that improving review comment generation requires more than dataset cleaning alone, motivating explicit validity criteria, richer contextual inputs, and evaluation practices aligned with review intent and actionability.
