Search papers, labs, and topics across Lattice.
The paper introduces SpecValidator, a parameter-efficiently fine-tuned small model, to detect defects in task descriptions used for LLM-based code generation across three categories: Lexical Vagueness, Under-Specification, and Syntax-Formatting. SpecValidator significantly outperforms larger models like GPT-3.5-turbo and Claude Sonnet 3 in defect detection (F1 = 0.804, MCC = 0.745) and demonstrates generalization to unseen Under-Specification defects, revealing that LLM robustness to defective descriptions depends more on the defect type and task description characteristics than model size. The study also highlights the importance of richer contextual grounding in task descriptions, as seen in LiveCodeBench, for enhancing LLM reliability in code generation.
Turns out, a tiny fine-tuned model can spot flaws in coding instructions that trip up even the biggest LLMs, suggesting we're over-relying on brute force for code generation.
Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we develop SpecValidator, a lightweight classifier based on a small model that has been parameter-efficiently finetuned, to automatically detect task description defects. We evaluate SpecValidator on three types of defects, Lexical Vagueness, Under-Specification and Syntax-Formatting on 3 benchmarks with task descriptions of varying structure and complexity. Our results show that SpecValidator achieves defect detection of F1 = 0.804 and MCC = 0.745, significantly outperforming GPT-5-mini (F1 = 0.469 and MCC = 0.281) and Claude Sonnet 4 (F1 = 0.518 and MCC = 0.359). Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used. Our results also show that the robustness of LLMs in task description defects depends primarily on the type of defect and the characteristics of the task description, rather than the capacity of the model, with Under-Specification defects being the most severe. We further found that benchmarks with richer contextual grounding, such as LiveCodeBench, exhibit substantially greater resilience, highlighting the importance of structured task descriptions for reliable LLM-based code generation.
