Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This tertiary review synthesizes 30 secondary studies on LLM-based code generation tasks (CGTs) published between 2017-2025, revealing a rapid growth in the field since 2023. The analysis, structured using SWEBOK and HELM, highlights strong benchmark accuracy but weak real-world generalization, fragile robustness, and under-reported toxicity and bias. The review identifies key challenges in economic feasibility, evaluation validity, and socio-technical integration, pointing towards future research directions focused on domain-aware model improvement and holistic evaluation.
LLM-generated code may ace benchmarks, but struggles with real-world generalization, fragile robustness, and under-reported toxicity, revealing a critical gap between research and practical application.
Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs.