Search papers, labs, and topics across Lattice.
This paper conducts a comprehensive empirical evaluation of six leading LLM-based coding assistants, analyzing their performance across 1,847 code generation tasks in various programming languages and complexity levels. The evaluation focuses on functional correctness, code quality, computational efficiency, and prompt robustness, revealing that while Claude 3.5 Sonnet leads in overall pass rates, GPT-4 outperforms in complex algorithmic challenges. Additionally, the study highlights significant performance drops on adversarial prompts and varying security vulnerability rates, offering a benchmark suite, CodeEval-1847, for future assessments.
Claude 3.5 Sonnet tops the charts in code generation accuracy, but GPT-4 reigns supreme in tackling complex algorithms, revealing a nuanced landscape of LLM capabilities.
Large Language Models (LLMs) have trans-formed software development through AI-powered code generation, yet systematic comparisons of their capabilities remain limited. We present a comprehensive empirical evaluation of six leading LLM-based coding assistants—GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, CodeLlama-70B, DeepSeek Coder, and Mistral Large—across 1,847 code generation tasks spanning five programming languages and eight complexity tiers. Our evaluation framework assesses functional correctness (pass@k), code quality (maintainability, security), computational efficiency, and prompt robustness. Key findings reveal: (1) Claude 3.5 Sonnet achieves the highest overall pass@1 rate (84.7%) but GPT-4 excels in complex algorithmic tasks; (2) all models exhibit significant performance degradation (18–34%) on adversarial prompt variations; (3) security vulnerability rates range from 3.2% (Claude) to 11.8% (CodeLlama); and (4) open-source models achieve 73–81% of proprietary model performance at substantially lower cost. We release our benchmark suite, CodeEval-1847, comprising novel problems to prevent data contamination. Our findings provide actionable guidance for practitioners selecting AI coding tools and highlight critical areas for model improvement.
