Search papers, labs, and topics across Lattice.
This study introduces LangChoiceBench, a benchmark designed to systematically measure programming-language preferences in large language models (LLMs), particularly focusing on Python. The evaluation of 25 diverse LLMs reveals a strong bias towards Python, low consistency between recommendations and implementations, and a notable tendency for smaller models to favor Python while exhibiting less language diversity. Additionally, the analysis of reasoning traces uncovers a phenomenon termed "phantom evidence," where models fabricate justifications for their language choices, highlighting significant flaws in their decision-making processes.
Python remains the overwhelming choice for code generation in LLMs, but many selections are based on convenience rather than project needs, revealing critical flaws in model reasoning.
Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.
