Search papers, labs, and topics across Lattice.
This study investigates the effects of prompt language on code generation quality from large language models (LLMs) by analyzing 460 coding tasks in Python and Java, translated into multiple languages. The research reveals that English prompts do not always yield the highest functional correctness or code quality, with outcomes varying based on both the programming language and the specific LLM used. Notably, the generated code often incorporates a mix of English and the prompt language, highlighting the complexities of language bias in code generation contexts.
English isn't always the best choice for generating high-quality code鈥攍anguage bias significantly impacts LLM performance across programming tasks.
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.
