Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of compound prompt constraints—formatting, persona, and urgency—on the reliability of code generation by large language models (LLMs). By conducting a full-factorial analysis across 27 combinations using 164 HumanEval+ problems, the authors reveal that these constraints can lead to architecture-dependent performance degradation that is not evident in single-factor assessments. Notably, the GPT-4o family shows significant super-additive effects, with reductions in pass rates that highlight the need for compound-prompt testing in LLM reliability evaluations.
Compound prompt constraints can degrade LLM performance by up to 12 percentage points, revealing that architecture matters more than size in code generation reliability.
Large language models (LLMs) are increasingly used in software engineering pipelines for code generation, where production prompts often combine multiple constraints. This paper presents a full-factorial empirical study of how output formatting, persona assignment, and urgency framing jointly affect LLM code-generation reliability. We evaluate all 27 combinations in a controlled 3x3x3 design and decompose each compound condition into an additive prediction and a residual interaction term that captures super-additive degradation. The study uses all 164 HumanEval+ problems across five OpenAI models from the GPT-4o family, GPT-4.1 family, and o3-mini, yielding 22,140 greedy-decoding evaluations. A format-aware extraction pipeline separates formatting failures from reasoning failures, and significance is assessed with McNemar's test, odds ratios, and 95% confidence intervals. Results show that compound constraints can produce architecture-dependent degradation not predictable from single-factor experiments. The GPT-4o family exhibits consistent super-additive effects, with pass@1 reductions 3-12 percentage points beyond additive predictions; the largest interaction is -12.2 pp on GPT-4o-mini for JSON + expert persona + moderate urgency. JSON combinations generally produce larger interactions than XML. In contrast, the GPT-4.1 family is largely resistant, while o3-mini shows a qualitatively different pattern in which structured output constraints can improve performance. These findings show that vulnerability is architecture-dependent rather than size-dependent, that individually neutral or beneficial constraints can combine to cause substantial degradation, and that compound-prompt testing should be standard in reliability assessment for LLM-assisted engineering pipelines.