Search papers, labs, and topics across Lattice.
This paper investigates the impact of system prompts on instruction-tuned language models (ILMs) and code language models (CLMs) for code generation across various configurations. The study reveals that increasing system prompt specificity does not guarantee improved correctness, few-shot examples can degrade performance in larger code-specialized models, and programming language significantly influences the sensitivity to system prompt variations. These findings challenge conventional wisdom and highlight the need for configuration-dependent and language-specific prompt engineering strategies.
Forget what you know about prompt engineering: more specific system prompts don't always improve code generation, and few-shot examples can actually *hurt* performance in large code models.
Instruction-tuned Language Models (ILMs) have become essential components of modern AI systems, demonstrating exceptional versatility across natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs -- commonly referred to as Code Language Models (CLMs) -- translate human intent into executable programs. While progress has been driven by advances in scaling and training methodologies, one critical aspect remains underexplored: the impact of system prompts on both general-purpose ILMs and specialized CLMs for code generation. We systematically evaluate how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect code assistant. Our experimental setting spans 360 configurations across four models, five system prompts, three prompting strategies, two languages, and two temperature settings. We find that (1) increasing system-prompt constraint specificity does not monotonically improve correctness -- prompt effectiveness is configuration-dependent and can help or hinder based on alignment with task requirements and decoding context; (2) for larger code-specialized models, few-shot examples can degrade performance relative to zero-shot generation, contrary to conventional wisdom; and (3) programming language matters, with Java exhibiting significantly greater sensitivity to system prompt variations than Python, suggesting language-specific prompt engineering strategies may be necessary.
