Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the generalization limitations of automated prompt optimization methods across different tasks and LLMs. Through a causal inference-inspired observational analysis of prompt edits, the authors identify consistent task-conditioned edit patterns associated with performance improvements or declines. They find that complexity-increasing edits hurt mathematical reasoning, while step-by-step edits improve logical reasoning, suggesting that prompt optimization failures stem from interactions between edit types and task characteristics.
Prompt optimization's failures aren't random flukes, but predictable consequences of how specific edits interact with task demands.
Automated prompt optimization methods (e.g., DSpy, TextGrad) can substantially improve the performance of large language model (LLM), however, their generalization ability across different tasks remains underperformed. In practice, the superiority of the optimized prompt on one benchmark often fails to transfer to another, and this limitation persists even when switching across different LLM backbones. To investigate the underexplored sources of heterogeneity in prompt performance, we conduct a causal inference-inspired observational analysis of optimized prompts across a diverse set of optimization frameworks, LLM backbones, and NLP benchmarks. To achieve the goal, we build upon the propensity-adjusted associational analysis together with multiple complementary representations of prompt edits, where the consistent task-conditioned edits patterns are identified. We find that complexity-increasing and meta-instructional edits are negatively associated with mathematical and multi-hop reasoning performance, whereas step-by-step and meta-cognitive edits improve logical and sequential reasoning tasks. These effects are robust across cognitive-load annotations, surface-level text features, and edit-motif analyses, and can generalize across optimization frameworks. Overall, these results indicate that prompt optimization failures arise from systematic interactions between edit families and task characteristics rather than random optimization artifacts, providing feature-level characterization of optimizer behavior and motivating future task-conditioned optimizer design.