Search papers, labs, and topics across Lattice.
The paper uses a custom Genetic Pareto (GEPA) algorithm to optimize prompts for scientific reasoning tasks in LLMs, aiming to understand how prompting affects reasoning behavior. They find that optimized prompts often induce model-specific heuristics ("local logic") that boost performance but fail to generalize across different LLMs. This highlights the challenge of relying on prompt engineering for robust scientific reasoning in LLMs.
Scientific reasoning gains from prompt engineering are often mirages, driven by model-specific hacks that don't generalize.
As Large Language Models (LLMs) achieve increasingly sophisticated performance on complex reasoning tasks, current architectures serve as critical proxies for the internal heuristics of frontier models. Characterizing emergent reasoning is vital for long-term interpretability and safety. Furthermore, understanding how prompting modulates these processes is essential, as natural language will likely be the primary interface for interacting with AGI systems. In this work, we use a custom variant of Genetic Pareto (GEPA) to systematically optimize prompts for scientific reasoning tasks, and analyze how prompting can affect reasoning behavior. We investigate the structural patterns and logical heuristics inherent in GEPA-optimized prompts, and evaluate their transferability and brittleness. Our findings reveal that gains in scientific reasoning often correspond to model-specific heuristics that fail to generalize across systems, which we call "local" logic. By framing prompt optimization as a tool for model interpretability, we argue that mapping these preferred reasoning structures for LLMs is an important prerequisite for effectively collaborating with superhuman intelligence.
