Search papers, labs, and topics across Lattice.
This paper investigates how prompt optimization affects the performance of LLMs acting as judges for legal question answering, using the LEXam benchmark. They optimize task prompts with the ProTeGi method, using feedback from Qwen3-32B and DeepSeek-V3 as judges, and find that automatic optimization outperforms human-designed prompts. Crucially, prompts optimized with lenient judges generalize better to strict judges, suggesting that judge disposition significantly impacts prompt transferability.
LLM judges aren't created equal: lenient judges yield optimized prompts that generalize better than those from strict judges, revealing a critical dependency on judge disposition for effective prompt optimization.
This work explores the role of prompt design and judge selection in LLM-as-a-Judge evaluations of free text legal question answering. We examine whether automatic task prompt optimization improves over human-centered design, whether optimization effectiveness varies by judge feedback style, and whether optimized prompts transfer across judges. We systematically address these questions on the LEXam benchmark by optimizing task prompts using the ProTeGi method with feedback from two judges (Qwen3-32B, DeepSeek-V3) across four task models, and then testing cross-judge transfer. Automatic optimization consistently outperforms the baseline, with lenient judge feedback yielding higher and more consistent gains than strict judge feedback. Prompts optimized with lenient feedback transfer better to strict judges than the reverse direction. Analysis reveals that lenient judges provide permissive feedback, yielding prompts with broader applicability, whereas strict judges produce restrictive feedback, leading to judge-specific overfitting. Our findings demonstrate algorithmically optimizing prompts on training data can outperform human-centered prompt design and that judges' dispositions during optimization shape prompt generalizability. Code and optimized prompts are available at https://github.com/TUMLegalTech/icail2026-llm-judge-gaming.
