Search papers, labs, and topics across Lattice.
This paper explores training-free prompt optimization as an alternative to RLHF for aligning LLMs as math tutors. They adapt 7 existing prompt optimization methods and introduce 5 new education-specialized methods, evaluating them on out-of-distribution math tutoring benchmarks. Results show that optimized prompts outperform a strong RL-trained baseline, with their ParetoGrad method achieving the best balance across key tutoring metrics, while also revealing differences in teaching knowledge patterns compared to RL-trained models.
Forget RLHF: strategically crafted prompts alone can unlock superior math tutoring performance in LLMs, even surpassing RL-trained models.
Aligning LLMs for math tutoring typically requires RL-based training with multi-GPU infrastructure. We investigate whether training-free prompt optimization-evolving only the system prompt via API calls-can serve as a practical alternative. We adapt 7 published methods and propose 5 education-specialized methods, evaluating these 12 methods under 5 conditions on 2 OOD benchmark suites. All 12 best-per-method configurations surpass the strongest RL-trained baseline (R_total = 0.633), and our ParetoGrad achieves the best Pareto balance across post-test solve rate, leak control, and helpfulness, rather than dominating any single component. Behavioral analysis with an 82-code educational codebook reveals that training-free methods rely on teaching-knowledge patterns at 2-3x the rate of RL-trained models, with a compensating ~10 percentage-point reduction in intent-level scaffolding. We also find a task-dependent reasoning mode effect consistent across training-free and RL-based paradigms. Our approach enables efficient development of pedagogically aligned LLM tutors with prompts alone and minimal compute.
