Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a multi-armed bandit (MAB) framework for adaptive prompt selection in automated essay scoring (AES) that significantly reduces operational costs while maintaining scoring accuracy. By treating different prompt types as arms in a bandit controller, the approach dynamically selects optimal strategies during inference, achieving a 78.4% reduction in LLM calls compared to exhaustive grid search. Experiments reveal that a multi-step grading approach with calibration examples yields the highest accuracy, providing valuable insights into balancing cost and reliability in educational technology applications.
A novel bandit-driven approach slashes LLM calls by over 78% while maintaining scoring accuracy in automated essay assessments.
Large Language Models (LLMs) demonstrate strong capabilities in automated essay scoring (AES), but contemporary approaches typically employ fixed prompt selection, failing to address operational cost concerns and evolving optimal configurations. We propose a novel cost-aware approach that treats each prompt type as an arm in a multi-armed bandit (MAB) controller, enabling adaptive selection of optimal prompting strategies during inference. Our experiments on IELTS Writing Task 2 essays show that the MAB framework achieves comparable scoring accuracy to exhaustive grid search while reducing LLM calls by 78.4\% to find the best grading approach. We implemented four distinct grading recipes (multi-step vs. single-step assessment, with vs. without calibration examples) and found that the multi-step approach with examples achieves the highest accuracy. By tracking token usage and latency alongside agreement metrics, we produce the first cost-reliability learning curves for essay scoring, providing actionable insights for educational technology platforms that must balance operational costs against assessment validity. This work represents the first application of online control mechanisms to adaptively select prompting strategies in AES, transforming prompt selection from an offline hyperparameter optimization problem into an efficient online learning task.