Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study identifies intervention bias as a critical failure mode in zero-shot large-language-model (LLM) educational advisory agents, where they incorrectly recommend actions despite an oracle policy indicating inaction. Through a six-arm ablation study on 800 students, it was found that zero-shot GPT-4o had a false-positive rate of 43%, suggesting significant miscalibration in its recommendations. In contrast, supervised policy learning using a trajectory-conditioned ONNX Decision Transformer achieved near-zero calibration error and high accuracy, effectively mitigating the intervention bias observed in zero-shot models.
Zero-shot LLMs can mislead educational interventions, recommending action for 73% of students when an oracle suggests none, leading to thousands of unnecessary advisor contacts.
We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction. In a six-arm ablation on the Open University Learning Analytics Dataset (N=800 students, four temporal cutoffs), at day 56 -- when the oracle designates 70.1% of students as needing no intervention -- zero-shot GPT-4o recommends action for 73%, a 43 percentage-point false-positive rate. Commercial RAG and SQL-augmented retrieval are comparably miscalibrated; at 10,000 students this implies about 4,300 unnecessary advisor contacts per cycle. Supervised policy learning eliminates this bias: a trajectory-conditioned ONNX Decision Transformer (DT) and a snapshot XGBoost classifier, trained on the same oracle-labelled trajectories under strict prefix-only features, both achieve near-zero calibration error. The DT reaches macro-F1 0.79 (macro-recall 0.85) across all five action classes, predicting even the rare load-reduction action without collapsing, at a 0% action flip rate and sub-5 ms CPU decision latency. The two supervised arms are on par; the DT's edge over XGBoost at the final cutoff is indicative only (unpaired across cohorts). Scope: we validate Stage-2 decision-making (EAV state vector to supervised policy) under controlled oracle input from structured OULAD data; high fidelity reflects feature-oracle alignment, not general high-stakes-AI capability. The most robust finding is the intervention-bias contrast, not the absolute accuracies. We also show an Evaluation Gap: LLM-as-judge scoring (DeepEval G-Eval) is blind to intervention bias, rewarding fluent over-prescription rather than decision quality.