Search papers, labs, and topics across Lattice.
The authors investigate budgeted rationale selection for medical QA using Root-Mean-Square Robustness-based Sample Prioritization (RMS-RSP), which prioritizes examples based on how gold-versus-distractor logit margins shift when perturbing hidden states strictly at rationale tokens. Addressing the high acquisition cost of rationale supervision, this framework tests whether selective attribution can match the utility of exhaustive rationale training on MedGemma-4B-IT across five benchmarks. While locked-budget standard accuracy gains over random selection remain marginal (60.61% vs. 60.08%), RMS-RSP consistently boosts robust accuracy (+1.91 points) and semantic consistency (+2.85 points) against answer-order permutations across all datasets using 29–254× fewer tokens than full supervision.
Full rationale supervision burns up to 254× more tokens without guaranteeing reliability, whereas isolating representations with high rationale-boundary sensitivity selectively immunizes medical LLMs against choice-order brittleness.
Medical question-answering datasets often contain answer labels, whereas high-quality rationales remain scarce, noisy, or costly to validate. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already-labeled questions should receive rationale supervision under a fixed token budget. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected. We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin. Across five medical QA datasets, MedGemma-4B-IT, three training seeds, ten budgeted non-RSP selectors, and an unbudgeted full-supervision reference, RMS-RSP provides a deliberately qualified result. Its locked-budget accuracy is 60.61% on average versus 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points). Its full-budget accuracy area is not better than Random. However, after three answer-option reorderings, RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, respectively, with the same direction on all five datasets. Training on every pool rationale raises macro accuracy to 63.74%, but consumes 29--254 times more rationale tokens and does not uniformly improve robustness. These findings do not establish universal accuracy gains; they instead suggest that rationale-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes.
