Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MARGIN, a novel runtime calibration technique for coordinating heterogeneous foundation models by learning model-specific confidence corrections from observed answer outcomes without retraining or requiring a calibration set. MARGIN addresses the issue of disparate confidence meanings across models and workloads by tracking accuracy-confidence ratios within dynamic bands and blending these corrections towards a global estimate. Evaluations across code generation, QA, and math tasks demonstrate MARGIN's superior performance against baselines in reducing post-shift calibration error and improving answer-selection accuracy in coordination scenarios.
Foundation models can be coordinated more effectively by learning to correct their self-reported confidence at runtime, significantly improving answer selection accuracy even when model performance shifts.
When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.