Search papers, labs, and topics across Lattice.
This study addresses the significant discrepancy between balanced-test precision and operational performance in Earth-observation classifiers, specifically for detecting internal solitary waves using Sentinel-1 data. By identifying that a balanced-test score of 0.794 does not reflect the true operational precision of 0.192, the authors propose a novel three-number reporting method that includes balanced-test, operational-prior, and real post-deployment metrics. The implementation of this prior-matched reporting method led to a substantial improvement in classifier precision, achieving 0.927 while maintaining a recall floor of 0.80, demonstrating its effectiveness in operational settings.
A balanced-test score can mislead operational performance assessments by over 60%, revealing a critical need for prior-matched evaluation in rare-event detection.
The Internal Waves Service screens the Sentinel-1 Wave-mode archive for internal solitary waves, routing detections to experts whose adjudication time is the resource the effort exists to conserve. Because attention is the cost of error, precision leads. Its classifier was trained and reported at a one-to-one class balance, fixed before the operational rate could be known. That rate has since emerged at roughly one scene in twenty, and a balanced-test score badly overstates the precision a validator meets. A model that scores 0.794 balanced-test precision scores 0.192 in real operation: the gap is a systematic artefact of reporting at the wrong prior, invisible to the metric most work quotes. We show the mismatch to be an evaluation problem in the costume of a training one at a fixed recall, prior correction and calibration cannot move precision, and answer it with a prior-matched reporting method based on three figures: balanced-test, operational-prior, and real post-deployment, whose contrast is the honest measure. A precision-first, leakage-controlled development cycle then improves the classifier lever by lever, each promoted only against a pre-registered margin; added capacity not clearing it, calibration inert, feature aggregation the one real lift, so the honest negatives are as much a result as the gain. Holding recall at a floor of 0.80 and certifying against a sealed, single-read lockbox, the promoted model reports 0.927 precision at the operational prior; an out-of-time check confirms discrimination transfers to unseen periods while a fixed operating point does not. Prior-matched reporting, begin balanced, then move to the prior as the stream reveals it, transfers to any operational Earth-observation service bootstrapping a rare-event detector under a prior it has yet to discover.
