Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study rigorously evaluates model rankings and input reliance in surface-water segmentation using the Sen1Floods11 dataset, employing repeated configuration comparisons and various analytical methods. The findings reveal that while aggregate metrics like global IoU are useful for ranking, they do not capture the stability of rankings or the degree to which predictions depend on specific inputs. Notably, the cross-modal student model achieves the highest mean IoU, yet the ordering of configurations varies significantly across different seeds and geographic contexts, highlighting the need for more nuanced evaluation methods.
Aggregate metrics can mislead; this study uncovers significant variability in model rankings and input reliance that could change how we evaluate segmentation systems.
Performance evaluation for surface-water segmentation commonly uses an aggregate metric such as global intersection-over-union (IoU) to rank model configurations. However, a configuration ranking does not by itself establish why one system performs better, whether a close ordering is stable, or how strongly predictions rely on individual inputs. We examine these distinctions primarily on Sen1Floods11 through repeated configuration comparisons, paired test-chip analysis, fixed-checkpoint input stress tests, and geographic reweighting, with a targeted secondary evaluation of supervised input configurations on GEOID-Flood. The cross-modal student achieves the highest three-seed mean IoU on Sen1Floods11, but close orderings vary across seeds and geographic weighting, while ancillary-input rankings differ between Swin-UNet and U-Net. The GEOID-Flood evaluation shows substantial agreement in supervised ancillary-input effects, although the exact architecture ordering remains configuration dependent. Fixed-checkpoint tests further establish reliance on terrain and WorldCover without establishing a clean-input performance benefit, while target semantics and the later WorldCover prior restrict the evaluation to retrospective all-water segmentation. These results show that aggregate metrics remain useful for ranking complete configurations, but ranking stability, component attribution, input reliance, and deployment scope require distinct evidence. Performance evaluation should therefore match the evidence reported to the claim being made.