Search papers, labs, and topics across Lattice.
The paper investigates failure modes in semantic segmentation models caused by spurious correlations between category and scene during training. They find that models often exhibit "semantic label flips," where foreground pixels are assigned the wrong foreground class identity while maintaining accurate object boundaries. They introduce a diagnostic metric, Flip, to quantify these label flips and an entropy-based flip-risk score to identify flip-prone cases at inference.
Semantic segmentation models can get object boundaries right but still confidently mislabel the object's identity when category and scene are spuriously correlated.
The robustness of machine learning models can be compromised by spurious correlations between non-causal features in the input data and target labels. A common way to test for such correlations is to train on data where the label is strongly tied to some non-causal cue, then evaluate on examples where that tie no longer holds. This idea is well established for classification tasks, but for semantic segmentation the specific failure modes are not well understood. We show that a model may achieve reasonable overlap while assigning the wrong semantic label, swapping one plausible foreground class for another, even when object boundaries are largely correct. We focus on this semantic label-flip behaviour and quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity while remaining predicted as foreground. In a setting where category and scene are correlated during training, increasing the correlation consistently widens the gap between common and rare test conditions and increases these within-object label swaps on counterfactual groups. Overall, our results motivate assessing segmentation robustness under distribution shift beyond overlap by decomposing foreground errors into correct pixels, flipped-identity pixels, and missed-to-background pixels. We also propose an entropy-based, ground truth label-free `flip-risk'score, which is computed from foreground identity uncertainty, and show that it can flag flip-prone cases at inference time. Code is available at https://github.com/acharaakshit/label-flips.
