Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to test-time canonicalization for pretrained vision models, addressing their vulnerability to affine transformations that do not alter object class. By reframing canonicalization as out-of-distribution (OOD) detection, the authors explore a wide array of OOD scoring functions and optimization techniques, ultimately identifying that distance-based scores with random search and local refinement yield the best performance. The method preserves in-distribution accuracy while enhancing robustness against transformed inputs through a gated mechanism that selectively applies canonicalization based on OOD scores.
Transforming inputs only when necessary can significantly boost model robustness without sacrificing in-distribution accuracy.
Pretrained vision models often misclassify inputs that are rotated, scaled, or sheared, even though these affine transformations leave the object class unchanged. Robustness is usually restored either by building equivariance into the architecture or by retraining with augmentation, both of which require changing or retraining the model. Test-time canonicalization instead leaves the classifier untouched. It undoes the transformation of each input, mapping it to a canonical form near the training distribution before classification. Existing canonicalizers, however, rely on a narrow set of logit-based energy scores and bespoke search procedures, leaving the design space of scoring functions and optimizers unexplored. We reframe canonicalization as out-of-distribution (OOD) detection, which lets any OOD score serve as the energy minimized over transformations. Across benchmarks ranging from handwritten characters and sketches to natural images and 3D point clouds, we systematically evaluate around twenty OOD scores and nine search algorithms, finding that distance-based scores paired with random search and local refinement perform best overall. Because canonicalizing an already-aligned input can hurt accuracy, we add a gated mechanism that transforms an input only when its OOD score indicates this is needed, preserving most in-distribution accuracy while retaining the robustness gains on transformed inputs. Code is available at github.com/johschm/its.
