Search papers, labs, and topics across Lattice.
The paper introduces ShapeY, a benchmark dataset of 68,200 rendered images of 200 3D objects designed to evaluate shape-based object recognition. ShapeY uses a nearest-neighbor matching task to probe the embedding space of object recognition systems, assessing their ability to cluster different views of the same object based on 3D shape similarity despite viewpoint and appearance changes. Experiments on 321 pre-trained networks reveal that even state-of-the-art models struggle with robust shape-based recognition, exhibiting inconsistent generalization across viewpoints and egregious matching errors.
Even the best vision models make shockingly bad shape recognition errors, like confusing a car with a chair, when evaluated on a new viewpoint-invariant shape recognition benchmark.
Object recognition (OR) in humans relies heavily on shape cues and the ability to recognize objects across varying 3D viewpoints. Unlike humans, deep networks often rely on non-shape cues such as texture and background, leading to vulnerabilities in generalization and robustness. To address this gap, we introduce ShapeY, a novel and principled benchmarking framework designed to evaluate shape-based recognition capability in OR systems. ShapeY comprises 68,200 grayscale images of 200 3D objects rendered from multiple viewpoints and optionally subjected to non-shape ``appearance''changes. Using a nearest-neighbor matching task, ShapeY specifically probes the fine-grained structure of an OR system's embedding space by evaluating whether object views are clustered by 3D shape similarity across varying 3D viewpoints and other non-shape changes. ShapeY provides a suite of quantitative and qualitative performance readouts, including error rate graphs, viewpoint tuning curves, histograms of positive and negative matching scores, and grids showing ordered best matches, which together offer a comprehensive evaluation of an OR system's shape understanding capability. Testing of 321 pre-trained networks with diverse architectures reveals significant challenges in achieving robust shape-based recognition: even state-of-the-art models struggle to generalize consistently across 3D viewpoint and appearance changes, and are prone to infrequent but egregious matches of objects of obviously completely different shape. ShapeY establishes a principled framework for advancing artificial vision systems toward human-like shape recognition capabilities, emphasizing the importance of disentangled and invariant object encodings.
