Search papers, labs, and topics across Lattice.
This paper critiques the reliance on image classification as the sole metric for evaluating Privacy-Enhancing Technologies (PETs) in computer vision, arguing that it overlooks important performance aspects relevant to diverse vision tasks. The authors introduce a compute-aware multi-task evaluation protocol that utilizes lightweight proxy tasks to assess PETs across various transformations, revealing significant discrepancies in task performance despite similar classification accuracies. Their findings underscore the necessity for more comprehensive evaluation methods that reflect the complexities of real-world vision applications beyond mere classification metrics.
PETs that achieve similar classification accuracy can perform drastically differently across various vision tasks, revealing hidden vulnerabilities in their effectiveness.
Privacy-Enhancing Technologies (PETs) in computer vision often rely on noise or image perturbations to protect visual data while securely processing it, creating a trade-off between task performance and protection. This trade-off is commonly evaluated using image classification, which primarily captures semantic separability and remains robust despite significant geometric, spatial layout or local boundary alterations. As a result, it is too simplistic as a proxy for generic vision tasks. Exhaustive downstream-task evaluation, however, is computationally expensive because models must often be trained for each PET transformation and parameter setting. We therefore propose a compute-aware multi-task protocol for evaluating PETs in model training. It combines lightweight proxy tasks that target complementary aspects of visual structure while remaining simple and fast to compute. Across irreversible privacy transformations, key-based block primitives, and learnable image encryption schemes, we demonstrate that PETs with similar classification accuracy can differ substantially on other tasks. The outcomes highlight the need for PET evaluation protocols that move beyond classification-only reporting.
