Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors present OKS Attack, a decision-based black-box adversarial attack tailored for human pose estimation and downstream keypoint-based action recognition pipelines. While existing video black-box attacks rely on coarse bounding-box overlap metrics like IoU, this approach uses Object Keypoint Similarity (OKS) feedback to directly exploit the vulnerability of fine-grained spatial keypoint structures. Across both top-down and single-stage pose estimators on Penn Action, the attack consistently degrades pose quality (reducing mean OKS by up to 0.1494) and slashes downstream action recognition accuracy by up to 13.86 percentage points.
Keypoint-based video pipelines can be reliably compromised in purely black-box settings by optimizing perturbations against skeletal geometry rather than bounding boxes, inducing up to a 13.86 percentage-point drop in downstream action recognition.
Human pose estimation and keypoint-based action recognition models are increasingly deployed as components of video understanding pipelines, yet their vulnerability to adversarial attacks remains insufficiently studied. Temporally coherent black-box attacks have been previously studied in visual object tracking, where the attack feedback can be defined using bounding-box overlap measures such as Intersection over Union (IoU). However, human pose estimation produces keypoint configurations rather than enclosing boxes, making box-level similarity poorly suited for measuring pose degradation. We propose OKS Attack, a decision-based black-box attack that uses Object Keypoint Similarity (OKS) as the attack feedback signal, directly targeting the spatial structure of human poses rather than their enclosing boxes. Experiments on the Penn Action dataset show that OKS Attack consistently reduces pose quality across evaluated pose estimators, with mean OKS decreases ranging from 0.0802 to 0.1494. In a downstream cross-dataset action-recognition evaluation, the attack reduces accuracy by 6.18 to 13.86 percentage points and outperforms query-matched random-noise perturbations. The attack is effective across both top-down and single-stage pose estimation models. The source code will be made publicly available at https://github.com/KacperM33/OKS_attack