Search papers, labs, and topics across Lattice.
This paper introduces ActPair, a novel three-stage framework for text-based person anomaly search that enhances the identification of individuals based on context-dependent behaviors. By fine-tuning a vision-language model with an action-aligned multi-task objective and employing a pairwise multimodal reranking approach, the method effectively bridges the pose-semantic gap and retains critical semantic details. Extensive experiments show that ActPair outperforms existing methods on the Pedestrian Anomaly Behavior (PAB) dataset and generalizes well to other datasets, highlighting its robustness and efficacy in real-world applications.
ActPair achieves state-of-the-art performance in person anomaly detection by integrating action-aligned retrieval with efficient pairwise multimodal reranking, redefining how we approach context-dependent behavior analysis.
Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.
