Search papers, labs, and topics across Lattice.
This paper introduces FineX, a novel framework for fine-grained human action recognition that integrates RGB appearance, pose heatmap geometry, and skeletal-graph topology through pairwise cross-attention and a latent sparse Mixture-of-Experts approach. By facilitating a structured exchange of information between different representation streams, FineX significantly enhances the model's ability to distinguish visually similar actions based on subtle differences in body configuration and timing. The method achieves state-of-the-art performance on multiple benchmarks, notably improving mean class accuracy on the challenging Gym288 dataset by 7.6 percentage points without relying on textual supervision or extensive vision-language pre-training.
FineX boosts fine-grained action recognition accuracy by over 7% on challenging datasets, leveraging a unique fusion of visual and skeletal representations.
Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.
