Search papers, labs, and topics across Lattice.
This paper introduces a novel cardiac video classification model that integrates deformable shape and texture representations through a bi-directional cross-attention mechanism, allowing for adaptive weighting of features based on spatio-temporal correspondence. By moving beyond simple concatenation, the model learns to dynamically adjust the contributions of each modality across different cardiac phases, enhancing the interpretability of the classification results. The proposed approach achieves state-of-the-art performance on a cine cardiac magnetic resonance video dataset, significantly improving diagnostic accuracy and interpretability compared to existing methods.
Adaptive weighting of shape and texture features in cardiac video classification leads to state-of-the-art performance and enhanced interpretability of critical cardiac phases.
Deformable shape representations have proven to be robust complements to texture features in cardiac image classification, offering geometric priors that are invariant to imaging artifacts and intensity variations. However, existing deep networks perform simple concatenation to combine these distinct feature representations, which neither fully exploits their complementary nature nor learns cross-modal feature dependencies. Furthermore, this results in uniform attention across all timepoints; hence ignoring the varying diagnostic importance across the cardiac phases. In this paper, we propose a novel cardiac video classification model that, for the first time, learns temporal features in an integrated space of deformable shape and image texture representations. In particular, we design a bi-directional cross-attention in the latent space to fuse latent deformable shape and image features, allowing each modality to adaptively weight the other based on spatio-temporal correspondence. In contrast to current methods that apply uniform weighting across all the cardiac phases, our approach learns to dynamically adjust the contributions of shape and texture representations, derived from images, over time. We demonstrate state-of-the-art classification performance on a cine cardiac magnetic resonance (CMR) video dataset, achieving improved interpretability from attention mechanisms that identify diagnostically critical cardiac phases and modality contributions.
