Search papers, labs, and topics across Lattice.
This paper addresses the challenges of surgical action triplet recognition in robot-assisted surgery by introducing the Mixture-of-Experts-guided Co-Optimization (MoeCo) framework, which employs knowledge-driven learning to enhance model performance. The authors tackle component-level and category-level optimization conflicts through a component-tailored adapter and a coordinated gradient learning strategy, respectively, while integrating multimodal large language model knowledge to improve interpretability and robustness. Experimental results on the CholecT45 and CholecT50 datasets demonstrate that MoeCo significantly outperforms existing methods, particularly in recognizing rare surgical actions.
Dynamic integration of multimodal knowledge boosts the recognition of rare surgical actions, overcoming critical optimization conflicts in robot-assisted surgery.
Surgical action triplet recognition constitutes a critical task in context-aware robot-assisted surgery, facilitating automatic surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (\textit{MoeCo}) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates multimodal large language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50 datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.
