Search papers, labs, and topics across Lattice.
This paper introduces EgoMed-Agent, a multi-agent system designed for interactive egocentric medical image segmentation (IEMIS) that addresses challenges of semantic ambiguity and visual variability in user instructions and video feeds. The system employs a Target Confirmation Workflow to validate user intentions and a Localization-Guided Propagation Workflow to maintain segmentation accuracy across frames. Experimental results demonstrate that EgoMed-Agent achieves a 71.34% average Dice score, significantly outperforming the best text-prompted baseline, which scored only 11.70%.
EgoMed-Agent achieves over six times better segmentation accuracy than existing text-prompted methods by effectively aligning user intent with visual data in real-time.
Interactive egocentric medical image segmentation (IEMIS) plays an important role in smart-glasses-assisted medical image review, segmenting the medical targets a clinician refers to from their egocentric view. Once it succeeds, the object-level visual evidence it provides strengthens the review and underpins fine-grained analysis and clinical decision-making. However, the instruction and the video both come from the user's egocentric perspective, which poses two challenges. (1) Semantic ambiguity leaves the model unable to confirm the user-intended target. (2) Visual variability makes the segmentation jump from frame to frame. In this paper, we propose EgoMed-Agent, a multi-agent system that understands the target from the human perspective through two workflows. (1) The \textit{Target Confirmation Workflow} grounds the instruction against candidate targets with a reliability score, confirming the target when the grounding is reliable and asking the user to clarify when it is not, thereby confirming the segmentation target. (2) The \textit{Localization-Guided Propagation Workflow} couples mask propagation with per-frame target localization, using the localized target to correct the propagated mask whenever the two diverge, so the segmentation stays on the target across the egocentric video. Extensive experiments show that EgoMed-Agent reaches 71.34\% average Dice, far above the best text-prompted baseline (11.70\%). Our code is available at \href{https://github.com/wdyyyyyy/EgoMed-Agent}{our project page}.
