Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces InstEditSeg, a novel generative framework that reformulates medical segmentation tasks as instruction-driven image editing, addressing challenges such as low contrast and ambiguous boundaries in polyp and skin lesion segmentation. By leveraging latent diffusion models and incorporating a DINOv3 visual encoder with a multi-scale feature pyramid, the model produces color-coded overlays instead of binary masks, enhancing alignment with natural image distributions. The results show that InstEditSeg achieves competitive accuracy with traditional methods while offering significant advantages in cross-domain generalization, multi-lesion segmentation, and task control through textual instructions.
Instruction-driven image editing can outperform traditional binary segmentation methods in medical imaging, achieving better generalization and flexibility.
Accurate segmentation of polyps and skin lesions is pivotal for clinical diagnosis, yet existing methods struggle with low contrast, ambiguous boundaries, and cross-domain distribution discrepancies. Discriminative networks and most diffusion-based segmentation approaches predict standalone binary masks, leaving the visual priors of large-scale pretrained generative models largely unexploited. We propose InstEditSeg, a unified generative framework that reformulates medical segmentation as an instruction-driven image editing problem. Instead of emitting a mask, the model renders a color-coded overlay on the original image, conditioned on a textual instruction, so that the edited output aligns with the natural image distribution learned by latent diffusion models and mitigates the domain gap between natural and medical imagery. To recover fine anatomical structures, we introduce DINOv3 as an auxiliary visual encoder and a DINO Feature Guidance Block that builds a multi-scale feature pyramid. The pyramid is fused into the diffusion U-Net by channel concatenation and zero-initialized convolution so that hierarchical discriminative priors can be injected without perturbing the pretrained weights. A dual-branch classifier-free guidance strategy requiring only two forward passes per denoising step reduces inference cost. On polyp and skin lesion benchmarks the framework achieves accuracy competitive with strong discriminative baselines, and it further demonstrates concrete advantages of the generative formulation: notably better cross-domain generalization on unseen data, more complete multi-lesion segmentation, instruction-conditioned task control, and sampling flexibility. We also analyze the strengths and limitations of the paradigm, including its color sensitivity and unsupported attribute-conditioned selection. Code is available at: https://github.com/wincharm001/InstEditSeg.