Search papers, labs, and topics across Lattice.
This paper introduces InsightSeg, a novel episodic memory mechanism that enhances guideline-consistent semantic segmentation by reusing successful correction insights from previous errors. By converting these corrections into directive natural-language insights and anchoring them to specific image regions, InsightSeg enables a segmenting agent to prevent recurring mistakes rather than merely correcting them. The approach significantly improves both initial and final segmentation performance on datasets like Waymo and Cityscapes while reducing the need for additional refinement steps, showcasing a more efficient use of multi-agent refinement systems.
InsightSeg transforms past correction episodes into actionable insights, enabling segmentation agents to prevent errors before they occur.
Guideline-consistent semantic segmentation requires more than category recognition, as real-world labeling policies demand fine-grained, task-specific decisions. Recent multi-agent refinement systems improve compliance with such textual guidelines by detecting and correcting errors. However, they are stateless: feedback from the critiquing agent is discarded, causing the same guideline-specific mistakes to be repeatedly rediscovered and corrected across the dataset at the cost of additional refinement. We introduce InsightSeg, an episodic memory mechanism that converts successful correction episodes into reusable, visually grounded insights. A meta-analyzer distills each qualifying episode into directive natural-language insights and anchors them to the local image regions that caused the error using patch-level visual concept vectors. On subsequent images, these concepts are matched against dense patch embeddings to retrieve relevant insights, which condition the segmenting agent before making its first prediction. This shifts the system from correcting recurring errors to preventing them, improving segmentation quality before any refinement occurs. Across Waymo and Cityscapes, InsightSeg improves both first-pass and final guideline-consistent segmentation performance while requiring fewer refinement steps, demonstrating that multi-agent refinement can become more accurate and efficient by drawing on past correction experience.
