Search papers, labs, and topics across Lattice.
The paper introduces SketchSense, a novel framework for sketch-guided image inpainting that addresses the challenges posed by imperfect sketches, which often contain a mix of reliable and unreliable strokes. By employing a bidirectional attention mechanism that synchronously denoises both RGB and structural streams, the model refines its interpretation of the sketch while generating images. Experimental results demonstrate significant improvements in both restoration quality and structural fidelity compared to existing methods, highlighting the effectiveness of the proposed approach in handling complex imagery.
SketchSense achieves remarkable gains in image inpainting by intelligently interpreting imperfect sketch guidance, outperforming traditional methods in both quality and structural accuracy.
Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.
