Search papers, labs, and topics across Lattice.
ReCap is introduced as a lightweight consistency framework for story visualization that enhances character stability and visual fidelity in generated image sequences without altering the base diffusion model. It uses a CORE module that selectively conditions on previous frames based on pronoun references and a SemDrift module that aligns denoiser representations with DINOv3 embeddings during training to correct semantic drift. ReCap achieves state-of-the-art character consistency on FlintstonesSV and PororoSV benchmarks, outperforming StoryGPT-V by 2.63% and 5.65% respectively, while adding only 149K parameters.
Achieve state-of-the-art character consistency in story visualization with a surprisingly lightweight framework that adds only 149K parameters.
Story Visualization aims to generate a sequence of images that faithfully depicts a textual narrative that preserve character identity, spatial configuration, and stylistic coherence as the narratives unfold. Maintaining such cross-frame consistency has traditionally relied on explicit memory banks, architectural expansion, or auxiliary language models, resulting in substantial parameter growth and inference overhead. We introduce ReCap, a lightweight consistency framework that improves character stability and visual fidelity without modifying the base diffusion backbone. ReCap's CORE (COnditional frame REferencing) module treats anaphors, in our case pronouns, as visual anchors, activating only when characters are referred to by a pronoun and conditioning on the preceding frame to propagate visual identity. This selective design avoids unconditional cross-frame conditioning and introduces only 149K additional parameters, a fraction of the cost of memory-bank and LLM-augmented approaches. To further stabilize identity, we incorporate SemDrift (Guided Semantic Drift Correction) applied only during training. When text is vague or referential, the denoiser lacks a visual anchor for identity-defining attributes, causing character appearance to drift across frames, SemDrift corrects this by aligning denoiser representations with pretrained DINOv3 visual embeddings, enforcing semantic identity stability at zero inference cost. ReCap outperforms previous state-of-the-art, StoryGPT-V, on the two main benchmarks for story visualization by 2.63% Character-Accuracy on FlintstonesSV and by 5.65% on PororoSV, establishing a new state-of-the-art character consistency on both benchmarks. Furthermore, we extend story visualization to human-centric narratives derived from real films, demonstrating the capability of ReCap beyond stylized cartoon domains.
