Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
Reference-guided diffusion editors frequently suffer from poor fidelity because edit-region queries often allocate less than 1% of their attention mass to reference tokens. To resolve this bottleneck, the authors design RefGAP, a training-free framework that dynamically computes layer-wise logit offsets during the forward pass to boost reference attention in edit regions while suppressing it in keep regions. Across seven image and video diffusion architectures, RefGAP matches the fidelity-preservation Pareto frontier of manually tuned constant biases across tasks like face swapping and virtual try-on without requiring per-model hyperparameter sweeps.
Diffusion editors routinely allocate less than 1% of their attention mass to conditioning images, a major identity bottleneck that can be resolved on the fly across multiple architectures using training-free dynamic logit offsets.
Reference-guided diffusion editors struggle to faithfully reproduce user-provided references. We identify a potential bottleneck in diffusion editors: many methods provide limited reference-attention allocation. For example, in LoomVideo, edit-region queries assign less than 1% of their attention mass to the reference. We introduce RefGAP, a training-free correction that determines logit-offset magnitudes online at each layer from the reference-attention mass measured during the forward pass. Positive offsets to reference logits strengthen reference usage by edit-region queries, while negative offsets for keep-region queries limit reference-induced changes outside the edit. Two global coefficients control the correction; they are selected once on validation data from four development diffusion editors and held fixed. Across seven diffusion-based image/video editors, RefGAP improves identity fidelity in head swapping and face swapping. RefGAP achieves a fidelity-preservation trade-off comparable to separately tuned constant edit-side biases, without per-approach strength sweeps. Additional experiments on virtual try-on and background replacement evaluate transfer beyond identity editing.