Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To mitigate update interference across heterogeneous evaluation dimensions in video post-training, this work establishes G$^3$-LoRA, a framework that clusters reward-weighted data by residual gradient compatibility and trains grouped LoRA experts that are consolidated via weight merging and on-policy distillation. The approach resolves a core failure mode in reward-weighted flow matching, where incompatible reward objectives demand conflicting velocity-field updates in overlapping noisy latent regimes. Evaluated on Wan2.1-T2V-1.3B and CogVideoX-2B, the consolidated adapter outperforms joint LoRA fine-tuning and heuristic semantic clustering across VBench2.0 dimensions while successfully avoiding cross-skill negative transfer.
Training a monolithic LoRA across multi-dimensional video rewards silently degrades capabilities by averaging conflicting velocity fields, a failure mode resolved by partitioning data along residual gradient compatibility.
Post-training foundation video models on heterogeneous reward-weighted data usually assume that all data categories induce compatible updates. This assumption is fragile when categories correspond to different skills, domains, or evaluation dimensions. We study this problem in text-to-video post-training, where VBench2.0 dimensions define data buckets and an external multimodal reward pipeline assigns sample weights. We propose G$^3$-LoRA (Gradient-Guided Grouped LoRA), a data organization procedure that probes category-level gradients induced by reward-weighted video samples, removes the shared global update direction, clusters categories by residual gradient compatibility, trains group-specific LoRA experts, and consolidates them into one adapter by weight merging followed by on-policy distillation from the experts. We motivate this procedure by viewing reward-weighted flow matching as velocity-field regression: incompatible reward dimensions may prefer different denoising directions in overlapping noisy latent regions, causing shared LoRA training to average capabilities. On Wan2.1-T2V-1.3B-Diffusers, the merged grouped adapter improves the matched VBench2.0 evaluation over the base model, a joint reward-weighted LoRA baseline, and random, semantic, and raw-gradient partitions trained with the same pipeline; an independent evaluator agrees, and on CogVideoX-2B grouping avoids the negative transfer of joint training. The gain is not uniform: merging compresses the largest specialist gains, distillation recovers part of this loss, and camera motion and several local-quality dimensions remain challenging. Together, these results suggest that gradient compatibility can serve as a practical diagnostic for organizing reward-weighted video post-training data.