Search papers, labs, and topics across Lattice.
This paper introduces a layered framework for knowledge infusion in multimodal iterative generative models, addressing the limitations of existing methods that often fail to systematically categorize knowledge incorporation. By identifying four distinct intervention layers鈥攕urface, trajectory, latent, and parametric infusion鈥攖he authors demonstrate how knowledge can be effectively integrated at various stages of the generative process. In experiments using diffusion models, the implementation of multiple layers led to a 70.97% reduction in knowledge-violating outputs, validating the framework's effectiveness and complementarity of the different layers.
Knowledge infusion in generative models can be strategically layered to achieve a dramatic 70.97% reduction in knowledge-violating outputs.
Multimodal generative models produce fluent outputs but remain unreliable when generation must respect structured, domain-specific, or safety-critical knowledge. Existing methods incorporate knowledge through mechanisms such as prompt augmentation, guidance, latent editing, or fine-tuning, yet they are typically categorized by technique rather than by the component of the generative process they modify. We argue that knowledge infusion in iterative generative models is fundamentally anintervention-layer problem. Since thegenerative process unfolds as a trajectory of internal states, knowledge can act on four structurally distinct components of this process: the input/output boundary, the transition function, the intermediate state, and the model parameters. This maps to four intervention layers: surface, trajectory, latent, and parametric infusion. We instantiate the framework in diffusion models, map representative methods to all four layers, and derive design principles for multi-layer composition. In a controlled safety-alignment experiment using a multimodal knowledge graph with two diffusion backbones, we implement three of the four layers cumulatively, surface (input-side and output-side) and trajectory--latent (mid-generation). We show empirically that each additional layer addresses failure classes that prior layers cannot reach, reducing knowledge-violating outputs by 70.97% compared to vanilla generation and empirically confirming the framework's complementarity prediction.
