Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel logit composition strategy for autoregressive models, inspired by techniques from diffusion models, to enable compositional generalization. By assuming factorized conditionals, the authors demonstrate that this composition method is projective, allowing component models to maintain control over specific output subspaces. They further prove that this projective property is preserved under smooth reparameterizations and that length generalization is maintained when factorization assumptions hold uniformly.
Merging or composing LLMs doesn't have to be a chaotic mess: this logit composition method offers a surprisingly clean way to combine models while preserving their individual control over specific outputs.
Composing autoregressive models remains a core challenge in understanding how large language models can combine behaviors or skills learned across tasks. We introduce a new and principled composition strategy for autoregressive systems, inspired by composition methods developed for diffusion models. Under a factorized-conditionals assumption, we show that the resulting composition is projective: each component model preserves control over its own designated subspace of the output distribution avoiding interference between models. This property is further preserved under smooth reparameterizations of the output space, yielding a feature-space theorem. Finally, we show that composition preserves length-generalizing behavior when the factorization assumptions and component guarantees hold uniformly at the target length. These results provide a principled understanding of when model composition and merging succeed in autoregressive systems and identify conditions under which their interactions remain stable.