Search papers, labs, and topics across Lattice.
This paper investigates the decoding dynamics of Masked Diffusion Language Models (MDLMs) and introduces TIE (Trajectory-based Iterative Ensembling), a framework that enhances knowledge fusion by tracking reliable decoding trajectories. The authors find that successful generations maintain stable confidence dynamics, while unreliable trajectories can be corrected by integrating intermediate states from other models. Their approach demonstrates strong performance across various reasoning tasks, highlighting TIE's effectiveness in addressing the challenges of MDLM ensembling.
Reliable decoding trajectories can be tracked and enhanced across models, leading to improved performance in complex reasoning tasks.
Masked Diffusion Language Models (MDLMs) have emerged as a distinct paradigm for sequence generation. As MDLMs become diverse in capabilities and knowledge coverage, an important question is how to combine their knowledge. Toward this, we first investigate the unique decoding dynamics of MDLMs. We find that successful generations exhibit stable confidence dynamics over answer-relevant positions, while unreliable trajectories can often be corrected by injecting promising intermediate states from other models. Guided by this observation, we propose TIE (Trajectory-based Iterative Ensembling), a knowledge fusion framework in which MDLMs iteratively identify reliable decoding trajectories and relay them across models. TIE tracks confidence dynamics over answer-relevant positions to determine which model currently follows a more reliable trajectory and selectively transfers partially denoised sequences across models. As the model on the more promising trajectory often changes across denoising steps, TIE allows different models to contribute complementary strengths at different stages of generation. Strong performance across diverse reasoning tasks, along with our analyses, suggests that TIE offers a practical approach to the underexplored problem of MDLM ensembling.
