Search papers, labs, and topics across Lattice.
This paper introduces a novel variable-length tokenizer that enhances the performance of diffusion transformers by merging tokens instead of truncating them, addressing the limitations of conventional variable-length tokenization. The proposed learnable global merging approach ensures that token semantics remain aligned across varying lengths, enabling a more effective representation in latent diffusion models. Experimental results demonstrate that this method achieves a superior trade-off between generated image quality and computational efficiency on the ImageNet dataset compared to existing variable-length tokenization methods.
Merging tokens rather than truncating them allows diffusion models to maintain semantic alignment across variable lengths, drastically improving image generation quality.
Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio. Variable-length tokenizers (VLTs) promise adaptive compression by varying token counts, allowing diffusion models to flexibly balance quality and compute. However, conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across lengths. This leads to a cross-length shift in the latent distribution that hinders a single variable-length diffusion model from operating effectively. To address this, we propose a novel variable-length tokenizer that modulates length by merging tokens. We show that encouraging similar tokens to merge enables direct cross-length representation alignment when the diffusion transformer operates according to the merging pattern. Since conventional merging methods are data-dependent, making the merging pattern inaccessible during generation, we introduce learnable global merging, which is data-independent, to ensure compatibility with diffusion transformers. On ImageNet 256$\times$256 generation, our merging-based variable-length tokenizer integrated with a diffusion transformer achieves a superior gFID-compute trade-off compared to prior VLT methods. Code is available at [this https URL](https://github.com/movinghoon/lgm)
