Pierre Ablin

A novel $\mathrm{SemiCat}$ architecture achieves discrete diffusion-level performance on TinyGSM and outperforms one-hot flows at scale by addressing self-conditioning and embedding geometry in continuous diffusion language models.

Masked diffusion language models can match autoregressive baselines while cutting sampling steps by over 20%—provided you train across multi-step continuous trajectories rather than isolated random masks.

Distilling long documents into compressed KV caches silently cripples base model capabilities on unrelated prompts, but a lightweight query router or modest off-target data mixing completely eliminates this derailment.

Stop guessing how much to pretrain vs. specialize your language model – scaling laws can now tell you the optimal compute allocation for maximizing performance on downstream tasks.

Tri-modal masked diffusion models can now be trained from scratch, achieving strong results in text generation, text-to-image, and text-to-speech, thanks to a systematic exploration of the design space and a novel SDE-based batch size reparameterization.

Factually incorrect SLMs can be made more truthful by teaching them *what* to delegate, not just minimizing loss.