Mila

Grokking's distinct phases—nuisance fitting, core learning, and delayed generalization—are analytically separable, with sharp conditions for occurrence and predictable plateau lengths.

Transformer and RNN tractography models achieve state-of-the-art performance by introducing a novel generation-validation phase for streamline-level supervision, overcoming the limitations of local loss functions.

This analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to $14$B parameters, Necessity and Plasticity anti-align across depth, whereas in the evaluated Mamba-style SSMs they point to overlapping regions.

LWVIC4Code is proposed, a non-contrastive representation learning approach specifically designed for Type-IV clone detection that achieves competitive or superior performance without negative samples, benefits from layer-wise supervision, and generalizes effectively from Python to other languages, particularly Java and C#.

Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate.

Static memory models are holding back performance—Proteus reveals that incrementally activating memory can drastically enhance context retention and reduce interference.

A simple yet powerful adaptation of attention for triangle meshes outperforms existing methods, achieving state-of-the-art results in geometry-processing tasks.

Transforming our understanding of Transformers, this work reveals that learnability may be as crucial as expressivity in optimizing large language models.

Allocating more capacity to earlier layers in language models can significantly enhance performance, challenging the long-held uniform layer design paradigm.

MambAdapter achieves superior performance in audio and speech tasks while drastically cutting down on computational resources.

Language specialization in multilingual MoEs happens mostly in the final layers, suggesting a surprisingly simple recipe for parameter-efficient adaptation.

DINOv2's representation space is so statistically well-behaved that you can train a vanilla diffusion transformer on it and beat specialized architectures with fewer parameters.

LLMs struggle with structured 2D tasks when inputs are serialized into 1D, revealing a surprising performance gap compared to vision-augmented models that directly process the 2D layout.

LLMs re-rank documents better when you learn to route each query to the specific attention heads that matter, instead of relying on static subsets or everything at once.

Looped LLMs don't just perform better reasoning, they also internally mirror the distinct inference stages of standard feedforward models, repeating them cyclically.

Merging experts in MoE LLMs can actually *improve* performance compared to pruning, offering a new path to compression that preserves capabilities.

MoEs don't always need learned routers: routing information can be embedded directly in the hidden state.

Ditch the text: WavSLM shows you can train a competitive speech language model using only distilled WavLM representations, unlocking a simpler, single-stream generative pretraining paradigm for speech.

Diagonal SSMs, despite their empirical success, provably fail to track states of non-Abelian groups, revealing fundamental limitations in their expressive power.

Takeuchi's Information Criterion (TIC) accurately predicts DNN generalization gaps, but only when models operate near the Neural Tangent Kernel (NTK) regime.

Attention-based re-ranking gets a boost: ReAttn's post-hoc re-weighting tames over-concentration and lexical bias, leading to more accurate and interpretable results without extra training.

Dramatically improve protein language models by simply post-training them to align with protein graphs, yielding a 59% increase in contact prediction accuracy.