Scaling Laws & Emergent Abilities - Weekly Roundup

Generalization after fine-tuning is dictated by the linear structure of knowledge in pretrained models, not just factual memorization.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.

Autonomous ML engineering agents can achieve frontier-model performance by learning from long experimental trajectories using outcome-guided supervision and windowed reinforcement learning.

Generalization after fine-tuning is dictated by the linear structure of knowledge in pretrained models, not just factual memorization.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.

Autonomous ML engineering agents can achieve frontier-model performance by learning from long experimental trajectories using outcome-guided supervision and windowed reinforcement learning.

Byte-level LLMs can achieve a 25% jump in math accuracy by patching where text is *new*, not just surprising.

Learned token visibility control dramatically boosts LLM reasoning accuracy, achieving state-of-the-art performance on GSM8K and MATH-500 with a tiny, frozen inference-time policy.

A single, elastic transformer model can now dynamically adapt its compute budget across a continuum of sparsity levels, achieving up to 1.80x higher throughput and 1.26x lower latency on A100 GPUs without retraining.

N-gram embeddings can be learned without any additional parameters or GPU memory, unlocking massive scaling and instant domain adaptation for LMs.

Larger momentum coefficients ($β$) are crucial for preserving optimization gains in LLM post-training with noisy rewards, outperforming conventional choices and boosting accuracy by nearly 2% on large models.

LLM reasoning can be boosted at inference time by separately sharpening reasoning paths and final answers, outperforming existing power-sampling methods across multiple benchmarks.

Tabular foundation models can achieve state-of-the-art in-context learning by selectively activating feature-specific experts, outperforming dense models on challenging datasets.

MoE models slash the performance cost of multilingual expansion by 83% when scaling from 12 to 200 languages, fundamentally altering the trade-offs in capacity allocation.

Language models encode task information in hidden states *before* their unembedding matrix can express it, suggesting current developmental timelines are misleading.

Gauge-equivariant GNNs generate internal representations that exhibit a surprising loop geometry, with contour length growing exponentially while spatial extent grows only logarithmically with depth.

Larger RLVR batches, despite appearing efficient with square-root learning rates, suffer a critical batch size that degrades sample efficiency and hastens failure due to curvature effects.

Explicitly modeling cross-layer information flow as a memory retrieval problem unlocks a linear-time attention alternative that outperforms existing methods.

Refresh timing, not just frequency, dictates Kronecker-preconditioned LLM training performance, with optimal placement flipping based on learning rate.

Sparse autoencoders recover known factors 80% better by training in the data's generative space, not its scaled readout space.