Scaling Laws & Emergent Abilities - Weekly Roundup

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

A 260M-parameter looped diffusion model achieves performance surpassing a 6.5x larger non-looped counterpart with significantly less inference compute.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

A 260M-parameter looped diffusion model achieves performance surpassing a 6.5x larger non-looped counterpart with significantly less inference compute.

Kilometer-scale extreme heat downscaling accuracy scales linearly with climatological distance, not simulation volume, revealing a path to cost-effective urban adaptation.

Wild AI-generated text can actively harm LLM performance on human text, with its value flipping from beneficial to detrimental as token ratios increase.

Trusting LLM accuracy gains from sampling multiple answers ($k$) requires vastly more samples than commonly assumed; this work cuts certification costs by up to 47% by accounting for inter-question variance.

LLM pre-training power laws are not fixed but dynamically shaped by learning-rate and batch-size schedules, with a novel forcing-memory surrogate accurately predicting loss across these regimes.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Mixing auxiliary data doesn't just offer more tokens—under the right spectral alignment and dataset growth ratio, it provably accelerates the downstream power-law scaling exponent itself.

Adding uninformative ambient dimensions completely breaks kurtosis-based projection pursuit, proving that high-order unsupervised discovery fundamentally requires second-order subspace pre-filtering to avoid hallucinating structure in pure Gaussian noise.

Pumping capacity into feature cross-networks yields diminishing returns, but harvesting non-shared estimator variance via recursive weight-sharing unlocks state-of-the-art CTR accuracy without standard ensemble deployment overhead.

Standard next-token adaptation fails on financial time series unless fine-tuning objectives explicitly penalize rollouts for distorting higher-order temporal structures like volatility clustering.

Optimizers can dramatically alter step-by-step training trajectories, but a conserved geometric invariant ($2\alpha_r + \alpha_t = 1$) locks every algorithm from SGD to Adam and Muon into the exact same $D^{-1/3}$ optimal data scaling wall.

A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.

Uniform layer looping wastes latent compute on trivial tokens, but dynamically routing iteration depth beats standard transformers on AIME with a 53% steeper test-time scaling slope.

Frontier pretraining runs routinely stake tens of millions of dollars on scaling laws, yet the empirical pipelines used to fit and extrapolate those curves have never been systematically benchmarked until now.

Parallel exploration beats sequential refinement when scaling agent test-time compute, but early-stage trajectory errors create persistent bottlenecks that deeper search and downstream model grafting struggle to overcome.

Pretrained visual encoders are a temporary crutch: encoder-free multimodal architectures close the performance gap with encoder-based systems at roughly $10^{22}$ FLOPs, well within current frontier pretraining budgets.

ZooWork-ShopRanker, a family of e-commerce rerankers aligned to judge-labeled shopping preference, and ShopRank-Bench, a contamination-limited benchmark of ~10,000 private-traffic preference pairs in both text formats, tiered by how many judge families committed to each label.