Scaling Laws & Emergent Abilities

Innate robot reflexes, designed for solitary tasks, actively impede the evolution of effective communication strategies in social foraging.

Looped Transformers can achieve tens of thousands of effective reasoning steps without performance collapse by learning to selectively retain and decay intermediate states.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Spectral analysis reveals that increasing latent dimensions in Koopman autoencoders improves dynamics learning by concentrating learned eigenvalues in low-residual regions, not just by reducing one-step prediction error.

Omni-modal foundation models achieve self-improvement via RL compute scaled to 1M context, leveraging asynchronous training and agentic grading for enhanced intelligence.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

Latent world models scale predictably with compute via a second-order power law, enabling accurate forecasting of robotic planning performance and zero-shot task execution on real hardware.

LLM capabilities diverge under latent sequence compression; language modeling, decodability, and reasoning do not share a single compression threshold.

LLMs develop inherent fault tolerance as they scale, learning to compute within error-correcting codes that remain efficient even for massive models.

Frontier LLMs might be accumulating alignment debt faster than previously assumed, as fitting alignment scaling laws on smaller models can significantly underestimate the true burden at scale.

Sparse MoE routers don't achieve adaptive expert selection through their maps alone; they co-evolve learned score distributions that critically determine token-wise expert participation.

Large HEP datasets are not just for training; they are foundational for unlocking power-law scaling, with smaller datasets offering little insight into high-compute performance.

Decoupling update magnitude from gradient norm in SGD initially accelerates learning but leads to marginal stability and distinct scaling regimes determined by model width and batch size.

The pursuit of LLM performance is shifting from raw scale to strategic resource allocation, revealing a critical need to re-evaluate scaling laws for parameter-efficient, compute-optimal training in constrained environments.

Two fixed attention blocks suffice for universal interpolation in deep residual networks, even with causal masking.

Quantum devices can exponentially increase their observable complexity over time with finite memory, outperforming classical systems by leveraging phase rotations to achieve perfect discrimination.

A single 10M-parameter Transformer policy achieves near-perfect win rates in competitive Super Smash Bros. Melee, outperforming specialized AIs and even a zero-delay opponent.

A novel gating mechanism allows LLMs to learn new tasks without forgetting old ones, even without access to prior training data.

Looped MoE models can scale to over 10 iterations, achieving substantial performance gains by stabilizing recurrence and diversifying expert selection.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

A 260M-parameter looped diffusion model achieves performance surpassing a 6.5x larger non-looped counterpart with significantly less inference compute.

Kilometer-scale extreme heat downscaling accuracy scales linearly with climatological distance, not simulation volume, revealing a path to cost-effective urban adaptation.

Wild AI-generated text can actively harm LLM performance on human text, with its value flipping from beneficial to detrimental as token ratios increase.

Trusting LLM accuracy gains from sampling multiple answers ($k$) requires vastly more samples than commonly assumed; this work cuts certification costs by up to 47% by accounting for inter-question variance.

LLM pre-training power laws are not fixed but dynamically shaped by learning-rate and batch-size schedules, with a novel forcing-memory surrogate accurately predicting loss across these regimes.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Mixing auxiliary data doesn't just offer more tokens—under the right spectral alignment and dataset growth ratio, it provably accelerates the downstream power-law scaling exponent itself.

Adding uninformative ambient dimensions completely breaks kurtosis-based projection pursuit, proving that high-order unsupervised discovery fundamentally requires second-order subspace pre-filtering to avoid hallucinating structure in pure Gaussian noise.

Pumping capacity into feature cross-networks yields diminishing returns, but harvesting non-shared estimator variance via recursive weight-sharing unlocks state-of-the-art CTR accuracy without standard ensemble deployment overhead.

Standard next-token adaptation fails on financial time series unless fine-tuning objectives explicitly penalize rollouts for distorting higher-order temporal structures like volatility clustering.

Optimizers can dramatically alter step-by-step training trajectories, but a conserved geometric invariant ($2\alpha_r + \alpha_t = 1$) locks every algorithm from SGD to Adam and Muon into the exact same $D^{-1/3}$ optimal data scaling wall.

A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.

Uniform layer looping wastes latent compute on trivial tokens, but dynamically routing iteration depth beats standard transformers on AIME with a 53% steeper test-time scaling slope.

Frontier pretraining runs routinely stake tens of millions of dollars on scaling laws, yet the empirical pipelines used to fit and extrapolate those curves have never been systematically benchmarked until now.

Parallel exploration beats sequential refinement when scaling agent test-time compute, but early-stage trajectory errors create persistent bottlenecks that deeper search and downstream model grafting struggle to overcome.

Pretrained visual encoders are a temporary crutch: encoder-free multimodal architectures close the performance gap with encoder-based systems at roughly $10^{22}$ FLOPs, well within current frontier pretraining budgets.

ZooWork-ShopRanker, a family of e-commerce rerankers aligned to judge-labeled shopping preference, and ShopRank-Bench, a contamination-limited benchmark of ~10,000 private-traffic preference pairs in both text formats, tiered by how many judge families committed to each label.