Scaling Laws & Emergent Abilities - Weekly Roundup

Unlimited OCR can transcribe dozens of pages in one go, overcoming the memory bottlenecks that plague traditional OCR models.

Solving for hyperparameters in spline regression can be done in closed form, achieving exhaustive search accuracy with dramatically less computation.

Achieving near-zero loss barriers in model merging could revolutionize how we connect and utilize billion-parameter transformers.

Switching to polynomial activation functions allows minimal neural networks to master Game of Life dynamics, challenging the notion that bigger is always better in neural network design.

Sublinearly structured DNNs not only achieve feature-learning consistency but also outperform traditional architectures, shedding light on the success of popular CNNs in image classification.

Unlimited OCR can transcribe dozens of pages in one go, overcoming the memory bottlenecks that plague traditional OCR models.

Solving for hyperparameters in spline regression can be done in closed form, achieving exhaustive search accuracy with dramatically less computation.

Achieving near-zero loss barriers in model merging could revolutionize how we connect and utilize billion-parameter transformers.

Switching to polynomial activation functions allows minimal neural networks to master Game of Life dynamics, challenging the notion that bigger is always better in neural network design.

Sublinearly structured DNNs not only achieve feature-learning consistency but also outperform traditional architectures, shedding light on the success of popular CNNs in image classification.

Achieving robust ultra-long-term time series forecasting, Diffusion-LLM outperforms traditional LLMs by leveraging distribution-aware regularization for enhanced generalization.

Extending the prediction horizon of chaotic systems by 2.3x reveals that chaos isn't inherently unpredictable—it's a matter of the right neural architecture.

Prime Fourier Embeddings reveal that modular arithmetic can be efficiently structured, leading to over 500x specialization between relevant and irrelevant channels in neural networks.

An optimal knowledge distribution can significantly enhance LLM knowledge boundaries, outperforming traditional synthesis methods across multiple benchmarks.

Mamba-based OCR can process long paragraphs 1.4 to 4.5 times faster than Transformers, but struggles with handwriting due to data limitations.

Diminishing returns on model size reveal that smarter compute allocation can outperform sheer scale in speech processing tasks.

Allocating more capacity to earlier layers in language models can significantly enhance performance, challenging the long-held uniform layer design paradigm.

Randomized YaRN boosts long-context reasoning performance by exposing models to out-of-distribution positional encodings, yielding impressive gains at extreme lengths.

SVD-Surgeon achieves superior compression of large language models without retraining, enhancing performance while reducing resource demands.

Energy consumption during Transformer fine-tuning can be accurately predicted across various configurations, revealing critical insights for sustainable AI development.

Achieving a 90% boost in prefill throughput for MoE models could redefine the efficiency of large-scale language model serving.

Small language models can outperform leading zero-shot LLMs in relation extraction tasks when fine-tuned on task-specific data, challenging the notion that bigger is always better.

A single fixed RNN can achieve any desired accuracy for continuous functions by simply running longer, challenging the need for new networks with improved target accuracy.

Neural scaling laws reveal unexpected insights into how deep learning models defy classical statistics, reshaping our understanding of model performance in scientific applications.

Compositionality in neural networks only emerges in a narrow depth-connectivity regime, with specific architectural constraints dictating success or failure.

Overparameterized PINNs can self-partition into ineffective modules, but a new training framework restores robust learning and achieves unprecedented accuracy.

Joint modeling of time series with vastly different scales can be achieved without sacrificing accuracy, thanks to a novel self-Adaptive Scale-handling module.

SAC achieves a staggering 2.1x throughput increase by fetching only the essential KV entries for sparse attention models, revolutionizing memory efficiency in LLM inference.

Activating only half the query heads in Transformers can match the performance of fully active models, slashing computational costs while maintaining accuracy.

Lower-order discontinuities in Sparse Mixture-of-Experts architectures dominate input space, significantly affecting model outputs and performance.

BCL achieves significant and consistent performance boosts in information extraction tasks, leveraging Bayesian updates to refine label representations systematically.

Training with large block sizes cripples reasoning performance, but a novel curriculum approach unlocks strong reasoning capabilities in diffusion models.

DLMs reveal surprising trade-offs between performance and computational efficiency that challenge conventional wisdom about language model design.

Sumi challenges the dominance of autoregressive models by achieving competitive performance on key benchmarks while being the first large-scale uniform diffusion language model.

A simple rescaling of the MLM-head can turn unstable training runs into competitive sparse retrieval models, challenging the notion that bigger encoders alone drive performance.

Nonuniform width allocation in transformers can lead to a 22% reduction in FLOPs while enhancing language modeling performance.

Routing accuracy in LLM assistants drops significantly as tool catalogs grow, but embedding-based shortlisting can recover up to 17 percentage points in performance.

Beyond a certain threshold, queue peaks grow logarithmically rather than quadratically, fundamentally altering our understanding of scheduling efficiency in stochastic networks.

Recursive depth in masked diffusion models can dramatically enhance parameter efficiency, enabling models to perform as well as much larger counterparts without the added computational burden.

Predicting unseen links in knowledge graphs can often hinge on just a single half-link, revealing critical insights into model generalization.

SoftMoE achieves comparable or superior performance to traditional sparse MoE while activating significantly fewer experts, revolutionizing expert allocation in LLMs.

A small trade-off in utility can lead to significantly enhanced safety alignment in LLMs, protecting against jailbreak attacks.

Small initialization can dramatically enhance reasoning performance in large language models, revealing a new lever for improving AI capabilities.

Fixed-budget evaluations can significantly underestimate the capabilities of advanced language models, with larger token budgets unlocking their true potential across diverse tasks.

TivTok redefines video tokenization by enabling the reuse of persistent information across frames, achieving unprecedented compression efficiency with minimal token usage.

Model merging can achieve superior stereo matching accuracy with minimal retraining time, slashing error rates while preserving module-specific knowledge.

A two-loop configuration in LoopCoder-v2 boosts code generation performance by over 50% compared to a non-looped baseline, while more loops actually hinder results.

Overtraining in SFT can lead to rank inversion, with GRPO performance plummeting despite improved pre-RL metrics, highlighting a critical failure mode in RLHF training.

JetSpec shatters the speed ceiling of speculative decoding, achieving up to 9.64x acceleration on complex tasks while maintaining high acceptance rates.

Qwen-RobotManip achieves a 20% relative improvement over the previous state-of-the-art in robotic manipulation, showcasing unprecedented generalization capabilities from diverse, open-source datasets.

Hyperball achieves a remarkable 20-30% speedup in language model pretraining, even as model sizes grow, challenging the limits of traditional optimizers.

Tying expert parameters across layers can halve memory usage in MoE models without sacrificing performance, revolutionizing how we scale LLMs.

A dual-path attention mechanism reveals that slower coupling does not improve performance but offers a structured mapping that could redefine hierarchical pretraining strategies.

Layer span variation can unlock a new dimension of deterministic rollout diversity, boosting performance by over 10 percentage points on reasoning tasks.

Achieving 72.1% fewer reverse steps without sacrificing accuracy, \textsc{LESS} redefines efficiency in diffusion language models.

SMEPilot achieves up to 3.94× faster LLM inference by intelligently balancing workloads between CPU and SME units based on operator characteristics.

Moderate geometric convergence in LLMs can coexist with near-perfect functional transfer, challenging assumptions about structural alignment across architectures.

ASRD leverages trusted Anchor Tokens to dramatically improve decoding quality and speed in diffusion LLMs, achieving a 6.4% accuracy boost while accelerating inference by 7.2x.

Achieving significant cost reductions in quantum computing by optimizing shot execution could revolutionize how we approach quantum algorithm deployment.

A passive optical system can outperform advanced digital forecasting models, achieving significant accuracy gains without complex sequence mixing.

Grokking in DNNs is driven by noise-induced escapes from metastable states, revealing a surprising connection between regularization strength and the emergence of learnable features.