Scaling Laws & Emergent Abilities

Targeting Super Weights in LLMs can lead to performance collapse, challenging assumptions about parameter importance and trainability.

Judge upgrades in LLMs aren't interchangeable—only specific parameter increases yield reliable improvements in evaluation consistency.

GPU frequency behavior reveals inter-kernel dependencies that could revolutionize latency-prediction models in machine learning.

Achieving extreme low-bit compression without lookup tables could revolutionize how we deploy large language models in memory-limited settings.

Layer patching can dramatically enhance model performance in size interpolation, revealing that simple strategies often outperform complex methods.

Information restriction is the key to understanding how diffusion models can generalize rather than memorize, revealing a precise phase boundary that could transform generative AI practices.

Memory compaction in LLMs is fundamentally flawed, with critical information often discarded before it's needed, revealing a systemic inefficiency across all layers.

UltraX achieves the highest average performance across datasets while using fewer training tokens, redefining efficiency in data refinement for LLMs.

Frontier models falter as task difficulty scales, revealing critical gaps in long-context reasoning capabilities.

Hidden Decoding achieves unprecedented performance improvements in large language models by scaling computation along the sequence length without modifying the Transformer architecture.

Random sampling can bridge the gap between small and large input sizes, revealing how models generalize across dimensions with surprising efficiency.

High guidance in classifier-free guidance can destabilize models, but a simple adjustment can stabilize performance without extra computational cost.

Autoregressive Chain-of-Thought learning can achieve optimal sample complexity without being hindered by rollout length, thanks to a new stability measure called parity dimension.

DiPhon enables the generation of large graphs from small training samples while preserving their core topological properties, revolutionizing scalable graph generation.

Concentrating model capacity on delegation roles can yield substantial performance gains in hierarchical search agents, revealing a critical bottleneck in task decomposition.

TF-Engram achieves a notable performance boost in LLMs by integrating scalable, train-free semantic memory without the typical overhead of retraining.

Achieving up to 31.8% lower latency in multi-GPU systems could redefine how we optimize Large Language Model serving under strict latency constraints.

Achieving over 96% inter-node weak scalability with a grid size of 1 billion cells transforms the capabilities of flow simulations in computational fluid dynamics.

Kimi Delta Attention with Muon outperforms other architectures in validation loss, while introducing Cross-Layer Value Routing reveals new avenues for optimizing memory management in linear attention models.

Scaling linear RNNs with Sparse Delta Memory leads to dramatic gains in long-context recall without increasing computational overhead.

Evolving LLMs into a heterogeneous intelligent ecology could fundamentally alter how we approach AI alignment and governance.

NTK regression can require exponentially more samples than necessary for compositional tasks, revealing a critical gap in our understanding of neural network performance.

Finite-width neural networks converge to their Gaussian-process limits with error bounds that shrink as the network width increases, revealing a surprising robustness across architectures.

Infinite-width Bayesian neural networks can learn complex functions with the same efficiency as their finite-width counterparts, revealing a surprising equivalence in learnability.

FreqDepthKV achieves a remarkable 3.9x effective compression ratio while preserving high accuracy in long-context LLM tasks, revolutionizing cache management.

Real-world multivariate data boosts zero-shot generalization in time series models, outperforming synthetic counterparts by a significant margin.

The best-informed LLM agent can absorb nearly the entire wealth pool in a coupled economy, revealing stark limitations in our understanding of agent dynamics.

MemDefrag reveals that a simple tracing mechanism can boost knowledge retention in LLMs by over 25% after multiple memory updates.

Learning from real-world environments follows a precise log-sigmoid scaling law, with agent performance and learning speed improving dramatically over time.

Hyperparameter transfer can significantly boost GNN performance, revealing that even small adjustments can yield large gains in training efficiency and effectiveness.

Grokking is not just a static property; it's a delicate phase transition that can be easily disrupted by minor changes in computational conditions.

Foundation models can decisively outperform classical methods in time series forecasting, but only under specific data conditions—knowing when to deploy them is crucial for efficiency.

Few-shot binding in transformers hinges on the interplay of input pathways and code readability, revealing that traditional symbolic routes may fail at critical readout stages.

Training a neural network with $L^4$ loss enables it to compute more functions than neurons, revealing a surprising efficiency in representation.

SCALA achieves human-level sample efficiency by mimicking cognitive selectivity, allowing models to excel in visual recognition with minimal data.

Depth in neural networks may not enhance expressivity as previously thought, revealing that functional diversity is constrained by norm-controlled complexity.

Continuous scoring from LLM-as-a-Verifier leads to state-of-the-art verification accuracy and improved sample efficiency in reinforcement learning tasks.

Consensus decision protocols significantly boost performance in knowledge-intensive tasks, reshaping how we leverage multi-agent systems in LLMs.

Simple Best-of-$N$ sampling can outperform complex guided search methods in text-to-image generation, revealing a critical flaw in how we evaluate efficiency in diffusion models.

Joint commitment in dLLMs can lead to substantial accuracy gains, especially in reasoning tasks, by coordinating token selection more effectively.

Energy consumption in video generation models can be accurately predicted without direct access to model details, revealing a surprising adherence to scaling laws.

Sangam slashes latency for diffusion language models by intelligently managing prefill and decode processes, revealing a new paradigm for efficient LLM serving.

Cold start latencies can be slashed by up to 99.3% with CoCoScale's innovative layer-wise scaling approach.

Pretraining loss misleads model selection, wasting compute, while a simple scaling rule unlocks superior performance in pixel-wise Earth-observation models.

HiLS-Attention achieves over 64x context length extrapolation with 90% retrieval accuracy, outperforming traditional full attention mechanisms.

A single-layer spiking neural network can achieve robust in-context learning without the need for attention mechanisms or deep architectures.

Scaling synthetic data isn't just about generating more; fixed-source synthesis reveals surprising limits to performance gains.

A simple data-driven approach outperforms complex models in ultrasound understanding, revealing the power of scale and alignment.

Organizing views into diversity-aware chunks can drastically enhance the performance of geometry transformers while slashing memory costs and inference times.

Scaling LLMs significantly boosts social simulation accuracy in well-represented domains, but fails to enhance calibration for human cognitive biases.