Training Efficiency & Optimization

A specific looped Transformer architecture and input injection strategy unlock significant gains in reasoning and context utilization, outperforming larger models with drastically fewer parameters.

Distilling explicit, reward-guided skills into VLMs without the student model ever seeing the skills themselves unlocks superior multimodal performance.

Generative time series models can now produce multiple plausible future trajectories as fast as quantile models query a single point, unlocking accurate dependency modeling for critical applications.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

Gating policy distillation with directional agreement between early and late RL shifts unlocks more effective knowledge transfer, boosting LLM performance on reasoning tasks.

Assigning reward credit to specific tokens, rather than uniformly, dramatically improves multi-reward RL for LLMs.

Diffusion models can learn new objectives by stochastically routing attention within their existing architecture, achieving 4x faster RL training without sacrificing generation quality.

Regularized importance sampling in LLM RL breaks reward shift invariance, but a covariance-based estimand recovers this property, with distinct centering strategies impacting bias and variance.

Simultaneously evaluate $K$ policies in linear contextual bandits with near-optimal sample complexity by sharing information across evaluations via a unified ridge regression estimator.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

A single model query can now estimate the full acceptance probability for parallel masked diffusion, slashing rejection sampling costs.

Lossless compression of sparse ReLU activations slashes LLM activation VRAM by up to 31% with near-zero overhead.

Exponential Hopfield networks learn exponentially more archetypes than pairwise models by leveraging softmax attention, but risk memorizing noisy examples over true archetypes.

A simple byte reordering strategy slashes multilingual boundary prediction errors by over 70% in scripts like Korean and Chinese, fundamentally altering how byte-level models perceive codepoint transitions.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

Standard MIL suppresses secondary findings in coexisting lesions; this novel framework recovers them by modeling the dominant-label annotation process.

Deep learning compilers are silently introducing bugs through state retention, with 35 stateful issues found in just four frameworks.

Offline-trained neural networks are mathematically proven to diverge in closed-loop control, but bounded online adaptation offers a practical solution.

A simple, fixed-budget best-arm identification algorithm can be made nearly as adaptive as state-of-the-art methods by optimizing its phase schedule, achieving $11/12$ of the optimal normalized rate for Gaussian arms.

Neural networks preferentially learn low-frequency sinusoidal patterns due to the inherent graded structure of their gradient descent dynamics, explained by an orthogonal polynomial framework.

Fine-grained data balancing in heterogeneous pipelines unlocks 2x training speedups for large models by overcoming memory bottlenecks and coarse workload assignments.

Hundreds of billions of midtraining tokens on a curated software engineering corpus unlock substantial gains in agentic capabilities, outperforming standard SFT alone.

A single weight-space encoder now masters 100 transformer architectures, outperforming specialized models and handling unseen, larger networks with less memory.

Continual learning, aiming to improve deepfake detector accuracy over time, systematically intensifies unfairness across demographic groups.

A single INR can now model complex, periodic deformations across entire image sequences, enabling self-regularization and outperforming prior methods.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

Adaptive planning horizons emerge naturally from policy confidence, outperforming fixed discount factors on challenging credit assignment problems.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Single-qubit quantum circuits, despite universal function approximation, get stuck in training due to "phase-locking," requiring orders of magnitude more optimization steps than multi-qubit equivalents.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.

A novel PI-DeepONet implementation slashes training time by up to $3.75\times$ and makes 3D physics-informed learning feasible, overcoming critical computational barriers.

Byte-level LLMs can achieve a 25% jump in math accuracy by patching where text is *new*, not just surprising.

Consolidating specialist LLMs into a single, more accurate generalist is possible with *zero* gradient updates, relying solely on forward passes and input statistics.

An adaptive acquisition function learned online via temporal difference learning discovers optimal perturbation experiments 2x more effectively than fixed methods.

A single LR gradient equation explains the shape of learning-rate transfer curves across diverse settings, revealing layer-specific sensitivities and theoretical bounds.

Generative models can inject crucial scientific knowledge into GP posteriors, recovering hidden modes and improving predictive accuracy where traditional GPs fail.

Cost-penalized active feature acquisition is shown to be a direct Lagrangian relaxation, with a computable certificate for greedy suboptimality and a structural guarantee in the near-submodular regime.

In-context learning emerges significantly earlier than specialized prediction, revealing a fundamental trade-off in how LLMs adapt to novel data based on pretraining scale.

Clients can achieve superior worst-decile accuracy in federated learning by stopping local training based on an optimal stopping framework that balances loss reduction against drift, outperforming tuned baselines and existing adaptive methods.

Offline risk-sensitive RL is now provably efficient, with new algorithms and tight bounds for dynamic and static optimized certainty equivalents.

A neural evaluator predicts Lean 4 proof acceptance with 87.8% accuracy and achieves 4.2x the throughput of a multi-worker Lean baseline, drastically accelerating proof discovery.

Full-page document parsing models can now pinpoint and correct local errors by assigning credit directly to tokens, outperforming baselines across six benchmarks.

A single forward pass now reveals which input variables matter for PFN predictions, achieving exact FDR control and outperforming lasso, BART, and other methods on sparse nonlinear regression.

Agentic RL agents can learn to efficiently manage context and subtasks across multiple sessions by estimating the marginal contribution of each session, outperforming naive advantage broadcasting.

Injecting uncertainty samples into DNNs during federated training unlocks robust, near-centralized performance for joint communication and computation resource allocation in XR systems, outperforming state-of-the-art.

Train LLMs with up to $1.455\times$ less memory and enable $3.5\times$ longer context lengths by losslessly compressing tensors on-the-fly.

Early stopping of Randomized Block Coordinate Descent can achieve optimal generalization rates in nonconvex learning, a feat previously unquantified.

A frozen, 16MB vocabulary can match or beat learned embeddings in small LLMs, unlocking dramatic speedups and memory reductions for edge deployment.

Most self-evolving LLM updates are useless, yet current methods inherit them all; new frameworks drastically cut harmful updates and preserve valuable experience.