Training Efficiency & Optimization - Weekly Roundup

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

Diffusion models can learn lattice field theories without ever seeing real data, matching HMC performance.

Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.

A single dimensionless motion prior generalizes across vastly different object families and data modalities, outperforming specialized methods without retraining.

Agents can now recursively evolve their own training environments, jointly building and validating tasks, contexts, and verifiers for robust long-horizon learning.

Diffusion models can learn lattice field theories without ever seeing real data, matching HMC performance.

Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.

A single dimensionless motion prior generalizes across vastly different object families and data modalities, outperforming specialized methods without retraining.

Agents can now recursively evolve their own training environments, jointly building and validating tasks, contexts, and verifiers for robust long-horizon learning.

Generative models, when framed as proposal distributions for planning, unlock efficient long-horizon reasoning in model-based RL, outperforming diffusion policies and model-free methods.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

A small, training-free "brief thinker" can cut LLM reasoning tokens by 31% without sacrificing accuracy.

Larger minibatch optimal transport couplings in flow matching, enabled by a novel entropic approximation, significantly boost training stability and generation performance.

Quality-aware pretraining strategies, particularly approach-aware episode conditioning, can significantly outperform plain behavioral cloning in VLA tasks, with optimal choices depending on label granularity and model architecture.

Neural networks now jointly infer CRN structure and kinetics from complex time-series data, outperforming traditional ABC methods.

A learned "Token-to-Go" progress meter enables multi-turn RL agents to effectively learn from sparse rewards, boosting performance by up to 7.2 points without increasing training costs.

Robots can't truly improve themselves yet; current systems lack the crucial ability to recursively refine their own learning processes based on physical experience.

Larger RLVR batches, despite appearing efficient with square-root learning rates, suffer a critical batch size that degrades sample efficiency and hastens failure due to curvature effects.

SDE Lyapunov estimates now directly translate to finite-time SGD convergence, simplifying analysis for novel optimizers.

Recursive message passing achieves 99% Sudoku-Extreme accuracy with just 16k parameters, outperforming larger neural baselines on N-queens and SAT with zero-shot generalization.

Offline Q-learning can achieve $\widetilde{O}(1/ε)$ sample complexity in finite-horizon KL-regularized MDPs, even without explicit pessimism, by leveraging KL regularization.

LLM agents can learn complex tasks more efficiently by adaptively injecting skills only when their utility is confirmed through outcome-aware likelihood comparisons on existing data, bypassing expensive environment rollouts.

Transformers can be quantitatively predicted by a Bayesian reference, revealing their failure modes on complex, interdependent sequential data.

Closed neighborhoods in planar graphs can be taught with at most two non-clashing examples, resolving a long-standing conjecture, yet distinguishing between one and two teaching examples is NP-hard.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Grokking's distinct phases—nuisance fitting, core learning, and delayed generalization—are analytically separable, with sharp conditions for occurrence and predictable plateau lengths.

A simple greedy approach offers a $\frac{1}{\sqrt{e}}$-approximation for optimal policies in heterogeneous asynchronous multi-player bandits, a problem previously intractable.

Token-wise importance sampling in off-policy RLVR is now provably unbiased and practically superior, outperforming GRPO by up to 2.58% on math benchmarks.

Manufacturing AI can now learn to avoid costly negative transfer by selectively leveraging prior knowledge, achieving 20% lower carbon footprint prediction error with minimal target data.

Heavy-tailed noise doesn't derail clipped SGD's last-iterate convergence, achieving near-optimal rates even with adversarial gradient attacks.

Pruning LLMs while preserving their influence functions is now over 28x faster and more effective, unlocking better data selection for downstream tasks.

Refresh timing, not just frequency, dictates Kronecker-preconditioned LLM training performance, with optimal placement flipping based on learning rate.

Training-free conditional sampling for Schrödinger Bridges is now possible, achieving state-of-the-art results on image generation and inverse problems by twisting SB's inherent endpoint prediction capabilities.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

A simple post-hoc geometric calibration method recovers lost out-of-domain capabilities and boosts in-domain performance after SFT, even on large, diverse models.

Self-distillation methods fail to generalize reasoning gains to unseen problems and their gradients offer weak policy improvement signals, challenging their efficacy beyond in-distribution imitation.

Optimistic FTRL achieves linear convergence to Nash equilibria in harmonic games, a significant improvement over prior sublinear bounds.

LLMs can achieve superior KV cache compression by intelligently allocating quantization budgets between layer-wise and token-wise strategies, outperforming uniform or single-granularity approaches.

A cheap diagnostic predicts model alignment and merging success with $0.91$ correlation, eliminating costly empirical search.

Robot foundation models can now learn new tasks rapidly and retain old ones, slashing real-world training time by up to 68.5% without significant forgetting.

Causal inference estimation rates are surprisingly sensitive to covariate distribution smoothness, with rough designs demonstrably slowing down pointwise CATE estimation even when treatment effects and nuisance functions are well-behaved.

LLM agents can now dynamically design their own post-training recipe spaces, leading to a 2.6x performance boost on complex multi-domain tasks.

Robust generative modeling is achieved by replacing squared error with geometric median regression in Flow Matching, outperforming standard methods under severe data contamination.

Deterministic forecasting embedded within generative models is unstable; DV-Flow decouples it as a parallel velocity, unlocking stable end-to-end training and superior probabilistic forecasts.

Multi-objective bandits can make finding the best arm *easier* than single-objective settings, even identifying larger optimal sets with fewer samples.

Global optimization unlocks LLM fine-tuning with drastically reduced compute and memory, matching or exceeding gradient-based gains.

Dependent sampling doesn't break KMM risk estimation; effective sample sizes derived from mixing properties provide robust finite-sample guarantees.

Circulant permutations unlock provably lower variance and storage for MinHash beyond CMinHash, with a new analytical tool that accurately predicts Minner estimator performance.

Quantum state preparation can achieve four orders of magnitude lower infidelities for specific states by embracing risk-seeking optimization, rather than solely optimizing for average fidelity.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

A theoretically grounded EI algorithm for continuous simulation optimization achieves simple-regret guarantees by learning unknown heteroscedastic noise and optimizing incumbent/recommendation strategies based on regret bounds.

Equivariant MLIPs just got a 250x speedup: LunaTP unlocks efficient tensor products by rethinking parallel execution.

Few labels per question are sufficient for near-optimal recalibration when information is pooled across related tasks, with identification regret decaying exponentially with label count.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

A conditional energy prior explicitly guides latent space towards perturbation-specific signals, unlocking an 83.2% improvement in perturbation recognition for cellular response prediction.

Adam's implicit bias offers a practical defense against training data reconstruction attacks, unlike SGD and Muon which leave reconstructible fingerprints.

Data replay methods can recover up to 88% of performance in task-incremental radar waveform classification, with synergistic combinations of replay mechanisms boosting accuracy by over 5%.

Representation reuse hinges not on source-task performance, but on preserving task-relevant distinctions and ensuring their consistent interpretability for downstream models.

Contextual bandits with lumpable structure can be solved with $r$ times less regret than previously thought possible.

Two residual streams in diffusion language models halve the performance gains of adding a second stream by enabling distinct read and write routing patterns.

Over-parameterization's impact on robustness is not monotonic; solution geometry and optimization path, not just parameter count, dictate sensitivity in matrix sensing.

Unknown GP hyperparameters can drastically distort tail-risk estimates in CVaR optimization, but a new algorithm robustly navigates this uncertainty to achieve near-oracle performance.

Distributed attention can achieve near-zero communication overhead for mask localization and compute-communication overlap, unlocking massive throughput gains for long-context LLMs.

Explicitly modeling copy operations unlocks substantial improvements in LLM efficiency and performance, even for models already adept at in-context learning.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Quantization strategies can achieve up to 4x energy savings for LLM inference, with peak efficiency observed for long inputs and short outputs.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

Embodied agents improve on complex tasks by recombining learned skills, but their ability to extract value from additional experience saturates, failing to scale with learning samples.

Slash LLM evaluation costs by over 270x without sacrificing accuracy, revealing that even top open-weight models struggle to outperform simple baselines on AML transaction fraud detection.

LLM distillation can achieve 18.55% higher math accuracy and 13.8 points better ALFWorld success by intelligently branching student trajectories at uncertain decision points, not just at turn starts.

Reasoning supervision can be compressed over 450x with uncertainty-guided contraction, boosting LLM efficiency without sacrificing capability.

Batching future MH proposals unlocks significant speedups for LLM power sampling without altering the core reasoning process.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

One million parallel environments are insufficient without intelligent sampling: Success Guided Sampling unlocks mega-scale RL for robots by focusing training on the edge of learned capabilities.

Abstracting human demonstrations into scene graphs unlocks zero-shot generalization for dexterous manipulation policies, achieving 71% better performance on unseen tasks than imitation-heavy methods.

A single recurrent Transformer block can match the accuracy of deep vision encoders with 70% fewer parameters by dynamically programming FFN experts via a continuous depth coordinate.

Policy optimization for adversarial MDPs now matches state-of-the-art regret bounds by stabilizing local updates with regularized Q-functions, eliminating the horizon gap.

Source-side training dynamics, not just current values or training time, prospectively predict OOD degradation, even across model architectures.

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

A single regional weather model can now leverage multiple global forecast drivers without retraining, achieving kilometer-scale accuracy.

Optimal oracle complexity for nonsmooth nonconvex composite optimization is finally achieved, matching noncomposite rates by cleverly incorporating the convex regularizer into online learning losses.

LLM value caches are surprisingly redundant, allowing for significant memory compression through simple symmetry exploitation without hurting inference speed.

Unifying density ratio estimation and distribution transport, this work reveals how a single displacement field can simultaneously capture statistical shifts and dynamical transformations between probability distributions.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Input-specific parameter generation unlocks adaptive time-series forecasting, outperforming static adaptation and improving zero-shot generalization.

Training higher-order Topological Neural Networks on massive graphs like Reddit is now possible, thanks to a novel clustering-based mini-batching strategy that slashes GPU memory requirements by over 80%.

Rare concepts in LLM fine-tuning are significantly hindered by "prior barriers" that require targeted supervision, not just more data.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Most published TTA methods for time-series forecasting are outperformed by simple, hyperparameter-free baselines under realistic label delays, with some even becoming harmful.

Reverse-mode gradients slash Bayesian inference time for expensive simulators, especially when parameters vastly outnumber outputs.

Spatio-temporal Bayesian optimization for point-process data is now scalable and efficient, achieving accurate peak discovery with minimal evaluations.

State-preserving constraints in Bayesian optimization are transformed into an efficient, anisotropic ellipsoid in design space, enabling robust optimization of complex engineering systems.

Reasoning models can learn to plan by reverse-engineering solution ideas from hindsight, even when initially unable to solve the problem.

LLMs can now autonomously learn to self-improve at test-time, nearly doubling utility by intelligently selecting and applying TTT strategies.

A training-free change detection method achieves near-optimal performance by integrating diffusion models to precisely estimate Hyvärinen scores, slashing detection delays by over 90% in simulations.

Weisfeiler-Lehman graph kernels unlock superior structural features for algorithm selection, outperforming traditional methods on challenging constraint satisfaction problems.

Gradient-based learning is now possible for variational sequential Monte Carlo without iterative solvers, thanks to a differentiable relaxation of systematic resampling.

QNN gate selection and arrangement, not just qubit count, critically dictate financial forecasting performance, with the best configuration outperforming classical BiLSTM and exhibiting significant noise sensitivity on current quantum hardware.

Random features and projections unlock efficient quadratic entropy estimation for complex distributions and distances, revealing nuanced interdisciplinary research landscapes.

End-to-end diffusion models can achieve state-of-the-art precipitation nowcasting with over 17x faster inference by forecasting directly in radar space.

Optimal regret in adversarial MDPs with stochastic hard constraints is finally achieved, bridging the gap between existing upper and lower bounds by achieving $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$ regret.

Iterative refinement in data space, not flow space, unlocks fast, high-fidelity generative sampling without manual timestep tuning.