Training Efficiency & Optimization - Weekly Roundup

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

Embodied agents improve on complex tasks by recombining learned skills, but their ability to extract value from additional experience saturates, failing to scale with learning samples.

Slash LLM evaluation costs by over 270x without sacrificing accuracy, revealing that even top open-weight models struggle to outperform simple baselines on AML transaction fraud detection.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

Embodied agents improve on complex tasks by recombining learned skills, but their ability to extract value from additional experience saturates, failing to scale with learning samples.

Slash LLM evaluation costs by over 270x without sacrificing accuracy, revealing that even top open-weight models struggle to outperform simple baselines on AML transaction fraud detection.

LLM distillation can achieve 18.55% higher math accuracy and 13.8 points better ALFWorld success by intelligently branching student trajectories at uncertain decision points, not just at turn starts.

Reasoning supervision can be compressed over 450x with uncertainty-guided contraction, boosting LLM efficiency without sacrificing capability.

Batching future MH proposals unlocks significant speedups for LLM power sampling without altering the core reasoning process.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

One million parallel environments are insufficient without intelligent sampling: Success Guided Sampling unlocks mega-scale RL for robots by focusing training on the edge of learned capabilities.

Abstracting human demonstrations into scene graphs unlocks zero-shot generalization for dexterous manipulation policies, achieving 71% better performance on unseen tasks than imitation-heavy methods.

A single recurrent Transformer block can match the accuracy of deep vision encoders with 70% fewer parameters by dynamically programming FFN experts via a continuous depth coordinate.

Policy optimization for adversarial MDPs now matches state-of-the-art regret bounds by stabilizing local updates with regularized Q-functions, eliminating the horizon gap.

Source-side training dynamics, not just current values or training time, prospectively predict OOD degradation, even across model architectures.

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

A single regional weather model can now leverage multiple global forecast drivers without retraining, achieving kilometer-scale accuracy.

Optimal oracle complexity for nonsmooth nonconvex composite optimization is finally achieved, matching noncomposite rates by cleverly incorporating the convex regularizer into online learning losses.

LLM value caches are surprisingly redundant, allowing for significant memory compression through simple symmetry exploitation without hurting inference speed.

Unifying density ratio estimation and distribution transport, this work reveals how a single displacement field can simultaneously capture statistical shifts and dynamical transformations between probability distributions.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Input-specific parameter generation unlocks adaptive time-series forecasting, outperforming static adaptation and improving zero-shot generalization.

Training higher-order Topological Neural Networks on massive graphs like Reddit is now possible, thanks to a novel clustering-based mini-batching strategy that slashes GPU memory requirements by over 80%.

Rare concepts in LLM fine-tuning are significantly hindered by "prior barriers" that require targeted supervision, not just more data.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Most published TTA methods for time-series forecasting are outperformed by simple, hyperparameter-free baselines under realistic label delays, with some even becoming harmful.

Reverse-mode gradients slash Bayesian inference time for expensive simulators, especially when parameters vastly outnumber outputs.

Spatio-temporal Bayesian optimization for point-process data is now scalable and efficient, achieving accurate peak discovery with minimal evaluations.

State-preserving constraints in Bayesian optimization are transformed into an efficient, anisotropic ellipsoid in design space, enabling robust optimization of complex engineering systems.

Reasoning models can learn to plan by reverse-engineering solution ideas from hindsight, even when initially unable to solve the problem.

LLMs can now autonomously learn to self-improve at test-time, nearly doubling utility by intelligently selecting and applying TTT strategies.

A training-free change detection method achieves near-optimal performance by integrating diffusion models to precisely estimate Hyvärinen scores, slashing detection delays by over 90% in simulations.

Weisfeiler-Lehman graph kernels unlock superior structural features for algorithm selection, outperforming traditional methods on challenging constraint satisfaction problems.

Gradient-based learning is now possible for variational sequential Monte Carlo without iterative solvers, thanks to a differentiable relaxation of systematic resampling.

QNN gate selection and arrangement, not just qubit count, critically dictate financial forecasting performance, with the best configuration outperforming classical BiLSTM and exhibiting significant noise sensitivity on current quantum hardware.

Random features and projections unlock efficient quadratic entropy estimation for complex distributions and distances, revealing nuanced interdisciplinary research landscapes.

End-to-end diffusion models can achieve state-of-the-art precipitation nowcasting with over 17x faster inference by forecasting directly in radar space.

Optimal regret in adversarial MDPs with stochastic hard constraints is finally achieved, bridging the gap between existing upper and lower bounds by achieving $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$ regret.

Iterative refinement in data space, not flow space, unlocks fast, high-fidelity generative sampling without manual timestep tuning.

Contextual contamination in few-shot vision-language learning is mitigated by a novel semantic purification and reinforcement-learning gating mechanism, unlocking superior performance.

KL regularization's failure in group policy optimization stems from subtle reward interactions, but a new calibration method unlocks performance gains.

Interpretable archetypes emerge from discrete data using a principled likelihood framework, unlocking insights in genomics and microbiome analysis previously inaccessible to standard AA.

A reverse Hebbian-inspired pruning method achieves 99% sparsity on ResNet-18 with only a 2.4% accuracy drop, dynamically allocating sparsity without layer-specific targets.

Dynamic critic pessimism, guided by geometric disagreement, unlocks superior locomotion learning and efficiency in off-policy actor-critic methods.

Heterogeneous experts can be dynamically coordinated via MoE gating to form composite model services, outperforming monolithic models and standard MoE in cost-aware marketplaces.

Adversarial outliers can be effectively handled in linear dynamical system identification by leveraging group-sparsity and robust estimation techniques, achieving provable guarantees.

Spectral analysis reveals that increasing latent dimensions in Koopman autoencoders improves dynamics learning by concentrating learned eigenvalues in low-residual regions, not just by reducing one-step prediction error.

Structured linear dynamical systems can be learned from drastically undersampled observations ($p=o(1)$) with trajectory lengths far shorter than previously thought possible.

Pruning LLMs for decoding speedup is significantly improved by calibrating sparsity with actual generation activations, not just static text, and optimizing SpMV kernels.

Out-of-distribution data can dramatically boost discrete diffusion model performance under extreme data scarcity by strategically injecting it at low noise levels.

LLMs adapt neural operator configurations using both a strong task-dependent prior and real-time experimental feedback, outperforming Bayesian optimization.

Quantizing AdamW's second moment in preconditioner space, not state space, slashes validation loss gaps by up to 70% in 4-bit training.

Kernel methods finally offer a scalable and interpretable alternative to black-box ANNs for learning Koopman embeddings, matching or surpassing ANN performance on complex dynamical systems.

Synergistic multimodal information, previously elusive, is unlocked by modeling higher-order statistical dependencies via tensor decomposition.

Auxiliary "ghost tasks" make any linear differential equation system effectively parametrizable for Gaussian Processes, solving a key limitation in physics-informed ML.

Diffusion models bypass Langevin's condition-number bottleneck by leveraging time-dependent score trajectories, achieving sampling error independent of target covariance ill-conditioning.

Agents learning to optimize local rewards in spatial public goods games can inadvertently create detrimental spatial patterns that reduce collective welfare by over 40% compared to random movement.

Interleaving source tasks with verification can save unboundedly many expected calls, but only if the verifier is exact; noisy verifiers flip this, demanding unbounded information and error.

Synthetic data alone can match real-world GAD performance, but unlocking further gains requires models that explicitly fuse topological and semantic evidence.

Uncertainty in loss distributions cripples standard risk-averse ML; this work shows credal sets can reliably prevent catastrophic decisions without sacrificing average performance.

Parameter-efficient adaptation methods can match full fine-tuning gains for tabular foundation models, while selective aggregation offers a clear path to improved predictions over ensemble averaging.

Spectral learning unlocks $\mathcal{O}(d\log d)$ covariance estimation for dense Gaussian dependencies, outperforming prior methods in forecasting and prediction tasks.

Self-testing a KV cache with targeted Q&A pairs, rather than rereading the whole context, dramatically boosts memory retention, especially under severe budget constraints.

Human tactile gloves costing under $11 can now generate robot-ready tactile data, slashing acquisition costs by over 95% and boosting efficiency 3.5x for dexterous manipulation.

Posterior sampling recovers signals from noisy one-bit compressed sensing with near-optimal sample complexity, even with learned priors that are only approximately correct.

Self-improving AI systems can achieve stable learning dynamics and persistent utility advantages by strategically managing objective drift through co-evolutionary symbolic archives.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Key contribution not extracted.

Key contribution not extracted.

DiffusionART achieves state-of-the-art RL fine-tuning for diffusion models, reaching peak performance with over 12x fewer GPU hours and substantially higher rewards than existing methods.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

Achieve state-of-the-art image generation quality at a fraction of the compute cost by dynamically adapting tokenization to visual complexity.

AI agents can learn to climb leaderboards in complex adversarial games by refining executable policies, but struggle with intricate rule sets

Language models can actively select replay targets to improve their own broad capability preservation, not just retain explicitly trained tasks.

LLM agents can learn from past mistakes by retroactively revising their internal "harness" state, significantly improving continual learning performance.

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

A novel contrastive feedback mechanism allows LLM agents to learn skill proposals that are aligned with the current actor's behavior without requiring expensive skill evaluations.

LLM agents can dynamically evolve their skill libraries, preventing obsolescence and improving long-horizon task performance by up to 7.8% through a fitness-driven lifecycle.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Adversarial shifts in bandit arm means, not just gaps, break standard best-arm identification algorithms, requiring a novel importance-weighting approach for robust performance.

Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Forget-only unlearning demands models memorize far more than standard training, challenging the premise of efficient deletion.

Self-distillation can degrade student performance if the teacher exploits privileged information unavailable to the student, but a novel KL-regularized objective ensures teacher guidance aligns with student capabilities.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Gradient clipping, often seen as a heuristic, unlocks optimal convergence and linear agent speed-up for decentralized SGD under heavy-tailed noise, outperforming normalization in this regime.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

Agnostic smoothed online learning is now achievable without knowing the base measure or smoothing parameter, matching statistical learning's flexibility with online learning's robustness.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

Geometric weighting by update norm offers a subtle but measurable improvement in personalized federated learning, outperforming uniform aggregation and FedPer-R1 on CIFAR-10's native client accuracy.

Average-reward RL for general multichain MDPs is now possible without model knowledge or reduction to discounted problems.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Fresh, shallow network activations can correct diffusion model feature forecasts, unlocking significant speedups and improved fidelity without any parameter updates.

Probabilistic forecasting models can now achieve 65% CRPS reduction with drastically fewer samples by directly learning the distribution shift between successive predictions.