Training Efficiency & Optimization - Weekly Roundup

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

A novel contrastive feedback mechanism allows LLM agents to learn skill proposals that are aligned with the current actor's behavior without requiring expensive skill evaluations.

LLM agents can dynamically evolve their skill libraries, preventing obsolescence and improving long-horizon task performance by up to 7.8% through a fitness-driven lifecycle.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Adversarial shifts in bandit arm means, not just gaps, break standard best-arm identification algorithms, requiring a novel importance-weighting approach for robust performance.

Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Forget-only unlearning demands models memorize far more than standard training, challenging the premise of efficient deletion.

Self-distillation can degrade student performance if the teacher exploits privileged information unavailable to the student, but a novel KL-regularized objective ensures teacher guidance aligns with student capabilities.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Gradient clipping, often seen as a heuristic, unlocks optimal convergence and linear agent speed-up for decentralized SGD under heavy-tailed noise, outperforming normalization in this regime.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

Agnostic smoothed online learning is now achievable without knowing the base measure or smoothing parameter, matching statistical learning's flexibility with online learning's robustness.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

Geometric weighting by update norm offers a subtle but measurable improvement in personalized federated learning, outperforming uniform aggregation and FedPer-R1 on CIFAR-10's native client accuracy.

Average-reward RL for general multichain MDPs is now possible without model knowledge or reduction to discounted problems.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Fresh, shallow network activations can correct diffusion model feature forecasts, unlocking significant speedups and improved fidelity without any parameter updates.

Probabilistic forecasting models can now achieve 65% CRPS reduction with drastically fewer samples by directly learning the distribution shift between successive predictions.

Pretraining neural CFD surrogates across diverse domains unlocks substantial gains in zero- and few-shot generalization, drastically reducing data requirements for new applications.

Recurrent non-stationarity unlocks data reuse, but its "safe" application hinges on a bias-variance tradeoff akin to classical settings, with a novel algorithm achieving near-optimal regret.

A scalar adjoint method slashes the computational cost of fine-tuning flow policies in RL, achieving state-of-the-art results on complex manipulation tasks.

Embedding compression can be dramatically improved by learning the optimal coding geometry from data, not just optimizing within a fixed one.

Derivative GPs can now achieve state-of-the-art accuracy with less computation and memory than function-only baselines by cleverly budgeting gradient information.

A single objective directly optimizes speculative decoding efficiency, eliminating auxiliary hyperparameters and achieving state-of-the-art performance across multiple tasks.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Physics-aligned electronic ground-state models shatter out-of-distribution generalization barriers for MLIPs, achieving near-chemical accuracy on reactive systems with minimal data.

Differentially private synthetic time-series forecasting utility hinges on forecaster-specific window assembly, not just generator quality.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Generative recommender training strategies that append missed targets can actively harm ranking performance by introducing detrimental probability competition.

Mamba's linear-time state-space models unlock superior multi-scale financial volatility forecasting, outperforming Transformers by capturing regime shifts and enabling longer context.

Gaussian regularizers like SIGReg can certify InfoNCE's optimal solutions but diverge significantly in their behavior away from the optimum, especially regarding nuisance channels and style retention.

Reweighted normalizing flows enable exponential convergence to arbitrary Boltzmann distributions via a novel WFR JKO scheme, bypassing common structural assumptions.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

LLMs develop inherent fault tolerance as they scale, learning to compute within error-correcting codes that remain efficient even for massive models.

A frozen graph backbone effectively preserves past structural knowledge in continual multi-agent reinforcement learning, dramatically reducing forgetting and boosting transfer across tasks with distinct graph structures.

Quadrupedal robots can achieve up to 20% lower energy consumption by learning speed-dependent gaits with a hierarchical RL policy that explicitly optimizes cost of transport.

Negative curvature can amplify PWGF approximation errors, but subsequent positive curvature can recover accuracy over growing time horizons, enabling stable optimization in non-convex landscapes.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

PPO-Clip's stationarity and critic tracking are jointly bounded by a common amplification factor dependent on algorithmic parameters, revealing a unified error mechanism.

A novel bias expansion for two-timescale stochastic approximation reveals a $\varepsilon^2/\eta$ term that fundamentally alters extrapolation strategies.

Inertial activity recognition models can now learn rich, global motion structure from unlabeled data by discovering and grouping motion primitives, achieving state-of-the-art performance with orders of magnitude less data.

Axis-aligned splits in BART are a bottleneck for non-axis-aligned functions; RoBART achieves adaptive posterior contraction by rotating each tree's coordinate system.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Bayesian inference for complex switching stochastic differential equations is now broadly tractable, even with non-analytic transition densities.

A single information-theoretic framework elegantly unifies multi-objective, multi-fidelity, and constrained Bayesian optimization, bypassing separate heuristics for each.

The asymptotic behavior of RBF kernel MLEs is finally understood, revealing minimax optimal convergence rates for all key parameters.

Diffusion models can achieve arbitrarily low error amplification at low noise levels, decoupling sampler performance from worst-case Lipschitz bounds.

Generative MARL teachers fail by interpolating coordination modes; a novel optimal transport approach forces them to commit to distinct strategies, boosting student performance.

A simple, training-free calibration wrapper boosts tensor-network LLM compression by 2-6x, closing up to 60% of the perplexity gap without altering the underlying decomposition solver.

Optimizing BO hyperparameters with BO itself reveals configurations that significantly reduce cumulative regret on GP-inferred sample paths.

Provably private tabular classification is now achievable with a single forward pass, outperforming baselines and drastically cutting fitting time.

Kinetic Langevin diffusions slash posterior sampling time for imaging inverse problems by 10x while maintaining reconstruction quality, even with complex diffusion model priors.

Intermediate retrieval signals, not just final outcomes, unlock efficient credit assignment for training LLM search agents.

Adversarial bandits with $m$-set choices can suffer drastically slower learning rates than previously thought, with regret scaling quadratically with $m$ in some settings.

Physical robot learning can now autonomously discover and exploit informative experiments to continuously improve foundation models, achieving 77% success on RoboCasa365 and zero-shot transfer to real hardware.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

Trillion-parameter agentic RL refits are now 12-40x faster, shrinking an 87.5-minute full checkpoint transfer to 150 seconds by sending only bit-exact weight deltas.

Jointly training path and flow networks with a novel alignment objective unlocks improved generative model performance by preventing path overfitting and low-entropy bottlenecks.

Humanoid locomotion learned from scratch and transferred to hardware in 10x less time, thanks to a novel off-policy flow RL approach.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Robust bandit learning is tractable in a specific setting, but even minor deviations lead to NP-hardness, pinpointing a critical boundary for efficient unrealizable learning.

Spurious forgetting in LLMs is a reversible access loss, not permanent knowledge erasure, and can be mitigated by decoupling shared representational shifts from fact-specific updates.

Generalization-heavy models are demonstrably more fragile during unlearning, suffering significantly higher performance degradation on retained data.

Sequential recommenders can dramatically boost catalog diversity by adaptively pruning action sets, achieving up to 5.21x diversity gains without sacrificing session depth.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

A single interpolation coefficient unlocks variance-optimal off-policy evaluation by seamlessly blending the stability of cluster-level weights with the unbiasedness of action-level importance sampling.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Parallel LLM agents can finally achieve near-linear speedups by strategically parallelizing only when token-based cost estimates predict a net gain, eliminating redundant work and reconciliation overhead.

A cubic memory rule for Adam, derived from a local model of gradient dynamics, slashes validation gaps by over 40% on diverse vision and language benchmarks.

Sequential hypothesis testing without explicit densities is now possible with anytime-valid error control and near-optimal performance.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Exploiting cyclic symmetries in multi-index models offers a polynomial sample complexity advantage, with data augmentation proving as sample-efficient as architectural weight sharing.

Privacy-preserving federated learning achieves segmentation performance on skin lesions comparable to centralized methods, without centralizing sensitive patient data.

Divergence choice in Shampoo-like optimizers fundamentally alters Kronecker approximation and finite-sample error compensation, explaining observed training behavior.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Quantum circuits can unlock compositional generalization in multimodal AI, even on noisy hardware.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

Occupancy measure formulation enables decision-focused learning in MDPs with closed-form gradients, overcoming KKT-based scalability bottlenecks and achieving state-of-the-art regret.

Counterfactual explanations can now achieve near-perfect numerical sparsity and proximity by treating feature changes as a sparse transport problem solved with flow matching.

Meta-learning unlocks efficient, resolution-independent PDE surrogate modeling for time-dependent systems by treating initial conditions as context, structuring latent spaces, and accelerating inference.

Encoding sensor geometry as a flow-matching prior dramatically improves generative models for brain signals, outperforming isotropic noise and even empirical data covariance.

Gradient-free few-shot adaptation emerges from decentralized cellular dynamics, achieving state-of-the-art OOD transfer by updating a spatial program through local neighborhood interactions.

AI rapidly identified differentiated peptide depot formulations with tunable 30-day release profiles by navigating a complex multidimensional design space.

Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.

Generate valid and unique discrete designs in a single forward pass with over $50\times$ the yield per operation compared to prior one-step methods.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

Segment reward feedback can drastically reduce RL regret under binary feedback by increasing segment granularity, a benefit surprisingly absent with sum feedback.