Training Efficiency & Optimization - Weekly Roundup

Asynchronous learning communities don't just fragment; they predictably fragment *before* converging around deadlines, a pattern driven by participation shifts, not just behavioral alignment.

GPU kernel optimization agents can regress by up to 19% when transferring distilled knowledge across DSLs, revealing that source attempt quality, not just success, dictates transfer efficacy.

Folding multi-branch redundancy slashes speculative decoding costs in diffusion LLMs, unlocking up to 2x faster generation.

Key contribution not extracted.

Achieve state-of-the-art LLM-guided program evolution for under $2, drastically cutting costs for complex optimization tasks.

GPU kernel optimization agents can regress by up to 19% when transferring distilled knowledge across DSLs, revealing that source attempt quality, not just success, dictates transfer efficacy.

Folding multi-branch redundancy slashes speculative decoding costs in diffusion LLMs, unlocking up to 2x faster generation.

Key contribution not extracted.

Achieve state-of-the-art LLM-guided program evolution for under $2, drastically cutting costs for complex optimization tasks.

LLMs can recursively rewrite their own solutions from specialized environments into generalizable training data, boosting performance on complex tasks by over 34% and improving pass@3 by up to 17 percentage points.

A few critical denoising steps in diffusion language models dictate the entire output, and training only these "pivots" dramatically boosts performance.

Adapters forget as much as they learn: a simple post-training normalization unlocks significant gains in both specialization and retained capabilities for LoRA.

A single tokenizer can now slash token counts for non-English scripts by 20-40% without increasing English tokenization cost.

The non-smooth pointwise behavior of linear optimization solvers smooths to a quadratic form under data distribution averaging, enabling closed-form curvature derivation.

Imperfect diffusion models can now achieve exact target distribution sampling at inference time by correcting inherent path and endpoint errors.

Asynchronous LLM training can be made robust to stale rollouts by minimizing a novel group-mass capping bias bound, outperforming existing methods on reasoning tasks.

Optimal $\mathcal{O}(\varepsilon^{-4-2/p})$ convergence for stochastic bilevel optimization is achieved with a single-loop first-order method, matching a new lower bound.

Adversarial distillation achieves state-of-the-art few-step visual generation by learning log-density ratios directly via classification, eliminating the need for auxiliary score fitting.

LLM workflows can be optimized without human labels by treating agent assignment and task decomposition as a bidding process, slashing optimization costs.

State costs in Schrödinger bridges on graphs can be analytically incorporated via a Feynman-Kac tilt, enabling exact, non-learned solutions with linear memory growth even for massive networks.

Teacher averaging in MOPD, not just raw gradients, dictates student learning, with Adam's momentum and BF16 precision masking significant parameter shifts.

Humanoid robots can learn entirely new complex manipulation skills at test-time by evolving reward programs, unlocking latent capabilities of pre-trained controllers.

Post-training LLMs for agentic tasks can drastically reduce solution coverage, a "Sharpening Tax" that a novel Bayesian sampler can mitigate.

Topology-preserving 3D generation now achieves 98% fewer tokens and 58.5% faster inference by encoding shapes as overlapping slice latents instead of expensive voxels.

Non-convex sampling from matrix spaces is now possible with provable exponential convergence using momentum preconditioning that spectrally tames gradients.

Multimodal sampling with Wasserstein gradient flows and forward diffusion is fundamentally limited by slow mixing, requiring exponential time to bridge well-separated modes.

A novel meta-network framework for data selection achieves unprecedented transferability across diverse LLM training tasks by overcoming the limitations of existing MTS objectives.

A single controller can now robustly learn stochastic transport across a range of initial distribution shifts, outperforming standard SB and noise augmentation on challenging tasks.

A novel hybrid deep learning approach achieves robust few-shot malware detection by leveraging autoencoder-derived features for rapid meta-learning adaptation.

A single token can bypass most safety filters.

Relational learning with tabular foundation models can now bypass feature explosion and interaction blindness by propagating information directly through the schema graph.

Adversarial training on Fourier phase perturbations unlocks over 20% domain generalization gains in medical imaging by decoupling spatial structure from domain-specific textures.

Discrete neural samplers achieve fixed-point convergence and drastically reduce mode collapse by leveraging Gibbs sampling within a masked diffusion framework.

Model-free Q-learning now guarantees optimal reachability policies in a broad class of MDPs, slashing sample complexity and memory needs.

Adaptive model search can now achieve statistically controlled adoption without sacrificing evidence efficiency, even with path-dependent challenger generation.

Kernel learners outperform feature learners for nonlinear in-context learning when task diversity is high and context length is short, but the opposite holds for low diversity and long contexts.

Per-user normalization of wrist EDA data can inflate affect recognition performance by up to 4x, masking the true benefit of pretraining and making published results incomparable.

LLMs can learn effective dynamic embeddings for massive combinatorial optimization problems with a shallow policy by selectively propagating memory across rollout steps, bypassing the need for expensive full recomputation or external labels.

Two fixed attention blocks suffice for universal interpolation in deep residual networks, even with causal masking.

Adaptive tensor product computation slashes MLIP runtime by $2.4\times$ and memory by $2.6\times$ without sacrificing accuracy, revealing computational depth's link to chemical complexity.

Harnessing earlier recurrent states as implicit weak supervision, a training-free contrastive decoding method boosts Looped Transformer performance by up to 11.45% and enables 22.5%-48.2% FLOP reduction.

Transformer and RNN tractography models achieve state-of-the-art performance by introducing a novel generation-validation phase for streamline-level supervision, overcoming the limitations of local loss functions.

LLM training CI is now 77.5% faster and 63.9% more GPU-efficient by intelligently pruning and prioritizing tests based on runtime evidence.

A novel policy optimization method bypasses critic action gradients entirely, achieving faster, more robust updates and improved exploration in continuous control.

Multimodal RL models trained without visual input can still achieve high accuracy on visual reasoning tasks when images are present at inference, but prolonged visual training can paradoxically erode grounding.

Free-boundary PDEs are solved with unprecedented accuracy and interface resolution using Kolmogorov-Arnold Networks, surpassing PINNs and residual networks.

A single 10M-parameter Transformer policy achieves near-perfect win rates in competitive Super Smash Bros. Melee, outperforming specialized AIs and even a zero-delay opponent.

Intrinsic motivation can be learned from system dynamics alone, unlocking unsupervised discovery of essential robotic behaviors and complex locomotion.

Sampling weight updates, not just tokens, unlocks significant test-time adaptation gains for LLMs.

Diffusion models can lose diversity when only later timesteps are updated during reward optimization, a finding that challenges prior assumptions and motivates a new training paradigm.

Jointly training on simulation and experimental data unlocks balanced prediction in physical systems, outperforming sequential transfer and preserving crucial simulation-only knowledge.

A data-driven weather model now rivals operational NWP for 1 km resolution forecasts over complex Alpine terrain, but struggles with localized wind extremes.

Iterative refinement of a class token with shared blocks slashes transformer parameters and computation while boosting audio recognition accuracy.

Stochastic rounding excels in Transformer MLPs, while round-to-nearest is better at the output head, enabling a 28% perplexity reduction via mixed-precision emulation.

A single deep generative model now captures the full joint distribution of mixed-type multivariate outcomes, outperforming specialized methods.

Asynchronous learning communities don't just fragment; they predictably fragment *before* converging around deadlines, a pattern driven by participation shifts, not just behavioral alignment.

Diffusion models can now achieve one-step, efficient action generation for online multi-agent coordination, shattering previous scalability barriers.

A learned linear skip connection eradicates spurious local minima in shallow ReLU networks, even when overparameterized, by confining student features to the teacher's subspace.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

Adaptive LLM fine-tuning can achieve superior performance and stability by intelligently searching step-sizes along a trusted gradient direction, costing less than a full line search.

Fine-tuning 30B+ LLMs on a single 80GB GPU is now possible with TACO, slashing optimizer state memory by $174\times$ without sacrificing accuracy.

LLM RL policy updates can be significantly misrepresented by standard masking, leading to substantial unseen drift; CARM's absolute-value approach reveals and corrects this.

Quasi-Newton methods now conquer deep learning's non-convexity and scale, outperforming Adam and others on challenging tasks like 136M-parameter physics-informed models.

Synthetic scenes with automatically grounded object identities unlock substantial real-world perceptual gains in MLLMs via on-policy self-distillation.

A 0.9B parameter omni-modal embedder achieves state-of-the-art text retrieval quality by distilling a frozen text backbone's embeddings into new modalities, bypassing the common modality-extension performance drop.

A novel gating mechanism allows LLMs to learn new tasks without forgetting old ones, even without access to prior training data.

Looped MoE models can scale to over 10 iterations, achieving substantial performance gains by stabilizing recurrence and diversifying expert selection.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

StreamMAE unlocks self-supervised learning on continuous video streams, matching i.i.d. MAE performance by tackling intra-batch similarity with novel regularization.

LLMs can learn to generate syntactically valid text without sacrificing semantic quality, even with underspecified prompts, by leveraging self-supervised rewards derived from their own likelihoods.

Task-specific LoRA-Experts and prototype-ensemble matching unlock rehearsal-free class-incremental learning with reduced interference and improved discrimination.

Recurrent memories can retain full Fisher information post-writing with an invertible hold, decoupling write and read optimality and achieving near-perfect task-specific performance.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

Moreau--Yosida unadjusted Langevin sampling achieves $\widetilde O(\varepsilon^{-1})$ iterations for $\varepsilon$-accurate convergence to target distributions, bypassing the need for third derivatives.

An RL agent learns to optimize SpTRSV parallelism by intelligently rewriting sparse matrix dependency graphs, achieving substantial performance gains with minimal structural changes.

Howard's policy iteration for deterministic MDPs is not strongly polynomial, exhibiting an exponential iteration complexity that separates it from the simplex method.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

Inductive Gromov-Wasserstein embeddings are now achievable, enabling out-of-sample generalization without recomputing costly optimal transport plans.

Most failed multi-turn agent trajectories contain a single, recoverable pivotal mistake that, if addressed, can restore task success, a phenomenon PivotOPD exploits for significant performance gains.

Multimodal models can achieve significant image generation improvements by learning from their own critiques during test-time compute, effectively acting as their own recursive self-improvement mechanism.

Faster computer-use agents are bottlenecked by their speed, but existing benchmarks are irreproducible; this new standardized framework reveals counterintuitive speed-efficiency trade-offs and that current open-weight models lag behind.

Physics-constrained generation now achieves comparable fidelity to multi-step methods in a single neural transport evaluation.

A single auxiliary variable per group of failures unlocks near-optimal $T^{-4/5}$ convergence for large-scale Cox regression, outperforming stochastic baselines.

An optimal threshold policy for irrevocable matching in growing trees loses at most twice the total-variation error of the true growth law, even when forecasts are misspecified.

Clinical MLLMs can be significantly improved by optimizing for AUROC via pairwise instance rankings, reversing accuracy-based prompt evolution's performance degradation.

Standard image ViTs, when adapted with masked Siamese learning on "super images," unlock state-of-the-art video representation learning with orders of magnitude less pretraining.

A novel decoupled autoencoder design unlocks 32x spatial compression for diffusion models while simultaneously boosting reconstruction fidelity and accelerating training convergence.

Untied embeddings, not weight tying, unlock superior performance and memory efficiency for decoder-only LLMs under DP-SGD.

Diffusion language models can now generate high-quality text with up to 49% fewer network evaluations by matching output distributions.

Kilometer-scale extreme heat downscaling accuracy scales linearly with climatological distance, not simulation volume, revealing a path to cost-effective urban adaptation.

Decentralized power allocation in complex MANETs is solved with a learned GNN that matches centralized optimization performance using only local, noisy channel data.

Explicitly constraining class geometry with differentiable prototypes significantly boosts tensor neural network performance under extreme label scarcity, particularly through training-time regularization.

Separating policy roles in offline RL unlocks significant performance gains by tailoring updates for value bootstrapping versus direct action execution.

Standard time-series forecasting losses systematically fail on pathological demand regimes, creating a bias irreducible by model capacity alone.

A low-rank residual connection allows frozen LLMs to learn from feedback with minimal parameters, achieving substantial error reduction in downstream tasks.

LLM pre-training power laws are not fixed but dynamically shaped by learning-rate and batch-size schedules, with a novel forcing-memory surrogate accurately predicting loss across these regimes.

End-to-end learned subspace projections shatter prior compression limits, achieving 70% weight reduction in Llama-2-7B with near-dense performance and drastically shrinking KV cache memory.

Zero-shot voice conversion now achieves $9\times$ faster inference with improved perceptual quality by jointly optimizing flow and content encoding.

Exact incremental BPE achieves $O(\log t)$ amortized append time by weighting centroid search intervals, a significant speedup over prior $O(\log^2 t)$ methods.

Neural operators can now learn directly from noisy, low-cost Monte Carlo estimates, achieving $10^4$-$10^5\times$ speedups for particle transport problems without sacrificing accuracy.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

Tree search on on-policy trajectories dramatically boosts policy gradient learning by uncovering rare high-return paths, outperforming PPO on Atari and MuJoCo with minimal bias increase.

Source-validation accuracy is a poor proxy for out-of-distribution reliability; re-ranking checkpoints by aggregated source reliability metrics significantly boosts target probability quality without needing target data.

Human password generation exhibits persistent linguistic regularities that can be exploited for a 16% boost in password recovery by adapting pre-trained language models.