Training Efficiency & Optimization - Weekly Roundup

Trillion-parameter agentic RL refits are now 12-40x faster, shrinking an 87.5-minute full checkpoint transfer to 150 seconds by sending only bit-exact weight deltas.

Jointly training path and flow networks with a novel alignment objective unlocks improved generative model performance by preventing path overfitting and low-entropy bottlenecks.

Humanoid locomotion learned from scratch and transferred to hardware in 10x less time, thanks to a novel off-policy flow RL approach.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Robust bandit learning is tractable in a specific setting, but even minor deviations lead to NP-hardness, pinpointing a critical boundary for efficient unrealizable learning.

Trillion-parameter agentic RL refits are now 12-40x faster, shrinking an 87.5-minute full checkpoint transfer to 150 seconds by sending only bit-exact weight deltas.

Jointly training path and flow networks with a novel alignment objective unlocks improved generative model performance by preventing path overfitting and low-entropy bottlenecks.

Humanoid locomotion learned from scratch and transferred to hardware in 10x less time, thanks to a novel off-policy flow RL approach.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Robust bandit learning is tractable in a specific setting, but even minor deviations lead to NP-hardness, pinpointing a critical boundary for efficient unrealizable learning.

Spurious forgetting in LLMs is a reversible access loss, not permanent knowledge erasure, and can be mitigated by decoupling shared representational shifts from fact-specific updates.

Generalization-heavy models are demonstrably more fragile during unlearning, suffering significantly higher performance degradation on retained data.

Sequential recommenders can dramatically boost catalog diversity by adaptively pruning action sets, achieving up to 5.21x diversity gains without sacrificing session depth.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

A single interpolation coefficient unlocks variance-optimal off-policy evaluation by seamlessly blending the stability of cluster-level weights with the unbiasedness of action-level importance sampling.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Parallel LLM agents can finally achieve near-linear speedups by strategically parallelizing only when token-based cost estimates predict a net gain, eliminating redundant work and reconciliation overhead.

A cubic memory rule for Adam, derived from a local model of gradient dynamics, slashes validation gaps by over 40% on diverse vision and language benchmarks.

Sequential hypothesis testing without explicit densities is now possible with anytime-valid error control and near-optimal performance.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Exploiting cyclic symmetries in multi-index models offers a polynomial sample complexity advantage, with data augmentation proving as sample-efficient as architectural weight sharing.

Privacy-preserving federated learning achieves segmentation performance on skin lesions comparable to centralized methods, without centralizing sensitive patient data.

Divergence choice in Shampoo-like optimizers fundamentally alters Kronecker approximation and finite-sample error compensation, explaining observed training behavior.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Quantum circuits can unlock compositional generalization in multimodal AI, even on noisy hardware.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

Occupancy measure formulation enables decision-focused learning in MDPs with closed-form gradients, overcoming KKT-based scalability bottlenecks and achieving state-of-the-art regret.

Counterfactual explanations can now achieve near-perfect numerical sparsity and proximity by treating feature changes as a sparse transport problem solved with flow matching.

Meta-learning unlocks efficient, resolution-independent PDE surrogate modeling for time-dependent systems by treating initial conditions as context, structuring latent spaces, and accelerating inference.

Encoding sensor geometry as a flow-matching prior dramatically improves generative models for brain signals, outperforming isotropic noise and even empirical data covariance.

Gradient-free few-shot adaptation emerges from decentralized cellular dynamics, achieving state-of-the-art OOD transfer by updating a spatial program through local neighborhood interactions.

AI rapidly identified differentiated peptide depot formulations with tunable 30-day release profiles by navigating a complex multidimensional design space.

Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.

Generate valid and unique discrete designs in a single forward pass with over $50\times$ the yield per operation compared to prior one-step methods.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

Segment reward feedback can drastically reduce RL regret under binary feedback by increasing segment granularity, a benefit surprisingly absent with sum feedback.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Collaborative edge-cloud LLM inference can achieve 275% higher throughput by dynamically synchronizing heterogeneous requests to batch their common cloud-side suffix.

LLM pruning can be task-aware: DIPrune's dual importance scoring jointly optimizes static feature saliency and dynamic semantic evolution, outperforming prior methods on LLaVA and Qwen-VL.

Retraining on observed data in performative prediction can lead to unstable models, but a worst-case objective with label confidence intervals ensures convergence.

Decentralized learning topologies can be optimized using only local neighborhood and friend-of-a-friend information, achieving competitive performance without global graph awareness.

Training-free LQEC recovers 84% of the FP16 accuracy gap at 2-bit quantization by decoupling calibration and compensation, outperforming baselines by over 20%.

Existing autoregressive post-training weights, often discarded after diffusion model conversion, can be directly applied to significantly boost dLLM performance, even outperforming dedicated diffusion post-training when combined.

End-to-end meta-learning of neural field representations reveals that first-order approximations discard crucial learning pathways, leading to improved few-shot adaptation and semantic prediction.

LLM agents struggle to translate general capabilities into cost-efficient, reusable solutions, often failing to outperform simple distillation baselines despite significant zero-shot prowess.

A single RL policy masters energy-efficient path following across diverse tracks and even complex vehicle dynamics, outperforming NMPC and baselines without explicit track-specific training.

Agent experience, even when uniformly unrewarded, can be distilled into predictive foresight to dramatically improve learning in sparse reward settings.

Expert counseling quality assessment transfers surprisingly well across domains, with LLM-extracted instrument-based constructs outperforming in-domain training and even beating generic dialogue features.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

Frozen medical AI models can now continuously learn and improve from real-world deployment experience without requiring weight updates or risking catastrophic forgetting.

MLLMs can now natively weave elastic visual representations, adaptively allocating detail to save 43% of tokens while preserving near-perfect performance.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Software engineering agents learn significantly more from failed attempts by conditioning recovery trajectories on explicit verifier feedback, boosting resolution rates by up to 10% on SWE-bench.

Data-free diffusion distillation now achieves FID 2.04 at 1-NFE by unifying generation and score estimation in a single student network.

External context is often unnecessary for model-based RL agents that implicitly learn to recover latent environment dynamics from their own experience.

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A novel initialization technique for diffusion samplers bypasses initial steps by directly sampling a softened clean-space posterior, achieving order-of-magnitude gains on rare-mode inverse problems.

Latent reasoning models can achieve 97% of explicit CoT accuracy at 25% of the latency by shaping a continuous latent space with flow matching and self-verification.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Aligning DLM generation with reference distributions via feature-space MMD significantly boosts performance on perplexity and reasoning tasks without retraining the entire model.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

A single-voice TTS model with 10x fewer parameters and 15x less compute than its teacher achieves near-identical audio quality, runnable on a single laptop.

Key contribution not extracted.

Private prediction can achieve logarithmic dependence on the time horizon, while private learning requires polynomial dependence, revealing a fundamental gap in their sample complexity for differentially private settings.

Neural PDE solvers can miss optimal solutions due to unbounded parameters, but adding kernel derivatives recovers these "missing limits."

Hierarchical MoE routing with a novel arrival-weighted mean estimation prevents expert collapse and enables localized co-activation, outperforming flat routing on large embedding models.

Prior-data fitted networks unlock state-of-the-art tabular anomaly detection without requiring anomalous samples during training.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

Sparse MoE routers don't achieve adaptive expert selection through their maps alone; they co-evolve learned score distributions that critically determine token-wise expert participation.

Large HEP datasets are not just for training; they are foundational for unlocking power-law scaling, with smaller datasets offering little insight into high-compute performance.

Even with only uniform state preparation and measurements, or computational basis access, we can fully reconstruct generic 2-local and many common nearest-neighbor Hamiltonians.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Logistic regression GD with a large stepsize achieves polylogarithmic convergence to zero loss on linearly separable data, bypassing the sublinear rates of prior theory.

Confident conflicts in LLM distillation are mitigated by dynamically mixing student and teacher distributions based on student entropy, boosting reasoning and preserving general capabilities.

Decoupling update magnitude from gradient norm in SGD initially accelerates learning but leads to marginal stability and distinct scaling regimes determined by model width and batch size.

Tabular ICL models can achieve near full-context performance with drastically reduced inference costs by aligning intermediate activations, even with minimal training data and no GPU.

Binary sources allow for strictly smaller optimal representations in the Information Bottleneck than previously thought.

A single token can bypass most safety filters, revealing a critical vulnerability in current LLM alignment techniques.

Hyper-connections create a fading memory in Transformers, with the effective context horizon directly tied to the second singular value of the stream-mixing matrices.

Single-neuron models can now capture multimodal voltage responses, including coexisting firing/non-firing states and depolarization block modes, a feat previously requiring massive ensembles or failing with neural operators.

Nighttime anomaly detection is significantly enhanced by fusing M-band and DNB satellite signals, achieving higher accuracy and broader applicability than thermal-only methods.

Amortized spike-and-slab inference for linear models with interactions achieves state-of-the-art variable selection accuracy with a single forward pass, eliminating costly per-dataset MCMC.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Frequency-specific conditional relationships in multivariate time series are unlocked by using inverse cross-spectral density matrices as graph shift operators.

The optimal LoRA rank is not intrinsic to the model but dynamically dictated by the data's geometric properties, enabling principled data selection for efficient fine-tuning.

Autoregressive latent priors learned directly from data, not fixed Gaussians, unlock VAEs for high-fidelity sequential generation with diffusion-level quality and faster inference.

Recover a full Pareto stationary front of multi-objective solutions with the computational cost of standard single-objective training.

Equivariant potentials can achieve transformer-level accuracy with 35x fewer parameters and 10x faster inference by leveraging Cartesian tensor products.

Physics-based secure key distribution can now sustainably support federated learning for medical imaging, reducing key material needs by 35x without sacrificing accuracy.

The pursuit of LLM performance is shifting from raw scale to strategic resource allocation, revealing a critical need to re-evaluate scaling laws for parameter-efficient, compute-optimal training in constrained environments.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Deep GNNs can achieve superior depth scaling and accuracy by leveraging controlled Turing instability or stable near-critical propagation to combat over-smoothing.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

Medical LLMs can achieve superior QA performance by dynamically selecting specialized adaptation components based on question context and clinical tags, surpassing static parameter-efficient methods.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Deterministic midpoint methods for Langevin sampling can achieve superior sample complexity in high dimensions compared to their randomized counterparts.

A dual VAE architecture effectively bridges the sim-to-real gap for robotic navigation, enabling high performance with minimal real-world data.

Wi-Fi sensing accuracy jumps to 92.83% by decomposing CSI into trajectory-guided tokens that capture complex temporal dynamics.

Closed-form KANs integrated into IGA dramatically slash PDE solution errors by orders of magnitude, outperforming pure neural solvers and standard IGA.

Self-supervised learning on mass spectrometry data unlocks state-of-the-art de novo peptide sequencing, outperforming fully supervised methods without any sequence labels.