Training Efficiency & Optimization - Weekly Roundup

Achieve real-time online learning for model predictive control with a novel spatio-temporal Gaussian Process approximation that maintains constant computational complexity.

AdaMuS overcomes the bias towards high-dimensional data in multi-view learning by adaptively pruning redundant parameters and sparsely fusing views, leading to improved performance on dimensionally unbalanced data.

LLMs can be actively trained to master specific knowledge domains with 50% less data and computation by focusing on what they *don't* know, not what they already do.

Even with a 98:1 test-to-train ratio, PEFT methods like QLoRA can unlock surprisingly strong generalization from billion-parameter vision models for agricultural image classification, suggesting underfitting is the bigger risk than overfitting.

Pre-trained models unlock surprisingly aggressive quantization in federated learning, slashing communication costs by 40% without sacrificing accuracy on MNIST and CIFAR-100.

Achieve real-time online learning for model predictive control with a novel spatio-temporal Gaussian Process approximation that maintains constant computational complexity.

AdaMuS overcomes the bias towards high-dimensional data in multi-view learning by adaptively pruning redundant parameters and sparsely fusing views, leading to improved performance on dimensionally unbalanced data.

LLMs can be actively trained to master specific knowledge domains with 50% less data and computation by focusing on what they *don't* know, not what they already do.

Even with a 98:1 test-to-train ratio, PEFT methods like QLoRA can unlock surprisingly strong generalization from billion-parameter vision models for agricultural image classification, suggesting underfitting is the bigger risk than overfitting.

Pre-trained models unlock surprisingly aggressive quantization in federated learning, slashing communication costs by 40% without sacrificing accuracy on MNIST and CIFAR-100.

Achieve better compression in low-bit quantization by considering not just numerical sensitivity, but also the structural role of each layer.

Quantum computers could finally unlock the full potential of machine learning for drug discovery by directly generating the quantum chemistry data that classical computers struggle to produce.

Federated recommendation systems can now better adapt to evolving user preferences without sacrificing privacy, thanks to a novel approach that retains historical knowledge and transfers insights between similar users.

LLMs can achieve state-of-the-art reasoning accuracy with significantly fewer tokens by rewarding intermediate reasoning steps that maximize information gain and maintain monotonic progress.

LLMs can predict multiple tokens in parallel without any training, simply by cleverly probing their embedding space with dynamically generated mask tokens.

Pruning vision tokens across both the ViT and LLM can yield a 62% efficiency boost in video VLMs with minimal performance loss, and without complex text conditioning.

Class reweighting and anatomy-guided decoding can substantially improve the performance of video analysis pipelines for rare events in imbalanced gastrointestinal datasets.

Stop struggling with the stability-plasticity dilemma in multilingual Speech-LLMs: Zipper-LoRA dynamically disentangles LoRA updates to boost low-resource ASR without sacrificing cross-lingual transfer.

Attention sinks aren't just a forward-pass phenomenon; they actively warp the training landscape by creating "gradient sinks" that drive massive activations.

Achieve near-optimal waveform optimization with 98.8% spectral efficiency using a 5-layer, AutoML-tuned unrolled proximal gradient descent network trained on just 100 samples.

Forget training separate models for each compression level; this framework achieves state-of-the-art extreme image compression with flexible bitrate control using a single diffusion-based arbitrary-scale super-resolution model.

Virtual cell perturbation prediction gets a 12x speedup in pretraining and a 12% boost in biological fidelity with SCALE, a new foundation model that prioritizes scalable infrastructure and biologically faithful evaluation.

LLMs can slash over 80% of their chain-of-thought tokens with a minor accuracy boost, thanks to a new RL-based method that targets the "Minimal Sufficient Length" of reasoning.

RL agents can learn far more efficiently by dynamically distilling and leveraging past experiences that co-evolve with the agent's growing capabilities.

NNVMC's promise for solving quantum many-body problems is currently bottlenecked by surprisingly mundane issues: low-intensity elementwise operations and data movement on GPUs.

Counterintuitively, the most *unreliable* samples in medical imaging datasets—those with fluctuating confidence and frequent forgetting during training—are the *most* informative for building accurate decision boundaries.

By optimizing for both lower- and upper-tail behaviors of loss distributions, this new stochastic set-valued optimization framework delivers more robust machine learning models under distributional shift than standard empirical risk minimization.

Training video diffusion models with pixel-wise losses just got a whole lot cheaper: ChopGrad reduces memory complexity from linear to constant with video length.

Forget painstakingly tuning quantization for each LLM – RAMP learns a quantization policy that generalizes across architectures, often outperforming target-specific training.

Drifting offers a surprisingly effective way to distill iterative Boltzmann sampling into a single forward pass, even with unknown normalization constants.

Convolutional Neural Operators (CNOs) surprisingly excel at capturing translated dynamics in the FitzHugh-Nagumo model, despite other architectures achieving lower training error or faster inference.

Quantum annealing could soon accelerate protein engineering: Q-BIOLAT formulates protein fitness as a QUBO problem, directly compatible with emerging quantum annealing hardware.

Infinite neural nets can be sparse, and this paper proves it, showing that total variation regularization provably yields sparse solutions in infinite-width shallow ReLU networks, with sparsity bounds tied to the geometry of the data.

Ditch the overconfident posteriors: Structured SIR offers a memory-efficient way to capture complex, multi-modal uncertainty in high-dimensional image registration, outperforming variational inference.

k-NN regression, a classic non-parametric method, can now be rigorously applied to complex survey data, expanding its applicability to a wider range of real-world statistical problems.

Ditch the feature engineering: Baguan-TS lets you use raw time series sequences directly for in-context forecasting, outperforming traditional methods.

Ditch quadratic attention bottlenecks: this new transformer variant achieves competitive time-series forecasting with O(N log N) complexity by representing sequence states on a unit circle.

By cleverly using readily available video segmentation masks, this method boosts DINOv2's point tracking performance by over 14% – a surprisingly effective way to inject temporal awareness into static image-pretrained models.

Ditch the temperature ladder: Generative Replica Exchange (GREX) uses normalizing flows to generate high-temperature configurations on-demand, slashing the computational cost of replica exchange simulations.

Instead of forcing modalities to imitate each other, IIBalance lets each modality contribute according to its intrinsic information budget, leading to better multimodal fusion.

Ditch fixed compute budgets: this new flow-matching method for robotic control adaptively allocates computation, speeding up simple tasks and focusing on complex ones.

Forget training behemoths: ADMs slash memory overhead to just twice the inference footprint while guaranteeing geometric correctness and continuous adaptation.

Achieve significant latency and energy savings in memory systems with an RL-based controller that also provides insights into *why* its decisions are optimal.

KANs get a 50x BitOps reduction without accuracy loss by quantizing their B-splines down to 2-3 bits and using lookup tables.

By explicitly modeling and predicting non-stationary factors in both time and frequency domains, TimeAPN significantly boosts the accuracy of long-term time series forecasting, outperforming existing normalization techniques.

Achieve SOTA LLM alignment in complex technical domains with a fraction of the compute by distilling knowledge into smaller models using a hybrid reward mechanism and targeted data augmentation.

Stop benchmarking algorithm discovery on the same old saturated datasets: DiscoGen offers millions of fresh, configurable tasks to truly test your ADA.

By federating distributional critics and using a Wasserstein barycenter trust region, TR-FedDistRL avoids the dangerous "mean-smearing" that can make federated RL unsafe in critical applications.

Forget SVD: CARE aligns low-rank attention approximations with input activations, boosting accuracy up to 1.7x and slashing perplexity by 215x when converting models to multi-head latent attention.

Exploiting geometric symmetries in tensegrity structures slashes computational cost and boosts accuracy in physics-informed neural networks.

Ditch slow diffusion policies: FMER achieves 7x faster training and superior performance in sparse reward RL by using flow matching and a tractable entropy regularization term.

Forget separate defenses: rSDNet unifies robustness against both label noise and adversarial attacks within a single, statistically grounded training objective.

Achieve 4K image-to-video generation with diffusion models without training by cleverly fusing tiled denoising with a low-resolution latent prior, balancing detail and global coherence.

A simple adaptive normalization technique can significantly improve continual learning performance on tabular data by mitigating catastrophic forgetting in dynamic environments.

Achieve state-of-the-art anomaly detection in multi-class and continual learning scenarios with AdapTS, a teacher-student framework that slashes memory overhead by up to 149x compared to existing methods.

Normalizing error signals, not just activations, is the key to unlocking the benefits of inhibition-mediated normalization for learning in neural networks.

Neural networks can accurately predict polymer free energies, even when traditional methods like Bennett Acceptance Ratio fail due to poor phase-space overlap.

YOLO can learn faster and better by strategically skipping redundant images during training, achieving a 1.43x speedup and improved accuracy with a new Anti-Forgetting Sampling Strategy.

Finally, a software energy profiler achieves both high accuracy and cross-platform portability, enabling practical algorithmic energy optimization across diverse languages and hardware.

By reorganizing 3D scenes into structurally-aware subscenes, S-VGGT offers a parallel geometric bridge for efficient processing, slashing global attention costs without compromising reconstruction fidelity.

Ditch the polar decomposition: MUD offers a surprisingly simple and efficient alternative for momentum whitening, speeding up transformer training by up to 50% compared to AdamW and Muon.

Achieve zero-shot adaptation to new tasks in complex control environments by learning a shared low-dimensional goal embedding that unifies policy and value function representations.

Achieve competitive video generation with Stable Diffusion using only 2.9% additional parameters by adapting temporal attention based on motion content, outperforming methods with explicit temporal consistency losses.

LLMs can maintain performance while skipping global attention for 80% of tokens, slashing compute costs and memory footprint in long-context scenarios.

Q-value policies, traditionally outperformed by state-value policies in planning, can surpass them with the right regularization, offering a faster alternative for policy evaluation.

Predicting permeability tensors from microstructure images just got 33% more accurate thanks to a physics-informed CNN-Transformer that learns faster and generalizes better via pretraining and differentiable constraints.

Mirror Descent, a workhorse of large-scale optimization, now has a Riemannian generalization with convergence guarantees, opening doors to efficient optimization on curved spaces.

Even without architectural modifications, a new gradient inversion attack, ARES, can reconstruct high-fidelity training samples in federated learning, exposing a significant privacy risk.

Differential attention and asymmetric loss functions can significantly improve the performance of BiomedCLIP on highly imbalanced video classification tasks like identifying rare pathologies in video capsule endoscopy.

By co-training flow and retrieval networks, WINFlowNets eliminates the need for pre-training, unlocking CFlowNets for dynamic robotic environments where data is scarce.

Genetic programming can discover unconventional multigrid cycles that outperform hand-tuned methods, suggesting automated algorithm design can unlock untapped performance in classical numerical solvers.

Robots can now plan 9x faster and achieve significantly higher success rates by decoupling action prediction from video generation in World-Action Models.

Online RLHF can match the performance of offline RLHF with 10x less data, and potentially 1000x at scale.

Forget dropout – Gaussian Chaos Noise offers provable control over representation deformation and boosts calibration in deep networks.

Instance-specific timestep schedules can significantly boost diffusion model performance, challenging the reliance on global discretization strategies.

Achieve state-of-the-art performance in multimodal remote sensing semantic segmentation with significantly fewer trainable parameters by using a novel parameter-efficient and modality-balanced symmetric fusion framework.

Autoregressive neural surrogates can now simulate dynamical systems for infinitely long horizons, thanks to a novel self-refining diffusion model that avoids error compounding.

Unlocking insights from massive molecular dynamics simulations just got easier: covariance matrix comparisons reveal key physical properties and phase transitions with remarkable data efficiency.

Ditch the data augmentation and decoders: R2-Dreamer's Barlow Twins-inspired objective delivers faster, more versatile MBRL, especially when spotting the small stuff matters.

Forget full finetuning: OPERA's dynamic pruning lets you adapt retrieval models to new domains with better ranking and recall, in half the time.

Jointly training audio watermarking and source separation unlocks robust multi-stream watermarking, enabling independent tracking of individual audio components within a mix.

Biased compression, previously overlooked in distributed learning with gradient coding, can actually boost performance when combined with error feedback to mitigate straggler effects and reduce communication costs.

Forget quadratic attention: FEAT achieves state-of-the-art performance on structured data with linear complexity and 40x faster inference.

Masked diffusion language models can now achieve 21.8x better compute efficiency than autoregressive models, thanks to binary encoding and index shuffling.

This Italian LLM punches way above its weight, matching the performance of models trained on 6-10x more data while using only 3B active parameters during inference.

Software energy consumption isn't just an aggregate number – it's a path-dependent journey, and this new model reveals hidden optimization opportunities that can slash energy use by up to 705x.

SympFormer achieves faster convergence in attention blocks by drawing inspiration from inertial Nesterov acceleration, offering a potential speedup without additional computational cost.

Unsupervised pretraining of drug-response models offers clear gains when adapting to patient tumors with very limited labeled data, despite providing limited benefit when source and target domains overlap substantially.

Solve Fokker-Planck equations on manifolds without meshes by pushing forward samples with neural networks.

DynamicGate-MLP learns to selectively activate MLP units based on the input, achieving better compute efficiency without sacrificing performance.

Fine-tune 123B+ parameter models on a single RTX 4090 with SlideFormer, a system that achieves up to 6x larger models and 8x larger batch sizes.

Unfolding the EM algorithm into a neural network yields a speaker localization method that's more robust and accurate than traditional Batch-EM, especially in challenging acoustic conditions.

Achieve near-linear scaling and 40x speedup for MP2 calculations on large molecules by unleashing multi-GPU parallelism for local correlation methods.

Deep learning slashes design time for high-efficiency Doherty power amplifiers, enabling complex pixelated combiners that extend the back-off efficiency range.

SNNs can now learn robust visual representations from unlabeled event data, rivaling supervised learning in low-data regimes, thanks to a new contrastive self-supervised learning framework.

Forget perplexity – ZipCal uses Zipf's law to curate calibration data for LLM compression, matching state-of-the-art performance at 240x the speed.

PKINet-v2 achieves state-of-the-art accuracy in remote sensing object detection while delivering a 3.9x FPS acceleration by fusing anisotropic and isotropic kernels into a single efficient depth-wise convolution.

Rank-1 LoRA fine-tuning can safely and efficiently adapt simulated locomotion policies to real-world robots, slashing fine-tuning time by nearly half while maintaining safety.

Train your UMM visual generation component on image-only data first and you'll get SOTA performance with a fraction of the compute.

Resource consumption vulnerabilities in LLMs can lead to degraded service availability and economic sustainability, demanding a systematic understanding and mitigation approach.

Ditch quadratic attention in your ViTs without sacrificing performance: ViT-AdaLA distills knowledge from pre-trained VFMs into linear attention architectures, achieving state-of-the-art results on classification and segmentation.

Forget expensive distillation – aligning language models can be as simple as carefully choosing the right mix of pretraining data based on log-likelihood differences.

MXFP4 quantization just got a whole lot better: BATQuant recovers up to 96.43% of full-precision performance in LLMs and MLLMs, even under aggressive W4A4KV16 settings, by preventing outlier propagation across quantization blocks.

Forget replay buffers: this method achieves state-of-the-art continual learning performance with zero additional memory by training on both concrete data and its abstract representations.

Reinforcement learning can now orchestrate the complex, whole-body movements of salamander robots, enabling seamless transitions between walking and swimming.