Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A single open-source model now performs multi-character scene dubbing, in-filling edits, and multi-turn revisions with natural language direction, all at 271 fps.

Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.

Prompt adaptation methods don't just change what LLMs say, they fundamentally rewire *how* they compute, with distinct mechanisms for text optimization, prompt tuning, and prefix tuning.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A single open-source model now performs multi-character scene dubbing, in-filling edits, and multi-turn revisions with natural language direction, all at 271 fps.

Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.

Prompt adaptation methods don't just change what LLMs say, they fundamentally rewire *how* they compute, with distinct mechanisms for text optimization, prompt tuning, and prefix tuning.

A single dimensionless motion prior generalizes across vastly different object families and data modalities, outperforming specialized methods without retraining.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

System One models can achieve direct, low-latency decision-making by treating answers as structured choices rather than serialized text, with Jeva demonstrating robust null detection and permutation invariance.

Larger minibatch optimal transport couplings in flow matching, enabled by a novel entropic approximation, significantly boost training stability and generation performance.

Gauge-equivariant GNNs generate internal representations that exhibit a surprising loop geometry, with contour length growing exponentially while spatial extent grows only logarithmically with depth.

Recursive message passing achieves 99% Sudoku-Extreme accuracy with just 16k parameters, outperforming larger neural baselines on N-queens and SAT with zero-shot generalization.

Transformers can be quantitatively predicted by a Bayesian reference, revealing their failure modes on complex, interdependent sequential data.

Explicitly modeling cross-layer information flow as a memory retrieval problem unlocks a linear-time attention alternative that outperforms existing methods.

A single unified framework achieves state-of-the-art performance in both action-conditioned world modeling and real-world robotic manipulation by dynamically routing specialized experts.

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Grokking's distinct phases—nuisance fitting, core learning, and delayed generalization—are analytically separable, with sharp conditions for occurrence and predictable plateau lengths.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

A cheap diagnostic predicts model alignment and merging success with $0.91$ correlation, eliminating costly empirical search.

Segmentation models implicitly learn powerful object-centric representations that outperform encoder-based feature extraction for medical image prediction, even without explicit downstream supervision.

Equivariant MLIPs just got a 250x speedup: LunaTP unlocks efficient tensor products by rethinking parallel execution.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

Physically informed GNNs can accurately forecast advection-diffusion dynamics on irregular grids, outperforming existing methods and enhancing solar power predictions.

Contextual bandits with lumpable structure can be solved with $r$ times less regret than previously thought possible.

Two residual streams in diffusion language models halve the performance gains of adding a second stream by enabling distinct read and write routing patterns.

Distributed attention can achieve near-zero communication overhead for mask localization and compute-communication overlap, unlocking massive throughput gains for long-context LLMs.

Explicitly modeling copy operations unlocks substantial improvements in LLM efficiency and performance, even for models already adept at in-context learning.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Generative image and video compression achieves significant memory and compute reduction by replacing dense Gaussian codebooks with implicit randomized Hadamard dictionaries, enabling real-time streaming video compression on a single GPU.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Generative representations need more than just semantic alignment; they require active exploitation, as shown by Representation Guidance outperforming larger models on ImageNet with a novel flow-matching target modification.

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

A single recurrent Transformer block can match the accuracy of deep vision encoders with 70% fewer parameters by dynamically programming FFN experts via a continuous depth coordinate.

Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

A single regional weather model can now leverage multiple global forecast drivers without retraining, achieving kilometer-scale accuracy.

Looped Transformers can achieve tens of thousands of effective reasoning steps without performance collapse by learning to selectively retain and decay intermediate states.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Input-specific parameter generation unlocks adaptive time-series forecasting, outperforming static adaptation and improving zero-shot generalization.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Most published TTA methods for time-series forecasting are outperformed by simple, hyperparameter-free baselines under realistic label delays, with some even becoming harmful.

A novel hierarchical decomposition into soft communities allows for scalable attributed graph generation that balances structural fidelity and downstream utility without memorizing the reference graph.

State-preserving constraints in Bayesian optimization are transformed into an efficient, anisotropic ellipsoid in design space, enabling robust optimization of complex engineering systems.

Gradient-based learning is now possible for variational sequential Monte Carlo without iterative solvers, thanks to a differentiable relaxation of systematic resampling.

QNN gate selection and arrangement, not just qubit count, critically dictate financial forecasting performance, with the best configuration outperforming classical BiLSTM and exhibiting significant noise sensitivity on current quantum hardware.

Learnable shape parameterizations unlock superior sample efficiency and design exploration in Bayesian optimization, outperforming hand-crafted approaches across diverse engineering domains.

Heterogeneous experts can be dynamically coordinated via MoE gating to form composite model services, outperforming monolithic models and standard MoE in cost-aware marketplaces.

Structured linear dynamical systems can be learned from drastically undersampled observations ($p=o(1)$) with trajectory lengths far shorter than previously thought possible.

A residual memory mechanism allows LLMs to retain crucial context for long-horizon tasks, achieving near full-context performance with only 5.2% of input positions.

Directly predicting conditional marginals with a single Transformer pass unlocks efficient decision-making under uncertainty, outperforming generative models and classical methods even with full model knowledge.

Kernel methods finally offer a scalable and interpretable alternative to black-box ANNs for learning Koopman embeddings, matching or surpassing ANN performance on complex dynamical systems.

Auxiliary "ghost tasks" make any linear differential equation system effectively parametrizable for Gaussian Processes, solving a key limitation in physics-informed ML.

Neural networks can learn interpretable, polytope-based latent representations that preserve performance and enable direct VQ training.

Spectral learning unlocks $\mathcal{O}(d\log d)$ covariance estimation for dense Gaussian dependencies, outperforming prior methods in forecasting and prediction tasks.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

A single forecasting backbone unifies multivariate and multimodal time series, outperforming 16 SOTA models by disentangling media identity and mastering temporal scales.

Decoding schedules for diffusion and autoregressive models can be ranked by a kernel of pairwise dependence estimated from pretrained weights, predicting performance across modalities before generation.

Learned cable simulators can achieve 15%+ prediction error reduction and halve segment length drift by biasing attention with physical distances.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

Static video regions can be denoised 50% faster without sacrificing quality by dynamically skipping computation on easy patches.

AI agents can now navigate the intricate trade-offs of post-route power optimization, learning to balance speculative power recovery with costly repair actions under strict timing and physical constraints.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

A quadtree visual tokenizer slashes token count for autoregressive image generation by adaptively focusing representational power on intricate image regions.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Global point set alignment is solved exactly by finding the farthest vertex of a novel "permutation polygon," resolving a long-standing open problem in assignment geometry.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Derivative GPs can now achieve state-of-the-art accuracy with less computation and memory than function-only baselines by cleverly budgeting gradient information.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Physics-aligned electronic ground-state models shatter out-of-distribution generalization barriers for MLIPs, achieving near-chemical accuracy on reactive systems with minimal data.

Model selection for scientific forecasting is fundamentally protocol-dependent, with optimal choices reversing based on dataset size, system complexity, and evaluation metrics like state refresh versus closed-loop performance.

Mamba's linear-time state-space models unlock superior multi-scale financial volatility forecasting, outperforming Transformers by capturing regime shifts and enabling longer context.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

Negative curvature can amplify PWGF approximation errors, but subsequent positive curvature can recover accuracy over growing time horizons, enabling stable optimization in non-convex landscapes.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

A novel bias expansion for two-timescale stochastic approximation reveals a $\varepsilon^2/\eta$ term that fundamentally alters extrapolation strategies.

Axis-aligned splits in BART are a bottleneck for non-axis-aligned functions; RoBART achieves adaptive posterior contraction by rotating each tree's coordinate system.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Bayesian inference for complex switching stochastic differential equations is now broadly tractable, even with non-analytic transition densities.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Small language models achieve high abstract reasoning scores by memorizing task distributions rather than learning generalizable rules, failing even simple out-of-distribution shifts.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Discovering task-specific, compact concept vocabularies from unsupervised pools unlocks inspectable AI without the need for manual annotations or massive, static concept sets.