Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Generative representations need more than just semantic alignment; they require active exploitation, as shown by Representation Guidance outperforming larger models on ImageNet with a novel flow-matching target modification.

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Generative representations need more than just semantic alignment; they require active exploitation, as shown by Representation Guidance outperforming larger models on ImageNet with a novel flow-matching target modification.

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

A single recurrent Transformer block can match the accuracy of deep vision encoders with 70% fewer parameters by dynamically programming FFN experts via a continuous depth coordinate.

Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

A single regional weather model can now leverage multiple global forecast drivers without retraining, achieving kilometer-scale accuracy.

Looped Transformers can achieve tens of thousands of effective reasoning steps without performance collapse by learning to selectively retain and decay intermediate states.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Input-specific parameter generation unlocks adaptive time-series forecasting, outperforming static adaptation and improving zero-shot generalization.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Most published TTA methods for time-series forecasting are outperformed by simple, hyperparameter-free baselines under realistic label delays, with some even becoming harmful.

A novel hierarchical decomposition into soft communities allows for scalable attributed graph generation that balances structural fidelity and downstream utility without memorizing the reference graph.

State-preserving constraints in Bayesian optimization are transformed into an efficient, anisotropic ellipsoid in design space, enabling robust optimization of complex engineering systems.

Gradient-based learning is now possible for variational sequential Monte Carlo without iterative solvers, thanks to a differentiable relaxation of systematic resampling.

QNN gate selection and arrangement, not just qubit count, critically dictate financial forecasting performance, with the best configuration outperforming classical BiLSTM and exhibiting significant noise sensitivity on current quantum hardware.

Learnable shape parameterizations unlock superior sample efficiency and design exploration in Bayesian optimization, outperforming hand-crafted approaches across diverse engineering domains.

Heterogeneous experts can be dynamically coordinated via MoE gating to form composite model services, outperforming monolithic models and standard MoE in cost-aware marketplaces.

Structured linear dynamical systems can be learned from drastically undersampled observations ($p=o(1)$) with trajectory lengths far shorter than previously thought possible.

A residual memory mechanism allows LLMs to retain crucial context for long-horizon tasks, achieving near full-context performance with only 5.2% of input positions.

Directly predicting conditional marginals with a single Transformer pass unlocks efficient decision-making under uncertainty, outperforming generative models and classical methods even with full model knowledge.

Kernel methods finally offer a scalable and interpretable alternative to black-box ANNs for learning Koopman embeddings, matching or surpassing ANN performance on complex dynamical systems.

Auxiliary "ghost tasks" make any linear differential equation system effectively parametrizable for Gaussian Processes, solving a key limitation in physics-informed ML.

Neural networks can learn interpretable, polytope-based latent representations that preserve performance and enable direct VQ training.

Spectral learning unlocks $\mathcal{O}(d\log d)$ covariance estimation for dense Gaussian dependencies, outperforming prior methods in forecasting and prediction tasks.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

A single forecasting backbone unifies multivariate and multimodal time series, outperforming 16 SOTA models by disentangling media identity and mastering temporal scales.

Decoding schedules for diffusion and autoregressive models can be ranked by a kernel of pairwise dependence estimated from pretrained weights, predicting performance across modalities before generation.

Learned cable simulators can achieve 15%+ prediction error reduction and halve segment length drift by biasing attention with physical distances.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

A quadtree visual tokenizer slashes token count for autoregressive image generation by adaptively focusing representational power on intricate image regions.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Global point set alignment is solved exactly by finding the farthest vertex of a novel "permutation polygon," resolving a long-standing open problem in assignment geometry.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Derivative GPs can now achieve state-of-the-art accuracy with less computation and memory than function-only baselines by cleverly budgeting gradient information.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Physics-aligned electronic ground-state models shatter out-of-distribution generalization barriers for MLIPs, achieving near-chemical accuracy on reactive systems with minimal data.

Model selection for scientific forecasting is fundamentally protocol-dependent, with optimal choices reversing based on dataset size, system complexity, and evaluation metrics like state refresh versus closed-loop performance.

Mamba's linear-time state-space models unlock superior multi-scale financial volatility forecasting, outperforming Transformers by capturing regime shifts and enabling longer context.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

Negative curvature can amplify PWGF approximation errors, but subsequent positive curvature can recover accuracy over growing time horizons, enabling stable optimization in non-convex landscapes.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

A novel bias expansion for two-timescale stochastic approximation reveals a $\varepsilon^2/\eta$ term that fundamentally alters extrapolation strategies.

Axis-aligned splits in BART are a bottleneck for non-axis-aligned functions; RoBART achieves adaptive posterior contraction by rotating each tree's coordinate system.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Bayesian inference for complex switching stochastic differential equations is now broadly tractable, even with non-analytic transition densities.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.