Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Generative representations need more than just semantic alignment; they require active exploitation, as shown by Representation Guidance outperforming larger models on ImageNet with a novel flow-matching target modification.

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.

Generative representations need more than just semantic alignment; they require active exploitation, as shown by Representation Guidance outperforming larger models on ImageNet with a novel flow-matching target modification.

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

A single recurrent Transformer block can match the accuracy of deep vision encoders with 70% fewer parameters by dynamically programming FFN experts via a continuous depth coordinate.

Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.

Exploit sparse derivative structure to slash AD passes for Jacobians and Hessians, offering drop-in replacements for JAX users.

A single regional weather model can now leverage multiple global forecast drivers without retraining, achieving kilometer-scale accuracy.

Looped Transformers can achieve tens of thousands of effective reasoning steps without performance collapse by learning to selectively retain and decay intermediate states.

Quantum classifiers grow to unpredictable depths regardless of training data size, and their generalization is not explained by active gate count alone.

Input-specific parameter generation unlocks adaptive time-series forecasting, outperforming static adaptation and improving zero-shot generalization.

Noise in Gaussian regression on Boolean cube polynomials can necessitate an exponential increase in sample complexity, requiring $(m+t)4^k\exp\{-O(k^{1/3})\}$ samples for parametric rates, a stark contrast to the $O((m+t)2^k)$ needed for noiseless identification.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Most published TTA methods for time-series forecasting are outperformed by simple, hyperparameter-free baselines under realistic label delays, with some even becoming harmful.

A novel hierarchical decomposition into soft communities allows for scalable attributed graph generation that balances structural fidelity and downstream utility without memorizing the reference graph.

State-preserving constraints in Bayesian optimization are transformed into an efficient, anisotropic ellipsoid in design space, enabling robust optimization of complex engineering systems.

Gradient-based learning is now possible for variational sequential Monte Carlo without iterative solvers, thanks to a differentiable relaxation of systematic resampling.

QNN gate selection and arrangement, not just qubit count, critically dictate financial forecasting performance, with the best configuration outperforming classical BiLSTM and exhibiting significant noise sensitivity on current quantum hardware.

Learnable shape parameterizations unlock superior sample efficiency and design exploration in Bayesian optimization, outperforming hand-crafted approaches across diverse engineering domains.

Heterogeneous experts can be dynamically coordinated via MoE gating to form composite model services, outperforming monolithic models and standard MoE in cost-aware marketplaces.

Structured linear dynamical systems can be learned from drastically undersampled observations ($p=o(1)$) with trajectory lengths far shorter than previously thought possible.

A residual memory mechanism allows LLMs to retain crucial context for long-horizon tasks, achieving near full-context performance with only 5.2% of input positions.

Directly predicting conditional marginals with a single Transformer pass unlocks efficient decision-making under uncertainty, outperforming generative models and classical methods even with full model knowledge.

Kernel methods finally offer a scalable and interpretable alternative to black-box ANNs for learning Koopman embeddings, matching or surpassing ANN performance on complex dynamical systems.

Auxiliary "ghost tasks" make any linear differential equation system effectively parametrizable for Gaussian Processes, solving a key limitation in physics-informed ML.

Neural networks can learn interpretable, polytope-based latent representations that preserve performance and enable direct VQ training.

Spectral learning unlocks $\mathcal{O}(d\log d)$ covariance estimation for dense Gaussian dependencies, outperforming prior methods in forecasting and prediction tasks.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

A single forecasting backbone unifies multivariate and multimodal time series, outperforming 16 SOTA models by disentangling media identity and mastering temporal scales.

Decoding schedules for diffusion and autoregressive models can be ranked by a kernel of pairwise dependence estimated from pretrained weights, predicting performance across modalities before generation.

Learned cable simulators can achieve 15%+ prediction error reduction and halve segment length drift by biasing attention with physical distances.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

Static video regions can be denoised 50% faster without sacrificing quality by dynamically skipping computation on easy patches.

AI agents can now navigate the intricate trade-offs of post-route power optimization, learning to balance speculative power recovery with costly repair actions under strict timing and physical constraints.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

A quadtree visual tokenizer slashes token count for autoregressive image generation by adaptively focusing representational power on intricate image regions.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Global point set alignment is solved exactly by finding the farthest vertex of a novel "permutation polygon," resolving a long-standing open problem in assignment geometry.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Derivative GPs can now achieve state-of-the-art accuracy with less computation and memory than function-only baselines by cleverly budgeting gradient information.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Physics-aligned electronic ground-state models shatter out-of-distribution generalization barriers for MLIPs, achieving near-chemical accuracy on reactive systems with minimal data.

Model selection for scientific forecasting is fundamentally protocol-dependent, with optimal choices reversing based on dataset size, system complexity, and evaluation metrics like state refresh versus closed-loop performance.

Mamba's linear-time state-space models unlock superior multi-scale financial volatility forecasting, outperforming Transformers by capturing regime shifts and enabling longer context.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

Negative curvature can amplify PWGF approximation errors, but subsequent positive curvature can recover accuracy over growing time horizons, enabling stable optimization in non-convex landscapes.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

A novel bias expansion for two-timescale stochastic approximation reveals a $\varepsilon^2/\eta$ term that fundamentally alters extrapolation strategies.

Axis-aligned splits in BART are a bottleneck for non-axis-aligned functions; RoBART achieves adaptive posterior contraction by rotating each tree's coordinate system.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Bayesian inference for complex switching stochastic differential equations is now broadly tractable, even with non-analytic transition densities.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Small language models achieve high abstract reasoning scores by memorizing task distributions rather than learning generalizable rules, failing even simple out-of-distribution shifts.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Discovering task-specific, compact concept vocabularies from unsupervised pools unlocks inspectable AI without the need for manual annotations or massive, static concept sets.

Exploiting cyclic symmetries in multi-index models offers a polynomial sample complexity advantage, with data augmentation proving as sample-efficient as architectural weight sharing.

Divergence choice in Shampoo-like optimizers fundamentally alters Kronecker approximation and finite-sample error compensation, explaining observed training behavior.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Quantum circuits can unlock compositional generalization in multimodal AI, even on noisy hardware.

Meta-learning unlocks efficient, resolution-independent PDE surrogate modeling for time-dependent systems by treating initial conditions as context, structuring latent spaces, and accelerating inference.

Encoding sensor geometry as a flow-matching prior dramatically improves generative models for brain signals, outperforming isotropic noise and even empirical data covariance.

Gradient-free few-shot adaptation emerges from decentralized cellular dynamics, achieving state-of-the-art OOD transfer by updating a spatial program through local neighborhood interactions.

Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.

Unlocking broad downstream generalization for 3D atomistic systems, Atom-JEPA achieves state-of-the-art property prediction by learning from unlabeled structures alone.

Generate valid and unique discrete designs in a single forward pass with over $50\times$ the yield per operation compared to prior one-step methods.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Sparse MoE models match dense models' code repair correctness with drastically fewer active parameters, challenging the sole reliance on total parameter count for evaluating code generation efficiency.

End-to-end meta-learning of neural field representations reveals that first-order approximations discard crucial learning pathways, leading to improved few-shot adaptation and semantic prediction.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

Generative flow matching directly refines 4D hand-object interactions during synthesis, integrating physical constraints and 2D evidence without post-hoc optimization.

Chemical reaction models can now strictly enforce valence conservation and enabling rules without any training, achieving superior accuracy on reaction prediction tasks.

MLLMs can now natively weave elastic visual representations, adaptively allocating detail to save 43% of tokens while preserving near-perfect performance.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Explicitly learning a shared perception-action representation boosts continuous-action VLA performance by over 13% on real-world tasks.

Generating molecular graphs in a pretrained VAE's latent space via flow matching unlocks significant efficiency gains and enables property-guided generation without retraining.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Wrist accelerometers can now yield population-scale sleep heart rate insights, with a novel transformer model outperforming existing methods in estimation availability.

Neural PDE solvers can miss optimal solutions due to unbounded parameters, but adding kernel derivatives recovers these "missing limits."

Hierarchical MoE routing with a novel arrival-weighted mean estimation prevents expert collapse and enables localized co-activation, outperforming flat routing on large embedding models.

A novel two-stream architecture disentangles temporal context from spatial refinement in EEG source imaging, achieving superior localization accuracy without compromising inference speed.