Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Generative time series models can now produce multiple plausible future trajectories as fast as quantile models query a single point, unlocking accurate dependency modeling for critical applications.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

Dual-view time-series representations unlock significant gains in temporal reasoning by preserving both absolute measurements and invariant temporal structure, outperforming dedicated TSLMs.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Generative time series models can now produce multiple plausible future trajectories as fast as quantile models query a single point, unlocking accurate dependency modeling for critical applications.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

Dual-view time-series representations unlock significant gains in temporal reasoning by preserving both absolute measurements and invariant temporal structure, outperforming dedicated TSLMs.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

Exponential Hopfield networks learn exponentially more archetypes than pairwise models by leveraging softmax attention, but risk memorizing noisy examples over true archetypes.

A simple byte reordering strategy slashes multilingual boundary prediction errors by over 70% in scripts like Korean and Chinese, fundamentally altering how byte-level models perceive codepoint transitions.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

A single weight-space encoder now masters 100 transformer architectures, outperforming specialized models and handling unseen, larger networks with less memory.

A single INR can now model complex, periodic deformations across entire image sequences, enabling self-regularization and outperforming prior methods.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Single-qubit quantum circuits, despite universal function approximation, get stuck in training due to "phase-locking," requiring orders of magnitude more optimization steps than multi-qubit equivalents.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.

A novel PI-DeepONet implementation slashes training time by up to $3.75\times$ and makes 3D physics-informed learning feasible, overcoming critical computational barriers.

Specialized experts in MoE LLMs, not dense model neurons, are the key to interpretable and controllable personalization.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

Deep learning architectures designed for semantic understanding can now solve complex problems in algebraic topology, identifying topologically equivalent finite group actions on Riemann surfaces.

Asymmetric temporal dynamics and expanded receptive fields are unlocked in spatio-temporal graph forecasting by a unified spectral operator on cellular sheaves.

Decentralized agent swarms can learn to self-orchestrate their collective behavior and individual capabilities simultaneously, outperforming fixed coordination strategies.

Streaming VLMs can now internalize long-term history as evolving LoRA weights, bypassing context window limitations and achieving superior performance on video understanding tasks.

Video generators can be directly transformed into powerful 4D perception systems without iterative refinement, achieving SOTA on tracking and pose estimation.

Spatial intelligence tasks are solved 5.6x faster with zero accuracy loss by reframing them as classification problems, eliminating autoregressive decoding entirely.

Aggressive KV cache compression can be achieved with near-zero accuracy loss by treating the cache as a trainable neural network optimized at test-time.

A simple, fixed query rule improves FID by over 2 points on CIFAR-10 without retraining, revealing prediction horizons are state-sensitive inference controls.

Spectral decomposition via a learnable Hodge-Laplacian unlocks a 47% reduction in interior MSE for steady-PDE surrogates on coarse meshes.

Continuous integration over a piecewise linear memory curve unlocks exact normalized attention with analytical gradients, outperforming softmax and focal attention in causal Transformers.

A single, elastic transformer model can now dynamically adapt its compute budget across a continuum of sparsity levels, achieving up to 1.80x higher throughput and 1.26x lower latency on A100 GPUs without retraining.

Simple feature propagation and label smoothing on relational graphs match much of the performance of complex GNNs, suggesting architectural complexity might be over-emphasized.

A single pretrained tabular foundation model can solve diverse power grid analysis tasks, from power flow to contingency screening, without fine-tuning.

Training-free neural emulators achieve solver-level accuracy and out-of-distribution robustness by directly encoding known polynomial physics.

N-gram embeddings can be learned without any additional parameters or GPU memory, unlocking massive scaling and instant domain adaptation for LMs.

A neural operator reconstructs continuous HRTF fields from sparse measurements with 18% lower LSD and 43% lower ILD error than leading baselines, achieving 373x speedup over neural fields.

Generative representation learning unlocks compute-optimal latent tokenization, achieving superior image quality with drastically fewer FLOPs than prior methods.

Tabular foundation models can achieve state-of-the-art in-context learning by selectively activating feature-specific experts, outperforming dense models on challenging datasets.

Quantizing MoE models with mixed precision is significantly improved by calibrating expert error sensitivity using an information-theoretic contrast target, outperforming simple ranking-based methods.

Speculative decoding drafters can achieve over 2x speedup on 256K context by treating distant tokens as shared far memory with sharpened attention.

Asynchronous EEG event detection is solved by a novel multiscale discrete spatiotemporal tokenization that captures both brief and sustained neural dynamics, outperforming prior methods across diverse datasets.

Generative models for PDEs can now recover fine-scale details and maintain long-horizon stability by learning residual distributions and aligning multiple trajectories.

Neural operators learn optimal derivative orders strictly below the PDE order, revealing a data-driven spectral regularization mechanism.

Spectral convolutional kernels unlock significantly greater accuracy gains from learned resolution allocation than their spatial counterparts, even when spatial area is normalized.

Deep learning DoA estimators can drastically cut inference time, but SVD-based beamforming remains the primary latency bottleneck in real-time ISAC.

RNA inverse design is fundamentally a topology-and-geometry co-design problem, and TANGO achieves a 52.7% improvement in secondary-structure fidelity by explicitly modeling this.

For star-free languages, transformers learn best when attention weights lag behind output layer updates, defying intuitive feature learning theories.

A single turbulent flow realization, when intelligently patched, can unlock extreme-scale super-resolution, outperforming models trained on multiple full snapshots.

The minimal width of ReLU networks for approximating functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ is $n+m$ for specific $n$ values, directly linking network capacity to the topological properties of real projective space.

MoE models slash the performance cost of multilingual expansion by 83% when scaling from 12 to 200 languages, fundamentally altering the trade-offs in capacity allocation.

The loss surface of a simple polynomial activation network exactly solves its own training dynamics, revealing a blueprint for training hundred-layer networks without normalization.

A replay-free GNN architecture effectively combats recency bias in class-incremental learning by routing nodes to experts via masked autoencoder reconstruction error, outperforming existing methods without task IDs.

A novel heterogeneous graph representation and message-passing mechanism unlocks 18.69% better routability prediction by jointly encoding circuit topology, layout, and routing grids.

Visualize KANs like never before: directly manipulate B-spline control points and see gradient descent rewire activation functions in real-time.

The cost of tracking changing comparators in exp-concave online learning is reduced by a factor of $d^{1/3}$, resolving the dimension's optimal dependence.

Neural operators are Bregman proximity operators, unlocking a new class of models with enhanced depth and accuracy.

Staged PINNs recover power converter parameters with 5% accuracy for L/C and 10% for ESR, outperforming simultaneous training by an order of magnitude in waveform RMSE.

Practical approximations to orthogonalization in spectral-norm steepest descent methods like Muon can flip their implicit bias from spectral to Frobenius max-margin.

Region-level learning with granular balls dramatically boosts TSVR's robustness against noise and outliers, outperforming point-wise methods on challenging real-world regression tasks.

Hyperbolic geometry enables RL agents to navigate complex mazes with 13x fewer parameters and 12x faster training than massive Euclidean networks, by aligning representation geometry with task demands.

Transformer context for in-context RL can be compressed to the Krylov dimension, revealing a $Θ(γ^m)$ decay in approximation error below this threshold.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A single open-source model now performs multi-character scene dubbing, in-filling edits, and multi-turn revisions with natural language direction, all at 271 fps.

Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.

Prompt adaptation methods don't just change what LLMs say, they fundamentally rewire *how* they compute, with distinct mechanisms for text optimization, prompt tuning, and prefix tuning.

A single dimensionless motion prior generalizes across vastly different object families and data modalities, outperforming specialized methods without retraining.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

System One models can achieve direct, low-latency decision-making by treating answers as structured choices rather than serialized text, with Jeva demonstrating robust null detection and permutation invariance.

Larger minibatch optimal transport couplings in flow matching, enabled by a novel entropic approximation, significantly boost training stability and generation performance.

Gauge-equivariant GNNs generate internal representations that exhibit a surprising loop geometry, with contour length growing exponentially while spatial extent grows only logarithmically with depth.

Recursive message passing achieves 99% Sudoku-Extreme accuracy with just 16k parameters, outperforming larger neural baselines on N-queens and SAT with zero-shot generalization.

Transformers can be quantitatively predicted by a Bayesian reference, revealing their failure modes on complex, interdependent sequential data.

Explicitly modeling cross-layer information flow as a memory retrieval problem unlocks a linear-time attention alternative that outperforms existing methods.

A single unified framework achieves state-of-the-art performance in both action-conditioned world modeling and real-world robotic manipulation by dynamically routing specialized experts.

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Grokking's distinct phases—nuisance fitting, core learning, and delayed generalization—are analytically separable, with sharp conditions for occurrence and predictable plateau lengths.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

A cheap diagnostic predicts model alignment and merging success with $0.91$ correlation, eliminating costly empirical search.

Segmentation models implicitly learn powerful object-centric representations that outperform encoder-based feature extraction for medical image prediction, even without explicit downstream supervision.

Equivariant MLIPs just got a 250x speedup: LunaTP unlocks efficient tensor products by rethinking parallel execution.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

Physically informed GNNs can accurately forecast advection-diffusion dynamics on irregular grids, outperforming existing methods and enhancing solar power predictions.

Contextual bandits with lumpable structure can be solved with $r$ times less regret than previously thought possible.

Two residual streams in diffusion language models halve the performance gains of adding a second stream by enabling distinct read and write routing patterns.

Distributed attention can achieve near-zero communication overhead for mask localization and compute-communication overlap, unlocking massive throughput gains for long-context LLMs.

Explicitly modeling copy operations unlocks substantial improvements in LLM efficiency and performance, even for models already adept at in-context learning.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Generative image and video compression achieves significant memory and compute reduction by replacing dense Gaussian codebooks with implicit randomized Hadamard dictionaries, enabling real-time streaming video compression on a single GPU.