Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

A specific looped Transformer architecture and input injection strategy unlock significant gains in reasoning and context utilization, outperforming larger models with drastically fewer parameters.

Reusable continuous-time summaries slash IMTSF inference latency by up to 87% and enable 52% faster forecasting over multiple calls, all with a fraction of the parameters.

Generative design can now produce moldable parts by transforming 3D shape generation into 2D paired depth image generation, bypassing undercuts by construction.

Recurrence's rapid learning of retrieval functions actively prevents attention from acquiring similar capabilities in hybrid architectures, necessitating staged training for optimal performance.

A novel Kronecker Delta Rule enables associative memories to revise targeted information with minimal interference, unlocking stable, long-context linear attention.

Reusable continuous-time summaries slash IMTSF inference latency by up to 87% and enable 52% faster forecasting over multiple calls, all with a fraction of the parameters.

Generative design can now produce moldable parts by transforming 3D shape generation into 2D paired depth image generation, bypassing undercuts by construction.

Recurrence's rapid learning of retrieval functions actively prevents attention from acquiring similar capabilities in hybrid architectures, necessitating staged training for optimal performance.

A novel Kronecker Delta Rule enables associative memories to revise targeted information with minimal interference, unlocking stable, long-context linear attention.

Compressing LLM embeddings by 50% without sacrificing performance is now achievable by reorganizing the feature space with a learned spectral basis.

General CPWL functions can be represented by ReLU networks with just two hidden layers, shattering previous depth conjectures.

LLMs can now achieve 66x storage reduction and 1.86x speedup at 128K context by internalizing memory, not just retrieving it.

Agents can now dynamically compose learned skills by selecting LoRA adapters as actions, overcoming catastrophic forgetting and boosting performance by up to 27.5% on complex benchmarks.

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.

Quantum Hamiltonian embeddings unlock hypothesis classes beyond classical reach by encoding datasets into parameterized Hamiltonians optimized for ground-state classification.

Lifelong generative models can now learn new tasks without forgetting old ones by directly evolving their velocity fields, bypassing expensive trajectory simulations.

Recursive pretraining with gated mixture recurrence unlocks substantial parameter efficiency and adaptive computation for protein language models, outperforming larger models on key structural prediction tasks.

Polynomial gradient orthogonalization achieves 15-18% faster runtimes than state-of-the-art methods by leveraging optimized polynomial circuits.

Gaussian process additive models can now precisely control the prior explained variance of each component, overcoming the inherent variance overestimation of standard Matérn priors.

Near-autoregressive quality is achievable with parallel decoding, unlocking significant inference speedups across model scales.

Neural networks for point clouds require quadratic representation orders to guarantee separation, a concrete bound finally established.

Nested recursive reasoning architectures fail to consistently improve ARC-AGI-1 performance over baselines at matched parameter counts, even with a single-GPU budget.

Continuum robot design can now be optimized with 5x less data and 30-50% higher accuracy, while also quantifying prediction uncertainty.

Few-shot prompting with historical snapshots enables graph foundation models to generalize contingency analysis to unseen power system topologies, overcoming limitations of pure regression approaches.

Explicitly conditioning sleep representations on global time-of-sleep dramatically boosts performance on challenging pediatric sleep tasks, even across different datasets and models.

A single foundation model can now decode plant stress signals across species and environments, unlocking unprecedented scalability for agricultural and ecological monitoring.

Adaptive refinement guided by collocation residuals in hybrid PINNs slashes solution errors for singular PDEs by up to 38x, outperforming uniform refinement.

Learned expand-and-sparsify representations with ReLU activation match dense layers at significantly reduced sparsity, and binarization costs learned sparse codes nothing.

Simple logit averaging during training, not complex attention mechanisms, unlocks state-of-the-art performance in non-invasive speech BCIs, outperforming prior winners by a significant margin.

A Transformer that treats coupon discount dimensions as queries unlocks 1.26% higher business value per user by effectively modeling cross-category demand shifts.

Adapting MoE experts directly, not through bypass adapters, unlocks the best downstream performance with minimal general capability loss.

Sparse attention in video diffusion models can be corrected using cached dense refinements, recovering fidelity without altering the reuse schedule.

Fine-scale automotive flow prediction can now leverage incremental states passed cross-layer, achieving superior accuracy by continuously integrating local details into global physics.

Iterative GNNs trained on solver trajectories outperform single-pass SOCP solvers and enable warm-starting classical optimizers.

Target Propagation's connection to Gauss-Newton optimization is established under a new implicit norm, with CGTP achieving KFAC-like performance using orders of magnitude fewer outer iterations.

A multimodal spatiotemporal transformer system achieves near-perfect detection of Pyropia bleaching seven days in advance, but its efficacy is critically dependent on the recency of nutrient survey data.

Self-supervised GNNs can directly learn distortion-aware precoders for satellite systems, outperforming traditional optimization methods in speed and operating range while maintaining QoS.

LLMs can now "think with visual latents" by generating masks directly from their internal representations, dramatically improving referring comprehension and segmentation.

A specific looped Transformer architecture and input injection strategy unlock significant gains in reasoning and context utilization, outperforming larger models with drastically fewer parameters.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Generative time series models can now produce multiple plausible future trajectories as fast as quantile models query a single point, unlocking accurate dependency modeling for critical applications.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

Dual-view time-series representations unlock significant gains in temporal reasoning by preserving both absolute measurements and invariant temporal structure, outperforming dedicated TSLMs.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

Exponential Hopfield networks learn exponentially more archetypes than pairwise models by leveraging softmax attention, but risk memorizing noisy examples over true archetypes.

A simple byte reordering strategy slashes multilingual boundary prediction errors by over 70% in scripts like Korean and Chinese, fundamentally altering how byte-level models perceive codepoint transitions.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

A single weight-space encoder now masters 100 transformer architectures, outperforming specialized models and handling unseen, larger networks with less memory.

A single INR can now model complex, periodic deformations across entire image sequences, enabling self-regularization and outperforming prior methods.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Single-qubit quantum circuits, despite universal function approximation, get stuck in training due to "phase-locking," requiring orders of magnitude more optimization steps than multi-qubit equivalents.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.

A novel PI-DeepONet implementation slashes training time by up to $3.75\times$ and makes 3D physics-informed learning feasible, overcoming critical computational barriers.

Specialized experts in MoE LLMs, not dense model neurons, are the key to interpretable and controllable personalization.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

Deep learning architectures designed for semantic understanding can now solve complex problems in algebraic topology, identifying topologically equivalent finite group actions on Riemann surfaces.

Asymmetric temporal dynamics and expanded receptive fields are unlocked in spatio-temporal graph forecasting by a unified spectral operator on cellular sheaves.

Decentralized agent swarms can learn to self-orchestrate their collective behavior and individual capabilities simultaneously, outperforming fixed coordination strategies.

Streaming VLMs can now internalize long-term history as evolving LoRA weights, bypassing context window limitations and achieving superior performance on video understanding tasks.

Video generators can be directly transformed into powerful 4D perception systems without iterative refinement, achieving SOTA on tracking and pose estimation.

Spatial intelligence tasks are solved 5.6x faster with zero accuracy loss by reframing them as classification problems, eliminating autoregressive decoding entirely.

A frozen, 16MB vocabulary can match or beat learned embeddings in small LLMs, unlocking dramatic speedups and memory reductions for edge deployment.

Explicitly modeling metrical structure alongside note events halves transcription error in expressive piano performances, bridging the gap between raw audio and symbolic notation.

Small, fast-moving objects and fine textures in dynamic volumetric video are finally recoverable by intelligently allocating Gaussians based on future motion, not just current positions.

Aggressive KV cache compression can be achieved with near-zero accuracy loss by treating the cache as a trainable neural network optimized at test-time.

A simple, fixed query rule improves FID by over 2 points on CIFAR-10 without retraining, revealing prediction horizons are state-sensitive inference controls.

Spectral decomposition via a learnable Hodge-Laplacian unlocks a 47% reduction in interior MSE for steady-PDE surrogates on coarse meshes.

Continuous integration over a piecewise linear memory curve unlocks exact normalized attention with analytical gradients, outperforming softmax and focal attention in causal Transformers.

A single, elastic transformer model can now dynamically adapt its compute budget across a continuum of sparsity levels, achieving up to 1.80x higher throughput and 1.26x lower latency on A100 GPUs without retraining.

Simple feature propagation and label smoothing on relational graphs match much of the performance of complex GNNs, suggesting architectural complexity might be over-emphasized.

A single pretrained tabular foundation model can solve diverse power grid analysis tasks, from power flow to contingency screening, without fine-tuning.

Training-free neural emulators achieve solver-level accuracy and out-of-distribution robustness by directly encoding known polynomial physics.

N-gram embeddings can be learned without any additional parameters or GPU memory, unlocking massive scaling and instant domain adaptation for LMs.

A neural operator reconstructs continuous HRTF fields from sparse measurements with 18% lower LSD and 43% lower ILD error than leading baselines, achieving 373x speedup over neural fields.

Generative representation learning unlocks compute-optimal latent tokenization, achieving superior image quality with drastically fewer FLOPs than prior methods.

Tabular foundation models can achieve state-of-the-art in-context learning by selectively activating feature-specific experts, outperforming dense models on challenging datasets.

Quantizing MoE models with mixed precision is significantly improved by calibrating expert error sensitivity using an information-theoretic contrast target, outperforming simple ranking-based methods.

Speculative decoding drafters can achieve over 2x speedup on 256K context by treating distant tokens as shared far memory with sharpened attention.

Asynchronous EEG event detection is solved by a novel multiscale discrete spatiotemporal tokenization that captures both brief and sustained neural dynamics, outperforming prior methods across diverse datasets.

Generative models for PDEs can now recover fine-scale details and maintain long-horizon stability by learning residual distributions and aligning multiple trajectories.

Neural operators learn optimal derivative orders strictly below the PDE order, revealing a data-driven spectral regularization mechanism.

Spectral convolutional kernels unlock significantly greater accuracy gains from learned resolution allocation than their spatial counterparts, even when spatial area is normalized.

Deep learning DoA estimators can drastically cut inference time, but SVD-based beamforming remains the primary latency bottleneck in real-time ISAC.

RNA inverse design is fundamentally a topology-and-geometry co-design problem, and TANGO achieves a 52.7% improvement in secondary-structure fidelity by explicitly modeling this.

For star-free languages, transformers learn best when attention weights lag behind output layer updates, defying intuitive feature learning theories.

A single turbulent flow realization, when intelligently patched, can unlock extreme-scale super-resolution, outperforming models trained on multiple full snapshots.

The minimal width of ReLU networks for approximating functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ is $n+m$ for specific $n$ values, directly linking network capacity to the topological properties of real projective space.

MoE models slash the performance cost of multilingual expansion by 83% when scaling from 12 to 200 languages, fundamentally altering the trade-offs in capacity allocation.

The loss surface of a simple polynomial activation network exactly solves its own training dynamics, revealing a blueprint for training hundred-layer networks without normalization.

A replay-free GNN architecture effectively combats recency bias in class-incremental learning by routing nodes to experts via masked autoencoder reconstruction error, outperforming existing methods without task IDs.

A novel heterogeneous graph representation and message-passing mechanism unlocks 18.69% better routability prediction by jointly encoding circuit topology, layout, and routing grids.

Visualize KANs like never before: directly manipulate B-spline control points and see gradient descent rewire activation functions in real-time.

The cost of tracking changing comparators in exp-concave online learning is reduced by a factor of $d^{1/3}$, resolving the dimension's optimal dependence.

Neural operators are Bregman proximity operators, unlocking a new class of models with enhanced depth and accuracy.

Staged PINNs recover power converter parameters with 5% accuracy for L/C and 10% for ESR, outperforming simultaneous training by an order of magnitude in waveform RMSE.

Practical approximations to orthogonalization in spectral-norm steepest descent methods like Muon can flip their implicit bias from spectral to Frobenius max-margin.

Region-level learning with granular balls dramatically boosts TSVR's robustness against noise and outliers, outperforming point-wise methods on challenging real-world regression tasks.

Hyperbolic geometry enables RL agents to navigate complex mazes with 13x fewer parameters and 12x faster training than massive Euclidean networks, by aligning representation geometry with task demands.

Transformer context for in-context RL can be compressed to the Krylov dimension, revealing a $Θ(γ^m)$ decay in approximation error below this threshold.

Speculative drafting can now dynamically query and integrate historical context from recurrent states, unlocking $3.16\times$ speedups on LLM generation.

A single global topology predictor can unlock substantial performance gains in Transformers, outperforming layer-wise routing and achieving better results with less data.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.