Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Task-specific LoRA-Experts and prototype-ensemble matching unlock rehearsal-free class-incremental learning with reduced interference and improved discrimination.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

Recurrent memories can retain full Fisher information post-writing with an invertible hold, decoupling write and read optimality and achieving near-perfect task-specific performance.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Task-specific LoRA-Experts and prototype-ensemble matching unlock rehearsal-free class-incremental learning with reduced interference and improved discrimination.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

Recurrent memories can retain full Fisher information post-writing with an invertible hold, decoupling write and read optimality and achieving near-perfect task-specific performance.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

A 260M-parameter looped diffusion model achieves performance surpassing a 6.5x larger non-looped counterpart with significantly less inference compute.

A single time-series language model now masters forecasting, contextual prediction, and temporal reasoning, outperforming specialized models on key benchmarks.

Untied embeddings, not weight tying, unlock superior performance and memory efficiency for decoder-only LLMs under DP-SGD.

Decentralized power allocation in complex MANETs is solved with a learned GNN that matches centralized optimization performance using only local, noisy channel data.

Explicitly constraining class geometry with differentiable prototypes significantly boosts tensor neural network performance under extreme label scarcity, particularly through training-time regularization.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

A low-rank residual connection allows frozen LLMs to learn from feedback with minimal parameters, achieving substantial error reduction in downstream tasks.

Exact incremental BPE achieves $O(\log t)$ amortized append time by weighting centroid search intervals, a significant speedup over prior $O(\log^2 t)$ methods.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.

A conformal PINN framework bypasses model dependence and deep learning pitfalls to extract fundamental hadronic physics, yielding a $a_μ^{ππ}$ contribution with unprecedented precision.

Recursive reasoning models achieve state-of-the-art algorithmic performance and OOD generalization with a simple optimization recipe, decoupling architectural complexity from recurrent state stabilization.

A novel cross-attention mechanism dynamically clusters queries to achieve sub-quadratic complexity in neural operators for PDEs without sacrificing spatial resolution or model flexibility.

Bayesian generative models can now estimate densities without invertible transformations, achieving state-of-the-art performance by adaptively focusing computation.

A single learned linear layer, not the shared Transformer weights, dictates distinct computations within each recurrent loop of a Looped Transformer, with attention patterns causally routing these operations.

Random hyperplane arrangements reveal that neural networks can achieve rank lifting with a width that scales independently of input dimensionality, dramatically improving efficiency for separable data.

A fixed topological descriptor unlocks state-of-the-art time-series classification with 1000x fewer parameters than large neural networks.

Physical degradation models can now learn reusable representations that generalize across entirely unseen system types, significantly improving prognostic accuracy with limited downstream data.

The Green's operator reveals how computation is globally routed and reused across tasks and time in recurrent networks, offering a new lens beyond activity geometry.

Streaming world models can now maintain consistent representations across minutes-long revisits without quadratic attention costs by leveraging a novel retrieval-based memory bank.

Reconstructing revisited scenes in streaming world models is now possible with constant memory, outperforming full-softmax attention by 30% on peak memory.

Future needs, not current context, are the key to unlocking effective long-horizon video generation.

Two shared scalar signals are sufficient to dynamically govern complex spectral memory, outperforming prior adaptive recurrences and accelerating training.

A single inference engine now unifies 15 sparse attention methods, unlocking over 10x throughput gains for long-context LLMs through intelligent KV cache management.

Weight decay, not just learning rate, dictates the peak magnitude and lifespan of massive activations in transformers.

Zero-shot in-context inference trained strictly on synthetic causal graphs can outperform fully tuned AutoML pipelines across diverse real-world tabular benchmarks.

Enforcing uniform load balancing across video tokens fundamentally breaks visual coherence, but bifurcating experts into semantic and generic tiers resolves patch fragmentation while driving emergent coarse-to-fine denoising.

Sub-quadratic linear attention no longer requires full-precision state caching: 8-bit recurrent states can match FP32 accuracy while delivering up to 3.70× kernel speedups without any retraining.

Fully factorizing self-attention across separate temporal, spectral, and spatial dimensions slashes compute while dominating SITS segmentation benchmarks—and uncovers that common tile-size discrepancies have quietly rendered most published baseline comparisons invalid.

Linear attention's recurrent state can be compressed down to 4 bits while outperforming uniform INT8 once precision is dynamically allocated to long-lived memory tracks and output-sensitive key rows.

Predicting dynamic expert activations two layers in advance eliminates transfer stalls, enabling massive MoE models to run up to 11.7× faster on a single commodity GPU.

Rigid coarse-to-fine hierarchies cripple multi-scale anomaly detection: letting arbitrary patch granularities interact symmetrically via bidirectional cross-scale attention yields a 9%+ VUS-PR leap over 50 baselines across 530 time series benchmarks.

Severe generation degradation in few-step parallel diffusion is an objective mismatch rather than an architectural limitation: swapping cross-entropy for a joint-aware $\alpha$-loss yields coherent generation in as few as four steps.

The exponential curse of dimensionality in rule-based machine learning can be collapsed to polynomial complexity $\mathcal{O}(mn^2)$ via Kolmogorov-Arnold decomposition, cutting parameters by up to 98% without forfeiting universal approximation guarantees.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Deterministic dependency graphs severely misalign with stochastic residual structures in decoupled diffusion forecasters, but dynamically sparsifying edges via residual uncertainty estimation eliminates redundant conditioning and repairs calibration.

Far from merely clustering tokens, full Transformers act as pattern-forming dynamical systems that natively amplify traveling and rotating waves—an inductive bias that can be engineered at initialization to accelerate training.

Most directions in a transformer's residual stream are functionally invisible to the terminal loss, exposing a thin observable subspace whose pullback Fisher geometry consistently outperforms standard attention weights for compression and distillation.

Continual learners no longer need oracle task IDs to dynamically scale capacity, as loss-driven width expansion coupled with persistent key-value attention prevents catastrophic forgetting across unsegmented task streams.

Non-factorized diffusion language models achieve superior few-step generation by leveraging a discrete latent space learned via Mixture-of-Experts routing, bypassing posterior collapse issues.

A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.

Optimizing soft memory directly for user behavior, not text reconstruction, unlocks state-of-the-art long-context personalization.

Video world models can now maintain persistent scene memory without ever-growing storage by warping and fusing low-rank features into a fixed-size HexMemory.

Key contribution not extracted.

INT4 KV caching for MLA models is now possible, achieving 3.59x compression at 128K context and enabling 5.168x higher throughput in cache-pressure serving workloads.

Latent communication between LLMs was largely an illusion of interface shortcutting until Draft-KV, which routes genuine drafting KV caches to let a frozen 0.5B model leap from 37% to 78% MMLU-Redux accuracy by tapping an 8B model's latent thoughts.

Uniform layer looping wastes latent compute on trivial tokens, but dynamically routing iteration depth beats standard transformers on AIME with a 53% steeper test-time scaling slope.

Transformer row representations fail to form an autonomous thermodynamic critical class during training, meaning popular physical scaling analogies collapse unless explicit optimizer memory and readout error budgets are accounted for.

Transformers actively protect their massive activation outliers through an upstream "read-blindness," where layers continually write to extreme residual coordinates while systematically refusing to read them.

Geometric deep learning has long been fragmented by bespoke, manifold-specific architectures, but a single unified formulation can now systematically derive valid linear and convolutional layers across ten distinct Riemannian geometries.

Linear attention models do not need larger hidden states to fix their memory bottleneck; simply assigning attention heads to distinct temporal resolutions unlocks up to an 18.9% boost on associative recall.

Standard early-exit architectures collapse completely outside their pre-selected exits, but stochastic prefix supervision transforms a normal Transformer into a valid, deployable language model at literally every intermediate layer depth.

Monolithic n-gram memory tables waste capacity and fail on polysemous context, but decomposing lookups into a shared basis dictionary with per-component gating unlocks scalable, context-aware parameter expansion.

Autonomous LLM agents guided by early learning-curve forecasting can discover generalist EEG architectures that outperform state-of-the-art foundation models across 14 diverse clinical and affective benchmarks.

Standard hybrid architectures order their layers backwards for multilingual data: placing a full-attention layer first triggers immediate cross-lingual representation alignment and accelerates distillation by up to 2.5X.

Nonlinear recurrent dynamics no longer require slow, iterative linearizations to parallelize: input-conditioned Riccati flows compose exactly via $2 \times 2$ Möbius matrix multiplication in a single associative scan.

Key contribution not extracted.

Standard feedforward networks require just $d+1$ total hidden neurons to approximate $d$-dimensional continuous functions to arbitrary precision, settling the exact theoretical minimum while maintaining near-optimal bit complexity.

Graph Transformer attention doesn't degrade into unstructured noise at scale: learned interaction matrices converge to continuous, predictable graphons that transfer directly to larger graphs.

Reading barely 20% of the KV cache is enough to match up to 99% of dense attention accuracy at 32K context, yielding an immediate 1.69× speedup over FlashAttention without any retraining.

Linear-complexity state-space modeling eliminates the trade-off between receptive field size and inference speed in physiological signal denoising, yielding performance advantages that actually widen on longer sequences and context-dependent ST/T pathologies.

Dynamic MoE load balancing can run at every single microbatch with zero CPU synchronization by having GPUs deterministically solve bitwise-identical expert replication plans locally, boosting cluster throughput by up to 11.4%.

Graph-level autoencoding no longer requires intractable node-matching or permutation-invariance hacks: casting graphs as hyperball clouds provides a canonical ordering that makes standard Transformers efficient, invertible graph compressors.

Long-horizon in-context RL no longer requires exploding context windows—compressing past learning trajectories into a fixed-size latent buffer matches full Algorithm Distillation performance at a fraction of the memory cost.

Sub-quadratic attention no longer requires sacrificing softmax-level expressivity: additive sorting kernels achieve infinite identity-approximation capacity while scaling quasi-linearly on long sequences.

Transformers modeling distribution-to-distribution maps hit a hard expressivity barrier on discrete atomic inputs, but achieve universal approximation in Wasserstein distance the moment input distributions satisfy a continuous level-set condition.

Pre-trained Softmax ViTs can directly bootstrap linear-attention models without expensive pre-training from scratch: simply copy the operator-agnostic MLP weights and distill the attention routing behavior.

A novel training strategy allows video segmentation models to learn from hours-long sequences, overcoming memory and gradient limitations to achieve 10x speedup over SOTA.

Control accent strength in zero-shot TTS with a single model and inference-time weight, outperforming cascaded systems on out-of-domain accents without sacrificing speaker identity.

Agents can achieve up to 5.8% higher success rates by learning a reusable "experience tree" directly from data, bypassing LLM-generated skills and enabling true generalization.