Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

A quadtree visual tokenizer slashes token count for autoregressive image generation by adaptively focusing representational power on intricate image regions.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Global point set alignment is solved exactly by finding the farthest vertex of a novel "permutation polygon," resolving a long-standing open problem in assignment geometry.

Multiplicative composition of Fourier-Chebyshev features unlocks richer PDE solution representations than implicit MLP composition.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

A quadtree visual tokenizer slashes token count for autoregressive image generation by adaptively focusing representational power on intricate image regions.

Decoupling denoising and context writing parameters in autoregressive video models unlocks unprecedented long-horizon generation capabilities, extending continuous output to 24 hours from just 5-second training rollouts.

Global point set alignment is solved exactly by finding the farthest vertex of a novel "permutation polygon," resolving a long-standing open problem in assignment geometry.

Quantum architecture search can now be 40% cheaper on real hardware by reducing circuit measurements without sacrificing optimization quality.

Evolutionary search discovers a 26x smaller, yet more accurate, model for satellite-based chlorophyll prediction than expert-designed alternatives.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Derivative GPs can now achieve state-of-the-art accuracy with less computation and memory than function-only baselines by cleverly budgeting gradient information.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Physics-aligned electronic ground-state models shatter out-of-distribution generalization barriers for MLIPs, achieving near-chemical accuracy on reactive systems with minimal data.

Model selection for scientific forecasting is fundamentally protocol-dependent, with optimal choices reversing based on dataset size, system complexity, and evaluation metrics like state refresh versus closed-loop performance.

Mamba's linear-time state-space models unlock superior multi-scale financial volatility forecasting, outperforming Transformers by capturing regime shifts and enabling longer context.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

Negative curvature can amplify PWGF approximation errors, but subsequent positive curvature can recover accuracy over growing time horizons, enabling stable optimization in non-convex landscapes.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

A novel bias expansion for two-timescale stochastic approximation reveals a $\varepsilon^2/\eta$ term that fundamentally alters extrapolation strategies.

Axis-aligned splits in BART are a bottleneck for non-axis-aligned functions; RoBART achieves adaptive posterior contraction by rotating each tree's coordinate system.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Bayesian inference for complex switching stochastic differential equations is now broadly tractable, even with non-analytic transition densities.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Small language models achieve high abstract reasoning scores by memorizing task distributions rather than learning generalizable rules, failing even simple out-of-distribution shifts.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Discovering task-specific, compact concept vocabularies from unsupervised pools unlocks inspectable AI without the need for manual annotations or massive, static concept sets.

Exploiting cyclic symmetries in multi-index models offers a polynomial sample complexity advantage, with data augmentation proving as sample-efficient as architectural weight sharing.

Divergence choice in Shampoo-like optimizers fundamentally alters Kronecker approximation and finite-sample error compensation, explaining observed training behavior.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Quantum circuits can unlock compositional generalization in multimodal AI, even on noisy hardware.

Meta-learning unlocks efficient, resolution-independent PDE surrogate modeling for time-dependent systems by treating initial conditions as context, structuring latent spaces, and accelerating inference.

Encoding sensor geometry as a flow-matching prior dramatically improves generative models for brain signals, outperforming isotropic noise and even empirical data covariance.

Gradient-free few-shot adaptation emerges from decentralized cellular dynamics, achieving state-of-the-art OOD transfer by updating a spatial program through local neighborhood interactions.

Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.

Unlocking broad downstream generalization for 3D atomistic systems, Atom-JEPA achieves state-of-the-art property prediction by learning from unlabeled structures alone.

Generate valid and unique discrete designs in a single forward pass with over $50\times$ the yield per operation compared to prior one-step methods.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Sparse MoE models match dense models' code repair correctness with drastically fewer active parameters, challenging the sole reliance on total parameter count for evaluating code generation efficiency.

End-to-end meta-learning of neural field representations reveals that first-order approximations discard crucial learning pathways, leading to improved few-shot adaptation and semantic prediction.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

Generative flow matching directly refines 4D hand-object interactions during synthesis, integrating physical constraints and 2D evidence without post-hoc optimization.

Chemical reaction models can now strictly enforce valence conservation and enabling rules without any training, achieving superior accuracy on reaction prediction tasks.

MLLMs can now natively weave elastic visual representations, adaptively allocating detail to save 43% of tokens while preserving near-perfect performance.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Generating molecular graphs in a pretrained VAE's latent space via flow matching unlocks significant efficiency gains and enables property-guided generation without retraining.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Wrist accelerometers can now yield population-scale sleep heart rate insights, with a novel transformer model outperforming existing methods in estimation availability.

Neural PDE solvers can miss optimal solutions due to unbounded parameters, but adding kernel derivatives recovers these "missing limits."

Hierarchical MoE routing with a novel arrival-weighted mean estimation prevents expert collapse and enables localized co-activation, outperforming flat routing on large embedding models.

A novel two-stream architecture disentangles temporal context from spatial refinement in EEG source imaging, achieving superior localization accuracy without compromising inference speed.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

Sparse MoE routers don't achieve adaptive expert selection through their maps alone; they co-evolve learned score distributions that critically determine token-wise expert participation.

Large HEP datasets are not just for training; they are foundational for unlocking power-law scaling, with smaller datasets offering little insight into high-compute performance.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Binary sources allow for strictly smaller optimal representations in the Information Bottleneck than previously thought.

LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.

Hyper-connections create a fading memory in Transformers, with the effective context horizon directly tied to the second singular value of the stream-mixing matrices.

Generative auctions can now achieve externality-aware whole-feed optimization with exact strategyproofness using a transformer, overcoming a critical deployment barrier.

Single-neuron models can now capture multimodal voltage responses, including coexisting firing/non-firing states and depolarization block modes, a feat previously requiring massive ensembles or failing with neural operators.

Amortized spike-and-slab inference for linear models with interactions achieves state-of-the-art variable selection accuracy with a single forward pass, eliminating costly per-dataset MCMC.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Frequency-specific conditional relationships in multivariate time series are unlocked by using inverse cross-spectral density matrices as graph shift operators.

Explicitly modeling the temporal rate of change in spatio-temporal graph models unlocks superior prediction accuracy and pattern preservation for propagating phenomena.

A single space token transforms Transformers from arithmetic failures to 89% accurate multipliers by enabling learnable carry propagation.

Autoregressive latent priors learned directly from data, not fixed Gaussians, unlock VAEs for high-fidelity sequential generation with diffusion-level quality and faster inference.

Transformer attention applied to geometry-aware segment tokens unlocks 2459x speedup for electromigration stress prediction with sub-8% error.

Pretrained time-series foundation models offer surprisingly inconsistent benefits for EMG tasks, with full fine-tuning often outperforming linear probing and few-shot adaptation only partially recovering performance on new users.

Equivariant potentials can achieve transformer-level accuracy with 35x fewer parameters and 10x faster inference by leveraging Cartesian tensor products.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Medical LLMs can achieve superior QA performance by dynamically selecting specialized adaptation components based on question context and clinical tags, surpassing static parameter-efficient methods.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Wi-Fi sensing accuracy jumps to 92.83% by decomposing CSI into trajectory-guided tokens that capture complex temporal dynamics.

Closed-form KANs integrated into IGA dramatically slash PDE solution errors by orders of magnitude, outperforming pure neural solvers and standard IGA.

Individual token selection and metadata caching unlock sparse attention for diffusion transformers without sacrificing generation quality, achieving up to $2.32\times$ speedups.

Routing robustness, not just load balancing, is the key to reliable sparse MoE scaling, with DRMoET boosting performance by 1.4% on a 10.3B parameter model.

Unknown traffic objects are detected with 2.6% higher recall than RT-DETR by fusing semantic and visual prompts symmetrically, enabling real-time open-world detection without NMS.

A few critical denoising steps in diffusion language models dictate the entire output, and training only these "pivots" dramatically boosts performance.

Adapters forget as much as they learn: a simple post-training normalization unlocks significant gains in both specialization and retained capabilities for LoRA.

State costs in Schrödinger bridges on graphs can be analytically incorporated via a Feynman-Kac tilt, enabling exact, non-learned solutions with linear memory growth even for massive networks.

Dynamically recomputing embeddings at each denoising step in diffusion transformers halves training compute while achieving state-of-the-art FID scores.

Molecular generation models can now achieve 100% validity and state-of-the-art distributional alignment by serializing higher-order topology into context-free grammar rules.

Topology-preserving 3D generation now achieves 98% fewer tokens and 58.5% faster inference by encoding shapes as overlapping slice latents instead of expensive voxels.

Non-convex sampling from matrix spaces is now possible with provable exponential convergence using momentum preconditioning that spectrally tames gradients.

A novel hybrid deep learning approach achieves robust few-shot malware detection by leveraging autoencoder-derived features for rapid meta-learning adaptation.

Malware analysis just got a new tool: S4 models unlock long-range dependency detection in execution flows, outperforming traditional deep learning for multi-class classification.