Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Jointly diffusing token and cluster embeddings unlocks significant gains in continuous diffusion language models, achieving state-of-the-art perplexity and reasoning performance with minimal overhead.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Sublinear regret in linear bandits is achievable even with strict sliding-window constraints by leveraging a novel "transition diameter" and a rare-switching algorithm, outperforming naive approaches with significantly fewer policy updates.

Complex-valued neural networks, powered by Wirtinger autodifferentiation and FFT-Hadamard convolutions, match or exceed real-valued FNet performance on language modeling with half the training steps.

Small language models achieve high abstract reasoning scores by memorizing task distributions rather than learning generalizable rules, failing even simple out-of-distribution shifts.

Nonlinear recurrent memories struggle with sequential test-time training, but layerwise delta updates unlock significant gains in language modeling and retrieval.

Discovering task-specific, compact concept vocabularies from unsupervised pools unlocks inspectable AI without the need for manual annotations or massive, static concept sets.

Exploiting cyclic symmetries in multi-index models offers a polynomial sample complexity advantage, with data augmentation proving as sample-efficient as architectural weight sharing.

Divergence choice in Shampoo-like optimizers fundamentally alters Kronecker approximation and finite-sample error compensation, explaining observed training behavior.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Quantum circuits can unlock compositional generalization in multimodal AI, even on noisy hardware.

Meta-learning unlocks efficient, resolution-independent PDE surrogate modeling for time-dependent systems by treating initial conditions as context, structuring latent spaces, and accelerating inference.

Encoding sensor geometry as a flow-matching prior dramatically improves generative models for brain signals, outperforming isotropic noise and even empirical data covariance.

Gradient-free few-shot adaptation emerges from decentralized cellular dynamics, achieving state-of-the-art OOD transfer by updating a spatial program through local neighborhood interactions.

Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.

Unlocking broad downstream generalization for 3D atomistic systems, Atom-JEPA achieves state-of-the-art property prediction by learning from unlabeled structures alone.

Generate valid and unique discrete designs in a single forward pass with over $50\times$ the yield per operation compared to prior one-step methods.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Sparse MoE models match dense models' code repair correctness with drastically fewer active parameters, challenging the sole reliance on total parameter count for evaluating code generation efficiency.

End-to-end meta-learning of neural field representations reveals that first-order approximations discard crucial learning pathways, leading to improved few-shot adaptation and semantic prediction.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

Generative flow matching directly refines 4D hand-object interactions during synthesis, integrating physical constraints and 2D evidence without post-hoc optimization.

Chemical reaction models can now strictly enforce valence conservation and enabling rules without any training, achieving superior accuracy on reaction prediction tasks.

MLLMs can now natively weave elastic visual representations, adaptively allocating detail to save 43% of tokens while preserving near-perfect performance.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Explicitly learning a shared perception-action representation boosts continuous-action VLA performance by over 13% on real-world tasks.

Generating molecular graphs in a pretrained VAE's latent space via flow matching unlocks significant efficiency gains and enables property-guided generation without retraining.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Wrist accelerometers can now yield population-scale sleep heart rate insights, with a novel transformer model outperforming existing methods in estimation availability.

Neural PDE solvers can miss optimal solutions due to unbounded parameters, but adding kernel derivatives recovers these "missing limits."

Hierarchical MoE routing with a novel arrival-weighted mean estimation prevents expert collapse and enables localized co-activation, outperforming flat routing on large embedding models.

A novel two-stream architecture disentangles temporal context from spatial refinement in EEG source imaging, achieving superior localization accuracy without compromising inference speed.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

Sparse MoE routers don't achieve adaptive expert selection through their maps alone; they co-evolve learned score distributions that critically determine token-wise expert participation.

Large HEP datasets are not just for training; they are foundational for unlocking power-law scaling, with smaller datasets offering little insight into high-compute performance.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Binary sources allow for strictly smaller optimal representations in the Information Bottleneck than previously thought.

LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.

Hyper-connections create a fading memory in Transformers, with the effective context horizon directly tied to the second singular value of the stream-mixing matrices.

Generative auctions can now achieve externality-aware whole-feed optimization with exact strategyproofness using a transformer, overcoming a critical deployment barrier.

Single-neuron models can now capture multimodal voltage responses, including coexisting firing/non-firing states and depolarization block modes, a feat previously requiring massive ensembles or failing with neural operators.

Amortized spike-and-slab inference for linear models with interactions achieves state-of-the-art variable selection accuracy with a single forward pass, eliminating costly per-dataset MCMC.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Frequency-specific conditional relationships in multivariate time series are unlocked by using inverse cross-spectral density matrices as graph shift operators.

Explicitly modeling the temporal rate of change in spatio-temporal graph models unlocks superior prediction accuracy and pattern preservation for propagating phenomena.

A single space token transforms Transformers from arithmetic failures to 89% accurate multipliers by enabling learnable carry propagation.

Autoregressive latent priors learned directly from data, not fixed Gaussians, unlock VAEs for high-fidelity sequential generation with diffusion-level quality and faster inference.

Transformer attention applied to geometry-aware segment tokens unlocks 2459x speedup for electromigration stress prediction with sub-8% error.

Pretrained time-series foundation models offer surprisingly inconsistent benefits for EMG tasks, with full fine-tuning often outperforming linear probing and few-shot adaptation only partially recovering performance on new users.

Equivariant potentials can achieve transformer-level accuracy with 35x fewer parameters and 10x faster inference by leveraging Cartesian tensor products.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Medical LLMs can achieve superior QA performance by dynamically selecting specialized adaptation components based on question context and clinical tags, surpassing static parameter-efficient methods.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Wi-Fi sensing accuracy jumps to 92.83% by decomposing CSI into trajectory-guided tokens that capture complex temporal dynamics.

Closed-form KANs integrated into IGA dramatically slash PDE solution errors by orders of magnitude, outperforming pure neural solvers and standard IGA.

Individual token selection and metadata caching unlock sparse attention for diffusion transformers without sacrificing generation quality, achieving up to $2.32\times$ speedups.

Routing robustness, not just load balancing, is the key to reliable sparse MoE scaling, with DRMoET boosting performance by 1.4% on a 10.3B parameter model.

Unknown traffic objects are detected with 2.6% higher recall than RT-DETR by fusing semantic and visual prompts symmetrically, enabling real-time open-world detection without NMS.

A few critical denoising steps in diffusion language models dictate the entire output, and training only these "pivots" dramatically boosts performance.

Adapters forget as much as they learn: a simple post-training normalization unlocks significant gains in both specialization and retained capabilities for LoRA.

State costs in Schrödinger bridges on graphs can be analytically incorporated via a Feynman-Kac tilt, enabling exact, non-learned solutions with linear memory growth even for massive networks.

Dynamically recomputing embeddings at each denoising step in diffusion transformers halves training compute while achieving state-of-the-art FID scores.

Molecular generation models can now achieve 100% validity and state-of-the-art distributional alignment by serializing higher-order topology into context-free grammar rules.

Topology-preserving 3D generation now achieves 98% fewer tokens and 58.5% faster inference by encoding shapes as overlapping slice latents instead of expensive voxels.

Non-convex sampling from matrix spaces is now possible with provable exponential convergence using momentum preconditioning that spectrally tames gradients.

A novel hybrid deep learning approach achieves robust few-shot malware detection by leveraging autoencoder-derived features for rapid meta-learning adaptation.

Malware analysis just got a new tool: S4 models unlock long-range dependency detection in execution flows, outperforming traditional deep learning for multi-class classification.

Standard GNNs and Transformers can now capture complex multi-way interactions by simply adding higher-order positional encodings derived from clique complexes, boosting performance on tasks requiring rich relational understanding.

A novel Four-Dimensional Architecture Selection Framework correctly recommends 70% of deployed autonomous driving architectures, offering critical guidance for system design.

Learned attention weights actively harm in-context learning performance in Chronos-2, while simple uniform pooling provides consistent benefits.

Relational learning with tabular foundation models can now bypass feature explosion and interaction blindness by propagating information directly through the schema graph.

Kernel learners outperform feature learners for nonlinear in-context learning when task diversity is high and context length is short, but the opposite holds for low diversity and long contexts.

LLMs can learn effective dynamic embeddings for massive combinatorial optimization problems with a shallow policy by selectively propagating memory across rollout steps, bypassing the need for expensive full recomputation or external labels.

Two fixed attention blocks suffice for universal interpolation in deep residual networks, even with causal masking.

Adaptive tensor product computation slashes MLIP runtime by $2.4\times$ and memory by $2.6\times$ without sacrificing accuracy, revealing computational depth's link to chemical complexity.

Harnessing earlier recurrent states as implicit weak supervision, a training-free contrastive decoding method boosts Looped Transformer performance by up to 11.45% and enables 22.5%-48.2% FLOP reduction.

Transformer and RNN tractography models achieve state-of-the-art performance by introducing a novel generation-validation phase for streamline-level supervision, overcoming the limitations of local loss functions.

Free-boundary PDEs are solved with unprecedented accuracy and interface resolution using Kolmogorov-Arnold Networks, surpassing PINNs and residual networks.

Sampling weight updates, not just tokens, unlocks significant test-time adaptation gains for LLMs.

A data-driven weather model now rivals operational NWP for 1 km resolution forecasts over complex Alpine terrain, but struggles with localized wind extremes.

Iterative refinement of a class token with shared blocks slashes transformer parameters and computation while boosting audio recognition accuracy.

Neural dynamics aren't monolithic; a new model precisely disentangles and quantifies multiple, behavior-dependent timescales from population recordings.

Ghost nodes and physics refiners unlock stable, accurate long-horizon hydrologic forecasts even with missing upstream boundary data.

Diffusion models can now achieve one-step, efficient action generation for online multi-agent coordination, shattering previous scalability barriers.

Novel dashboards can bridge the gap between complex topological data analysis and user interpretation by explicitly acknowledging and managing methodological uncertainty.

A single denoising process now unifies discrete token generation and continuous latent trajectories, enabling diffusion models to jointly reason about token dependencies and global structure.

Looped MoE models can scale to over 10 iterations, achieving substantial performance gains by stabilizing recurrence and diversifying expert selection.

Unified LLMs can now achieve state-of-the-art 3D generation and understanding by treating geometry as an explicit sequence of sparse octree tokens, without compromising core language abilities.

Task-specific LoRA-Experts and prototype-ensemble matching unlock rehearsal-free class-incremental learning with reduced interference and improved discrimination.

Trillion-token scale looped MoE models achieve equivalent reasoning performance to 2x larger non-looped MoE models with matched training compute, unlocking test-time scaling via recurrence.

Recurrent memories can retain full Fisher information post-writing with an invertible hold, decoupling write and read optimality and achieving near-perfect task-specific performance.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.