Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

Prematurely abandoned in modern scaling recipes, layer dropout can actually slash LLM pre-training compute by 25% while natively unlocking 1.5x faster inference through zero-shot layer skipping.

Neural rendering no longer requires per-scene fitting or specialized shaders to capture complex transport effects like caustics and participating media across arbitrary geometries.

VLA-Precision, an efficient real-world online RL framework featuring the Asymmetric Co-Bootstrapping (ACoB) algorithm and the ACoB-Stream architecture, and ACoB-Stream, a closed-loop experience--policy architecture that establishes invariant-state decoupling and on-demand streaming as design principles, deliver up to 10.9% improvements in throughput and computational efficiency.

A single layer of attention can only compute \( k \)-bit parity with \( k \) heads, revealing fundamental limits on attention mechanisms in neural architectures.

A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

Prematurely abandoned in modern scaling recipes, layer dropout can actually slash LLM pre-training compute by 25% while natively unlocking 1.5x faster inference through zero-shot layer skipping.

Neural rendering no longer requires per-scene fitting or specialized shaders to capture complex transport effects like caustics and participating media across arbitrary geometries.

VLA-Precision, an efficient real-world online RL framework featuring the Asymmetric Co-Bootstrapping (ACoB) algorithm and the ACoB-Stream architecture, and ACoB-Stream, a closed-loop experience--policy architecture that establishes invariant-state decoupling and on-demand streaming as design principles, deliver up to 10.9% improvements in throughput and computational efficiency.

A single layer of attention can only compute \( k \)-bit parity with \( k \) heads, revealing fundamental limits on attention mechanisms in neural architectures.

Attention parameterization can trap models in uninformative states, but symmetry-breaking mechanisms can dramatically enhance recovery efficiency.

Correlated weight initialization can fundamentally alter the asymptotic behavior of deep residual networks, revealing critical hyperparameters that influence performance.

LevelSyn achieves a remarkable 6.89% power reduction and 27.48% timing improvement by integrating physical awareness into logic synthesis, revolutionizing design closure efficiency.

Loop Memory Attention enables models to revisit earlier computations, leading to a 2.2% accuracy boost in MathQA tasks compared to fixed loop depths.

The proposed geometric framework reveals that convolutional filter subspaces can be parametrized as smooth projective varieties, offering new insights into filter redundancy and low-rank structures.

LLMs can achieve over 80% accuracy in extracting architectural insights from code commits, but often miss the critical rationale behind design choices.

Closed-architecture audio processors may be powerful, but they obscure the rich narratives and complexities of their internal processes, risking a loss of artistic and technical insight.

Lngram v2 achieves substantial memory efficiency gains while preserving semantic integrity, allowing for scalable transformer architectures without the typical parameter bloat.

Trust in AI-generated cryptographic artifacts can be achieved without relying on the authorship of the AI, as demonstrated by a zero-failure rate in defect detection.

NACRE achieves secure containerization without sacrificing performance, maintaining syscall metrics within 3.5% of the baseline while enhancing confidentiality.

Bridging cognitive theory and ergonomic design, this framework enhances decision-making and adaptability in human-machine interactions, crucial for high-stakes environments.

Prospective coding in deep recurrent networks can boost performance and efficiency, achieving state-of-the-art results with fewer parameters than conventional methods.

Achieving over 2x throughput improvements in attention mechanisms could redefine efficiency benchmarks for large-scale model training.

Sparse Top-K routing can optimize expert specialization while controlling computational costs, reshaping our understanding of MoE architectures.

Task-adaptive pruning can cut model size by nearly 25% while boosting accuracy in histopathology tasks, challenging the notion that bigger always means better.

The admission gate in cache prefetching outperforms the predictor, revealing that better proxies do not guarantee improved system performance.

Semantic-Aware Subgraph State Space Model achieves unprecedented accuracy in histopathology classification by preserving the spatial organization of tissue structures.

Recurrent linear attention does not compound quantization drift over long contexts—delta-rule updates and non-linear gates actively overwrite and compress noise, allowing an entire 27B hybrid model to run in NVFP4 W4A4 with near-zero quality loss up to 64K tokens.

Auxiliary draft models are no longer necessary for speculative decoding: distilling lightweight diffusion heads directly into standard LLMs yields lossless 3× generation speedups even at peak batch sizes.

Mutation analysis as an adequacy metric for kernel-benchmark oracles is introduced: deterministic rules inject 10 compilable faults into verified CUDA implementations of 188 KernelBench problems, 7{,}384 of them with an independent kill witness; any test protocol is scored by the fraction it detects.

Language models can now self-select relevant context, slashing attention costs by over 50% while maintaining performance.

Retrieving as few as 4 out of 4,096 tokens per head via dynamic pointer chasing is enough to match—and even marginally beat—dense Transformer pretraining loss.

Training stability in Transformers can be achieved by constraining residual connections to orthogonal transformations, preventing the loss of diversity across layers.

Routing decisions in sparse MoE models are governed by a shared geometric structure that enhances predictive accuracy across layers, challenging the notion of independence in expert selection.

Sparsifying communication topologies in LLM multi-agent systems can lead to significant efficiency gains without sacrificing accuracy, challenging conventional design approaches.

XMerge outperforms existing depth-compression techniques, achieving top rankings in task performance while enabling aggressive layer removal without sacrificing quality.

Continuous mixture collapse in LLMs reveals that even minor distortions can lead to significant reasoning failures, challenging the reliability of latent-state methods.

TAME achieves a 4.0 increase in retrieval accuracy on MSR-VTT by effectively modeling temporal dynamics in video data, outpacing traditional CLIP-based approaches.

Nearly orthogonal domain gradients can still wreak havoc in MoE+LoRA setups, but a new approach effectively curbs this interference.

FastMTP speculative decoding doubles the decoding speed on low-budget GPUs while maintaining high accuracy in document parsing.

Adversarial attacks can be mitigated by up to 60% in Mixture-of-Experts models through a novel shared expert mechanism that anchors safety alignment.

RACE-AIMC cuts energy use by 69% while maintaining accuracy comparable to digital systems by intelligently selecting the best AIMC accelerator and certifying its reliability.

Neural networks prune themselves during optimization not through gradual simplification, but via abrupt percolation phase transitions where architectural symmetries force subnetworks to merge in discrete, cascading blocks.

Polish ModernBERT sets a new standard for Polish language encoders, outperforming RoBERTa with fewer parameters and better efficiency.

Routing tokens through a contrastive lens boosts expert specialization and yields up to 1.77 points improvement in zero-shot reasoning accuracy.

Looping the middle layers of MoE Transformers can save up to 18% in training FLOPs while boosting performance beyond validation loss predictions.

Achieving significant network compression without sacrificing performance, LRNBA allows for the creation of deeper and wider models under the same parameter budget.

One-layer transformers can seamlessly replicate the performance of one-nearest-neighbor classifiers in multiclass scenarios, bridging a crucial theoretical gap.

A surprising 3.98–8.00% of Transformer heads achieve near closure under composition, revealing a hidden algebraic structure that could redefine our understanding of attention mechanisms.

FPGAs could revolutionize Transformer model deployment by offering superior energy efficiency and latency compared to traditional hardware.

Optimal hyperparameter choices for supervised fine-tuning can vary dramatically with model scale and architecture, challenging conventional wisdom in AI training practices.

TREVIS is proposed, an approach for learning decision trees with respect to complex objectives, based on the exploration of the latent space of a Tree Transformer Variational Auto-Encoder (TTVAE), that replaces the discrete search space with a continuous one, enabling gradient-based optimization via a differentiable surrogate model.

Attention isn't just for transformers—it's the secret sauce behind denoising diffusion models too, enabling faster and more efficient image generation.

More heads in attention layers not only enhance parameter identification but also reveal that some biases can be entirely irrelevant to model performance.

HiLRP achieves 0.97 Pointing accuracy in class activation mapping, far surpassing the 0.55 of existing methods, revealing the critical need for unified attribution in diverse ViT architectures.

A novel Mixture-of-Experts VLM slashes document processing costs by over 80%, outperforming human annotation and existing models.

Shifting from monolithic expert selection to fine-grained path composition, PCoMoE boosts inference speed by 1.31x while enhancing accuracy by 10%.

Outperforming prior open models, Instella-MoE sets a new standard for efficiency and performance in language modeling with its innovative architecture and training methods.

Transformers may not be as efficient in token transport as optimal transport theory suggests, especially in early layers.

Agents can now switch models and environments without losing their identity or memory, revolutionizing how we think about long-lived AI systems.

Achieving Monte Carlo-level accuracy in proton dose calculations while significantly reducing computational time could revolutionize treatment planning in radiation therapy.

Achieving over 24 times better energy efficiency in LLM inference could redefine the scalability of AI applications.

Substantial behavioral diversity in SPEC CPU 2026 reveals critical scale-dependent effects that could redefine performance optimization strategies for next-gen datacenter processors.

CRISP recovers up to 28% improvement on retrieval tasks while achieving a remarkable 5.30x speedup in attention computation for long-context LLMs.

The Hydration Proxy Pattern could revolutionize how enterprises manage conversational state, ensuring data sovereignty while enhancing scalability.

A columnar relational engine can outperform traditional graph databases by two-to-four orders of magnitude, challenging the need for specialized graph engines in enterprise workloads.

Photonic interconnects can deliver up to 5.8x latency improvements for large-scale AI inference, revolutionizing how we handle complex models.

AI design agents can now directly discover and reuse analog circuit designs, significantly enhancing the efficiency of circuit development and defect detection.

Achieving a 15.1% accuracy boost in neural network training while enhancing energy efficiency by over 33 times could redefine on-chip training paradigms.

FALCON achieves robust fault tolerance in edge AI by seamlessly integrating in-memory computing with stochastic techniques, ensuring reliable performance even in the face of significant noise and variations.

AInfer-PD slashes rollout completion times by up to 35.3% in distributed MoE environments, revolutionizing the efficiency of reinforcement learning inference.

Optimizing satellite learning architectures reveals that tailored federated learning can drastically improve wildfire detection across varying orbital altitudes.

FLaG achieves state-of-the-art performance by harnessing the power of frequency-domain representations, revealing how low- and high-frequency sensitivities can be strategically utilized across various tasks.

Achieving nearly 4x energy savings and over 4x latency improvements for LLMs on edge devices without sacrificing performance opens new avenues for deploying advanced AI in real-world applications.

A single-tower speech codec can outperform complex dual-tower architectures, achieving low-bitrate efficiency without sacrificing semantic quality.

A hybrid CNN-BiLSTM model with intermediate layers achieves unprecedented accuracy in State of Health estimation for battery systems.

Silent data corruption can derail LLM training, but TrainSDC offers a targeted solution that maintains performance even under fault conditions.

Achieving superior performance with one-third the resources, Qwen3.8-Flash-Next redefines efficiency in large-scale language models.

DynaNDE accelerates MoE inference by leveraging dynamic scheduling and expert reuse, resulting in up to 2.6x throughput improvements.

U-PAST achieves state-of-the-art speech enhancement performance with a fraction of the parameters, outperforming larger models in challenging acoustic conditions.

Allocating synaptic weights based on neuron firing rates can slash power consumption by over 60% in Spiking Neural Networks without significant area costs.

A fixed-dimensional latent space can achieve vanishing approximation error in neural networks, challenging conventional beliefs about the necessity of high-dimensional representations.

TMB-based neural network mixed-effects models eliminate the burdensome manual derivations, streamlining complex modeling while enhancing statistical accuracy.

Achieving perfect length generalization with a transformer model of just 280 parameters could revolutionize how we approach algorithmic tasks in neural networks.

Learning beamformers from sub-6 GHz CSI can outperform conventional methods relying on full mmWave channel information, revolutionizing beamforming strategies in massive MIMO systems.

Q-Strata outperforms existing quantization methods by directly optimizing inter-block dependencies, leading to superior model performance with lower perplexity.

Sparse, quantized linear-attention models can outperform dense counterparts on neuromorphic hardware, achieving up to 37× higher throughput and 16× lower power consumption.

Fixed equivariant architectures for multi-head self-attention can lead to severe expressivity loss, with an explosion of irreducible components limiting their effectiveness.

Turing-20B-A2B achieves exceptional long-context capabilities with only a fraction of its parameters activated, pushing the boundaries of efficiency in physical AI applications.

Traditional video world models falter in tracking hidden states beyond a limited training horizon, but two innovative mechanisms show promise for effective state retention and extrapolation.

Hand-written behavioral clock gating fails at gate level, while tool-inserted ICG cells achieve robust power savings across all simulation corners.

CHIPSMORE accelerates LLM inference by achieving 2.38x higher throughput and 27x better energy efficiency compared to Nvidia H100, without duplicating weights for concurrent requests.

Achieving competitive performance in full-key side-channel attacks on uncropped datasets with a simple transformer model could revolutionize the field by making advanced techniques more accessible.

Achieving a 95.5% pass rate on hardware design benchmarks, this framework transforms how LLMs can tackle complex hardware challenges.

VARA slashes energy consumption by over 60% while maintaining accuracy, revolutionizing how we utilize activation sparsity in ReRAM architectures.

Beacon slashes design costs by up to 93.5% in multi-chiplet accelerators by transforming detailed execution reports into actionable insights through LLM-driven agents.

PRIME reveals that preserving the original Dense path while adding conditional residuals can significantly enhance CTR model performance without sacrificing stability.

Achieving up to 42% power reduction in high-speed adiabatic logic circuits could redefine energy efficiency benchmarks in advanced chip design.