Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Achieving a 2.37% boost in Union accuracy over the best baseline, Robust CurveMoE reveals a new paradigm for balancing adversarial defenses across multiple norm constraints.

Pointwise convolutions, which dominate parameter volume in large-kernel CNNs, can be drastically reduced through a novel group-sharing strategy, enabling efficient deployment on edge devices.

The required rank for effective low-rank adaptation in Transformer attention can be much smaller than expected, especially under softmax saturation conditions.

Geometry-constrained KANs not only outperform traditional fixed-basis models in symbolic regression but also adaptively learn their geometric properties, revealing deeper insights into the data structure.

EXAONE Tabular outperforms tuned ensembles and larger models, achieving top rankings in multiple benchmarks while drastically reducing inference costs.

Achieving a 2.37% boost in Union accuracy over the best baseline, Robust CurveMoE reveals a new paradigm for balancing adversarial defenses across multiple norm constraints.

Pointwise convolutions, which dominate parameter volume in large-kernel CNNs, can be drastically reduced through a novel group-sharing strategy, enabling efficient deployment on edge devices.

The required rank for effective low-rank adaptation in Transformer attention can be much smaller than expected, especially under softmax saturation conditions.

Geometry-constrained KANs not only outperform traditional fixed-basis models in symbolic regression but also adaptively learn their geometric properties, revealing deeper insights into the data structure.

EXAONE Tabular outperforms tuned ensembles and larger models, achieving top rankings in multiple benchmarks while drastically reducing inference costs.

Foresight pruning can significantly enhance the performance of sparse PINN solvers by focusing on the sensitivity of PDE residuals rather than just output dynamics.

Achieving state-of-the-art performance with 72% fewer parameters, CrossMambaTuning redefines efficiency in adapting image compression models for machine vision.

Syn2Logic achieves groundbreaking speed and efficiency in neuromorphic computing without requiring any hardware description language coding.

Flipping just a few bits can inflate LLM output by over 5900%, revealing a critical vulnerability in Mixture-of-Experts architectures.

Achieving a mean AUC of 0.8750, this hierarchical MoE model outperforms traditional methods by effectively integrating imaging and EHR data for ILD classification.

DESCENT achieves unprecedented accuracy in airport surface movement prediction by leveraging adaptive context sampling to navigate complex operational environments.

A stealthy backdoor attack exploits batch-dependent behaviors in Vision MoE, remaining dormant during audits but activating with alarming success during deployment.

Maximizing $L_p$-norms over zonotopes is W[1]-hard for all fixed rational $p \in (1, \infty)$, revealing deep computational challenges in neural network sensitivity analysis.

Maia 200 achieves unprecedented AI acceleration while slashing energy costs, redefining the landscape for high-performance computing.

Tighter verification bounds for neural networks could finally enable safe deployment in critical systems, challenging the limitations of current relaxation methods.

Achieving rapid earthquake magnitude estimation with a single station, SeisMamba outperforms traditional models in both speed and accuracy, even in unseen regions.

Ternary multiplicative adaptation recovers lost performance in quantized models while maintaining extreme efficiency, outperforming traditional low-bit methods.

Heterogeneous expert families can significantly boost interpretability and predictive performance in machine learning models, adapting to local data structures more effectively than traditional homogeneous approaches.

A causal streaming anomaly detector achieves real-time performance on edge hardware while revealing that detection responsiveness is governed more by an event boundary gate than by the base decay rate.

ALPHABET achieves Bayes oracle performance with a mere 6,437 parameters, revolutionizing efficiency in sequence modeling.

Achieving 90% accuracy in optical card reading without manual intervention could redefine the capabilities of physical Turing Machines.

Generative AI in architecture isn't just about design—it's a complex interplay of natural language, programming languages, and annotations that can redefine how we conceptualize space.

Formally-verified NoCs can now be generated effortlessly, eliminating the tedious verification process while ensuring strong liveness guarantees.

MoE-based feature adaptation can dramatically enhance the accuracy of coronary artery segmentation in challenging X-ray angiography scenarios.

Task-structured routing in MoTE allows for interpretable and efficient multi-task video-language learning, outperforming traditional dense activation methods.

Retaining the largest attention weights in KV-cache eviction is nearly optimal, challenging the perceived complexity of selection strategies.

MoE models may encode moral content robustly, but they are surprisingly fragile, collapsing under noise that dense models easily withstand.

Mamba-based models can rival state-of-the-art transformers in fake news detection while slashing resource demands by nearly half.

HSM design choices can significantly impact the security and performance of automotive systems, revealing critical trade-offs that could reshape industry standards.

Despite diverging philosophies, leading agent harnesses are converging on a shared architecture, highlighting a critical gap in external verifiability for trust in AI systems.

Depth pruning doesn't have to mean sacrificing accuracy—SHIFT-LLM recovers lost performance with minimal overhead.

GaussVLA achieves a 19.7% improvement in spatial manipulation success rates while being more parameter-efficient than previous models.

A new open-source benchmark suite for 3D-ICs could revolutionize how researchers tackle the complexities of heterogeneous integration technologies.

Eliminating thermal-induced tuning stalls in optical interconnects can boost MoE model performance by up to 3.8x, unlocking new potential for large-scale AI systems.

Degradation-aware image restoration can boost bridge damage detection accuracy in low-light conditions by nearly 60%.

ProxyFormer achieves a staggering 0.7 million token context length while retaining up to 95% retrieval accuracy, revolutionizing the scalability of generative models.

Sigmoid attention, while less effective in dense modeling, dramatically improves KV-cache eviction performance, challenging conventional wisdom about attention mechanisms.

Routing-induced bias in Mixture-of-Experts models can be corrected to improve fairness without sacrificing predictive accuracy.

By integrating machine learning with traditional simulation methods, this framework boosts resonant antenna designs from 40% to 52%, showcasing a leap in design efficiency.

Achieving a mean error of just 0.41 mmHg for diastolic BP, this framework sets a new standard for cuffless blood pressure estimation.

HAWKEYE nearly doubles the baseline performance in temporal link prediction by leveraging deeper structural insights that traditional methods overlook.

CNNs outperform larger models in real-time plasma equilibrium prediction, achieving a remarkable balance of speed and accuracy essential for tokamak control.

Achieving up to a 72.81× speedup in long-sequence inference could redefine efficiency standards for Diffusion Language Models.

Motion-Guided Mamba redefines video frame interpolation by aligning feature propagation with dynamic motion trajectories, achieving unprecedented accuracy in complex scenarios.

Dynamic integration of multimodal knowledge boosts the recognition of rare surgical actions, overcoming critical optimization conflicts in robot-assisted surgery.

VIPER slashes design-space exploration time from hours to under a minute while achieving less than 10% error in performance predictions for Processing-in-Memory systems.

Traditional attention-mask checks miss critical causality violations, while our new audit method pinpointed 100% of failures across multiple models.

Achieving 25% higher throughput than dense models, Giga-Embeddings redefines efficiency in high-quality text embedding generation.

Content-sensitive attention constraints in transformers can dramatically enhance alignment with human reading times, revealing critical limitations in existing models.

Achieving lower latency in secure IoT data aggregation, Phi-PHE-BC redefines the performance landscape for homomorphic blockchain architectures.

Channel selection in convolutional networks can dramatically reduce computational costs while maintaining or exceeding performance benchmarks.

Disabling thread affinity can significantly improve the scaling of parallel applications with work imbalance on hybrid CPU architectures.

Achieving a 25% footprint reduction while enhancing performance metrics positions this SRAM design as a game-changer for future memory technologies at the 2nm node.

SYNTLOG achieves up to 66% fewer LUTs and synthesizes massive circuits in minutes, while Vivado fails to finish after hours.

Achieving a 2.00x reduction in weight bandwidth while decoding at 5.94 tokens/s could redefine efficiency benchmarks for autoregressive models on CPU architectures.

Achieving a 99% correlation with real silicon, this simulation framework reveals crucial insights into the architectural evolution of GPUs for AI workloads.

NOVA's innovative architecture delivers 4.5x higher throughput and 69.8% lower latency for hybrid LLMs, pushing the boundaries of memory processing efficiency.

Targeted approximation in floating point multipliers can yield up to 92% hardware footprint savings without sacrificing CNN accuracy.

Routing-guided exploration boosts software fix resolution rates by nearly 8%, outperforming traditional sampling methods without needing explicit answer forms.