Architecture Design (Transformers, SSMs, MoE) - Weekly Roundup

Synthetic-to-real pretraining bridges the chronic scarcity of longitudinal clinical imaging, enabling transformers to reliably forecast up to 15 years of spinal disease progression.

Global building layers show stark, resolution-dependent divergence: Overture dominates vector accuracy (0.786 F1), but raster rankings invert between 10m and 100m resolutions while temporal drift silently suppresses benchmark scores by up to 0.060 F1 in expanding cities.

Fysiverse-3D-Vision is proposed, a unified vision-language-geometry framework for generative 3D scene reconstruction and executable asset construction from a single image that establishes a shared representation where spatial reasoning and geometric reconstruction mutually enhance each other, allowing object layouts to be inferred beyond the constraints of individual asset generators.

It is found that expert selection is far more redundant than the field's operating points assume, and how much expert computation fine-grained MoEs can dispense with is revealed, and when dynamic allocation earns its complexity is established, informing both practical deployment and future pruning methods.

Evaluations across seven tasks and three Qwen scales show higher mean accuracy than RandOpt on Countdown, GSM8K, and MATH-500 at every scale, and perturbation geometry is highlighted as a key design choice for population-efficient, gradient-free search around pretrained models.

An upper bound on squared Hellinger risk for looped sieve maximum likelihood and a minimax lower bound over the tuned untied family are established, revealing a parameter--iteration--accuracy tradeoff: repeated computation can improve approximation without adding parameters, while increasing computational cost and fitted-class complexity.

The Adaptive Decomposition Network (ADNet), a component-specific forecasting framework that adaptively disentangles traffic dynamics into dominant and residual components, is proposed, and the effectiveness of adaptive decomposition and component-specific modeling for multi-step traffic forecasting is demonstrated.

A sharp width--magnitude tradeoff at fixed depth is established using one elementary bounded $1$-Lipschitz Dyadic--Triangular Activation and yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size.

Recurrent token mixing via GTR achieves 58.9 AP on COCO with sub-2ms latency, offering a practical, efficient alternative to quadratic self-attention for high-resolution dense prediction.

Benchmarking saliency map faithfulness is so sensitive to perturbation design and class-conditioning that minor evaluation choices can completely flip which explainability method appears most trustworthy in medical ViTs.

A leakage-aware protocol with vessel-, time- and region-disjoint splits is built and applied to two corpora with different traffic: 31 days of Danish national AIS traffic and 30 days of US Gulf coast traffic off Houston and Galveston.

A benchmark study of six TSFMs evaluated on energy, traffic, and financial datasets reveals that while TSFMs outperform statistical methods and supervised models, they are subject to a fundamental trade-off between point accuracy and probabilistic reliability.

It is proved that LEMA transformers with chain of thought can simulate word-RAMs, as was recently shown for the less restrictive rightmost hard attention, and dictionary-based inference is implemented for LEMA transformers with dictionaries residing in main memory rather than VRAM.