Architecture Design (Transformers, SSMs, MoE)

Reusable continuous-time summaries slash IMTSF inference latency by up to 87% and enable 52% faster forecasting over multiple calls, all with a fraction of the parameters.

Generative design can now produce moldable parts by transforming 3D shape generation into 2D paired depth image generation, bypassing undercuts by construction.

Recurrence's rapid learning of retrieval functions actively prevents attention from acquiring similar capabilities in hybrid architectures, necessitating staged training for optimal performance.

A novel Kronecker Delta Rule enables associative memories to revise targeted information with minimal interference, unlocking stable, long-context linear attention.

Compressing LLM embeddings by 50% without sacrificing performance is now achievable by reorganizing the feature space with a learned spectral basis.

General CPWL functions can be represented by ReLU networks with just two hidden layers, shattering previous depth conjectures.

LLMs can now achieve 66x storage reduction and 1.86x speedup at 128K context by internalizing memory, not just retrieving it.

Agents can now dynamically compose learned skills by selecting LoRA adapters as actions, overcoming catastrophic forgetting and boosting performance by up to 27.5% on complex benchmarks.

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.

Quantum Hamiltonian embeddings unlock hypothesis classes beyond classical reach by encoding datasets into parameterized Hamiltonians optimized for ground-state classification.

Lifelong generative models can now learn new tasks without forgetting old ones by directly evolving their velocity fields, bypassing expensive trajectory simulations.

Recursive pretraining with gated mixture recurrence unlocks substantial parameter efficiency and adaptive computation for protein language models, outperforming larger models on key structural prediction tasks.

Polynomial gradient orthogonalization achieves 15-18% faster runtimes than state-of-the-art methods by leveraging optimized polynomial circuits.

Gaussian process additive models can now precisely control the prior explained variance of each component, overcoming the inherent variance overestimation of standard Matérn priors.

Near-autoregressive quality is achievable with parallel decoding, unlocking significant inference speedups across model scales.

Neural networks for point clouds require quadratic representation orders to guarantee separation, a concrete bound finally established.

Nested recursive reasoning architectures fail to consistently improve ARC-AGI-1 performance over baselines at matched parameter counts, even with a single-GPU budget.

Continuum robot design can now be optimized with 5x less data and 30-50% higher accuracy, while also quantifying prediction uncertainty.

Few-shot prompting with historical snapshots enables graph foundation models to generalize contingency analysis to unseen power system topologies, overcoming limitations of pure regression approaches.

Explicitly conditioning sleep representations on global time-of-sleep dramatically boosts performance on challenging pediatric sleep tasks, even across different datasets and models.

A single foundation model can now decode plant stress signals across species and environments, unlocking unprecedented scalability for agricultural and ecological monitoring.

Adaptive refinement guided by collocation residuals in hybrid PINNs slashes solution errors for singular PDEs by up to 38x, outperforming uniform refinement.

Learned expand-and-sparsify representations with ReLU activation match dense layers at significantly reduced sparsity, and binarization costs learned sparse codes nothing.

Simple logit averaging during training, not complex attention mechanisms, unlocks state-of-the-art performance in non-invasive speech BCIs, outperforming prior winners by a significant margin.

A Transformer that treats coupon discount dimensions as queries unlocks 1.26% higher business value per user by effectively modeling cross-category demand shifts.

Adapting MoE experts directly, not through bypass adapters, unlocks the best downstream performance with minimal general capability loss.

Sparse attention in video diffusion models can be corrected using cached dense refinements, recovering fidelity without altering the reuse schedule.

Fine-scale automotive flow prediction can now leverage incremental states passed cross-layer, achieving superior accuracy by continuously integrating local details into global physics.

Iterative GNNs trained on solver trajectories outperform single-pass SOCP solvers and enable warm-starting classical optimizers.

Target Propagation's connection to Gauss-Newton optimization is established under a new implicit norm, with CGTP achieving KFAC-like performance using orders of magnitude fewer outer iterations.

A multimodal spatiotemporal transformer system achieves near-perfect detection of Pyropia bleaching seven days in advance, but its efficacy is critically dependent on the recency of nutrient survey data.

Self-supervised GNNs can directly learn distortion-aware precoders for satellite systems, outperforming traditional optimization methods in speed and operating range while maintaining QoS.

LLMs can now "think with visual latents" by generating masks directly from their internal representations, dramatically improving referring comprehension and segmentation.

A specific looped Transformer architecture and input injection strategy unlock significant gains in reasoning and context utilization, outperforming larger models with drastically fewer parameters.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Generative time series models can now produce multiple plausible future trajectories as fast as quantile models query a single point, unlocking accurate dependency modeling for critical applications.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

Dual-view time-series representations unlock significant gains in temporal reasoning by preserving both absolute measurements and invariant temporal structure, outperforming dedicated TSLMs.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

Exponential Hopfield networks learn exponentially more archetypes than pairwise models by leveraging softmax attention, but risk memorizing noisy examples over true archetypes.

A simple byte reordering strategy slashes multilingual boundary prediction errors by over 70% in scripts like Korean and Chinese, fundamentally altering how byte-level models perceive codepoint transitions.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

A single weight-space encoder now masters 100 transformer architectures, outperforming specialized models and handling unseen, larger networks with less memory.

A single INR can now model complex, periodic deformations across entire image sequences, enabling self-regularization and outperforming prior methods.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

A closed-form depth-width law for neural collapse reveals how representation compression scales with the L/N ratio in deep linear networks.

Single-qubit quantum circuits, despite universal function approximation, get stuck in training due to "phase-locking," requiring orders of magnitude more optimization steps than multi-qubit equivalents.

Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.