Yu-Liang Liu

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

NCP-ArchPreview is introduced, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP) and learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation.

Scaling Hyper-Connections beyond four streams is now feasible, yielding substantial performance gains without prohibitive costs.