Tuo Dai

Multi-head attention applied to network depth unlocks substantial performance gains and compute efficiency in large MoE models.

Scaling Hyper-Connections beyond four streams is now feasible, yielding substantial performance gains without prohibitive costs.