Search papers, labs, and topics across Lattice.
The paper introduces PowLU, a novel activation function designed to improve the stability of large language model (LLM) pre-training by addressing numerical instability issues associated with SwiGLU's quadratic-like behavior for large positive inputs. PowLU uses a rational power function to achieve adaptive nonlinearity, thereby controlling the output range and mitigating outlier amplification. Experiments on the Ling architecture (7.9B and 124B parameters) demonstrate that PowLU achieves competitive performance against SwiGLU and SwiGLU-Clip, while also improving training scalability.
Ditch SwiGLU's quadratic instability: PowLU offers a rational power function that stabilizes LLM pre-training without sacrificing performance.
In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. For large positive inputs, SwiGLU approximates the quadratic function $x^2$, providing strong nonlinearity and expressive capacity. However, this property also causes numerical instability as the input or model scale increases, particularly in low-precision LLM training. The main reason is its approximate quadratic amplification, which enlarges the output range and exacerbates outliers. To address this issue, we propose a stable activation function, Power Linear Unit (PowLU), for large-scale LLM pre-training. Specifically, PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Moreover, we provide theoretical justification for several key properties of PowLU. Scaling law experiments confirm that the performance is consistent across model sizes, and further experimental results with the Ling architecture (7.9B and 124B total parameters) demonstrate that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip in large-scale training of LLMs. In addition, the experimental results also show that PowLU effectively improves the scalability of the large-scale training of LLMs.
