Search papers, labs, and topics across Lattice.
This study investigates the integration of lightweight depthwise convolutions into the Qwen3 Transformer architecture to enhance local inductive bias in large language models (LLMs) without significantly increasing model size. By applying convolution to the projected queries, keys, and values prior to attention, the authors achieve improved accuracy across seven downstream benchmarks while maintaining a negligible parameter increase. The findings indicate that this convolutional approach enhances the sensitivity of repeated token IDs to their immediate context, thereby optimizing short-range token interactions within LLMs.
Adding depthwise convolutions to Transformers can boost accuracy on downstream tasks while barely increasing model size.
Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size. Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation. Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01\%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.
Tsinghua AI