Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper extends the equivalence between one-layer transformers and nearest-neighbor classifiers from binary to multiclass settings by employing simplex encoding. The authors demonstrate that one-layer transformers with an argmax classification head function identically to a one-nearest-neighbor classifier, addressing limitations in previous research that utilized a non-standard rounding approach. This finding not only clarifies the theoretical underpinnings of one-layer transformers but also enhances their practical applicability in multiclass classification tasks.
One-layer transformers can seamlessly replicate the performance of one-nearest-neighbor classifiers in multiclass scenarios, bridging a crucial theoretical gap.
We extend recent work establishing an equivalence between one-layer transformers and nearest-neighbor classifiers in the binary setting to the multiclass case. By leveraging the simplex encoding, we show that one-layer transformers with an argmax classification head behave identically to a one-nearest-neighbor classifier in the multiclass setting. This closes a gap left by prior work, whose multiclass result relied on a non-standard rounding-based approach rather than the typical argmax head used in practice.