Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper demonstrates that Partial Least Squares (PLS) can be represented as a linearized form of self-attention. This equivalence allows PLS to be analyzed using neural network frameworks, potentially offering new insights into its behavior. The authors suggest that PLS's dimensionality reduction properties imply that self-attention mechanisms inherently perform a degree of dimensionality normalization, which contributes to improved learning.
PLS, the workhorse of chemometrics, turns out to be secretly doing linearized self-attention, hinting at implicit dimensionality normalization within neural networks.
This note provides an interesting observation on casting partial least square (PLS) as a linearized self-attention so that PLS may be studied within the neural network paradigm. On the other hand, the dimensionality reduction and selection of predictors in PLS may indicate that self-attention includes certain degree of dimensionality normalization toward improved learning.