Search papers, labs, and topics across Lattice.
Rationally Linearized Multilayer Perceptrons (RaLiMPs) leverage low-degree rational approximations of activation functions to efficiently linearize MLPs. This approach constructs kernel representations with narrow bottlenecks, allowing for efficient composition across layers and providing theoretical approximation guarantees. Experimental results show RaLiMPs achieve competitive or superior accuracy with significantly reduced inference costs across diverse applications.
Low-degree rational approximations enable MLPs to be linearized into efficient, composable kernel representations that outperform polynomial approximations.
We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.
