Search papers, labs, and topics across Lattice.
This paper introduces a diagonalization-based optimization technique for Linear Echo State Networks (ESNs) that reduces the computational complexity of reservoir state updates from O(N^2) to O(N). By transforming the recurrent dynamics into the eigenbasis of the recurrent matrix, the authors achieve element-wise updates, eliminating costly matrix multiplications. The paper proposes three methods: Eigenbasis Weight Transformation (EWT), End-to-End Eigenbasis Training (EET), and Direct Parameter Generation (DPG), all of which maintain predictive accuracy while providing substantial computational speedups.
Linear Echo State Networks can now achieve O(N) per-step computational complexity, opening the door to faster training and inference without sacrificing accuracy.
We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from O(N^2) to O(N). By reformulating reservoir dynamics in the eigenbasis of the recurrent matrix, the recurrent update becomes a set of independent element-wise operations, eliminating the matrix multiplication. We further propose three methods to use our optimization depending on the situation: (i) Eigenbasis Weight Transformation (EWT), which preserves the dynamics of standard and trained Linear ESNs, (ii) End-to-End Eigenbasis Training (EET), which directly optimizes readout weights in the transformed space and (iii) Direct Parameter Generation (DPG), that bypasses matrix diagonalization by directly sampling eigenvalues and eigenvectors, achieving comparable performance than standard Linear ESNs. Across all experiments, both our methods preserve predictive accuracy while offering significant computational speedups, making them a replacement of standard Linear ESNs computations and training, and suggesting a shift of paradigm in linear ESN towards the direct selection of eigenvalues.
