Search papers, labs, and topics across Lattice.
This paper introduces a novel perturbation-based learning rule for online self-supervised learning in echo state networks (ESNs), addressing the challenge of variance growth in high-dimensional systems. By applying an orthogonal decomposition to the self-supervised learning cost, the method effectively reduces the perturbation dimension from the reservoir size to the input size, enabling efficient adaptation without compromising performance. The key finding demonstrates that this approach allows for scalable, memory-efficient learning while maintaining the benefits of online and self-supervised adaptation in complex environments.
Reducing perturbation dimensions in echo state networks could revolutionize online self-supervised learning by minimizing variance while maximizing adaptability.
Intelligent systems should not only solve tasks but also adapt under real-world constraints. Autonomous adaptation via self-supervised learning, sequential adaptation via online learning, and memory-efficient implementation via perturbation-based learning are important requirements for such systems. However, these requirements are generally in tension for high-dimensional systems, because perturbation-based learning suffers from variance that grows with the dimension of the perturbed variables. In this study, we focus on echo state networks (ESNs), where this tension naturally arises in large reservoirs. We propose a perturbation-based learning rule for online self-supervised learning in ESNs. The proposed rule is derived from an orthogonal decomposition of the self-supervised learning cost, which separates an input-dependent component from a redundant component determined by the fixed ESN parameters. By perturbing only the input-dependent component, the effective perturbation dimension is reduced from the reservoir dimension to the input dimension. Thus, the proposed method preserves self-supervised adaptation, online learning, and scalar-feedback perturbation learning, while avoiding reservoir-size-dependent variance growth. This suggests a design principle for scalable and hardware-compatible learning: online learning should be restricted to the dynamically necessary low-dimensional component of the objective.
