Search papers, labs, and topics across Lattice.
The paper introduces direct Kolen-Pollack predictive coding (DKP-PC), a novel variant of predictive coding that addresses the limitations of slow error propagation and vanishing updates in deep networks. DKP-PC uses learnable direct feedback connections from the output layer to all hidden layers, enabling faster error transmission and mitigating exponential decay. Empirical results demonstrate that DKP-PC achieves comparable or superior performance to standard PC while improving latency and computational efficiency.
Forget slow error propagation: this new predictive coding variant slashes error signal delay from O(L) to O(1) with direct feedback alignment.
Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.
