Search papers, labs, and topics across Lattice.
This paper introduces WF-Act-PC, a novel predictive coding method that computes the Jacobian transpose locally, thus eliminating the reliance on non-local autograd operations in deep networks. By leveraging local weight updates and specific assumptions about layer structure, the authors demonstrate that their approach not only closes the transport gap but also improves accuracy with increased depth on CIFAR datasets. The results show WF-Act-PC outperforming traditional predictive coding methods and matching or exceeding backpropagation performance on various architectures, highlighting its potential for more efficient training in deep learning.
WF-Act-PC achieves superior accuracy with deeper networks while eliminating the need for non-local error transport, challenging the dominance of backpropagation.
Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC. We show that this dependency is largely avoidable. For any layer $f(x)=\mathrm{Act}(\mathrm{Norm}(L(x)))$ with frozen normalization statistics, the exact $J^\top$ factors into three locally available terms, $J^\top v = L^\top(s \odot \sigma'(z) \odot v)$, where $\sigma'$ is the activation derivative, $z$ is the pre-activation, and $s=\gamma/\sigma_{\mathrm{run}}$ is the normalization gain. Prior weight-feedback methods omitted both corrections; restoring them closes the transport gap for this layer class. Locality here holds up to three assumptions, which we state upfront: weight symmetry ($L^\top$ mirrors the forward operator, as assumed by all PC), a soft spectral-norm control that is not synapse-local, and a nearest-neighbour approximation for MaxPool. Substituting the identity into PC yields WF-Act-PC, which removes the autograd backward pass from error transport. On CIFAR-10/100 (50 epochs, 5 seeds), WF-Act-PC is the only PC method whose accuracy improves with depth, surpassing iPC - the strongest classical PC baseline - by 2.7-22.3 pp on CIFAR-10. With both methods tuned per architecture, it matches or exceeds a comparably-tuned backpropagation baseline on the deeper CIFAR-10 architectures (VGG-9: 93.57% vs. 92.43%; ResNet-18: 92.76% vs. 91.54%) and on the harder Tiny-ImageNet benchmark, while trailing tuned BP on the deeper CIFAR-100 VGG cells. Our WF-Act-PC implementation is publicly available at https://github.com/jlshen025/pcax
