Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper re-derives Target Propagation (TP) within a trust region framework, revealing its connection to Gauss-Newton (GN) optimization under a novel implicit norm. The proposed Conjugate Gradient Target Propagation (CGTP) method leverages conjugate gradients for parallel, layer-local subproblem solving, with a single adaptive parameter controlling the trust region and weight update bound. CGTP achieves competitive accuracy with ADAM and KFAC on MLP tasks while significantly reducing outer iteration requirements and demonstrating stability across network depths.
Target Propagation's connection to Gauss-Newton optimization is established under a new implicit norm, with CGTP achieving KFAC-like performance using orders of magnitude fewer outer iterations.
Target propagation (TP) is a backpropagation alternative in which local targets carry the global learning signal. Prior works claim that TP performs Gauss-Newton (GN) optimisation on these targets, but the correspondence has only been established under unrealistic assumptions where hitting them exactly is trivial. Practical variants instead descend on a least-squares problem via a single gradient step. In this paper, we rederive TP from a trust region under an implicit, previously unexamined norm, and introduce CGTP (Conjugate Gradient Target Propagation), which uses conjugate gradients to solve the layer-local subproblems in parallel. A single adaptive parameter jointly controls this trust region and a matching bound on the weight update; KFAC emerges exactly as the special case where the per-sample Jacobians are replaced by their batch mean, giving its factored damping a variational grounding. On MNIST and Fashion-MNIST MLPs, CGTP matches the accuracy of ADAM and KFAC while needing orders of magnitude fewer outer iterations than ADAM (about 10x fewer backward-pass-equivalents overall), while remaining stable across depth.