Search papers, labs, and topics across Lattice.
This paper analyzes the convergence properties of a basic forward-backward-splitting (FBS)-induced deep neural network, derived from iterative optimization, to its deep-layer limit system in a learning context. It establishes a 螕-convergence argument, demonstrating that optimal learning parameters of the network converge to solutions of the deep-layer limit system's learning problem under mild assumptions. The paper also provides a qualitative analysis of perturbation stabilities for these learning problems, validated by a numerical experiment.
Unveiling the convergence of FBS-induced networks to their deep-layer limits provides theoretical guarantees for training deep unfolded architectures.
Deep unfolding neural networks derived from iterative optimization schemes and numerical ordinary/partial differential equations (ODEs/PDEs) have attracted much attention in data science over the last decade. Therein, numerous important network architectures were constructed from the basic forward-backward-splitting (FBS) algorithm. In this paper, we continue our research on the most basic FBS-induced network, an architecture unrolled from the original FBS algorithm by incorporating direct parameter relaxations. Following the difference/differential inclusion formulations in our previous forward system analyses, we here consider some theoretical aspects of corresponding learning problems. Under some mild assumptions, we establish a general convergence property of the training problem of the basic FBS-induced network to the learning problem of the deep-layer limit system, implying a $螕$-convergence argument showing that any cluster point of the optimal learning parameters for the network is a solution to the learning problem of the deep-layer limit system. A qualitative analysis of perturbation stabilities of these learning problems is also presented. A simple numerical experiment is conducted to validate our main general convergence result.
