Search papers, labs, and topics across Lattice.
This paper introduces a constrained variable-projection framework for structured data-science models by interpreting variable projection as a collapsed bilevel optimization problem. The authors derive exact reduced-gradient formulas that are compatible with automatic differentiation and propose a conditional-gradient algorithm, demonstrating convergence under standard assumptions. Numerical experiments across various applications, including sparse autoencoding and few-shot learning, show significant improvements in both wall-clock and data efficiency compared to traditional joint-optimization methods.
A new constrained variable-projection approach achieves superior efficiency in structured data-science models, outperforming conventional optimization techniques.
Variable projection is a classical technique for separable nonlinear least-squares problems, in which variables that enter linearly are eliminated exactly, yielding a reduced nonlinear problem. By expressing this framework as a particular instance of a broader class of bilevel optimization problems, we develop a constrained variable-projection framework for data-science models, where the remaining variables are subject to convex constraints and the eliminated variables arise from a lower-level least-squares problem. In particular, by interpreting variable projection as a collapsed bilevel optimization problem, we derive exact reduced-gradient formulas compatible with automatic differentiation and propose a conditional-gradient algorithm for the resulting constrained reduced problem. We establish convergence guarantees under standard smoothness and compactness assumptions, and discuss extensions to structured lower-level variables. Numerical experiments on sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning suggest that the method can improve wall-clock efficiency and data efficiency relative to natural joint-optimization baselines.
