Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes zeroth-order (ZO) optimization algorithms from a dynamical systems perspective, demonstrating that under certain conditions, they can achieve convergence rates comparable to first-order (FO) methods without suffering from extra dimension dependencies. They formulate the average of a ZO algorithm as the average of its FO counterpart with bounded perturbations. By leveraging input-to-state stability properties, they prove that ZO methods converge to a neighborhood of the FO method's fixed point, with the radius of this neighborhood dependent on the perturbation bound.
Zeroth-order optimization, long considered cursed with extra-dimensional dependencies, can actually achieve first-order convergence rates under specific conditions, challenging conventional wisdom.
While it is generally understood that zeroth-order (ZO) algorithms have an extra dependency on their number of iterations for any choice of parameters, compared to their first-order (FO) counterparts, in this work, we show that under several conditions, in expectation, ZO methods do not suffer from extra dimension dependencies in their convergence rates with respect to their FO counterparts. We look at optimisation algorithms from the dynamical systems perspective and analyse the conditions under which one can formulate the average of a ZO algorithm as the average of its FO counterpart with bounded perturbations with values dependent on design parameters. Then, using input-to-state stability properties, we show ZO methods follow the same decay rate as their FO counterparts and converge to a neighbourhood of the fixed point of FO methods, where its radius depends on the bound of the norm of the perturbations, which can be made arbitrarily small. The theoretical findings are illustrated via numerical examples.