Search papers, labs, and topics across Lattice.
This paper formalizes the self-refinement process of foundation models as an optimal stopping problem, allowing for the determination of the ideal number of refinement iterations based on expected improvements versus costs. By deriving optimal stopping policies and employing stochastic approximation for efficient computation, the authors demonstrate significant cost efficiency in their approach. Experimental evaluations on a coding benchmark reveal that these new stopping policies outperform previous methods, highlighting their practical advantages in model refinement.
Stopping policies derived from optimal stopping theory can drastically enhance the cost-efficiency of self-refining foundation models, outperforming traditional methods.
Foundation models can improve their outputs through a self-refinement process driven by external feedback. In this process, the model is embedded in an iterative loop where it generates outputs, receives feedback from verifiers, and refines its responses through in-context learning. Following a novel approach, we formalize this process as an optimal stopping problem where the number of refinement iterations is decided based on expected improvement relative to cost. We derive optimal stopping policies and show that they can be efficiently computed through stochastic approximation. To evaluate our approach experimentally, we apply it to a coding benchmark for foundation models. The empirical results show that our stopping policies are significantly more cost-efficient than stopping policies proposed in prior work.
