Search papers, labs, and topics across Lattice.
This paper introduces Task Specialization Fine-Tuning (TSFT), an innovative framework for optimizing fine-tuning in Contextual Reinforcement Learning (CRL) by leveraging a pretrained policy. By employing a parametric model to predict fine-tuning performance and solving the budget allocation problem through integer linear programming, TSFT addresses the challenges of heterogeneous marginal returns and sample inefficiency. Experimental results across various domains, including combinatorial optimization and continuous control, show that TSFT significantly enhances task coverage and approaches oracle performance, marking a significant advancement in model-based CRL.
TSFT redefines how we allocate fine-tuning resources in CRL, achieving near-oracle performance while maximizing task coverage.
Contextual Reinforcement Learning (CRL) seeks to generalize classical RL by maximizing task coverage across a context space of related tasks. While prior works often train from scratch and rely on either multi-task learning for a single policy or strategically training multiple policies, we advocate for a unified alternative: pretraining a single policy with good initial performance, followed by fine-tuning multiple policies for task specialization. This new paradigm, however, introduces unique challenges, such as heterogeneous marginal returns and sample inefficiency. This raises a critical research question: given a pretrained policy and a constrained budget, how much fine-tuning should each task region receive to enable sample-efficient CRL? To this end, we propose Task Specialization Fine-Tuning (TSFT), an online framework that predicts fine-tuning performance with a simple parametric model and exactly solves the resulting discrete budget allocation problem via integer linear programming. Extensive experiments across diverse decision domains, including combinatorial optimization, continuous control, and LLM fine-tuning, demonstrate that TSFT significantly outperforms baselines in task coverage and approaches oracle performance. Our work charts a new direction for model-based CRL, aligning with the modern pretrain-finetune era.
