Search papers, labs, and topics across Lattice.
This research introduces Preference Tree Optimization (PTO), a framework that enhances goal-oriented dialogue systems by generating preference data through simulations using a Preference Tree with Look-Ahead. Focusing on Motivational Interviewing (MI), the study employs virtual patients and an oracle evaluator to create rich datasets that improve agent decision-making capabilities via Direct Preference Optimization (DPO). Experimental results show that agents trained with PTO significantly outperform baselines in session satisfaction and working alliance, particularly benefiting from deeper look-ahead configurations for long-term planning.
Agents trained with the Preference Tree Optimization framework achieve unprecedented improvements in goal-oriented dialogue, outperforming traditional methods in both satisfaction and strategic planning.
Developing dialogue systems capable of engaging in multi-turn, goal-oriented conversations remains a significant challenge, especially in specialized domains with limited data. This research proposes a novel framework called Preference Tree Optimization (PTO), designed to iteratively improve agent models in such dialogue systems, by generating preference data using a method called Preference Tree with Look-Ahead. Focusing on Motivational Interviewing (MI) -- a counseling technique aimed at facilitating behavioral change -- we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference datasets. By combining this method with Direct Preference Optimization (DPO), we aim to enhance the agent's decision-making capabilities over iterative training cycles. The proposed framework addresses data scarcity and advances the development of more nuanced and effective dialogue systems in goal-oriented domains. Experimental evaluations demonstrate that the PTO framework enhances dialogue agents'performance in goal-oriented conversations within the domain of Motivational Interviewing (MI). Models trained with PTO consistently outperformed the baseline in key metrics such as session satisfaction and working alliance. Additionally, incorporating look-ahead simulations led to improved long-term planning and more effective conversational strategies, with deeper look-ahead configurations yielding the most stable and high-scoring results.
