Search papers, labs, and topics across Lattice.
This paper introduces FiMI Banking, a controlled environment specifically designed for Indian retail banking, addressing the limitations of general-purpose language models in handling sensitive banking tasks. By leveraging vetted banking documents and structured ground truth, the authors evaluate two post-training methods: preference optimization and reinforcement learning with verifiable rewards. The findings reveal that preference optimization significantly enhances safe behavior, while reinforcement learning improves performance on edge cases and order-sensitive tasks, demonstrating a complementary approach to developing reliable banking agents.
Preference optimization boosts safe responses in banking agents from 52% to 80%, while reinforcement learning enhances edge-case performance significantly with fewer tokens generated.
Banks need conversational systems that can answer product questions, assist customers with account-related requests, and operate safely within strict operational and regulatory constraints. General-purpose language models do not reliably meet these requirements. They fall short when a task requires grounded information, correct tool use, or cautious handling of bank-specific sensitive situations. We introduce FiMI Banking, a controlled Indian retail-banking setting. We build it from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. We evaluate two post-training approaches: preference optimization for response-level behavior, and reinforcement learning with verifiable rewards for multi-turn tool-use tasks. Preference optimization improves safe behavior substantially: out-of-scope refusal rises from 52% to 80%. Reinforcement learning improves edge-case performance from 0.509 to 0.718 and order-sensitive task performance from 0.590 to 0.679, while using 29% fewer generated tokens. These results show that preference optimization and verifiable-reward reinforcement learning address complementary requirements for reliable banking agents.
