Search papers, labs, and topics across Lattice.
This paper presents a comprehensive framework for developing customer support AI agents at scale, specifically tailored for Nubank's 100M+ user base. By integrating structured context engineering, human-in-the-loop prompt iteration, and rigorous evaluation methodologies, the authors demonstrate that the quality of the evaluation pipeline significantly influences the speed of iteration and overall effectiveness of the AI agents. Results from five distinct deployments show substantial improvements in customer satisfaction metrics, with one deployment achieving a 37 percentage-point increase in Net Promoter Score, indicating that a well-structured evaluation process can reliably predict and enhance production outcomes.
Evaluation-driven development can boost AI agent performance, achieving customer satisfaction levels nearly on par with expert human agents.
The rapid rise in LLM capabilities has made AI agents increasingly viable across a broad range of tasks. Among the most promising applications is building production-ready customer-facing agents, a challenge that demands coordinated excellence in evaluation methodology, context engineering, training, and online measurement. Yet these critical pillars are typically developed in isolation, creating blind spots that only surface after deployment. In this paper, we present a unified framework that bridges offline development with online impact for customer support AI agents at Nubank, a company with 100M+ users. Our approach integrates several key components: (1) structured context engineering tailored to customer support agents, (2) systematic human-in-the-loop prompt iteration, (3) rigorous LLM judge evaluation with measured inter-rater agreement and GEPA optimization for consistency, and (4) ideation-to-production validation. A central insight is that evaluation-pipeline quality directly determines iteration velocity. We present results from five production deployments spanning distinct domains: card delivery, debt management, credit-limit support, card management, and product explanation. These deployments deliver consistent customer-satisfaction gains while substantially accelerating iteration. In our card-delivery deployment, large-scale A/B testing yields a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate over prior agent variants, alongside a strong correlation between offline simulation metrics and online outcomes, demonstrating that eval-driven development reliably predicts production impact. On most use cases, AI satisfaction reaches within a few percentage points of expert human agents.
