Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
[Sztuczna inteligencja] Autorzy: Edesio Alcoba, Kevin Rossell, Aman Gupta i 13 więcej
Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simula…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja