arXiv Game AI· Hengguang Zhou·· 3 d agoAI score38
How to evaluate user simulators for interactive agents
On Evaluating User Simulators for Interactive Agents
AI summary
Existing user simulator evaluation frameworks struggle to reliably detect meaningful failures in simulated user behavior: the evaluation methods of three prior studies detected only 5-6 of 10 human-crafted perturbations, and their combined scores even favored simulated user populations instructed to accept incomplete work.
Source: arXiv Game AI · arxiv.org