New evaluation suite detects all perturbations in user simulators
What happened
Hengguang Zhou et al. propose a new user simulator evaluation suite for evaluating user simulators of interactive agents. The study reports that the evaluation methods of three existing studies detect only 5–6 of 10 artificial perturbations, and that their combined scores even favor simulated user populations that are instructed to accept incomplete work. The new suite combines the utility of user simulators for evaluating agents with two fidelity metrics against real users, and evaluates 6 user simulators on 351 real multi-turn sessions; these sessions come from 348 distinct users across 23 languages, in which users collaborate with agents to create and iterate on artifacts such as full-stack web applications, games, design assets, and automated workflows.
AI-written from the coverage · updated 3 d ago
Coverage timeline
Follow the coverage to see the story from different sides.
- arXiv Game AIHow to evaluate user simulators for interactive agents
Existing user simulator evaluation frameworks struggle to reliably detect meaningful failures in simulated user behavior: the evaluation methods of three prior studies detected only 5-6 of 10 human-crafted perturbations, and their combined scores even favored simulated user populations instructed to accept incomplete work.
Heat trend for this story
- Current (comparable)
- 3
- Peak (comparable)
- 7Nov 19 13:00
- 24-hour change
- -51%
The trend only compares participants observed continuously and completely, so it may cover fewer than the current heat count. Hover over or tap the chart to see each hour's heat; with a keyboard, use the left and right arrow keys.