Skip to content
Hot storyPast story

New evaluation suite detects all perturbations in user simulators

1 report1 sourceUpdated 3 d ago

What happened

AI overview

Hengguang Zhou et al. propose a new user simulator evaluation suite for evaluating user simulators of interactive agents. The study reports that the evaluation methods of three existing studies detect only 5–6 of 10 artificial perturbations, and that their combined scores even favor simulated user populations that are instructed to accept incomplete work. The new suite combines the utility of user simulators for evaluating agents with two fidelity metrics against real users, and evaluates 6 user simulators on 351 real multi-turn sessions; these sessions come from 348 distinct users across 23 languages, in which users collaborate with agents to create and iterate on artifacts such as full-stack web applications, games, design assets, and automated workflows.

AI-written from the coverage · updated 3 d ago

Coverage timeline

Follow the coverage to see the story from different sides.

Nov 19
  1. arXiv Game AI
    How to evaluate user simulators for interactive agents

    Existing user simulator evaluation frameworks struggle to reliably detect meaningful failures in simulated user behavior: the evaluation methods of three prior studies detected only 5-6 of 10 human-crafted perturbations, and their combined scores even favored simulated user populations instructed to accept incomplete work.

Heat trend for this story

Current (comparable)
3
Peak (comparable)
7Nov 19 13:00
24-hour change
-51%

The trend only compares participants observed continuously and completely, so it may cover fewer than the current heat count. Hover over or tap the chart to see each hour's heat; with a keyboard, use the left and right arrow keys.