ARC-AGI-3 think-aloud dataset: humans vs. LLMs exploring unfamiliar games
What happened
Shuzhi Liu and colleagues built and released a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics, using ARC-AGI-3 as the test platform. The researchers say it is the first dataset to record how humans discover and use the mechanics of a new environment. The dataset contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The researchers' analysis finds that among successful trajectories, qualitative representation accounts for 50.0% for humans and 14.3% for LLM agents. Humans more often explore when they have no specific hypothesis.
AI-written from the coverage · updated 26 d ago
Coverage timeline
Follow the coverage to see the story from different sides.
- arXiv Game World Models & GenerationPickHow humans learn unfamiliar worlds: a think-aloud dataset on ARC-AGI-3
Researchers built a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics. It contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The analysis shows that qualitative representation makes up 50.0% of successful human trajectories versus 14.3% of successful LLM trajectories, and humans more often explore without a specific hypothesis.