Skip to content
Hot storyPast story

ARC-AGI-3 think-aloud dataset: humans vs. LLMs exploring unfamiliar games

1 report1 sourceUpdated 27 d ago

What happened

AI overview

Shuzhi Liu and colleagues built and released a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics, using ARC-AGI-3 as the test platform. The researchers say it is the first dataset to record how humans discover and use the mechanics of a new environment. The dataset contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The researchers' analysis finds that among successful trajectories, qualitative representation accounts for 50.0% for humans and 14.3% for LLM agents. Humans more often explore when they have no specific hypothesis.

AI-written from the coverage · updated 26 d ago

Coverage timeline

Follow the coverage to see the story from different sides.

Oct 26
  1. arXiv Game World Models & GenerationPick
    How humans learn unfamiliar worlds: a think-aloud dataset on ARC-AGI-3

    Researchers built a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics. It contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The analysis shows that qualitative representation makes up 50.0% of successful human trajectories versus 14.3% of successful LLM trajectories, and humans more often explore without a specific hypothesis.