AceAction-5M: A dataset for hierarchical-goal generalist action learning across multiple 3D game worlds
AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
The authors present the AceAction dataset, a collection of human gameplay demonstrations across multiple 3D games that links overall task instructions, ordered subgoals and local skill instructions with time-aligned visual observations and keyboard-and-mouse actions, covering navigation, reconnaissance, combat, interaction and recovery. The team uses a vision-language model pipeline to identify subgoals from changes in goals or interactable objects, anchor navigation goals to recognizable waypoints, and compose adjacent subgoals into task instructions. Review of 500 trajectories achieved a 95.8% pass rate in independent evaluation.
The dataset strings overall task instructions, ordered subgoals and local skill instructions together with time-aligned frames and keyboard-and-mouse actions, so it can be used to observe how an agent executes, composes and switches skills.
Source: arXiv Game AI · arxiv.org