TaskFrame proposes hierarchical planner and controller framework connecting task-level planning to frame-level keyboard-and-mouse control
TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
Guangzhi Wang proposes TaskFrame, a hierarchical planner and controller framework that connects task-level planning with frame-level keyboard and mouse control. A vision-language task planner breaks high-level goals into ordered atomic instructions, optimized with supervised learning and dependency-aware GRPO; the hidden state of a special goal token that terminates each atomic instruction is cached as a continuous goal embedding and reused in subsequent control steps.
The paper presents a three-stage training pipeline for the planner and controller along with a cached goal embedding approach, so readers can judge how the hierarchical structure connects vision-language reasoning with keyboard-and-mouse control.
Source: arXiv Game AI · arxiv.org