TaskFrame hierarchical planner-controller framework proposed
What happened
Guangzhi Wang et al. propose TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard and mouse control for game agents. A vision-language task planner breaks high-level goals into ordered sequences of atomic instructions and is optimized with supervised learning and dependency-aware GRPO; a causal Transformer controller outputs discrete key presses and continuous camera movement frame by frame. In Minecraft, its instruction decomposition is on par with general vision-language models, and its combat success rate in closed-loop evaluation is higher than comparison baselines.
AI-written from the coverage · updated 23 d ago
Coverage timeline
Follow the coverage to see the story from different sides.
- arXiv Game AITaskFrame: Connecting task-level planning with frame-level control for game agents
TaskFrame is a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard and mouse control for game agents. A vision-language task planner breaks high-level goals into ordered sequences of atomic instructions and is optimized with supervised learning and dependency-aware GRPO; a causal Transformer controller outputs discrete key presses and continuous camera movement frame by frame. In Minecraft, its instruction decomposition is on par with general vision-language models, and its combat success rate in closed-loop evaluation is higher than comparison baselines.