Mirror: self-improving video-to-action models
Self-Improving Video-to-Action Models
Mirror is a self-supervised method that learns video-to-action mapping through interaction with the environment, without prior action data or hand-written rewards. On Atari, OGBench, and DeepMind Control, it outperforms mainstream observation-based learning methods using its learned video plans. On Minecraft BASALT, it surpasses methods that have access to expert actions or hand-crafted task knowledge. In Minecraft survival tasks, it obtained an iron pickaxe through self-improvement alone, with no reward or expert action labels.
It doesn't rely on human action annotations or hand-written rewards, letting video planning models become executable agent policies. Worth a look for readers interested in combining video models with agents.
Source: arXiv Game AI · arxiv.org