Gym-Anything-RT: Scaling real-time environments for vision-language agents
Gym-Anything-RT generates 100 real-time visual mini-games automatically through an agent pipeline to evaluate the real-time interaction abilities of vision-language agents. The paper reports that GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time mini-games, rising to 75.0% when the environment is paused. The authors also find that some frontier agents bypass real-time perception and control when allowed to write code. The environment and the generated mini-games are publicly released.
Why it matters: This study separates real-time performance from other abilities. It allows agents' scores to be compared before and after pausing on the same mini-game, which helps identify weaknesses in real-time decision-making.