Skip to content

Fields

World models latest news

Interactive generative environments and world models: playable worlds generated in real time, physics and consistency, and what they mean for game engines.

92 picks47 in the last 30 days123 collected in all

Updated

Picks archive · Page 3

Oct 27Tue41–60
  1. arXiv Game World Models & Generation37

    How humans learn unfamiliar worlds: a think-aloud dataset on ARC-AGI-3

    Researchers built a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics. It contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The analysis shows that qualitative representation makes up 50.0% of successful human trajectories versus 14.3% of successful LLM trajectories, and humans more often explore without a specific hypothesis.

    Why it matters: This paper uses think-aloud human game trajectories to compare with the failure and recovery processes of LLM agents, offering a reference for examining the limitations of existing game-agent benchmarks.

  2. arXiv Game World Models & Generation45

    WorldPager: zero-stall tiered memory for long-horizon world models on a single consumer GPU

    WorldPager stores the int4 KV memory of interactive world models across GPU, DRAM and NVMe tiers and prefetches it based on camera pose. On a single RTX 4090, it continuously generates 37.5 minutes of video without stalls, with host memory staying constant and a speed of about 9.5 frames per second. On a closed test set, int4 storage showed no measurable quality loss. On long-interval revisits, image quality dropped by about 1 MUSIQ point.

    Why it matters: The paper places a world model's KV memory across GPU, DRAM and NVMe tiers with pose-based prefetching, which is useful for readers following long-horizon interactive video generation.

  3. arXiv Game World Models & Generation53

    BoxWorld: Let World Engines Control and Video Generators Render

    BoxWorld hands user-defined interaction rules to the Box World Engine, then has the Box-State generative renderer autoregressively render a real-time interactive video world from the object states. The paper says it clearly outperforms end-to-end world models on evaluations including PairedStateBench and remains competitive on several public benchmarks. Training has three stages: learning object localization and state transitions from automatically annotated open-domain and game videos, then rollout distillation on self-generated histories.

    Why it matters: Separating interaction reasoning from generative rendering, and comparing it with end-to-end video world models, offers one way to divide the work in controllable game worlds.

Oct 26Mon
  1. arXiv Game World Models & Generation56

    WorldPager: zero-stall tiered memory for long-horizon world models on a single consumer GPU

    WorldPager stores the int4 KV memory of interactive world models across GPU, DRAM and NVMe tiers and prefetches it based on camera pose. On a single RTX 4090, it continuously generates 37.5 minutes of video without stalls, with host memory staying constant and a speed of about 9.5 frames per second. On a closed test set, int4 storage showed no measurable quality loss. On long-interval revisits, image quality dropped by about 1 MUSIQ point.

    Why it matters: The paper places a world model's KV memory across GPU, DRAM and NVMe tiers with pose-based prefetching, which is useful for readers following long-horizon interactive video generation.

  2. arXiv Game World Models & Generation49

    How humans learn unfamiliar worlds: a think-aloud dataset on ARC-AGI-3

    Researchers built a think-aloud dataset that records how humans explore and learn unfamiliar game mechanics. It contains 144 human trajectories from 64 participants across 25 public games, plus 250 LLM agent trajectories. The analysis shows that qualitative representation makes up 50.0% of successful human trajectories versus 14.3% of successful LLM trajectories, and humans more often explore without a specific hypothesis.

    Why it matters: This paper uses think-aloud human game trajectories to compare with the failure and recovery processes of LLM agents, offering a reference for examining the limitations of existing game-agent benchmarks.

Oct 25Sun
  1. arXiv Game AI50

    WMEval: Evaluating interactive world models grounded on physical simulations

    WMEval evaluates interactive world models against Unity physics simulation as a reference. The benchmark covers 9 scenes and assesses action controllability, scene and character consistency, prompt following, kinematics, fluid dynamics, physical causality, and lighting. By executing matched action sequences in the simulator and in the world models and comparing the results, the paper argues that existing models still have substantial room for improvement in dimensions that standard perceptual metrics fail to capture.

    Why it matters: Existing world model evaluations mostly look at perceptual similarity of frames; this work instead uses Unity physics simulation as a reference, adding a check of physical causal consistency.

Oct 23Fri
  1. arXiv Game World Models & Generation34

    Can LLMs serve as event world models? A study from single-actor environments to multi-actor event worlds

    A paper asks whether large language models can predict future observations from event logs and the actions an agent plans to take, using benchmarks built from human Diplomacy games and real Polymarket trades. When only the agent's own actions are given and other actors strongly shape outcomes, model predictions fall well short, and larger models and longer histories help only modestly. When the recorded actions of all actors are provided, fine-tuning and more data improve predictions, but models still overestimate the success rate of planned actions, while imagined experience from world models still improves policy learning.

    Why it matters: The paper argues that when given only the agent's own actions, models struggle to predict outcomes strongly shaped by other actors, a key gap for event world modeling with multiple actors

Oct 22Thu
  1. arXiv Game AI45

    AnyGameRenderer: entity-consistent generative rendering across games and an entity-level benchmark

    AnyGameRenderer is a video-to-video generative renderer that converts grey-box or flat-color game videos made by AI coding agents into footage with characters, motion details and effects, while keeping the layout, camera, and the positions and timing of entities unchanged.

    Why it matters: It turns grey-box prototypes generated by agents into footage with characters and effects, and uses an entity-level benchmark to measure temporal correspondence, which helps assess how feasible it is to go from prototype to finished visuals.

  2. arXiv Game AI45

    AnyGameRenderer proposes entity-consistent generative rendering across games and an entity-level benchmark

    AnyGameRenderer is a video-to-video generative renderer that turns prototype footage such as gray-box and colored-geometry scenes produced by AI agents into high-quality visuals with characters, motion detail, and effects, while keeping the layout, camera, and each entity's position and timing unchanged.

    Why it matters: The paper presents a method for turning gray-box prototype videos into high-quality visuals, along with an entity-level benchmark and quantitative comparisons, making it possible to judge whether generative rendering preserves entity correspondences.

  3. arXiv Game World Models & Generation39

    LiveAct: heterogeneous action interfaces for real-time multi-agent world models

    LiveAct introduces a heterogeneous action interface that lets a multi-agent video world model generate joint visual futures for multiple controlled agents, handling control inputs ranging from game buttons to robot commands. The method reaches agent attribution accuracy of 0.898 on games and 0.942 on multi-robot manipulation, without relying on agent position or identity cropping. Its causal generator runs at 17.70 FPS at 512×448 resolution.

    Why it matters: The work binds each action stream to its corresponding visual agent and reports streaming generation at 17.70 FPS, making it a useful comparison point for existing single-agent interactive world models.

  4. arXiv Game AI43

    WorldStream: streaming interactive 3D worlds with hierarchical planning and persistent state

    WorldStream introduces an open-ended 3D world model that can expand playable terrain at runtime, respond to open-ended language instructions, and restore visited locations along with their edits. A hierarchical planner outputs macro and micro structural decisions, which deterministic local code then compiles into terrain, roads, settlements, vegetation, and entities. The world record stores plans, seeds, object identities, mutable state, and boundary constraints, so returning chunks can be recompiled from it. Stateful NPCs also update as they act.

    Why it matters: The paper puts terrain generation, NPC state, and records of edits to visited areas into one shared world state. It is useful for comparing approaches to streaming open-world generation.

  5. arXiv Game World Models & Generation40

    Can LLMs serve as event world models? A study from single-actor environments to multi-actor event worlds

    A paper asks whether large language models can predict future observations from event logs and the actions an agent plans to take, using benchmarks built from human Diplomacy games and real Polymarket trades. When only the agent's own actions are given and other actors strongly shape outcomes, model predictions fall well short, and larger models and longer histories help only modestly. When the recorded actions of all actors are provided, fine-tuning and more data improve predictions, but models still overestimate the success rate of planned actions, while imagined experience from world models still improves policy learning.

    Why it matters: The paper argues that when given only the agent's own actions, models struggle to predict outcomes strongly shaped by other actors, a key gap for event world modeling with multiple actors

Oct 21Wed
  1. arXiv Game World Models & Generation50

    LiveAct: heterogeneous action interfaces for real-time multi-agent world models

    LiveAct introduces a heterogeneous action interface that lets a multi-agent video world model generate joint visual futures for multiple controlled agents, handling control inputs ranging from game buttons to robot commands. The method reaches agent attribution accuracy of 0.898 on games and 0.942 on multi-robot manipulation, without relying on agent position or identity cropping. Its causal generator runs at 17.70 FPS at 512×448 resolution.

    Why it matters: The work binds each action stream to its corresponding visual agent and reports streaming generation at 17.70 FPS, making it a useful comparison point for existing single-agent interactive world models.

  2. arXiv Game AI42

    SolarRenderer: Few-Step Generative Reference Rendering Based on Structural Conditions

    SolarRenderer is a structure-conditioned video renderer that turns layouts, actions, and camera trajectories from a game engine or coding agent into realistic video. Built on a frozen MiniMax-H3 33B backbone with two lightweight LoRAs, it renders in four steps, 12.3 times faster than 50-step sampling, and its streaming variant renders 768p video in real time. The paper reports state-of-the-art results on an in-distribution test set and a generalization benchmark covering 16 structural representations, and generalization to 11 unseen representations such as pose skeletons, sketches, and point clouds.

    Why it matters: The paper turns the coarse structure output by a game engine into video, and its four-step sampling is 12.3 times faster than 50 steps, making it a reference for real-time generative rendering.

  3. Tripo38

    Eden-Live now officially available on @reactorworld.

    Tripo's real-time generative rendering tool Eden-Live is now officially available on the Reactor platform, with a live online demo to try. According to a quoted post, it can repaint 3D scenes or prototypes into interactive worlds, change styles in real time using reference images and text, and allow prompts to be edited during runtime.

    Why it matters: Eden-Live offers an online demo on the Reactor platform, useful for developers interested in real-time visual repainting and scene stylization who want to evaluate integration options.

  4. arXiv Game World Models & Generation41

    ActionRoPE: Using world coordinates for spatially consistent control in planar-camera game world models

    ActionRoPE turns action-driven camera displacement into world-coordinate rotary positions, so tokens at the same world position keep the same positional encoding over time. The paper also introduces Coordinate-Conditioned Noise Scheduling (CNS), which stabilizes observed regions earlier and generates new regions by referencing them. Compared with four representative action-conditioning mechanisms, the method follows commanded actions more faithfully and cuts revisit drift by 66% relative to the strongest baseline. It also transfers to video backbones with 1.3B to 33B parameters and to out-of-distribution scenarios.

    Why it matters: Converting action displacements into world-coordinate rotary positions is worth a look for action control in world models of top-down and isometric games.

  5. arXiv Game AI42

    WorldStream: streaming interactive 3D worlds with hierarchical planning and persistent state

    WorldStream introduces an open-ended 3D world model that can expand playable terrain at runtime, respond to open-ended language instructions, and restore visited locations along with their edits. A hierarchical planner outputs macro and micro structural decisions, which deterministic local code then compiles into terrain, roads, settlements, vegetation, and entities. The world record stores plans, seeds, object identities, mutable state, and boundary constraints, so returning chunks can be recompiled from it. Stateful NPCs also update as they act.

    Why it matters: The paper puts terrain generation, NPC state, and records of edits to visited areas into one shared world state. It is useful for comparing approaches to streaming open-world generation.

  6. Viggle AI49

    Viggle AI releases JST-2, unifying 3D character, motion and scene generation

    Viggle AI has released JST-2, a world foundation model that is now live in its API. Developers can get started at https://viggle.ai/developers. The company says JST-2 builds its full model stack in 3D space rather than pixel or video space, generating 3D scenes within seconds and expanding them in real time. The same general architecture can also generate 3D characters with built-in motion.

    Why it matters: Unlike most world models that learn in pixel or video space, JST-2 builds its entire model stack directly in 3D space, and its API is now open.

  7. arXiv Game AI51

    SolarRenderer: Few-Step Generative Reference Rendering Based on Structural Conditions

    SolarRenderer is a structure-conditioned video renderer that turns layouts, actions, and camera trajectories from a game engine or coding agent into realistic video. Built on a frozen MiniMax-H3 33B backbone with two lightweight LoRAs, it renders in four steps, 12.3 times faster than 50-step sampling, and its streaming variant renders 768p video in real time. The paper reports state-of-the-art results on an in-distribution test set and a generalization benchmark covering 16 structural representations, and generalization to 11 unseen representations such as pose skeletons, sketches, and point clouds.

    Why it matters: The paper turns the coarse structure output by a game engine into video, and its four-step sampling is 12.3 times faster than 50 steps, making it a reference for real-time generative rendering.

  8. arXiv Game AI41

    OverSeer: Multi-agent world models beyond what agents see

    OverSeer is a multi-agent world model that adds fixed-position auxiliary camera views beyond each agent's first-person view, improving temporal and cross-agent consistency in multi-agent video generation. It uses a delta-VAE that encodes only changes relative to a near-static reference image, reducing the number of tokens needed to process the auxiliary views. Experiments in Minecraft and Doom show improved world and agent consistency, and qualitative results on real-world multi-camera capture indicate the method does not depend on a game engine.

    Why it matters: Multi-agent video world models struggle to stay consistent across viewpoints; this work uses a fixed auxiliary viewpoint as a proxy for world state, which is a useful reference for multi-view generation.