Skip to content
Hot storyPast story

LiveFoley: real-time video-to-audio generation on a single GPU with 156 ms latency

1 report1 sourceUpdated 18 d ago

What happened

AI overview

Monash University, Nanjing University, Shengshu Technology and other institutions proposed LiveFoley, a streaming video-to-audio model that generates sound effects from a real-time video stream. The research team says that on a single A800 GPU with a hop size of 1, the end-to-end latency is 156 ms. The model uses History LocDiT by default, taking past video chunks as context without adding latency; an optional Future-Warmup LocDiT uses the current hidden state and the next visual token to improve quality, but adds 0.08 seconds of lookahead, raising the estimated latency to 236 ms. The team says both configurations are below the human perception threshold. Unless otherwise noted, demo audio is streamed by LiveFoley-48k at 48 kHz stereo, performing one autoregressive step every 0.08 seconds of video, using no future frames other than the lookahead.

AI-written from the coverage · updated 16 d ago

Coverage timeline

Follow the coverage to see the story from different sides.

Nov 4
  1. Grok Bot ResearchPick
    LiveFoley: a real-time video-to-audio model for interactive worlds with 156 ms end-to-end latency on a single A800

    Monash University, Nanjing University, Shengshu Technology and other institutions proposed LiveFoley, a streaming video-to-audio (V2A) model that generates sound effects from real-time video streams, with an end-to-end latency of 156 ms on a single A800 GPU.