MMAudio

MMAudio is a research-grade open-source model for video-to-audio synthesis, developed by Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji from the University of Illinois Urbana-Champaign and Sony AI. The model accepts video input and optional text prompts and generates a synchronized audio track that aligns with the visual events, motion, and context of the video at the frame level. Its key technical innovation is multimodal joint training — training simultaneously on large-scale video-audio datasets including AudioSet, VGGSound, and WavCaps as well as text-audio datasets — which improves both audio quality and semantic alignment compared to models trained on video data alone. A frame-level conditional synchronization module further ensures that generated sounds correspond to specific visual events in the correct timing window rather than playing out of sync with the action on screen. MMAudio generates an 8-second audio clip in approximately 1.23 seconds, with only 157 million parameters, making it notably efficient relative to its output quality. See related options. Video creators use MMAudio to add ambient soundscapes, environmental sound effects, and synchronized audio to AI-generated or silent video content without manual sound design. Game developers use it to prototype dynamic sound generation for scenes where audio should respond to visual changes and player interactions. Archival researchers working with silent historical footage use it to add contextually plausible audio to silent recordings. VFX artists and post-production teams use it in early editorial phases to generate scratch audio tracks for editorial timing decisions before the final sound design session. AI researchers use it as a benchmark model for evaluating new approaches to audio-visual synchronization Find related tools.