MMAudio

MMAudio is a research-grade open-source model for video-to-audio synthesis, developed by Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji from the University of Illinois Urbana-Champaign and Sony AI. The model accepts video input and optional text prompts and generates a synchronized audio track that aligns with the visual events, motion, and context of the video at the frame level. Its key technical innovation is multimodal joint training — training simultaneously on large-scale video-audio datasets including AudioSet, VGGSound, and WavCaps as well as text-audio datasets — which improves both audio quality and semantic alignment compared to models trained on video data alone. A frame-level conditional synchronization module further ensures that generated sounds correspond to specific visual events in the correct timing window rather than playing out of sync with the action on screen. MMAudio generates an 8-second audio clip in approximately 1.23 seconds, with only 157 million parameters, making it notably efficient relative to its output quality. Video creators use MMAudio to add ambient soundscapes, environmental sound effects, and synchronized audio to AI-generated or silent video content without manual sound design. Game developers use it to prototype dynamic sound generation for scenes where audio should respond to visual changes and player interactions. See related options. Archival researchers working with silent historical footage use it to add contextually plausible audio to silent recordings. VFX artists and post-production teams use it in early editorial phases to generate scratch audio tracks for editorial timing decisions before the final sound design session. AI researchers use it as a benchmark model for evaluating new approaches to audio-visual synchronization. MMAudio is freely available as open-source software on GitHub at hkchengrex/MMAudio, with interactive online demos accessible via Hugging Face and Replicate for users who want to try the model without local installation. Local installation requires a Linux environment, PyTorch 2.5.1 or later, and a GPU with at least 8GB of VRAM for inference. The model is provided for research and personal use; the training datasets it was trained on carry their own license terms, and Adobe does not guarantee the model's suitability for commercial applications — users should review the repository license and training dataset terms before any commercial deployment Find related tools.