Skip to content

About

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Resources

Stars

363 stars

Watchers

1 watching

Forks

Latest commit

 

History

44 Commits

Folders and files

Repository files navigation



KandinskyLab Report Diffusers HF Demo ComfyUI vLL-Omni SGLang FastVideo

Kandinsky 6.0: A family of diffusion models for Video + Audio generation

We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (TI2AV) modes. A plug-in super-resolution model raises the output resolution to Full HD (1920×1080).

Project Updates

Quick start

An NVIDIA GPU. You need uv and just. Each path has its own environment and generate command:

Path Guide
Pipeline docs/quickstart/quick_start.md
vLLM-Omni docs/quickstart/quick_start_vllm.md
SGLang docs/quickstart/quick_start_sglang.md
FastVideo docs/quickstart/quick_start_fastvideo.md

The recipe index is docs/just.md. The default call is text to video with audio. One image starts from a reference frame, two images set the first and last frames, an audio file drives the video, and a video file drives the audio. The Python calls and just generate flags are in Generation modes.

git clone https://github.com/kandinskylab/kandinsky-6.git
cd kandinsky-6
just setup
just download pro-distill
just generate 'A cat is lying on a mat, turns toward the camera, opens its mouth and says <S>Hello<E>'

Performance

Working time (s) for a 5-second text-to-video clip with audio. Weight loading and MP4 encoding are excluded. The base clip is 480×864. Full HD super-resolved at scale 2.25. Compile is torch. Prompt expansion uses Qwen2.5.

Distilled checkpoints use PiFlow, 10 steps. The other checkpoints use 50 steps and MagCache.

The H100 column is just bench: FlashAttention 3, Qwen in BF16, module offload. Each cell is median of three generations.

RTX 4090, RTX 5090, and A100 use SageAttention and Qwen in BF16. Lite stays resident on the RTX 5090 and the A100, and uses module offload on the RTX 4090. Pro uses block offload on the RTX 4090 and the RTX 5090, and module offload on the A100. Each cell is one generation after warmup.

The RTX 5060 Ti uses SageAttention and Qwen in NF4. Lite stays resident. Pro uses block offload. Each cell is one generation after warmup.

RTX 4090 24 GB RTX 5060 Ti 16 GB RTX 5090 32 GB A100 80 GB H100 80 GB
Lite Distill SD 68.0 173.7 44.2 78.6 31.2
Lite Distill Full HD 208.9 608.4 140.3 207.1 78.0
Lite SD 267.9 834.9 201.3 371.6 143.2
Lite Full HD 408.7 1286.7 296.2 508.0 192.2
Pro Distill SD 235.0 717.6 174.5 236.0 87.8
Pro Distill Full HD 381.0 1153.0 269.5 375.4 133.2
Pro SD 955.4 3143.6 768.0 980.4 351.5
Pro Full HD 1096.7 3571.4 862.8 1109.8 399.7

Module breakdowns:

Community

About

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Resources

Stars

363 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages