GitHub - zai-org/CogVideo: text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

CogVideo & CogVideoX

中文阅读

日本語で読む

Experience the CogVideoX-5B model online at 🤗 Huggingface Space or 🤖 ModelScope Space

📚 View the paper and user guide

👋 Join our WeChat and Discord

📍 Visit QingYing and API Platform to experience larger-scale commercial video generation models.

Project Updates

• 🔥🔥 News : 2025/03/24 : We have launched CogKit , a fine-tuning and inference framework for the CogView4 and CogVideoX series. This toolkit allows you to fully explore and utilize our multimodal generation models.

• 🔥 News : 2025/02/28 : DDIM Inverse is now supported in CogVideoX-5B and CogVideoX1.5-5B . Check here .

• 🔥 News : 2025/01/08 : We have updated the code for Lora fine-tuning based on the diffusers version model, which uses less GPU memory. For more details, please see here .

• 🔥 News : 2024/11/15 : We released the CogVideoX1.5 model in the diffusers version. Only minor parameter adjustments are needed to continue using previous code.

The CogVideoX1.5-5B series supports 10-second videos with higher resolution, and CogVideoX1.5-5B-I2V supports video generation at any resolution. The SAT code has already been updated, while the diffusers version is still under adaptation. Download the SAT version code here .

• 🔥 News : 2024/11/08 : We have released the CogVideoX1.5 model. CogVideoX1.5 is an upgraded version of the open-source model CogVideoX.

4090 GPU, cogvideox-factory , has been released. It supports fine-tuning with multiple resolutions. Feel free to use it!

• 🔥 News : 2024/10/13 : A more cost-effective fine-tuning framework for CogVideoX-5B that works with a single

click here to view it. More training details and a demo have been added. To see the demo, click here .

- 🔥 News : 2024/10/09 : We have publicly released the technical documentation for CogVideoX fine-tuning on Feishu, further increasing distribution flexibility. All examples in the public documentation can be fully reproduced.

• 🔥 News : 2024/10/10 : We have updated our technical report. Please

This model can take an image as a background input and generate a video combined with prompt words, offering greater controllability.

With this, the CogVideoX series models now support three tasks: text-to-video generation, video continuation, and image-to-video generation. Welcome to try it online at Experience .

• 🔥 News : 2024/9/19 : We have open-sourced the CogVideoX series image-to-video model CogVideoX-5B-I2V .

model CogVLM2-Caption , used in the training process of CogVideoX to convert video data into text descriptions, has been open-sourced. Welcome to download and use it.

• 🔥 2024/9/19 : The Caption

significantly optimized the model's inference performance, greatly lowering the inference threshold. You can run CogVideoX-2B on older GPUs like GTX 1080TI , and CogVideoX-5B on desktop GPUs like RTX 3060 .

Please strictly follow the requirements to update and install dependencies, and refer to cli_demo for inference code. Additionally, the open-source license for the CogVideoX-2B model has been changed to the Apache 2.0 License .

• 🔥 2024/8/27 : We have open-sourced a larger model in the CogVideoX series, CogVideoX-5B . We have

almost no loss.

• 🔥 2024/8/6 : We have open-sourced 3D Causal VAE , used for CogVideoX-2B , which can reconstruct videos with

**.

• 🔥 2024/8/6 : We have open-sourced the first model of the CogVideoX series video generation models, **CogVideoX-2B

the CogVideo branch). This is the first open-source large Transformer-based text-to-video generation model. You can access the ICLR'23 paper for technical details.

• 🌱 Source : 2022/5/19 : We have open-sourced the CogVideo video generation model (now you can see it in

Table of Contents

Jump to a specific section:

• Quick Start

• Prompt Optimization

• SAT

• Diffusers

• Gallery

• CogVideoX-5B

• CogVideoX-2B

• Model Introduction

• Friendly Links

• Project Structure

• Quick Start with Colab

• Inference

• finetune

• sat

• Tools

• CogVideo(ICLR'23)

• Citation

• Model-License

Quick Start

Prompt Optimization

Before running the model, please refer to this guide to see how we use large models like GLM-4 (or other comparable products, such as GPT-4) to optimize the model.

This is crucial because the model is trained with long prompts, and a good prompt directly impacts the quality of the video generation.

SAT

Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.

Follow instructions in sat_demo : Contains the inference code and fine-tuning code of SAT weights. It is recommended to improve based on the CogVideoX model structure. Innovative researchers use this code to better perform rapid stacking and development.

Diffusers

Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.

pip install -r requirements.txt

Then follow diffusers_demo : A more detailed explanation of the inference code, mentioning the significance of common parameters.

For more details on quantized inference, please refer to diffusers-torchao . With Diffusers and TorchAO, quantized inference is also possible leading to memory-efficient inference as well as speedup in some cases when compiled.

A full list of memory and time benchmarks with various settings on A100 and H100 has been published at diffusers-torchao .

Gallery

CogVideoX-5B

5b_1.mp4

5b_2.mp4

5b_3.mp4

5b_4.mp4

5b_5.mp4

5b_6.mp4

5b_7.mp4

5b_8.mp4

CogVideoX-2B

1.mp4

2.mp4

3.mp4

4.mp4

To view the corresponding prompt words for the gallery, please click here

Model Introduction

CogVideoX is an open-source version of the video generation model originating from QingYing . The table below displays the list of video generation models we currently offer, along with their foundational information.

Model Name CogVideoX1.5-5B (Latest) CogVideoX1.5-5B-I2V (Latest) CogVideoX-2B CogVideoX-5B CogVideoX-5B-I2V

Release Date November 8, 2024 November 8, 2024 August 6, 2024 August 27, 2024 September 19, 2024

Video Resolution 1360 * 768 Min(W, H) = 768 768 ≤ Max(W, H) ≤ 1360 Max(W, H) % 16 = 0 720 * 480

Number of Frames Should be 16N + 1 where N <= 10 (default 81) Should be 8N + 1 where N <= 6 (default 49)

Inference Precision BF16 (Recommended) , FP16, FP32, FP8*, INT8, Not supported: INT4 FP16*(Recommended) , BF16, FP32, FP8*, INT8, Not supported: INT4 BF16 (Recommended) , FP16, FP32, FP8*, INT8, Not supported: INT4

Single GPU Memory Usage

SAT BF16: 76GB diffusers BF16: from 10GB* diffusers INT8(torchao): from 7GB* SAT FP16: 18GB diffusers FP16: 4GB minimum* diffusers INT8 (torchao): 3.6GB minimum* SAT BF16: 26GB diffusers BF16 : 5GB minimum* diffusers INT8 (torchao): 4.4GB minimum*

Multi-GPU Memory Usage BF16: 24GB* using diffusers

FP16: 10GB* using diffusers

BF16: 15GB* using diffusers

Inference Speed (Step = 50, FP/BF16) Single A100: ~1000 seconds (5-second video) Single H100: ~550 seconds (5-second video) Single A100: ~90 seconds Single H100: ~45 seconds Single A100: ~180 seconds Single H100: ~90 seconds

Prompt Language English*

Prompt Token Limit 224 Tokens 226 Tokens

Video Length 5 seconds or 10 seconds 6 seconds

Frame Rate 16 frames / second 8 frames / second

Position Encoding 3d_rope_pos_embed 3d_sincos_pos_embed 3d_rope_pos_embed 3d_rope_pos_embed + learnable_pos_embed

Download Link (Diffusers) 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel

Download Link (SAT) 🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel SAT

Data Explanation

scheme has not been tested for actual memory usage on devices outside of NVIDIA A100 / H100 architectures. Generally, this scheme can be adapted to all NVIDIA Ampere architecture and above devices.

If optimizations are disabled, memory consumption will multiply, with peak memory usage being about 3 times the value in the table. However, speed will increase by about 3-4 times. You can selectively disable some optimizations, including:

• While testing using the diffusers library, all optimizations included in the diffusers library were enabled. This

pipe.enable_sequential_cpu_offload() pipe.vae.enable_slicing() pipe.vae.enable_tiling()

• For multi-GPU inference, the enable_sequential_cpu_offload() optimization needs to be disabled.

video quality loss, though inference speed will significantly decrease.

• Using INT8 models will slow down inference, which is done to accommodate lower-memory GPUs while maintaining minimal

We recommend using the precision in which the model was trained for inference.

• The CogVideoX-2B model was trained in FP16 precision, and all CogVideoX-5B models were trained in BF16 precision.

used to quantize the text encoder, transformer, and VAE modules to reduce the memory requirements of CogVideoX. This allows the model to run on free T4 Colabs or GPUs with smaller memory! Also, note that TorchAO quantization is fully compatible with torch.

compile , which can significantly improve inference speed. FP8 precision must be used on devices with NVIDIA H100 and above, requiring source installation of torch , torchao Python packages. CUDA 12.4 is recommended.

• PytorchAO and Optimum-quanto can be

increases by about 10%. Only the diffusers version of the model supports quantization.

• The inference speed tests also used the above memory optimization scheme. Without memory optimization, inference speed

refinement.

• The model only supports English input; other languages can be translated into English for use via large model

Friendly Links

We highly welcome contributions from the community and actively contribute to the open-source community. The following works have already been adapted for CogVideoX, and we invite everyone to use them:

• LeMiCa : a diffusion model inference acceleration solution developed by China Unicom Data Science and Artificial Intelligence Research Institute. By leveraging cache-based techniques and global denoising path optimization, LeMiCa provides efficient inference support for CogVideoX, achieving nearly 2.5x lossless acceleration while maintaining visual consistency and quality.

RIFLEx extends the video with just one line of code: freq[k-1]=(2np.pi)/(Ls) . The framework not only supports training-free inference, but also offers models fine-tuned based on CogVideoX.

By fine-tuning the model for just 1,000 steps on original-length videos, RIFLEx significantly enhances its length extrapolation capability.

• RIFLEx-CogVideoX :

CogVideoX architecture, supporting flexible resolutions and multiple launch methods.

• CogVideoX-Fun : CogVideoX-Fun is a modified pipeline based on the

supports more functional Web UIs.

• CogStudio : A separate repository for CogVideo's Gradio Web UI, which

framework, allowing you to easily deploy your own models or the latest cutting-edge open-source models with just one click.

• Xorbits Inference : A powerful and comprehensive distributed inference

CogVideoX into your workflow.

• ComfyUI-CogVideoXWrapper Use the ComfyUI framework to integrate

infrastructure for video generation, with full pipeline support and continuous integration of the latest models and techniques.

• VideoSys : VideoSys provides a user-friendly, high-performance

Space image provided by community members.

• AutoDL Space : A one-click deployment Huggingface

is a fine-tuned model based on CogVideoX, specifically designed for interior design.

• Interior Design Fine-Tuning Model :

on multiple GPU Clusters. xDiT supports real-time image and video generations services. cogvideox-factory : A cost-effective fine-tuning framework for CogVideoX, compatible with the diffusers version model.

Supports more resolutions, and fine-tuning CogVideoX-5B can be done with a single 4090 GPU.

• xDiT : xDiT is a scalable inference engine for Diffusion Transformers (DiTs)

structure, aimed at providing greater flexibility for keyframe interpolation generation.

• CogVideoX-Interpolation : A pipeline based on the modified CogVideoX

restructured the architecture, including text encoders, UNet, VAE, etc., enhancing computational performance while maintaining compatibility with open-source community models. The framework has been adapted for CogVideoX.

• DiffSynth-Studio : DiffSynth Studio is a diffusion engine. It has

• CogVideoX-Controlnet : A simple ControlNet module code that includes the CogVideoX model.

• VideoTuna : VideoTuna is the first repo that integrates multiple AI video generation models for text-to-video, image-to-video, text-to-image generation.

• ConsisID : An identity-preserving text-to-video generation model, bases on CogVideoX-5B, which keep the face consistent in the generated video by frequency decomposition.

• A Step by Step Tutorial : A step-by-step guide on installing and optimizing the CogVideoX1.5-5B-I2V model in Windows and cloud environments. Special thanks to the FurkanGozukara for his effort and support!

Project Structure

This open-source repository will guide developers to quickly get started with the basic usage and fine-tuning examples of the CogVideoX open-source model.

Quick Start with Colab

Here provide three projects that can be run directly on free Colab T4 instances:

CogVideoX-5B Text-to-Video Colab code.

• CogVideoX-5B-T2V-Colab.ipynb :

CogVideoX-5B Quantized Text-to-Video Inference Colab code, which takes about 30 minutes per run.

• CogVideoX-5B-T2V-Int8-Colab.ipynb :

CogVideoX-5B Image-to-Video Colab code.

• CogVideoX-5B-I2V-Colab.ipynb :

CogVideoX-5B Video-to-Video Colab code.

• CogVideoX-5B-V2V-Colab.ipynb :

Inference

common parameters. All of this is covered here.

• dcli_demo : A more detailed inference code explanation, including the significance of

Quantized model inference code that can run on devices with lower memory. You can also modify this code to support running CogVideoX models in FP8 precision.

• cli_demo_quantization :

• diffusers_vae_demo : Code for running VAE inference separately.

interpolation and super-resolution tools integrated.

• space demo : The same GUI code as used in the Huggingface Space, with frame

Since CogVideoX is trained on long texts, we need to transform the input text distribution to match the training data using an LLM. The script defaults to using GLM-4, but it can be replaced with GPT, Gemini, or any other large language model.

• convert_demo : How to convert user input into long-form input suitable for CogVideoX.

CogVideoX-2B / 5B model to generate videos. Similar to our Huggingface Space, you can use this script to run a simple web application for video generation.

• gradio_web_demo : A simple Gradio web application demonstrating how to use the

finetune

• finetune_demo : Fine-tuning scheme and details of the diffusers version of the CogVideoX model.

sat

improve based on the CogVideoX model structure. Innovative researchers use this code to better perform rapid stacking and development.

• sat_demo : Contains the inference code and fine-tuning code of SAT weights. It is recommended to

Tools

This folder contains some tools for model conversion / caption generation, etc.

• convert_weight_sat2hf : Converts SAT model weights to Huggingface model weights.

text.

• caption_demo : Caption tool, a model that understands videos and outputs descriptions in

Adapter in diffusers format.

• export_sat_lora_weight : SAT fine-tuning model export tool, exports the SAT Lora

Adapter.

• load_cogvideox_lora : Tool code for loading the diffusers version of fine-tuned Lora

open-source local large language model + Flux + CogVideoX.

• llm_flux_cogvideox : Automatically generate videos using an

Supported by xDiT , parallelize the video generation process on multiple GPUs.

• parallel_inference_xdit :

CogVideo(ICLR'23)

The official repo for the paper: CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers is on the CogVideo branch

CogVideo is able to generate relatively high-frame-rate videos. A 4-second clip of 32 frames is shown below.

cogvideo.mp4

The demo for CogVideo is at https://models.aminer.cn/cogvideo , where you can get hands-on practice on text-to-video generation. The original input is in Chinese.

Citation

🌟 If you find our work helpful, please leave us a star and cite our paper.

@article{yang2024cogvideox,

title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer},

author={Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others},

journal={arXiv preprint arXiv:2408.06072},

year={2024}

} @article{hong2022cogvideo,

title={CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers},

author={Hong, Wenyi and Ding, Ming and Zheng, Wendi and Liu, Xinghan and Tang, Jie},

journal={arXiv preprint arXiv:2205.15868},

year={2022}

}

Model-License

The code in this repository is released under the Apache 2.0 License .

The CogVideoX-2B model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License .

The CogVideoX-5B model (Transformers module, include I2V and T2V) is released under the CogVideoX LICENSE .

— 本文由 AI 根据公开来源辅助整理，命令、版本与许可证请在使用前到原始页面复核。