🔥🔥 We release the codes, models and demos for MotionCtrl on Stable Video Diffusion (SVD).

Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.

More results are in showcase_svd and our Project Page.

conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt

• Run local inference script

1. Download the weights of MotionCtrl motionctrl_svd.pth and put it to ./checkpoints.
2. Running scripts: sh configs/inference/run.sh

• Run local gradio demo

  python -m app --share
  

❗❗❗ Noted ❗❗❗

1. If the motion control is not obvious, try to increase the speed in the run.sh or Motion Speed in the gradio demo.
2. If the generated videos are distored severely, try to descrease the speed in the run.sh or Motion Speed in the gradio demo. Or increase FPS.

If you make use of our work, please cite our paper.

@inproceedings{wang2024motionctrl,
  title={Motionctrl: A unified and flexible motion controller for video generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Li, Yaowei and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Ying},
  booktitle={ACM SIGGRAPH 2024 Conference Papers},
  pages={1--11},
  year={2024}
}

The current version of MotionCtrl is built on SVD. We appreciate the authors for sharing their awesome codebase.

For any question, feel free to email wzhoux@connect.hku.hk or zhouzi1212@gmail.com.