ControlNeXt is our official implementation for controllable generation, supporting both images and videos while incorporating diverse forms of control information. In this project, we propose a new method that reduces trainable parameters by up to 90% compared with ControlNet, achieving faster convergence and outstanding efficiency. This method can be directly combined with other LoRA techniques to alter style and ensure more stable generation. Please refer to the examples for more details.

📢 We are initially releasing our code with weights, and further details will be presented in our upcoming paper. Please stay tuned for updates.

This project is still undergoing iterative development. The code and model may be updated at any time. More information will be provided later.

• ControlNeXt-SVD [ Link ] : Generate the video controlled by the sequence of human poses. This can be seen as an attempt to replicate the implementation of AnimateAnyone. However, our model is built upon Stable Video Diffusion, employing a more concise architecture.

• ControlNeXt-SD1.5 [ Link ] : Controllable image generation. Our model is built upon Stable Diffusion 1.5. Fewer trainable parameters, faster convergence, improved efficiency, and can be integrated with LoRA.

• ControlNeXt-SDXL [ Link ] : Controllable image generation. Our model is built upon Stable Diffusion XL . Fewer trainable parameters, faster convergence, improved efficiency, and can be integrated with LoRA.

• ControlNeXt-SD3 [ Link ] : Stay tuned.

• ControlNeXt-CN [ Link ] : Stay tuned.

If you can't load the videos, you can also directly download them from here.