-
Notifications
You must be signed in to change notification settings - Fork 590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
社区盼开源图生视频模型如久旱盼甘霖 | We really need an image2video CogVideoX! #182
Comments
Your work is very helpful. The earth and the sky will praise your generosity, and countless algorithm engineers will praise you, the selfless devotee, the great architect. |
I am also trying to port the train inpaint solution from the Open Sora Plan to CogVideoX. This solution is similar to EasyAnimate, but I hope the official team can provide a fine-tuning code for Diffusers, as modifying the code based on SAT is not very intuitive. The forward process involves different modules, which are all in different .py files. |
期待I2V,首尾帧,视频延长,我觉得我能做出一部大片。。。。。 |
2B版本不是已经可以图生视频了吗,我没有本地部署,但是我有使用官方的智谱APP试用了一下,由于效果不是太理想就没再用了。难道说开源的版本不能使用图生视频功能吗,只有API才可以用吗?我不是太清楚。另外发行介绍说3060就可以用5B版本,用试过的朋友了吗,效果如何 |
Yes, the open-source version does not support image2video at this moment. If there is an open-source strong I2V model, there will be a community finetuning it (like Stable Diffusion & LLaMA). I also tried OpenSoraPlan 1.2 and OpenSora 1.2 for image2video. Limited by resources, their models are not as strong as CogVideoX, though they are very good developers. |
我们收到了这个意见,我们会继续调研一下,感谢你们的支持 |
图生视频并没有开源,目前只是开源了t2v,但大部分人都是需要i2v保持视频的统一性和可控性,所以文生视频显得非常鸡肋,用处不大。 非常期待清影能开源i2v。 |
comfyui我们很快会支持 |
Feature request / 功能建议
希望能看到CogVideoX的图生视频版本!!!
Motivation / 动机
社区盼开源图生视频模型如久旱盼甘霖,只生成6s短视频的文生视频模型在生产中用处比较有限,用6s短视频表达清楚创作者的意思是困难的。
只有具备图生视频能力,才能在拼接多段短视频时保持人物和场景的一致性,才能创作出长视频。从文生视频到图生视频,在技术上只是小小一步,但对创作者而言是能用和不能用的区别。
至于训练生成更长短视频的模型(比如8秒,10秒)相对而言不是那么重要,一般一个镜头时长不会超过6秒。
Your contribution / 您的贡献
您发布初始版本以后,我可以试用或者提PR改进网络结构(如果您这边愿意提供微调数据)。
比如OpenSoraPlan为了实现根据前后帧inpaint中间帧,采用了如下的方式训练I2V模型:
![68747470733a2f2f7332312e617831782e636f6d2f323032342f30382f31322f7041705a5a4a662e706e67](https://private-user-images.githubusercontent.com/33491471/361825293-21122c1a-994a-4f1e-afe6-24aac9b6086b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyNjkxNjAsIm5iZiI6MTcyNTI2ODg2MCwicGF0aCI6Ii8zMzQ5MTQ3MS8zNjE4MjUyOTMtMjExMjJjMWEtOTk0YS00ZjFlLWFmZTYtMjRhYWM5YjYwODZiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAyVDA5MjEwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE5YzIyYTEzOGYwOTM0OWVjOWVkZWVkN2U3Yzk3YWIxYzA0NzhiYTQ2OTVlM2RjZTI5YjA5YzM1ZGU4ODA4MjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Uw0v5IXPu3I2mWqUsJ1TZ1FZ7xUo6cm-DoJnp3KX2QA)
The text was updated successfully, but these errors were encountered: