Details of finetuning an Image-to-Video model

[tedfeng424]

Really awesome work!

The Appendix of the paper mentions how to finetune an Image-to-Video model from T2V model. Similar to SVD, the noised version of the condition image is concatenated channel wise.

Is the condition image added to the model through other methods? (For instance, SVD also replaces the text embeddings with CLIP image embeddings, does CogVideoX I2V also replace the original text embeddings with some kind of image embeddings?)

Thank you.

[tedfeng424]

And just a follow-up question, approximately how much resources is needed to finetune a I2V model from a T2V model?

Thanks!

[tengjiayan20]

SVD replaces the text embeddings with CLIP image embeddings, then the model will no longer support text prompt input, which is something people don’t want to see.
As for the optimal training amount, we are still in the exploration stage.

[tedfeng424]

Thank you for the reply. And just to clarify, the first frame is only concatenated to the input channel wise after 3D VAE and nothing else?

[tengjiayan20]

Thank you for the reply. And just to clarify, the first frame is only concatenated to the input channel wise after 3D VAE and nothing else?

yes

[hw-liang]

Great work! I noticed in your implementation, to add the image condition, there are some options:

if self.noised_image_all_concat: image = image.repeat(1, x.shape[1], 1, 1, 1) else: image = torch.concat([image, torch.zeros_like(x[:, 1:])], dim=1)

By default, self.noised_image_all_concat = False. Did you also try setting self.noised_image_all_concat = True? How is the performance?