I tried to train text to video and image to video models to generate 2D game animations. I used public game animation data and 3D mixamo model rendered animations to train the animation generation models. I'm open sourcing the model, training data, training code and data generation code. More details can be found in my blog here.

You can try the games here

a dino jumping cacti in the desert

Donald Trump jumping trash can in new york city

Harry Potter jumping tree in Hogwarts castle

Tylor Swift jumping microphone in hotel room

1. Clone the repository.

   git clone https://github.com/lyogavin/godmodeanimation.git

2. Install the necessary dependencies.

   cd godmodeanimation/i2v
   pip install -r requirements.txt

3. Prepare the datasets by unzipping the files and placing them in the /root/ucf_ds directory.
4. Download the pretrained model in the /root/vc2/model.ckpt directory.
5. Run the training script.

   train_t2v_run_jump.sh # for rum jump model
   train_t2v_spinkick.sh # for spin kick model
   train_t2v_sword_wield.sh # for sword wield model

DC I2V model is based on DynamiCrafter. Please follow the instructions in the DynamiCrafter repository to train the DC I2V model.

1. Install Python blender: See here for details.

2. Render the animation:

   cd godmodeanimation/data_generation
   python render_mixamo_videos.py # --help to see more options
   

We created Replicate public models for Sword Wield, Spin Kick, Run Jump and Run. You can try it from Replicat platform, find the details in Replicate.

If you find This work useful in your research and wish to cite it, please use the following BibTex entry:

@software{godmodeanimation2024,
  author = {Gavin Li},
  title = {2D Game Animation Generation: All you need is repeat the same motion 1000 times},
  url = {https://github.com/lyogavin/godmodeanimation/},
  version = {1.0},
  year = {2024},
}

Welcome contributions, ideas and discussions!

If you find this work useful or interesting to you, please ⭐ or buy me a coffee! It's very important to me! 🙏