[Project Page] | [Video (narrated)] | [Video (results only)] | [Paper] | [Arxiv]
conda env create -f environment.yml
conda activate 4DGen
pip install -r requirements.txt
# 3D Gaussian Splatting modules, skip if you already installed them
# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization
pip install ./simple-knn
# install kaolin for chamfer distance (optional)
# https://kaolin.readthedocs.io/en/latest/notes/installation.html
# CHANGE the torch and CUDA toolkit version if yours are different
pip install kaolin -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.1_cu116.htmlWe release our collected data in Google Drive.
Each test case contains two folders: {name}_pose0 and {name}_sync. pose0 refers to the monocular video sequence. sync refers to the pseudo labels generated by SyncDreamer.
We recommend using Practical-RIFE if you need to introduce more frames in your video sequence.
To preprocess your own images into RGBA format, one can use preprocess.py or preprocess_sync.py
# for monocular image sequence
python preprocess.py --path xxx
# for images generated by syncdreamer
python preprocess_sync.py --path xxxpython train.py --configs arguments/i2v.py -e rosepython render.py --skip_train --configs arguments/i2v.py --skip_test --model_path "./output/xxxx/"This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!
- https://github.com/dreamgaussian/dreamgaussian
- https://github.com/hustvl/4DGaussians
- https://github.com/graphdeco-inria/gaussian-splatting
- https://github.com/graphdeco-inria/diff-gaussian-rasterization
- https://github.com/threestudio-project/threestudio
If you find this repository/work helpful in your research, please consider citing the paper and starring the repo ⭐.
@article{yin20234dgen,
title={4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency},
author={},
journal={arXiv preprint: 2312.17225},
year={2023}
}}