中文版

     

This repo contains source code (training / inference) of 3D diffusion model, pretrained weights and gradio demo code of our 3D mesh generation project, you can find more visualizations on our project page and try our demo and tutorial. If you have high-quality 3D data or some other ideas, we very much welcome any form of cooperation.

• Pretrained Models
• Gradio & Huggingface Demo
• Inference
• Training
• Data Prepration
• Video
• Acknowledgement
• Citation

git clone https://github.com/wyysf-98/CraftsMan.git

conda create -n CraftsMan python=3.10
conda activate CraftsMan
conda install -c pytorch pytorch=2.3.0 torchvision=0.18.0 cudatoolkit=11.8 && \
pip install -r docker/requirements.txt

We provide the training and the inference code here for future research. The latent set diffusion model is heavily build on the same structure of Michelangelo, which is based on a perceiver and with 104M parameters.

Currently, We provide the models with 4 view images as condition and inject camera information via ModLN to the clip feature extractor. We will consider open source the further models according to the real situation.

## you can just get the model using wget:
wegt https://huggingface.co/wyysf/CraftsMan/blob/main/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6/config.yaml
wegt https://huggingface.co/wyysf/CraftsMan/blob/main/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6/model.ckpt
wegt https://huggingface.co/wyysf/CraftsMan/blob/main/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae/config.yaml
wegt https://huggingface.co/wyysf/CraftsMan/blob/main/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae/model.ckpt

## or you can git clone the repo:
git lfs install
git clone https://huggingface.co/wyysf/CraftsMan

If you download the models using wget, you should manually put them under the ckpts/image-to-shape-diffusion directory.

We provide gradio demos with different text/image-to-MV diffusion models, such as CRM, Wonder3D and LGM. You can select different models to get better results. To run a gradio demo in your local machine, simply run:

python gradio_app.py --model_path ./ckpts/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae

To generate 3D meshes from images folders via command line, simply run:

python inference.py --input eval_data --device 0 --model ./ckpts/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae

You can modify the used mv-images model by:

python inference.py --input eval_data --mv_model 'ImageDream' --device 0  # support ['CRM', 'ImageDream', 'Wonder3D'] --model ./ckpts/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae

We use rembg to segment the foreground object by default. If the input image already has an alpha mask, please specify the no_rembg flag:

python inference.py --input 'apps/examples/1_cute_girl.webp' --device 0 --no_rembg --model ./ckpts/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae

If you have images from other views (left, right, bacj), you can specify images by:

python inference.py --input 'apps/examples/front.webp' --device 0 --right_view 'apps/examples/right.webp' --model ./ckpts/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6-aligned-vae

For more configs, please refer to the inference.py.

We provide our training code to facilitate future research. And we provide a data sample in data. For the occupancy part, you can download from Objaverse-MIX for easy use. For more training details and configs, please refer to the configs folder.

### training the shape-autoencoder
python launch.py --config ./configs/shape-autoencoder/l256-e64-ne8-nd16.yaml \
                 --train --gpu 0

### training the image-to-shape diffusion model
python launch.py --config .configs/image-to-shape-diffusion/clip-mvrgb-modln-l256-e64-ne8-nd16-nl6.yaml \
                 --train --gpu 0

We are diligently working on the release of our mesh refinement code. Your patience is appreciated as we put the final touches on this exciting development." 🔧🚀

You can also find the video of mesh refinement part in the video.

Q: Tips to get better results.

1. CraftsMan takes multi-view images as condition of the 3D diffusion model. By our experiments, compared with the reconstruction model like (Wonder3D, InstantMesh), our method is more robust to multi-view inconsistency. As we rely on the image-to-MV model, the facing direction of input images is very important and it always leads to good reconstruction.
2. If you have your own multi-view images, it would be a good choice to use your own images rather than the generated ones
3. Just like the 2D diffusion model, try different seeds, adjust the CFG scale or different scheduler. Good Luck.
4. We will provide a version that conditioned on the text prompt, so you can use some positive and negative prompts.

• Inference code
• Training code
• Gradio & Hugging Face demo
• Model zoo, we will release more ckpt in the future
• Environment setup
• Data sample
• Code for mesh refine

• Thanks to LightIllusion for providing computational resources and Jianxiong Pan for data preprocessing. If you have any idea about high-quality 3D Generation, welcome to contact us!
• Thanks to Hugging Face for sponsoring the nicely demo!
• Thanks to 3DShape2VecSet for their amazing work, the latent set representation provides an efficient way to represent 3D shape!
• Thanks to Michelangelo for their great work, our model structure is heavily build on this repo!
• Thanks to CRM, Wonder3D and LGM for their released model about multi-view images generation. If you have a more advanced version and want to contribute to the community, we are welcome to update.
• Thanks to Objaverse, Objaverse-MIX for their open-sourced data, which help us to do many validation experiments.
• Thanks to ThreeStudio for their great repo, we follow their fantastic and easy-to-use code structure!

CraftsMan is under AGPL-3.0, so any downstream solution and products (including cloud services) that include CraftsMan code or a trained model (both pretrained or custom trained) inside it should be open-sourced to comply with the AGPL conditions. If you have any questions about the usage of CraftsMan, please contact us first.

@misc{li2024craftsman,
title         = {CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner}, 
author        = {Weiyu Li and Jiarui Liu and Rui Chen and Yixun Liang and Xuelin Chen and Ping Tan and Xiaoxiao Long},
year          = {2024},
archivePrefix = {arXiv preprint arXiv:2405.14979},
primaryClass  = {cs.CG}
}