Kevin Chen, Christopher Choy, Manolis Savva, Angel Chang, Thomas Funkhouser, Silvio Savarese
Citing
If you find this code useful in your work, please cite us:
@article{chen2018text2shape,
title={Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings},
author={Chen, Kevin and Choy, Christopher B and Savva, Manolis and Chang, Angel X and Funkhouser, Thomas and Savarese, Silvio},
journal={arXiv preprint arXiv:1803.08495},
year={2018}
}
Code for the turking interface can be found here.
Data
The following data files can be downloaded from the project webpage and are required for running the code:
Due to the complicated and highly variable nature of the meshes collected in ShapeNet, a select few of the models in the ShapeNet dataset result in unexpected voxelizations. We have noted which of these models this phenomenon occurs in the text2shape data zip. If you find any additional such models, please report them to [email protected]. Thank you!
Third Party
Please download and build the SmartScenes Toolkit.
Under tools/scripts/render.sh, edit the $TOOLKIT_PATH to the path to the SmartScenes Toolkit.
Additionally, set up the configuration file in lib/config.py. The following need to be edited:
General
__C.DIR.DATA_PATH: Directory containing train/val/test splits and json.__C.DIR.TOOLKIT_PATH: Path to SmartScenes Toolkit.
ShapeNet Paths
__C.DIR.RGB_VOXEL_PATH: Directory containing the RGB voxelizations.__C.DIR.RAW_CAPTION_CSV: ShapeNet captions CSV.
Primitives Paths
__C.DIR.PRIMITIVES_RGB_VOXEL_PATH: Directory containing the RGB voxelizations and descriptions.
We store our voxelizations in NRRD format. We read the data in Python using pynrrd.
To visualize your voxels, you can visualize the NRRD using the ssc/render-voxels.js script from
SmartScenes Toolkit, or use your own method (e.g. save in a different format and visualize point
clouds, etc.).
# Train
./run_lba_encoder.sh 0 LBA1 'shapenet/encoder_logdir' 'train encoder on shapenet' '--dataset shapenet --validation --visit_weight 0.25 --learning_rate 2e-4 --lba_mode MM --num_epochs 100 --decay_steps 2500 --lba_test_mode shape --batch_size 100 --lba_unnormalize'
# Generate text and shape embeddings in a subdirectory under the model path called train/val/test
./tools/scripts/generate_text_embeddings.sh LBA1 outputs/shapenet/encoder_logdir/model.ckpt-50 '--dataset shapenet --visit_weight 0.25 --lba_mode MM --num_epochs 10000 --lba_test_mode text --lba_unnormalize'
# Train
./run_lba_encoder.sh 0 LBA1 'primitives/encoder_logdir' 'our full method on primitives' '--dataset primitives --validation --visit_weight 0.25 --learning_rate 2e-4 --lba_mode MM --num_epochs 100 --decay_steps 5000 --lba_test_mode shape --batch_size 100 --lba_unnormalize'
# Generate text embeddings
./tools/scripts/generate_text_embeddings.sh LBA1 outputs/primitives/encoder_logdir/model.ckpt-500 '--dataset primitives --visit_weight 0.25 --lba_mode MM --num_epochs 10000 --lba_test_mode text --lba_unnormalize'
# Generate shape embeddings
./tools/scripts/generate_text_embeddings.sh LBA1 outputs/primitives/encoder_logdir/model.ckpt-500 '--dataset primitives --visit_weight 0.25 --lba_mode MM --num_epochs 10000 --lba_test_mode shape --lba_unnormalize'
# Train
./run.sh 0 shapenet/cwgan_logdir "CWGAN1 (improved WGAN) on ShapeNet" "--model CWGAN1 --dataset shapenet --cfg ./cfgs/improved_wgan.yaml --shapenet_ct_classifier --learning_rate 5e-5 --queue_capacity 20 --noise_size 8 --uniform_max 0.5 --decay_steps 10000"
# Render
./tools/scripts/render.sh CWGAN1 outputs/shapenet/cwgan_logdir
# Train
python main.py --shapenet_ct_classifier --model Classifier128 --classifier --dataset shapenet --validation --log_path outputs/shapenet/shapenet_ct_classifier --batch_size 64
# Train on all splits
./run_classifier.sh 0 primitives/classifier128 'classifier on primitives dataset (full dataset with train/val/split)' '--dataset primitives'
# or run:
python main.py --model Classifier128 --batch_size 64 --num_epochs 10000 --learning_rate 1e-3 --decay_steps 10000 --log_path ./outputs/primitives/classifier128 --dataset primitives --validation --classifier