This repository is the pytorch implementation of our paper, GaussianGrasper.

GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping

Yuhang Zheng · Xiangyu Chen · Yupeng Zheng · Songen Gu · Runyi Yang · Bu Jin · Pengfei Li · Chengliang Zhong · Zengmao Wang · Lina Liu · Chao Yang · Dawei Wang · Zhen Chen · Xiaoxiao Long📧 · Meiqing Wang📧
Preprint, 2024 (📧 indicates corresponding authors)

The pipeline of our proposed method. (a) is our proposed pipeline where we scan multi-view RGBD images for initialization and reconstruct 3D Gaussian field via feature distillation and geometry reconstruction. Subsequently, given a language instruction, we locate the target object via open-vocabulary querying. Grasp pose candidates for grasping the target object are then generated by a pre-trained grasping model. Finally, a normal-guided module that uses surface normal to filter out unfeasible candidates is proposed to select the best grasp pose. (b) elaborates on EFD where we leverage contrastive learning to constrain rendered latent feature L and only sample a few pixels to recover features to the CLIP space via an MLP. Then, the recovered features are used to calculate distillation loss with the CLIP features. (c) shows the normal-guided grasp that utilizes Force-closure theory to filter out unfeasible grasp poses.

This reposity will be updated soon, including:

• Initialization.
• Uploading the Scanned Data.
• Uploading the Code for processing.
• Uploading the Code for reconstructing feature fields.
• Uploading the Code for grasping.

You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.7 of CUDA. You can find more information about installing CUDA here

GaussianGrasper requires python >= 3.8. We recommend using conda to manage dependencies. Make sure to install Conda before proceeding.

conda create --name gsgrasper -y python=3.8
conda activate gsgrasper

Install PyTorch with CUDA (this repo has been tested with CUDA 11.7) and tiny-cuda-nn. cuda-toolkit is required for building tiny-cuda-nn.

pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Easy option:

pip install nerfstudio

OR if you want the latest and greatest:

git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
pip install --upgrade pip setuptools
pip install -e .

pip install -r requirements.txt

python ./scripts/generate_data.py

We provide example scripts to train our model.

sh train.sh

We provide example scripts to render depth, normal and feature maps.

sh render.sh

We provide example scripts to update the scene.

sh update.sh

If you find our work useful in your research, please consider citing:

@article{zheng2024gaussiangrasper,
      title={GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping},
      author={Zheng, Yuhang and Chen, Xiangyu and Zheng, Yupeng and Gu, Songen and Yang, Runyi and Jin, Bu and Li, Pengfei and Zhong, Chengliang and Wang, Zengmao and Liu, Lina and others},
      journal={arXiv preprint arXiv:2403.09637},
      year={2024}
}

Our code is built on top of open-source GitHub repositories. We thank all the authors who made their code public, which tremendously accelerates our project progress. If you find these works helpful, please consider citing them as well.

nerfstudio-project/nerfstudio

graspnet/anygrasp_sdk