🤖 OccRWKV

Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity

We will open source the complete code after the paper is accepted ！

• [2024/09.28]: OccRWKV's logs are available for download:

• [2024/09.15]: The pre-trained model can be downloaded at OneDrive.
• [2024/09.01]: 🔥 We released the code of OccRWKV. The First Receptance Weighted Key Value (RWKV)-based 3D Semantic Occupancy Network

OccRWKV introduces an efficient semantic occupancy prediction network that smartly decouples semantics and occupancy with specialized RWKV blocks, achieving superior accuracy and real-time performance for 3D semantic scene understanding. Its use of BEV space and sparse feature projection significantly reduces computational demands, making it highly suitable for enhancing real-time autonomous navigation in robotic systems.

@article{wang2024occrwkv,
title={OccRWKV: Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity},
author={Wang, Junming and Yin, Wei and Long, Xiaoxiao and Zhang, Xinyu and Xing, Zebing and Guo, Xiaoyang and Qian, Zhang},
year={2024}
      } 

Please kindly star ⭐️ this project if it helps you. We take great efforts to develop and maintain it 😁.

conda create -n occ_rwkv python=3.10 -y
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
pip install spconv-cu120
pip install tensorboardX
pip install dropblock
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cu121.html
pip install -U openmim
mim install mmcv-full
pip install mmcls==0.25.0

Please download the Semantic Scene Completion dataset (v1.1) from the SemanticKITTI website and extract it.

Or you can use voxelizer to generate ground truths of semantic scene completion.

The dataset folder should be organized as follows.

SemanticKITTI
├── dataset
│   ├── sequences
│   │  ├── 00
│   │  │  ├── labels
│   │  │  ├── velodyne
│   │  │  ├── voxels
│   │  │  ├── [OTHER FILES OR FOLDERS]
│   │  ├── 01
│   │  ├── ... ...

Clone the repository:

https://github.com/jmwang0117/OccRWKV.git

$ cd <root dir of this repo>
$ bash scripts/run_train.sh

$ cd <root dir of this repo>
$ bash scripts/run_val.sh

Since SemantiKITTI contains a hidden test set, we provide test routine to save predicted output in same format of SemantiKITTI, which can be compressed and uploaded to the SemanticKITTI Semantic Scene Completion Benchmark. You can provide which checkpoints you want to use for testing. We used the ones that performed best on the validation set during training. For testing, you can use the following command.

$ cd <root dir of this repo>
$ bash scripts/run_test.sh

Many thanks to these excellent open source projects:

• RWKV
• Vision-RWKV
• SSC-RS
• Co-Occ
• AGRNav
• HE-Nav