Unifying Voxel-based Representation with Transformer for 3D Object Detection

Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, Jiaya Jia

[ arXiv ] [ BibTeX ]

This project provides an implementation for the NeurIPS 2022 paper " Unifying Voxel-based Representation with Transformer for 3D Object Detection " based on mmDetection3D . UVTR aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection.

This project is based on mmDetection3D , which can be constructed as follows.

• Install PyTorch v1.7.1 and mmDetection3D v0.17.3 following the instructions .
• Copy our project and related files to installed mmDetection3D:

cp -r projects mmdetection3d/
cp -r extra_tools mmdetection3d/

• Prepare the nuScenes dataset following the structure .
• Generate the unified data info and sampling database for nuScenes dataset:

python3 extra_tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes_unified

You can train the model following the instructions . You can find the pretrained models here if you want to train the model from scratch. For example, to launch UVTR training on multi GPUs, one should execute:

cd
 /path/to/mmdetection3d
bash extra_tools/dist_train.sh 
${CFG_FILE}
 ${NUM_GPUS}

or train with a single GPU:

python3 extra_tools/train.py 
${CFG_FILE}

You can evaluate the model following the instructions . For example, to launch UVTR evaluation with a pretrained checkpoint on multi GPUs, one should execute:

bash extra_tools/dist_test.sh 
${CFG_FILE}
 ${CKPT}
 ${NUM_GPUS}
 --eval=bbox

or evaluate with a single GPU:

python3 extra_tools/test.py 
${CFG_FILE}
 ${CKPT}
 --eval=bbox

We provide results on nuScenes val set with pretrained models.

We would like to thank the authors of mmDetection3D and DETR3D for their open-source release.

UVTR is released under the Apache 2.0 license .

Consider cite UVTR in your publications if it helps your research.

@inproceedings{li2022uvtr,
  title={Unifying Voxel-based Representation with Transformer for 3D Object Detection},
  author={Li, Yanwei and Chen, Yilun and Qi, Xiaojuan and Li, Zeming and Sun, Jian and Jia, Jiaya},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
}