GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction

Sicheng Zuo^*, Wenzhao Zheng^*$\dagger$, Yuanhui Huang, Jie Zhou, Jiwen Lu

^* Equal contribution. $\dagger$ Project leader

GaussianWorld reformulates 3D occupancy prediction as a 4D occupancy forecasting problem conditioned on the current sensor input and propose a Gaussian World Model to exploit the scene evolution for perception.

To exploit the scene evolution for perception, we reformulate the 3D occupancy prediction as a 4D occupancy forecasting problem conditioned on the current visual input. We propose a Gaussian World Model (GaussianWorld) to explicitly exploit the scene evolution in the 3D Gaussian space and predict 3D occupancy in a streaming manner. Our GaussianWorld demonstrates state-of-the-art performance compared to existing methods without introducing additional computation overhead.

Follow instructions HERE to prepare the environment.

1. Download nuScenes V1.0 full dataset data HERE.

2. Download the occupancy annotations from SurroundOcc HERE and unzip it.

3. Download pkl files HERE.

4. Download the pretrained weights for the image backbone HERE and put it inside pretrain

Folder structure

GaussianWorld
├── ...
├── data/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── v1.0-test/
|   |   ├── v1.0-trainval/
│   ├── surroundocc/
│   │   ├── samples/
│   │   |   ├── xxxxxxxx.pcd.bin.npy
│   │   |   ├── ...
│   ├── nuscenes_temporal_infos_train.pkl
│   ├── nuscenes_temporal_infos_val.pkl
├── pretrain/
│   ├── r101_dcn_fcos3d_pretrain.pth

We provide the following checkpoints trained on the SurroundOcc dataset:

Evaluate the single-frame model GaussianFormer on the SurroundOcc validation set:

bash scripts/eval_base.sh config/nusc_surroundocc_base_eval.py out/ckpt_base.pth out/xxxx

Evaluate the streaming model GaussianWorld on the SurroundOcc validation set:

bash scripts/eval_stream.sh config/nusc_surroundocc_stream_eval.py out/ckpt_stream.pth out/xxxx

Train the single-frame model GaussianFormer on the SurroundOcc validation set:

bash scripts/train_base.sh config/nusc_surroundocc_base.py out/xxxx

Train the streaming model GaussianWorld on the SurroundOcc validation set:

bash scripts/train_stream.sh config/nusc_surroundocc_stream.py out/xxxx

Install packages for visualization according to the documentation.

Visualize the single-frame model GaussianFormer on the SurroundOcc validation set:

bash scripts/vis_base.sh config/nusc_surroundocc_base_visualize.py out/ckpt_base.pth scene-0098 out/xxxx

Visualize the streaming model GaussianWorld on the SurroundOcc validation set:

bash scripts/vis_stream.sh config/nusc_surroundocc_stream_visualize.py out/ckpt_stream.pth scene-0098 out/xxxx

Our work is inspired by these excellent open-sourced repos: TPVFormer PointOcc SelfOcc GaussianFormer SurroundOcc OccFormer BEVFormer

If you find this project helpful, please consider citing the following paper:

@article{zuo2024gaussianworld,
    title={GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction},
    author={Zuo, Sicheng and Zheng, Wenzhao and Huang, Yuanhui and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:2412.10373},
    year={2024}
}