Dynamic 6-DoF Volumetric Video Generation: Software Toolkit and Dataset

Mufeng Zhu, Yuan-Chun Sun, Na Li, Jin Zhou, Songqing Chen, Cheng-Hsin Hsu, Yao Liu
| Webpage | Full Paper |

This repository contains the official implementation for Dynamic 6-DoF Volumetric Video Generation: Software Toolkit and Dataset.

Neural-based approach dataset generation

Dependencies

Blender > 3.5
Numpy
Mathutils

This code is for Blender version > 4.0

For Blender version < 3.9

To import PLY into Blender correctly, follow the instructions from import ply as verts. If you install Blender through Snap in Ubuntu, try Overlay to get permission to modify the source files of Blender. Then modify

bpy.ops.wm.ply_import(filepath = mesh_path)

to

bpy.ops.import_mesh.ply(filepath=mesh_path, use_verts=True)

in initialization() function.

Set

bpy.context.scene.view_settings.view_transform = 'Raw'

Running a Script in Linux Terminal

To execute a script in the Linux terminal, use the following command format:

blender -b /path/to/.blend_file -P /path/to/script.py

Example command:

blender -b lego.blend -P lego_train.py

To generate datasets for point clouds, use train.blend.

Original dataset

Since we are unable to release the Blender models and point clouds, you can download the original datasets in the following links:

• 8iVFB
• vsenseVVDB2
• Synthetic Dynamic Scenes
  • Lego
  • Pig
  • Amy

Following the instructions above, you can generate datasets by yourself.

Generated dataset

As we mentioned in our paper, we generate the training and testing datasets for Blender models. We also generate the point cloud sequences for these models. You can find them here. Each folder contains two .zip files. "model.zip" includes training and testing images along with transform.json for training nerual-based methods. Format and coordinate convertion may required for latest NeRF models. "model_PLY.zip" contains the dynamic point cloud sequences.

Point cloud sequence dataset generation (Synthetic Dynamic Scenes)

If you want to generate the dataset by yourself, you can follow the README descriptions. And the codes to generate the dataset are all in ./SyntheticDynamicScenes.

Citation

@

@inproceedings{zhu2024dynamic,
      title        = {Dynamic 6-DoF Volumetric Video Generation: Software Toolkit and Dataset},
      author       = {Zhu, Mufeng and Sun, Yuan-Chun and Li, Na and Zhou, Jin and Chen, Songqing and Hsu, Cheng-Hsin and Liu, Yao},
      booktitle    = {2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP)},
      pages        = {1--6},
      year         = {2024},
      organization = {IEEE},
      url          = {https://6-dof-dynamic-content-software.github.io/}
    }