Our framework contains several stages:
- Preprocess: collect or estimate depths and camera information from images and videos
- Reconstruction: build physically interactable scenes from images and videos
- Motion: process trajectory and generate grasp poses
- Simulation: import physical scenes into the simulator, collect robotic demonstrations
- We will start with the provided examples in the
data/folder.
You can also prepare your own data following the same structure:
data/
key.jpg or .png # for single image input
key.mp4 # for video input
key.mp4 + key_depth.png # for video input with GT depths
and make sure the key is also in config/config.yaml.
- Make sure we have pulled or built the docker images following the docker setup:
python3 init.py setup
We will use openreal2sim/reconstruction:dev image for the reconstruction and openreal2sim/simulation:dev image for the IsaacLab simulation.
Every step below is one python3 init.py run <service> -- <command> call from the
repository root on the host: init.py starts the container, runs the command in it
and removes it. Add --gpus 0,1 to pick GPUs and --port host:container for the
GUI steps. python3 init.py shell reconstruction opens a bash prompt instead, if
you prefer to run several commands by hand.
Note: We have mounted several volumes so that checkpoints will be cached even if the container is removed. You can change the volume mappings by modifying the docker/compose.yml file.
The pretrained checkpoints are downloaded by python3 init.py setup, which runs
docker/installation/install.py in the reconstruction container for you.
Running this scripts for all preprocessing steps:
python3 init.py run reconstruction -- python openreal2sim/openreal2sim.py preprocess
This will provide estimated depths and camera intrinsics/extrinsics for the reconstruction stage.
We can check the point cloud reconstruction quality at outputs/{key}/geometry/dynamic_pcd.ply:
We first need to segment objects that needs to be reconstructed. We provide a GUI for this purpose, served on port 7860:
python3 init.py run reconstruction --port 7860:7860 -- python openreal2sim/gui/segmentation_annotator.py
How to use the GUI annotator:
-
Input
key(e.g.demo_image) in theOutput-keytextbox and pressloadto load image frames -
Select the objects you want to segment by simply clicking on the image
-
Modify the object name from the default
pc_objto the class name in thePoint-click nameand pressConfirm maskandSave mask_dict. -
If you are processing a video, press the
PROPAGATE & SAVEbutton to propagate the segmentation masks across frames.
Please note that we must have a ground mask annotated, since we need this to find the ground plane for reconstruction.
Example annotated masks are in outputs/{key}/annotated_images:
Then, run the whole physical scene reconstruction pipeline:
python3 init.py run reconstruction -- python openreal2sim/openreal2sim.py reconstruction
We will get a portable scene assets folder at outputs/{key}/simulation and camera & scene information at outputs/{key}/simulation/scene.json.
We can check the mesh reconstruction quality at outputs/{key}/simulation/scene_optimized.glb:
First, please run the motion processing pipeline.
python3 init.py run reconstruction -- python openreal2sim/openreal2sim.py motion.
We also provide a GUI for grasp pose annotation. Please follow the instructions on the interface to annotate the grasping pose for selected scene.
Please run python3 init.py run reconstruction --port 8000:5000 -- python openreal2sim/gui/gripper_pose_editor.py. This will open the 5000 port of the container and the 8000 port of the server.
How to use the GUI annotator:
-
Input
key(e.g.demo_video) in the blank textbox and pressLoad Sceneto load image frames -
Use the keyboard to move the gripper till a good pose.
-
Roll Down the left bar and press
Save Current Pose. -
You can press
Load Saved Graspsto visualize all the saved grasps and delete them in the left bar if necessary.
We support importing the reconstructed scenes into different physics simulators and collecting robotic trajectories by cross-embodiment transfer from videos. Please refer to the simulation part for more details.
Current supported physics simulators:
- IssacLab (IssacLab v2.0.2 & IssacSim v4.5.0)
- Maniskills
- Mujoco [WIP]
The simulation image is pulled by python3 init.py setup; build it only when you
change a Dockerfile. The packages are public, so the pull is anonymous and needs
no GitHub token or login.
On the host machine and before entering the container, run
xhost +local:
This is to allow the container to access the host's X server for IsaacSim GUI.
The simulation stage runs usd conversion, heuristic manipulation rollout and randomized generation.
Please run
python3 init.py run simulation -- python openreal2sim/steps/simulation/simulation.py
If you want to start from a given step, please add the --stage parameter, and choose among usd_conversion, sim_heuristic_manip and sim_randomize_rollout. --key processes a single key and --config points at another config file.
The tunable params are stored in openreal2sim/steps/simulation/isaaclab/envs/running_cfg.py, please refer to that file for detailed instructions.
Note: when running the above script for the first time, it may take quite a while to load IsaacSim. Just be patient.
You may observe something like this in the IsaacSim GUI:
and you can find the visuo-motor trajectories at outputs/{key}/demos/, as well as h5py/{key}/.
Note: The heuristic policy can be very un-stable, largely depending on the video object pose estimation quality. Please check the object pose estimation results at outputs/{key}/reconstruction/objects/*.mp4.
If the results are not good, a solution might be re-running the reconstruction stage from object pose estimation.
python3 init.py run reconstruction -- python openreal2sim/openreal2sim.py reconstruction --stage "scenario_fdpose_optimization"
by tuning the fdpose_est_refine_iter and fdpose_track_refine_iter in config/config.yaml.
You can also try selecting a new traj key. This requires going back to the motion stage and set the traj_key parameter in config.py.
init.py only assembles docker compose calls. The equivalent raw commands are:
docker compose -f docker/compose.yml pull reconstruction simulation
HOST_UID=$(id -u) HOST_GID=$(id -g) DOCKER_GID=$(getent group docker | cut -d: -f3) \
docker compose -p "$USER" -f docker/compose.yml run --rm --publish 7860:7860 reconstruction bash
HOST_UID=$(id -u) HOST_GID=$(id -g) docker compose -p "$USER" -f docker/compose.yml up -d simulation
HOST_UID=$(id -u) HOST_GID=$(id -g) docker compose -p "$USER" -f docker/compose.yml exec simulation bash
and then run the python commands above inside the container, dropping the
python3 init.py run <service> -- prefix.








