This repository contains a simulation environment for a Universal Robot (UR) manipulator arm while performing pick and place on a conveyor belt. The robot shares its space with a human worker. MoveIt's stock perception pipeline builds obstacles from an octomap, which is slow to update, determines the human as the same generic obstacle type as any other static obstacle in the scene, and only checks collisions at planning time. On top of that we track the human from pose keypoints and publish tight and loose obstacle zones straight into the planning scene, and monitor the trajectory while it's running so the robot can stop or slow down as the human moves, instead of relying on stale octomap data and planning-time checks alone.
Developed and tested on Ubuntu 22.04 with Docker installed.
CPU-only is fine: Gazebo rendering uses the iGPU/dGPU via /dev/dri (no CUDA), and human pose inference runs on CPU (device_name: cpu) by default. A modern multi-core CPU is recommended since Gazebo, motion planning, and pose inference all compete for CPU at the same time (tested on an Intel Core i9). No GPU/CUDA setup is required to run the project as-is; if you want GPU-accelerated pose inference instead, you'd need a CUDA-enabled onnxruntime/mmdeploy-runtime build and device_name: cuda in mission_planner/configs/hpe_params.yaml — not covered by the included Dockerfile.
If you don't already have the ROS workspace, create it and place this project in its src folder:
mkdir -p ~/workspaces/sim_ws/src
cd ~/workspaces/sim_ws/src
# copy/clone this project's contents into this directory
Build the image from the included Dockerfile — it installs ROS deps, ONNX Runtime, the human pose estimation Python deps, and compiles the full workspace (ur_moveit_config first, to take priority over the binary-installed version, then everything else), so the container is ready to launch immediately:
cd ~/workspaces/sim_ws/src
docker build -t hri_ros_gz -f Dockerfile .
Running the docker in the 'src' directory of a ROS workspace:
cd ~/workspaces/sim_ws/src
xhost +local:root
docker run --rm -it --net=host --device /dev/dri:/dev/dri --group-add video -e DISPLAY=$DISPLAY -e QT_X11_NO_MITSHM=1 -v /tmp/.X11-unix:/tmp/.X11-unix:ro --shm-size=1g -v "$PWD":/ws/src --workdir /ws hri_ros_gz bash
The container drops you straight into a shell with the workspace already sourced — skip straight to Run below.
src/ is bind-mounted live into the container, so editing source on the host doesn't update the baked-in build automatically. After editing, rebuild from inside the container:
colcon build --symlink-install
If you add/change a package dependency, also re-run rosdep install --ignore-src --from-paths src -y first.
Terminal 1 (the same shell from setup) — Gazebo + UR arm + MoveIt + cameras:
ros2 launch hri_simulation hri_sim.launch.py
Once Gazebo/MoveIt/RViz are fully up, open a second terminal into the same container for the perception + planning stack:
docker exec -it <container> bash
ros2 launch mission_planner sorting.launch.py