English | 中文
Synchronised egocentric capture: a head camera, five VIVE trackers and a pair of gloves recorded together on one host clock. Two independent glove rigs are supported:
| Rig | Glove data | Machines | Entry point |
|---|---|---|---|
| Wuji | 21-node skeleton + touch | Linux recorder + Windows tracker sender | record_ego_wuji.py (Linux), tracker_sender_windows.py (Windows) |
| MANUS | 21-node skeleton + joint angles | One Windows machine | record_ego_manus.py |
Both rigs store the hand skeleton as 21 nodes (MediaPipe topology), so the processing pipeline treats them the same way.
- How it works
- Files
- Wuji rig
- MANUS rig
- Recording
- Session output and timestamps
- Uploading data
- Replaying a session
- Troubleshooting
In the Wuji rig, Linux is the only capture coordinator:
Wuji gloves ─UDP/Zenoh─> Linux Wuji SDK ──────────────> gloves.npz
HDMI capture card ─────> Linux GoProHDMICamera ───────> MP4 + video.npz
HTC Vive Tracker ──────> Windows SteamVR/OpenXR ─TCP─> Linux NetworkTracker
├──> tracker.npz
└──> session.json
Clock sync. Windows sends its wall-clock time with every pose. Every 2 s the two machines exchange an NTP-style four-timestamp sample; Linux estimates the Windows − Linux offset from the lowest-RTT sample and maps tracker times onto the Linux wall clock, the same clock as the video and the Wuji t_host_us. Until the offset is known, the Linux receive time is used; it is also the fallback when the estimated offset jumps by more than 5 s.
The MANUS rig runs on one Windows machine: the MANUS sender pushes hand data over UDP, and the trackers are read locally through OpenXR.
capture/
├── record_ego_wuji.py # Wuji rig entry point (Linux)
├── record_ego_manus.py # MANUS rig entry point (Windows)
├── record_ego_app.py # orchestrates devices, web server and recording tasks
├── record_ego_page.html # live monitoring and control page
├── tracker_sender_windows.py # Windows OpenXR tracker sender
├── visualize_session.py # session replay
├── record_ego/ # config, device lifecycle, recording, upload, web server
├── devices/
│ ├── wuji_glove.py # Wuji SDK driver
│ ├── manus_glove.py # MANUS UDP receiver and packing
│ ├── network_tracker.py # Linux-side tracker TCP receiver
│ ├── vive_tracker.py # Windows OpenXR driver
│ └── gopro_camera.py # capture-card driver
├── network_bridge/protocol.py # TCP/MessagePack framing
├── ManusClient/ # MANUS UDP sender (download separately, see ManusClient/README.md)
├── wuji_calibration/ # import/export of Wuji calibration bundles (bundles are not committed)
├── scripts/
│ ├── setup_wuji_gloves.sh # Wuji network setup and checks on Linux
│ ├── wuji_calibration_tool.py # Wuji glove calibration, bundle export/import
│ ├── wuji_launch.sh # one-command start for the Wuji rig
│ ├── start_tracker_sender.ps1 # Windows tracker sender launcher
│ ├── rclone_linux.sh # bulk upload of data/ by day (Linux)
│ └── rclone_win.bat # bulk upload of data/ by day (Windows)
├── task_setting/example.json # example task list: a JSON array of strings
├── tests/ # unit tests
├── requirements.txt # recorder dependencies
└── requirements-tracker-windows.txt # Windows sender dependencies
All commands below are run from capture/.
Use Ubuntu 22.04 or later (as supported by the Wuji SDK) and Python 3.10/3.11.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txtWith the gloves, cables and USB Ethernet adapters connected, run the setup script. It skips the default-route and other in-use interfaces, probes each physical port for the left glove at 192.168.1.100 and the right glove at 192.168.1.101, creates persistent NetworkManager profiles and /32 glove routes, and finally scans, connects and disconnects through the Wuji SDK to verify.
./scripts/setup_wuji_gloves.sh # configure automatically
./scripts/setup_wuji_gloves.sh --check # check only, do not touch NetworkManager
./scripts/setup_wuji_gloves.sh --interface enx001 --interface enx002 # name the USB adapters when auto-detection is ambiguous
./scripts/setup_wuji_gloves.sh --expected-gloves 2 --allow-ufw # require both gloves; open the SDK's LAN UDP ports if UFW is on- The script never runs the
ip addr flushfrom the official example, so it cannot wipe addresses on your other interfaces; automatic mode also leaves alone any interface that carries the default route or another IPv4 subnet. - If the glove IPs were changed, pass the actual subnet with
--left-ip,--right-ipand--host-prefix. - See
./scripts/setup_wuji_gloves.sh --helpfor all options.
A bundle is everything the Wuji SDK holds for one SDK user — hand models, touch models and calibration records — so that every capture machine uses the calibration of the gloves attached to it. Close Wuji Studio first.
python scripts/wuji_calibration_tool.py users # list SDK users (* = current)
python scripts/wuji_calibration_tool.py create-user "rig-A" # create a user for this pair of gloves
python scripts/wuji_calibration_tool.py calibrate --user <user_id> # calibrate each glove, following the SDK prompts
python scripts/wuji_calibration_tool.py export --user <user_id> # export to wuji_calibration/calibration_bundle.zip
python scripts/wuji_calibration_tool.py preview # inspect the bundle without importing itYou can also calibrate in Wuji Studio and then export the same user. To set up another capture machine, copy the bundle into its wuji_calibration/.
On start-up the recorder imports wuji_calibration/calibration_bundle.zip and switches to the bundle's user. Use --wuji-calibration for a different path, or --no-wuji-calibration to skip the import and keep the SDK's current user.
python record_ego_wuji.py \
--task-setting task_setting/example.json \
--tracker-host 0.0.0.0 \
--tracker-port 8766 \
--tracker-token "<shared secret>"-
--tracker-host 0.0.0.0listens on every interface; find the Linux LAN IP withhostname -I. -
To keep the secret out of your shell history, set it as an environment variable instead:
export EGO_TRACKER_TOKEN="<shared secret>". -
The control page is at
http://127.0.0.1:8770/. To open it from another computer, add--host 0.0.0.0and allow the web port through the firewall. -
If UFW is enabled, allow the Windows machine to reach TCP 8766:
sudo ufw allow from <WINDOWS_IP> to any port 8766 proto tcp
Warning
Never expose port 8766 to the internet. Even on a trusted LAN, set a long --tracker-token.
Windows needs SteamVR set as the active OpenXR runtime, with a supported role assigned to every tracker. Install a separate environment:
python -m venv .venv-tracker
.\.venv-tracker\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements-tracker-windows.txtStart the Linux recorder first, then run either of these on Windows:
.\scripts\start_tracker_sender.ps1 -LinuxHost 192.168.1.20 -Port 8766 -Token "<shared secret>"
python tracker_sender_windows.py --host 192.168.1.20 --port 8766 --token "<shared secret>"- Host, port and token must match on both sides.
- If the connection drops, Windows reconnects every 2 s; the Linux recorder keeps running and keeps recording video and gloves.
- Windows Defender usually only needs to allow Python's outbound LAN connections; no inbound port is opened on Windows.
Send rate. Trackers are sent at up to 90 FPS by default (-SendFps / --send-fps). The cap matters: on some headless OpenXR runtimes wait_frame returns immediately, and without a limit the sender produces nearly a thousand duplicate poses per second, starving the Linux event loop that receives glove data and control events. Linux additionally caps the tracker buffer at 120 FPS by default (--tracker-record-fps) to stay compatible with older senders.
Recording from the Windows keyboard. The sender listens to the Windows terminal and controls the Linux recorder over the same authenticated TCP connection:
| Key | Action |
|---|---|
r |
Start recording (countdown from the Linux --record-countdown) |
e |
Stop and save |
Esc |
Stop listening to the keyboard; poses keep streaming |
Override the countdown from Windows with -Countdown 3 / --countdown 3, or disable keyboard control with -NoKeyboard / --no-keyboard. Setting the same non-empty token on both sides stops other devices on the LAN from sending control commands.
-
Download the MANUS UDP sender (
SDKMinimalClient_Windows.exeandManusSDK.dll) intoManusClient/. They are not distributed with this repository; seeManusClient/README.mdfor how to get them and for the packet format. -
Point the sender at the recorder's IP and port, then start the recorder:
python record_ego_manus.py \ --task-setting task_setting/example.json \ --manus-ip 0.0.0.0 \ --manus-port 9000
- The recorder receives MANUS's 25-node skeleton, drops the four non-thumb metacarpals and stores 21 nodes (the same layout as Wuji / MediaPipe), together with the joint angles. No touch channels are produced.
- Trackers are driven locally through
pyopenxr; pass--no-trackerto record without them — camera and gloves are unaffected. - See
python record_ego_manus.py --helpfor all options.
Tracker sampling. Local trackers are sampled on the host clock (--tracker-sample-fps, default 90; 0 = unlimited). Headless SteamVR may only produce compositor frames at ~15 Hz, but the trackers themselves report at 90+ Hz and xrLocateSpace can be queried at any time, so sampling is decoupled from the compositor. A full wait/begin/end frame runs only every 0.5 s to keep the session alive, and xrSyncActions is called only inside those keep-alive frames: outside a frame SteamVR blocks it until the next compositor frame (~66 ms measured), which would drag sampling back down to ~15 Hz.
Before you start
- Wuji: close Wuji Studio and check that skeleton and touch data look right. MANUS: check that the UDP sender is running.
- Start SteamVR and the Windows sender, and check that the top of the page shows the tracker as connected.
- Check the camera preview and the 3D tracker poses, then start recording.
The Wuji page shows the camera preview by default, limited to 640×360 (the recording keeps its full resolution); it can be hidden from the top-right corner.
Control-page keys
| Key | Action |
|---|---|
r |
Start a new session after the countdown |
e |
Stop and save the current session |
d |
Abort and discard the current session |
q |
Quit; any unsaved recording is discarded |
Common options
| Option | Default | Description |
|---|---|---|
--task-setting |
required | JSON array of task names |
--save-dir |
./data |
Root folder for sessions |
--record-countdown |
3 |
Countdown before recording, in seconds |
--host |
127.0.0.1 |
Address the control page listens on |
--cam-device |
0 |
Capture-card index or /dev/video* path |
--cam-width / --cam-height |
2560 / 1440 |
Requested resolution |
--cam-fps |
60 |
Requested capture frame rate |
--video-codec |
h264 |
h264 or mp4v |
--h264-preset |
ultrafast |
Software H.264 speed preset |
--h264-threads |
4 |
H.264 encoder threads; 0 = auto |
--no-camera |
off | Do not connect a camera |
--preview-width / --preview-height |
Wuji 640 / 360, MANUS 0 / 0 |
Maximum preview size; 0 = capture resolution; does not affect the recording |
--tracker-host |
0.0.0.0 |
Tracker TCP listen address (Wuji) |
--tracker-port |
8766 |
Tracker TCP port (Wuji) |
--tracker-token |
empty | Shared secret; or EGO_TRACKER_TOKEN |
--tracker-record-fps |
120 |
Tracker buffer rate cap; 0 = unlimited (Wuji) |
--no-tracker |
off | Record without trackers |
--wuji-calibration |
default bundle | Wuji calibration bundle path |
--no-wuji-calibration |
off | Skip the bundle and use the SDK's current user |
See python record_ego_wuji.py --help or python record_ego_manus.py --help for everything else.
data/session_YYYYMMDD_HHMMSS_TASK/
├── session_....mp4 # head-camera video
├── session_....video.npz # per-frame timestamps
├── session_....npz # gloves: skeleton, joint angles / touch
├── session_....tracker.npz # poses of the five trackers
└── session_....session.json # session metadata
- Video and Wuji
t_host_uscome straight from Linuxtime.time(). - Wuji subscribes only to the
tactileandhand_skeletonstreams the dataset uses, so the other ~120 Hz derived streams do not compete with video capture for CPU. - Video capture and encoding run on separate threads with a bounded queue. MP4 timestamps (variable frame rate) come from the per-frame
t_host_us, so gaps in the input never shorten playback. - Tracker send times come from Windows
time.time(). After correction they are written totracker_0__pose__t_host_us; the raw Windows wall clock is kept intracker_0__pose__t_source_host_usfor sync diagnostics. - Trackers have no hardware clock of their own, so
t_dev_usis-1andt_usfalls back to the correctedt_host_us. - Dropped tracker packets and the transport used are recorded in the session's tracker metadata.
Bulk upload. scripts/rclone_linux.sh and scripts/rclone_win.bat are thin wrappers around rclone copy --progress. Edit LOCAL_PATH (local folder) and REMOTE_PATH (remote bucket path) at the top of the script, then run it (bash scripts/rclone_linux.sh, or double-click scripts\rclone_win.bat). To filter files (--include) or delete local copies after upload (move), edit the single rclone command in the script.
Upload from the recorder. The recorder can also upload finished sessions in the background with rclone:
| Option | Description |
|---|---|
--rclone-remote |
Remote target, e.g. myremote:bucket/ego; or EGO_RCLONE_REMOTE |
--rclone-bin |
rclone executable, default rclone; or EGO_RCLONE_BIN |
--rclone-config |
Optional rclone.conf path; or EGO_RCLONE_CONFIG |
--storage-limit-gb |
Upload and clean up automatically once the data folder exceeds this size; default 50, 0 disables |
--storage-check-interval |
Minutes between size checks; default 30 |
python visualize_session.py data/session_YYYYMMDD_HHMMSS_TASKThe glove type is detected from session.json, the glove NPZ metadata and the node count: current MANUS and Wuji data both use 21 nodes, and older MANUS data is recognised by its 25 nodes. Both are aligned to video and trackers by t_host_us. The page title shows the detected type, node count and the streams actually recorded.
Options: --no-open, --port 8000, --max-fps 30, --build-only. Demo sessions can be generated for either glove:
python visualize_session.py /tmp/manus_demo --demo --demo-type manus
python visualize_session.py /tmp/wuji_demo --demo --demo-type wujiLinux keeps waiting for the tracker
- The
--hoston Windows must be the Linux LAN IP, not127.0.0.1. - Port and token must match exactly on both sides.
- Test from Windows:
Test-NetConnection <LINUX_IP> -Port 8766. - Check the Linux firewall, and that both machines are on a LAN / VLAN that can reach each other.
- Make sure the Linux receiver was started first.
No tracker poses on Windows
- Check that
requirements-tracker-windows.txtis installed. - SteamVR must be the active OpenXR runtime, with every tracker paired and assigned a role.
- The "Handheld object" role produces no usable pose; use waist, wrist, ankle or similar.
- The sender prints every serial and role it finds, which helps debug the SteamVR setup.
Linux cannot find the Wuji gloves
- Check that
wuji-sdkis installed on native Linux and that no other program holds the SDK. - Run
./scripts/setup_wuji_gloves.sh --checkto test interfaces, routes, ping and the SDK scan. - Check the left/right serial numbers and the current SDK user's calibration data.