add magic_window: segment people out of the camera onto a swappable backdrop#387
Open
salmanmkc wants to merge 15 commits into
Open
add magic_window: segment people out of the camera onto a swappable backdrop#387salmanmkc wants to merge 15 commits into
salmanmkc wants to merge 15 commits into
Conversation
New demo that uses MediaPipe ImageSegmenter (multiclass selfie model) to cut people out of the device-camera feed in real time and composite them onto a swappable backdrop, shown on a floating window in the scene. Per frame it grabs a camera snapshot, runs the segmenter to get a category mask, and composites in a shader (person pixels kept, background replaced with passthrough, a solid colour, or a gradient). Press B to cycle backdrops. A webcam fallback gives the simulator a real person to segment; on a headset the world-facing camera feeds it instead.
Replaces the keyboard-only backdrop cycling with a spatial control panel you can grab and place. Uses the uiblocks ManipulationBehavior card pattern with a tap-to-cycle backdrop button that updates its label in place, and pins the nested label/icon render order + depthTest so the text isn't occluded by its own backplate. Keyboard B still works as a fallback. Adds the uiblocks importmap entries (the @pmndrs/* uikit stack, signals, yoga) and the missing three/ subpath mapping uikit needs.
Collaborator
|
Love it! Thank you Salman!! |
Contributor
Author
my pleasure! thanks for the quick review! |
ruofeidu
reviewed
Jun 17, 2026
| // Webcam fallback for desktop / simulator mode. | ||
| // | ||
| // The SDK reads frames from `xb.core.deviceCamera.getSnapshot()`, which on a | ||
| // real Quest returns frames from the passthrough cameras. On desktop the |
Collaborator
There was a problem hiding this comment.
nit: avoid using Quest in the comments, but XR device
Contributor
Author
There was a problem hiding this comment.
sure will change it!
|
|
||
| function shouldUseWebcamFallback() { | ||
| // Only activate when the simulator addon is in charge (i.e. we're not in | ||
| // an immersive WebXR session). On a real Quest this returns false and the |
Collaborator
There was a problem hiding this comment.
nit: avoid using Quest in the comments, but XR device
| // If the user denies, the SDK keeps using the original `getSnapshot` | ||
| // (which returns blank/scene frames) and the demo degrades to "no face | ||
| // detected" without crashing. | ||
|
|
Collaborator
There was a problem hiding this comment.
you can have this for now.
Yet I remember we can simply set simulator webcam in the camera module.
the next camera is kind of broken --- I'll talk with Nels in the afternoon.
https://xrblocks.github.io/docs/templates/Camera/
Contributor
Author
There was a problem hiding this comment.
oo great thanks, looks like Nels did the fix #392 I will adjust it now
…ng getSnapshot
enableCamera('user') makes XRDeviceCamera go straight to the real webcam in
the simulator (facingMode 'user' bypasses the simulator camera), so the
getSnapshot monkeypatch in WebcamFallback is no longer needed. camera cycling
to the real device before permission was the blocker, fixed in google#392.
…stretched the old webcam fallback forced a 4:3 stream to match the 4:3 plane. the camera module defaults to 1280x720, so a 16:9 webcam was getting squished onto the 4:3 window. resize the feed plane and its border to the actual frame aspect (no-op for a 4:3 feed).
the camera module streams 1280x720 by default, so starting the window at 16:9 avoids a visible resize the moment the first frame lands. applyAspect_ still corrects to whatever the webcam actually delivers.
the dark frame was a solid plane behind the feed, so the passthrough backdrop discarded the feed background only to reveal the dark plane. build the frame as a rectangle with a feed-sized hole so the centre is empty and the real world shows through.
the mask stores category ids and the shader thresholds id >= 0.5. linear filtering interpolated between category ids, so a boundary against category 5 crossed the threshold much earlier than one against category 1, bleeding the silhouette unevenly. nearest keeps category edges crisp and correct.
add a dispose() (the Script teardown hook) that removes the keydown listener and releases the segmenter, textures, geometries and materials. also dispose the old mask DataTexture before allocating a new one on a resolution change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New demo. uses MediaPipe ImageSegmenter (the multiclass selfie model, which wasn't wired up anywhere in the SDK yet) to cut people out of the device-camera feed in real time and drop them onto a swappable backdrop, shown on a floating window in the scene.
per frame it grabs a camera snapshot, runs the segmenter for a category mask, and composites in a shader: person pixels kept, background replaced with passthrough (see straight through to the room), a solid colour, or a gradient. the camera comes straight from the SDK camera module (
enableCamera('user')), so the desktop simulator pulls the real front webcam and a headset uses its world-facing camera, no demo-side getSnapshot patching. the window sizes itself to the camera aspect so the feed isn't stretched.A draggable uiblocks control panel swaps backdrops, same ManipulationBehavior card + renderOrder/depthTest treatment as the other demos so the labels aren't occluded by their backplate. keyboard B cycles them too.
prototyped demo-first; the segmenter wrapper (
SegmenterController) is kept self-contained so it can be lifted into aworld/segmentationaddon later.Passthrough should work on XR devices (the window frame is a ring now so the cut-out shows the real world through the middle, not a dark backplate).
updated passthrough:
ignore the passthrough being black in the video below (older video)
imagesegment2.mp4