Skip to content

racer: fill the gaps in GPUDirect model loading - #1009

Merged
Jordan Olshevski (jveski) merged 69 commits into
mainfrom
docs/racer-gpudirect-guide
Oct 28, 2026
Merged

Jordan Olshevski (jveski) merged 69 commits into
mainfrom
docs/racer-gpudirect-guide

Conversation

@jveski

@jveski Jordan Olshevski (jveski) commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Makes GPUDirect model loading with Racer and vLLM work out of the box. The released racer-dataplane image now includes CUDA and cuFile, and in the default auto mode it touches no GPU until the first GPU read, so one image serves GPU and CPU nodes. The operator now checks racer-dataplane-config, the webhook mounts the client socket into init containers, and GPU, cuFile, and landing metrics are exported. The racer-vllm plugin is now a single module, and the Python packages ship as wheels on each GitHub release instead of PyPI.

Covers node setup for GPUDirect Storage and GPUDirect RDMA, the CUDA
dataplane image, GPU config keys, warming, and how to verify each path.
Marks functional gaps as TODOs.
…uffers

rdma-verbs now holds several device pools per rail, keyed by an id (the
GPU ordinal). Each decrypting GPU's landing buffer is split into one
slice per rail, so every GPU can land over every rail. Rails on the
GPU's NUMA node get a slice first when slots are short. Landing buffers
are freed once the RDMA node closes and drains, and registered again at
the next activation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because an unpublished finding indicates that required GPU E2E runs can pass without exercising the dataplane when the socket is unset.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate a critical gap in signed release inventory and digest-pinned signature-verification coverage for the new racer-vllm image.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because checkpoint layout changes can bypass artifact pinning and silently mix new weights with previously fetched configuration and tokenizer files.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shard discovery mishandles formatted rank placeholders, omitting other ranks’ ETags and causing valid models to fail pin checks with ModelChangedError.

1 open issue

Review effort: Balanced

Comment thread clients/python/racer-vllm/src/racer_vllm.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Native CUDA/cuFile interfaces, cross-container GPU identity, and multi-GPU RDMA buffer lifetimes require hardware review because simulation and same-process tests cannot establish driver/NIC correctness or DMA safety.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@jveski Jordan Olshevski (jveski) changed the title racer: GPUDirect model loading guide and the missing GPU features racer: fill the gaps in GPUDirect model loading Oct 28, 2026
@jveski
Jordan Olshevski (jveski) added this pull request to the merge queue Oct 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 28, 2026
@jveski
Jordan Olshevski (jveski) added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit dca6c96 Oct 28, 2026
43 checks passed
@jveski
Jordan Olshevski (jveski) deleted the docs/racer-gpudirect-guide branch October 28, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants