You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Benchmark of gantry P2P against gantry + Racer on 1497 AKS nodes, plus two fixes found along the way.
Changes
internal/gantry/racer: retry unavailable Racer reads with backoff and resume broken blob streams at the same offset and ETag. New racer_retry_budget setting (default 30s) and gantry_racer_mirror_retries_total{stage}. Without this, busy owners made containerd fall back to the origin: 34 of 1497 nodes finished an 8 GiB pull in 15 minutes.
cmd/racer-dataplane: recreate slab files whose size no longer matches RACER_SLAB_BYTES instead of crashing with InvalidConfiguration; delete slab files of workers that no longer exist.
hack/gantry-racer-bench: benchmark tools (origin registry, image generator, pull DaemonSet, disk prep, metrics and NIC stats).
Add an opt-in Slab::reset_mismatched_size that truncates and recreates a
regular slab file whose size differs from the configured capacity, after
validating every other dimension. Device-backed slabs are unaffected and
the default remains strict.
Open per-worker slab files with reset_mismatched_size so a changed
RACER_SLAB_BYTES recreates the file instead of failing startup with
InvalidConfiguration. Checkpoints recorded for the old geometry or worker
set are already skipped by recovery, so the node starts cold. Remove slab
files left by workers outside the current plan. Device mode is unchanged.
Retry Stat and Get on racersdk.ErrUnavailable with jittered exponential
backoff bounded by a per-request budget and the request context. After
headers are committed, resume a failed WriteTo from the exact byte count
accepted by the response, pinned to the original ETag, and abort if the
resumed metadata differs. Add gantry_racer_mirror_retries_total{stage}.
Expose the Racer mirror retry budget via racer_retry_budget,
GANTRY_RACER_RETRY_BUDGET, and --racer-retry-budget (default 30s, zero
disables) and report retries through gantry_racer_mirror_retries_total.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Retry handling can prematurely block blob resumption or double the backoff allowance, while benchmark scripts can hide failed uploads, accept incomplete metrics, and interrupt cleanup or disk preparation.
This PR improves Gantry’s Racer-backed image pulls and Racer’s slab-file startup recovery, alongside tools and results from a 1,497-node AKS benchmark.
Changes:
Adds configurable retries, version-pinned stream resumption, and retry metrics.
Recreates mismatched slab files and removes files belonging to obsolete workers.
Adds benchmark orchestration, image generation, disk preparation, and metrics analysis.
File
Description
internal/gantry/racer/retry_test.go
Tests retries, cancellation, stream resumption, and metrics.
internal/gantry/racer/racer.go
Implements retry policies, stream resumption, and counters.
internal/gantry/config/racer_test.go
Tests retry-budget configuration and validation.
internal/gantry/config/config.go
Exposes the retry budget through configuration, environment, and flags.
hack/gantry-racer-bench/slabwipe.yaml
Adds a manual slab-cleanup DaemonSet.
hack/gantry-racer-bench/scrape.sh
Collects fleet-wide Gantry and Racer metrics.
hack/gantry-racer-bench/RESULTS.md
Documents benchmark results, tuning, and observations.
hack/gantry-racer-bench/origin.yaml
Deploys benchmark registries and a tools pod.
hack/gantry-racer-bench/nicstats.py
Summarizes per-node network samples.
hack/gantry-racer-bench/genimage/main.go
Generates and uploads random OCI images.
hack/gantry-racer-bench/gantry-override.yaml
Pins the benchmark image and raises its retry budget.
hack/gantry-racer-bench/diskprep.sh
Prepares worker resource disks for Racer.
hack/gantry-racer-bench/bench.sh
Orchestrates preparation, image generation, pulls, and reporting.
cmd/racer-loadgen/gantry_test.go
Disables retries to preserve callback-count assertions.
cmd/racer-dataplane/src/app/tests.rs
Tests stale-worker slab-file cleanup.
cmd/racer-dataplane/src/app/recovery.rs
Adds tests for incompatible checkpoint geometry.
cmd/racer-dataplane/src/app.rs
Enables slab-size reset and stale-file pruning.
cmd/racer-dataplane/alloc/src/slab.rs
Implements and tests opt-in slab-file recreation.
cmd/gantry/agent_racer.go
Wires retry configuration and telemetry into the handler.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Critical slab-lifecycle flaws can unlink files still used by live workers and restore stale checkpoint references to discarded pages after a resize and rollback.
Earlier metrics scrape failures are silently ignored
hack/gantry-racer-bench/bench.sh:44
A failed metrics request for any replica except the last is silently ignored: the piped for loop returns the last request's status, so pipefail does not detect an earlier failure. cmd_run then records a partial total at lines 319 and 461-462, producing incorrect or even negative origin-egress deltas. Explicitly propagate each failed scrape instead of accepting a partial sum.
xargs runs annotation command when no disks are available
hack/gantry-racer-bench/bench.sh:263
If every worker reports DISK ... none ... (for example, a pool with no resource disks), this pipeline has no input, but xargs still invokes the annotation command once with sh as the node name. That failure exits cmd_diskprep under set -e before its summary and DaemonSet cleanup. Add -r so the supported no-disk case skips annotation.
Missing PodIPs are excluded from scrape completeness checks
hack/gantry-racer-bench/scrape.sh:23
A pod without a PodIP is silently excluded from the completeness check: the JSONPath at line 12 emits an empty line for it, the scrape loop skips it, and grep -c . excludes it from want. Consequently a pending or restarting pod can produce a successful but partial fleet total. Count every emitted pod row, including empty ones, so a missing IP fails the scrape.
Recovered plaintext reads remain incorrectly marked as aborted
internal/gantry/racer/racer.go:631
A successful resume after a short upstream read on the plaintext fast path can still be reported as aborted: ReadFrom sets racerResponseWriter.failed (racer.go:1075-1081), and WrapHTTP includes that sticky flag in HTTPObservation.Aborted (racer.go:865). Distinguish recovered source failures from terminal downstream failures, and assert Aborted == false in the HTTP resume test.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Scrape failures and pods without IPs can silently produce incomplete totals, while empty raw-disk input incorrectly invokes kubectl and breaks disk preparation for slab-file-only nodes.
Propagate scrape failures from all origin replicas
hack/gantry-racer-bench/bench.sh:44
When an earlier origin replica's scrape fails but the last succeeds, this helper still returns success when called through command substitution (bench.sh:319,461). This produces an incomplete ORIGIN_TX total and can make the pre/post delta misleading or negative. Explicitly propagate each scrape failure rather than relying on set -e.
Prevent xargs from invoking diskprep with empty input
hack/gantry-racer-bench/bench.sh:263
On a fleet with no usable raw disks, every record is filtered out, but xargs still invokes the shell once with no arguments. This attempts to annotate node sh with selector ^$, causing diskprep to fail instead of completing normally for slab-file-only nodes. Add -r, as the lease-pruning pipeline already does, so empty input does not invoke kubectl.
Fail when selected pods lack scrapeable IPs
hack/gantry-racer-bench/scrape.sh:23
A selected pod without a podIP produces a blank line in the IP file. Both the download loop and this nonblank-line count skip it, so the scrape succeeds with a partial fleet total during scheduling or restarts, despite the promised all-pod check. Count every emitted record: the JSONPath on line 12 emits one newline per selected pod, including pods that cannot yet be scraped.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to evaluate potentially destructive symlink-following cleanup, incorrect abort reporting after successful recovery, and silent benchmark failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Benchmark of gantry P2P against gantry + Racer on 1497 AKS nodes, plus two fixes found along the way.
Changes
internal/gantry/racer: retry unavailable Racer reads with backoff and resume broken blob streams at the same offset and ETag. Newracer_retry_budgetsetting (default 30s) andgantry_racer_mirror_retries_total{stage}. Without this, busy owners made containerd fall back to the origin: 34 of 1497 nodes finished an 8 GiB pull in 15 minutes.cmd/racer-dataplane: recreate slab files whose size no longer matchesRACER_SLAB_BYTESinstead of crashing withInvalidConfiguration; delete slab files of workers that no longer exist.hack/gantry-racer-bench: benchmark tools (origin registry, image generator, pull DaemonSet, disk prep, metrics and NIC stats).