racer: refactoring/DSTs for core Racer logic by jveski · Pull Request #1010 · Azure/unbounded

Jordan Olshevski (jveski) · 2026-10-27T23:50:47Z

Summary

Adds two crates under cmd/racer-dataplane:

racer-core (core/): Racer's read path, routing, cache, liveness, and admission rules as sans-IO code. No I/O, clocks, threads, or OS randomness. Callers pass time (time::Now) and an RNG (time::SplitMix64). core/clippy.toml and core/tests/purity.rs enforce this.
racer-sim (sim/): a deterministic discrete-event simulator. It runs the real node logic for up to 50,000 nodes in one thread, models the control plane, and checks invariants on every run.

The production dataplane and the simulator call the same racer-core code, so the simulator tests production behavior, not a model of it. Design doc: designs/racer-core-simulation.md.

Shared logic

Rule or state machine	Path	Production caller	Simulator caller
Page fill walk (`Fill`: owner targets, candidate walk, hedges, stale membership, fallback and waits, origin, publication)	`core/src/read/walk.rs`	`src/read/candidates.rs` (`CandidatePolicy::drive`)	`sim/src/node/world.rs` (`NodeWorld::drive`)
Singleflight steps (join gates, wait/elect, handover, publish, fail) and `PageFlights`	`core/src/read/flight.rs`	`src/read/flight.rs`	`sim/src/node/state.rs`
Budgets, route link limits, hot page shedding (`shed_hot_page`), helper fanout	`core/src/read/budget.rs`	`src/peer/server.rs`	`sim/src/node/world.rs`
Route rules (`decide_hop`, relay plans, `MAX_LINKS`)	`core/src/route.rs`	`src/peer/forwarding.rs`, `peer-wire` (`MAX_HOPS`)	every simulated hop
Serve, persist, response classes, origin authority	`core/src/read/mod.rs`	`src/read/`	`sim/src/node/world.rs`
Orphan notices and successor hints	`core/src/read/hints.rs`	`src/read/candidates.rs`, `src/read/flight.rs`	`sim/src/node/world.rs`
Silence, rank 0 watch, ping and offer caps	`core/src/liveness.rs`	`src/read/candidates.rs`	`sim/src/node/world.rs`
Caches (`TickLru`, `OrderedLru`, segment value, expiry)	`core/src/cache.rs`	`src/memory.rs`	per-node memory index
Admission floors (`Rules`)	`core/src/admission.rs`	`src/admission.rs`	`sim/src/node/state.rs`
Subscriptions scheduler	`core/src/subscriptions.rs`	`src/peer/subscriptions.rs`	-
Placement and routes (`Placement<R>`, `Paths<S>`)	`topology/`	`src/topology.rs`	phantom rankings and shared routes, `sim/src/phantom.rs`
Disk segments (`Segments`, `SegmentClock`)	`alloc/`	`src/store/`	`PhantomEntries` (feature `simulation`)
Control plane steps (seat, primary, admission, authz)	`cluster-control/src/step/`	cluster-control server	`sim/src/control.rs` (`ControlSim`)

The walk runs with Features::PRODUCTION in production (src/read/candidates.rs) and by default in the simulator.

Simplification pass

This revision rebases on main (after #1009) and simplifies the earlier version: 149 -> 117 files, net +41.8k -> +35.7k lines vs main.

One fill machine. racer_core::read::walk::Fill owns the whole acquisition (walk setup, probes, stale membership, hedges, fallback, origin). CandidatePolicy::drive in production and NodeWorld::drive in the simulator both feed it events. The old FillPlan/FillStep/WalkResult helpers and the sim-only SIM_* feature bits are gone, so the simulator now follows production's stale-membership semantics.
One flight table. The singleflight step functions in core/src/read/flight.rs are shared by production Flights and the simulator's PageFlights. The duplicate admission types (FlightAdmission, Unlimited, sim Admit/PageCharges) are removed.
Fewer pieces in racer-core. 34 -> 16 files. Single-use traits, facts structs, accessors and re-export shims are removed. subscriptions::Fault is a From<Kind> bound, and hot-page shedding (shed_hot_page) and the ping and offer caps are each decided in one place.
Sim-only code lives in the sim. The control plane cost model, Follower, phantom rankings, shared routes and the fake API server's rate limit are now in racer-sim. cluster-control/src/step/ went from 10 to 5 files. The simulator drives one ControlSim model on a single queue and RNG.
Tests. The walk tests are table-driven with golden probe traces, and flight, control and combined tests share fixtures. Scenario tests that ran the same simulation were merged, and every assertion was kept. Each config.node.* switch still has a test that turns it off and shows the failure.

Production behavior is unchanged. Hash-preserving refactors were checked against fixed trace hashes (15 scenarios x 2 seeds at 200 nodes). Where hashes moved, in the control and combined scenarios, every run still has zero hard invariant failures, and outcome metrics stay within ~5%.

Simulator performance (seed 1, same machine): zipf 5,000 nodes 134s -> 123s, 1.60 -> 1.66 GB RSS; storm 20,000 nodes 1067s -> 884s, 5.38 -> 5.40 GB; full-cold-start 5,000 nodes, interleaved A/B at similar load: 86.8s vs 87.8s user CPU, 0.80 -> 0.77 GB.

Running

make racer-sim ARGS='--scenario full-churn --nodes 200 --seed 7 --duration 300s'
make racer-sim ARGS='--scenario zipf --nodes 50000 --config cfg.json --out zipf.json'

Flags: --scenario, --nodes, --seed, --duration, --config (JSON, unknown keys are errors, "node" and "control" sub-objects), --out. RACER_SIM_TIMEOUT (default 7200s) bounds the target.

Scenarios:

Data plane: smoke, caches-only, zipf, storm, churn, faults, empty
Control plane: enrollment-storm, failover, ctl-churn, lagging
Combined: full-cold-start, full-failover, full-churn, full-lagging

Invariants

Data plane: budget_never_grows, route_within_hop_limit, relay_route_valid, only_owner_hits_origin, origin_within_scope, origin_for_live_waiter, origin_repeat_explained, membership_version_monotonic, flight_leader_valid.

Control plane: ctl_single_primary, ctl_terms_monotonic, ctl_sequence_monotonic, ctl_client_membership_monotonic, ctl_client_converges.

None is relaxed.

Determinism

All time comes from the event clock and all randomness from per-role SplitMix64 streams seeded from --seed. Every event feeds an order-sensitive trace hash. Same scenario, nodes, seed, duration, and config give the same hash and report. Replay tests: lib.rs replay_is_deterministic, control replays_exactly_and_seed_matters, combined replays_exactly, contention_scenarios same_seed_replays_and_seeds_differ and seed_matrix_replays_identical_reports.

Production fixes found with the simulator

Each fix has a switch under config.node (sim/src/config.rs). All are on by default, matching production. The named test turns the switch off and shows the simulator catches the problem.

fanout: readers of a hot page fetch from helpers instead of piling on its owners. hot_page_readers_fan_out_through_helpers, overloaded_hot_page_owners_do_not_stream_to_abandoned_requesters.
shed_busy: readers do not go back first to an owner that just shed them. readers_do_not_return_first_to_an_owner_that_shed_them.
flight_handover: an expired fill leader hands its fill to a live waiter. expired_leaders_hand_their_fill_to_later_waiters, expired_fills_cancel_their_origin_request.
silence, rank0_watch: hung peers are detected and used again after they recover. hung_peers_are_found_and_recovered_peers_are_used_again.
stale_fallback, stale_wait, stale_link_refund: readers handle owners that are behind on membership. readers_handle_owners_that_are_behind.
successor_hints: rank 0 asks a successor that recently missed it before going to origin. failover_readers_are_deferred_to_orphan_holders.
successor_targets: owners ask hinted later owners copy-only first. failover_owners_copy_orphans_from_successors.
orphan_notices, offer_cut_groups: owners above rank 0 offer origin copies to rank 0, and offers wait out a cut toward the target site. failover_holders_offer_orphans_to_rank_zero.
rank0_failure_hold: owners leave origin retries to a failing rank 0. owners_leave_origin_retries_to_a_failing_rank_zero.
lineage: walks ask past owners of pages that moved. lineage_targets_find_copies_of_pages_that_moved.
lineage_grants, demoted_targets, copy_join_rank: more moved copies are found. demoted_targets_and_lineage_grants_find_more_moved_copies.
parallel_granted, lineage_cap, unknown_age_targets: deep lineage probes in parallel find more copies without more latency. parallel_deep_lineage_probes_find_more_copies_without_more_latency.

Tests are in sim/tests/contention_scenarios.rs and sim/src/combined.rs. Other production rules from this work: new members join only after they serve peer links (mark_serving), and joins are batched by join_hold_secs.

Testing

cd cmd/racer-dataplane
cargo test --locked --workspace
cargo check --workspace --all-targets --all-features
cargo test -p racer-core -p racer-sim
make racer-sim ARGS='--scenario full-cold-start --nodes 2000'

racer-core unit tests cover the walk, flights, budgets, caches, liveness, and admission, plus core/tests/purity.rs and core/tests/lru_memory.rs.
racer-sim runs every small scenario in cargo test (smoke_scenarios_hold_invariants, control and combined scenario tests) with zero invariant failures.
sim/tests/phantom_memory.rs checks per-node phantom cache memory. empty_substrate_50k_rss checks the 50,000-node baseline.

Copilot

🟡 Changes recommended

Missing authorization and primary-role rechecks after write-slot waits can expose key material to deleted Pod or Node identities and let former primaries serve replica images.

3 open issues

What changed in this PR

This PR extracts shared Racer decision logic into racer-core and introduces a deterministic simulator targeting 50,000 nodes within 32 GiB. Production adapters use the shared rules, with related reliability fixes and regression coverage.

Changes:

Share read, routing, admission, and cache policies with explicit time and randomness.
Add simulator integration, memory measurements, and production/core equivalence tests.
Improve membership readiness, fill handover, peer recovery, and control-plane load handling.

File	Description
Makefile	Adds the `racer-sim` target.
cmd/racer-dataplane/topology/tests/phantom_memory.rs	Measures placement and route-cache memory.
cmd/racer-dataplane/topology/src/lib.rs	Exposes simulator APIs and tracks membership lineage.
cmd/racer-dataplane/src/telemetry.rs	Adds offer, successor, and silence metrics.
cmd/racer-dataplane/src/store/catalog.rs	Shares segment scoring and tests eviction equivalence.
cmd/racer-dataplane/src/runtime.rs	Adds explicit-time request checks.
cmd/racer-dataplane/src/retention.rs	Shares retention rules and accepts explicit time.
cmd/racer-dataplane/src/read/tests/persist_policy.rs	Tests persistence against core policy.
cmd/racer-dataplane/src/read/tests/fill.rs	Covers caller cancellation and copy-only joining.
cmd/racer-dataplane/src/read/range_stream.rs	Adapts budget handling to core errors.
cmd/racer-dataplane/src/read/metadata.rs	Uses shared refresh purposes and settlement rules.
cmd/racer-dataplane/src/read/fill.rs	Integrates shared policies, handover, and copy offers.
cmd/racer-dataplane/src/read/dispatch.rs	Delegates worker selection to core.
cmd/racer-dataplane/src/rdma/links/tests.rs	Tests restart recovery and setup cleanup.
cmd/racer-dataplane/src/rdma/links.rs	Adds abandoned setup cleanup.
cmd/racer-dataplane/src/peer/tests/opaque.rs	Tests transport-independent page streaming counts.
cmd/racer-dataplane/src/peer/tests.rs	Tests explicit-clock admission and link health.
cmd/racer-dataplane/src/peer/protocol.rs	Adds Ping/Offer modes and shared response types.
cmd/racer-dataplane/src/peer/forwarding.rs	Uses shared response-validation rules.
cmd/racer-dataplane/src/peer.rs	Integrates shared hop rules and silence tracking.
cmd/racer-dataplane/src/lib.rs	Removes the contention test-module declaration.
cmd/racer-dataplane/src/error.rs	Adds core error conversions.
cmd/racer-dataplane/src/control/session.rs	Defers rail reporting until listeners open.
cmd/racer-dataplane/src/config.rs	Uses shared admission limits and defaults.
cmd/racer-dataplane/src/app/tests/dst.rs	Improves fault targeting and restart setup.
cmd/racer-dataplane/src/app/tests.rs	Adjusts coordinator access for worker cleanup.
cmd/racer-dataplane/src/app/native.rs	Defers native activation until membership admission.
cmd/racer-dataplane/src/app.rs	Integrates readiness, silence, offers, and cleanup.
cmd/racer-dataplane/sim/src/node/mod.rs	Defines simulator node modules.
cmd/racer-dataplane/sim/src/node/ids.rs	Maps simulated identities to wire types.
cmd/racer-dataplane/sim/src/main.rs	Adds CLI execution and JSON reporting.
cmd/racer-dataplane/sim/Cargo.toml	Defines the simulator crate.
cmd/racer-dataplane/peer-wire/src/tests.rs	Tests wire route limits.
cmd/racer-dataplane/peer-wire/src/lib.rs	Shares the maximum route length.
cmd/racer-dataplane/peer-wire/src/forwarding.rs	Delegates route checks to core.
cmd/racer-dataplane/peer-wire/Cargo.toml	Adds the core dependency.
cmd/racer-dataplane/flow/src/subscriptions.rs	Generalizes waiter notifications for simulation.
cmd/racer-dataplane/flow/src/lib.rs	Exports the admission clock trait.
cmd/racer-dataplane/flow/src/admission.rs	Supports caller-provided clock implementations.
cmd/racer-dataplane/core/tests/purity.rs	Checks sources for forbidden runtime calls.
cmd/racer-dataplane/core/tests/lru_memory.rs	Measures payload-free cache memory.
cmd/racer-dataplane/core/src/time.rs	Adds explicit time and deterministic randomness.
cmd/racer-dataplane/core/src/read/worker.rs	Extracts canonical worker selection.
cmd/racer-dataplane/core/src/read/serve.rs	Defines serving and copy-join policies.
cmd/racer-dataplane/core/src/read/response.rs	Adds payload-generic peer responses.
cmd/racer-dataplane/core/src/read/policy.rs	Shares failure, retention, and persistence rules.
cmd/racer-dataplane/core/src/read/mod.rs	Exposes shared read modules.
cmd/racer-dataplane/core/src/read/error.rs	Defines core read errors.
cmd/racer-dataplane/core/src/peer/mod.rs	Exposes peer decisions and fault classes.
cmd/racer-dataplane/core/src/lib.rs	Defines the sans-IO crate surface.
cmd/racer-dataplane/core/src/cache/segment.rs	Extracts segment eviction scoring.
cmd/racer-dataplane/core/src/cache/mod.rs	Exposes cache policy modules.
cmd/racer-dataplane/core/clippy.toml	Disallows direct clocks and thread spawning.
cmd/racer-dataplane/core/Cargo.toml	Defines the shared core crate.
cmd/racer-dataplane/controlplane/tests/core.rs	Tests inline preparation behavior.
cmd/racer-dataplane/controlplane/src/published.rs	Adds recently superseded part lookup.
cmd/racer-dataplane/controlplane/src/lib.rs	Exports preparation mode selection.
cmd/racer-dataplane/controlplane/src/feed.rs	Supports inline preparation for simulation.
cmd/racer-dataplane/cluster-control/src/tests.rs	Marks fixture dataplanes as reported.
cmd/racer-dataplane/cluster-control/src/step/renew.rs	Extracts seat renewal decisions.
cmd/racer-dataplane/cluster-control/src/step/mod.rs	Exposes control-plane steps and jitter.
cmd/racer-dataplane/cluster-control/src/step/cost.rs	Models notification and write-queue costs.
cmd/racer-dataplane/cluster-control/src/step/contend.rs	Extracts primary contention decisions.
cmd/racer-dataplane/cluster-control/src/shared.rs	Adds notification, entropy, and authorization state.
cmd/racer-dataplane/cluster-control/src/server.rs	Queues poll writes and reduces bearer rechecks.
cmd/racer-dataplane/cluster-control/src/limits.rs	Adds authorization and write-queue limits.
cmd/racer-dataplane/cluster-control/src/lib.rs	Exposes simulation-related control modules.
cmd/racer-dataplane/cluster-control/src/kube/mod.rs	Distinguishes API throttling responses.
cmd/racer-dataplane/cluster-control/src/entropy.rs	Adds injectable entropy sources.
cmd/racer-dataplane/cluster-control/src/control.rs	Initializes new shared control state.
cmd/racer-dataplane/cluster-control/src/authority/publication.rs	Shares delta-versus-full response selection.
cmd/racer-dataplane/cluster-control/src/authority/keyring.rs	Accepts injected key-generation entropy.
cmd/racer-dataplane/cluster-control/src/authority/election.rs	Accepts explicit wall time.
cmd/racer-dataplane/cluster-control/src/annotations.rs	Adds committed-sequence notification metadata.
cmd/racer-dataplane/Cargo.toml	Registers core and simulator workspace members.
cmd/racer-dataplane/Cargo.lock	Records new crate dependencies.
cmd/racer-dataplane/alloc/tests/phantom_memory.rs	Measures phantom segment-entry memory.
cmd/racer-dataplane/alloc/src/retention.rs	Supports deterministic retention hashers.
cmd/racer-dataplane/alloc/src/lib.rs	Exports phantom segment entries.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

A critical cancellation race before driver submission can leave a retained flight and its admission charge unreleased, keeping the flight fenced and preventing drain from completing.

2 open issues

2 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Recovery pings can remain blocked by silent-peer routing or delayed by RDMA allocation, while canceling the last fill adopter can leave fetches and reservations active until a deadline or another I/O event.

1 open issue

Exempt bodyless pings from receive admission

1 resolved since last review

Complete retained operation when handover becomes stale

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because mixed-version ping handling and refreshed-owner ordering can misclassify healthy destinations and skip orphan-notice or failure-cooldown bookkeeping.

No open issues 🎉

1 resolved since last review

Exempt bodyless pings from receive admission

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because the flight join policy can bypass retained data and trigger unnecessary origin refetches when memory publication fails and no disk copy exists.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Cross-crate changes to authenticated peer modes, membership admission, and RDMA teardown require maintainer review and integrated transport, rolling-upgrade, and failure-recovery validation beyond source inspection.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Coupled changes to authenticated peer routing, managed-node admission, and RDMA teardown require human validation of mixed-version compatibility and runtime lifecycle invariants beyond source inspection and deterministic models.

No open issues 🎉

_{Review effort: Balanced}

A successor probe that hung held the requester waiting on this owner past its own attempt timeout (DST traffic churn oracle seed 7, 503). The probe now takes at most a fifth of the time left, and any error of it other than this request ending is a miss.

A younger hint names an owner whose own fetch is most likely still running, so the probe only missed: 5k zipf sent 131 successor probes and copied nothing (failed reads 21 to 29). With PROBE_AFTER zipf is unchanged and full-failover copies 31 of 33 probes.

…nd 212

Churn tests that ran the same (nodes, retained, seed) configurations now share one table-driven loop; the lagging-owner ablations share their default run; the two search-limit tests and the two expired-leader tests share their identical runs. Every assertion is kept.

The watch, silence and recovery tests ran the same hang setup three times, including one identical run twice. Every assertion is kept.

Merge option and duration parsing tests, and the smoke and per-scenario feature tests, which ran the same 200-node reports twice. build_config takes the config text directly; run reads the file.

The per-probe outcome strings and failed-fill dumps were ad-hoc debug output that cost a Vec<String> per fill. Note the in-flight ping and offer caps in the design doc.

shed_hot_page now takes the ranked owners, the local node, the requester and a lazy helper check, so the peer server and the sim no longer each derive owner and exempt from the first three ranked owners. Production decisions are unchanged.

…ge launch, table skip counters

… silence evidence test

…phase

…e helpers Remove ApiBudget::stats/qps()/try_reserve, admission::take/newer and Lottery::attempt(), and an empty if block in install_record. Tests assert the same rules through the remaining state.

The control sim used the fake API Store only for its token bucket and request counts. It now keeps a small ApiLimit, so Store loses the bucket, latency and stats and moves back into kube::fake under cfg(test). Drop unasserted control metrics and the image_bytes_per_member setting.

Production bootstraps are Slots::new, a WriteQueue with no waiters; the sim now models them the same way, so the separate Gate type goes.

…ts; table combined plans

No host overrides it outside its own test; feeds keep preparing on a thread as on main.

…m accessors Merge the API-limit and budget storms, and the queued and slow write runs, into one test each with every assertion kept. Combined reads the control engine directly instead of through pass-through accessors.

Production keeps these fields in shared::Serving; only the simulator used step::seat::Follower. Its tests move with it, and the step tests assert admit's rollback rule directly.

Fix stale paths and names after the racer-core simplification: shared shed_hot_page and ping/offer caps, PageFlights in racer-core, removed SIM_* divergence bits, current replay test names, API rate limit model.

The comment-condensing pass rewrote documentation that predates this PR. Restore the origin/main text verbatim where the code it describes is unchanged.

Bring back the rules the condensing pass truncated: copy-offer and miss-hint handling, hedge race preconditions, notice and liveness return semantics, handover ordering in Flights::retain. Drop the Known issue reference.

The comment-condensing pass cut doc blocks to their first sentences and dropped per-outcome rules (OrphanNotices::offer_result), gate orders (flight joins and waits), budget invariants (refunds never mint credits), route wire/search forms, and the AcquisitionBudget compile_fail doctest. Restore them concisely and fix intra-doc links to removed or private items so cargo doc is warning-free.

Restore the module overview and the classify_repeat doc (the condensing pass left the doc of an inlined helper on it), finish a truncated comment, drop a Known issue reference, and collapse a nested if and a needless reborrow.

Jordan Olshevski (jveski) requested review from a team and a balanced review from Copilot October 27, 2026 23:50

Copilot started reviewing on behalf of Jordan Olshevski (jveski) October 27, 2026 23:51

View session

github-advanced-security Bot found potential problems Oct 27, 2026

View reviewed changes

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread cmd/racer-dataplane/cluster-control/src/server.rs Outdated

Comment thread cmd/racer-dataplane/cluster-control/src/server.rs Outdated

Comment thread cmd/racer-dataplane/src/peer.rs

Jordan Olshevski (jveski) requested a balanced review from Copilot October 28, 2026 00:23

Copilot started reviewing on behalf of Jordan Olshevski (jveski) October 28, 2026 00:23

View session