Repository navigation
Benchmark Fanout v2026.10 on throwaway Hetzner VMs with separate generators - #1
Merged
Merged
Conversation
Port the harness to the current Fanout release and make a short-lived Hetzner Cloud VM the standard publication host. - Start Fanout with its environment-only configuration: one FANOUT_ADDR listener, chat disabled, and a per-run FANOUT_METRICS_TOKEN for the private /metrics endpoint. Setup must return the ingest token. - Accept release versions with a maturity suffix, such as v2026.10.1-alpha. - Replace the removed observability reads with the operations the UI issues: a System overview refresh (batched panel queries plus annotations), an exemplar drill-down, and trace detail within the discovered trace's own window. Latency covers the whole operation, any failed request or panel fails it, and the sustained read gate fails when an operation kind never succeeds. - scripts/hetzner.sh creates one ccx33 VM, runs one or more `--`-separated benchmark groups on it, and always deletes it. Hetzner bills every started hour, so a publication run shares one VM between its ingest-only and read-under-write groups. - Reports record the host description and redact the SSH address from the reproduction command.
Bash defers a trap until the foreground child exits, so TERM on the script left the benchmark running and the VM billing until it finished. Run each group as a background child and wait for it; INT or TERM now forwards the signal, waits for the child, skips the remaining groups, and the existing cleanup deletes the VM.
Provisioned runs placed telemetrygen on the Fanout VM, so the generators took a large share of its CPUs and the result was a same-host ceiling. scripts/hetzner.sh now creates a private network with a Fanout-only VM (ccx33) and a generator VM (ccx43). The managed runner on the Fanout host binds Fanout to its private address and launches every telemetrygen process on the generator host over SSH, with a run-scoped key, keepalives, a client deadline and a remote timeout. A per-run sshd drop-in lifts MaxStartups for the largest phase's simultaneous launches. Reports record both hosts and the generator placement, and redact every address in the reproduction command. Cleanup removes the key, the drop-in, both VMs and the network, including after an interrupted create. Plain --host runs stay same-host.
ShellCheck before 0.11 reports functions invoked only by traps as unreachable (SC2317); 0.11 uses SC2329. Disable both on those functions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FANOUT_ADDR, private/metricswith a per-run token, the current setup and ingest-token flow, and release tags with a maturity suffix.scripts/hetzner.shandjust hetzner: throwaway Hetzner Cloud VMs that are always deleted, including when the script is stopped or a create is interrupted.telemetrygenprocess is launched on the generator VM over SSH with a run-scoped key, keepalives and deadlines. Reports record both hosts and redact every address.Verification
just test,go vet,bash -nand ShellCheck pass.