Repository navigation
Conversation
GarageReplication copies objects between two independent Garage clusters through S3, for replication and migration. This change adds the API only: the CRD, its schema validation and the admission webhook. The controller that runs the rclone Jobs follows separately; until then objects are accepted but not reconciled. Following the review of rajsinghtech#580: - Scope is replication and migration (Copy and Mirror). The Archive mode and the backup wording are removed. - The API is correct without the webhook: bucket names, endpoint URLs, Secret and ConfigMap names, data keys, lengths, the mode/mirror pairing and unique destinations are schema rules, pinned against a real API server by an envtest suite. - buckets[] and status.buckets are list-maps keyed by destination, which is now required. Status fields are typed enums. - Credentials Secrets must opt in (GarageKey-generated, or labelled garage.rajsingh.info/replication-credentials: "true"); the controller enforces it. CA bundles come from ConfigMaps only. - A safety hold (including AwaitingTrigger for a first Mirror run) is released only by the trigger-sync annotation, never by a spec edit. - The webhook rejects a destination bucket already written by another GarageReplication on the same destination cluster, checks reference grants, implicit-namespace self-loops and cron expressions.
Kubernetes 1.25 costs CEL string-extension functions such as lowerAscii() as unbounded, so the self-loop rule pushed the GarageReplication CRD past the cost budget and the chart's 1.25 smoke install failed. Compare the endpoint URLs exactly in CEL; the webhook still catches URLs that differ by case, a trailing slash or a default port.
- Require the replication-credentials label on every credentials Secret, GarageKey-generated ones included: otherwise anyone allowed to create a GarageReplication could copy out another application's buckets with its key. A GarageKey opts in through spec.secretTemplate.labels. - Warn on admission that the object is not reconciled while no replication controller runs (GarageReplicationValidator.ControllerEnabled). - Do not name a GarageReplication of another namespace in the destination conflict error, and document that destination exclusivity is cluster-wide and first come, first served. - Cap podTemplate.nodeSelector at 64 entries like labels and annotations. - PROJECT: GarageReplication has no controller yet.
Run GarageReplications as one rclone Job per bucket and run, behind --enable-replication (chart value replication.enabled, off by default). - Controller: scheduled and triggered runs that never overlap, per-bucket status keyed by destination, Job history pruning, metrics. Credentials Secrets must carry the replication-credentials label (CredentialsNotAllowed otherwise); CA bundles come from ConfigMaps; destination exclusivity is re-checked on resolved URLs (DestinationConflict). Holds are released only by trigger-sync, and a bucket that has not run in Mirror mode yet waits for one (AwaitingTrigger). - Job: non-root, read-only root filesystem, no capabilities, credentials through secretKeyRef only. The script classifies failures by rclone exit code first and reports a bounded termination message. New optional mirror.maxDeleteSize; sizes are bytes (rclone reads a bare number as KiB). - The webhook warns while the controller is disabled. - CI: the Garage Contract job runs the Job's container in the pinned rclone image against two Garage processes, and a Kind "replication" e2e shard runs two GarageClusters through Copy, Mirror, both guards and the migration sequence. - Docs: how-to guide, Helm values, annotations and conditions, design record.
littlejo
force-pushed
the
replication-controller
branch
from
November 15, 2026 15:36
2ba7256 to
2c089e1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
GarageReplicationcontroller: one rclone Job per bucket and run,behind
--enable-replication(chart valuereplication.enabled, off bydefault).
Stacked on #580 (the API). Only the last commit (
feat(replication): add the GarageReplication controller) is new here; the two PRs are meant to mergetogether.
It addresses the blocking points from the review of #580:
garage.rajsingh.info/replication-credentials: "true"label on every credentials Secret, on every reconcile (Ready=False/CredentialsNotAllowed). CA bundles come from ConfigMaps only.trigger-syncstatus.buckets[].hold; only the annotation does.clusterRefand anendpointreaching the same cluster conflict (Ready=False/DestinationConflict). An object in another namespace is not named.test/contract/replication_contract_test.goruns the Job's container in the pinned rclone image with Docker, as UID 65532 on a read-only root filesystem, against two Garage processes. The workflow setsREPLICATION_CONTRACT_REQUIRED=true, so it fails instead of skipping without a runtime.replicatione2etest/e2e/replication_e2e_test.go, newreplicationshard (excluded from thelayoutcatch-all): two GarageClusters, the not-reconciled warning, the opt-in, Copy, exclusivity, Mirror's first trigger,maxDeletesand empty-source holds with release, and the migration sequence (scheduled Copy, final Mirror + Check on a trigger, suspend).ControllerEnabledfrom--enable-replication.Non-blocking items, all taken:
First Mirror run waits for a trigger (
hold: AwaitingTrigger), alsoafter switching an existing bucket from Copy to Mirror. This is tracked by
the new
status.buckets[].lastMode. Switching back to Copy drops the hold,since Copy never deletes.
Size-based delete cap: new optional
mirror.maxDeleteSize, passed torclone
--max-delete-size. It fails withMaxDeletesExceededand holds thebucket, like
maxDeletes.Classification by rclone exit code first, as measured against rclone
1.75.2 and pinned by the contract test:
BucketNotFoundEndpointUnreachable--max-deleteand--max-delete-sizeraise it)MaxDeletesExceededExit code 1 covers both a 403 and a missing destination bucket (a PUT
answered
NoSuchBucket), so the log decides only that case.Typed status enums and list-maps (already in feat(replication): add the GarageReplication API #580).
Resolved URLs for the same-cluster self-loop and for destination
conflicts.
API changes on top of #580:
status.buckets[].lastMode;spec.mirror.maxDeleteSize;SetupWebhookWithManagerreplaced bySetupGarageReplicationWebhookWithManager(mgr, controllerEnabled);PROJECTlistscontroller: trueagain.Fix found while testing: rclone reads a bare size as KiB, so
bandwidthLimit: "1024"would have meant 1 MiB/s. The controller now appendsB, and the docs say K/M/G are binary multiples.Related issue or design
Stacked on #580. Design record:
docs/design/2026-11-09-garage-replication-design.md. Its "Review decisions"section now covers both reviews.
Compatibility and operational impact
Off by default. Without
--enable-replication:RBAC with
replication.enabled:batch/jobscreate/delete/get/list/watch;garagereplicationspatch, to consume thetrigger-syncannotation;garagereplications/statusget/update/patch.The rules are in both
clusterrole.yamlandnamespace-rbac.yaml, and thegenerated
config/rbac/role.yamlstays the superset.New flags:
--enable-replication/ENABLE_REPLICATIONand--replication-image/REPLICATION_IMAGE. Chart valuesreplication.enabledandreplication.image.Image:
docker.io/rclone/rclone:1.75.2, pinned by digest. A new Renovateregex manager tracks it with automerge off: the engine's flags, exit
codes and log format are part of the safety contract.
Jobs:
account token, RuntimeDefault seccomp, so they pass the
restrictedPodSecurity Standard (the e2e namespace enforces it);
secretKeyRef; the operator never readsSecret values.
Data safety:
mode: Mirrorplus a firsttrigger-sync;maxDeletesand optionallymaxDeleteSize, and theempty-source guard refuses to empty a destination;
podFailurePolicy) and holdsthe bucket;
Rollback: turn
replication.enabledoff. Running Jobs finish or can bedeleted, and no objects are touched.
Validation
Commands run:
make manifests generate api-referenceandmake verify-generatego test $(go list ./... | grep -v '/e2e$')(envtest 1.36.2), including21 new controller unit tests
golangci-lint run ./...(v2.15.0), plus thegaragecontractande2ebuild tags on the new files
GARAGE_BIN=… REPLICATION_CONTRACT_REQUIRED=true go test -tags garagecontract ./test/contract/...(Garage v2.4.1, podman): all 20 contract tests, including the 15 replication
cases
kube-apiserver 1.25.16
go vet -tags e2e ./test/e2e/Not run:
replicatione2e shard locally (no Kind on my machine); CI runs it.helm lint/helm templatelocally (covered by the Helm workflow).mkdocs buildlocally (covered by the docs workflow).Checklist
robustness fixes.
(
docs/how-to/replication.md, Helm values, annotations and conditions,custom resources, design record).