Skip to content

feat(replication): add the GarageReplication controller - #590

Open
littlejo wants to merge 4 commits into
rajsinghtech:mainfrom
littlejo:replication-controller
Open

littlejo wants to merge 4 commits into
rajsinghtech:mainfrom
littlejo:replication-controller

Conversation

@littlejo

Copy link
Copy Markdown
Contributor

Summary

Adds the GarageReplication controller: one rclone Job per bucket and run,
behind --enable-replication (chart value replication.enabled, off by
default).

Stacked on #580 (the API). Only the last commit (feat(replication): add the GarageReplication controller) is new here; the two PRs are meant to merge
together.

It addresses the blocking points from the review of #580:

Review point Where
Credentials opt-in, GarageKey Secrets included Controller requires the garage.rajsingh.info/replication-credentials: "true" label on every credentials Secret, on every reconcile (Ready=False/CredentialsNotAllowed). CA bundles come from ConfigMaps only.
Hold released only by trigger-sync No spec edit clears status.buckets[].hold; only the annotation does.
Destination exclusivity, resolved URLs The controller re-checks after admission and also compares resolved URLs, so a clusterRef and an endpoint reaching the same cluster conflict (Ready=False/DestinationConflict). An object in another namespace is not named.
rclone in the contract job test/contract/replication_contract_test.go runs the Job's container in the pinned rclone image with Docker, as UID 65532 on a read-only root filesystem, against two Garage processes. The workflow sets REPLICATION_CONTRACT_REQUIRED=true, so it fails instead of skipping without a runtime.
Kind replication e2e test/e2e/replication_e2e_test.go, new replication shard (excluded from the layout catch-all): two GarageClusters, the not-reconciled warning, the opt-in, Copy, exclusivity, Mirror's first trigger, maxDeletes and empty-source holds with release, and the migration sequence (scheduled Copy, final Mirror + Check on a trigger, suspend).
Warn while disabled The webhook gets ControllerEnabled from --enable-replication.

Non-blocking items, all taken:

  • First Mirror run waits for a trigger (hold: AwaitingTrigger), also
    after switching an existing bucket from Copy to Mirror. This is tracked by
    the new status.buckets[].lastMode. Switching back to Copy drops the hold,
    since Copy never deletes.

  • Size-based delete cap: new optional mirror.maxDeleteSize, passed to
    rclone --max-delete-size. It fails with MaxDeletesExceeded and holds the
    bucket, like maxDeletes.

  • Classification by rclone exit code first, as measured against rclone
    1.75.2 and pinned by the contract test:

    rclone exit code Reason
    3 (missing source bucket) BucketNotFound
    5 (retryable errors left after the retries) EndpointUnreachable
    7 in Mirror (only --max-delete and --max-delete-size raise it) MaxDeletesExceeded

    Exit code 1 covers both a 403 and a missing destination bucket (a PUT
    answered NoSuchBucket), so the log decides only that case.

  • Typed status enums and list-maps (already in feat(replication): add the GarageReplication API #580).

  • Resolved URLs for the same-cluster self-loop and for destination
    conflicts.

API changes on top of #580:

  • status.buckets[].lastMode;
  • spec.mirror.maxDeleteSize;
  • SetupWebhookWithManager replaced by
    SetupGarageReplicationWebhookWithManager(mgr, controllerEnabled);
  • PROJECT lists controller: true again.

Fix found while testing: rclone reads a bare size as KiB, so
bandwidthLimit: "1024" would have meant 1 MiB/s. The controller now appends
B, and the docs say K/M/G are binary multiples.

Related issue or design

Stacked on #580. Design record:
docs/design/2026-11-09-garage-replication-design.md. Its "Review decisions"
section now covers both reviews.

Compatibility and operational impact

  • Off by default. Without --enable-replication:

    • no controller, no Job informer, no Job RBAC;
    • objects are validated but not reconciled, and admission warns about it.
  • RBAC with replication.enabled:

    • batch/jobs create/delete/get/list/watch;
    • garagereplications patch, to consume the trigger-sync annotation;
    • garagereplications/status get/update/patch.

    The rules are in both clusterrole.yaml and namespace-rbac.yaml, and the
    generated config/rbac/role.yaml stays the superset.

  • New flags: --enable-replication / ENABLE_REPLICATION and
    --replication-image / REPLICATION_IMAGE. Chart values
    replication.enabled and replication.image.

  • Image: docker.io/rclone/rclone:1.75.2, pinned by digest. A new Renovate
    regex manager tracks it with automerge off: the engine's flags, exit
    codes and log format are part of the safety contract.

  • Jobs:

    • run in the GarageReplication's namespace;
    • Pods are non-root, read-only root filesystem, no capabilities, no service
      account token, RuntimeDefault seccomp, so they pass the restricted Pod
      Security Standard (the e2e namespace enforces it);
    • credentials come only through secretKeyRef; the operator never reads
      Secret values.
  • Data safety:

    • nothing destructive happens without mode: Mirror plus a first
      trigger-sync;
    • deletes are capped by maxDeletes and optionally maxDeleteSize, and the
      empty-source guard refuses to empty a destination;
    • a tripped guard fails the Job without retry (podFailurePolicy) and holds
      the bucket;
    • deleting a GarageReplication never touches objects.
  • Rollback: turn replication.enabled off. Running Jobs finish or can be
    deleted, and no objects are touched.

Validation

Commands run:

  • make manifests generate api-reference and make verify-generate
  • go test $(go list ./... | grep -v '/e2e$') (envtest 1.36.2), including
    21 new controller unit tests
  • golangci-lint run ./... (v2.15.0), plus the garagecontract and e2e
    build tags on the new files
  • GARAGE_BIN=… REPLICATION_CONTRACT_REQUIRED=true go test -tags garagecontract ./test/contract/...
    (Garage v2.4.1, podman): all 20 contract tests, including the 15 replication
    cases
  • Schema suite (27 specs) and installation of every chart CRD against
    kube-apiserver 1.25.16
  • go vet -tags e2e ./test/e2e/

Not run:

  • The Kind replication e2e shard locally (no Kind on my machine); CI runs it.
  • helm lint / helm template locally (covered by the Helm workflow).
  • mkdocs build locally (covered by the docs workflow).

Checklist

  • The PR addresses one coherent problem and includes its necessary
    robustness fixes.
  • Tests cover the changed behavior.
  • User-facing docs and samples are updated where needed
    (docs/how-to/replication.md, Helm values, annotations and conditions,
    custom resources, design record).
  • Generated files were regenerated and committed where needed.
  • Release-only chart version fields are unchanged.

GarageReplication copies objects between two independent Garage clusters
through S3, for replication and migration. This change adds the API only:
the CRD, its schema validation and the admission webhook. The controller
that runs the rclone Jobs follows separately; until then objects are
accepted but not reconciled.

Following the review of rajsinghtech#580:

- Scope is replication and migration (Copy and Mirror). The Archive mode
  and the backup wording are removed.
- The API is correct without the webhook: bucket names, endpoint URLs,
  Secret and ConfigMap names, data keys, lengths, the mode/mirror pairing
  and unique destinations are schema rules, pinned against a real API
  server by an envtest suite.
- buckets[] and status.buckets are list-maps keyed by destination, which
  is now required. Status fields are typed enums.
- Credentials Secrets must opt in (GarageKey-generated, or labelled
  garage.rajsingh.info/replication-credentials: "true"); the controller
  enforces it. CA bundles come from ConfigMaps only.
- A safety hold (including AwaitingTrigger for a first Mirror run) is
  released only by the trigger-sync annotation, never by a spec edit.
- The webhook rejects a destination bucket already written by another
  GarageReplication on the same destination cluster, checks reference
  grants, implicit-namespace self-loops and cron expressions.
Kubernetes 1.25 costs CEL string-extension functions such as lowerAscii()
as unbounded, so the self-loop rule pushed the GarageReplication CRD past
the cost budget and the chart's 1.25 smoke install failed. Compare the
endpoint URLs exactly in CEL; the webhook still catches URLs that differ by
case, a trailing slash or a default port.
- Require the replication-credentials label on every credentials Secret,
  GarageKey-generated ones included: otherwise anyone allowed to create a
  GarageReplication could copy out another application's buckets with its
  key. A GarageKey opts in through spec.secretTemplate.labels.
- Warn on admission that the object is not reconciled while no replication
  controller runs (GarageReplicationValidator.ControllerEnabled).
- Do not name a GarageReplication of another namespace in the destination
  conflict error, and document that destination exclusivity is cluster-wide
  and first come, first served.
- Cap podTemplate.nodeSelector at 64 entries like labels and annotations.
- PROJECT: GarageReplication has no controller yet.
Run GarageReplications as one rclone Job per bucket and run, behind
--enable-replication (chart value replication.enabled, off by default).

- Controller: scheduled and triggered runs that never overlap, per-bucket
  status keyed by destination, Job history pruning, metrics. Credentials
  Secrets must carry the replication-credentials label
  (CredentialsNotAllowed otherwise); CA bundles come from ConfigMaps;
  destination exclusivity is re-checked on resolved URLs
  (DestinationConflict). Holds are released only by trigger-sync, and a
  bucket that has not run in Mirror mode yet waits for one
  (AwaitingTrigger).
- Job: non-root, read-only root filesystem, no capabilities, credentials
  through secretKeyRef only. The script classifies failures by rclone exit
  code first and reports a bounded termination message. New optional
  mirror.maxDeleteSize; sizes are bytes (rclone reads a bare number as KiB).
- The webhook warns while the controller is disabled.
- CI: the Garage Contract job runs the Job's container in the pinned rclone
  image against two Garage processes, and a Kind "replication" e2e shard
  runs two GarageClusters through Copy, Mirror, both guards and the
  migration sequence.
- Docs: how-to guide, Helm values, annotations and conditions, design
  record.
@littlejo
littlejo force-pushed the replication-controller branch from 2ba7256 to 2c089e1 Compare November 15, 2026 15:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant