Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/garage-contract.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ name: Garage Contract
# rules and the status/health fields. The released versions gate pull
# requests; the main-v2 entry is informational (an upstream change there
# should be triaged, not block unrelated work).
#
# The replication contract runs the GarageReplication Job script in the
# operator's pinned rclone image with Docker, as the Job's non-root user on a
# read-only root filesystem: copy/sync semantics, the safety guards and their
# exit codes, the failure classification and the termination message.

on:
push:
Expand Down Expand Up @@ -70,5 +75,16 @@ jobs:
} >> "$GITHUB_ENV"
echo "### Garage contract: \`${image}\`" >> "$GITHUB_STEP_SUMMARY"

- name: Pull the replication image
run: |
set -euo pipefail
image="$(sed -n 's/^const DefaultReplicationImage = "\(.*\)"$/\1/p' internal/controller/garagereplication_job.go)"
test -n "${image}"
docker pull --quiet "${image}"
echo "### Replication image: \`${image}\`" >> "$GITHUB_STEP_SUMMARY"

- name: Run contract tests
env:
# Fail rather than skip when the container runtime is missing.
REPLICATION_CONTRACT_REQUIRED: "true"
run: make test-contract
4 changes: 2 additions & 2 deletions .github/workflows/helm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,8 @@ jobs:
grep -q -- 'port: 9443' <<<"$rendered"
grep -q -- '--cosi-namespace=shadows' <<<"$rendered"
webhook_render="$(sed -n '/kind: MutatingWebhookConfiguration/,$p' <<<"$rendered")"
test "$(grep -c '^ namespaceSelector:' <<<"$webhook_render")" -eq 14
test "$(grep -c ' - "shadows"' <<<"$webhook_render")" -eq 14
test "$(grep -c '^ namespaceSelector:' <<<"$webhook_render")" -eq 15
test "$(grep -c ' - "shadows"' <<<"$webhook_render")" -eq 15

cosi_rbac="$(helm template scoped charts/${{ env.CHART_NAME }} \
--namespace operators \
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/test-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,13 @@ jobs:
kind_config: ""
- name: layout
runner: ubuntu-latest
filter: "!(gateway || manager || webhooks || external-gateway || dual-version || manual-mode || management-handle || node-local-pools || factor-migration || garage-versions || k8s-discovery)"
filter: "!(gateway || manager || webhooks || external-gateway || dual-version || manual-mode || management-handle || node-local-pools || factor-migration || garage-versions || k8s-discovery || replication)"
kind_config: ""
# replication runs two GarageClusters and real rclone Jobs
# (GarageReplication) with the operator's --enable-replication.
- name: replication
runner: ubuntu-latest
filter: "replication"
kind_config: ""
- name: node-local-pools
runner: ubuntu-latest
Expand Down
55 changes: 55 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ UPSTREAM CODEBASE ../garage
| `GarageNode` | `gn` | Node layout (zone, capacity, gateway) |
| `GarageAdminToken` | `gat` | Admin API tokens |
| `GarageReferenceGrant` | `grg` | Cross-namespace access grants (v1beta1+) |
| `GarageReplication` | `grepl` | Cross-cluster object replication/migration (rclone Jobs, `--enable-replication`) |

### Development Commands

Expand Down Expand Up @@ -456,6 +457,60 @@ and a completed `LegacySTSMigrated` condition short-circuits future runs.

---

## Cross-cluster replication (GarageReplication)

Copies objects between two **independent** Garage clusters through S3, for
replication and migration. It is **not** a backup (no snapshots, no Archive
mode; maintainer decision on #580) and not Garage federation. Design record:
`docs/design/2026-11-09-garage-replication-design.md`; user guide:
`docs/how-to/replication.md`.

- Controller behind `--enable-replication` (`ENABLE_REPLICATION`, chart
`replication.enabled`, default off). It gates the controller, the Job
informer and the chart's `batch/jobs` + status RBAC; the webhook is always
registered and warns while the flag is off
(`GarageReplicationValidator.ControllerEnabled`).
- The API must be correct **without the webhook**: formats, enums, list keys
and the mode/mirror pairing are schema rules, pinned against a real API
server by `internal/controller/garagereplication_schema_test.go`. The webhook
only adds context checks (reference grant, destination exclusivity across
objects, implicit-namespace self-loop, cron).
- CEL on the shared `ClusterReference` (unbounded strings) blows the rule cost
budget; envtest's CRD install catches it. Keep such checks in Go.
Kubernetes 1.25 (the chart's smoke job) is stricter still: it costs CEL
string-extension functions such as `lowerAscii()` as unbounded, which the
1.36 envtest does not catch.
- `buckets[]` and `status.buckets` are list-maps keyed by `destination`; Jobs
carry the destination in an annotation and the controller keys everything
on it.
- Credentials Secrets must carry the label
`garage.rajsingh.info/replication-credentials: "true"`, GarageKey-generated
ones included (maintainer, #580: otherwise any GarageKey in the namespace can
be copied out), checked by the controller on every reconcile
(`Ready=False/CredentialsNotAllowed`): otherwise `create garagereplications`
would amount to `get secrets`. CA bundles come from ConfigMaps only.
- Destination exclusivity is cluster-wide, first come first served; the
webhook compares spec keys, the controller also resolved URLs
(`Ready=False/DestinationConflict`). Errors must not name an object in
another namespace.
- Holds (`status.buckets[].hold`) are released only by `trigger-sync`, never by
a spec edit. `AwaitingTrigger` gates the first Mirror run of a bucket whose
`lastMode` is not Mirror (new, or switched from Copy) and is dropped when the
mode goes back to Copy.
- The Job script classifies failures by rclone exit code first (3 missing
source bucket, 5 retryable/unreachable, 7 in Mirror = delete cap; measured
against rclone 1.75.2), log text only for exit 1 (auth vs `NoSuchBucket`).
rclone reads a bare size as KiB, so the controller appends `B`
(`rcloneSize`). Re-measure with the contract test when bumping the image
(Renovate does not automerge it).
- Contract test: `test/contract/replication_contract_test.go` runs the Job's
container with docker/podman (non-root, read-only rootfs, tmpfs work dir)
and reads the 0600 termination log through a root container. CI sets
`REPLICATION_CONTRACT_REQUIRED=true`. Kind e2e: `test/e2e/replication_e2e_test.go`,
shard `replication`.

---

## Configuration Reference

### GarageCluster Options
Expand Down
9 changes: 9 additions & 0 deletions PROJECT
Original file line number Diff line number Diff line change
Expand Up @@ -54,4 +54,13 @@ resources:
kind: GarageAdminToken
path: github.com/rajsinghtech/garage-operator/api/v1beta1
version: v1beta1
- api:
crdVersion: v1
namespaced: true
controller: true
domain: rajsingh.info
group: garage
kind: GarageReplication
path: github.com/rajsinghtech/garage-operator/api/v1beta1
version: v1beta1
version: "3"
6 changes: 3 additions & 3 deletions api/v1beta1/garagekey_webhook.go
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ func validateImportKey(ik *ImportKeyConfig) error {
return fmt.Errorf("importKey.secretRef.name is required")
}
for field, key := range map[string]string{
"accessKeyIdKey": ik.AccessKeyIDKey, "secretAccessKeyKey": ik.SecretAccessKeyKey,
accessKeyIDKeyField: ik.AccessKeyIDKey, secretAccessKeyKeyField: ik.SecretAccessKeyKey,
} {
if key != "" {
if problems := utilvalidation.IsConfigMapKey(key); len(problems) > 0 {
Expand Down Expand Up @@ -429,8 +429,8 @@ func validateSecretTemplate(template *SecretTemplate) error {
includeWebsiteURL := template.IncludeWebsiteURL != nil && *template.IncludeWebsiteURL
includeCredentialsFile := template.IncludeCredentialsFile != nil && *template.IncludeCredentialsFile
keys := []generatedKey{
{field: "accessKeyIdKey", value: defaultString(template.AccessKeyIDKey, defaultAccessKeyIDSecretDataKey), enabled: true},
{field: "secretAccessKeyKey", value: defaultString(template.SecretAccessKeyKey, defaultSecretAccessKeySecretDataKey), enabled: true},
{field: accessKeyIDKeyField, value: defaultString(template.AccessKeyIDKey, defaultAccessKeyIDSecretDataKey), enabled: true},
{field: secretAccessKeyKeyField, value: defaultString(template.SecretAccessKeyKey, defaultSecretAccessKeySecretDataKey), enabled: true},
{field: "endpointKey", value: defaultString(template.EndpointKey, "endpoint"), enabled: includeEndpoint},
{field: "hostKey", value: defaultString(template.HostKey, "host"), enabled: includeEndpoint},
{field: "schemeKey", value: defaultString(template.SchemeKey, "scheme"), enabled: includeEndpoint},
Expand Down
3 changes: 2 additions & 1 deletion api/v1beta1/garagereferencegrant_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ type ReferenceGrantFrom struct {
// Kind is the resource kind allowed to make cross-namespace references.
// GarageAdminToken remains in the schema for compatibility, but the static
// credential path is namespace-local and does not accept cross-namespace grants.
// +kubebuilder:validation:Enum=GarageKey;GarageBucket;GarageAdminToken
// GarageReplication may reference a GarageCluster in another namespace.
// +kubebuilder:validation:Enum=GarageKey;GarageBucket;GarageAdminToken;GarageReplication
// +required
Kind string `json:"kind"`

Expand Down
Loading
Loading