You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -19,6 +19,7 @@ Add first-class Multi-AZ support for the Kubernetes control plane LoadBalancer i
19
19
- Managing or provisioning DNS records.
20
20
- Provider-specific topologies such as ACTIVE_STANDBY across fault domains.
21
21
- Service type LoadBalancer for worker Services.
22
+
- Automatic creation of the network and subnets; the operator is responsible for creating the network and subnets before enabling multi-AZ LoadBalancer.
22
23
23
24
## User Stories
24
25
1) As a platform engineer, I want per-AZ LBs so a full AZ outage leaves the cluster reachable via DNS multi-A records that resolve to the remaining AZs.
@@ -48,130 +49,85 @@ flowchart LR
48
49
end
49
50
```
50
51
51
-
## Integration with External Global Server Load Balancing (GSLB)
52
-
53
-
External GSLB systems (e.g., Route 53 health-checked records, Akamai GTM, Cloudflare Load Balancing, NS1, F5 DNS/GTM) pair naturally with this Multi-AZ LB design:
54
-
55
-
- Clear targets: Each AZ has its own LB with a stable IP (floating IP or provider VIP) and deterministic name. These per-AZ endpoints are ideal GSLB health-check targets.
56
-
- Health-aware failover: GSLB continuously probes each per-AZ LB (TCP 6443 or an alternative port configured via additionalPorts) and automatically removes unhealthy AZ endpoints from DNS responses.
57
-
- Improved blast-radius isolation: An AZ outage only affects the corresponding AZ LB. GSLB maintains service by answering with remaining healthy AZ LB IPs.
- Same-region/same-AZ endpoints for lowest latency
60
-
- Spillover to other AZs only on failure
61
-
- Weighted distribution across AZs for capacity utilization
62
-
63
-
Recommended GSLB patterns
64
-
- Record model: Use a single control plane FQDN (the cluster’s spec.controlPlaneEndpoint.Host) and publish multiple A/AAAA records—one per AZ LB IP.
65
-
- Health checks:
66
-
- Protocol: TCP on the API port (default 6443). For providers that support L7 checks, TCP is generally sufficient for the Kubernetes API.
67
-
- Source IPs: Ensure GSLB checker IPs are permitted if using allowedCIDRs on listeners.
68
-
- TTL guidance:
69
-
- Use low TTL (e.g., 30–60s) to accelerate failover while balancing resolver load.
70
-
- Be aware that some clients cache beyond TTL; plan operationally for a brief grace period during failover.
71
-
- IP sourcing:
72
-
- Floating IPs typically simplify routing and are stable across LB re-creation.
73
-
- If using fixed VIPs (no floating), ensure they are routable to your GSLB health-check network and external resolvers that must reach them.
74
-
- Automation hooks:
75
-
- Deterministic LB naming (per-AZ suffix) and tags facilitate discovery by GSLB automation to register/update record sets.
76
-
- A controller or out-of-band job can list per-AZ LBs and synchronize GSLB records and health checks.
77
-
78
-
Failure scenarios and behavior
79
-
- Single AZ failure: The corresponding per-AZ LB becomes unhealthy; GSLB health checks fail; DNS answers exclude that AZ until recovery. Existing connections may break depending on client TCP retry behavior; new connections will target healthy AZs.
80
-
- Partial AZ degradation (e.g., only some members or monitor thresholds): Octavia monitor status influences LB health; ensure GSLB health thresholds align with Octavia monitor sensitivity to avoid premature removal or flapping.
81
-
- Network partitions from health-check vantage points:
82
-
- If GSLB checkers reside outside the cloud, confirm egress paths to per-AZ IPs and allowedCIDRs permit probes from those checkers.
83
-
- Consider diverse checker regions to avoid false positives due to upstream routing issues.
84
-
85
-
Operational considerations
86
-
- Access control: When using allowedCIDRs, include:
87
-
- Management cluster egress IPs (so CAPO can reconcile listeners/pools/monitors)
88
-
- Bastion/router IPs as needed for administration
89
-
- GSLB health-check source IP ranges
90
-
- Observability:
91
-
- Track per-AZ LB health and GSLB health check status together to diagnose discrepancies (LB marked healthy, but GSLB marks unhealthy often indicates ACL/routing issues).
92
-
- Multi-region future: This proposal focuses on multi-AZ within a region. If multi-region is introduced later, the same per-AZ model composes naturally: per-AZ LBs per region, with GSLB distributing across regions using latency- or geo-based policies and regional failover priorities.
93
-
94
-
This integration enables operators to achieve health-aware, low-latency, and failure-tolerant access to the Kubernetes API without CAPO managing DNS, while leveraging the explicit per-AZ LB separation for precise GSLB control.
95
-
96
52
## API Changes (additive)
97
53
98
54
All changes are confined to the OpenStackCluster API and are backward compatible. Proposed changes in:
- availabilityZoneSubnets []AZSubnetMapping (required to enable multi-AZ)
58
+
### Spec additions on `APIServerLoadBalancer`
59
+
-`availabilityZoneSubnets []AZSubnetMapping` (required to enable multi-AZ)
104
60
- Explicit mapping; each entry includes:
105
-
- availabilityZone string
106
-
- subnet SubnetParam
107
-
- The LB network MUST be specified when using this mapping via spec.apiServerLoadBalancer.network. Each mapped subnet MUST belong to that network.
108
-
- allowCrossAZLoadBalancerMembers *bool
109
-
- Default false.
110
-
- When true, register control plane nodes to all per-AZ LBs; otherwise same-AZ only.
111
-
- additionalPorts []int
61
+
-`availabilityZone string`
62
+
-`subnet SubnetParam`
63
+
- The LB network MUST be specified when using this mapping via `spec.apiServerLoadBalancer.network`. Each mapped subnet MUST belong to that network.
64
+
-`allowCrossAZLoadBalancerMembers *bool`
65
+
- Default `false`.
66
+
- When `true`, register control plane nodes to all per-AZ LBs; otherwise same-AZ only.
67
+
-`additionalPorts []int`
112
68
- Optional extra listener ports besides the Kubernetes API port.
113
-
- allowedCIDRs []string
69
+
-`allowedCIDRs []string`
114
70
- Optional VIP ACL list when supported by the Octavia provider.
115
71
116
72
Notes:
117
-
- The existing single-value availabilityZone field (if present) is treated as a legacy single-AZ shorthand; multi-AZ requires availabilityZoneSubnets.
73
+
- The existing single-value `availabilityZone` field (if present) is treated as a legacy single-AZ shorthand; multi-AZ requires `availabilityZoneSubnets`.
118
74
119
75
### Status additions
120
-
- apiServerLoadBalancers []LoadBalancer
121
-
- A list-map keyed by availabilityZone (kubebuilder listMapKey=availabilityZone).
- Set Octavia AvailabilityZone hint when supported by the provider.
157
-
- Create or adopt listeners, pools, and monitors for the API port and any additionalPorts.
112
+
- Set Octavia `AvailabilityZone` hint when supported by the provider.
113
+
- Create or adopt listeners, pools, and monitors for the API port and any `additionalPorts`.
158
114
- If floating IPs are not disabled, allocate and associate a floating IP to the LB VIP port when needed.
159
-
- Update or insert the AZ entry in status.apiServerLoadBalancers, including name, id, internalIP, optional ip, tags, allowedCIDRs, and loadBalancerNetwork.
115
+
- Update or insert the AZ entry in `status.apiServerLoadBalancers`, including `name`, `id`, `internalIP`, optional `ip`, `tags`, `allowedCIDRs`, and `loadBalancerNetwork`.
- When multi-AZ is enabled (availabilityZoneSubnets provided), rename legacy resources to the AZ-specific name for the first configured AZ, or adopt correctly named resources if they already exist.
- When multi-AZ is enabled (`availabilityZoneSubnets` provided), rename legacy resources to the AZ-specific name for the first configured AZ, or adopt correctly named resources if they already exist.
165
121
- Emit clear events and warnings; ensure idempotent operation.
166
122
167
123
### Member registration behavior
168
124
- Determine the machine failure domain (AZ) from the owning control plane machine.
169
-
- Default behavior: register the node only with the LoadBalancer whose availabilityZone matches the node’s AZ; if the legacy LB exists without an AZ, include it as a fallback.
170
-
- When allowCrossAZLoadBalancerMembers is true: register the node with all per-AZ LBs.
171
-
- Reconcile membership across the API port and any additionalPorts.
125
+
- Default behavior: register the node only with the LoadBalancer whose `availabilityZone` matches the node's AZ; if the legacy LB exists without an AZ, include it as a fallback.
126
+
- When `allowCrossAZLoadBalancerMembers` is `true`: register the node with all per-AZ LBs.
127
+
- Reconcile membership across the API port and any `additionalPorts`.
172
128
173
129
### Control plane endpoint
174
-
- Preserve a user-provided DNS in spec.controlPlaneEndpoint when set and valid.
130
+
- Preserve a user-provided DNS in `spec.controlPlaneEndpoint` when set and valid.
175
131
- Otherwise choose:
176
132
- The LB floating IP if present, else the VIP for an LB.
177
133
- If no LB host is available and floating IPs are allowed, allocate or adopt a floating IP for the cluster endpoint when applicable.
@@ -185,7 +141,7 @@ For each AZ in availabilityZoneSubnets:
185
141
186
142
## Example configurations
187
143
188
-
Explicit AZ→Subnet mapping (required for multi-AZ)
144
+
### Explicit AZ→Subnet mapping (required for multi-AZ)
- Default behavior remains single-LB when no multi-AZ mapping is provided.
249
186
- Enabling multi-AZ:
250
-
- Operators add availabilityZoneSubnets (and optionally additionalPorts, allowedCIDRs, allowCrossAZLoadBalancerMembers) and must specify the LB network.
187
+
- Operators add `availabilityZoneSubnets` (and optionally `additionalPorts`, `allowedCIDRs`, `allowCrossAZLoadBalancerMembers`) and must specify the LB network.
251
188
- Controller renames or adopts legacy resources into AZ-specific naming.
252
-
- status.apiServerLoadBalancers is populated alongside legacy status until further cleanup.
189
+
- `status.apiServerLoadBalancers`is populated alongside legacy status until further cleanup.
253
190
- Disabling multi-AZ:
254
191
- Remove the mapping; controller maintains single-LB behavior.
255
-
- Per-AZ LBs are not automatically deleted; operators may clean up unused resources.
192
+
- Per-AZ LBs are not automatically deleted to prevent accidental data loss and to allow operators to gracefully migrate traffic before cleanup. Operators are responsible for manually cleaning up unused LoadBalancer resources.
- Mapping/network mismatches: reject with clear validation messages; enforce via CRD CEL where feasible and in-controller checks.
@@ -282,15 +219,15 @@ Test code locations:
282
219
283
220
## Rollout plan
284
221
1) API and CRD changes:
285
-
- Add new fields and list-map keyed status to OpenStackCluster types in [api/v1beta1/](api/v1beta1/).
286
-
- Update CRDs in [config/crd/bases/](config/crd/bases/) and patches in [config/crd/patches/](config/crd/patches/).
222
+
- Add new fields and list-map keyed status to OpenStackCluster types in [api/v1beta1/](../../api/v1beta1/).
223
+
- Update CRDs in [config/crd/bases/](../../config/crd/bases/) and patches in [config/crd/patches/](../../config/crd/patches/).
287
224
2) Controller implementation:
288
-
- VIP network/subnet resolution and explicit AZ mapping in [controllers/openstackcluster_controller.go](controllers/openstackcluster_controller.go).
289
-
- Per-AZ LB reconciliation, rename/adoption, member selection, and optional floating IPs in [pkg/cloud/services/loadbalancer/](pkg/cloud/services/loadbalancer/).
225
+
- VIP network/subnet resolution and explicit AZ mapping in [controllers/openstackcluster_controller.go](../../controllers/openstackcluster_controller.go).
226
+
- Per-AZ LB reconciliation, rename/adoption, member selection, and optional floating IPs in [pkg/cloud/services/loadbalancer/](../../pkg/cloud/services/loadbalancer/).
290
227
3) Documentation:
291
-
- Update configuration guide and examples in [docs/book/src/clusteropenstack/configuration.md](docs/book/src/clusteropenstack/configuration.md).
228
+
- Update configuration guide and examples in [docs/book/src/clusteropenstack/configuration.md](../book/src/clusteropenstack/configuration.md).
292
229
4) Testing:
293
-
- Unit tests across controller and services; e2e suite updates in [test/e2e/](test/e2e/).
230
+
- Unit tests across controller and services; e2e suite updates in [test/e2e/](../../test/e2e/).
294
231
5) Optional metrics:
295
232
- Add observability for per-AZ LB counts and reconciliation timings (non-breaking).
0 commit comments