Operational runbook follow-up split out from the §8.2 egress lockdown work (client-runtime#102/#103/#104, chart 1.7.0/1.7.1). Gates the per-fleet lockdown flip.
Why
Enabling the lockdown per fleet flips egressProxy.routeWorkloads=true + networkPolicy.training.allowExternalHttps=false. From the 2026-06-12 dev-EKS incident:
- Flipping
routeWorkloads=true injects HTTPS_PROXY=egress-proxy-service:3128 into every training pod spawned during the window.
- A later rollback/downgrade that removes the egress-proxy leaves those in-flight pods CrashLooping on a dead proxy (
ProxyError: Failed to resolve 'egress-proxy-service').
- jobs-manager consumes experiments from the queue once and does not re-spawn failed/deleted jobs — so disruption = lost runs, and
kubectl delete job on a stuck job orphans the experiment.
Ask
Author a runbook (client/docs/runbooks/egress-lockdown-flip.md) covering:
- Pre-flip: drain in-flight training (quiet window / wait for active jobs) before enabling
routeWorkloads on a shared cluster.
- Flip ordering:
routeWorkloads=true → verify a training run routes via the gateway (squid TCP_TUNNEL) → allowExternalHttps=false → verify G2 (helm test).
- Rollback ordering: revert
allowExternalHttps first, drain, then remove the gateway — never pull the egress-proxy out from under dependent pods. Recovery for already-CrashLooping pods: temporarily re-apply egress-proxy (configmap+service+deployment) so they resolve it and finish, then remove.
- Hard rule: never
kubectl delete job a stuck training job (jobs-manager won't recreate it → orphaned experiment).
Acceptance
Related
Operational runbook follow-up split out from the §8.2 egress lockdown work (client-runtime#102/#103/#104, chart 1.7.0/1.7.1). Gates the per-fleet lockdown flip.
Why
Enabling the lockdown per fleet flips
egressProxy.routeWorkloads=true+networkPolicy.training.allowExternalHttps=false. From the 2026-06-12 dev-EKS incident:routeWorkloads=trueinjectsHTTPS_PROXY=egress-proxy-service:3128into every training pod spawned during the window.ProxyError: Failed to resolve 'egress-proxy-service').kubectl delete jobon a stuck job orphans the experiment.Ask
Author a runbook (
client/docs/runbooks/egress-lockdown-flip.md) covering:routeWorkloadson a shared cluster.routeWorkloads=true→ verify a training run routes via the gateway (squidTCP_TUNNEL) →allowExternalHttps=false→ verify G2 (helm test).allowExternalHttpsfirst, drain, then remove the gateway — never pull the egress-proxy out from under dependent pods. Recovery for already-CrashLooping pods: temporarily re-apply egress-proxy (configmap+service+deployment) so they resolve it and finish, then remove.kubectl delete joba stuck training job (jobs-manager won't recreate it → orphaned experiment).Acceptance
client/docs/.Related