Repository navigation
Retry creates rejected right after the CRD is created (OwnerReferencesPermissionEnforcement) - #2103
Open
matthewhudsonedb wants to merge 3 commits into
Open
matthewhudsonedb wants to merge 3 commits into
matthewhudsonedb wants to merge 3 commits into
Conversation
…sionEnforcement For a short window after the CHI CRD is created (observed 5-23s), the OwnerReferencesPermissionEnforcement admission plugin rejects creates carrying a blockOwnerDeletion owner reference with "cannot find RESTMapping", because its cached RESTMapper does not know the kind yet. The preliminary common ConfigMaps failed (logged only), leaving pods in Init until after the 300s StatefulSet wait, and a failed per-host ConfigMap aborted the reconcile with nothing to requeue it. Retry ConfigMap, Service, PDB, Secret and StatefulSet creates on that error and on transient API errors via CreateWithRetry (1s doubling, capped at 10s, ~35s total), sharing the loop with GetWithRetry. Any other Forbidden stays terminal. Fixes Altinity#2094
…rmissionEnforcement CHK objects get the same blockOwnerDeletion owner reference (Kind ClickHouseKeeperInstallation), so a CHK applied right after its CRD is created hits the same admission window. controller-runtime's back-off does not cover it: reconcileCRAuxObjectsPreliminary only logs common ConfigMap failures, so nothing is returned to requeue on. Wrap the CHK ConfigMap, PDB, Secret, Service and StatefulSet creates in CreateWithRetry, as for CHI.
On kind the CHK fix needed 6 of 7 attempts: the admission window closed during a 10s back-off. Allow 10 retries (1s, 2s, 4s, 8s, then 10s) so a slower apiserver does not run out.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2094
Right after the CHI or CHK CRD is created, clusters with
OwnerReferencesPermissionEnforcement(OpenShift turns it on) reject the operator's creates for a short time withcannot find RESTMapping. Every object the operator creates has an owner reference withblockOwnerDeletion: true, so they all hit it. The rejection clears by itself after about 5-25 seconds, but the operator doesn't try again:Abortedand stays that way.This retries the create when it hits that error, the same way
GetWithRetryalready retries reads. It waits 1s, 2s, 4s, 8s, then 10s between tries, for about 75s in total. Any other Forbidden error still fails straight away. It covers ConfigMap, Service, PDB, Secret and StatefulSet creates for both CHI and CHK. PVCs aren't included because the operator doesn't give them an owner reference.I tested it on kind with
--enable-admission-plugins=NodeRestriction,OwnerReferencesPermissionEnforcement, applying the CRDs, the operator and a 1-node CHI or CHK at the same time. The rejection lasted about 16-19s in each run.Completedafter 54sAbortedafter about 5.5 minCompletedafter 61sIn the CHK run the ConfigMap needed 6 tries before it went through, which is why the retry window is about 75s rather than something shorter.