Skip to content

Commit 02900fb

Browse files
GGh41thGGh41th
committed
feat/static-pod-initialization (#125)
Integrate with npd using a custom problem daemon to ensure that static pods are mirrored before scheduling on the node.
1 parent 6fa1e93 commit 02900fb

File tree

8 files changed

+377
-0
lines changed

8 files changed

+377
-0
lines changed

‎examples/staticpods-readiness/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Node Readiness Examples (Static Pods)
2+
3+
This example demonstrates how NRC can be used alongside [NPD (Node Problem Detector)](https://github.com/kubernetes/node-problem-detector) to make sure that all static pods are mirrored in the API server to avoid over-committing issues. See [#115325](https://github.com/kubernetes/kubernetes/issues/115325), [#47264](https://github.com/kubernetes/website/issues/47264), and [#126870](https://github.com/kubernetes/kubernetes/pull/126870) for reference.
4+
5+
## Deployment Steps
6+
7+
1. Deploy a testing cluster. For this example we use Kind with a mounted static pod manifest for testing:
8+
9+
```bash
10+
kind create cluster --config examples/staticpods-readiness/kind-cluster-config.yaml
11+
```
12+
13+
2. Install NRC:
14+
15+
```bash
16+
VERSION=v0.2.0
17+
kubectl apply -f https://github.com/kubernetes-sigs/node-readiness-controller/releases/download/${VERSION}/crds.yaml
18+
kubectl apply -f https://github.com/kubernetes-sigs/node-readiness-controller/releases/download/${VERSION}/install.yaml
19+
```
20+
3. Taint The node:
21+
```bash
22+
kubectl taint node kind-worker readiness.k8s.io/StaticPodsMissing=pending:NoSchedule
23+
```
24+
4. Deploy NPD as a DaemonSet with the static pods readiness configuration:
25+
26+
```bash
27+
./examples/staticpods-readiness/setup-staticpods-readiness.sh
28+
```
29+
30+
This deploys NPD with:
31+
- Init container that downloads `kubectl`, `yq`, and `jq`
32+
- Custom plugin that checks if static pods are mirrored
33+
- RBAC permissions to read pods and nodes
34+
35+
5. Apply the node readiness rule:
36+
```bash
37+
kubectl apply -f examples/staticpods-readiness/staticpods-readiness-rule.yaml
38+
```
39+
If the static pods was successfully created, NRC would read the condition added by NPD and the taint should be removed.

‎examples/staticpods-readiness/kind-cluster-config.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
nodes:
4+
- role: control-plane
5+
- role: worker
6+
extraMounts:
7+
- hostPath: examples/staticpods-readiness/test-staticpod.yaml
8+
containerPath: /etc/kubernetes/manifests/test-staticpod.yaml
9+
readOnly: true
10+

‎examples/staticpods-readiness/npd/npd-ds.yaml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: node-problem-detector-staticpods
5+
namespace: kube-system
6+
labels:
7+
app: node-problem-detector
8+
component: staticpods-monitor
9+
spec:
10+
selector:
11+
matchLabels:
12+
app: node-problem-detector
13+
component: staticpods-monitor
14+
template:
15+
metadata:
16+
labels:
17+
app: node-problem-detector
18+
component: staticpods-monitor
19+
spec:
20+
serviceAccountName: node-problem-detector-staticpods
21+
tolerations:
22+
- operator: Exists
23+
effect: NoSchedule
24+
- operator: Exists
25+
effect: NoExecute
26+
initContainers:
27+
# Download tools (kubectl, yq, jq) for the check script
28+
- name: install-tools
29+
image: busybox:1.36
30+
command:
31+
- sh
32+
- -c
33+
- |
34+
set -e
35+
ARCH=$(uname -m)
36+
case $ARCH in
37+
x86_64) ARCH="amd64" ;;
38+
aarch64) ARCH="arm64" ;;
39+
esac
40+
41+
# Install kubectl
42+
KUBECTL_VERSION="v1.35.0"
43+
wget -O /tools-bin/kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${ARCH}/kubectl"
44+
chmod +x /tools-bin/kubectl
45+
46+
# Install yq
47+
YQ_VERSION="v4.44.1"
48+
wget -O /tools-bin/yq "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH}"
49+
chmod +x /tools-bin/yq
50+
51+
# Install jq
52+
JQ_VERSION="1.7.1"
53+
wget -O /tools-bin/jq "https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${ARCH}"
54+
chmod +x /tools-bin/jq
55+
volumeMounts:
56+
- name: tools-bin
57+
mountPath: /tools-bin
58+
containers:
59+
- name: node-problem-detector
60+
image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.19
61+
command:
62+
- /node-problem-detector
63+
- --logtostderr
64+
- --config.custom-plugin-monitor=/config/staticpods-plugin.json
65+
securityContext:
66+
privileged: true
67+
env:
68+
- name: NODE_NAME
69+
valueFrom:
70+
fieldRef:
71+
fieldPath: spec.nodeName
72+
- name: PATH
73+
value: /tools-bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
74+
volumeMounts:
75+
- name: config
76+
mountPath: /config
77+
readOnly: true
78+
- name: plugin
79+
mountPath: /config/plugin
80+
readOnly: true
81+
- name: static-pods
82+
mountPath: /etc/kubernetes/manifests
83+
readOnly: true
84+
- name: tools-bin
85+
mountPath: /tools-bin
86+
readOnly: true
87+
resources:
88+
limits:
89+
cpu: 100m
90+
memory: 128Mi
91+
requests:
92+
cpu: 20m
93+
memory: 64Mi
94+
volumes:
95+
- name: config
96+
configMap:
97+
name: npd-staticpods-config
98+
items:
99+
- key: staticpods-plugin.json
100+
path: staticpods-plugin.json
101+
- name: plugin
102+
configMap:
103+
name: npd-staticpods-config
104+
defaultMode: 0755
105+
items:
106+
- key: check-staticpods.sh
107+
path: check-staticpods.sh
108+
- name: static-pods
109+
hostPath:
110+
path: /etc/kubernetes/manifests
111+
type: DirectoryOrCreate
112+
- name: tools-bin
113+
emptyDir: {}

‎examples/staticpods-readiness/npd/npd-rbac.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: node-problem-detector-staticpods
5+
namespace: kube-system
6+
---
7+
apiVersion: rbac.authorization.k8s.io/v1
8+
kind: ClusterRole
9+
metadata:
10+
name: node-problem-detector-staticpods
11+
rules:
12+
- apiGroups:
13+
- ""
14+
resources:
15+
- nodes
16+
verbs:
17+
- get
18+
- apiGroups:
19+
- ""
20+
resources:
21+
- nodes/status
22+
verbs:
23+
- patch
24+
- apiGroups:
25+
- ""
26+
resources:
27+
- pods
28+
verbs:
29+
- get
30+
---
31+
apiVersion: rbac.authorization.k8s.io/v1
32+
kind: ClusterRoleBinding
33+
metadata:
34+
name: node-problem-detector-staticpods
35+
roleRef:
36+
apiGroup: rbac.authorization.k8s.io
37+
kind: ClusterRole
38+
name: node-problem-detector-staticpods
39+
subjects:
40+
- kind: ServiceAccount
41+
name: node-problem-detector-staticpods
42+
namespace: kube-system

‎examples/staticpods-readiness/npd/npd-staticpods-config.yaml‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: npd-staticpods-config
5+
namespace: kube-system
6+
data:
7+
staticpods-plugin.json: |
8+
{
9+
"plugin": "custom",
10+
"pluginConfig": {
11+
"invoke_interval": "30s",
12+
"timeout": "10s",
13+
"max_output_length": 80,
14+
"concurrency": 1
15+
},
16+
"source": "staticpods-monitor",
17+
"conditions": [
18+
{
19+
"type": "StaticPodsMissing",
20+
"reason": "AllStaticPodsMirrored",
21+
"message": "All static pods have mirror pods in API server"
22+
}
23+
],
24+
"rules": [
25+
{
26+
"type": "permanent",
27+
"condition": "StaticPodsMissing",
28+
"reason": "StaticPodsNotMirrored",
29+
"path": "/config/plugin/check-staticpods.sh"
30+
}
31+
]
32+
}
33+
34+
check-staticpods.sh: |
35+
#!/bin/bash
36+
37+
set -euo pipefail
38+
39+
# Exit code constants
40+
readonly OK=0
41+
readonly NONOK=1
42+
readonly UNKNOWN=2
43+
44+
STATIC_PODS_DIR="/etc/kubernetes/manifests"
45+
NODE_NAME="${NODE_NAME:-$(hostname)}"
46+
47+
48+
# Check required tools
49+
for tool in kubectl yq jq; do
50+
if ! command -v $tool &> /dev/null; then
51+
echo "$tool not found in PATH"
52+
exit $UNKNOWN
53+
fi
54+
done
55+
56+
parse_json_manifest() {
57+
58+
# The yaml file might be a multi-document yaml , but given that kubelet
59+
# only reads the first document we will skip the rest.
60+
NAME=$(jq -r '.metadata.name' $1)
61+
NAMESPACE=$(jq -r '.metadata.namespace // "default"' $1)
62+
63+
# if NAME is empty , we'll suppose that the file is
64+
# syntaxically incorrect and exit.
65+
if [ "$NAME" = "" ];then
66+
echo "manifest $1 is invalid"
67+
exit $UNKNOWN
68+
fi
69+
70+
FNAME="$NAME-$NODE_NAME"
71+
72+
RESULT=$(kubectl get pod "$FNAME" -n "$NAMESPACE" -o=jsonpath='{.metadata.name}' 2>/dev/null)
73+
if [ "$RESULT" = "" ];then
74+
echo "Static pod $FNAME is missing a mirror pod"
75+
exit "$NONOK"
76+
fi
77+
}
78+
79+
80+
parse_yaml_manifest() {
81+
82+
NAME=$(yq -r '.metadata.name' $1 | head -1)
83+
NAMESPACE=$(yq -r '.metadata.namespace // "default"' $1 | head -1)
84+
85+
# if NAME is empty , we'll suppose that the file is
86+
# syntaxically incorrect and exit.
87+
if [ "$NAME" = "" ];then
88+
echo "manifest $1 is invalid"
89+
exit $UNKNOWN
90+
fi
91+
92+
FNAME="$NAME-$NODE_NAME"
93+
94+
RESULT=$(kubectl get pod "$FNAME" -n "$NAMESPACE" -o=jsonpath='{.metadata.name}' 2>/dev/null)
95+
if [ "$RESULT" = "" ];then
96+
echo "Static pod $FNAME is missing a mirror pod"
97+
exit "$NONOK"
98+
fi
99+
}
100+
101+
for file in "$STATIC_PODS_DIR"/*
102+
do
103+
# we read the first non blank line , if it starts with '{'
104+
# after trimming then it's json , otherwise we will consider it
105+
# as yaml. (It's not 100% accurate , but it's a good guess work).
106+
107+
# we read the first line and strip the leading whitespaces
108+
fline=$(grep -m 1 . $file | awk '{$1=$1};1')
109+
110+
fchar="${fline:0:1}"
111+
if [ $fchar = "{" ]; then
112+
parse_json_manifest "$file"
113+
else
114+
parse_yaml_manifest "$file"
115+
fi
116+
117+
done
118+
exit $OK

‎examples/staticpods-readiness/setup-staticpods-readiness.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
3+
# Copyright The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
set -e
18+
19+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
20+
21+
kubectl apply -f "$SCRIPT_DIR/npd/npd-rbac.yaml"
22+
kubectl apply -f "$SCRIPT_DIR/npd/npd-staticpods-config.yaml"
23+
kubectl apply -f "$SCRIPT_DIR/npd/npd-ds.yaml"

‎examples/staticpods-readiness/staticpods-readiness-rule.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: readiness.node.x-k8s.io/v1alpha1
2+
kind: NodeReadinessRule
3+
metadata:
4+
name: static-pods-readiness-rule
5+
spec:
6+
nodeSelector:
7+
matchLabels: {}
8+
conditions:
9+
- type: "StaticPodsMissing"
10+
requiredStatus: "False"
11+
taint:
12+
key: "readiness.k8s.io/StaticPodsMissing"
13+
effect: "NoSchedule"
14+
15+
enforcementMode: "bootstrap-only"

‎examples/staticpods-readiness/test-staticpod.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: v1
2+
kind: Pod
3+
metadata:
4+
name: test-static-pod
5+
labels:
6+
app: test-static-pod
7+
spec:
8+
containers:
9+
- name: pause
10+
image: registry.k8s.io/pause:3.9
11+
resources:
12+
requests:
13+
cpu: 10m
14+
memory: 16Mi
15+
limits:
16+
cpu: 10m
17+
memory: 16Mi

0 commit comments

Comments
 (0)