Skip to content

Commit e0463ce

Browse files
GGh41thGGh41th
committed
feat/static-pod-initialization (#125)
Integrate with npd using a custom problem daemon to ensure that static pods are mirrored before scheduling on the node.
1 parent 6fa1e93 commit e0463ce

File tree

5 files changed

+333
-0
lines changed

5 files changed

+333
-0
lines changed

‎examples/staticpods-readiness/README.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Node Readiness Examples (Static Pods)
2+
3+
This example demonstrates how NRC can be used alongside [NPD (Node Problem Detector)](https://github.com/kubernetes/node-problem-detector) to make sure that all static pods are mirrored in the API server to avoid over-committing issues. See [#115325](https://github.com/kubernetes/kubernetes/issues/115325), [#47264](https://github.com/kubernetes/website/issues/47264), and [#126870](https://github.com/kubernetes/kubernetes/pull/126870) for reference.
4+
5+
## Deployment Steps
6+
7+
1. Deploy a testing cluster if you haven't already. For this example we are using a Kubernetes 1.35 Kind cluster.
8+
We need to also mount our problem daemon script and its respective config. On Kind this can be done using this config:
9+
10+
```yaml
11+
kind: Cluster
12+
apiVersion: kind.x-k8s.io/v1alpha4
13+
nodes:
14+
# This is a 1 cluster node for testing purposes
15+
- role: control-plane
16+
extraMounts:
17+
- hostPath: /path/to/check-staticpods-synced.sh
18+
containerPath: /opt/check-staticpods-synced.sh
19+
- hostPath: /path/to/staticpods-syncer.json
20+
containerPath: /opt/staticpods-syncer.json
21+
- hostPath: /path/to/staticPodsManifestsDir
22+
containerPath: /etc/kubernetes/manifests
23+
```
24+
25+
2. Install NRC and apply the node readiness rule:
26+
27+
```bash
28+
kubectl apply -f nrr.yaml
29+
```
30+
31+
3. Deploy NPD, either as a DaemonSet using the official Helm chart, or in standalone mode. Keep in mind that if you are going to go the Helm way, you need to modify the NPD image to include the binaries that your script depends on (in our case, we need `curl` and `kubectl`), or download them in your script (this is not recommended since you'll have to set a high timeout in the custom plugin monitor config, at least for the initial script run).
32+
33+
### Standalone Mode
34+
35+
In this example, and since I have a 1-node Kind cluster, I will go the standalone way. Note that this is the default shipping method in GKE and AKS.
36+
37+
```bash
38+
# Exec into your kind node
39+
docker exec -it container_id bash
40+
41+
# Download NPD (change the arch if you are working on arm!)
42+
curl -LO https://github.com/kubernetes/node-problem-detector/releases/download/v1.35.2/node-problem-detector-v1.35.2-linux_amd64.tar.gz
43+
mkdir /opt/npd && tar -xf node-problem-detector-v1.35.2-linux_amd64.tar.gz -C /opt/npd && rm -f node-problem-detector-v1.35.2-linux_amd64.tar.gz
44+
45+
# Start NPD with the custom problem daemon
46+
/opt/npd/bin/node-problem-detector \
47+
--apiserver-override="https://127.0.0.1:6443?inClusterConfig=false&auth=/etc/kubernetes/admin.conf" \
48+
--config.custom-plugin-monitor=/opt/staticpods-syncer.json
49+
```
50+
51+
### Using Helm
52+
53+
Use the `values.yaml` file to override the default Helm values, and don't forget to add the required binaries to the NPD image (`curl`, `kubectl`).
54+
55+
```bash
56+
helm repo add deliveryhero https://charts.deliveryhero.io/
57+
helm repo update
58+
helm install --generate-name deliveryhero/node-problem-detector -f values.yaml
59+
```
60+
61+
> **Note:** For a more robust and reliable version of the shell script, you can check this [standalone binary](https://github.com/GGh41th/NPD-staticpods-syncer).

‎examples/staticpods-readiness/check-staticpods-synced.sh‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
#!/bin/bash
2+
3+
# Copyright The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
18+
# This script will check that all static pods are mirrored in the
19+
# API server.
20+
21+
# Exit code constants
22+
readonly OK=0
23+
readonly NONOK=1
24+
readonly UNKOWN=2
25+
26+
function usage {
27+
echo ""
28+
echo "Usage: $0 --staticPodsDir"
29+
echo ""
30+
echo "Check if all static pods are mirrored in the Api server"
31+
echo ""
32+
echo "options:"
33+
echo -e "\t--staticPodsDir\t Path of static pods manifests (Default: /etc/kubernetes/manifests)"
34+
exit $UNKOWN
35+
}
36+
37+
function die {
38+
echo $1
39+
exit $2
40+
}
41+
42+
STATIC_PODS_DIR="/etc/kubernetes/manifests"
43+
NODE_NAME=$(hostname)
44+
45+
while [[ $# -gt 0 ]]; do
46+
case $1 in
47+
--staticPodsDir) STATIC_PODS_DIR=$2; shift ;;
48+
-h|--help) usage ;;
49+
*) die "Unknown option: $1" $UNKNOWN ;;
50+
esac
51+
done
52+
53+
54+
55+
YQ_VERSION="v4.52.4"
56+
YQ_PATH="/opt/yq"
57+
58+
JQ_VERSION="jq-1.8.1"
59+
JQ_PATH="/opt/jq"
60+
61+
62+
if [ ! -f "$YQ_PATH" ]; then
63+
64+
echo "yq not found at $YQ_PATH, downloading..."
65+
OS=$(uname -s | tr [:upper:] [:lower:])
66+
ARCH=$(uname -m)
67+
68+
case $ARCH in
69+
x86_64)
70+
ARCH="amd64"
71+
;;
72+
73+
arm|aarch64)
74+
ARCH="arm64"
75+
;;
76+
77+
# This script is for demonstration purposes , YQ supports
78+
# the majority of archs , add case statements as desired.
79+
*)
80+
echo "Unsupported Arch: $ARCH"
81+
exit $UNKOWN
82+
esac
83+
84+
YQ_BINARY="yq_${OS}_${ARCH}"
85+
86+
curl -sL "https://github.com/mikefarah/yq/releases/download/$YQ_VERSION/$YQ_BINARY" -o "$YQ_PATH"
87+
chmod +x "$YQ_PATH"
88+
89+
fi
90+
91+
92+
93+
if [ ! -f "$JQ_PATH" ]; then
94+
95+
echo "jq not found at $JQ_PATH, downloading..."
96+
OS=$(uname -s | tr [:upper:] [:lower:])
97+
ARCH=$(uname -m)
98+
99+
case $ARCH in
100+
x86_64)
101+
ARCH="amd64"
102+
;;
103+
104+
arm|aarch64)
105+
ARCH="arm64"
106+
;;
107+
108+
# This script is for demonstration purposes , YQ supports
109+
# the majority of archs , add case statements as desired.
110+
*)
111+
echo "Unsupported Arch: $ARCH"
112+
exit $UNKOWN
113+
esac
114+
115+
JQ_BINARY="jq-${OS}-${ARCH}"
116+
117+
curl -sL "https://github.com/jqlang/jq/releases/download/$JQ_VERSION/$JQ_BINARY" -o "$JQ_PATH"
118+
chmod +x "$JQ_PATH"
119+
120+
fi
121+
122+
parse_json_manifest() {
123+
124+
# The yaml file might be a multi-document yaml , but given that kubelet
125+
# only reads the first document we will skip the rest.
126+
NAME=$($JQ_PATH -r '.metadata.name' $1)
127+
NAMESPACE=$($JQ_PATH -r '.metadata.namespace // "default"' $1)
128+
129+
# if NAME is empty , we'll suppose that the file is
130+
# syntaxically incorrect and exit.
131+
if [ "$NAME" = "" ];then
132+
echo "manifest $1 is invalid"
133+
exit $UNKNOWN
134+
fi
135+
136+
FNAME="$NAME-$NODE_NAME"
137+
138+
RESULT=$(kubectl get pod "$FNAME" -n "$NAMESPACE" -o=jsonpath='{.metadata.name}' 2>/dev/null)
139+
if [ "$RESULT" = "" ];then
140+
echo "Static pod $FNAME is missing a mirror pod"
141+
exit "$NONOK"
142+
fi
143+
}
144+
145+
146+
parse_yaml_manifest() {
147+
148+
NAME=$($YQ_PATH -r '.metadata.name' $1 | head -1)
149+
NAMESPACE=$($YQ_PATH -r '.metadata.namespace // "default"' $1 | head -1)
150+
151+
# if NAME is empty , we'll suppose that the file is
152+
# syntaxically incorrect and exit.
153+
if [ "$NAME" = "" ];then
154+
echo "manifest $1 is invalid"
155+
exit $UNKNOWN
156+
fi
157+
158+
FNAME="$NAME-$NODE_NAME"
159+
160+
RESULT=$(kubectl get pod "$FNAME" -n "$NAMESPACE" -o=jsonpath='{.metadata.name}' 2>/dev/null)
161+
if [ "$RESULT" = "" ];then
162+
echo "Static pod $FNAME is missing a mirror pod"
163+
exit "$NONOK"
164+
fi
165+
}
166+
167+
for file in "$STATIC_PODS_DIR"/*
168+
do
169+
# we read the first non blank line , if it starts with '{'
170+
# after trimming then it's json , otherwise we will consider it
171+
# as yaml. (It's not 100% accurate , but it's a good guess work).
172+
173+
# we read the first line and strip the leading whitespaces
174+
fline=$(grep -m 1 . $file | awk '{$1=$1};1')
175+
176+
fchar="${fline:0:1}"
177+
if [ $fchar = "{" ]; then
178+
parse_json_manifest "$file"
179+
else
180+
parse_yaml_manifest "$file"
181+
fi
182+
183+
done
184+
exit $OK

‎examples/staticpods-readiness/nrr.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: readiness.node.x-k8s.io/v1alpha1
2+
kind: NodeReadinessRule
3+
metadata:
4+
name: static-pods-readiness-rule
5+
spec:
6+
nodeSelector:
7+
matchLabels: {}
8+
conditions:
9+
- type: "StaticPodsMissing"
10+
requiredStatus: "False"
11+
12+
taint:
13+
key: "readiness.k8s.io/StaticPodsMissing"
14+
effect: "NoSchedule"
15+
16+
enforcementMode: "bootstrap-only"

‎examples/staticpods-readiness/staticpods-syncer.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"plugin": "custom",
3+
"pluginconfig": {
4+
"invoke_interval": "1m",
5+
"timeout": "1m",
6+
"max_output_length": 80,
7+
"concurrency": 1
8+
},
9+
"source": "staticpods-custom-plugin-monitor",
10+
"conditions": [
11+
{
12+
"type": "StaticPodsMissing",
13+
"reason": "mirrorpodsfound",
14+
"message": "all mirros pods were created"
15+
}
16+
],
17+
"rules": [
18+
{
19+
"type": "permanent",
20+
"condition": "StaticPodsMissing",
21+
"reason": "mirrorpodmissing",
22+
"path": "/opt/check-staticpods-synced.sh"
23+
}
24+
]
25+
}

‎examples/staticpods-readiness/values.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Use this file to override the values provided by the helm chart.
2+
3+
# If you need specific binaries in your npd container , use the
4+
# npd image as a base image , then add your desired binaries.
5+
# image: npd_custom_image
6+
7+
extraVolumes:
8+
- name: static-pod-syncer
9+
hostPath:
10+
path: /opt/check-staticpods-synced.sh
11+
type: File
12+
13+
extraVolumeMounts:
14+
- name: static-pod-syncer
15+
mountPath: /opt/check-staticpods-synced.sh
16+
17+
settings:
18+
custom_monitor_definitions:
19+
staticpods-syncer.json: |
20+
{
21+
"plugin": "custom",
22+
"pluginconfig": {
23+
"invoke_interval": "1m",
24+
"timeout": "1m",
25+
"max_output_length": 80,
26+
"concurrency": 1
27+
},
28+
"source": "staticpods-custom-plugin-monitor",
29+
"conditions": [
30+
{
31+
"type": "StaticPodsMissing",
32+
"reason": "mirrorpodsfound",
33+
"message": "all mirros pods were created"
34+
}
35+
],
36+
"rules": [
37+
{
38+
"type": "permanent",
39+
"condition": "StaticPodsMissing",
40+
"reason": "mirrorpodmissing",
41+
"path": "/opt/check-staticpods-synced.sh"
42+
}
43+
]
44+
}
45+
46+
custom_plugin_monitors:
47+
- /custom-config/staticpods-syncer.json

0 commit comments

Comments
 (0)