Skip to content

Commit 33216e3

Browse files
ali-a-aali-a-a
authored
feat: constrained impersonation for secure node status updates (#143)
1 parent 65a81b8 commit 33216e3

File tree

8 files changed

+335
-1
lines changed

8 files changed

+335
-1
lines changed

‎cmd/readiness-condition-reporter/main.go‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ const (
3737
envConditionType = "CONDITION_TYPE"
3838
envCheckEndpoint = "CHECK_ENDPOINT"
3939
envCheckInterval = "CHECK_INTERVAL"
40+
envImpersonateNode = "IMPERSONATE_NODE"
4041
defaultCheckInterval = 30 * time.Second
4142
defaultHTTPTimeout = 10 * time.Second
4243
)
@@ -90,6 +91,14 @@ func main() {
9091
os.Exit(1)
9192
}
9293

94+
// Set the constrained impersonation config
95+
if os.Getenv(envImpersonateNode) == "true" {
96+
config.Impersonate = rest.ImpersonationConfig{
97+
UserName: "system:node:" + nodeName,
98+
}
99+
klog.InfoS("Node impersonation enabled", "impersonating", config.Impersonate.UserName)
100+
}
101+
93102
clientset, err := kubernetes.NewForConfig(config)
94103
if err != nil {
95104
klog.ErrorS(err, "Failed to create client")

‎config/testing/kind/kind-constrained-impersonation-config.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Kind cluster configuration for testing Constrained Impersonation (KEP-5284).
2+
# Requires Kubernetes v1.35+ (Alpha) or v1.36+ (Beta).
3+
# The ConstrainedImpersonation feature gate must be enabled on the API server.
4+
kind: Cluster
5+
apiVersion: kind.x-k8s.io/v1alpha4
6+
name: nrr-constrained-impersonation
7+
kubeadmConfigPatches:
8+
- |
9+
kind: ClusterConfiguration
10+
apiServer:
11+
extraArgs:
12+
feature-gates: "ConstrainedImpersonation=true"
13+
nodes:
14+
- role: control-plane
15+
- role: worker
16+
kubeadmConfigPatches:
17+
- |
18+
kind: JoinConfiguration
19+
nodeRegistration:
20+
kubeletExtraArgs:
21+
register-with-taints: "readiness.k8s.io/NetworkReady=pending:NoSchedule"
22+
- role: worker
23+
kubeadmConfigPatches:
24+
- |
25+
kind: JoinConfiguration
26+
nodeRegistration:
27+
kubeletExtraArgs:
28+
register-with-taints: "readiness.k8s.io/NetworkReady=pending:NoSchedule"

‎docs/book/src/SUMMARY.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
<!-- - [Storage Drivers](./examples/storage-readiness.md) -->
1616
- [Security Agent](./examples/security-agent-readiness.md)
1717
<!-- - [Device Drivers](./examples/dra-readiness.md) -->
18+
- [Constrained Impersonation](./examples/constrained-impersonation.md)
1819

1920
# Releases
2021

@@ -24,7 +25,7 @@
2425

2526
- [Monitoring](./operations/monitoring.md)
2627
<!-- - [Troubleshooting](./operations/troubleshooting.md) -->
27-
<!-- - [Security](./operations/security.md) -->
28+
- [Security](./operations/security.md)
2829

2930
<!-- # Development -->
3031

‎docs/book/src/examples/constrained-impersonation.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Secure Status Reporting
2+
3+
By default, a readiness reporter's ServiceAccount is granted broad permissions to update the status of **any** Node in the cluster. If a node is compromised, an attacker can manipulate the readiness status of every other node.
4+
5+
[Constrained Impersonation (KEP-5284)](https://github.com/kubernetes/enhancements/issues/5284) solves this by allowing the reporter to impersonate only the Node it runs on. The API server enforces this at the authorization layer, so that no other node's status can be touched.
6+
7+
This guide walks through a CNI readiness example that uses constrained impersonation instead of broad RBAC. It is a hardened variant of the [CNI Readiness](./cni-readiness.md) example.
8+
9+
> **Prerequisites**: Kubernetes v1.35+ with the `ConstrainedImpersonation` feature gate enabled, or v1.36+ where it is Beta and enabled by default.
10+
11+
> **Note**: You can find all the manifests used in this guide in the [`examples/constrained-impersonation`](https://github.com/kubernetes-sigs/node-readiness-controller/tree/main/examples/constrained-impersonation) directory.
12+
13+
## Step-by-Step Guide
14+
15+
### 1. Create a Kind Cluster
16+
17+
Create a cluster with the `ConstrainedImpersonation` feature gate enabled and worker nodes that join with a startup taint:
18+
19+
```sh
20+
kind create cluster \
21+
--config config/testing/kind/kind-constrained-impersonation-config.yaml \
22+
--image kindest/node:v1.35.0
23+
```
24+
25+
### 2. Install the CRDs and Controller
26+
27+
```sh
28+
make install
29+
make deploy
30+
```
31+
32+
### 3. Deploy the Example
33+
34+
```sh
35+
cd examples/constrained-impersonation
36+
kubectl apply -f .
37+
```
38+
39+
### 4. RBAC Explained
40+
41+
The RBAC consists of two ClusterRoles:
42+
43+
**Impersonation role** — allows the reporter to impersonate its own Node:
44+
45+
```yaml
46+
apiVersion: rbac.authorization.k8s.io/v1
47+
kind: ClusterRole
48+
metadata:
49+
name: node-readiness-impersonator
50+
rules:
51+
- apiGroups: ["authentication.k8s.io"]
52+
resources: ["nodes"]
53+
verbs: ["impersonate:associated-node"]
54+
```
55+
56+
**Constrained action role** — restricts what the impersonated identity can do:
57+
58+
```yaml
59+
apiVersion: rbac.authorization.k8s.io/v1
60+
kind: ClusterRole
61+
metadata:
62+
name: node-status-patcher-constrained
63+
rules:
64+
- apiGroups: [""]
65+
resources: ["nodes"]
66+
verbs: ["impersonate-on:associated-node:get"]
67+
- apiGroups: [""]
68+
resources: ["nodes/status"]
69+
verbs: ["impersonate-on:associated-node:update"]
70+
```
71+
72+
### 5. Verification
73+
74+
**Check that the reporter is running:**
75+
76+
```sh
77+
kubectl -n kube-system get pods -l app=cni-reporter
78+
```
79+
80+
**Check node conditions:**
81+
82+
```sh
83+
kubectl get nodes -o custom-columns='NAME:.metadata.name,CALICO_READY:.status.conditions[?(@.type=="projectcalico.org/CalicoReady")].status'
84+
```
85+
86+
### 6. Security Verification
87+
88+
**Verify the ServiceAccount has no direct permissions:**
89+
90+
```sh
91+
kubectl auth can-i get nodes --as=system:serviceaccount:kube-system:cni-reporter
92+
# no
93+
kubectl auth can-i update nodes/status --as=system:serviceaccount:kube-system:cni-reporter
94+
# no
95+
```
96+
97+
The SA cannot read or update any node directly; all access goes through constrained impersonation.
98+
99+
**Verify the reporter can still update its own node (via impersonation):**
100+
101+
```sh
102+
kubectl get nodes -o custom-columns='NAME:.metadata.name,CALICO_READY:.status.conditions[?(@.type=="projectcalico.org/CalicoReady")].status'
103+
```
104+
105+
The `CalicoReady` condition should appear on every node.
106+
This proves the reporter is successfully impersonating its local node identity and writing status.
107+
108+
## Comparison with Broad RBAC
109+
110+
For a deeper discussion, see [Security](../operations/security.md).

‎docs/book/src/operations/security.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Security
2+
3+
The Node Readiness Controller relies on external **reporters**, lightweight components running on each node, to publish readiness information as Node conditions. Because these reporters must patch Node status objects, the RBAC they require needs careful attention.
4+
5+
## The Threat Model
6+
7+
Reporters typically run as DaemonSet pods or sidecars on the nodes they monitor.
8+
They use the Kubernetes API to update `nodes/status` with condition data that the controller consumes.
9+
10+
With **broad RBAC** (the default in Kubernetes < v1.35), a reporter's ServiceAccount is granted `patch` and `update` on all `nodes/status` resources cluster-wide. This means that if a single node is compromised, an attacker can:
11+
12+
- Mark **other** nodes as ready or not-ready, influencing scheduling decisions across the cluster.
13+
- Inject false conditions to bypass readiness gates on nodes they do not control.
14+
15+
This violates the principle of least privilege: a reporter should only be able to modify the status of the node it runs on.
16+
17+
## Constrained Impersonation (KEP-5284)
18+
19+
[Constrained Impersonation](https://github.com/kubernetes/enhancements/issues/5284) (KEP-5284) introduces authorization rules that restrict a ServiceAccount to impersonating only the Node identity associated with the pod's bound service account token, and to performing only specific actions during that impersonation.
20+
21+
### How It Works
22+
23+
Two ClusterRoles are used together:
24+
25+
1. **Impersonation role** — grants the reporter the ability to impersonate the identity of its own Node and nothing else.
26+
27+
```yaml
28+
apiVersion: rbac.authorization.k8s.io/v1
29+
kind: ClusterRole
30+
metadata:
31+
name: node-readiness-impersonator
32+
rules:
33+
- apiGroups: ["authentication.k8s.io"]
34+
resources: ["nodes"]
35+
verbs: ["impersonate:associated-node"]
36+
```
37+
38+
The `impersonate:associated-node` verb tells the API server to validate that the pod's bound service account token references the same node (via the `authentication.kubernetes.io/node-name` extra key).
39+
A reporter on `node-A` cannot impersonate `node-B`.
40+
41+
2. **Constrained action role** — restricts what the impersonated identity is allowed to do.
42+
43+
```yaml
44+
apiVersion: rbac.authorization.k8s.io/v1
45+
kind: ClusterRole
46+
metadata:
47+
name: node-status-patcher-constrained
48+
rules:
49+
- apiGroups: [""]
50+
resources: ["nodes"]
51+
verbs: ["impersonate-on:associated-node:get"]
52+
- apiGroups: [""]
53+
resources: ["nodes/status"]
54+
verbs: ["impersonate-on:associated-node:update"]
55+
```
56+
57+
The `impersonate-on:associated-node:<verb>` verbs permit only specific operations during the impersonated session.
58+
The reporter can get its own Node and update its status, but cannot modify labels, taints, the spec, or any resource other than `nodes/status`.
59+
60+
Both roles are bound to the reporter's ServiceAccount via ClusterRoleBindings.
61+
62+
### Reporter Configuration
63+
64+
The reporter must be configured to use impersonation by setting the `IMPERSONATE_NODE` environment variable to `"true"`. When enabled, the reporter sends `Impersonate-User: system:node:<nodeName>` headers on every API request, which triggers the constrained impersonation authorization flow in the API server.
65+
66+
```yaml
67+
env:
68+
- name: IMPERSONATE_NODE
69+
value: "true"
70+
```
71+
72+
When `IMPERSONATE_NODE` is not set, the reporter uses its ServiceAccount identity directly (the pre-v1.35 behavior).

‎examples/constrained-impersonation/rbac.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Constrained Impersonation RBAC (KEP-5284)
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: ClusterRole
4+
metadata:
5+
name: node-readiness-impersonator
6+
rules:
7+
- apiGroups: ["authentication.k8s.io"]
8+
resources: ["nodes"]
9+
verbs: ["impersonate:associated-node"]
10+
---
11+
apiVersion: rbac.authorization.k8s.io/v1
12+
kind: ClusterRole
13+
metadata:
14+
name: node-status-patcher-constrained
15+
rules:
16+
- apiGroups: [""]
17+
resources: ["nodes"]
18+
verbs: ["impersonate-on:associated-node:get"]
19+
- apiGroups: [""]
20+
resources: ["nodes/status"]
21+
verbs: ["impersonate-on:associated-node:update"]
22+
---
23+
apiVersion: rbac.authorization.k8s.io/v1
24+
kind: ClusterRoleBinding
25+
metadata:
26+
name: cni-reporter-impersonator-binding
27+
roleRef:
28+
apiGroup: rbac.authorization.k8s.io
29+
kind: ClusterRole
30+
name: node-readiness-impersonator
31+
subjects:
32+
- kind: ServiceAccount
33+
name: cni-reporter
34+
namespace: kube-system
35+
---
36+
apiVersion: rbac.authorization.k8s.io/v1
37+
kind: ClusterRoleBinding
38+
metadata:
39+
name: cni-reporter-constrained-binding
40+
roleRef:
41+
apiGroup: rbac.authorization.k8s.io
42+
kind: ClusterRole
43+
name: node-status-patcher-constrained
44+
subjects:
45+
- kind: ServiceAccount
46+
name: cni-reporter
47+
namespace: kube-system

‎examples/constrained-impersonation/readiness-rule.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: readiness.node.x-k8s.io/v1alpha1
2+
kind: NodeReadinessRule
3+
metadata:
4+
name: network-readiness-rule
5+
spec:
6+
conditions:
7+
- type: "projectcalico.org/CalicoReady"
8+
requiredStatus: "True"
9+
taint:
10+
key: "readiness.k8s.io/NetworkReady"
11+
effect: "NoSchedule"
12+
value: "pending"
13+
enforcementMode: "continuous"
14+
nodeSelector:
15+
matchExpressions:
16+
- key: node-role.kubernetes.io/control-plane
17+
operator: DoesNotExist

‎examples/constrained-impersonation/reporter-ds.yaml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: cni-reporter
5+
namespace: kube-system
6+
---
7+
apiVersion: apps/v1
8+
kind: DaemonSet
9+
metadata:
10+
name: cni-reporter
11+
namespace: kube-system
12+
labels:
13+
app: cni-reporter
14+
spec:
15+
selector:
16+
matchLabels:
17+
app: cni-reporter
18+
template:
19+
metadata:
20+
labels:
21+
app: cni-reporter
22+
spec:
23+
hostNetwork: true
24+
serviceAccountName: cni-reporter
25+
tolerations:
26+
- operator: Exists
27+
containers:
28+
- name: cni-status-patcher
29+
image: registry.k8s.io/node-readiness-controller/node-readiness-reporter:v0.1.1
30+
imagePullPolicy: IfNotPresent
31+
env:
32+
- name: NODE_NAME
33+
valueFrom:
34+
fieldRef:
35+
fieldPath: spec.nodeName
36+
- name: CHECK_ENDPOINT
37+
value: "http://localhost:9099/readiness"
38+
- name: CONDITION_TYPE
39+
value: "projectcalico.org/CalicoReady"
40+
- name: CHECK_INTERVAL
41+
value: "5s"
42+
- name: IMPERSONATE_NODE
43+
value: "true"
44+
resources:
45+
limits:
46+
cpu: "10m"
47+
memory: "32Mi"
48+
requests:
49+
cpu: "10m"
50+
memory: "32Mi"

0 commit comments

Comments
 (0)