Generate RBAC based on the current config

[tfadeyi]

Generate RBAC based on the current config, and in such a way that the code can be shared with the backend.

[j-fuentes]

More context on this one.

platform.jetstack.io has a wizard that guides the user to connect a cluster to the platform.

It generates a command for the user to run like this one:

$ { kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: jetstack-secure
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
  namespace: jetstack-secure
data:
  config.yaml: |-
    server: "https://platform.jetstack.io"
    organization_id: "my-organization"
    cluster_id: "my-cluster"
    data-gatherers:
    # gather k8s apiserver version information
    - kind: "k8s-discovery"
      name: "k8s-discovery"
    # pods data is used in the pods and application_versions packages
    - kind: "k8s-dynamic"
      name: "k8s/pods"
      config:
        resource-type:
          resource: pods
          version: v1
    # gather services for pod readiness probe rules
    - kind: "k8s-dynamic"
      name: "k8s/services"
      config:
        resource-type:
          resource: services
          version: v1
    # gather higher level resources to ensure data to determine ownership is present
    - kind: "k8s-dynamic"
      name: "k8s/deployments"
      config:
        resource-type:
          version: v1
          resource: deployments
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/replicasets"
      config:
        resource-type:
          version: v1
          resource: replicasets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/statefulsets"
      config:
        resource-type:
          version: v1
          resource: statefulsets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/daemonsets"
      config:
        resource-type:
          version: v1
          resource: daemonsets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/jobs"
      config:
        resource-type:
          version: v1
          resource: jobs
          group: batch
    - kind: "k8s-dynamic"
      name: "k8s/cronjobs"
      config:
        resource-type:
          version: v1beta1
          resource: cronjobs
          group: batch
    # gather resources for cert-manager package
    - kind: "k8s-dynamic"
      name: "k8s/secrets"
      config:
        resource-type:
          version: v1
          resource: secrets
    - kind: "k8s-dynamic"
      name: "k8s/certificates"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: certificates
    - kind: "k8s-dynamic"
      name: "k8s/ingresses"
      config:
        resource-type:
          group: networking.k8s.io
          version: v1
          resource: ingresses
    - kind: "k8s-dynamic"
      name: "k8s/certificaterequests"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: certificaterequests
    - kind: "k8s-dynamic"
      name: "k8s/issuers"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: issuers
    - kind: "k8s-dynamic"
      name: "k8s/clusterissuers"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: clusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/googlecasissuers"
      config:
        resource-type:
          group: cas-issuer.jetstack.io
          version: v1beta1
          resource: googlecasissuers
    - kind: "k8s-dynamic"
      name: "k8s/googlecasclusterissuers"
      config:
        resource-type:
          group: cas-issuer.jetstack.io
          version: v1beta1
          resource: googlecasclusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/awspcaissuer"
      config:
        resource-type:
          group: awspca.cert-manager.io
          version: v1beta1
          resource: awspcaissuers
    - kind: "k8s-dynamic"
      name: "k8s/awspcaclusterissuers"
      config:
        resource-type:
          group: awspca.cert-manager.io
          version: v1beta1
          resource: awspcaclusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/mutatingwebhookconfigurations"
      config:
        resource-type:
          group: admissionregistration.k8s.io
          version: v1
          resource: mutatingwebhookconfigurations
    - kind: "k8s-dynamic"
      name: "k8s/validatingwebhookconfigurations"
      config:
        resource-type:
          group: admissionregistration.k8s.io
          version: v1
          resource: validatingwebhookconfigurations
---
apiVersion: v1
kind: Secret
metadata:
  name: agent-credentials
  namespace: jetstack-secure
data:
  credentials.json: <REDACTED>
EOF
} && kubectl apply -f https://platform.jetstack.io/agent_installation.yaml

That long command is actually two commands concatenated. The first one applies this yaml with kubectl:

apiVersion: v1
kind: Namespace
metadata:
  name: jetstack-secure
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
  namespace: jetstack-secure
data:
  config.yaml: |-
    server: "https://platform.jetstack.io"
    organization_id: "my-organization"
    cluster_id: "my-cluster"
    data-gatherers:
    # gather k8s apiserver version information
    - kind: "k8s-discovery"
      name: "k8s-discovery"
    # pods data is used in the pods and application_versions packages
    - kind: "k8s-dynamic"
      name: "k8s/pods"
      config:
        resource-type:
          resource: pods
          version: v1
    # gather services for pod readiness probe rules
    - kind: "k8s-dynamic"
      name: "k8s/services"
      config:
        resource-type:
          resource: services
          version: v1
    # gather higher level resources to ensure data to determine ownership is present
    - kind: "k8s-dynamic"
      name: "k8s/deployments"
      config:
        resource-type:
          version: v1
          resource: deployments
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/replicasets"
      config:
        resource-type:
          version: v1
          resource: replicasets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/statefulsets"
      config:
        resource-type:
          version: v1
          resource: statefulsets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/daemonsets"
      config:
        resource-type:
          version: v1
          resource: daemonsets
          group: apps
    - kind: "k8s-dynamic"
      name: "k8s/jobs"
      config:
        resource-type:
          version: v1
          resource: jobs
          group: batch
    - kind: "k8s-dynamic"
      name: "k8s/cronjobs"
      config:
        resource-type:
          version: v1beta1
          resource: cronjobs
          group: batch
    # gather resources for cert-manager package
    - kind: "k8s-dynamic"
      name: "k8s/secrets"
      config:
        resource-type:
          version: v1
          resource: secrets
    - kind: "k8s-dynamic"
      name: "k8s/certificates"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: certificates
    - kind: "k8s-dynamic"
      name: "k8s/ingresses"
      config:
        resource-type:
          group: networking.k8s.io
          version: v1
          resource: ingresses
    - kind: "k8s-dynamic"
      name: "k8s/certificaterequests"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: certificaterequests
    - kind: "k8s-dynamic"
      name: "k8s/issuers"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: issuers
    - kind: "k8s-dynamic"
      name: "k8s/clusterissuers"
      config:
        resource-type:
          group: cert-manager.io
          version: v1
          resource: clusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/googlecasissuers"
      config:
        resource-type:
          group: cas-issuer.jetstack.io
          version: v1beta1
          resource: googlecasissuers
    - kind: "k8s-dynamic"
      name: "k8s/googlecasclusterissuers"
      config:
        resource-type:
          group: cas-issuer.jetstack.io
          version: v1beta1
          resource: googlecasclusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/awspcaissuer"
      config:
        resource-type:
          group: awspca.cert-manager.io
          version: v1beta1
          resource: awspcaissuers
    - kind: "k8s-dynamic"
      name: "k8s/awspcaclusterissuers"
      config:
        resource-type:
          group: awspca.cert-manager.io
          version: v1beta1
          resource: awspcaclusterissuers
    - kind: "k8s-dynamic"
      name: "k8s/mutatingwebhookconfigurations"
      config:
        resource-type:
          group: admissionregistration.k8s.io
          version: v1
          resource: mutatingwebhookconfigurations
    - kind: "k8s-dynamic"
      name: "k8s/validatingwebhookconfigurations"
      config:
        resource-type:
          group: admissionregistration.k8s.io
          version: v1
          resource: validatingwebhookconfigurations
---
apiVersion: v1
kind: Secret
metadata:
  name: agent-credentials
  namespace: jetstack-secure
data:
  credentials.json: <REDACTED>

We can see there 3 different yaml objects. A Namespace, a ConfigMap and a Secret.

The ConfigMap embeds config.yaml, which is the configuration file for the agent.

Now, let's have a look at the second command:

kubectl apply -f https://platform.jetstack.io/agent_installation.yaml

It is applying another yaml. If we inspect the content (e.g. by doing curl -L https://platform.jetstack.io/agent_installation.yaml) we will see something like:

apiVersion: v1
kind: Namespace
metadata:
  name: jetstack-secure
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-cert-manager-reader
rules:
- apiGroups: ["cert-manager.io"]
  resources:
  - certificates
  - certificaterequests
  - issuers
  - clusterissuers
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-googlecas-reader
rules:
- apiGroups: ["cas-issuer.jetstack.io"]
  resources:
  - googlecasissuers
  - googlecasclusterissuers
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-awspca-reader
rules:
- apiGroups: ["awspca.cert-manager.io"]
  resources:
  - awspcaissuers
  - awspcaclusterissuers
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jetstack-secure-agent-get-webhooks
rules:
- apiGroups: ["admissionregistration.k8s.io"]
  resources:
  - validatingwebhookconfigurations
  - mutatingwebhookconfigurations
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-secret-reader
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-secret-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-cert-manager-reader
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-cert-manager-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-googlecas-reader
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-googlecas-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-awspca-reader
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-awspca-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-get-webhooks
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-get-webhooks
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-node-reader
roleRef:
  kind: ClusterRole
  name: jetstack-secure-agent-node-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jetstack-secure-agent-cluster-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: ServiceAccount
  name: agent
  namespace: jetstack-secure
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
  namespace: jetstack-secure
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: agent
  template:
    metadata:
      labels:
        app.kubernetes.io/name: agent
    spec:
      serviceAccountName: agent
      volumes:
      - name: config
        configMap:
          name: agent-config
      - name: credentials
        secret:
          secretName: agent-credentials
      containers:
      - name: agent
        image: quay.io/jetstack/preflight:v0.1.31
        args:
        - "agent"
        - "-c"
        - "/etc/jetstack-secure/agent/config/config.yaml"
        - "-k"
        - "/etc/jetstack-secure/agent/credentials/credentials.json"
        - "-p"
        - "0h1m0s"
        volumeMounts:
        - name: config
          mountPath: "/etc/jetstack-secure/agent/config"
          readOnly: true
        - name: credentials
          mountPath: "/etc/jetstack-secure/agent/credentials"
          readOnly: true
        resources:
          requests:
            memory: "200Mi"
            cpu: "200m"
          limits:
            memory: "200Mi"
            cpu: "200m"

This last yaml contains the actual workload for the agent (a Deployment) and also the RBAC resources to let the agent gather data from the cluster.

This RBAC configuration needs to be aligned with the datagatherers configured in the agent configuration (remember config.yaml).

What we are trying to achieve here is to create a generator for these two things: default datagatherers configuration and default RBAC.

[charlieegan3]

I've been working a little with Wenlin on this today and my plan was, if there was more time, to build this incrementally with TDD in stages, e.g.

1. GVRs in, RBAC string templated out
2. GVRs in, RBAC structs out
3. GVRs and Namespaces in, scoped rbac generated based on some list of known namespaces in the case of excluded namespaces
4. means of getting GVRs from loaded dynamic data gatherers.
5. adding a command to spit this all out.
6. profit

[j-fuentes]

3. GVRs and Namespaces in, scoped rbac generated based on some list of known namespaces in the case of excluded namespaces

I think this does not make much sense. At generation time is not possible to know all the existing namespaces so you can filter out the exclusion list.

@charlieegan3 I am not sure I follow this.

[charlieegan3]

You are correct, I was imagining that it'd be possible to supply a list of namespaces that comes from kubectl get ns if the agent were running locally using the installers credentials.

I'm thinking that the first use of this is going to be a command in the agent, and if the agent, running in that command had access to the installers creds, it could get a list of all namespaces.

We might not be able to do this when using it from the platform side, however I think it's a feature worth adding here in the agent.

[tfadeyi]

The RBAC generation command has been added to the agent CLI command in #258 we are going to continue the work, new default config generation, In following PRs.