https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-google-cloud-gke
This document is intended to help support engineers attempting to reproduce issues related to the Vault Secrets Operator authenticating with Vault via GCP Auth Method and Workload Identity Federation
If you are looking for a less involved method for provisioning this infrastrcuture, consider taking a look at the published terraform module. This module performs the same actions documented below via terraform instead of the command line:
- TODO Terraform Module GOES HERE
- TODO Github Repository GOES HERE
- Make sure you have the following binaries installed on your local machine and you are familiar with the basic usage:
brew install google-cloud-sdk
brew install kubernetes-cli
brew install helm- Login to the gcloud cli
gcloud auth login- Set the following variables
# This can be found by running `gcloud compute zones list`. Choose a region close to you.
export GCP_COMPUTE_ZONE="us-central1-a"
# Name for the create GCP project (usually this is done via doormat)
export GCP_PROJECT_NAME="support-max-test"
# Choose a name for the cluster
export GKE_CLUSTER_NAME="test-cluster"
# Custom role for managain GCP/Vault auth method
export GCP_CUSTOM_IAM_ROLE_VAULT_AUTH="vault_gcp_auth_manager_role"
# Add your license here
export VAULT_LICENSE=""
# Choose a name for the k8s service account
export KSA_NAME="vault-auth-k8s-sa"
# Choose a name for the GCP IAM service account
export GSA_NAME="vault-auth" - Set the project ID
export GCP_PROJECT_ID=$(gcloud projects list --filter="name:$GCP_PROJECT_NAME" --format="value(projectId)")
gcloud config set project $GCP_PROJECT_ID- Create a GKE cluster and add the credentials to your local .kubeconfig
gcloud services enable container.googleapis.com
gcloud container clusters create $GKE_CLUSTER_NAME \
--num-nodes=1 \
--machine-type=e2-small \
--zone=$GCP_COMPUTE_ZONE \
--workload-pool=$GCP_PROJECT_ID.svc.id.goog
gcloud components install gke-gcloud-auth-plugin
gcloud container clusters get-credentials $GKE_CLUSTER_NAME- Create IAM service account K8s service account and annotate K8s service account with IAM service account information.
gcloud iam service-accounts create $GSA_NAME
kubectl create sa $KSA_NAME
kubectl annotate serviceaccount vault-auth-k8s-sa \
--namespace default \
iam.gke.io/gcp-service-account=$GSA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com- Grant IAM service account permissions to manage Vault GCP auth method, grant K8s service account the workload identity user role.
gcloud iam roles create $GCP_CUSTOM_IAM_ROLE_VAULT_AUTH \
--project=$GCP_PROJECT_ID \
--permissions="iam.serviceAccounts.get,iam.serviceAccountKeys.get"
gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \
--member="serviceAccount:$GSA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--role="projects/$GCP_PROJECT_ID/roles/$GCP_CUSTOM_IAM_ROLE_VAULT_AUTH"
gcloud iam service-accounts add-iam-policy-binding $GSA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com \
--role "roles/iam.workloadIdentityUser" \
--member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[default/$KSA_NAME]"- Create a secret for the vault enterprise license
kubectl create secret generic vault-ent-license --from-literal="license=$VAULT_LICENSE"- Deploy Vault using helm
cat <<EOF | tee values.yaml | helm install vault hashicorp/vault --values -
injector:
enabled: false
server:
enterpriseLicense:
secretName: vault-ent-license
image:
repository: hashicorp/vault-enterprise
tag: 1.17.3-ent
affinity: ""
serviceAccount:
create: false
name: $KSA_NAME
ha:
enabled: true
raft:
enabled: true
setNodeId: true
config: |
storage "raft" {
path = "/vault/data/"
retry_join {
leader_api_addr = "http://vault-0.vault-internal.default.svc.cluster.local:8200"
}
retry_join {
leader_api_addr = "http://vault-1.vault-internal.default.svc.cluster.local:8200"
}
retry_join {
leader_api_addr = "http://vault-2.vault-internal.default.svc.cluster.local:8200"
}
}
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_disable = "true"
}
service_registration "kubernetes" {}
EOF- Initialize and unseal vault, login as root, and configure the GCP auth method
kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > init.json
export VAULT_UNSEAL_KEY=$(cat init.json | jq -r '.unseal_keys_hex[]')
kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
kubectl exec vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
kubectl exec vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY- Login as root, configure the GCP auth method, create a static KV-v2 secret
export VAULT_ROOT_TOKEN=$(cat init.json | jq -r '.root_token')
kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKEN
kubectl exec vault-0 -- vault auth enable gcp
kubectl exec vault-0 -- sh -c 'vault policy write viewer - <<EOF
path "*" {
capabilities = [ "read" ]
}
EOF'
kubectl exec vault-0 -- vault write auth/gcp/role/test bound_service_accounts="*" type=iam max_jwt_exp=60m token_policies=viewer
kubectl exec vault-0 -- vault secrets enable -version=2 kv
kubectl exec vault-0 -- vault kv put kv/webapp/config password=1234- Deploy the Vault Secrets Operator via helm, and create associated resources
helm install vault-secrets-operator hashicorp/vault-secrets-operator
kubectl apply -f - <<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
name: default
namespace: default
spec:
address: http://vault.default.svc.cluster.local:8200
skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: default
namespace: default
spec:
vaultConnectionRef: default
method: gcp
mount: gcp
gcp:
role: test
workloadIdentityServiceAccount: $KSA_NAME
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: vault-kv-app2
namespace: default
spec:
type: kv-v2
mount: kv
path: webapp/config
destination:
name: secretkv
create: true
refreshAfter: 1h
vaultAuthRef: default
syncConfig:
instantUpdates: true
EOF
- Verify the K8s secret is created
kubectl get secret secretkv -o jsonpath='{.data.password}' | base64 -dgcloud container clusters delete $GKE_CLUSTER_NAME --zone $GCP_COMPUTE_ZONE
gcloud iam service-accounts delete $GSA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com
rm init.json