Pivotal Engineering Journal

Technical articles from Pivotal engineers.

Let's use Vault - Part 1: Deploying Vault

How to deploy Vault on Kubernetes using Google Cloud Storage as its backend

Posted on by
Categories:   GCP    Kubernetes    Vault    Credhub   
Edit this post on GitHub.

Why are we doing this?

The reason for this post’s existence is to encourage more teams to make the effort to move their credentials, certificates and other relevant files - think bbl up to a proper secret store.

Most of us are guilty of taking the easy way out by checking in all our secrets and certs into a private git repo . And although we install tools like the cred-alert-cli git hook to keep us in check, let us face the reality of our frequent usage of the --no-verify flag.

Eventually our hasty human nature coupled with --no-verify will lead us to expensing the costly mistake of leaking credentials into a repo that is public or may become public in the future - OSS for the win!.







We need to take the stance of not storing any credentials in git.







This post is one of a multi-part post that will be published to help us deploy, configure and use Vault as part of our workday lives and CI systems. This will NOT be a post to debate the pros and cons of other tools like credhub or lpass and how those tools can be used to mitigate this problem.

Agenda

Create bucket and artifacts

  1. Log into your GCP account.
  2. Create a service account and restrict its access to the storage.objectAdmin role.
    • We are going to store the service account key in k8s as a secret. Vault will use this key to CRUD secrets into the object store bucket.
  3. Create a bucket in GCS.
    • In this example, we just named it myteam-vault-bucket
  4. Genereate TLS certs for the ingress service so we can expose it via https.
    • Our team decided to generate certs from SSL For Free which leverages Let’s Encrypt for generating the certs.
    • Follow the instructions to generate the cert for the appropriate domain and add the TXT record in Route 53.
    • In this example, the URL the cert was generated for was https://vault.myteam.ci.cf-app.com

Configure the helm chart values.

There are comments in the config to explain the what and why of the properties.

# We create an Ingress service to expose vault and restrict it to https
# connections.
service:
  type: NodePort
ingress:
  enabled: true
  hosts:
  - vault.myteam.ci.cf-app.com
  annotations:
    kubernetes.io/ingress.allow-http: false
  tls:
  - hosts:
    - vault.myteam.ci.cf-app.com
    # The LetsEncrypt signed certs are stored as a k8s secret as well.
    secretName: vault-tls
vault:
  # By default the Vault Helm Chart will install in dev mode. Make sure to
  # turn this `dev` mode off!!
  dev: false
  customSecrets:
  # This secret will be uploaded into k8s as part of the Installation.
  - secretName: vault-gcs-service-account
    mountPath: /vault/sa
  config:
    api_addr: "https://vault.myteam.ci.cf-app.com"
    storage:
      gcs:
        bucket: myteam-vault-bucket
        ha_enabled: \"true\"
        credentials_file: /vault/sa/key.json

Helm Install

export NAMESPACE_NAME="myteam-vault"
kubectl create namespace "$NAMESPACE_NAME"

kubectl create secret generic vault-gcs-service-account \
    --from-file=key.json="/tmp/sa.json" \
    --namespace "$NAMESPACE_NAME"

kubectl create secret tls vault-tls \
    --cert "/tmp/vault-tls.crt" \
    --key "/tmp/vault-tls.key" \
    --namespace "$NAMESPACE_NAME"

helm repo add incubator \
    http://storage.googleapis.com/kubernetes-charts-incubator

helm install incubator/vault \
    --name vault \
    --values "/tmp/helm-config-values.yml" \
    --namespace "$NAMESPACE_NAME"

Other Manual Steps Required

After vault is deployed we need to manually edit the ingress service. See the github issue below descrbing the reason for this change.

This will show you the ingress service.

kuebctl get ingresses --namespace $NAMESPACE_NAME

By default the name of the ingress will be <ReleaseName>-<ChartName> which in this case will be vault-vault. If you’d like to override it, specify the property nameOverride in the helm config.

Edit the ingress service and remove path: /

kubectl edit ingresses vault-vault --namespace $NAMESPACE_NAME

More Info:

  • nameOverride: If you’d like to override the ingress service name.
  • Github Issue: the reason we have to remove path: /

Tada!!

You may now target your vault using the vault CLI

export VAULT_ADDR=https://vault.myteam.ci.cf-app.com
vault status

Understandably this is a bit of a 🐔 and 🥚 problem where we need some secrets to stand up our secret store. We’ve decided to store these secrets in LastPass.

Cleanup

  • Make sure to destroy sensitive information like the service account key and certs from your local machine.

Next…

Let’s configure vault for your team