1

Keeping Secrets:GKE with

Hashicorp Vault

2

{Aaron Bowden: Customer Engineer}

Agenda

Secret Management and Vault Overview

Run Vault on GKE

Connect to Vault from GKE

Round up

3

01 Secret Management and Vault overview

4

Secrets Overview

What is a “Secret”?

5

Username and Password

API Token

TLS Certificate

Credit Card

Tax File Number

Passport Number

Authorize and Authenticate

Confidential and sensitive

Secrets Overview

Secrets management

6

How do applications get secrets?

How do people get secrets?

How do the secrets get updated?

How do secrets get revoked?

When were secrets used and by whom?

What happens if a secret is compromised?

Secrets Overview

Where is the world now?

7

Secrets sprawled

Decentralized

Limited access control

Poor Visibility

8Ref: A Beautiful Mind, 2001 - https://www.imdb.com/title/tt0268978/

Introducing Hashicorp Vault

So What does Vault do?

9

Single Source of Secrets

Programmatic Access (API)

Manual Access (CLI/UX)

Cloud Friendly

Introducing Hashicorp Vault

Secure Storage

10

Data encrypted in transit

Data encrypted at rest

Hierarchical key/value store

TLS 1.2 for clients

256 bit AES-GCM

$ vault write secret/foo Aaron_pw=Amiga1200Success! Data written to: secret/foo

$ vault read secret/fooKey Value---- -----Refresh_interval 768h0m0sAaron_pw Amiga1200

Introducing Hashicorp Vault

CLI Example

11

Writing and reading keys from Vault. Keys are stored in a hierarchical layout, and encryption is transparent to the client.

$ curl \--header ‘x-vault-token: …’ \--data ‘{“data”: {“Aaron_pw”: “Amiga1200”}}

\$VAULT_ADDR/v1/secret/data/foo

$ curl \--header ‘x-vault-token: …’ \$VAULT_ADDR/v1/secret/data/foo

Introducing Hashicorp Vault

API Example

12

Most interactions with Vault are also available via the API.

Vault Overview

13

What if ... ? An operator copies passwords to ~/passwords.txt

An application logs the database credentials to disk?

A server error includes the API token in the diagnostic output?

Vault Overview

14

Dynamic Secrets Dynamic secrets are ephemeral and unique

Unique credentials give us a tight audit trail

Ephemeral secrets time-bound access

Access can be revoked early in case of a breach

Vault Overview

15

Dynamic Secrets

Request Credentials

CredentialsAudit Request

Create Credentials

Vault Overview

16

Secrets Backends Cloud Providers

GoogleAWSAzure

Databases

(*)SQLOracleMongoDB

Other Systems

RabbitMQSSHTLS\PKIActive DirectoryHashiCorp NomadHashiCorp Consul

Vault Overview

17

Authentication

Vault clients provide their identity and authenticate before accessing secrets. Vault has platform specific integrations to asset client identity

Authorization

Client identity is mapped to groups and policies which govern which secrets are accessible. A fine-grained authorization language allows control of which resources and operations are available.

Auditing

Rich audit logging on requests and responses provides a full audit trail of which secrets were accessed by which clients.

02 Run Vault on GKEPower of Google Kubernetes Engine plus security of Hashicorp Vault

18

Recommended Architecture on Kubernetes

19

Vault on GKE

Architecture for deploying Vault inside Kubernetes in a highly available manner.

github.com/sethvargo/vault-on-gke

KubernetesEngine

Cloud

KMSCloudStorage

Cloud LoadBalancing

HashiCorp

Vault Server

HashiCorp

Vault Server

HashiCorp

Vault Server

Recommended Architecture Highlights

20

Dedicated GKE Cluster

HashiCorp Vault runs in a dedicated Kubernetes cluster, in a dedicated Google Cloud project to which access is tightly controlled. No other apps run in this cluster.

github.com/sethvargo/vault-on-gke

Google Cloud Storage

Vault stores its encrypted data and configuration in Google Cloud Storage, which enables Vault to run in high availability mode. Google Cloud Spanner is also an option.

Google Cloud KMS

Vault servers are automatically initialized and unsealed by a sidecar container. The initial root token and unseal keys are encrypted with Google Cloud KMS key.

Recommended Architecture Highlights

21

Custom CA for TLS

TLS is managed via a custom certificate authority (CA), All Vault communication should be secured via TLS.

This could be replaced by a trusted CA like Let’s Encrypt.

github.com/sethvargo/vault-on-gke

L4 Load Balancing

GCP L4 regional load balancer delegates TLS to Vault instead of terminating TLS at a L7 load balancer.

This gives Vault full control over TLS and aids in using client certificates as a means for authentication.

Vault as a Service

Vault is available at an IP/DNS address for consumers. K8S is simply an implementation detail.

Specifically, we do not need to leverate K8S-specific features like service discovery.

---apiVersion: apps/v1kind: StatefulSetmetadata: # ...

Deployment Details

StatefulSet

22

The StatefulSet gives us predictable naming, exclusive semantics and a free circuit breaker.

spec: affinity: podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:-labelSelector: matchExpressions:

- key: app operator: In values: [‘vault’]

Deployment Details

Anti-Affinity Rules

23

Require each Vault server runs on a unique host. This provides added availability in case on node goes down.

containers:- name: vault-init image: sethvargo/vault-init:0.1.1

- name: vault image: vault:0.10.3 Args: [‘server’]

Deployment Details

Auto Unsealing

24

A sidecar container automatically initializes and unseals each Vault server.

containers:- # ...

securityContext: capabilities: add: [‘IPC_LOCK’]

Deployment Details

Memory Locking

25

Containers are granted the proper security capabilities to perform memory locking.

readinessProbe: httpGet: path: /v1/sys/health?standbyok=true port: 8200 Scheme: HTTPS initialDelaySeconds: 5 periodSeconds: 5

Deployment Details

Readiness Checks

26

Check the status, allowing standby servers to be “healthy” so they will be added to the load balancer.

---apiVersion: v1kind: Servicespec: type: LoadBalancer loadBalancerIP: 11.22.33.44 Selector: App: vault

Deployment Details

Load Balancer

27

The load balancer binds to the IP address of the external IP address from Google Cloud.

02 Demo time.

28

03 Connect to Vault from GKEAcquiring Vault secrets from inside Kubernetes pods and services

29

Vault Auth Methods

Vault Tokens

30

User or machines supply information that is validated by Vault. Vault then uses metadata to assign policies to a token.

HashiCorp

Vault Server

User or Machine

Third Party System

Vault Policy3. Map policy to metadata

1.Send information

2. Verify Information

4. Return Vault token

Vault Auth Methods

Auth to Kubernetes

31

Leverage the K8s TokenReviewer API to authenticate a pod to Vault and get a Vault Token.

HashiCorp

Vault Server

Service Account

Vault Policy3. Map policy to metadata

1.Send information

2. Verify Information

4. Return Vault token

TokenReviewer

initContainers: - name: vault-authenticator image: sethvargo/vault-kubernetes-authenticator volumeMounts: - name: vault-token mountPath: /home/vault

Deployment Details

Vault Authenticator

32

An init container pills the pod’s service account from disk, authenticates to Vault, and puts the Vault token inside a shared volume.

Yeah, but... What happens if credentials change? How will I signal my application to re-read the secrets file?

33

X_VAULT_TOKEN=$(</home/vault/.vault-token)

GCP_CREDS=$(curl -k -H "X-Vault-Token: "$X_VAULT_TOKEN"" -X GET "$VAULT_ADDR"/v1/gcp/key/ksdemo-key-roleset |jq -r '.data.private_key_data' | base64 --decode; echo)

echo $GCP_CREDS > /home/demo gcp.json

Deployment Details

Using Credentials

34

Authentication flow with token. Pick up the token, use it to obtain a secret and then apply it for use.

03 Demo time

35

Thank you