Chickens & EggsManaging secrets in AWS with Hashicorp Vault

Jeff Horwitz Jornaya

jhorwitz@jornaya.com

Applications need secrets

• Where do we store the secrets?

• How do we manage the secrets?

• How do servers/applications obtain the secrets?

Jornaya Environment

• Applications run in AWS in Autoscaling groups

• Configuration management via Chef server

• No PCI, HIPAA or other compliance requirements

• Secrets include passwords, private keys, API keys

Vault“Vault secures, stores, and tightly controls access to tokens,

passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing. Through a unified API, users can access an encrypted

Key/Value store and network encryption-as-a-service, or generate AWS IAM/STS credentials, SQL/NoSQL databases, X.509

certificates, SSH credentials, and more.”

https://www.vaultproject.io/

Vault✓ Platform independent

✓ High availability

✓ User & server authentication options

✓ Fine-grained access control

✓ Good language and tooling support

✓ Under active development

Other Options• Instance userdata

• Chef encrypted databags

• Chef vault

• Citadel

• KMS

Vault Server• Responds to client requests

• Interacts with backends

• storage, authentication, secret, audit

• Encrypts/Decrypts secrets with master key

• Master key is never stored on disk

Playing with Vault~$ vault server --dev==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.Vault is configured to only have a single unseal key. The roottoken has already been authenticated with the CLI, so you canimmediately begin using the Vault CLI.

The only step you need to take is to set the followingenvironment variables:

export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are reproduced below in case youwant to seal/unseal the Vault or play with authentication.

Unseal Key: e95cf9d02c044c67dfd5d379d3a56cee7a0209f8cc681fca1435b6c022fcf028Root Token: 611f6670-f871-ef2d-2110-87a54261d407

Talking to Vault• Vault provides an HTTP(S) RESTful API

• JSON responses

• The vault command is a user-friendly wrapper

• Modules available for various languages

• Config management (e.g. Chef via Ruby gem)

Reading & Writing Secrets~$ export VAULT_ADDR=http://127.0.0.1:8200~$ vault write secret/phillydevops/luggage-combo value=12345Success! Data written to: secret/phillydevops/luggage-combo

~$ vault read secret/phillydevops/luggage-combo

Key Value--- -----refresh_interval 720h0m0svalue 12345

HTTP API~$ curl -X POST -d '{"value": "12345"}' \ -H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407' http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo

HTTP API~$ curl -k -H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407' \ http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo | jq{ "request_id": "ac71da58-cc11-431b-598e-7eb3fe3e37a3", "lease_id": "", "renewable": false, "lease_duration": 2592000, "data": { "value": "12345" }, "wrap_info": null, "warnings": null, "auth": null}

Shamir’s Secret Sharing• https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing

• Splits a key into n shards (unseal keys)

• k shards needed to derive original key (k < n)

• No one person can easily obtain the original key

• No need to revoke shards if people leave

Unsealing Process

Unsealing Process• Vault server does not store the master key on disk

• Given the master key via the unsealing process

• Need minimum number of shards to unseal

• Key is only stored in memory

• Restarting will lose the key and “seal” the vault.

Unsealing Demo~$ vault read secret/phillydevops/luggage-comboError reading secret/phillydevops/luggage-combo: Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo

Code: 503. Errors:

* Vault is sealed~$ vault unsealKey (will be hidden): ********Sealed: falseKey Shares: 1Key Threshold: 1Unseal Progress: 0

Backends

Storage backends

Secret backends

Authentication backends

Audit backends

Storage Backends

• Physical storage for secrets

• Only see encrypted secrets

• High availability features

Storage BackendsBackend Support HA?Consul Official Yes

Zookeeper Community Yesetcd Community Yes

DynamoDB Community MaybeS3 Community No

Swift Community NoAzure Community Nomysql Community No

postgresql Community Noinmem Official No

file Official No

Storage BackendsBackend Support HA?Consul Official Yes

Zookeeper Community Yesetcd Community Yes

DynamoDB Community MaybeS3 Community No

Swift Community NoAzure Community Nomysql Community No

postgresql Community Noinmem Official No

file Official No

Storage BackendsBackend Support HA?Consul Official Yes

Zookeeper Community Yesetcd Community Yes

DynamoDB Community MaybeS3 Community No

Swift Community NoAzure Community Nomysql Community No

postgresql Community Noinmem Official Nofile Official No

Clustering• Storage backend must support high availability

• Active and standby servers

• State maintained by Consul (storage backend)

• Use Consul DNS or API to discover active server

• Standby will redirect to active by default

Cluster Status$ vault statusSealed: falseKey Shares: 5Key Threshold: 3Unseal Progress: 0Version: Vault v0.6.1Cluster Name: vault-cluster-abcd0123Cluster ID: deadbeef-beef-beef-beef-deadbeefcafe

High-Availability Enabled: trueMode: activeLeader: https://vault-1.example.com:8200

Secret Backends

• Static or dynamic secrets

• Backend configuration

• Multiple backend mountpoints

Generic Backend• Default backend

• Mounted at secret

• Stores and retrieves static secrets as k/v pairs

• Great for:

• username/password

• RSA private keys

• API keys

Tokens• Authentication in vault is performed via tokens

• UUIDs returned by authentication backends

• Tokens can expire and must be renewed

• Tokens can be revoked

• A single “root” token never expires

Authentication Backends

• Returns a token based on credentials

• Tokens may have a TTL (backend-dependent)

• Options to fit different workflows and platforms

Authentication Backends• AppRole (App ID)

• AWS EC2

• GitHub

• LDAP

• TLS Certificates

• Tokens

• Username & Password

LDAP Backend~$ vault auth -method ldap username=jhorwitzPassword (will be hidden):Successfully authenticated! You are now logged in.The token below is already saved in the session. You do notneed to "vault auth" again with the token.token: deadbeef-cafe-beef-beef-deadbeafcafetoken_duration: 28799token_policies: [admins, default]

Lookup a Token~$ vault token-lookup deadbeef-cafe-beef-beef-deadbeafcafeKey Value--- -----accessor 5776eb4b-05b1-3bab-98d3-08d34040a806creation_time 1480366497creation_ttl 28800display_name ldap-jhorwitzexplicit_max_ttl 0id deadbeef-cafe-beef-beef-deadbeafcafemeta map[policies:admins,default username:jhorwitz]num_uses 0orphan truepath auth/ldap/login/jhorwitzpolicies [admins default]renewable truettl 27379

aws-ec2 Backend

• Good for server-level authentication

• Fixes the chicken & egg problem on EC2 instances

• Let AWS do the dirty work for you

Instance Identity$ curl http://169.254.169.254/latest/dynamic/instance-identity/document{ "privateIp" : “10.0.0.1”, "devpayProductCodes" : null, "availabilityZone" : "us-east-1a", "accountId" : "1234567890", "version" : "2010-08-31", "instanceId" : "i-deadbeef", "billingProducts" : null, "instanceType" : "t2.nano", "pendingTime" : "2016-09-09T17:30:35Z", "architecture" : "x86_64", "imageId" : "ami-cafebeef", "kernelId" : null, "ramdiskId" : null, "region" : "us-east-1"}

Identity Signature$ curl http://169.254.169.254/latest/dynamic/instance-identity/pkcs7MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggGjewogICJwcml2YXRlSXAiIDogIjEwLjUwLjAuNjgiLAogICJkZXZwYXlQcm9kdWN0Q29kZXMiIDogbnVsbCwKICAiYXZhaWxhYmlsaXR5Wm9uZSIgOiAidXMtZWFzdC0xYSIsCiAgImFjY291bnRJZCIgOiAiMjk4Nzg1NDUzNTkwIiwKICAidmVyc2lvbiIgOiAiMjAxMC0wOC0zMSIsCiAgImluc3RhbmNlSWQiIDogImktYzRkMDQwZjUiLAogICJiaWxsaW5nUHJvZHVjdHMiIDogbnVsbCwKICAiaW5zdGFuY2VU…

Whitelisting

• Vault maintains a whitelist of instance IDs

• Cannot reauthenticate using an ID in the whitelist

• You can turn this off or remove ID from whitelist

• Use a nonce to prevent replay attacks

Roles• Clients specify a role with authentication requests

• Roles are “bound” to instance properties

• IAM role or instance profile

• AMI ID

• Vault cross-checks with EC2 before returning token

Creating a Role$ vault write auth/aws-ec2/role/deployer \ bound_iam_role_arn=arn:aws:iam::1234567890:instance-profile/deployer \ policies=deployer

Success! Data written to: auth/aws-ec2/role/deployer

Policies

• ACLs that are applied to roles, users and groups

• Applied at token creation

• Tokens can have multiple policies

Sample User Policypath "secret/test/*" { policy = "write"}

path "secret/*" { policy = "read"}

path "secret/classified/*" { capabilities = ["deny"]}

Sample Server Policypath "secret/env/prod/*" { policy = "read"}

path "secret/shared/*" { policy = "read"}

path "secret/ssl/private-key" { policy = "read"}

Integration

• Chef authenticates instance to aws-ec2 backend

• Chef stores token

• local filesystem (for scripts and apps)

• node run state (for Chef recipes)

Get PKCS7 Signaturedef instance_identity_pkcs7 uri = URI.parse('http://169.254.169.254/latest/dynamic/instance-identity/pkcs7') resp = Net::HTTP.get_response(uri) resp.body.delete("\n", '')end

Generate Noncenonce = SecureRandom.uuid

Send Login Requestdef aws_ec2_login(address, role, pkcs7, nonce) # find the leader, since Net::HTTP doesn't handle redirects uri = URI.parse("#{address}/v1/sys/leader") resp = Net::HTTP.get_response(uri) leader = JSON.parse(resp.body)['leader_address']

uri = URI.parse("#{leader}/v1/auth/aws-ec2/login") data = { 'role' => role, 'pkcs7' => pkcs7, 'nonce' => nonce }

req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json') req.body = data.to_json

resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http| http.request(req) end …

Write Token to Filefile node['leadid_vault']['client']['token_file'] do content lazy { node.run_state['token'] || node['leadid_vault']['client']['token'] } owner node['leadid_vault']['client']['uid'] group node['leadid_vault']['client']['gid'] mode '0440' sensitive true action :create_if_missing only_if { node['leadid_vault']['client']['enabled'] }end.run_action(:create)

Using the Token

• Chef recipes use the vault gem

• Other apps use consul-template

Configure Vault Chef gemdef configure(opts = {}) require 'vault' Vault.configure do |config| config.address = opts[:address] config.ssl_ca_cert = opts[:ssl_ca_cert] config.ssl_verify = opts[:ssl_verify] if opts[:token] config.token = opts[:token] elsif opts[:token_file] config.token = File.read(opts[:token_file]) else raise 'must specify either token or token_file' end endend

Helper to read secretsdef read(path) require 'vault' Vault.logical.read(path)end

Reading secrets in ChefChef::Recipe.send(:include, VaultHelpers)

if node['leadid_vault']['client']['enabled'] # Grab the Datadog API and Application Key from vault api_key = read( "#{node['leadid_base']['datadog']['secret_path']}" + "/datadog/api-key").data node.default['datadog']['api_key'] = api_key[:value]end

…

include_recipe 'datadog::dd-agent'include_recipe 'datadog::dd-handler'

consul-template$ consul-template \ -consul consul.example.com \ -template "/opt/jornaya/app.yml.ctmpl:/opt/jornaya/app.yml:service nginx restart"

consul-templatedatabase: host: db.example.com dbname: myapp{{with secret "secret/leadid/env/prod/db-user"}} user: {{.Data.username}} password: {{.Data.password}}{{end}}

consul-templatedatabase: host: db.example.com dbname: myapp user: donald password: WeHaveTheBestPasswords

Learnings• Don't use a self-signed certificate. There is pain.

• For SSL, name your servers or use static IP addresses.

• Clustering was the easiest part. Do it.

• Set up your path hierarchy ahead of time.

• Be prepared to RTFM.

• Different backends behave differently.

• Tokens do not live forever.

Questions?