Upload
gareth-rushgrove
View
226
Download
0
Embed Size (px)
Citation preview
(without introducing more risk)
The Challenges of Container Configuration
PuppetGareth Rushgrove
New capabilities and associated problems
(without introducing more risk)
Gareth Rushgrove
@garethr
(without introducing more risk)
Gareth Rushgrove
(without introducing more risk)Configuration
What is it and why should I care
(without introducing more risk)
Gareth Rushgrove
(without introducing more risk)synonyms: design, grouping, marshalling
Gareth Rushgrove
(without introducing more risk)Marshalling your containers
Gareth Rushgrove
(without introducing more risk)
- Immutability and containers- Runtime vs build time- Who configures the orchestrator?
Gareth Rushgrove
(without introducing more risk)Mainly Docker and Kubernetes examples, but should be generally applicable
Gareth Rushgrove
(without introducing more risk)
Everything is immutable now?
Assumptions vs reality
(without introducing more risk)
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker run -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"
(without introducing more risk)
Gareth Rushgrove
$ docker exec a7a01beb14de touch /tmp/surprise
(without introducing more risk)
Gareth Rushgrove
$ docker diff a7a01beb14deC /tmpA /tmp/surprise
(without introducing more risk)Containers are notimmutable by default
Gareth Rushgrove
(without introducing more risk)Containers are not immutable by default
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker run --read-only -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"
(without introducing more risk)
Gareth Rushgrove
$ docker exec 379150b2cf05 touch /tmp/surprisetouch: cannot touch '/tmp/surprise': Read-only file system
(without introducing more risk)SuggestionEnable read-only where possible
Gareth Rushgrove
(without introducing more risk)Many applications won’t start with a read-only filesystem
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
1 import logging2 from logging.handlers import RotatingFileHandler3 4 from flask import Flask5 6 app = Flask(__name__)7 8 @app.route('/')9 def home():10 app.logger.info('log request')11 return "We'll never get to here"1213 if __name__ == '__main__':14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)15 handler.setLevel(logging.INFO)16 app.logger.addHandler(handler)17 app.run(debug=True, host='0.0.0.0')
(without introducing more risk)
Gareth Rushgrove
1 import logging2 from logging.handlers import RotatingFileHandler3 4 from flask import Flask5 6 app = Flask(__name__)7 8 @app.route('/')9 def home():10 app.logger.info('log request')11 return "We'll never get to here"1213 if __name__ == '__main__':14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)15 handler.setLevel(logging.INFO)16 app.logger.addHandler(handler)17 app.run(debug=True, host='0.0.0.0')
(without introducing more risk)
Gareth Rushgrove
$ docker run --read-only -p 5000:5000 garethr/flaskapp Traceback (most recent call last): File "app.py", line 14, in <module> handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1) File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__ BaseRotatingHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open stream = open(self.baseFilename, self.mode)IOError: [Errno 30] Read-only file system: '/app/app.log'
(without introducing more risk)tmpfs support added inDocker 1.10
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker run --read-only --tmpfs /tmp \ -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"
(without introducing more risk)
Gareth Rushgrove
$ docker exec 222331443d28 touch /tmp/surprise
(without introducing more risk)
Gareth Rushgrove
$ docker diff 222331443d28
(without introducing more risk)
Gareth Rushgrove
$ docker exec 222331443d28 ls /tmpsurprise
(without introducing more risk)SuggestionUse tmpfs only where needed
Gareth Rushgrove
(without introducing more risk)
RememberWithout technical controls you only have social guaranteesof immutability
Gareth Rushgrove
(without introducing more risk)Build vs Run
And the relationship between them
(without introducing more risk)
Given an image- What machine built this image?- Are all the licenses compatible?- Who supports this image?- Does this image contain malware?
Gareth Rushgrove
(without introducing more risk)
Given a running container- Who built it?- How was it built?- What software does it contain?- Is the software up-to-date?
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask
COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]
(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask
COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]
Where did this base image come from?
(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask
COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]
What packages are installed? At what version?
Where are those packages from?
(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask
COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]
What version of flask is this?
(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask
COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]
What was in this folder at build time?
(without introducing more risk)The importance of time
Gareth Rushgrove
(without introducing more risk)How often are images rebuilt?
Gareth Rushgrove
(without introducing more risk)Rebuilding only on code change ignores environmental factors
Gareth Rushgrove
(without introducing more risk)Versioning and metadata
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"
(without introducing more risk)
Gareth Rushgrove
LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"
What time? What timezone?
(without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}" \ 4fa6e0f0c678 | jq{ "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12"}
(without introducing more risk)
SuggestionDecide upon and enforce metadata standards
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
LABEL com.example.git.repository="https://github.com/puppetlabs/puppet" \ com.example.git.sha="dc123cfb5ed4dca43a84be34a99d7c1080126fa4" \ com.example.build.time="2016-04-24T15:43:05+00:00" com.example.build.builder=“jenkins1.example.com" \ com.example.docs="https://github.com/puppetlabs/docker ...
(without introducing more risk)SuggestionEmbed Dockerfiles in images
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq{ "net.morethanseven.dockerfile": "/Dockerfile",}
(without introducing more risk)
Gareth Rushgrove
$ docker run -i -t garethr/alpine cat /DockerfileFROM alpineLABEL net.morethanseven.dockerfile="/Dockerfile"RUN apk add --update bash && rm -rf /var/cache/apk/*COPY Dockerfile /
(without introducing more risk)SuggestionProvide an API for your containers
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq{ "com.example.api.packages": "apk info -vv"}
(without introducing more risk)
Gareth Rushgrove
$ docker run -i -t garethr/alpine apk info -vvmusl-1.1.11-r2 - the musl c library (libc) implementationbusybox-1.23.2-r0 - Size optimized toolbox of many common UNIX utilitiesalpine-baselayout-2.3.2-r0 - Alpine base dir structure and init scriptsopenrc-0.15.1-r3 - OpenRC manages the services, startup and shutdown of a hostalpine-conf-3.2.1-r6 - Alpine configuration management scripts
(without introducing more risk)DEMO
(without introducing more risk)
Who configures the scheduler?
Higher level configuration
(without introducing more risk)
Schedulers/orchestrators abstract you from- Where individual containers run- Balancing due to new resources- Balancing due to failed resources
Gareth Rushgrove
(without introducing more risk)This results in a constraintsbased system
Gareth Rushgrove
(without introducing more risk)Which means those constraints need to be explicit and correct
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"
(without introducing more risk)
Gareth Rushgrove
$ docker run -d -P \ -e constraint:storage==ssd --name db mysql
(without introducing more risk)
Gareth Rushgrove
1 template:2 metadata:3 labels:4 app: guestbook5 tier: frontend6 spec:7 containers:8 - name: php-redis9 image: gcr.io/google_samples/gb-frontend:v410 resources:11 requests:12 cpu: 100m13 memory: 100Mi14 env:15 - name: GET_HOSTS_FROM16 value: dns17 # If your cluster config does not include a dns service, then to18 # instead access environment variables to find service host19 # info, comment out the 'value: dns' line above, and uncomment the20 # line below.
(without introducing more risk)How do you manage properties for all of your hosts?
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"
Does this machine really have an SSD?
What if someone swaps the drive?
(without introducing more risk)SuggestionUse properties of hosts as labels
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ facter | head -n 20 architecture => x86_64domain => localfacterversion => 2.4.6fqdn => Pro.localgid => staffhardwareisa => i386hardwaremodel => x86_64hostname => Proid => garethrinterfaces => lo0,gif0,stf0,en0,p2p0,awdl0,en1,en2,bridge0,vboxnet0,vboxnet1,vboxnet10,vboxnet11,vboxnet12,vboxnet13,vboxnet14,vboxnet2,vboxnet3,vboxnet4,vboxnet5,vboxnetipaddress => 192.168.0.5ipaddress_en0 => 192.168.0.5ipaddress_lo0 => 127.0.0.1
(without introducing more risk)
Gareth Rushgrove
$ facter -j os | jq { "os": { "name": "Darwin", "family": "Darwin", "release": { "major": "14", "minor": "5", "full": "14.5.0" } }}
(without introducing more risk)
Gareth Rushgrove
$ docker daemon \ --label net.example.os=`facter operatingsystem` \ --label net.example.virtual=`facter is_virtual` \ --label net.example.kernel=`facter kernelversion` \ ...
(without introducing more risk)
Orchestrators also tend to introduce new higher-level primitives
Gareth Rushgrove
(without introducing more risk)
Docker Networks, Kubernetes Services, ReplicationControllers, Chronos Jobs
Gareth Rushgrove
(without introducing more risk)Many with imperative interfaces
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
$ kubectl get pod mypod -o yaml \ | sed 's/\(image: myimage\):.*$/\1:v4/' \ | kubectl replace -f -
(without introducing more risk)
Gareth Rushgrove
$ docker network create bobc0a0f4538d259515813b771264688d37aaedb41098379a0d73ec0ca08926fe68
$ docker network create bobError response from daemon: network with name bob already exists
And everything configuredin YAML
Gareth Rushgrove
Code plus data has advantagesover data alone
Gareth Rushgrove
The language to represent the data shouldbe a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language
Gareth Rushgrove
Borg, Omega, and Kubernetes, ACM Queue, Volume 14, issue 1 http://queue.acm.org/detail.cfm?id=2898444
“
Avoid repetitionCombine external inputsCorrectnessAbstractions
Gareth Rushgrove
----
SuggestionUse a higher level programming tool for generating config data
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
jsonnet.org
(without introducing more risk)
Gareth Rushgrove
$ cat test.jsonnet// Example jsonnet file{ person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: "Bob" },}
(without introducing more risk)
Gareth Rushgrove
$ jsonnet test.jsonnet | jq { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }}
(without introducing more risk)
Gareth Rushgrove
$ jsonnet test.jsonnet | json2yaml --- person1: name: "Alice" welcome: "Hello Alice!" person2: name: "Bob" welcome: "Hello Bob!"
(without introducing more risk)DEMO
(without introducing more risk)
Gareth Rushgrove
garethr/kubernetes
(without introducing more risk)
Gareth Rushgrove
kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] },}
(without introducing more risk)
Gareth Rushgrove
apiVersion: v1kind: Podmetadata: namespace: default name: sample-podspec: container: - image: nginx name: container-name
(without introducing more risk)
Gareth Rushgrove
controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379,}
(without introducing more risk)
Gareth Rushgrove
apiVersion: v1kind: Servicemetadata: name: redis-master labels: app: redis tier: backend role: masterspec: ports: # the port that this service should serve on - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master---apiVersion: v1kind: ReplicationControllermetadata: name: redis-master # these labels can be applied automatically # from the labels in the pod template if not set labels: app: redis role: master tier: backendspec: # this replicas value is default # modify it according to your case replicas: 1 # selector can be applied automatically # from the labels in the pod template if not set # selector: # app: guestbook # role: master # tier: backend template:
(without introducing more risk)DEMO
(without introducing more risk)Conclusions
New technology means old problems
(without introducing more risk)
The difference between how you think something works and howit actually works riskshard-to-debug production issues
Gareth Rushgrove
(without introducing more risk)Containers introduce new and old configuration problems
Gareth Rushgrove
(without introducing more risk)
Configuration managementis the discipline aimed atminimising those risks
Gareth Rushgrove
(without introducing more risk)Start with principles
Gareth Rushgrove
(without introducing more risk)
- Identification- Control- Status accounting- Verification
Gareth Rushgrove
Military Handbook Configuration Management Guidance MIL-HDBK-61B
(without introducing more risk)
Apply them to yourcontainer basedinfrastructure today
Gareth Rushgrove
(without introducing more risk)Questions?And thanks for listening