A Survey of Container Security in 2016: A Security Update on Container Platforms

A Survey of Container Security in 2016: A Security Update on Container PlatformsSalman A. Baset, Research Staff Member, @salman_basetPhil Estes, Senior Technical Staff Member, @estesp

Recap and Background

2015• Unraveling Docker Security: Lessons from Production

Cloud (OpenStack Summit Tokyo, October 2015)

2016• Container Security (Docker London, June 2016)• How Secure is Your Container? (ContainerCon Berlin,

August 2016)

Today’s talk:• Container Security: Survey and Overview, and a

developer focused comparison of secure application deployment & configuration (OpenStack Summit Barcelona, October 2016)

Outline• Container Overview• Usage and Threat Models• Container Security Evolution• Developer Focus: Running Applications (Apache) Inside

Containers vs Host• Why Containers are Better for Security?• What are the New Security Issues?

• Docker vs. LXC vs. Rocket (NCC group report)• Ongoing work in Container Security

Raise your hand if you think that…• Running applications in hosts (VM / baremetal) vs. running

them inside containers on a VM / baremetal is: • more “secure”• no difference• less “secure”

• We think that running applications inside a container is more “secure” than running them directly on a VM / baremetal

• This is because containers allow deep visibility, and usable and developer-focused security.

But first what is a container?

What makes a container?

pid mount

IPC

user network

uts

> Linux kernel namespaces provide the isolation (hence “container”) in which we place one or more processes

> Linux kernel cgroups (“Control groups”) provide resource limiting and accounting (CPU, memory, I/O bandwidth, etc.)

Container usage and threats?

Single tenant, trusted codeLess concern regarding

malicious attack vectorsMulti-tenant, (organizationally)

trusted code ”Noisy neighbor” a higher priority

than protection from unknownsMulti-tenant, untrusted code

Similar to internet-facing application, but with addition of potential malicious “neighbors”

DoS Host (use up CPU, memory, disk), Forkbomb

Container, even with isolation, is able to use up host OS resources

Access host/private information

Container is allowed to access (maybe inadvertently) host details

Kernel modification/insert module

Given the shared kernel underlying containers; modifying the kernel’s configuration/code/inserting code

Administrative access (API socket access)

Access to the container “control” socket/system, allowing full admin privileges of host system

Exploits in OS libraries/code in container

Latent/unpatched exploits in libraries and/or binaries in the container’s filesystem

Container provenance

Validating/certifying the container image I am running is what I expect

THREAT DESCRIPTION

Kernel and Container Security Evolution• 2015

• Content Addressability (image spec, phase 1) – Docker 1.6• Default ulimits for all containers – Docker 1.6• Docker Content Trust – (notary) image provenance/signing – Docker 1.8

• 2016• Full migration to content addressability for images/layers – Docker 1.10• User namespaces – Docker 1.10• Secure computing (libseccomp) – Docker 1.10• --pids-limit (cgroups pid limitation) – Docker 1.11 (kernel 4.3 +)• cgroups “v2” – (kernel 4.5+)• --no-new-privileges (limit process escalation) – Docker 1.11• Storage driver quotas (limited) – Docker 1.12• Secure by default multi-node orchestration (mutual TLS) – Docker 1.12

Kernel and – save chart until we decide-2015 2016

Kernel – 3.13 Docker – 1.6 Kernel – 4.4 Docker – 1.12

cgroup pid limits

Configuring for Security: Hard & Tedious Work!

• Center for Internet Security (CIS) security “benchmarks”. • Consensus driven approach for secure configuration for host and key applications.• https://www.cisecurity.org/

• Operating Systems• Red Hat 7 CIS benchmark – v2.1.0 – June 2nd, 2016

• 304 pages• Ubuntu 14.04 CIS benchmark – v2.0.0 – Sep 30, 2016

• 296 pages

• Apache HTTP Server 2.4 benchmark – v1.3.0 – May 31st, 2016• 163 pages

• Docker 1.12 CIS benchmark – v1.0.0 – August 15, 2016• 189 pages

• OWASP rules for Apache• 60 mod security rules

Installing Applications in a Container vs. Directly on Host• Host configuration and hardening is needed whether

applications run in a container on a host, or directly on host. This is typically a one-time operation.

• Launching applications in a container on a host [is] a recurring operation. Container makes it easy to secure host from applications, making security usable.

HostHost

Container

• Apache webserver is among the most popular web servers in use today.

• We think that deploying Apache web server in a container is more secure than deploying directly on a host.

• Let’s compare Apache installation in a container on a host versus directly on a host to better understand our assertion.

Running Apache web server on a container vs. host (1/10)

Filesystem Size Mounted on CIS rules/dev/disk1 100G /

/dev/disk1 10G /tmp Ubuntu CIS 1.1.2

/dev/disk2 100G /var/log Ubuntu CIS 1.1.10

/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11

/dev/disk4 100G /mnt/httpd

HOST

STEP 1: • Configuring host file system• Configure separate partitions for Docker• Install Docker Engine

Preparing host list is not an exhaustive list and is meant as an illustration.

Filesystem Size Mounted on CIS rules

/dev/disk1 100G /

/dev/disk1 10G /tmp Ubuntu CIS 1.1.2

/dev/disk2 100G /var/log Ubuntu CIS 1.1.10

/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11

/dev/disk4 100G /var/lib/docker Docker CIS 1.1

/dev/disk5 100G /var/dockervols

CONTAINER (DOCKER) HOST

Agents / daemonsSSHDockerRunc

Agents / daemonsSSH

STEP 1: • Configuring host file system• Configure separate partitions for Apache

Goal: Configure separate partition


Configure Config file Description CIS rulesSSH /etc/ssh/sshd_config Configure SSH

properlyUbuntu CIS 5.2

User accounts

/etc/passwd, /etc/shadow

Configure user accounts per strict policy

Ubuntu CIS 5.4

Apache user Configure Apache user and run apache under it

Apache CIS 1.3

Configure Config file Description CIS rulesSSH /etc/ssh/sshd_config Configure SSH

properlyUbuntu CIS 5.2

User accounts

/etc/passwd, /etc/shadow

Configure user accounts per strict policy

Ubuntu CIS 5.4

User namespace user

Configure Docker engine with user namespace

Docker CIS 2.8

Apache Docker Image

Prepare Docker image for Apache with Apache user.

Apache CIS 1.3

HOST

STEP 2: • Manually configure Docker engine with user

namespace if non-root user not defined in Docker image.

• Use of kernel namespaces is by default.


Agents / daemonsSSHDockerRunc

Agents / daemonsSSH

STEP 2: • Configure non-root user for Apache.• Manually configure kernel namespaces.

Goal: Run processes as non-root utilizing namespaces. Essential for deprivileging


Name Configuration files

Description CIS rules

Audit daemon

/etc/audit/auditd.conf

Record system call activity Ubuntu CIS 4.1

Rsyslog /etc/rsyslog.conf A daemon for sending logs to a central server

Ubuntu CIS 4.2

Logging and log rotation

Configure log rotation to avoid disk full.

Ubuntu CIS 4.3Apache CIS 1.6

Monitoring Configure monitoring Apache CIS 1.6



Audit daemon

/etc/audit/auditd.conf

Record system call activity

Ubuntu CIS 4.1Docker CIS 1.7-1.15

Rsyslog /etc/rsyslog.conf A daemon for sending logs to a central server

Ubuntu CIS 4.2

Logging and log rotation

Configure log rotation to avoid disk full.

Ubuntu CIS 4.3Docker CIS 2.12

Monitoring Configure monitoring of containers

Docker CIS 6.2

HOST

STEP 3: • Configure audit, log collection, rotation, monitoring• Configure audit for Docker Engine• Collect Docker Engine and container logs


Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring

STEP 3: • Configure audit, log collection, rotation, monitoring• Collect Apache logs

Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring

Goal: Track system call activity, logs, and monitoring


Configure Config file Description CIS rules

NTP /etc/ntp.conf Ubuntu CIS 2.2.1

DNS DNS server not enabled. Ubuntu CIS 2.2.8

/proc/sys/net /etc/sysctl.conf /proc/net

Network related configs, e.g., net.ipv4.ip_forward=0

Ubuntu CIS 3.1/3.2

Firewalls iptables / security groups

Configure host firewalls Ubuntu CIS 3.6

Configure Config file Description CIS rules

NTP /etc/ntp.conf Ubuntu CIS 2.2.1

DNS DNS server not enabled. Ubuntu CIS 2.2.8

/proc/sys/net

docker

/etc/sysctl.conf /proc/net/etc/docker

Network related configs, e.g., net.ipv4.ip_forward=1--sysctl net.ipv4.ip_forward=0

Ubuntu CIS 3.1/3.2

Firewalls iptables / security groups

Configure host firewalls Ubuntu CIS 3.6

HOST

STEP 4: • Configure network and related services• Disable IP forwarding when starting container

engine.



STEP 4: • Configure network and related services

7. NTP8. DNS9. /etc/default/docker10. Firewalls

Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS

7. /proc/sys/net8. Firewalls

Goal: Configure network services, firewalls, IP forwarding


Name Configuration files Description CIS rulesSystem updates

/etc/apt/sources.list.d/*

Configure system updates using an agent

Ubuntu CIS 1.8

Antivirus / Malware detector

A daemon for detecting any malware on host

Name Configuration files Description CIS rulesSystem updates

/etc/apt/sources.list.d/*

Configure system updates using an agent

Ubuntu CIS 1.8

Antivirus / Malware detector

A daemon for detecting any malware on host

HOST

STEP 5: • Configure patching, and malware / antivirus agents

on host



STEP 5: • Configure patching, and malware / antivirus agents

on host

7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent


7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent

Goal: Configure software updates and malware / antivirus




Configure AppArmor (MAC)

Configure Apache AppArmor profile

/etc/apparmod.d/* Configure apparmor daemon and all profiles in enforcing

Ubuntu CIS 1.6

Apache CIS 1.12



Configure AppArmor (MAC)

Do not disable AppArmor

/etc/apparmod.d/* Configure apparmor daemon and all profiles in enforcing

Ubuntu CIS 1.6

Docker CIS 5.1

HOST

STEP 6: • Do not disable default Docker AppArmor profile.

Applied to running containers.



STEP 6: • Create an AppArmor profile for Apache. Complicated!

7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent13. AppArmor


7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor

Goal: Configure mandatory access control for defense in depth


Denial of service Description / mitigation CIS rulesNetwork Configure Apache for network DoS

mitigation.Apache CIS 1.9

Memory systemctl MemoryLimit=500M …

Resources (e.g., fork)

CPU systemctl CPUShares=…

Disks systemctl IO[Read|Write]BandwidthMax=…

ulimits Set ulimits

Denial of service Description / Mitigation CIS rulesNetwork Open issue #26767 (WIP)

Memory docker run -m

Resources (e.g., fork)

docker run --pids-limit

CPU docker run --cpu-set-cpus, --cpu-shares

Disks docker run --device-[read|write]-[i|b]ps

ulimits Set ulimits for open file descriptors Docker CIS 5.18

HOST

STEP 7: • Allow resource limits with an easy to use interface

to cgroups and ulimits for ANY container.



STEP 7: • Use systemd / upstart or manually configure

resource control using cgroups and ulimits using.Cumbersome.




Goal: Prevent Apache from DoSing host or other containers

Kernel: 4.5+ “cgroups v2”

https://github.com/docker/docker/issues/26767

https://github.com/docker/docker/issues/26767


Capabilities Description / mitigation CIS rules

Various Manual configuration

Capabilities Description / Mitigation CIS rules

Various Load a module, mount, network admin. Docker starts container with limited capabilities.

Docker CIS 5.3

HOST

STEP 8: • Launch containers as non-root with limited Linux

capabilities. Defaults are good.



STEP 8: • Manual selection and setting of capabilities is

cumbersome.




Goal: Further deprivilege running process and containers through limited Linux capabilities.

Although Apache is launched by a non-root user in container as well as on host, as part of defense in depth, it is good practice to further reducing capabilities available to the user.


Descriptiom CIS rules

Manual configuration of seccomp

Preparing host list is not an exhaustive list and is meant as an illustration.


Do not disable default seccomp profile Docker CIS 5.12

HOST

STEP 9: • Launch containers with limited set of allowed Linux

system calls. Defaults are good.



STEP 9: • Manually configuring undesired system calls

is cumbersome.

7. NTP8. DNS9. /etc/default/docker 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor



Goal: Further deprivilege running containers by limiting Linux system calls (Seccomp).

Although Apache is launched by a non-root user in container as well as on host with limited capabilties, as part of defense in depth, it is good practice to further reduce system calls that can be invoked.


STEP 10: • Configure Apache in a container. • Configuring Apache firewalls is a breeze.


STEP 10: • Configure Apache on host.• Potential cumbersome iptables configuration.

7. NTP8. DNS9. /etc/default/docker 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor14. Apache


7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor 12. Apache

Goal: Configure Apache for optimum security.

Apache config description CIS rules

Minimize Apache modules Apache CIS 1.2

Run Apache as a non-root user even if user name space is used

Apache CIS 1.3

Apache Access Control Apache CIS 1.4

Minimize features, content, and options Apache CIS 1.5

Configure TLS Apache CIS 1.7

Configure limits Apache CIS 1.8

HOSTCONTAINER (DOCKER) HOST

Not an exhaustive list and is meant as an illustration.

Apache config description CIS rules

Minimize Apache modules Apache CIS 1.2

Run Apache as a non-root user. Apache CIS 1.3

Apache Access Control Apache CIS 1.4

Minimize features, content, and options Apache CIS 1.5

Configure TLS Apache CIS 1.7

Configure limits Apache CIS 1.8

Running Apache web server on a container vs. host• A host is isolated from a misbehaving Apache running inside a

container. Various agents running on host are likely to remain functional, allowing root-cause analysis later.

• Same mechanisms are used to isolate one container from another on the same host.

HostHost

Container

Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring7. NTP8. DNS9. /etc/default/docker /proc is mounted read only for unprivileged containers 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor14. Apache

Agents / daemons1. SSH

2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS7. /proc/sys/net

8. Firewalls 9. Patching agent10. Malware agent11. AppArmor12. Apache

Running Application in a Container vs. Host – Defense in Depth

Configure partitions

Run as non-root, utilizing user and kernel namespaces

Configure log, monitoring, audit

Configure network, related services, disable IP forwarding

Configure patching and anti-virus agents

Configure mandatory access control

Prevent application from DoSing a host

Further deprivilege application through subset of Linux capabilities

Further Deprivilege application by limiting allowed system calls

Configure application security

HostContainer

Host

Manual

Manual

Manualper app

One time(in Docker)

Manualper app

ManualOne time

Step 1 Step 2 Step 3

Manual

Manual

Step 4

Manual

Manual

Step 5 Step 6

Manualper app

Out of box(in Docker)


Step 7

Manualper app

Step 8

Manualper app


Manualper app


Step 9 Step 10

Manualper app

Manual

What are the New Security Issues When Running Applications Inside Containers?

• Running containers as non-root• If a Linux user is not setup inside Dockerfile or Appc file to use the

application, it is best to enable user name spaces in Docker Engine.• We recommend to always enable user namespaces in Docker Engine.

• While using Docker, IP forwarding for containers should be disabled in Docker Engine.

• Docker / Rocket API is powerful. Enable with caution.

Observations from Production

• Application security misconfiguration is among the top causes for container compromise.

• IBM Vulnerability Advisor can identify vulnerable application packages and application security misconfigurations for containers in IBM Containers platform.

Docker vs LXC vs rkt

• Users/packagers won’t turn on security if it’s difficult (AppArmor profiles are hard to write; SELinux can be even harder)

• Sane defaults are tricky as well - someone’s app won’t work and they will complain

• Docker painstakingly tries to find a balance (e.g. DCT off by default, allowance for insecure registries)

* NCC Group report “Understanding and Hardening Linux Containers”, v1.1, p. 97, section 9.13

Ongoing work in Container Security• Not yet namespaced

• /dev, /proc/sys/*• However /proc is mounted as read-only

• Container security work == Kernel security work• Containers share kernel and make use of kernel isolation features

• Use hypervisor isolation for containers• runC is a step in this direction• Replace it with a container run-time that leverages hypervisor isolation,

e.g., Intel Clear Containers• Unikernels• Fully unprivileged containers

Thank You

Technology

A Survey of Container Security in 2016: A Security Update on Container Platforms