Containerize all the things

Agenda● Part 1: The concept of containers ● Part 2: Threats and Vulnerabilities● Part 3: Mitigating threats and best practices● Part 4: Take aways

Part 1: Docker on a calm see

What are containers ?● A way to isolate applications/processes from other

collocated processes or the host they are running on● Enables the packaging of an application and all its

dependencies● Portable unit: can run on a boat, truck, be lifted by a

tower crane,...

Containers’ history

Namespaces and cgroups● Namespaces = kernel-enforced view on different objects

from user space:– File system (mnt userspace)– Running processes (pid namespace)– Network stack (net namespace)– Users (user namespace)– Interprocess communication (ipc namespace)– Hostname configuration (uts namespace)

Namespaces and cgroups● cgroups = enforces limitation/s on a/group of

processe/s:– Memory– CPU– Network bandwidth– Disks I/O

Docker ?● A complete container management toolbox● Enables you to

– Build container images (with open specifications)– Execute/Run images as containers– Manage images and containers– Share and publish images

Docker images● Container Images are:

– A standardized unit of software, packaging● Code● Dependencies

– Packaging units/Images are● Consistent● Portable

“Package once, run everywhere”

Docker Containers● Docker Containers are:

– An instantiation of an image– Process(es) running in an isolated environment

● Docker containers should be:– Ephemeral– Stateless

“Dies like Frankenstein, resurrects as a baby”

Docker components

Containers vs Virtual Machines

Building a beautilful app

Why so hipster?

“Mature” technology with a rich ecosystem, changing the game on:

– Development and test automation– Integration and deployment– Software architecture– Infrastructure architecture

Part 2 – Containers Attack surface

Docker components

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Usage of untrusted container images (e.g.: CVE-2019-5736)– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses, (D)DoS minion– Application vulnerability– Image vulnerability– Usage of untrusted images

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image vulnerability– Usage of untrusted images

Usage of dangerous switches:

● --privileged

● --cap-add SYS_ADMIN

● --pid=host, --network=host,…

● --user root

Usage of dangerous volumes mappings● -v /:/tmp/a

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image vulnerability– Usage of untrusted images

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images

CVE-2019-5736

Source: https://github.com/Frichetten/CVE-2019-5736-PoC

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images

Source: https://blog.docker.com

Threats and Vulnerabilities

Source: https://blog.docker.com

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images

Source: https://cvedetails.com

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images

Containers with unlimited resources:

● CPU

● Memory

● Network Bandwidth

could abuse these resources and bring

the underlying system on its knees

Source: https://blog.docker.com

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images

Containers do not magically

● Make your application flawless

● Remove Programming errors

● Patch dependencies and libraries

● ...

Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS

– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment

● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images

Containers do not magically

● Protect yourself from avoiding running

Untrusted components

Part 3 – Controls and mitigation

(s)SDLC

Build phase● During build

– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source

● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

Security of images should be assessed to

● Identify unsafe settings in images manifests

● Identify vulnerable binaries/libraries

● Perform compliance checks on images

Possible solutions:

● CoreOS Clair

● Anchore

● Aqua Microscanner

● OpenScap

Build phase

Image source: https://anchore.com

Build phase● During build

– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source

● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

How to ensure only “compliant” images are used

● Images manifest must be version controlled

● Use a private/dedicate image registry

● Security tests could make the build fail

● Only a successful build can push to the registry

● Users can only pull from registry

And now, what about tomorrow for:

● Running containers

● Already validated images

Build phase

Image source: https://www.anchore.com

Reducing the attack surface● During build

– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source

● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

Build you images using minimal parent images

● scratch

● distroless-based

● Alpine based

Reducing attack surface● During build

– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source

● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

Issue with minimal images (non-distro based)

● Debug sessions are tricky

● More difficult to maintain

● Scanners could fail to identify vulnerabilities

Reducing attack surface● During build

– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source

● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

How can you trust the images you are downloading?

● Content should be audited/verified

● Image builds should be traceable

● Image should be rebuild-able

● Only pull signed images: Docker Content Trust

Reducing attack surface● At runtime

– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source

● Up to date/patched Operating System

● Up to date/patch container runtime

● Do not run application/containers as root

● Network isolation: iptables, NetworkPolicies,…

● Log all the things!

● Beef it up with container ‘IDS/ADS’ (eg. sysdig/falco)

Reducing attack surface● At runtime

– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source

● Disable intercontainers and host communication:

● dockerd --icc=false

● Re-map UID of root and other local users

● /etc/{subuid, subgid}

● Ensure docker API is not exposed and control docker

group membership

● Giving access to dockerd = giving root access

Reducing attack surface● At runtime

– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source

● Linux Capabilities = broken down root authorizations

● Docker by default disables a lot of capabilities

● Don’t add capabilities to your container

● Seccomp = syscall filter

● Firewall between user land and kernel

● Avoid at all costs --security-opt seccomp=unconfined

● 25 Kernel CVE prevented thanks to these layers

Source: https://docs.docker.com/engine/security/non-events/

Reducing attack surface● At runtime

– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source

Use kernel hardening modules

● SELinux

● AppArmor

● GRSec

Reducing attack surface● At runtime

– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits

● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source

Use cgroups limitations on your containers to:

● Avoid resources DoS on host

● Reduce impact by pausing the container

Take-aways (TLDL;)● Containers are:

– already in the vein of your infrastructure– Impacting software architectures and your development, build, integration and deployments

● Smooth ci/cd pipelines integrated with SSDLC– Security involved at build time: secure that deployment package– Security monitoring/detection at runtime

● Best practices also apply in container world:– System Container/Hardening– Vulnerability scanning– System and application logging– But you should/could go beyond this

Beyond docker...● Container orchestrators:

– Docker Swarm– Kubernetes

● Gvisor: Moving ring0 in ring3● Kata containers: s/docker/qemu● Cloud-Native?

Questions ?

Back-up slides

References● CVE-2019-5736:

https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html ● Dirtyc0w, container escape via vdso shared library:

https://github.com/scumjr/dirtycow-vdso ● Sysdig/Falco: https://github.com/falcosecurity/falco ● Sysdig/Images scanning tools:

https://sysdig.com/blog/container-security-docker-image-scanning/ ● Anchore: https://anchore.com● Clair:● Distroless: https://github.com/GoogleContainerTools/distroless

CVE-2019-5736

Source: https://www.twistlock.com/labs-blog/breaking-docker-via-runc-explaining-cve-2019-5736/