@MimmingCodes -- mimming.com
Kubernetes Runtime Security
@MimmingCodes -- mimming.com
Kubernetes Runtime Security
Jen TongSecurity AdvocateGoogle Cloud Platform
@MimmingCodesmimming.com
About me
https://twitter.com/MimmingCodeshttps://mimming.com
How many of you● use Kubernetes in production?● use containers?● are security engineers?● gotten a shell on a system?● have ever discovered a long ago
compromised system?
@MimmingCodes -- mimming.com
Agenda
Security overview
Containers & Kubernetes
Impact on security
Demo of a sad day
Fix low hanging fruit
Discuss higher up fruit
Security overviewoffense vs defense
@MimmingCodes -- mimming.com
Offensive Security
Goal
@MimmingCodes -- mimming.com
Offensive Security
GoalInt
egrit
y
@MimmingCodes -- mimming.com
Offensive Security
GoalInt
egrit
y Availability
@MimmingCodes -- mimming.com
Offensive Security
GoalInt
egrit
y Availability
Confidentiality
@MimmingCodes -- mimming.com
Offensive Security
GoalDefensive measure Intermediate resources
@MimmingCodes -- mimming.com
Offensive Security
Goal
@MimmingCodes -- mimming.com
Offensive Security
Goal
@MimmingCodes -- mimming.com
Offensive Security
Goal
Kill chain
@MimmingCodes -- mimming.com
It feels like development on a terrible API
@MimmingCodes -- mimming.com
Defensive security
Goal
@MimmingCodes -- mimming.com
Defensive security
Goal
@MimmingCodes -- mimming.com
Defensive security
Goal
@MimmingCodes -- mimming.com
Defensive security
Goal
Lessons
@MimmingCodes -- mimming.com
Defensive security
Goal
Lessons
Containers & Kubernetes… or as much as I can cover in 5 min
@MimmingCodes -- mimming.com
The promises of virtualization, but it
actually works
@MimmingCodes -- mimming.com
Virtualization
Hardware
Host OS
Guest OS
Libraries
Application
Guest OS
Libraries
Application
@MimmingCodes -- mimming.com
Containers
Hardware
Host OS
Libraries
Application
Libraries
Application
@MimmingCodes -- mimming.com
Lots of containers
@MimmingCodes -- mimming.com
@MimmingCodes -- mimming.com
Nodes
@MimmingCodes -- mimming.com
Pods
@MimmingCodes -- mimming.com
Pods
Pod
@MimmingCodes -- mimming.com
Management infrastructure
etcd
scheduler
controllers
apiserver
Master
@MimmingCodes -- mimming.com
Management infrastructure
etcd
scheduler
controllers
apiserver
Master kubelet
@MimmingCodes -- mimming.com
UI
CLI
API
All together
Users
etcd
scheduler
controllers
apiserver
Master Nodes
Impact on securityContainerization changes some stuff
@MimmingCodes -- mimming.com
Dynamic
@MimmingCodes -- mimming.com
Dynamic
@MimmingCodes -- mimming.com
Dynamic
@MimmingCodes -- mimming.com
Dynamic
@MimmingCodes -- mimming.com
Some things are harder for both sides
● Offense○ Kill chains have less time to execute○ More layers to break out of
● Defence○ Old tricks don't work as well○ More complexity -- bigger attack surface
@MimmingCodes -- mimming.com
DevelopmentDeployment
Runtime
@MimmingCodes -- mimming.com
During development
Tools for securely building containerized services
● Identity, RBAC (role based access control)● Secure inter-service communication● Secret access control & rotation
@MimmingCodes -- mimming.com
During deployment
Secure supply chain to prevent threats from entering
● Detect known vulnerable in dependencies● Add metadata to images● Verify the build pipeline
@MimmingCodes -- mimming.com
During runtime
Detect and respond to threats in running containers
● Proper configuration● Security context● Security centric monitoring
DemoOf a really bad day :(
Low hanging fruit
@MimmingCodes -- mimming.com
Never do this
$ kubectl create -f https://foo.com/bar.yml
@MimmingCodes -- mimming.com
Disable the Kubernetes Dashboard
@MimmingCodes -- mimming.com
Restrict the GCP service account
● Currently has project editor permission● Only need a few narrow permissions
○ monitoring.viewer○ monitoring.metricWriter○ logging.logWriter
@MimmingCodes -- mimming.com
Network policies
So an attacker can't hop between pods
Great list of examples:github.com/ahmetb/kubernetes-network-policy-recipes
https://github.com/ahmetb/kubernetes-network-policy-recipes
Demo 2.0
Higher up fruitIf you have more time
@MimmingCodes -- mimming.com
Security context
Further restrict permissions with
● AppArmor● SELinux● Seccomp
kubernetes.io/docs/tasks/configure-pod-container/security-context/
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
@MimmingCodes -- mimming.com
Security monitoring
● Hook into your cluster● Log a bunch of stuff● More policies
○ alerts○ automatic remediation○ forensics
● Mostly commercial products… for now
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Pod
Container
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Pod
Management Container
Container
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Privileged Container
Pod
Management Container
Container
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Pod
Management Container
Container
Kernel module
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Pod
Management Container
Container
● Network events● System calls
@MimmingCodes -- mimming.com
Deployment models
Node
Pod
Container
Kernel
User space
Pod
Management Container
Container
Ring buffer
Local database
Persistent disk
Hosted database
@MimmingCodes -- mimming.com
Open source options
● Sysdig ○ sysdig○ Inspect○ Falco
● Cilium● Capsule8
https://github.com/draioshttps://github.com/cilium/ciliumhttps://github.com/capsule8
@MimmingCodes -- mimming.com
What we discussed
Security overview
Containers & Kubernetes
Impact on security
Low hanging fruit
Higher up fruit
@MimmingCodes -- mimming.com
Thank you!
@MimmingCodes -- mimming.com