Orchestration Ownage - RSAC 2017

SESSIONID:SESSIONID:

#RSAC

BryceKunz

OrchestrationOwnage:ExploitingContainer-CentricDatacenterPlatforms

CSV-R03

SeniorThreatSpecialistAdobe

MikeMellorDirector,InformationSecurityAdobe

Presenter’s Company Logo – replace or

delete on master slide

#RSAC

Intro

2

MikeMellor

Director,InformationSecurity@Adobe

BryceKunz

SeniorThreatSpecialist@Adobe



#RSAC

Containers- TheFutureisNow!

3

2016Surveys:

15-16%ofallorganizationsarealreadyusingcontainersinproduction

35%organizationshavedoneaproof-of-concept

TheFutureisNow!

Containersareinproductionnow

Containersarecontinuingtogrowinpopularity



#RSAC

Containersappearmoresecure

4

Thebiggestdrivers:

39%toincreasedeveloperefficiencyand

36%tosupportmicroservices

Organizationswanttoavoidcloudplatformlock-in

2016Surveys:

Many(42%)organizationsgainvalueinthe“secure/isolated”capabilitiesthatcontainersprovide



#RSAC

ButmanagingContainersfeelscomplex

5

2016Survey:

Themoreexposureanorganizationhastocontainers,

Themorecomplexitiesareexposed.

Respondentssaidtheyfoundcontainers…

“toocomplextointegrateintoexistingenvironments,”

andrequire“toomanyskilledresources tomanage.”



#RSAC

Andareverychallengingtomanageatscale

6

2016Survey:

The#1challengeofcontainers,accordingtothe53%ofrespondentswhoareeitherusingorevaluatingcontainers,is…

“ContainerManagement.”



#RSAC

ProbableSecurityNightmare

7

“TooComplex”+ChallengingtoManage

=

ProbableSecurityNightmare

“ComplexitytheWorstEnemyofSecurity”- BruceSchneier



#RSAC

ContainerandClusterManagementOptions

8

Technology Design Pros Cons

PublicCloudContainerServices

Container Centric Easy, Scalable Vendor Lock-in;Proprietary

DockerSwarm Docker Centric NativeClustering LimitedbyAPI

Kubernetes ClustersofContainerized Apps

Worksw/Docker;Mounts persistentvolumes

Custom overlayrequiresmorespecialization

Mesos &DC/OS ClusterManagement

Works w/Docker,Kubernetes, &NativeApps;VeryFlexible

Additionallayersaddsmorecomplexities



#RSAC

ServerServer Server

ClusterManagement

9

CoreOSLinuxOS

Manyserversin…DataCenterAWSAzureetc…

Howdoweeffectivelyusealloftheseresources?Datacenter,Azure,AWS,

GCE,etc…



#RSAC

ServerServer Server

Mesos Master&Agents

10

Mesos Master5050/TCPbydefaultDistributesTasks

Mesos Agent5051/TCPbydefaultExecutesTasks

CoreOSLinuxOS

Datacenter,Azure,AWS,GCE,etc…

Master

AgentAgent



#RSAC

ServerServer Server

Mesos istheKernelofDC/OS

11

Mesos isthekernelofthedistributedoperatingsystemknownasDC/OS


Kernel:Master

AgentAgent



#RSAC

ServerServer Server

Frameworks

12

Frameworks

providethelogic

Init Jobs— Marathon

Cron JobsChronosMetronomeDatacenter,Azure,AWS,

GCE,etc…

Kernel:

Frameworks:

Master

AgentAgent



#RSAC

ServerServer Server

Supporting:ConfigurationStores

13

ConfigurationStores

keepeveryoneonthesamepage

ZooKeeper

Etcd


Kernel:

Frameworks:

Supporting:

Master

AgentAgent



#RSAC

ServerServer Server

Supporting:Discovery

14

Discovery

Enablesthefindingofotherserviceswithinthecluster

Mesos DNS


Kernel:

Frameworks:

Supporting:

Master

AgentAgent



#RSAC

ServerServer Server

DC/OSDesign

15

Containersw/Apps

DockerContainers

WebAppsetc…


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp



#RSAC

ServerServer Server

InternetAccessibleContainers

16

Containersw/Apps

PublicInternetAccessible

PrivateInternal


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet



#RSAC

ServerServer Server

Scenario

17

InitialAccess(RCE)

Viaavulnerablewebapplication

Intoacontainer

Aslimiteduser(e.g.www-data)


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

Scenario:RCEviawebappwithinacontainer

18

e.g.JBoss,Tomcat,OSGi Console,Axis2,etc…



#RSAC

ServerServer Server

ReconviaMesos DNS

19

Queryviapivot:

Mesos DNS53/UDP&TCP— DNSservice


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

.mesos TLD

20

Theeasywaytofindserviceswithinthecluster



#RSAC

ServerServer Server

ReconviaMesos DNS

21

Queryviapivot:

Mesos DNS

8123/TCPbydefault— DNSviaRESTAPI

ServiceDiscover— withintheCluster


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

Undocumented?

22

/v1/enumerate->allmesos dns information



#RSAC

EnumerateMesos DNSusingRESTAPI

23




#RSAC

FindIP&RHPTCPportsofallservices

24




#RSAC

ServerServer Server

Secure:DisableRiskyMesos DNSFeatures

25

Disablethe…

“AXFR”

“Enumerate”

APICalls

• Harderforattackertodiscoverallservices

• Applicationsshouldn’tcommonlybeusingtheseAPIcallsDatacenter,Azure,AWS,

GCE,etc…

Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

ServerServer Server

ReconviaMesos Master

26

Queryviapivot:

Mesos Master5050/TCPbydefaultDistributesTasks


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

EnumerateMesos Master

27

RequestviatheRESTAPI



#RSAC

EnumerateMesos Master

28

Response:json w/allMesos Agent’sIPaddresseswithinthecluster



#RSAC

ServerServer Server

ReconviaMesos DNS

29

Queryviapivot:

Mesos Agent5051/TCPbydefaultExecutesTasks


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

EnumerateMesos Agent

30




#RSAC

EnumerateMesos Agent

31

Response:json w/whatcontainersarecurrentlyrunningontheserver(i.e.basic0012)



#RSAC

ServerServer Server

Secure:LogicalInternalNetworkSegmentation

32

Separatesoutthenetworkintozones:

Appsw/Data

Management

CommonlywithCalico,Canal,orFlannelDatacenter,Azure,AWS,

GCE,etc…

Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp



#RSAC

ServerServer Server

SecretsviaConfigurationStore

33

EtcdRHP/TCPbydefault— 2379/TCP

client/server— 2380/TCPpeersConfigurationStore— CoreOSFleetsUnits— Applications

ZooKeeper2181/TCPbydefault— BinaryProtocol


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

EnumerateEtcd

34

RequestviatheRESTAPIrecursively



#RSAC

EnumerateEtcd

35

Response:json frequentlycontainingsecretsincludingcredentials



#RSAC

ServerServer Server

Secure:SeparateConfigurationStores

36

Separateouttheconfigurationstoresintozones:

Appsw/Data

Management

Enforceseparationvia…

AuthenticationCredentialsand

LogicalNetworkSegmentationDatacenter,Azure,AWS,

GCE,etc…

Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

ServerServer Server

Frameworks

37

MarathonLongRunningServices— e.g.Containers

Ensuresalwaysrunning

ChronosCron fortheClusterBatchJobs


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

RCEviaMarathonJobs

38




#RSAC

ServerServer Server

RCEviaMarathonJobs

39





Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE

RCE



#RSAC

RCEviaMarathonJobs

40

Response:json withthemaliciousjobstatus



#RSAC

ServerServer Server

RCEviaChronos Jobs

41





Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE

RCE



#RSAC

ServerServer Server

Secure:EnforceAuthentication

42

Applicationsmust…

supportandbeconfiguredtouseauthenticationaswell

securelystoreandusecredentials

bedeployedsecurelyand/orretrievecredentialssecurely

Alertonbruteforceattempts


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

ServerServer Server

CredsviaMitM withARPSpoofing

43

AnotherContainerhastheCredsforMarathon


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE



#RSAC

ServerServer Server


44

AttackerusesARPspoofingtoredirectthatcontainerstraffictothecompromisedcontainer

Attackercollectthecredentials


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE ARP



#RSAC

ServerServer Server


45

AttackercannowcreatemaliciousMarathonjobsNegatingauthenticationsecuritycontrols


Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE ARP

RCE



#RSAC

ServerServer Server

Secure:TLSforInternalCommunications

46

EnableTLSw/validcertificatesforstrongHTTPScommunications

AnythingusingcredentialsneedsTLS!

ValidateCertificates

Failclosedonbadcertificates

AlertoncertificateserrorsDatacenter,Azure,AWS,

GCE,etc…

Kernel:

Frameworks:

Apps:

Supporting:

Master

AgentAgent

CntrApp

CntrApp

CntrApp

CntrApp

Internet

RCE ARP



#RSAC

StrategicActions

47

Nextweek:AssesswhichservicesyoucanenableAuthentication&TLSon— w/obreakingyourexistingapplicationswithinthecluster

Threemonthsfromnow:ImplementAuthentication&TLSonsafeservicesandframeworks— Focusingonservicesresponsiblefororchestrationwithinthecluster

DeployseparateserviceswherepossibleforAppsthatdonotsupportTLS&Auth

Sixmonthsfromnow:RetrofitallApplicationswithintheclustertouseTLS&AuthenticationEnforcetheuseofTLS&Authenticationinternaleverywhere(disableclear-text)



#RSAC

BigPicture

48

ContainerAdoptionIsMaturing,especiallyinEnterprises

Enterprisesareusingcontainersinproduction.



#RSAC

BigPicture

49

Pivoting fromacompromisedservicewithintheclusterNocontainerbreakout/0day/exploitneededJ

Mayenableanattackertocompletelycompromisethecluster



#RSAC

BigPicture

50

LookingBeyondtheBorder

withaDefenseinDepthstrategy

SecurestheFuture&thecluster



#RSAC

Thankyou!

51

Thankyou!



#RSAC

FutureResearch

52

TestingMitM fromcompromisedcontainerNCCGroup’sreportstatesthisispossibleforco-hostedcontainers

TestdowngradeHTTPScommunicationsCanwedowngradefromHTTPStoHTTPandcapturecredsfromanothercontainer?

TestCerts(e.g.cancertpinningbeenabled?)toRESTAPIsCanweMitM andimpersonatetheAPIservice?

TestAuthenticationBruteforceattacksFairlycertaintherearenolockouts,canweenablebetterauthenticationsecurity?— Writemoduletobrute-forceandguesscreds

TestLogicalNetworkSegmentationToolsCalico,Canal,Flannel— Note:theseshouldworkasadvertisedbutprobablyweshouldindependentlyverify



#RSAC

References

53

https://www.cloudfoundry.org/wp-content/uploads/2016/06/Cloud-Foundry-2016-Container-Report.pdf

https://clusterhq.com/assets/pdfs/state-of-container-usage-june-2016.pdf

http://www.rightscale.com/blog/cloud-industry-insights/new-devops-trends-2016-state-cloud-survey