www.isaca.org Slide 1 of 41

Server Virtualization Assessment Server Virtualization Assessment – Tools and Techniques– Tools and Techniques

Chicago ISACA Chapter 8/11/2011Chicago ISACA Chapter 8/11/2011

Michael Hoesing CISA,Michael Hoesing CISA,

CISSP, CCP, ACDA, CIA, CFSA, CMA, CPACISSP, CCP, ACDA, CIA, CFSA, CMA, CPA

mhoesing@mail.unomaha.eduAnything discussed herein should be tested thoroughly in a lab environment before use in production. Opinions are those of the author and not conference sponsors, employers, clients, past, present or future. Don’t sue me; I have no money.

Slide 2 of 41

Server Virtualization Assessment -Server Virtualization Assessment -ObjectivesObjectives

• Virtualization Definitions, Background, Scope• Risks and Controls • Assessment Approaches and Tools:

• Assessment Examples– VM (Guest) Sprawl– ESX Console Operating System

(COS)Configuration

• Notes for vSphere 5

Slide 3 of 41

Background, Scope

Slide 4 of 41

BACKGROUNDBACKGROUND• 2004 ++ Virtualization Spreads• 2007 Gartner declares virtualization security

important• 2007 to Today risk and security/control

techniques and products related to virtualization evolve

• Now , and before now, we should evaluate how effective are those security techniques and controls (assessment)

• Business can’t live without speed to deployment

Slide 5 of 41

SCOPESCOPE

• Virtualization Scope – ESX servers hosting guests

• Not Included – (only so much can be done in 1.5 hours) VDI, Hyper-V, Xen (Citrix & other variants)

• Some risk topics reach beyond ESX (policy, process, procedure) if you are going to secure an ESX environment you must think beyond the COS

• Some topics should be in scope but their complexity is best covered separately (storage, backups)

Slide 6 of 41

Risks & Controls

Slide 7 of 41

RISKS & CONTROLS – a list of 10 RISKS & CONTROLS – a list of 10

1. VM/Guest Sprawl

2. Host Mis-Configuration

3. Network Segmentation

4. Remote Access

• Policies, Procedures, Inventory Practices, Reporting, Assessment

• Standards, Monitoring, Assessment

• Deploy Segregated Management, Production and IP Storage Networks

• SSH , SSL, access & account controls

Slide 8 of 41

RISKS & CONTROLS – a list of 10 RISKS & CONTROLS – a list of 10 (cont) (cont)

5. User Account Access & Roles

6. Single Point of Failure

7. Integration

8. Staff Skills9. Architecture (Blue Pill)10. Software Licensing11. I lied # 11 Appliances12. #12 Guest Escape

VMSA-2009-0006

• Policies, Procedures, Least Privilege

• Backups, Continuity Planning

• Strategic Architecture, Capacity Planning

• Training• Physical Security• Policy, Monitoring• QA, Certification

Processes, Vendor Mgmt• Patch Process

Slide 9 of 41

Assessment Approaches and Tools

Slide 10 of 41

ASSESSMENT APPROACHASSESSMENT APPROACH

• The Approach - 1.) a standard 2.) gather metrics 3.)compare metrics to the standard and cite variances

• Standard – • a.) yours, if you have created a document, congratulations• b.) VMware Hardening Guide(s)

http://www.vmware.com/files/pdf/vi35_security_hardening_wp.pdf http://communities.vmware.com/docs/DOC-15413

• c.) CIS – ESX 3.0, 3.5, 4.x Xen http://cisecurity.org/benchmarks.html (also has an XCCDF assessment tool (CIS-CAT) for members for 3.5 and 4.x)

• d.) DISA STIG http://iase.disa.mil/stigs/stig/esx_server_stig_v1r1_final.pdf

Slide 11 of 41

ASSESSMENT APPROACH (cont)ASSESSMENT APPROACH (cont)

• More Standards – • e.) NIST 800-125 Jan 2011

http://csrc.nist.gov/publications/nistpubs/800-125/SP800-125-final.pdf

• f.) PCI/DSS June 2011 https://www.pcisecuritystandards.org/documents/Virtualization_InfoSupp_v2.pdf

• g.) NSA http://www.nsa.gov/ia/_files/support/I733-009R-2008.pdf

• h.) Vendors (HyTrust), consultants, books (Ed Haletky, Scott Lowe, Siebert…)

Slide 12 of 41

ASSESSMENT APPROACH – Audit ASSESSMENT APPROACH – Audit ProgramsPrograms

• ISACA – Whitepaper – issued Oct 2010 risks, audit approacheshttp://www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/Virtualization-Benefits-and-Challenges.aspx Audit program issued Jan 2011, GRC levelhttp://www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/VMware-Server-Virtualization-Audit-Assurance-Program.aspx

• SANS talk through http://www.sans.org/reading_room/analysts_program/VMware_ITAudit_Sep09.pdf

• Mine (come to the hands-on class), mix of process/procedure and detailed metrics

Slide 13 of 41

GATHERING METRICS – SOME GATHERING METRICS – SOME THOUGHTSTHOUGHTS

• In 50 mins all I can do is name-drop, you do the research in your environment/strategy/risk appetite

• Not a Bake-off, not a Best-of, I can only relate what worked in my lab (see bullet one above), any product not mentioned just means I have not installed it yet

• Good News, lots of products to chose from , list grows almost daily (bad news, that expands due diligence time)

• Some tools Work in a Virtual Appliance, some tools have both a physical and virtual appliance

• Key – does ingress and egress to/from the Guest allow the product to do its Job (patching, AV, config assessment)

Slide 14 of 41

GATHERING METRICS – SOME GATHERING METRICS – SOME THOUGHTS (cont)THOUGHTS (cont)

• Free Tools – great price, don’t scale well• Some tools inventory the Virtual Center

database, some tools enumerate raw data• No one tool does everything, run multiple tools

for corroboration and completeness• Tools that use a RHEL baseline, take care in

reviewing, but maybe 80-90% correct• In a lab, build an ESX server (and vCenter) with

the vendor defaults, and build a second ESX server with your organization’s standard build, for education purposes and to calibrate tools

Slide 15 of 41

METRIC GATHERING TOOLSMETRIC GATHERING TOOLS

• Interviewing and Document Review for policies, standards, procedures, training

• !Free! Tools – • console CLI, and vSphere remote CLI• CIS-CAT 2.2.7 June/30/2011 (for members)

ESX 3.5 and 4.x benchmark XCCDF test scripts

• VIToolkit & Powershell, (now called vSphere PowerCLI 4.1 U1)

• esxcfg-xxx commands various (i.e. esxcfg-firewall –q)

• esxcfg-info – dump of everything, load into ACL and search

Slide 16 of 41

METRIC GATHERING TOOLS (cont)METRIC GATHERING TOOLS (cont)

• More Free Tools:• vmware-vim-cmd hostsvc/ = grep /net/info or

grep /storage/info (careful, many of these commands change settings, stick with the ones with the word ‘info’)

• Configuresoft (Ionix) ComplianceChecker, Tripwire configcheck, (ESX 3)

• From VMware - VI API, VIX API (allows files xfer from guest) , Perl API, CIM API (risks of rolling your own = script storage security, stored passwords, change management, version management)

Slide 17 of 41

METRIC GATHERING TOOLS (cont)METRIC GATHERING TOOLS (cont)

• More Free Tools:• Bastille – remember to run in the –assess mode,

not the harden mode (3.0.9-1.0)• DISA – SRR (security readiness review evaluation

script) watch these, they may harden if not run correctly

• LSAT – works on 3.5 and before, but the MD5 process will try to analyze the very large vmdk disk files, this is time consuming and could crash running guests (note : does not work in vSphere, C compiler is removed)

Slide 18 of 41

METRIC GATHERING TOOLS (cont)METRIC GATHERING TOOLS (cont)

Slide 19 of 41

Assessment Examples

VM (Guest) Sprawl

Slide 20 of 41

SPRAWL - CLI SPRAWL - CLI

• Free Tools – Command Line Interface (CLI) – / / /* | ls lR vmfs volumes grep vmx

• Or the ‘find’ command (does not follow sym links)

• -rwxrwxrwx 1 root root 4831838208 Jul 7 2007 BLVS-flat.vmdk• -rwxrwxrwx 1 root root 331 Jul 7 2007 BLVS.vmdk• -rwxrwxrwx 1 root root 8589934592 Jul 7 2007 BLVSMgr-flat.vmdk• -rwxrwxrwx 1 root root 336 Jul 7 2007 BLVSMgr.vmdk• -rw------- 1 root root 872415232 Sep 23 10:10 Reflex-VSA-Template-flat.vmdk• -rw------- 1 root root 480 Sep 23 10:10 Reflex-VSA-Template.vmdk• -rw------- 1 root root 4294967296 Oct 8 11:37 Reflex-vsc-flat.vmdk• -rw------- 1 root root 499 Oct 8 00:50 Reflex-vsc.vmdk• -rw------- 1 root root 6442450944 Sep 29 01:59 RHEL-4-4-ES-flat.vmdk• -rw------- 1 root root 339 Sep 29 01:57 RHEL-4-4-ES.vmdk• -rw------- 1 root root 16791552 Mar 17 2008 SLES10-SP1-000001-delta.vmdk

Slide 21 of 41

SPRAWL – CIS-CAT SPRAWL – CIS-CAT

Free Tools CIS-CAT (if a member) will list VM’s with non-compliant vmx config files (not a complete inventory but a good start on what needs correction)

Slide 22 of 41

SPRAWL – PowerCLI 4.0 SPRAWL – PowerCLI 4.0

VI Tools for Windows & Powershell now named vSphere PowerCLI 4.1 (partial script)

• $VC = Connect-VIServer 192.168.1.21 -User XXXXXX -Password XXXXXX

• $VMs = Get-VM | format-table -property name • $Datastores = Get-Datastore | Format-Table -property Name• $VMXlist = " "• $i = 1; while ($i -le $Datastores.length-4)• $Datastore = Read-Host "Enter Data Store Name, like storage1*

from the list above "• get-childitem -recurse -include *.vmx | format-table -property

name >> c:\vmxlist• $i +=1• Then compare the two files (VM list and vmx list) with diff, ACL, or

manually

Slide 23 of 41

SPRAWL – vCenter SPRAWL – vCenter

• Existing Management Tools - Virtual Center

Slide 24 of 41

SPRAWL – Reflex SPRAWL – Reflex

• Third Party Security Tools – Reflex

Slide 25 of 41

SPRAWL – Configuresoft (IONIX) SPRAWL – Configuresoft (IONIX)

• Commercial Configuration Assessment Tools – Configuresoft (Ionix)

Slide 26 of 41

SPRAWL – EcoraSPRAWL – Ecora

• Commercial Assessment Tools – Ecora

Slide 27 of 41

SPRAWL – Honorable MentionSPRAWL – Honorable Mention

• Anything that Monitors Usually has an Inventory Component• Akorri Balance Point BMC Performance Manager

• CA ASM (Unicenter) eG Innovations Enterprise Suite

• Embotics V-Commander HP Operations Orchestration

• IBM Tivoli Monitoring for Virtual Servers

• ManageIQ EVM Suite Netuitive SI for VMware

• Quest vFoglight Symantec Altiris

• Tideway Foundation Veeam Monitor

• SPI for VMware vmInformer

Slide 28 of 41

Assessment Examples

Host Configuration

Slide 29 of 41

HOST CONFIGURATION – CIS-CAT HOST CONFIGURATION – CIS-CAT CategoriesCategories

• CIS-CAT 9 Categories (3.5)

Slide 30 of 41

HOST CONFIGURATION – CIS-CAT HOST CONFIGURATION – CIS-CAT Benchmark ItemsBenchmark Items

• CIS-CAT 29 Benchmark Items (3.5)

Slide 31 of 41

HOST CONFIGURATION – CIS-CAT Detail HOST CONFIGURATION – CIS-CAT Detail Assessment Test and ResultsAssessment Test and Results

• 1.2.3 Recommended Boot services (3.5)

Slide 32 of 41

HOST CONFIGURATION – CIS-CAT HOST CONFIGURATION – CIS-CAT CategoriesCategories

• CIS-CAT 12 Categories (4.1)

Slide 33 of 41

HOST CONFIGURATION – CIS-CAT HOST CONFIGURATION – CIS-CAT Benchmark ItemsBenchmark Items

• CIS-CAT Benchmark 65 Items (4.x) (partial)

Slide 34 of 41

HOST CONFIGURATION – CIS-CAT Detail HOST CONFIGURATION – CIS-CAT Detail Assessment Test and ResultsAssessment Test and Results

• 9.1 Recommended Boot services (4.x)

Slide 35 of 41

HOST CONFIGURATION – TripwireHOST CONFIGURATION – Tripwire

• Commercial Assessment Tools – Tripwire

Slide 36 of 41

vSphere 5

Slide 37 of 41

vSphere 5vSphere 5

• Released July 2011• Memory based pricing is new, and not popular• ESX COS is gone, ESXi the only choice• ESXi has hypervisor and console all on the same

partition, faster (vendor says)• ESXi 5 has a firewall (iptables) ESXi 1-4 did not• No (if configured as suggested) console access,

all access is remote • Use vMA, remote CLI, and PowerCLI for audit

metric gathering or vCenter

Slide 38 of 41

vSphere 5 (cont)vSphere 5 (cont)

• TPM (Trusted Processing Module) recognition available (Intel’s TXT or AMD’s SEM , soon)

• Hope they Fixed These in 5 (ESXi 4.1 issues) Logs removed upon reboot root password not set

during installation Tech Support Mode (from console) Remote Tech Support Mode (SSH), accesses Single

User Mode (root without any password if not set at default, even with password root SSH is enabled)

Reset System Configuration – resets an empty root password (watch iLO and iDRAC)

Slide 39 of 41

Conclusion

Slide 40 of 41

SUMMARYSUMMARY

• Virtualized Infrastructure is Important to the Organization and worthy of secure configuration and periodic assessment of that state

• Standards are available for a starting point to create/edit your organization's policy

• Tools are available, in all price ranges, to gather metrics from an ESX environment

• Get the tools, gather the metrics, compare to the policy/standard, cite the differences, improve your security posture

Slide 41 of 41

Q and AQ and A

– If the question comes to you later mhoesing@mail.unomaha.edu

– ?– ?– ?– ?– ?– ?