BROADCOM PROPRIETARY & CONFIDENTIAL1
Data Center Infrastructure Management(DCIM)
(CFRT Peer Review Workshop) 11/20/08
BROADCOM PROPRIETARY & CONFIDENTIAL2
Schedule– 9:30am to 10:00am Registration and CFRT Networking– 10:00am to 10:30am CFRT Membership Business - Membership Status
and Planning– 10:30am to 11:00am Broadcom’s Data Center Journey
(speaker: Dave Dalesandro)– 11:00am to 11:30am Broadcom’s Review of Data Center Management Software (speaker: Dave Holub)– 11:30am to 12:15pm Interactive Demo - Broadcom’s DCIM application
(presenter: Dave Eastman)– 12:15pm to 1:30pm Lunch and Round Table Interaction Working
Lunch (served in meeting room) – 1:35pm to 2:10pm Panel Q&A (Sorena, Holub, Eastman, Dalesandro, Vippin)– 2:15pm to 2:45pm CFRT Wrap-up Networking and Informal Q&A– 2:50 pm to 3:15pm Broadcom Data Center tour (for those interested)
BROADCOM PROPRIETARY & CONFIDENTIAL3
Presenters• Dave Dalesandro
– Director, IT X-Functional Architecture – Data Center Design and Management– ([email protected])
• Dave Holub – Principal Project Manager, - IT – Data Center infrastructure design architect– ([email protected])
• Dave Eastman – Sr. Principal Project Manager, - IT – DCIM Prototype Developer– ([email protected])
BROADCOM PROPRIETARY & CONFIDENTIAL4
Goal
Provide data and build processes to operationalize the management of our data
centers
BROADCOM PROPRIETARY & CONFIDENTIAL5
Why? Data Center Infrastructure Optimization
• Broadcom data centers have been self service from inception – This only works ‘if’ you have excess capacity – With excess capacity ‘Optimization’ is not as critical of an issue
• Data centers cannot be self service when you near capacity– Finding available infrastructure is ‘not’ a trivial task
• We need to get the most out of our “large $” investments in data centers
• We need to protect the data center from overload or unbalanced situations
BROADCOM PROPRIETARY & CONFIDENTIAL6
Compute Growth Over Timecompute power
0
50000
100000
150000
200000
250000
300000
2000 2001 2002 2003 2004 2005 2006 2007 2008
compute power
BROADCOM PROPRIETARY & CONFIDENTIAL7
Compute Power• Processor Per-core EDA Points• Xeon 2.8GHz 32-bit 6• Xeon 3.0GHz 32-bit 6.4• Opteron 2.2GHz 11• Opteron 2.4GHz 12• Opteron 2.6GHz 13• Opteron 2.8GHz 13.9• Intel 2.66GHz 17.4• Intel 3.16GHz 19.1
BROADCOM PROPRIETARY & CONFIDENTIAL8
Global Disk TrendsDoubling storage capacity every 2 years!!!
GB
0
500000
1000000
1500000
2000000
2500000
2006
0329
2006
0628
2006
0927
2006
1227
2007
0328
2007
0627
2007
0926
2007
1226
2008
0326
2008
0625
2008
0723
Site (All) StorageType (All)
Date
Data
BROADCOM PROPRIETARY & CONFIDENTIAL9
Data Center Infrastructure• Data Centers are valuable resources as they get close to capacity
those resources must be carefully managed• Infrastructure includes
– Racks– Switches and switch ports– VLANs– Patch panels and cables (of all types)– Power utilization and monitoring– Generators– High voltage power components – HVAC components
• By accurately tracking the usage of systems and their placement in the data center we can ensure that overload conditions do not occur
BMS
BROADCOM PROPRIETARY & CONFIDENTIAL10
How did we get here?• Company growth and technology changes means growth in storage
and compute• Evaluated several commercial applications• We did not know what we needed until we took the journey• We had rules (3840W for example) but did not know the cascading
impact• We were always doing fire drills to meet the demands of the business• Lead times for data center expansion are long• Our largest data centers were designed 10+ years ago• Finding all the components to install just a single system in the data
center was becoming increasingly time consuming
BROADCOM PROPRIETARY & CONFIDENTIAL11
BROADCOM PROPRIETARY & CONFIDENTIAL12
Data Center Management
A Software Schlep
David HolubPrincipal Project
ManagerIS Infrastructure
11/20/08
BROADCOM PROPRIETARY & CONFIDENTIAL13
Software SmorgasbordVarious Buckets - Where have we looked, What have we Implemented?
Building Management SystemControls Integration – Outside the Scope of this Meeting/DiscussionMechanical Electrical Plant Monitoring and AlertsPropriety – Not Implemented Everywhere / Consistently Across FacilitiesMulti-Protocol – Modbus, SNMPCostly Integration and CustomizationVarious Scales – UPS’s from 50-500kVA modulesCritical Information for Management – Must Have DataCommercial Application VendorsLarge – Expensive/Comprehensive Packages“I Need A Bus Load of Consultants to get by”Other Extreme - Open Source – Best at Monitoring & AlertsOS Airlines.. Unix Airline - Each passenger brings a piece of the airplane and a box of tools to the airport. They gather on the tarmac…
Heat and Power modeling packagesInstrumentation – Tie dyeCFD AnalysisVisualization PackagesDrawing PicturesFlat Databases – hard to populate and keep current..Cable Plant ManagementTelecom-CentricProprietary Patch – Panel SystemsDesigners
BROADCOM PROPRIETARY & CONFIDENTIAL14
Customized Views• Data Center Floor – Not in all Places
BROADCOM PROPRIETARY & CONFIDENTIAL15
PDU Panel Example• BCM
– Yes/NoProbably notin all places
BROADCOM PROPRIETARY & CONFIDENTIAL16
Rack Power Over Time• 5 Months .5 Amps Variation @ 110 Volts (Lost in the Sauce)
BROADCOM PROPRIETARY & CONFIDENTIAL17
BRCM over Time• 5 Months - 1.8 Amps @ 110 Volts (Wrong Granularity…)
BROADCOM PROPRIETARY & CONFIDENTIAL18
Larger Electrical Scale• Static Transfer Switch
BROADCOM PROPRIETARY & CONFIDENTIAL19
6 Months in the life of a ASTS• Power Growth – Refresh Cycles – Great Data
BROADCOM PROPRIETARY & CONFIDENTIAL20
Open Source Monitoring & Alerts• Nice Graphs – Alerts, Thresholds, Notification lots of work…
BROADCOM PROPRIETARY & CONFIDENTIAL21
Poor Man’s CFDInstrument the Data CenterUnderstand Changew/o Blankingw/Blanking
BROADCOM PROPRIETARY & CONFIDENTIAL22
Uber Data Center Application• Expensive in $ and Resources
– To Buy– To Implement
• Simple Tasks Become:• Cumbersome• Require Great Expertise• Canonical Issues
– Training– Support– Waiting for Godot
BROADCOM PROPRIETARY & CONFIDENTIAL23
• Who Cares?
• Some People Do..
Cable Management Icky Problems…
BROADCOM PROPRIETARY & CONFIDENTIAL24
Cable Management Applications• Highly Generalized
– Not Data Center Specific– Carriers– Contractors– Vendors Specific
BROADCOM PROPRIETARY & CONFIDENTIAL25
Visualization Packages
BROADCOM PROPRIETARY & CONFIDENTIAL26
Theoretical Models
BROADCOM PROPRIETARY & CONFIDENTIAL27
Cool But I Need…• Need to Roll Up the Data - Pull Together Various Pieces on a Dashboard
– Live Within Your Constraints• Thresholds• Deployment Rules• Remediation – “Houston We Have a Problem” -Find All These
– Planning• Build Consensus – Team Understanding of Data Center Ecology• Capacity - Relationship to Redundancy and Resiliency
– Prediction• ‘What if..?’ Various scenarios for: Refresh / Deployment / Growth
– Hierarchical Relationships• Achieving Balance Across Infrastructure UPS’s, PDU’s, Floor Layouts/Cooling• Efficiently using resources and capital – Getting the most out of Data Centers
– Change management• Run a Project – We need to Deploy “X” (Space, Power, Cables Ports)• Day –to- Day tasks – Translation into Work Orders
BROADCOM PROPRIETARY & CONFIDENTIAL28
Data Center Infrastructure Management
The Application Story
Dave Eastman; Sr. Principal Project Manager’ IS Infrastructure; [email protected]
Due to the proprietary nature of the application and the fact that a decision to commercialize the product is pending, the application slides have been removed from this copy of the presentation. However, as discussed in the workshop if there is interest to learn more and possibly partner with Broadcom in the further development of the application contact should be made directly with Dave Eastman at Broadcom.
BROADCOM PROPRIETARY & CONFIDENTIAL29
Direct Benefits• Thresholds are understood and tracked• Efficient use of data center Infrastructure• Predictable consumption enables non-reactive procurement• Change Management process is key to success• Issues that we had no idea of before the application are now
identified issues can be rectified during upgrades or planned outages
• What if scenario provides alternatives (stretches the $DC)– Adjust thresholds to understand the impact– Refresh opportunities– Some redundancy can be reduced if the impact is understood
BROADCOM PROPRIETARY & CONFIDENTIAL30
Indirect Benefits• Unbalanced or over load conditions rectified before they become
issues• The journey has allowed us to get commitment from all involved• Next generation design improvements
– Cabling– Power distribution– Density– Efficiency– Instrumentation is critical and there are options – Modular/POD designs allow latest technology to be implemented by only
building what you need when you need it
• Rebates for refresh from Power companies
BROADCOM PROPRIETARY & CONFIDENTIAL31
What Next?• This is a Peer Review Workshop so . . .
• Tell us what you’re thinking . . . – Fill out the 1-page participant response survey
BROADCOM PROPRIETARY & CONFIDENTIAL32
LunchRound Table Discussion
BROADCOM PROPRIETARY & CONFIDENTIAL33
Panel Q & A• Dave Dalesandro
– Director, IT X-Functional Architecture
– Data Center Design and Management
• Dave Holub – Principal Project Manager, - IT – Data Center infrastructure design
architect– ([email protected])
• Dave Eastman – Sr. Principal Project Manager, - IT – DCIM Prototype Developer– ([email protected])
• Al Sorena – Manager, Critical Facilities – ([email protected])
• Vippin Chadha • Regional IT Manager, North
America• ([email protected])
BROADCOM PROPRIETARY & CONFIDENTIAL34
CFRT Wrap-up Networking and Informal Q&A
BROADCOM PROPRIETARY & CONFIDENTIAL35
Broadcom Data Center Tour
(for those interested)
THE END