Upload
thaddeus-lockard
View
223
Download
5
Tags:
Embed Size (px)
Citation preview
BayCare Health SystemDesigns
Next Generation Data CenterAugust 8, 2007
Doug LauterbachData Center Director, Bay Care Health System
Matt Schuster
Principal Consultant, BT INS
In today’s presentation…
• About BayCare• Challenges that necessitated a new data center• Planning the new facility• A look at the new data center• Your questions
About BayCare
• 9 hospitals in Tampa Bay area• Largest full service community-based health
system in the region• 17,000 team members• 2,707 Beds • 355,523 ER visits • 6.3 million lab tests
Challenges that necessitated a new facility
Business needs driving IT
• In the process of a 7 year electronic medical record project (moving towards paperless)
• Constant growth of digital imaging• Increasing compliance regulations - JCAHO
concerns about information management and availability
• Increased clinical reliance on IT systems requires high availability for quality patient care– Pharmacy– Lab– Radiology– Food and Nutrition– Bed tracking– Scheduling
Business Drivers of the new Facility
• Best Practices• Business Continuity• Business Expansion• Capacity Planning or Performance Analysis• Change in Application Architecture, Providers or Vendors• Compliance• Disaster Recovery• Expense Management• Increased Capacity• Operational Optimization• Regulatory• Reliability• Risk Management and Mitigation• Technology Growth or Consolidation• Technology Refresh
Existing facility challenge: space
• Short on space– 8 separate spaces for IT equipment– 4,700 sq. ft data center split into 4 spaces
• 30% of data center space used for circulation
• Large command center using a large portion of raised floor space
• Building’s physical resiliency not up to hurricane standards
• Design aspect to 40 watts/sf• Shallow cabinets limit the ability to expand and
distribute power and low-voltage cabling• Structure cabling unable to support:
– Blades– VoIP– Anticipated SAN growth
Existing facility challenge: power load
• Design didn’t support heat and power load• Downtime related to power design and capacity
– circuit capacity lacking– Distribution: not dual bus all the way to
equipment• Lack of available power threatening to limit new
technology adoption• No standard power configurations• No means or method to manage/measure power
at the equipment racks
Existing facility challenge: cooling
• Cooling was inefficient– Not using hot-aisle/cold-aisle design– Uneven temperatures through space– Hot spots creeping up
• Equipment lifespan likely shortened due to heat• 13-year old design, not optimized for blade
configurations• Not easily expandable• Low 12” raised floor and under-floor low-voltage
cabling restricting air supply• No available physical space for additional cooling• Legacy CRACs were not communicating to
efficiently cool the space.
Existing facility challenge: visibility & control
• Monitoring – currently focuses on circuits that are tripped (reactive), not designed for proactive data
• Temperature monitoring handled at the return air of the precision air conditioner (not looking at hotspots)
• Little remote control for servers• Lights-out management (iLO)• Existing Liebert SiteScan equipment was not
managing non-Liebert equipment• No temperature sensors at equipment location or
around the room• No branch circuit monitoring• 100% of lighting on 24x7
Existing facility challenge: managing change
• Provisioning is not standardized, automated or managed
• System deployments are based on:– looking through racks– personal knowledge– gut feeling– where uneducated users want placement
• Not managing capacity of:– space– circuits– power– weight– heat loads– network switch ports
Planning the New Facility
Objectives
• Provision for planned growth – addition of 100 servers per year
• Gain complete visibility and control and improve administration
• Improve system availability through full redundancy of the power and cooling systems
• Improve fire detection and suppression• Implement hot-aisle/cold-aisle design for maximum
capacity and energy efficiency• Improve planning and provisioning• Improve cable management• Increased Mechanical and Electrical efficiencies• Reduced operational costs• Operate as a nearly lights-out facility• Remote access into all systems
Building a Team
• Doug LauterbachImaging Services Director (serving as Data Center Project Director), BayCare Health System
• Chris Jenkins, Director Technology, BayCare Health System
• Matt Schuster, Principal Consultant, BT INS• Thomas Gooch, Local Liebert Representative,
Emerson Network Power• Paul Carastro, Carastro & Assoc, MEP Engineers• Mike Montecalvo, General Contractor, Solutions,
Inc.• Paul Schnitzlein, Architect, Harvard Jolly
Documenting Existing Equipment
• Document existing devices, power supplies and port counts per device at the rack level
• Determine/Graph device averages per rack to determine cabinet profiles
• Each piece of equipment assigned a number and quantities and loads were considered. Classifications included: – Legacy– Low– Medium– High– Blade– IBM RISC– Network
Planning Process: Modeling
• Mapped watts per square foot via CFD several times, tweaking each time to determine placement of precision air conditioners
• Room designed for 800-1,000 kW; CFD proves capacity
• 24-inch raised floor, multiple precision air conditioning units
• White space to grow into: 40 to 50 percent allows for doubling of capacity
• Rerun CFD every three to six months as changes are required using actual values collected from thermistors, branch circuit monitoring and other tools
Existing facility: Servers per Cabinet
Planning Process: Profile Planning
Cabinet Profiles
Best Practice: Technology Infrastructure
• 44U, 45” deep cabinets to allow for distribution of power and low-voltage cabling
• Metered Power Strips– Every cabinet has at least two strips, one for
each bus– Some cabinets with additional strip behind a
static transfer switch– Each strip has colored tape to quickly
determine which bus• KVM over IP in each cabinet
– Centralized– Decentralized
• iLO switch in each cabinet ($38/Ethernet port)
Best Practice: Power System Design
Liebert Precision Power Center
225 kVA480 Volt Input
120/208 Ouptut
Liebert 610 UPS 225 KVA
480v Input480v Output
BUS A
Dual Power Supply Servers
Automatic Transfer Switch
# 1RSC
Liebert 610 UPS 225 KVA
480v Input480/ Output
BUS B
70 kW A/C
BUSA
70 kW A/C
70 kW A/C
70 kW A/C
Main Electrical
distribution panel
Automatic Transfer Switch
# 3Building
800 A EmergencyDistribution
Panel480v
New caterpillarGenerator1000 KW
Automatic Transfer Switch
#2SPAWAR
Liebert Precision Power Center
225 kVA480 Volt Input
120/208 Ouptut
Dual BusRemote
Distribution Cabinet
BUSB
BUSA
Dual BusRemote
Distribution Cabinet
BUSB
Dual Power Supply Servers
Best Practice: Dual-Bus EPO Systems
Best Practice: Fire Suppression
• Very Early Smoke Detection Apparatus (VESDA)• Clean Agent Suppression system
– People-friendly– Environmentally friendly– Lower quantity of agent vs. FM-200– Over 3,000 pounds of FE-25 agent
• Four zones• Over 150+ detection points (above and below
floor)• Sprinklers were required, therefore pre-action with
double-interlock was installed
Best Practice: Remote Management
• Integrated Lights Out – allows remote control of servers from anywhere in the world
• People-free facility benefits:– Removed NOC Command Center from raised
floor– Minimizes human error disruptions– Allows managers to monitor and control
environment remotely
Best Practice: Proactive Monitoring
• Followed some of ASHRAE’s TC9.9 recommendations for monitoring temperature in racks/cabinets
• Temperature and Relative Humidity• Branch Circuit Monitoring• SiteScan I/O-32 and additional units• Leak Detection• VESDA• Fire Suppression• Fire Detection• Liebert Environmental Control Systems ICOMs
Control Network
Best Practice: Labeling
• Mechanical equipment• Electrical equipment• Electrical receptacles• Grid Coordinate• Racks and Cabinets• Structured Cabling
Tools: Rackwise/DCM
• Used tool to document existing inventories• Tracking relocation from one facility to another• Manage asset location by rack unit• Provisioning limitations per cabinet• Reporting capabilities
– Space and Assets– Power and Cooling– Cost and Capacity Planning– Rack Assembly with Cabling– Service & Warranty Contracts– Department & Customer Information
• Long term objective: to integrate a change/configuration management tool to track and manage assets
Tools: Rackwise/DCM
Tools: Computational Fluid Dynamics
• Allowed modeling the complete room for the initial 6 CRACs and full deployment of 12
Tools: Computational Fluid Dynamics
• Initial assessment indicates potential problems related to placement of precision cooling units…
Low Pressure Zone
CRACs
1 2
34 5 6
Tools: Computational Fluid Dynamics
• CFD Program allows for virtual change - validating proposed solution prior to installation.
CRAC Moved To New Location
Low Pressure Zone Removed
Tools: Computational Fluid Dynamics
• Follow-on Modeling Provides Proof of Operational Success Assuming Full Load – 735 kW.
Clouds Per Fahrenheit Degree
Tools: Computational Fluid Dynamics
• Modeling allows for proof of design concept at full load – prior to implementation
Pathlines from CRACs
Temperature (oF) in 1 foot increments
CRAC Network Communications
New Data Center – Construction
New Data Center – Construction
Current Data Center – Raised Floor
Current Data Center – Raised Floor
Current Data Center – Raised Floor
New Data Center – Raised Floor
Current Data Center – Operations
New Data Center – Operations
Current Data Center – Generator
Current Data Center – Network Rack
Current Data Center – Server Cabinets
Current Data Center – Server Cabinets
Current Data Center – Server Cabinets
Current Data Center – UPS Room
New Data Center – UPS Room
Current Data Center – Yard
New Data Center – Yard Construction
New Data Center – Yard Outside
Critical Success Factors
• Teamwork between BayCare, INS and Emerson Network Power/Liebert enabled success
• Maximization of modeling tools allowed a solid design, even before the first nail was hammered
• Have done virtualization study – partially executing recommendations ahead of time to make move easier
• Can’t convert everything to dual corded, so a Liebert STS was installed to provide redundancy
• Space for a second generator is provisioned and an ATS is in place to accommodate expansion
• Move in progress!
Questions and Answers