Columbia University’s Advanced Concept Data Center Pilot Project

May 15, 2009

22

Agenda1:00pm Introductions

1:15pm Welcome and Opening Remarks Candace Fleming, Vice President, CUITDavid Hirsh, Executive Vice President for ResearchNilda Mesa AVP, Environmental Stewardship

1:35pm Green DC Program presentation Alan Crosswell, Chief Technologist, CUIT

2:20pm CU Data Center Ian Katz, Manager, CUIT Data Center

Current Status

2:35pm NYSERDA Project Overview Richard Hall, Project Manager CUIT

3:00pm Closing Comments & Questions from guests Victoria Hamilton, Research Initiatives Coordinator

The “Big Potential” Comments from NYSERDA (tentative)

Introductions

May 15, 2009

44

External Visiting Committee

• Laurie Kerr, Senior Policy Advisor for Energy and Green Buildings, NYC Mayor’s Office of Long-Term Planning and Sustainability

• Vace Kundakci, Assistant Vice President for Information Technology and Chief Information Officer, City College of New York/City University of New York

• Timothy Lance, President and Board Chair, New York State Education and Research Network (NYSERNet)

• Marilyn McMillan, Associate Provost and Chief Information Technology Officer, New York University

Role: The External Visiting Committee is charged with maintaining skepticism about our plans and our

accomplishments from the perspective of external institutions. To the extent we demonstrate feasibility and potential impact, they will serve as powerful ambassadors to other institutions.

55

Internal Advisory Group• Wilmouth A. Elmes, Associate Vice President of Engineering/Technical Services, Manhattanville

Development Project

• Doug McKean, Director, Capital Project Management

• Arthur M. Langer, School of Engineering and Applied Science, Senior Director of the Center for Technology, Innovation, and Community Engagement and Faculty & Assoc Director, Executive Masters of Science in Technology Management at the School of Continuing Education

• Nilda Mesa, Assistant Vice President, Environmental Stewardship

• Scott W. Norum, Chief Administrative Officer for Arts and Sciences and Vice President, Office of the Vice President for Arts and Sciences

• Leonard Peters, School of Business, Associate Dean and Chief Information Officer – Information Technology

Role: The Internal Advisory Group serves two primary functions. Not only do we anticipate they will prove a

valuable sounding board for the execution of the pilot (implementation guidance/expert judgment), but we expect them to pose the hard questions about issues surrounding scale-up (feedback).

66

Research Faculty User Group

• Liam Paninski, Associate Professor, Dept. of Statistics

• Lei Cong, Staff Associate, Dept. of Statistics

• Kathryn V. Johnston, Associate Professor, Dept. of Astronomy

• Mary Putman, Clare Boothe Luce Associate Professor of Astronomy

• Greg L. Bryan, Assistant Professor , Dept. of Astronomy

Role:

Responsible for vetting the results of the combined HPC cluster for the Department of Astronomy and the Department of Statistics

77

Working Group

• Principal InvestigatorAlan Crosswell, Associate Vice President and Chief Technologist, CUIT

• Co-Principal Investigator Victoria Hamilton, Director of the Office for Research Initiatives (ORI)

• Richard Hall, Project Manager• Anthony Cirillo, AVP Systems and Operations• Peter Miner, Senior Director, Systems Engr• Ian Katz. DC Facilities Manager• Melissa Metz, Director of Systems Engineering • Fran Ovios, Director of Systems Engineering • Stew Feuerstein, Director and Chief Architect • Joseph Rini, Director of Network Engineering • Megan Andersen, Research Systems Administrator • Fran Fitzgerald, CU Facilities Project Manager

Role: Day-to-day responsibility for planning, executing and measuring required by the proposal. The

Columbia University in-house team will perform much of the work.

8

Project Structure

Welcome & Opening Remarks

Candace FlemingVice President and CIO, Columbia University Information Technology

David HirshExecutive Vice President for Research

Nilda Mesa AVP, Environmental Stewardship

Green DC Program presentation

Alan Crosswell, Assoc VP and Chief Technologist, CUIT

CU Data Center - Current Status

Ian Katz, Facility Manager, CUIT Data Center

NYSERDA Project Overview

Richard Hall, Project Manager, CUIT

1313

Project Objectives• Become well versed in data center efficiency, design techniques and assessment

metrics such as those espoused by Green Grid, ASHRAE and others (PUE, DCeP, etc.).

• Establish baselines and continuously measure several power and cooling variables to enable the study of historical trends and produce a data set to aid in further analysis of data center electrical and heat loading changes over time.

– Implement the recommended and advanced best practice improvements and measure the degree of efficiency achieved when these techniques are applied in our data center

• Validate claims of the degree of power and cooling efficiency improvements that are achievable through best practices and advanced data center techniques.

• Utilize results to inform subsequent phases of the central data center re-fitting and design of future data center space

• Provide training and guidance for other NYS data center operators, especially peer higher education institutions

1414

Project Scope• Include participation and critique from influential skeptics

• Update the Data Center’s aging infrastructure (modular approach)– Replacement of thirty servers, each of which is at the end of their useful life – Implementation of nine high-density equipment racks

• support current and future research computing clusters • consolidating the University’s server needs for High Density Computing

(HDC). – Address electrical distribution systems to increase efficiency– Improve and or replace inefficient cooling systems– Installation of instrumentation for power and cooling systems to monitor

consumption and performance to establish metrics to measure our progress toward PlaNYC.

• Realize energy and environmental gains while demonstrating economic and operational feasibility in infrastructure improvements, innovative computational equipment, centralized administrative systems and dispersed research clusters.

1515

Project PlanTo achieve the overall project objectives, CUIT has broken down the project into major tasks.

• Inventory

– Create detailed physical inventory of existing in-scope server racks

– Create data center baseline (formulated from above inventoried equip)

• Instrument server power consumption

– Install network monitored power monitors for each server

– Perform data collection at 5-min intervals using one or more of the following methods:

• Instrument server input air temperature and overall data center chilled water heat load

– Install server input ambient air temperature for each server

– Install BTU metering on data center supply and return lines

– Perform data collection at 5-min intervals

• Establish overall Data Center profile

– Utilize equipment load results to establish baselines

– Develop PUE ratio for entire data center & inventoried servers

• Implement 9 racks of high power density in-row cooling

– Install 9 server racks outfitted for high power density

– Provide in-row cooling subsystem for 9 server racks

– Implement measurement system for in-row cooling, BTU rejection and electrical consumption

1616

Project Plan Cont’d• Replace 30 “old” servers and measure efficiency improvement.

– Consolidate the replacement servers into high density racks and re-implement the same IT services

– Take measurements of before-and-after power consumption

– Document expected and actual efficiency improvement

• Compare old and new research clusters

– Benchmark applications on new Astronomy/Statistics HPC cluster

– “kWh consumed” comparison with old Astronomy server cluster

• Implement server power management

– Install server BIOS/high level power management feature upgrades to servers (identified in task 2)

– Document benchmarks

• Increase chilled water set point for high power density racks and measure

– Document measured before-and-after BTUs

• Communicate results

– Share results with key stakeholders

1717

Project Schedule

• Project Approval 4/1/09• Inventory 5/15/09• Instrument Power 7/30/09• Instrument Cooling 9/30/09• Data Center Profile 9/30/09• Nine Racks 1/31/10• Replace 30 Servers 5/30/10• Compare Clusters 10/31/09• Server Power Mgmt 9/30/10• Increase 5-degrees 3/31/10• Project Review 9/30/10• Project close 10/1/10

Closing Comments

Victoria Hamilton, Director of the Office for Research Initiatives (ORI)