ShiningaLightonSmallDataCentersintheU.S.

Authors:

MohanGaneshalingam,ArmanShehabi,Louis-BenoitDesroches

EnergyAnalysisandEnvironmentalImpactsDivisionLawrenceBerkeleyNationalLaboratory

June2017

ThisworkwassupportedbytheFederalEnergyManagementProgramoftheU.S.DepartmentofEnergyunderLawrenceBerkeleyNationalLaboratoryContractNo.DE-AC02-05CH1131.

LBNL-2001025

1

AbstractLargedatacentersarewellknownfortheirhigh-energyintensityandhavemadedramaticefficiencyimprovementsoverthepastdecade.Smallclosetandroomdatacentershavereceivedmuchlessattention,yetconstituteasignificantfractionofthetotalnumberofserversintheUnitedStates.Theoftenmakeshift,adhocnatureofsmalldatacentersoftenresultsinlittleattentionpaidtoenergyefficiencyandinadequatecoolingequipment.Thesmallphysicalfootprintofthesedatacenters,typicallyembeddedwithinalargerbuilding,makesitdifficulttoidentifyandtargetforefficiencymeasures.Theseconditionsmakesmalldatacentersnotoriouslyinefficientrelativetotheirlargercounterparts.Inthisreport,wepresentananalysisofsmallandmidsizedatacentersintheUS,drawingfromsurveysofcommercialbuildingstock.Wefindthatserversinsmalldatacentersmakeupapproximately40%ofinstalledserverstock,withthevastmajorityofsitesutilizingonly1-2servers.Weidentifyindustrieswheresmalldatacentersaremostprevalent,findingthatthehighestsaturationsareinmedical,retail,office,andeducationsectors.Smalldatacenterstypicallylackdedicatedcoolingequipment,oftenrelyingonbuildingairconditioningandventilationequipmentforcooling.Wefurtherfindthatthetypeofcoolingequipmentusedishighlycorrelatedwiththenumberofoperationalserverracks,withlessefficientcoolingoptionsusedwithfewerracks.WedevelopgeospatialmapsofsmallandmidsizedatacenterstovisuallyidentifyregionsofhighserverconcentrationandcalculateassociatedCO2emissions.Smalldatacentersconsume13billionkWhofenergyannually,emitting7millionmetrictons(MMT)ofcarbondioxide–theequivalentemissionsofapproximately2.3coal-firedplants.WediscussefficiencymeasuresthatcouldbeimplementedandestimatepotentialenergyandCO2savings.

2

Acknowledgements

TheauthorswouldliketothankHeidiFuchsandBlakeRingeisenforusefuldiscussionsregardingtheservermarket.RichardBrownprovidedhelpfulsuggestionsandfeedbackthatimprovedtheclarityofthispaper.

3

1. IntroductionThedigitizationofthemodernU.S.economyhasmadedatacentersessentialinfrastructureforcommercialbusinessesacrossallindustries.Moderndatacentersarepopulatedbycomputingequipment,whichoperatecontinuouslytosupporton-demandnetworkrequests.Theyrangeinsizefromindividualserversfoundinliteralclosetstoexpansivewarehousesfilledwiththousandsofservers.Datacentersofallsizesareusedtomanagecommunications,supportbusinessoperations,andcontrolaccesstodata.Attheheartofdatacenteroperationsareservers,networking,anddatastorageequipment.Serverstypicallyrundedicatedapplicationsandprocessrequestsreceivedviaadistributednetwork.Evenduringperiodsoflowutilization,thepowerdrawofaconventionalservermaybeasmuchas150-350W(Brownetal.,2008;Koomey,2011;Masanetetal.,2011;Shehabietal.,2016).Factoringinspaceconditioningandotherfacilityoverheadequipmentcandoubledatacenterelectricityconsumption(CenterofExpertiseforEnergyEfficiencyinDataCenters,n.d.).Consequently,buildingswithdatacentershaveahighenergyintensityrelativetoothertypesofcommercialbuildings(EIA,n.d.).Inaggregate,datacentersareestimatedtocurrentlyconsume70billionkWhintheU.S.alone,representingapproximately1.8%ofU.S.electricityconsumption(Shehabietal.,2016).Severalstudieshaveanalyzedtheevolvinglandscapeofdatacentersmostlyfocusingonlargestdatacentersthathousethousandsofservers(Brownetal.,2008;Koomey,2011;Masanetetal.,2011;Shehabietal.,2016).Inarecentstudy,Shehabietal.(2016)foundthatalthoughthereisoverallgrowthinthesizeandnumberofdatacentersovertime,thetotalenergyconsumptionattributedtodatacentershasremainedrelativelyconstantforthepastfewyears.Thisresultisaproductofconcertedeffortstooptimizeoperationsforthelargestdatacenters,whichareabletoreaplargeoperationalcostsavingsfromefficiencyimprovements.Technologycompaniesoperatinglargedatacentersbenefitfromeconomiesofscaleandhavetheresourcestoinvestinefficiencymeasures.Givenhighoperationalcostsoflargedatacenters,upfrontcapitalexpendituresonefficiencymeasuresarequicklyrecoupedbyoperatingcostsavings.Googlepresentsacasestudyinwhicha$25,000expenditureonefficiencymeasuresledtoannualsavingsof$67,000-apaybackof5months(Google,2011).Forthisparticulardatacenter,Googleestimatedayearlyenergysavingsof670MWh.Whenconsideringthelargenumberofdatacentersoperatedbysimilartechnologycompanies,undertakingefficiencymeasuresleadstomillionsofdollarsinsavings.Significantlylessattentionhasbeenpaidtosmall-scaledatacenters,typicallyreferredtoasdataclosetsanddatarooms(or“embedded”datacenters).Thesmallphysicalfootprintofthesedatacenters,typicallyembeddedwithinalargerbuilding,makesitdifficulttoidentifyandtargetforefficiencymeasures.Thesespacesaregenerallyrunbyinstitutionslessfamiliarwithinformationtechnologysystemsandbestpracticesfordatacentermanagement.Theoftenmakeshift,adhocnatureofsmalldatacentersresultsinlittleattentionpaidtoenergyefficiencyandinadequatecoolingequipment.Theseconditionsmakesmalldatacentersconsiderablylessefficientrelativetotheirlargecounterparts.

4

AstudyperformedbyCheungetal.(2014)surveyed30smallserverclosetsandroomswithfoursitesselectedfordetailedassessments.Theauthorsfoundthatmostsiteswerenotdesignedforefficientserveroperationandnotedmanyexamplesofpoorserverroommanagement.Forexample,inmanycasesconditionedcoolairwasinefficientlydirectedwithintheserverroomleadingtomixingwithexhaustedwarmair.ManyofthepotentialefficiencymeasuresoutlinedinCheungetal.rangefromno-tolow-cost.However,theauthorsnotebarriersimpedingmoreefficientoperationareorganizationalratherthantechnological.Feworganizationshadpoliciestopromoteefficiencyandmostlackedproperlytrainedstafftoresearchandimplementefficiencymeasures.BennettandDelforge(2012)performedasurveyof30businessesoperatingsmalldatacenterslookingspecificallyatthepenetrationofefficientoperatingpractices.Thesurveycovereddatacentersoperatingbetween1-30servers.Ingeneral,theyfoundalackofawarenessandorganizationalprioritizationofefficiencymeasures.Datacenteroperatorsweretypicallynotresponsibleforpayingenergybills,limitingmotivationforpursuingefficiency.SimilartothefindingsofCheungetal.,barrierstoefficiencywereorganizationalasopposedtotechnological.BennettandDelforgehighlighttheneedforlocalandstatepolicymeasurestocreateincentivestopromoteefficiencymeasuresspecificallyaimedtowardssmalldatacenters.Todate,nostudyhasanalyzedarepresentativesampleofcommercialbusinessesoperatingsmalldatacentersduetoadearthofdata.Thelackofdatahaslimitedtheabilityofpolicymakerstoeffectivelycraftpolicytoaddressenergywasteinsmalldatacenters.Inthisreport,weuserecentlyreleasedsurveydataofcommercialbuildingstocktoanalyzecharacteristicsofsmallandmidsizedatacenters.Weusetherecentlyreleased2012CommercialBuildingEnergyConsumptionSurvey(CBECS)administeredbythefederalEnergyInformationAdministration(EIA)toanalyzehowserversaredistributedgeographicallyandinwhattypesofbuildingstheyarelocated(EIA,2015).ThesedatarepresentanationallyrepresentativesurveydetailingenergyconsumptionpracticesofU.S.commercialbuildings.CombiningthesedatawithoccupationdatafromtheBureauofLaborStatistics(BLS)weconstructmapsofgeographiclocationofsmalldatacentersincommercialenterprises,andofserverlocationforsmallanddatacentersintheU.S.Additionally,weperformananalysisofthe2014CommercialBuildingStockAssessment(CBSA)administeredbytheNorthwestEnergyEfficiencyAlliance(NEEA).1AlthoughlimitedtodatacentersinthePacificNorthwestregion,thesurveyprovidesadetailedglimpseintothespacecoolingandservervirtualizationpracticeswithinsmalldatacenters.Thereportisorganizedasfollows.InSection2,wediscussthedatasourcesandmethodologyusedinouranalysisofdatacenters.Althoughthefocusofthisreportissmalldatacenters,wealsoincludeanalysisofmidsizedatacenters,whichfallinthegapbetweensmalldataclosetsandroomsandlargehyper-scaledatacenters.InSection3,wediscusstheresultsofouranalysisoftheCBECSandCBSAsurveys.InSection4,wepresentgeospatialmapsofserverintensityofsmallandmidsizedatacenters.WethendiscussaggregateenergyandCO2emissionsfromsmalldatacenterconsumptionandpotential1http://neea.org/resource-center/regional-data-resources

5

energysavingsfromefficiencymeasures.Section5providesasummaryofresultspresentedinthisreport.

2. DataandMethodsInthissectionwedescribethedatausedinouranalysisofsmallandmidsizedatacenters,aswellasthemethodologytoestimatenationalenergyconsumptionandCO2emissions.WewillusetheCBECSandCBSAstudiesofcommercialbuildingstocktocharacterizevariousaspectsofthesmallandmidsizedatacentermarket.WewillalsoutilizethenationallyrepresentativeCBECSdatatoestimatenationalenergyuseandCO2emissionsfromsmallandmidsizedatacenters.WestartwithU.S.commercialbuildingstocksurvey,estimatethenumberofservers,anddisaggregategeographicallybasedonregionalemploymentstatistics.

CommercialBuildingEnergyConsumptionSurvey2012CBECSisanationallyrepresentativesurveyofcommercialbuildingsintheUnitedStatesconductedbytheEIA.Thesurveyprovidesasnapshotofenergy-relatedbuildingcharacteristicsofU.S.commercialbuildingstock.CBECSdefinescommercialbuildingsasthosethatarenotprimarily(i.e.,>50%offloorspace)usedforresidential,industrial,oragriculturalpurposes.ForCBECS2012,EIAsurveyed6,720buildingsandweightedtheirsampletobenationallyrepresentativeofcommercialbuildingstock(EIA,2015).Thepubliclyreleasedmicrodatausedforthisanalysisisanonymizedtoremoveanycharacteristicsthatcouldpossiblybeusedtoidentifyindividualbuildings.Aspartofthisprocess,thelocationofbuildingsisonlymadeavailableattheCensusdivisionlevel(groupsof4-9states).

In2012,EIAincludedsurveyquestionstospecificallytargetserverusageincommercialbuildings.Whenquestioningrespondents,CBECSdefinedserversas“usuallyjusttheCPU,or‘case,’portionofacomputerthatmanagesnetworkresourcessuchascomputerfiles,printers,databases,ornetworktraffic;serversdonotrequiremuchhumanoperation,somostdonothavekeyboardsormonitors.”

Buildingswith500ormoreserversarecodedinCBECSwith‘9,995’inplaceoftheactualnumberofservers.Thesebuildingscorrespondto26recordsrepresentingapproximately2,000buildings.Thesesitesmayrepresentlargeenterpriseand‘hyper-scale’datacentersoutsidethescopeofthisreportandomittedfromouranalysis.However,theyarenotclearlyidentifiedasdatacentersbyactivityinthedataset.ItisalsopossiblethatthesedatacentersareclassifiedasindustrialfacilitiesoutsidethescopeofCBECS.

CBECScharacterizestheprincipalbuildingactivity(PBA)ofeachsampledbuildingintocategoriestogroupbuildingswithsimilarenergyconsumptionpatterns.Activitiescoverabroadrangeofcategoriesfrom‘Education’to‘WarehouseandStorage’.2,3AlthoughCBECS

2Fordefinitionsseehttp://www.eia.gov/consumption/commercial/building-type-definitions.cfm3CBECSalsoprovidesanadditionalmoredetailedcharacterizationofthebuildingactivitycodedasPBAPLUS.Forexample,the‘Education’PBAisfurthersubdividedinto‘elementary/middleschool’,‘highschool’,

6

includesaPBAfor‘DataCenters’(separatefromaquestionofwhetherthereisadatacenterorserverfarminthebuilding),therearenobuildingscategorizedassuchinthedataset.

Forthiswork,weadoptaclassificationsystemofeitherasmallormidsizedatacenterforeachbuildinginCBECSwithatleastoneserverbasedonthenumberofreportedservers.OurclassificationisrootedinthedatacentertaxonomydefinedbymarketreportsreleasedbytheInternationalDataCorporation(IDC).IDCcategorizesdatacentersinto5types,rangingfromsmallclosets(1-4servers)toenterprise-level,high-enddatacenters(>500servers).Indescriptionsofeachspace,IDCreportsthatbothclosetandroomdatacentersarehousedinrelativelysmallspaces(<500squarefeet),oftenlackdedicatedspacecooling(Baileyetal.,2007),andareusedprimarilybysmallbusinesses.Forourpaper,wedefineasmalldatacenterashavingbetween1-25servers,capturingbothclosetandroomservers.Allotherdatacentershavingmorethan25servers,butlessthan500areclassifiedasamidsizedatacenter.Table1providestypicalpropertiesforIDC’sdatacentertaxonomydatacentersandtherelationshiptoourclassification.Table1:Datacentercharacteristics

ClassificationThisWork

TaxonomyIDC

SpaceDimensions(squarefeet)

TypicalNumberofServers

SmallDataCenter

Closet <=100 1-4Room 101-1000 5-25

MidsizeDatacenter

LocalizedDatacenter 1,001-2000 26-100

Mid-tierDatacenter 2,001-20,000 101-499

NotAnalyzed High-endDatacenter >20,000 >500

CBECSreportsdatacentersquarefootageforthosebuildingswhererespondentsidentifiedadedicatedspaceforservers.Ideally,thesedatacanbeusedtoinformourserverspacecategorization.However,wefindthesedatanotusefulforcategorizingserverspaceforallbuildingsthatreporthavingservers.Forexample,98%ofrespondentswith5orlessserversdidnotidentifyhavingadatacenterandthusdidnotreportadatacentersquarefootage.Respondentswhoidentifiedadatacenterintheirbuildingappearbiasedtolargerspacetypes.Basedonthelackofreportingforallbuildingswithservers,wedonotusethedatacentersquarefootageestimatesfromCBECSinourcategorizationofserverspacetype.

‘university/college’,and‘othereducation’.Ingeneral,ouranalysisdoesnotbenefitgreatlyfromtheextradetailprovidedbythesemorespecificPBAs.Ingeneral,wewillpresentresultsintermsofPBA.However,whennoteworthy,wehighlightinterestingtrendsbythemorespecificbuildingactivitywithindiscussionofresults.

7

ItisunclearhowCBECSreportsbuildingswithmultipleseparatedatacenters.WeassumethatCBECSreportsthetotalnumberofserversinthebuildingcorrectly,butdoesnotdifferentiatebetweenmultipledatacenters.Althoughthisaddsalevelofuncertaintytoourestimatesofthenumberofdatacenters,wedonotexpectthistogreatlyimpactourdatacenterclassificationofbuildingsgiventhebroadcategorizationusedinourclassification.ItispossiblethatCBECSresultsmaysufferfromvariousformsofreportingbiasdependingonhowrespondentschosetoestimatethenumberofserversintheirbuilding.RespondentsansweringquestionsinCBECSarenotnecessarilytheindividualsmaintainingITequipment.CBECSreliesonestimatesfromrespondentsandresponsesarenotverifiedindependently.Inparticular,ourdatacentercategorizationsassumethattherespondentcan(1)positivelyidentifyserversand(2)provideareasonablyaccurateestimateofthenumberofserverson-premise.Thereislittlewecandotowardscharacterizinguncertaintiesduetothefirstassumption.Thesurveycouldpotentiallyunder-orover-reportthenumberofserversdependingoneachrespondent’sabilitytorecognizeandquantifyservers.Forthesecondassumption,ourclassificationofdatacentersaseithersmall,midsize,orhigh-endde-emphasizestheneedforexactestimatesforclassification.Forexample,arespondentmaypotentiallyconfuse5serversfor7or8servers,butisunlikelytoconfuse5serversformorethan25.

CommercialBuildingStockAssessment2014The2014CBSAsurveywasconductedbyNEEAinordertoprovideasnapshotofenergy-consumingdevicesfoundinregionalcommercialsites(NavigantConsulting,2014;NorthwestEnergyEfficiencyAlliance,2014).ThesurveyisspecifictocommercialbuildingsinthePacificNorthwestcoveringOregon,Washington,Idaho,andMontana.Detailedauditsofcommercialstockweregatheredfor859randomlyselectedcommercialsitesacrosstwelvebuildingtypes.Selectedsitescoveredbothurbanandruralcommerciallocations.DatawereanonymizedtoavoididentificationofindividualsitesandweightedtoberepresentativeofthePacificNorthwest.AlthoughlimitedtodatacentersinthePacificNorthwestregion,thesurveyprovidesadetailedglimpseintothespacecoolingandservervirtualizationpracticeswithinsmalldatacenters.CBSA2014includedmanyquestionsspecifictodatacentercharacteristicsandoperation.UnlikeCBECS,whichprovidesserverdataforeachbuilding,CBSAdataareprovidedatthedatacenterlevel,allowingforasinglecommercialsitetoreportmultipledatacenters.Foreachdatacenter,thesurveyrecordedthetotalnumberofracksintheserverroom,datacenterfloorspace,andtypeofspacecooling.Althoughtherewereadditionalquestionsregardingthepresenceofuninterruptiblepowersupplies(UPS),mostresponseswereleftblank.Thesurveydidnotincluderackdimensionsorthenumberofserversineachdatacenter,makingdirectcomparisonstoCBECSdifficult.NEEAspecificallysurveyedheating,ventilation,andairconditioning(HVAC)characteristicsforeachsite.Onsitesurveyorsrecordedthepresenceofdedicateddatacentercoolingandthetypeofconditioningprovided(e.g.,computerroomAC,buildingtransfer,waterchillers,etc.),evenifconditionedairwassharedwithspaceoutsideofthedatacenter.

8

Datacollectionforhospitalsanduniversitiesfollowedadifferentsurveyprotocolthanotherbuildingtypesduetothecomplexitiesinrecordingresultsforalargecampus.Forhospitalsanduniversities,CBSArecordsthedatacentersquarefootage,butdoesnotrecordthenumberofracks.Additionaldatafor521buildingswascollectedforthe‘CBSAOversampleStudy’(NorthwestEnergyEfficiencyAlliance,2014).Thesedatawerecollectedasautility-orstakeholder-fundedextensionoftheoriginalsurvey.Althoughthesedatarepresentasignificantadditionofthedataset,NEEArecommendsuseoftheCoredataset,presumablytoavoidintroducingbiasfromtheadditionoftargetedcommercialsites.OuranalysismakesuseoftheCoredataset.

EnergyCalculationsThemostcommonindustrymetricforquantifyingoveralldatacenterenergyefficiencyismeasurementofthePowerUsageEffectiveness(PUE)coefficient(TheGreenGrid,2012).PUEisdefinedastheratioofthetotalenergyusedfordatacenteroperationstotheelectricitydrawnbyITequipment:

𝑃𝑈𝐸 = 𝐸!"#"$%&#%'

𝐸!"

PUEcapturestheoverheadenergyrequiredtomaintaindatacenteroperationsincludingspacecooling,lighting,electricitydistribution,etc.Forexample,aPUEof2indicatesthatforeverykilowatt-hourdrawnbyITequipment,anotherkilowatt-hourisrequiredtomaintainserveroperations.TheenergydrawofITequipmentcanbeexpressedasthesumoftheenergyrequiredtopowerservers,network,anddatastorageequipment

𝐸!" = 𝐸!"#$"#! + 𝐸!"#$%&' + 𝐸!"#$%&' Thetotaldatacenterenergyconsumptioncanbeexpressedas

𝐸!"#"$%&#%' = 𝐸!" × 𝑃𝑈𝐸 𝐸!"#"$%&#%' = 𝐸!"#$"#! + 𝐸!"#$%&' + 𝐸!"#$%&' ×𝑃𝑈𝐸

Bytheverynatureofdatacenteroperations,serversoperatecontinuouslyinanactivemode.Serverpowerdrawdependsonutilization.Studieshavefoundthatserversrarelyoperateathighlevelsofutilization(Brownetal.,2008;WSPEnvironment&EnergyandNaturalResourcesDefenseCouncil,2012).Shehabietal.(2016)foundthatactiveserversinsmalldatacenterstypicallyoperateataround10-15%utilizationassuming10%ofserversare“zombie”serverswhichareconnectedtopowerbutneverutilized(KoomeyandTaylor,2015).Weassumethatatypicalserverfoundinasmallormidsizedatacenterconsumes,onaverage,approximately180W(Shehabietal.,2016),basedonthepower-

9

scalingabilityandcomputationalprocessorcountgenerallyfoundinthesedatacentertypes.FollowingMasanetetal.(2011),weuseatop-downapproachtoestimateenergyconsumptionfromnetworkandstoragedevices.Weassumenetworkandexternalstorageenergyconsumptionisproportionaltotheenergyconsumptionofserverhardware.Weassumenoexternalstorageinsmalldatacentersand20%formidsizedatacentersbasedontheircalculationsforthenumberofexternalharddrivesderivedfromshipmentsinBaileyetal.(2007).Ontopoftheserverenergyconsumption,weattributeanadditional5%ofserverenergyconsumptiontonetworkequipmentinsmalldatacentersand10%ofserverenergyconsumptiontonetworkequipmentinmidsizedatacenters.PreviousworkhasfoundthatsmallerdatacentersaremorelikelytoexhibithigherPUEvaluesduetopoorspacecooling(Cheungetal.,2014;Shehabietal.,2016).DetailedonsitecasestudiesofsmalldatacentersperformedbyCheungetal.(2014)foundthatdataroomsandclosetswereoftenfashionedoutofrepurposedspacenotintendedforservers.Manyclosetsandroomsstartedastemporaryhousingforsmall-scaleserverneedswhichincrementallyexpandedovertimetoadedicatedserverspace.ThelackofattentionpaidtoproperspacecoolingledtowarmairexhaustfromITequipmentmixingwithcooledairfromHVACequipment.Insomecases,theauthorsnotedthatservercoolingwasprovidedbybuildingcoolingequipment,theimpactofwhichisreducedcoolingefficiency.DetailedassessmentsoffoursmalldataspacesfoundPUEsrangingfrom1.5-2.1,withtwodatacentershavingPUEsof2.1.Shehabietal.(2016)reportvaluesof2.5forclosetdatacentersand2.0forroomandlocalizeddatacenters.Forcomparison,verylargedatacenteroperatorsreportedPUEsaslowas1.1throughconcertedefficiencyefforts(Google,2016).Forthiswork,weadoptatypicalPUEvalueof2.1forsmallandmidsizedatacenters.

OccupationalEmploymentStatisticsCBECSprovidesgeographicdataattheresolutionofCensusdivision(i.e.,groupsof4-9states)allowingustoquantifyserverownershipasafunctionofcensusdivisionandPBA.Toproducehigherspatialresolutions,wemaketheassumptionthatserversarespatiallydistributedproportionallytoworkerswhowouldworkinabuildingwithaserver.Forthispurpose,wemakeuseofoccupationalemploymentdata,availableatahighergeographicresolution,toapproximatethenumberofserversbyU.S.zipcode.

WeutilizeOccupationEmploymentStatistics(OES)releasedinMay2014fromtheBureauofLaborStatistics(BLS,2015).Thesurveyisconductedsemi-annuallyviamailanddesignedtocollectemploymentandwageestimatesfor821occupations.Thedataareprovidedforover650geographicalareasthatareacombinationofmetropolitanstatisticalarea(MSA)andnon-metropolitanareas.TheU.S.OfficeofManagementandBudgetdefinesMSAsashavingarelativelyhighpopulationdensitywithcloseeconomictiesthroughthearea.MSAsgenerallyincludealargecityandassociatedsurroundingareas,oftencoveringmultiplecounties.RegionswithineachstatethatdonotfallintoanMSAaredefinedas“nonmetropolitanareas”.Althoughastatemayhavemultiplenon-metropolitanareas,wecombineemploymentstatisticsforallnon-metropolitanareaswithinastate.

10

TheBLScollectsdataonthenumberofemployedpersonsin821detailedoccupationsthataregroupedinto22majoroccupationcategoriesforeachregionalunit.WeusethemajoroccupationcategoriesprovidedbyBLSandidentifyCBECSPBAswherethoseoccupationsaremostlikelytobefound.ForsomeoccupationsthereisadirectcorrespondencebetweenoccupationandPBA.Forexample,“LegalOccupations”canbemappeddirectlyto“Office”buildings.However,amajorityofoccupationsinOEShaveaone-to-manymappingofoccupationtoPBAs.Insuchcases,workersforthatoccupation,inagivenMSA/non-metropolitanarea,aresplitproportionallyaccordingthenumberofworkersinthoseCBECSPBAsforthecensusdivisioncorrespondingtothearea.Table2showsourmappingofoccupationscodestoPBA.Table2:MappingofOESjobcodestoCBECSPBA

OESJobCode CBECSPBAManagementOccupations OfficeBusinessandFinancialOperationsOccupations OfficeComputerandMathematicalOccupations Office,LaboratoryArchitectureandEngineeringOccupations Office,LaboratoryLife,Physical,andSocialScienceOccupations Office,LaboratoryCommunityandSocialServiceOccupations Office,ReligiousworshipLegalOccupations OfficeEducation,Training,andLibraryOccupations Education,PublicassemblyArts,Design,Entertainment,Sports,andMediaOccupations Publicassembly

HealthcarePractitionersandTechnicalOccupations

Outpatienthealthcare,Inpatienthealthcare,Nursing

HealthcareSupportOccupations Outpatienthealthcare,Inpatienthealthcare,Nursing

ProtectiveServiceOccupations PublicorderandsafetyFoodPreparationandServingRelatedOccupations Foodservice

BuildingandGroundsCleaningandMaintenanceOccupations Office,Laboratory,Lodging

PersonalCareandServiceOccupations Service

SalesandRelatedOccupationsFoodsales,Retailotherthanmall,Enclosedmall,Stripshoppingmall,Foodservice

OfficeandAdministrativeSupportOccupations Office,LodgingFarming,Fishing,andForestryOccupations AgriculturalConstructionandExtractionOccupations IndustrialInstallation,Maintenance,andRepairOccupations Other

ProductionOccupations Non-refrigeratedwarehouse,Refrigeratedwarehouse,Other

11

TransportationandMaterialMovingOccupations Non-refrigeratedwarehouse,Other

Wefirstdeterminethenumberofworkersinagivenmetropolitan/non-metropolitanarea,𝑀𝑆𝐴! ,andinagivenPBA,𝑃𝐵𝐴! ,whichwedenoteas𝑁! 𝑀𝑆𝐴! ,𝑃𝐵𝐴! :

𝑁! 𝑀𝑆𝐴! ,𝑃𝐵𝐴! = 𝑇(𝑂𝐶𝐶! → 𝑃𝐵𝐴!) ×𝑁!(𝑀𝑆𝐴! ,𝑂𝐶𝐶!)Inthisequation,weapplyourmappingofoccupationstoPBA,𝑇(𝑂𝐶𝐶! → 𝑃𝐵𝐴!),to𝑁!(𝑀𝑆𝐴! ,𝑂𝐶𝐶!)whichisthenumberofworkersin𝑀𝑆𝐴! ineachoccupationcode,𝑂𝐶𝐶! .𝑇(𝑂𝐶𝐶! → 𝑃𝐵𝐴!)isamatrixof0sand1swhichdescribeshoweachoccupationcodeisrelatedtoeachPBA.Nextwecalculatetheparameter𝜂,whichisthefractionofworkersinagiven𝑀𝑆𝐴! and𝑃𝐵𝐴! relativetothetotalnumberofworkersassociatedwith𝑃𝐵𝐴! inthecensusdivisionthatcontains𝑀𝑆𝐴! ,𝐶𝐷𝐼𝑉!"!!:

𝜂 𝑀𝑆𝐴! ,𝑃𝐵𝐴! = 𝑁!(𝑀𝑆𝐴! ,𝑃𝐵𝐴!)

𝑁!(𝑀𝑆𝐴! ,𝑃𝐵𝐴!)!"!!∈!"#!!"!!

FromCBECS,wehavethenumberofserversisagivencensusdivisionandPBA.Togetthenumberofserversinagiven𝑀𝑆𝐴! and𝑃𝐵𝐴! ,wemultiplythenumberofserversfoundinthe𝐶𝐷𝐼𝑉!"!! and𝑃𝐵𝐴! by𝜂:

𝑁! 𝑀𝑆𝐴! ,𝑃𝐵𝐴! = 𝑁! 𝐶𝐷𝐼𝑉!"!! ,𝑃𝐵𝐴! ×𝜂(𝑀𝑆𝐴! ,𝑃𝐵𝐴!)Andfinallytodeterminethenumberofserversin𝑀𝑆𝐴! ,wesumoverallPBAs:

𝑁! 𝑀𝑆𝐴! = 𝑁!(𝑀𝑆𝐴! ,𝑃𝐵𝐴!)!!!"" !"#$

Asimplifiedexamplefollowstoillustratetheprocess,beginningbymappingoccupationalcodestoagivenPBA.Considerauniversewheretherearetwooccupations(𝑂𝐶𝐶!and𝑂𝐶𝐶!)withtwopossiblePBAs(𝑃𝐵𝐴!and𝑃𝐵𝐴!).Allworkersin𝑂𝐶𝐶!areassumedtomaponto𝑃𝐵𝐴!.Workersin𝑂𝐶𝐶!canbeineither𝑃𝐵𝐴!or𝑃𝐵𝐴!.Weassumethat50%ofworkersin𝑂𝐶𝐶!willbein𝑃𝐵𝐴!and50%willbein𝑃𝐵𝐴!.Soifweassumethatthereare100workersin𝑂𝐶𝐶!and100in𝑂𝐶𝐶!,150workerswillbein𝑃𝐵𝐴!and50workerswillbein𝑃𝐵𝐴!.Next,consideracensusdivisionthatcontains3MSAs(𝑀𝑆𝐴!,𝑀𝑆𝐴!,and𝑀𝑆𝐴!).Inthiscensusdivision,wefind100serversassociatedwiththeretailPBA.Fromoccupationdata,wefind100workersin𝑀𝑆𝐴!,150in𝑀𝑆𝐴!,and250in𝑀𝑆𝐴!resultingin500totalworkersinourcensusdivisionassociatedwithretail.Wewouldthenassumethat20%(100/500)of

12

the100serversassociatedwithretailarein𝑀𝑆𝐴!.Similarly,30serversarefoundin𝑀𝑆𝐴!and50serversarein𝑀𝑆𝐴!.Lastly,forthepurposesofcreatinggeospatialmaps,serverswithinanMSAaredistributedevenlyacrossthezipcodesinthatMSA.Ourmethodtodisaggregateserverlocationbyzipcodeexplicitlyassumesthatthespatialdistributionofemployedworkersinbuildingswithserversisareasonableproxyforservers.

eGRID2012TheEmissionsandGenerationIntegratedDatabase(eGRID)administeredbytheU.S.EnvironmentalProtectionAgency(EPA)collectselectricitygenerationandemissionsdataforthemajorityofpowerplantsthatsupplyelectricityontheU.S.electricgrid(EPA,n.d.).ThesedataprovidedetailedprofilesofelectricitygenerationandassociatedgreenhousegasemissionratesacrosstheU.S.Dataarecollectedatthepowerplantlevel,thenaggregatedinto26eGRIDsubregionsdefinedbytheEPA.Variationsinemissionfactorsamongsubregionsareduetothemixofresourcesusedtopowertheregionalgrid.Subregionswithanelectricityportfoliowithsignificantcontributionsfromrenewableresources(i.e.,hydro,wind,solar,nuclear)haveloweremissionfactorsincomparisontoregionsthatrelypredominantlyoncoalandnaturalgas.

eGRIDprovidesemissionfactorsforbothbaseloadandnon-baseloadelectricitydemand.Forourestimates,weusebaseloademissionfactorswhencalculatingcurrentCO2emissionsfromservers.Reductionsinelectricityconsumptionimpactpowersuppliedbyancillarypowerplantsthatoperatecoincidentlywithpeakdemand.Weusenon-baseloadfactorswhencalculatingpotentialreductionsinCO2fromreducedelectricitydemand.eGRIDemissionfactorsarebasedonelectricitygenerationatthepowerplantandnotatthepointofconsumption.Tocapturelossesfromdistributionandtransmissionfromthepowerplanttotheconsumer,eGRIDderivesgrossgridlossfactorsfor5sectionsoftheU.S.electricgrid:Alaska,Hawaii,EasternU.S.,WesternU.S.,andERCOTwhichcoversmostofTexas.Thegrossgridlossfactorsarethenmappedontothe26eGRIDsubregionsandappliedtoourestimatesforgreenhousegasemissionstoaccuratelyaccountforalllosses.TotalCO2emissionsaretherefore:

𝐶𝑂! 𝑒𝑚𝑖𝑠𝑠𝑖𝑜𝑛𝑠 =𝐸!"#$

1− 𝐺𝐺𝐿!"#$%&× 𝐹 !"#$%&

!!!

where𝐸!"#$ representsenergyatthepointofconsumption,𝐺𝐺𝐿!"#$%&isthegridlossfactor,and𝐹 !"#$%&

!!! istheemissionsfactorrelatingtheamountofCO2emittedperunitofelectricitygenerated.Notethat𝐺𝐺𝐿!"#$%&and𝐹 !"#$%&

!!! arefunctionsofeGridsubregionasdenotedbytheirsubscript.

13

WeuseEPA’sPowerProfilerspreadsheettooltomapU.S.zipcodetoeGRIDsubregion(EPA,2015).Inaminorityofcases(2801outof41335),azipcodeiscoveredbymultipleutilitiesbelongingtodifferenteGRIDsubregions.Inthesecases,weusethelisted“PrimaryeGRIDSubregion”assumingthisrepresentsamajorityoftheelectricitysuppliedtothezipcode.

3. ResultsServersbydatacenterspacetypeInTable3,wereportgeneralstatisticsforsmallandmidsizedatacentersfoundinCBECS.Wefindapproximately3.7millionserversresideinsmalldatacenterscorrespondingto72%ofinstalledstockinCBECSbuildings.Smalldatacentersarereportedin31%ofallcommercialbuildingsinCBECSandin99%ofbuildingswithservers.Table3:SummaryoftheNumberofDatacentersandServersinCBECS*

DataCenterType

NumberofBuildings(millions)

NumberofServers(millions)

AverageNumber

ofServers

%ofServersinCBECS

NumberofCBECSRecords

Small(1-25servers)

1.725 3.70 2.1 72% 3163

Midsize(26-500servers)

0.021 1.47 69.4 28% 360

*Excludesbuildingsthatcontainmorethan500servers.IDCestimatesthetotalnumberofvolumeserversinstalledin2012acrossdatacentersofalltypestobe13.2million(IDC,2014a).OurestimateofthetotalnumberofserversinsmallandmidsizedatacentersinCBECSis5.2millionservers,implying40%ofserversarefoundinsmallandmidsizedatacenters.Figure1providesadetailedlookatthedistributioninthenumberofserverswithinourdatacenterclassifications.Thetoptwopanelsdisplaythedistributioninthenumberofserversperbuildingrelativetothetotalnumberofserversforsmallandmidsizedatacenters,respectively.Eachdistributionisnormalizedbythetotalnumberofserversinthatdatacenterspacetype.Thebottompanelsdisplaythedistributioninthenumberofserversperbuildingrelativetothenumberofsmallandmidsizedatacenters.Inthesepanels,eachdistributionisnormalizedbythetotalnumberofdatacentersineachdatacenterspacetype.

14

Figure1:Distributioninthenumberofserversbydatacentertype

Thetop-leftpanelshowsthatapproximately30%ofthetotalnumberofserversinsmalldatacentersarefoundinlocationswithonlyoneserver.Datacenterswithasingleservermayrepresentausecasewhereoperatorsneedjustoneserverforfilesharingacrossanofficeorfore-mailorwebhosting.AswithotherresultsfromCBECS,thisresultisdependentonthereportingaccuracyfromthesurveyrespondent.Aninterestingfeatureinthetop-leftpanelofFigure1istheslight,yetsignificant,increaseinfrequencyat10and20servers.Thismaybeareportingbiastowardsavalueroundedtothenearest10orpossiblyanindicationthatserveroperatorsarebiasedtowardsowningserversinincrementsof10servers.Asseeninthebottom-leftpanel,anoverwhelmingmajorityofcommercialbuildingswithsmalldatacentersoperateoneserver(~70%).Thefrequencyofsmalldatacenterswithmultipleserversdropsoffdramatically,withlessthan6%havingmorethan5servers.Thisresultindicatesthatalthoughsingleserverdatacentersonlymakeup30%ofthefractionofserversinsmalldatacenters,suchdatacentersarefarmoreprevalentandrepresentapopularusecaseforbusinesses.

15

Whenlookingatmidsizedatacenters,mostlocationshavelessthan100servers.Thereisanotablerelativedearthofdatacenterswithintherangeof200-500serversindicatingthismaybeanintermediatescaleofinfrastructurethatdoesnotserveapracticalpurposeinbusinessorindustry.CBSAreports36,500datacentersinitssampleofPacificNorthwestcommercialsites.TheCBSAsurveydidnotrecordthenumberofserversineachdatacenter,insteadcountingthenumberofserverracksinoperation(seesection2fordetails).ThisdifferencefromCBECS,whichrecordsthenumberofservers,likelyindicatesthatCBSAmaybeexcludinglocationsthatoperatewithonlyafewservers.Asdiscussedpreviously,ourCBECSanalysisshowsthatsingle-serveroperationisamajorusecase.Figure2showsthedistributionofdatacentersasafunctionofthereportednumberofoperationalserverracks.Themajorityofdatacentersreporthavingbetween1-5racks(84%),withoverhalfhavingasinglerack(55%).Detailsregardingthedimensionsoftherackwerenotrecorded.Dependingonthesizeandtypeofrack,anindividualrackcouldholdanywherefrom1serverupto40serversmakingitdifficulttotranslaterackcounttoservercountforacomparisontoresultsfromCBECS.However,theseresultsqualitativelysupporttheassertionthatthemajorityofdatacentersarerelativelysmallinsize.

Figure2:DistributioninthenumberofracksinCBSAdatacenters

Notably,1%ofidentifieddatacenters(correspondingto11records)donotreporthavinganyoperationalracks.Thisleadstoquestionsregardinghowhardwareisoperatedandspaceconditioned.Thesesitesmaybeoperatinghardwareinadhocspacesnotintendedforserveroperationandmaybenefitfromsimpleefficiencymeasures.

16

CBSAclassifieseachsiteaseitherurbanorruralbasedontheRural-UrbanContinuumCodeofthesitecounty.4Approximately33%ofsurveyeddatacentersarefoundinrurallocations.Figure3showstheurbanandruralbreakdownforeachrackcategory.Ofdatacentersthatreported0racks,59%arefoundinrurallocations,significantlymorethantheoverallbreakdown.Forotherdatacentersthatreportedhavingarack,thefractionofresponsesinrurallocationsisroughly33%,asexpectedfromtheoverallbreakdownbetweenurbanandrurallocations.Ruralareashavealargerfractionofdatacenterswith0rackscomparedtourbanareas.Theskewtowardssmaller,lessefficient(intermsofPUE)datacentersinruralareaspotentiallyindicatesanopportunityforenergysavings.

Figure3:UrbanandruralbreakdownbyracksizeinCBSAdata

ServersbyCensusDivisionInthissectionwereportgeographicresultsfromdatacentersinCBECSaggregatedbycensusdivision.Table4showsthepercentageofserversineachcensusdivisionfortheconsidereddatacentertypes.Wealsoshowthepercentageofcommercialbuildingsineachcensusdivision(CBECSWeight)forcomparison.Table4:GeographicdistributionofdatacentersbyCensusDivision.

CensusDivision SmallDatacenters

MidsizeDatacenters

CBECSWeight

NewEngland 5.8% 2.7% 5.4% MiddleAtlantic 9.2% 19% 9.1% EastNorthCentral 13% 12% 13% WestNorthCentral 6.7% 4.8% 9.0% 4Formoredetails,seehttp://www.ers.usda.gov/data-products/rural-urban-continuum-codes.aspx

17

SouthAtlantic 21% 8.2% 20% EastNorthCentral 5.6% 7.3% 6.7% WestSouthCentral 13% 16% 14% Mountain 6.1% 10% 6.1% Pacific 20% 20% 17% ThespatialdistributionofsmalldatacentersgenerallytracksthedistributionofcommercialbuildingsinCBECSwithafewnotableexceptions.WefindthattherearemoresmalldatacentersintheSouthAtlanticandPacificrelativetothenumberofcommercialbuildings.TherearefewersmalldatacenterserversinWestSouthCentralandEastNorthCentralrelativetonumberofcommercialbuildings.Overall,thesimilarityinspatialdistributionsunderscorestheprevalenceofsmalldatacentershousedwithincommercialspacesacrosstheU.S.Contrarytoourfindingsforsmalldatacenters,thespatialdistributionofmidsizedatacentersdifferssignificantlyfromthatofcommercialbuildings.MidsizedatacentersaresignificantlyoverrepresentedintheMid-AtlanticandMountainrelativetothenumberofcommercialbuildingsintheseregions.Giventhatmidsizedatacentersarefoundlessfrequentlythansmalldatacenters(asshowninTable1),midsizedatacentersmayonlybeutilizedbycertainindustriesthatrequiresignificantcomputingpoweranddonotnecessarilytrackthegeographicdistributionofallCBECsbuildings.

ServersbyPrincipleBuildingActivityInthissectionweinvestigatehowserversaredistributedbyCBECSbuildingactivity.CBECSprovidesaprincipalbuildingactivitycodeandamorespecificbuildingactivitycode.Inouranalysis,weanalyzedresultsbybothPBAandspecificPBA.Fortheresultspresentedinthissection,wefocusourattentiononresultsbyPBAaslittlemeaningfulinformationisgainedbythefinergradationprovidedbythespecificbuildingactivitycode.However,whennoteworthy,wediscussresultsbyspecificbuildingactivity.Table5showsthepercentageofserversbyPBAforeachdatacentertype.Althoughsmalldatacentersarefoundinadiversevarietyofsettings,thepluralityofserversassociatedwithsmalldatacentersarefoundinofficebuildings.Thisisunsurprisinggiventheutilitythatserversprovidewithinofficesettingssuchasfilestorageacrossanetwork,accesstodatabases,andrunningsharedenterpriseapplications.LookingatspecificPBAswithinofficebuildings,wefindthat26%ofserversinsmalldatacentersarefoundinadministrativeoffices.Similartosmalldatacenters,serversinmidsizedatacentersareprimarilyfoundinofficebuildings,specificallywithinadministrativeoffices(38%).Medicalbuildingsalsohouseasignificantnumberofserverswithinlargedatacenters,withmostassociatedwithInpatientCare(12%).

18

Table5:Percentageofserversbybuildingactivityforeachconsidereddatacentertype

BuildingActivity

SmallDataCenters

MidsizeDataCenters

Education 9.8% 7.4%FoodSales 1.8% 0.0%FoodService 3.0% 0.0%Lodging 1.2% 0.9%Medical 6.9% 22%Office 43% 57%PublicAssembly 3.3% 5.4%PublicOrder 2.1% 1.7%ReligiousWorship 2.0% 0.0%Retail 13% 0.1%Service 4.4% 0.2%Warehouse 8.2% 5.4%Vacant 0.71% 0.0%Other 1.4% 0.3%Total 100% 100%AnotherwayofinvestigatingtheprevalenceofdatacentersisbylookingatthemarketsaturationofdatacentersineachPBA.Foragivenbuildingactivity,thedatacentersaturationisdefinedastheratioofthenumberofbuildingsofthatactivitywithadatacentertothetotalnumberofbuildingsofthatactivity. Table6showssaturationratesofsmallandmidsizedatacentersacrossPBA.Forsmalldatacenters,saturationratesrangefrom14%forreligiousworshipupto59%formedicalcarefacilities(notincludingadministrativemedicaloffices)withtypicalvaluesbetween20-30%.Theseresultsshowthatsmalldatacentersarerelativelycommoninawiderangeofindustries.Again,wefindthatsmalldatacentersareespeciallyprevalentinofficesettings.Notethatwefindhighsaturationratesformedical(56%),courthouses(94%),andgovernmentbuildings(55%),whichhouseactivitieswherefederalprivacylawsmayimpedeadoptionofoff-sitecolocation,highlightingtheneedtocustomizeefficiencystrategiesforthesesmalldatacenters.Forexample,privacylawmaypreventmedicalcompaniesfrommovingtheirin-housedatacentertoacloud-basedsystemoperatingfromalarger,moreefficienthyperscaledatacenter.Insuchcases,therearepotentiallymethodsofleveragingthecomputationalpowerofhyperscaledatacentersthatinteractwithon-siteprotecteddata,whichcouldyieldsignificantenergysavings.Bycontrast,highsaturationratesforprivatebusinessesinofficeandretailspacesmayprovideanopportunityforsubstantialenergysavingsviacloud-basedsolutionswithouthavingtoaddressasmanyobstaclesregardingprivacy.

19

Table6:Serversaturationbybuildingactivityforeachconsidereddatacentertype

BuildingActivity SmallDataCenters

MidsizeDataCenters

Education 38% 0.54%FoodSales 31% 0.0%FoodService 22% 0.0%Lodging 22% 0.21%Medical 59% 2.2%Office 56% 1.1%PublicAssembly 20% 0.21%PublicOrder 28% 0.55%ReligiousWorship 14% 0.0%Retail 40% 0.0%Service 19% 0.0%Warehouse 21% 0.24%Vacant 3.0% 0.0%Other 32% 0.0%

SpaceCoolinginDatacentersTheCBSAsurveyallowsustoanalyzeHVACequipmentasafunctionofspacetypeandnumberofserverracks.AlthoughthesedataareweightedtoberepresentativeofthePacificNorthwest,wenotethatthedatamaynotreflectspaceconditioningacrossothergeographicregions.DifferentgeographicregionswillhavedifferentspacecoolingrequirementswhendeterminingHVACneedsforthebuilding.ThePacificNorthwestcovers‘Marine’and‘Cold’climatezonesasdefinedbytheBuildingAmericaprogram(PacificNorthwestNationalLaboratory,2015),sponsoredbytheU.S.DepartmentofEnergy’sOfficeofEnergyEfficiencyandRenewableEnergy(EERE).Buildingsinthesezonesgenerallyhavelowercoolingloadscomparedtohighertemperatureclimatezones,whichwillbefactoredintoHVACsystemdesigns.Thiscouldpotentiallyimpactspaceconditioningfordatacentersites,particularlythosethatutilizeeconomizersforcooling.

20

Figure4:SpacecoolingcharacteristicsofdatacentersintheCBSA

ThetoppanelsofFigure4displayresultsfordatacenterswithdedicatedairconditioning.Whencategorizingdatacentersbysquarefootage(leftpanel),wefindthatamajorityofdatacenterslessthan101squarefeetdonothavededicatedspaceconditioning,relyingoncoolingsystemsthatservicealargerpartofthebuildingornocoolingsystem.Surprisingly,datacentersinthe101-1000squarefootrangehavealargerproportionofdedicatedcooling(79%)comparedtodatacenterslargerthan1000squarefeet(62%),althoughwe

21

notethatthereareonly13recordsofdatacenterslargerthan1000squarefeetwithnodedicatedcooling.Furtherinvestigationfindsthatonlyonerecordofthe13reportshavingnotypeofHVACsystem.Inthetop,rightpanel,wefindthattheproportionofdatacenterswithoutdedicatedcoolingdecreasesasthenumberofoperationalserverracksincreases.ThebottompanelsshowtheproportionofdatacentersthatusedifferenttypesofHVACsystems.Notethatthesesystemsdonotnecessarilycorrespondtodedicatedconditioningforonlytheserverspaceandmayalsobeusedforconditioningotherpartsofthebuilding.Theleftpanelshowsthefractionofresponsesbysquarefootageandtherightpanelbynumberofoperationalserverracks.Amajorityofdatacenterslessthan1000squarefeetuseair-cooledcomputerroomairconditioners(CRACs),whicharelabeled‘AirDX’inthefigure.Asizablefractionofdatacenterslessthan1000squarefeetrelyonbuildingair-handlingunits(labeled‘BuildingTransfer’).Datacenterslessthan101squarefeethavethelargestfractionofrespondentsthatusenoconditioning(27%).Datacenterslargerthan1000squarefeetdatacentersrelymostlyonwater-cooleddirectexpansionsystems(42%).Thebottom,rightpanelshowssimilarresultsbynumberofoperationalserverracks.Over80%ofdatacenterswith0racksrepliedtheyhavenoHVACsystem.Curiously,wefind67recordsfordatacentersthatreporthavingeitheranair-cooledorwater-cooledCRAC,butalsoreporteitherhaving‘unknown’ornodedicatedairconditioning.ThisissurprisinggiventhatCRACsaredesignedspecificallyforuseindatacentersandwouldbeillsuitedforgeneralbuildingcooling.ItisunclearwhetherrespondentsansweredthequestionsincorrectlyincludingothertypesofcoolingsystemswithCRACsorifthereareofficeswhereCRACsarealsousedmoregenerallyinabuilding.

ServerVirtualizationServervirtualizationallowsasinglephysicalservertoactasmultiplevirtualmachinesrunningindependenttasksandapplications,ratherthandedicateindividualserversforeachapplication.Bycreatingvirtualmachines,datacenteroperatorscanconsolidatehardwareandmorefullyutilizeseverprocessingpower.However,previousresearchonsmallbusinessdatacenterpracticesindicatesthatservervirtualizationisnotheavilypracticed.BennettandDelforge(2012)foundonly37%ofsmallbusinessespracticeservervirtualizationinasurveyperformedin2011.Theauthorsnotethatmanysmallbusinesseshadneverheardofservervirtualizationordidnotthinktheinvestmentinvirtualizationwouldbecost-effective.Asurveyoflarge-scaleenterprisesin2011foundthatapproximately92%ofbusinessesutilizedvirtualization(VansonBourne,2011)andamorerecentsurveyin2016foundthat75%ofbusinessesofallsizesutilizedvirtualization(Spiceworks,n.d.).TheCBSAstudyrecordedthedegreeofvirtualizationundertakenbyITstaffforeachdatacenter.Unfortunately,itisunclearwhetherthedegreeofvirtualizationreferstothenumberofserversthatarevirtualserversorthenumberofphysicalserversthatcontainvirtualizationsoftware.Notably,mostsitesdidnotprovidearesponse.Giventhelackofvirtualizationinsmallbusinesseswithvirtualizationknowledge(BennettandDelforge,2012),itislikelythatmostofthesesitesdonotmakeuseofdatavirtualization.However,theremaybesomesiteswheretherespondentmaynothavebeenanITprofessional

22

capableofproperlyansweringthequestion.Giventheuncertainty,unknownresponsesareseparatelyidentifiedinourreporting.InFigure5,weshowresultsforthereporteddegreeofvirtualizationbysquarefootage.Datacenterslessthan1000squarefeetshowahighproportionofunknownandnovirtualizationresponses.Only8%ofdatacenterslessthan101squarefeethaveanydegreeofvirtualization,with7%reportingmorethan50%virtualization.Approximately15%ofdatacentersoperatewithservervirtualization.Datacenterslargerthan1000squarefeetoperatewiththehighestproportionofservervirtualizationwith31%.Wewouldexpectthatlargercentershavemoreopportunitytoconsolidate,sincetheywouldberunningalargernumberofapplications,andwouldbenefitthemostfromvirtualization.

Figure5:Degreeofservervirtualizationbydatacenterspacetype

4. DiscussionSpatialDistributionofServersInthissectionwecombinedatafromCBECSandOEStoconstructspatialmapsofserverintensity(i.e.,serversperunitarea).AsdiscussedinSection2,occupationstatisticsatthezipcodelevelareusedasaproxytoestimatethespatialdistributionofservers.Figure6showsthecombinedgeographicserverintensityforbothsmallandmidsizedatacenters.Serverintensitycloselytrackspopulationcenterswithhighintensitiesinurbanareas.Thisisunsurprisinggiventhatmostjobsthatwouldutilizeaserverarefoundincities.

23

MapsofCO2emissionintensitiesdisplayedaverysimilardistributiontowhatwefindforserverintensity.TheyarenotincludedherebecausedifferencesbetweentheserverintensityandCO2emissionmapswerehardtodiscern.However,itisworthmentioningthattherearesubtledifferencesbetweenthegeographicdistributionforserverandcarbondioxideemissionintensity.Forexample,thePacificNorthwesthasloweremissionsperserverduetotheregion’srelianceonhydropower.TheDenvermetropolitanregioninColoradohashigheremissionsperserverduetotheregion’srelianceoncoal.

Figure6:Geographicdistributionofservers

Inthenextsection,wewillcombineourderivedspatialdistributionofserverswithtypicalpowerconsumptionvaluesandgeographic-specificwithcarbonemissionfactorsfromeGridtoestimateregionalcarbondioxideemissions.

NationalEnergyUseandCarbonDioxideEmissionsInthissection,weaggregateourresultstoestimatetheannualenergyusedueonlytoservers,totalsiteenergyuseincludingenergyusedtomaintainandoperatethedatacenter,andtotalCO2emissionsincludinggrossgridlossesfortheconsidereddatacenters.ResultsforsmallandmidsizedatacenterscanbefoundinTable7.

Table7:AggregateannualenergyconsumptionandCO2emissionsbydatacentertype

DataCenterType

NumberofServers(millions)

ServerEnergyUse(billionkWh)

TotalEnergyUse(billionkWh)

TotalCO2Emissions(MMT)

Small 3.70 5.83 12.9 6.71Midsize 1.47 2.32 6.34 3.29Total 5.17 8.15 19.2 10.0

24

Inaggregate,smallandmidsizedatacentersuseapproximately20billionkWhandemit10millionmetrictonsofCO2peryear,equivalenttotheannualelectricityuseof1.5millionhomes.5Theaverageserverinasmalldatacenterisresponsiblefor1.81metrictonsofCO2peryearandinamidsizedatacentertheaverageserverisresponsiblefor2.23metrictonsofCO2peryear.

ComparisontoIDCMarketReportMarketresearchfirmIDCauthoreda2014studyanalyzingthestateofthedatacenterindustry(IDC,2014b).ThestudyprovidesacensusofUSdatacentersbasedonin-depthinterviewswithITbuyersandsuppliersincludingvendorsthatmanufacturepower,cooling,server,storage,andnetworkingequipment.Thestudypresentshistoricaldataandprojectionsofthedatacentermarketbasedonunderlyingindustrytrends.WeusedatapresentedintheIDCstudytocomparetotheestimatesderivedfromCBECS.IDCreportsthenumberofdatacentersbyspacetypeusingataxonomythatcapturestherelativelybroaddefinitionofdatacenter.Fromsmallesttolargest,theirtaxonomyincludescloset,room,localizedinternal,mid-tierinternal,andhigh-endinternaldatacenters.AsdescribedinSection2,thetaxonomyusedinthisworkcombinesclosetandroomdatacentersinto“small”andlocalizedinternalandmid-tierinternalinto“midsize”.InTable8,wecomparethenumberofdatacentersreportedbyIDCandourestimatesfromCBECS.Therearelargediscrepanciesbetweenthetwodatasets.IDCreportsnearlyonemillionmoresmalldatacenterscomparedtothenumberofbuildingsclassifiedasasmalldatacenterinCBECS.Formidsizedatacenters,CBECSreportsroughlyaquarterofdatacentersfoundbyIDC.Table4:SummaryofcomparisonindatacentercountsbetweenIDCandthiswork

Withinsmalldatacenters,IDCreportsaroughlyevensplitbetweenclosetandroomdatacenters.However,assumingbuildingsinCBECSwithfewerthan5serverscorrespondtoaclosetdatacenter,wefindthat91%ofsmalldatacentersareclosets.Theskewtowards

5UsingEPA’sgreenhousegasequivalencycalculator:https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator

IDCTaxonomy IDCDataCenterCount

Taxonomy(thiswork)

IDCDataCenterCount

CBECS(thiswork)

Closet 1,512,829 Small 2,846,125 1,724,783Room 1,333,296Localized 70,419 Midsize 80,383 21,203

Mid-tier 9,964

High-end 8,283 NotAnalyzed

8,283 NotAnalyzed

25

closetdatacenterscanbeseeninthebottomleftpanelofFigure1whichshowsthevastmajorityofsmalldatacentershaveonly1server.IfweapplytheIDCdefinitionofclosetdatacentertoCBECSbuildings(i.e.,adatacenterwithlessthan5servers),weestimate1.56millionclosetdatacenters.ThisvalueisveryclosetoIDC’sreportedestimateof1.51milliondatacenters.However,asinthecaseofmidsizeandhigh-enddatacenters,CBECSreportssignificantlyfewerroomdatacentersbyapproximatelyafactorof8.IDC’srangeinthenumberofserversisintendedasaroughguidetocategorizedatacentersandnotasastrictdefinition.However,itisdifficulttoseehowmisalignedclassificationsofdatacentersinCBECScouldpotentiallyexplainthedifferenceforlargeandhigh-enddatacentersbetweenthetwodatasets.Forthesedatacenters,CBECSreportsroughlyaquarterfewerdatacenters.OnepossibleexplanationforthedifferenceisCBECSgivesacountbybuilding,whereasIDCisestimatingthenumberofdatacenters.Thatis,abuildinginCBECSwithmultipledatacenterswouldbeaggregatedandrecordedasasinglerecord.AsCBECSgivesestimatesofthetotalnumberofserversineachbuilding,wehavenowayofinferringthetruenumberofdatacentersinbuildingswithmultipleseparatedatacenters.ThesystematicdifferencescouldpotentiallyberelatedtodifferencesinsurveymethodologybetweenIDCandCBECS.IDCreliesoninformationfromtheITindustrytodevelopestimatesofdatacenters,whereasCBECSrespondentsarecommercialbuildingownersandoperators.However,notethatsinceourenergyuseandCO2calculationsrelyonthenumberofservers,ourestimatesarenotsensitivetothisdiscrepancyinthenumberofdatacenters.InadditiontotheIDCdatacenterreport,wealsoobtaineddataoninstalledserverstock(IDC,2014a).IDCestimatesapproximately13.4millionserversin2012.FromCBECSdata,weestimate5.17millionserversinstalledinsmallandmidsizedatacentersimplyingtheseembeddeddatacenterspacetypesaccountforapproximately40%ofserverstock.Thelargefractionofserversinsmallandmidsizedatacentersagainunderscorestheneedtobetterunderstandthesetypesofdatacenterstoachieveenergysavingspotential.

EnergySavingOpportunities

ServerVirtualizationResultsfromtheCBSA2014showthatservervirtualizationremainsmostlyunderutilizedinsmallandmidsizedatacenters.Similarly,BennettandDelforge(2012)report26%servervirtualizationsaturationinsmallbusinessessurveyedin2011.OuranalysisofCBSAdatainSection3foundapproximately10%ofdatacentersutilizedsomelevelofservervirtualization.Theselowestimatesofvirtualizationindicatethereissignificantavailableenergysavingspotentialfromincreasingsaturation.BennettandDelforge(2012)estimatethatdatacenterswith10ormoreserverswouldbenefitfromvirtualization.Asascenario,weassumethat50%ofdatacenterswith10ormoreserversinCBECSareabletoreachvirtualizationratioof5to1,reportedasan

26

“average”usecasebyWSPandtheNationalResourceDefenseCouncil(2012).Inthisscenario,weestimateannualsiteenergysavingsof3.8billionkWh,correspondingto2millionmetrictonsofCO2emissions.Therearemanybarrierstotheadoptionofvirtualizedserversinsmallbusinesses.Inmanycases,businessespayaflatfeeforenergyserviceseliminatingenergycostsasanincentivetoreduceenergydemand.OrganizationbarriersmayincludelackofcapitaltoupgradeandlackofITknowledge.Additionally,virtualizationwillnotbenefitsmalldatacentersthatoperatefewserverswithlittleroomtoeliminateexcesshardware.

CloudComputingTheriseofcloudcomputingoverthepastdecadehasprovidedanewmeansforbusinessestomeetITneeds.TheavailabilityofSoftwareasaService(SaaS),InfrastructureasaService(IaaS),andPlatformasaService(PaaS)cloudsolutionshasmovedcomputingdemandawayfromsmall,on-premisedatacentersintohighlyoptimizedcloud-computingdatacenters.Inasmalldatacenter,individualserversareoftenusedforrunningindividualapplicationsortasks.Thisleadstoserversthat,onaverage,operateatverylowutilizationlevels.Inlarge,hyper-scaledatacenters,applicationsandtasksaredistributedacrossahugenumberofserversandoperationalizedtorunatutilizationlevelscloserto50%(Shehabietal.,2016).Cloud-computingdatacentersbenefitfromeconomiesofscaleandareabletomakeupfrontinfrastructureinvestmentstosupportefficientdatacenteroperations(WSPEnvironment&EnergyandNaturalResourcesDefenseCouncil,2012).Giventhelargecostofoperatingthousandsofservers,evensmallimprovementsinefficiencycanleadtosignificantoperatingcostsavings.TheUptimeInstitute(2014)estimatedanaveragePUEof1.7fordatacentersthatself-reportedthePUEfortheirlargestdatacenter.SomecloudcomputingdatacentershaveachievedevenlowerPUEs.Forexample,Googlehasreachedafleet-wideaveragePUEof1.12(Google,2016).Pushingthelimitsevenfurther,Gao(2014)presentsaframeworkforapplyingmachine-learningartificialintelligencetooperationaldatatopotentiallydecreaseGoogledatacenterPUEsbelow1.10.Forcomparison,smalldatacentershavetypicalPUEscloserto2.0(Cheungetal.,2014;Shehabietal.,2016).Hereweperformasimpleanalysistoapproximatethepotentialsavingsfrommovingcomputingneedsawayfromasmalldatacentertoacloud-computingdatacenter.Weassumethatserverloadsfromprivate-runofficeandretailstorescanbeeasilyshiftedfromin-housedatacenterstocloudcomputingdatacenters.Shehabietal.(2016)estimateacloud-computingserverrunningundertypicalconditionsconsumesapproximately360W.However,movinganapplicationortaskfromasmalldatacentertoacloud-computingserverwillonlyconsumeamarginalfractionoftheserver’scomputingpower.Previousstudiesestimatethattheloadofasmalldatacenterserverisapproximatelyequivalentto1/5theloadofhigh-endcloudserver(Masanet,2014;Shehabietal.,2016).Ifweassumeconservativelya10%reductioninserversfromindustriesotherthanfinancial,health,andgovernmentrelatedfields,weestimateapotential1.2billionkWhreductioninenergyconsumptionand0.85MMTofCO2emissionsperyear.

27

Asurveyofbusinessownerswithsmalldatacentersfoundthatformanyowners,amainbarriertocloudadoptionstemsfromprivacyandsecurityconcerns(BennettandDelforge,2012).However,theseareconcernsthatarenotnecessarilyrootedinreality.Cloudcomputingdatacentersoftentakemoremeasurestoensurecustomerdataismaintainedsecurelyanddedicatestaffresourcesforthispurpose.Additionally,manycloudcomputingoperatorsofferprivatecloudsolutionsforcompaniesthatwishtosilotheirdatafrompubliccloudresources.IncreasingpenetrationofclouddeploymentwithinsmallerdatacenterswilltakeaconcertedefforttoeducateITprofessionals.Althoughcloudcomputingoffersenergysavingspotentialbyshiftingloadsawayfromsmalldatacenterstomoreefficienthyperscaledatacenters,easyaccesstocloudservicescouldleadtoincreaseddemandforcomputingneedsthatwerenotneededpreviously.Forexample,businessesthatpreviouslyreliedonpaperrecordsthatmovetocloudserviceswilladdtooverallenergydemandorstartprovidingservicesthatrequirecloud-computingresources.Thesebusinesseswilllikelybenefitfromtheefficiencyofdigitizingrecordsandonlinemanagement.Forsomenewusers,thecloudoffersaneaseofsetupandmaintenanceinthatitdoesnotrequireadedicatedITprofessionaltoinstallanewsystem.

5. ConclusionTargetingsmallerdatacentersremainschallengingduetotheadhocnatureofsmalldatacenters,whichareoftensituatedinconfinedspaceswithinlargerbuildings.WemakeuseofrecentlyreleaseddataonserversanddatacentersincommercialbuildingstocharacterizethesmallandmidsizedatacentermarketandconstructspatialmapsoftheirlocationsacrosstheUnitedStates.Wecategorizebuildingswithlessthan25serversassmalldatacentersandbuildingswithmorethan25butfewerthan500datacentersasmidsizedatacenters.Wefindapproximately5millionservers(approximately40%ofinstalledstock)insmallandmidsizedatacenters,consistentwithpreviousestimates(Baileyetal.,2007;DelforgeandWhitney,2014).Thevastmajorityofserversinsmall&mediumdatacentersarefoundinsmalldatacenters.Amajorityofsmalldatacentersarefoundinofficesandretailbuildings.Thehighestsaturationsareinmedical,retail,office,andeducationsectors.Wefindthatapproximately26%ofserversinsmalldatacentersarefoundinadministrativeoffices.BasedondatafromtheCBSAconductedinthePacificNorthwest,amajorityofsmalldatacenterswithmorethanoneoperationalrackmakeuseofair-cooleddirectexpansionCRACsforspaceconditioning.ThemajorityofdatacenterswithoneorlessoperationalserverrackdidnotreportusinganyspecializedcoolingsystemandarelikelyrelyingonthebuildingHVACsystem.Siteswithtwoormoreoperationrackstendtorelyondedicatedspacecoolingwithair-cooledCRACsbeingthemostcommon.Water-cooledCRACsaremainlyusedindatacentersthatarelargerthan1000squarefeet,makingupapproximately40%ofcoolingunitsinthesesites.

28

Inaggregate,wefindthatsmallandmidsizedatacentersconsumeapproximately20billionkWhofsiteenergy.UsingdatafromtheBureauofLaborofStatistics,wecorrelatethenumberofserversfoundintheCBECSdatasetstoemploymentdatawithinoccupationalfieldsidentifiedasusingdatacenters.ThisallowsustodisaggregateresultsbyzipcodetocreatedetailedgeospatialmapsofserverintensityacrosstheUS.Unsurprisingly,highestserverintensitiesarelocatedindensemetropolitanareas,withhighestconcentrationsontheEastandWestCoasts.Intotal,smallandmidsizedatacentersemitover10MMTonCO2emissionsannually.Servervirtualizationprovidesameansforanindividualservertoactasmultiplemachinesallowingforserverconsolidation.Althoughapracticalmeanstodecreasedatacenterenergyconsumption,wefindthatitisnotacommonpracticeinsmalldatacenters.Thevastmajorityofdatacentersrespondedthattheydidnotemployvirtualizationordidnotknowthedegreeofvirtualizationwithintheirdatacenter.Only15%ofsmalldatacentersreportedhavinganylevelofvirtualizationwithintheirfleet.Modestconsolidationofhardwarethroughvirtualizationindatacenterswithmorethan10serverscouldsave3.8billionkWhannually,correspondingto2millionmetrictonsofCO2.Themostchallengingbarriertoadoptionofvirtualizationisnottechnical.Mostsmallbusinesseshaveeitherneverheardofservervirtualizationordidnotthinktheinvestmentinvirtualizationwouldbecost-effective(BennettandDelforge,2012).Theeraofcloudcomputingrepresentsanopportunitytoshiftresourcesfrominefficientsmalldatacenterstohighlyoptimizedhyper-scaledatacenters.Although,thereareindustrieswhereprivacyregulationsmaypreventtheuseofthirdpartyclouddatacenters,suchasinhealthcare,manyindustriesthatcurrentlyrelyonsmalldatacentersshouldbeabletomovecomputingtocloudcenters.Assumingofficesotherthanfinancial,healthcare,andgovernmentofficescanshift10%oftheirloadtocloud-computingresources,wecalculateapproximateenergysavingsof1.2billionkWhperyear.Despiteincreasingconsolidationofsmalldatacentersintolargerones,thereremainasizablenumberofsmallandmidsizedatacenters.Thisreporttakesafirststeptowardsidentifyingandcharacterizinghowthesecentersoperate.However,asdiscussedthroughoutthisreport,amainbarrierimpedingefficientoperationislackofeducationaboutbestpracticesandopportunities.Futureprogresswillbemadebymeasuresandpoliciesthatareabletodirectlytargetsmalldatacenters.

29

ReferencesBailey,M.,Eastwood,M.,Grieser,L.,Turner,V.,Gray,R.C.,2007.IDCSpecialStudy:Data

CenteroftheFuture(No.IDC#06C4799).InternationalDataCorporation(IDC).Bennett,D.,Delforge,P.,2012.SmallServerRooms,BigEnergySavings:Opportunitiesand

BarrierstoEnergyEfficiencyontheSmallServerRoomMarket.NaturalResourcesDefenseCouncil.

BLS,2015.OccupationalEmploymentStatisticsHomePage[WWWDocument].URLhttps://www.bls.gov/oes/(accessed6.27.17).

Brown,R.,Energy,A.toS.,Incorporated,I.C.F.,Incorporated,E.R.G.,Agency,U.S.E.P.,2008.ReporttoCongressonServerandDataCenterEnergyEfficiency:PublicLaw109-431.LawrenceBerkeleyNatl.Lab.

CenterofExpertiseforEnergyEfficiencyinDataCenters,n.d.CenterofExpertiseforEnergyEfficiencyinDataCenters[WWWDocument].URLhttps://datacenters.lbl.gov/(accessed3.27.17).

Cheung,I.,Greenberg,S.,Mahdavi,R.,Brown,R.,Tschudi,W.,2014.EnergyEfficiencyinSmallServerRooms:FieldSurveysandFindings.Presentedatthe2014ACEEESummerStudyonEnergyEfficiencyinBuildings.

Delforge,P.,Whitney,J.,2014.DataCenterEfficiencyAssessment-ScalingUpEnergyEfficiencyAcrosstheDataCenterIndustry-EvaluatingKeyDriversandBarriers.NaturalResourcesDefenseCouncil.

EIA,2015.EnergyInformationAdministration(EIA)-CommercialBuildingsEnergyConsumptionSurvey(CBECS)Data[WWWDocument].URLhttps://www.eia.gov/consumption/commercial/data/2012/index.php?view=microdata(accessed6.27.17).

EIA,n.d.Officebuildingswithdatacentersusesignificantlymoreelectricitythanotheroffices-TodayinEnergy-U.S.EnergyInformationAdministration(EIA)[WWWDocument].URLhttps://www.eia.gov/todayinenergy/detail.php?id=28232(accessed5.8.17).

EPA,2015.PowerProfilerZipcodeTool[WWWDocument].URLfromhttp://www2.epa.gov/sites/production/files/2015-10/power_profiler_zipcode_tool_2012_v6-0.xlsx(accessed10.21.16).

EPA,n.d.Emissions&GenerationResourceIntegratedDatabase(eGRID)[WWWDocument].URLhttps://www.epa.gov/energy/emissions-generation-resource-integrated-database-egrid(accessed5.8.17).

Gao,J.,2014.MachineLearningApplicationsforDataCenterOptimization.Google,2016.Efficiency:Howwedoit–DataCenters–Google[WWWDocument].URL

https://www.google.com/about/datacenters/efficiency/internal/(accessed9.1.16).

Google,2011.Google’sGreenDataCenters:NetworkPOPCaseStudy.IDC,2014a.IDC’sWorldwideQuaterlyServerShipmentTracker,2010-2018.International

DataCorporation(IDC),Framingham,MA.IDC,2014b.U.S.DatacenterCensusandConstruction2014–2018Forecast:Realigning

Workloads,ManagingObsolescence,andLeveragingHyperscale(No.252712).Koomey,J.,2011.Growthindatacenterelectricityuse2005to2010.Rep.Anal.Press

Complet.Req.N.Y.Times9.

30

Koomey,J.,Taylor,J.,2015.Newdatasupportsfindingthat30percentofserversare“Comatose”,indicatingthatnearlyathirdofcapitalinenterprisedatacentersiswasted.

Masanet,E.,2014.TheEnergyEfficiencyPotentialofCloud-BasedSoftware:AU.S.CaseStudy.eScholarship.

Masanet,E.R.,Brown,R.E.,Shehabi,A.,Koomey,J.G.,Nordman,B.,2011.EstimatingtheenergyuseandefficiencypotentialofUSdatacenters.Proc.IEEE99,1440–1453.

NavigantConsulting,2014.2014CommercialBuildingStockAssessment:FinalReport.NorthwestEnergyEfficiencyAlliance.

NorthwestEnergyEfficiencyAlliance,2014.NorthwestEnergyEfficiencyAlliance:2014CBSADatabase[Dataset].Portland,OR.

PacificNorthwestNationalLaboratory,2015.GuidetoDeterminingClimateRegionsbyCounty,BuildingAmericaBestPracties.

Shehabi,A.,Smith,S.J.,Horner,N.,Azevedo,I.,Brown,R.,Koomey,J.,Masanet,E.,Sartor,D.,Herrlin,M.,Lintner,W.,2016.UnitedStatesDataCenterEnergyUsageReport(No.LBNL-1005775).

Spiceworks,n.d.2016LatestTrendsinIT&TechnologyAdoption[WWWDocument].URLhttps://www.spiceworks.com/marketing/resources/reports/2016-state-of-it/(accessed5.8.17).

TheGreenGrid,2012.PUE:AComprehensiveExaminationoftheMetric.UptimeInstitute,2014.2014DataCenterIndustrySurvey[WWWDocument].UptimeInst.

EJournal.URLhttps://journal.uptimeinstitute.com/2014-data-center-industry-survey/(accessed9.9.16).

VansonBourne,2011.VeeamlaunchesV-indextomeasurevirtualizationpenetrationrate[WWWDocument].VeeamSoftw.URLhttps://www.veeam.com/news/veeam-launches-v-index-to-measure-virtualization-penetration-rate.html(accessed9.8.16).

WSPEnvironment&Energy,NaturalResourcesDefenseCouncil,2012.TheCarbonEmissionsofServerComputingforSmall-toMedium-SizedOrganizations:APerformanceStudyofOn-Premisevs.TheCloud.WSPEnvironment&EnergyandNaturalResourcesDefenseCouncil.