MassivelyParallelLiquidCoolingforAISupercomputing

DeepakBoggavarapu,PhDLiquidBrain

deepak@liquidbrain.coolSanFrancisco,CA

Summary:Ournewmicrochannelliquidcoolingsystemdelivers10Xperformanceimprovementoveraircooling.Large12cmx12cmceramicmicrochannelplatesthatareonly1mmthickremove300W/cm2heatfluxwhilestayingbelowchipjunctiontemperaturelimits.Massiveparallelismallows10kWofheatloadtoberemovedinareathesizeofahumanhand.Wesolvecurrentthermalproblemsandenablenewarchitecturesforfuturesystems.

IntroductionFor over 40 years Moore’s Law has propelledcomputing, but in recent years growth hasreachedaplateau.Thisisshowninthefollowinggraph [1] which shows transistor countincreasing inyellow.However frequency is flatoverthelastdecadewhilethenumberofcoresin black has increased. This is due to thermallimits shown in red which limit operatingfrequency. It is simply not possible to removetheheatloadfromthechip.

While Moore’s exponential is slowing, a newcomputing explosion has begun with AI/MLtaking off. Artificial Intelligence (AI) is pushingtheboundariesofcomputingsystemswitheverexpanding models and training sets. The nextphaseofcloudcomputinggrowthisbeingdrivenbyAI.OnestudyshowsAImodelswith300,000Xincrease in compute resources over the last 7years[2].Directliquidchipcoolingcansolvethispressingproblem.

The current state of the art in data centercooling is forced air cooling. The fundamentalproblem with this approach is that HVACsystemsandchilledwater systemsareusedatthe facilities level but fan based air cooling isused in the building and the rack. Fan coolingcanconsumeup to25%extraparasiticpower.Air is inherently a good insulator and impedesthe flow of heat: water is 4000 times moreefficient than air in transferring heat. Directliquid cooling at the chip level significantlyimproves performance and energy efficiency.Withtheultimategoalofcomputingsystemstoreach and exceed human brain levelperformance,liquidcoolingistheonlypossibleapproachintheforseeablefuture.

MicrochannelCoolingHerewedescribeanewmassivelyparallelliquidcooling system that enables large numbers ofCPU’s, GPU’s or other dedicatedAI processingchipstooperateatveryhighpackingdensityandatthermalperformancefarinexcessofcurrenttechnologies.Microchannelcoolinghasbeenusedinlaserchipcooling and solar energy to achieve very highperformance. In this approach, small channelsontheorderofmicronstohundredsofmicronsinsizearefabricatedthatpassliquidascloseaspossibletotheheatsourcetoremoveheat.Thepurpose is to increase surface area of thechannelsasmuchaspossibletoaidheattransfer

while maintaining flow and pressure atacceptablelevels.However,thesesophisticatedcooling systems have not been adopted incomputerchipcooling.SystemsEngineeringMany complex and conflicting engineeringrequirements must be met when designingcoolingsystems.Theseinclude:

1) Choice of materials and choice ofconductor or an insulator. A concernwithmetalsisthepossibilityofgalvaniccorrosion in the system. Insulatorsenableawholenewrangeofdesignandarchitecture options. Materials choiceimpactsthecoolantchoiceaswell.

2) Coefficient of thermal expansion – thecooling system material should matchthe CTE of the silicon chip. CTEmismatchesleadtostructuralfailuresofsolder bonds after many temperaturecycles. Thus matched CTE improvesreliability.

3) Temperature uniformity – the spatialtemperature variation across the chipshould beminimal. Localized hot spotscreatepointsoffailurethatspread.

4) Low fluid pressure drop – the systemshould be designed with minimalpressuredroptoreduceparasiticpowerdraws and improve reliability of seals,materials,andconnections.

5) Lowflowrate–lowflowrateisdesirableand increases the effectiveness of thecooling system. Improved effectivenessresults in higher fluid exit temperaturewhich can allow the extracted ‘wasteheat’ tobeused forotherusefulwork.This also allows the heat exchangerrejectingheattotheenvironmenttobemoreefficient.

6) Lowthermal resistance– the lower thethermalresistanceofthecoolingsystemand the entire heatflow path allowsmoreheattobedissipatedwhilekeepingthe chip junction temperature atspecifiedlevels.

7) Parallelism–parallelflowpathspreventsinglepointsoffailureandcreaterobustandreliablesystems.

8) Large area – chip cooling systemshavetraditionallyaddressedthedesignissuesofasinglechipatatime.Butthefuturewill see new architectures with largearraysofchipsorwaferscaleintegration.

9) Choiceofcoolant–Thecoolantimpactsstorage temperatures, operatingtemperatures, corrosion, and foulingissues.

These are just some of the many contraryconstraints that must be solved as we peerforward to the future of datacenters,supercomputing,andAI.TheSolutionBuilding on the previous work our team hasdone in high power laser diode cooling, solarenergy systems, and complex systemsengineering,wehavedesignedacoolingsystemtomeettheconstraintsmentioned.OursolutionconsistsofaceramicmicrochannelplatethatisCTEmatched to the chip, ismassivelyparallel,largearea,andmuchmore.Themicrochannelceramicplateisdesignedsuchthat each chip receives fresh coolant and ismassively parallel with no serial liquid paths.Within each chip area the liquid flow is alsoparallelsotherenosinglepointsoffailureandnohotspots.Theplatehastensofthousandsofliquid flowchannelsembedded in thematerialandislikelythelargestmonolithicmicrochannelsystemof itskind(figure1and2).Theplate is

12x12cmand1mmthickandcanbeconfiguredformuchlargerareas(30cmx30cm),differentgeometries, chip sizes, etc. The systemcanbeconfiguredforlargenumbersofdiscretechipsorwafer scale large area cooling with 100% fillfactor.Thesystemcandissipate>4kWheatloadwith heat flux of >250 W/cm2 while keepingjunction temp rise below 55C. It is capable ofheat loads exceeding 10kW in this same size.Totalareais144cm2whichisabouttheareaofahumanhand.

Figure1.Largeareamicrochannelplate.12cmx12cm area, 1mm thick and with tens ofthousandsoffluidchannels.

Figure 2. Large areamicrochannel plate (blue)with36chips(black)andinterconnects.Thesemicrochannel systemshavebeen testedin harsh outdoor conditions in highconcentration photovoltaic power systemsunderconditionsmuchmorechallengingthanadatacenter(figure3and4).Thesesystemsweredesigned to produce solar electric powerwithveryhighefficiencyandrecoverthewasteheatwith very high effectiveness. In this case thewasteheatcouldbeusedforwaterdesalination.Recovery and re-use of the waste heat fromdatacentersisanotherwaytohelpdecarbonizedatacenters.

Figure3.Highconcentrationsolarphotovoltaicdishsystemwithmicrochannelliquidcoolinginrugged outdoor environments. The liquidcoolingsystem isat the focalpointof thedishandkeepstriplejunctionsolarcellscool.

Figure4.Testingofmicrochannelliquidcoolingsystem at National Renewable EnergyLaboratorythatsetperformancerecords.We have met all 9 of the design constraintssimultaneously in one product that candramatically improve compute performance.For example an Intel Xeon 8168 has thermaldesignpower(TDP)of205Watts,estimateddiesizeof7cm2forheatfluxof30W/cm2.Atypicalair cooled server with this chip with ambienttemperature of 35C would operate at chipjunction temperature of 85C (delta T of 50C).Withourmicrochannelsystem,thechipwouldoperateat40Cwitha5CdeltaT.WithatypicaldeltaTbudgetof55C,ourmicrochannelsystemallowsthechip tooperateat300W/cm2heatfluxwhichfarsurpassescurrentchipheatflux.The graph below shows the relativeperformanceourcooling technology (figure5).Our first gen technology is10Xbetter thanair

Applicant Name: KriyaLab Control Number: 0670-5388

Page 2 of 4 Contains Confidential, Proprietary, or Privileged Information Exempt from Public Disclosure.

OTu2B.4, march 2012). The use of novel low power optical interconnects with data rates of 100 GB/sec enables >4X reduction in interconnect power consumption.

Figure 1. Fabricated array of large area chips on mounted on monolithic liquid cooling system.

36 high current III-V chips are used in place of CPUs for this test. These boards have been operated to 4kW heat load per board and tested to 250 W/cm2 flux density.

To show the radical changes enabled by our technology we compare to the current state of the art in shipping container based data centers. Google has stated they use shipping containers to house servers and that each container contains 1160 servers that consume a total of 250kW. Hewlett Packard has developed the POD containerized data center. Each POD has 22 racks with ~27kW heat load per rack with a total power consumption per container of 600kW. Each POD contains 2800 servers (2 CPUs per server so 5600 CPUs) with peak compute capacity of ~205 Teraflops. The POD is air cooled. Using the dense array CPU cooling concept outlined above, we can combine 12 boards shown above to hold 432 CPUs that will fit in 2U of rack space. This will generate 40kW of heat that is easily managed with our liquid cooling system. A full rack will hold >5600 CPUs and direct liquid cooling can easily handle this heat load in one rack. Such high density levels are not possible with air cooling. The high speed vcsel optical interconnect system enables the servers to be fully utilized without the latency and switching limitations of conventional interconnects. Project Phases: The overall project is to demonstrate a dense CPU array in 12 months. Phase 1: Design and build optimized liquid cooling system for 36 CPU chip array. Phase 2: Design and build integrated optical interconnect system using high speed VCSEL lasers. Phase 3: Integrate interconnects with CPU array and measure compute performance and energy consumption improvement compared to conventional servers. This phased approach will allow us to systematically address the risks of this technology. Work to date has significantly de-risked the core thermal technology. The fundamental thermal technology has been validated and our previous work in interconnects gives us confidence about this technology. The remaining risks are in the various packaging steps to fabricate the array with interconnects and demonstrating long term operational stability. This project would not be built without ARPA-E funding since it is early stage.

cooling. Our second gen technology will haveanother2ximprovement(LBv2)

Figure 5. Relative performance of ourmicrochannelcoolingtechnology.Our firstgentechnologyis10Xbetterthanaircooling(greenLBv1).Thechartbelowcomparestheheatfluxseveralhighperformancechips.AnIntelserverchiphasfluxof30W/cm2,andNvidiaGPUoperatesat31W/cm2. Recent advances in AI chipdevelopmentofferlargewaferscaleintegrationas one approach. But such wafer level partsoperate at similar average heat flux of 32W/cm2. As mentioned above, our coolingsystemenablesheatfluxesof300W/cm2,a10Ximprovementandoverlargeareas.

Figure6.HeatfluxofhighperformancechipsThe opportunity exists to combine largenumbersofchipswithnovelinterconnectsandachieve performance levels that exceed otherapproaches and with high manufacturingscalability. Heterogeneous integration of largearrays of chips each optimized for specificfunctionality as in figure 2 may far exceedperformance of wafer scale chip fabricationapproaches.

ThePathForwardOur microchannel technology offers a pathforward from solving today’smost challengingthermal problems with proven technology tonewarchitectures thatgetus closer tohumanbrainlikeperformance.-Wecool300W/cm2heatloadswithjunctiontemperaturebelowlimits.A10Ximprovement.- Large arraysof chips cooledwith 10kWheatloadinthesizeofhumanhand.Morethan10Xincreaseinareaandheatflux.- This technology enables new systempossibilities with 1000 GPUs or AI optimizedchipsinavolumeofacubicfoot.-Achievingbrainlikeperformancewillrequirea100,000Xincreaseincomputepowerincomingdecadeswhichwill require novel architecturesandinterconnects.Wecantakeinspirationfrombiologywhereneurons,synapses,andcapillariesare all integrated and intermixed in the brain.Microchannels are the capillaries of electroniccomputing from which new architectures willarise.Optimizedthermaltechnologycomplimentsandmakesanychipapproachbetter.References[1]https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/[2]https://openai.com/blog/ai-and-compute/