16
IHEP(Beijing LCG2) Site Report IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Fazhi.Qi, Gang Chen Computing Center,IHEP Computing Center,IHEP

IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Embed Size (px)

Citation preview

Page 1: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

IHEP(Beijing LCG2) Site ReportIHEP(Beijing LCG2) Site Report

Fazhi.Qi, Gang ChenFazhi.Qi, Gang ChenComputing Center,IHEPComputing Center,IHEP

Page 2: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

OutlineOutline

• InfrastructureInfrastructure

• Local Cluster StatusLocal Cluster Status

• LCG Tire2 Site StatisticsLCG Tire2 Site Statistics

•Management & Operation Management & Operation

• SummarySummary

Chen Gang/CC/IHEP 23/4/21 - 2

Page 3: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

InfrastructureInfrastructure

• Serving more than 1000 usersServing more than 1000 users

• Power supply capacity: 1800Kw Power supply capacity: 1800Kw

Shi,Jingyan/CC/IHEP 23/4/21 3

• Cooling: water Cooling: water cooling rack for cooling rack for the blade serversthe blade servers

Page 4: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

• Water cooling rackWater cooling rack

• Inter-row air conditioningInter-row air conditioning

• Cooling capacity per rack: Cooling capacity per rack:

28kw28kw

Infrastructure UpgradeInfrastructure Upgrade

Shi,Jingyan CC--IHEP 23/4/21 - 4

Power Capacity: 1800kwPower Capacity: 1800kw

Cooling SystemCooling System

Page 5: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Chen Gang/CC/IHEP 23/4/21 5

Local Cluster --Computing Local Cluster --Computing NodesNodes

• Most for BES,YBJ,DYB,Atlas,CMS Most for BES,YBJ,DYB,Atlas,CMS

experimentsexperiments

• Some small projects addedSome small projects added

• Blade system IBM/HP/DellBlade system IBM/HP/Dell

• Blade links with GigE/IBBlade links with GigE/IB

• Chassis links to central switch with 10GigEChassis links to central switch with 10GigE

• 886 computing nodes: 7082 CPU-886 computing nodes: 7082 CPU-

corescores

• Most running SL5.5 (64 bit)Most running SL5.5 (64 bit)

• Intend to migrate to SL5.8Intend to migrate to SL5.8

• A small part stayes in running A small part stayes in running

SL4.5 (32 bit)SL4.5 (32 bit)

Page 6: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

• Torque: 2.5.5Torque: 2.5.5

• Maui: 3.2.6Maui: 3.2.6

• Intend to upgrade to 3.4.4 Intend to upgrade to 3.4.4

or higher to support MPI or higher to support MPI

jobsjobs

• Tools developed to Tools developed to

monitor the resources monitor the resources

usage, queue status usage, queue status

etc.etc.

• Accounting tool Accounting tool

developeddeveloped

Chen Gang/CC/IHEP 23/4/21 - 6

Local Cluster -- Local Cluster -- SchedulerScheduler

Page 7: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Scheduler

• 50 queues to fit various requests

• Besides serial jobs, MPI, GPU jobs are also supported

• Testbed

• Integration of Torque and openstack

• Managing and scheduling VM nodes in batch-cloud

Chen.Gang/CC/IHEP 23/4/21 7

Page 8: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Local Cluster -- StorageLocal Cluster -- Storage

• Gluster system installed Gluster system installed

• Storage provided less than 4 months Storage provided less than 4 months

• Keeps optimizing performanceKeeps optimizing performance

• Adjust to deal with the new bugsAdjust to deal with the new bugs

• Total space: 153TB, Used space: 145TBTotal space: 153TB, Used space: 145TB

Chen Gang/CC--IHEP 23/4/21 - 8

Page 9: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Beijing LCG Tier II SiteBeijing LCG Tier II Site

• For CMS, ATLAS experimentsFor CMS, ATLAS experiments

• 1000+ Job slots1000+ Job slots

• Storage: Storage:

• 320TB dCache 320TB dCache

• 320TB dpm 320TB dpm

• 1T disks were replaced by 2T disks1T disks were replaced by 2T disks

Chen Gang/CC--IHEP 23/4/21 - 9

Page 10: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Beijing LCG Tier II SiteBeijing LCG Tier II Site

• CPU TimeCPU Time

Page 11: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Orient+

Network ConnectionNetwork Connection

DayaBay

BeijingCSTNet

HongKong

IHEP

USA

GLORIAD 10GASGC

IPv4 10G IPv6

BeijingTsinghua

YBJ

EUR.

2.5G

155M

155M

10G

Others

EDU.CN10G

Chen Gang/CC--IHEP 23/4/21 - 11

Page 12: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Perfsonar @IHEPPerfsonar @IHEP• Two hosts for perfsonarTwo hosts for perfsonar

• Perfsonar.ihep.ac.cn for Perfsonar.ihep.ac.cn for

Bandwidth testBandwidth test

• Perfsonar2.ihep.ac.cn for Perfsonar2.ihep.ac.cn for

Latency testLatency test

• Network performance Network performance

tuning is in progress tuning is in progress

between IHEP and EU. Sitesbetween IHEP and EU. Sites

• http://twiki.ihep.ac.cn/twiki/http://twiki.ihep.ac.cn/twiki/

bin/view/bin/view/

InternationalConnectivity/InternationalConnectivity/

IHEP-CCIN2P3IHEP-CCIN2P3

Chen Gang/CC--IHEP 23/4/21 - 12

Page 13: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Network Research (SDN@IHEP)Network Research (SDN@IHEP)• GoalGoal

• A flexible, reliable and high performance HEP data transfer A flexible, reliable and high performance HEP data transfer network (virtual and private) and system platform in Chinanetwork (virtual and private) and system platform in China

• IPv4 and IPv6 supportedIPv4 and IPv6 supported• The traffic can be switched between IPv4 and IPv6 The traffic can be switched between IPv4 and IPv6

infrastructure and physical path automatically or manually infrastructure and physical path automatically or manually based the network performance and applicationsbased the network performance and applications

• SDN@IHEP SDN@IHEP IHEPDTN IHEPDTN

• End user networkEnd user network• Backbone networkBackbone network (( IPv6 & IPv4IPv6 & IPv4 ))• SDN Switch (L2VPN gateway & Openflow supported)SDN Switch (L2VPN gateway & Openflow supported)• Control center (API to Application)Control center (API to Application)• Applications(FTS/NMS/…….)Applications(FTS/NMS/…….)

• MembersMembers

• IHEP/SJU/SDU/TsingHua/……IHEP/SJU/SDU/TsingHua/……• Network VendorNetwork Vendor :: Ruijie NetworksRuijie Networks

Page 14: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

SDN@IHEP modelSDN@IHEP model

Page 15: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

• Most part of computing environment running Most part of computing environment running

wellwell

• New gLuster system is in productionNew gLuster system is in production

• Network performance between IHEP-Eur. got Network performance between IHEP-Eur. got

an clear improvement an clear improvement

• New Management and Operation system will New Management and Operation system will

be deployed to improve the efficiencybe deployed to improve the efficiency

SummarySummary

Chen Gang/CC/IHEP 23/4/21 - 15

Page 16: IHEP(Beijing LCG2) Site Report Fazhi.Qi, Gang Chen Computing Center,IHEP

Thank you!Thank you!

Questions?Questions?

Chen Gang/CC/IHEP 23/4/21 - 16