Southeast University and AMS-02 Experiment · PDF fileSoutheast University and AMS-02 Experiment Fang Dong School of Computer Science and Engineering, Southeast University [email protected]

Southeast University and AMS-02 Experiment

Fang Dong

School of Computer Science and Engineering, Southeast University

[email protected]

The 14th International Conference on Cryptology and Network Security

(CANS 2015)

n  SEU’s contribution to AMS

n  Conclusion and future work

n  Background of AMS

Outline

2

~90% of the matter in the

Universe is not visible, called Dark Matter

Collisions of Dark Matter will yield characteristic distributions

of charged cosmic rays which can be detected by AMS.

The Origin of Dark Matter ~ 90% of Matter in the Universe is not visible and is called Dark Matter

A Galaxy as seen by telescope If we could see Dark Matter in the Galaxy

AMS is the only large-scale physics experiment that directly searches for dark matter in the space

Alpha Magnetic Spectrometer

4

Help scientists find the source of the dark matter and measuring cosmic ray sources as well as antimatter universe

USA A&M FLORIDA UNIV. JOHNS HOPKINS UNIV. MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER UNIV. OF MARYLAND-DEPRT OF PHYSICS UNIV. OF MARYLAND-E.W.S. S.CENTER YALE UNIV. - NEW HAVEN

MEXICO UNAM

DENMARK UNIV. OF AARHUS

FINLAND HELSINKI UNIV. UNIV. OF TURKU

FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE

GERMANY RWTH-I RWTH-III MAX-PLANK INST. UNIV. OF KARLSRUHE

ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA

NETHERLANDS ESA-ESTEC NIKHEF NLR PORTUGAL LAB. OF INSTRUM. LISBON

ROMANIA ISS UNIV. OF BUCHAREST

RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV.

SPAIN CIEMAT - MADRID I.A.C. CANARIAS.

SWITZERLAND ETH-ZURICH UNIV. OF GENEVA

中国 KOREA EWHA KYUNGPOOK NAT.UNIV.

中科院电⼯工所中科院⾼高能所东南⼤大学上海交⼤大中⼭山⼤大学⼭山东⼤大学航天部⼀一院航天部五院中⼭山科学院中央研究院中央⼤大学成功⼤大学太空计划室交通⼤大学

AMS is US Dept of Energy (DOE) led International Collaboration 16 Countries and regions, 60 Institutes and 600 Physicists, 17 years

6 5m x 4m x 3m

7.5 tons Silicon layer

7 Silicon layers

TRD

TOF 1, 2

TOF

RICH

ECAL

Magnet

300,000 electronic channels 650 microprocessors

Radiators 11,000 Photo Sensors

Silicon layer

6

STS-134 launch May 16, 2011 @ 08:56 AM

May 19: AMS installation completed at 5:15 CDT, start taking data 9:35 CDT

Huntsville, AL

Regional SOC Ø Data storage Ø Data reconstruction Ø Monte Carlo production Ø Data analysis

SEU RWTH … CIEMAT Milano

POCC Ø AMS Flight Status Monitoring

POCC Ø AMS Flight Status Monitoring

SOC Ø Data reconstruction Ø Data analysis Ø Data distribution Ø Monte Carlo production

CERN JSC

太空粒⼦子束

Tracking and Data Relay Satellite,TDRS

Marshall Space Flight

Center

KU band

AMS Data Flow

AMS Scientific Operation Center

9

Sim ulation D ata

D ata R econstruction

AMS-02Package

C alibration

D etector Sim ulation

R aw D ataD A Q D ata

R econstruction

GEANTPhysics A nalysis

ROOT

R esults

V isulization

M C Sim ulation

q Data storage、data reconstruction、Monte Carlo simulation、physics analysis、massive data transfer

Missions of AMS-SOC

10




Outline

11

Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment.

AMS 1. AMS-C

2. AMS-AIS

SEU

USA MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV.

- NEW HAVEN

MEXICO UNAM




GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE


NETHERLANDS ESA - ESTEC NIKHEF NLR

ROMANIA ISS, UNIV. OF BUCHAREST



SWITZERLAND ETH - ZURICH UNIV. OF GENEVA

CHINA

KOREA EWHA

KYUNGPOOK NAT.UNIV.

PORTUGAL LAB. OF INSTRUM. LISBON

SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu)

BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan)

NLAA (Beijing) SEU JSC

CERN

AMS-SOC@SEU

AMS-C@SEU

12

13

•  SEU’s Collaboration with AMS and MIT lasts for more than a decade

•  ~40 SEU Professors/Assistant Professors/students visited MIT

•  MIT faculties including Nobel laureate Professor Samuel Ting、Research Professor Vitaly Choutko visited SEU regularly in the past few years

Prof. Samuel Ting visited SEU

Talents exchange between SEU and AMS

Talents exchange between SEU and AMS

14

Prof. Li Qi at MIT Prof. Luo Junzhou at NASA

Prof. Vitaly Choutko at SEU Discussions at CERN

High-tech breakthroughs at SEU

15

AMS-02 silicon trackers AMS-02 Control System

AMS-C at SEU Prof. Samuel Ting at SEU

Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment.

AMS 1. AMS-C

2. AMS-AIS

3. AMS-SOC

SEU

USA MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV.

- NEW HAVEN

MEXICO UNAM




GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE


NETHERLANDS ESA - ESTEC NIKHEF NLR

ROMANIA ISS, UNIV. OF BUCHAREST



SWITZERLAND ETH - ZURICH UNIV. OF GENEVA

CHINA

KOREA EWHA

KYUNGPOOK NAT.UNIV.

PORTUGAL LAB. OF INSTRUM. LISBON

SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu)

BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan)

NLAA (Beijing) SEU JSC

CERN

AMS-SOC@SEU

AMS-C@SEU

16

Docker

q  SEU’ data center dedicated to AMS experiment q  5500 CPU cores (70 TFLOPS at peak) and 920TB storage.

q  Docker 、OpenStack、Hadoop q  1Gb/s dedicated network between SEU and CERNET Beijing

gateway, 2.5Gb between CERNET and CERN

q  Cloud based AMS-SOC for better supporting AMS data storage、computing and data transfer

AMS-02 SOC at SEU

17

…… ……

4 × IBM 42U Rack40Gbps QDR

Infiniband Switch

1Gpbs Ethernet Switch

IBM X3850 X5

IBM X3550 M3

IBM H22 Blade Server

IBM X3650 M3

8 × IBM 42U RackFC SAN Switch

IBM DS5300

To CERNET Beijing Exchange Point

40Gbps Infiniband 1Gbps Ethernet

8Gbps Fiber Channel 1Gbps Dedicated LinkTo SEU Campus Network

18

SEU’s AMS Data Transfer Network

Guangzhou

NanjingSEU

Shanghai

Wuhan

NLAACERN

10GPOS

2.5GPOS

Tier 1 Tier 2

1GPOS Dedicated for AMS

IPv6 Link

BeijingCNGI-6IX

Topology of China Education and Research NETwork(CERNET) Backbone

19

A Glance at SEU’s AMS-SOC (cost: RMB 25M)

Prof. Ting and the MIT delegation visited SEU’s AMS-SOC to discuss the data analysis solutions

20

Sim ulation D ata

D ata R econstruction

AMS-02Package

C alibration

D etector Sim ulation

R aw D ataD A Q D ata

R econstruction

GEANTPhysics A nalysis

ROOT

R esults

V isualization

M C Production

Data processing at SEU’s AMS SOC

21

CERN

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

…

…

Database

Data Storage

c01 c12 c17

1. MC Job request

LSF Server

2. MC Job/Env. download

4. Env. update

3. Job/Env. registration

5. Job submit

6. production

7. status update

7. status update

8. data transfer

Data Transfer Servers

Ser

OS

Ser

OS

Ser

OS

Ser

OS

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Cluster(VMs)

c18

HPC Cluster(Physical nodes)

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

c02

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

c16

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter

1 2 3 4 5 6 7 8 9 10 11 12 13 14

…

…

Facility for AMS MC production：HPC Cluster（168 physical servers） and cloud based VM Cluster（more than180 instances）

AMS [email protected]

3 4REC jobs downlad copy REC jobs

21

http://pcamss0.cern.ch/rc.html

IBM Cluster@SEUSugon Cluster@SEU

5

[email protected]

Remotely transfer jobs

v5.00mcscripts.tar.gzv5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz


Filemonitor &Database6AMS reconstruction jobs


Remotely transfer data

Facility for AMS data reconstruction ：HPC Cluster（168 physical servers）， 1Gb/s network between SEU and CERN

AMS userLogin node

Login

Physics analysis jobs

Job submission

SEU’s AMS SOC

Computing resources

Data analysis

Job scheduling

Software resources

Analysis results

168 IBM HS22 blade

servers

LSF ROOT/Geant/...

42 IBM Flex240 blade

servers

Data source: http://ams.cern.ch/ProdPlot/index.php

Data processing at SEU’s AMS SOC

SEU

Data Production at SEU CPU hours spent at SEU

q 696TB data are produced at SEU, including 138TB Raw data，368TB Monte Carlo data and 190TB Reconstructed data

q 1.53M CPU hours are spent at SEU: 1st place in 6 regional AMS SOC

22

•  Data transfer architecture

–  CERNèSEU( Raw Data ) •  client：LFTP •  server：VSFTP •  Request: 200GB/day

23

Data transfer at SEU’s AMS SOC

*.*.23.242

Base server at CERN

CERN SEU

Server(VSFTP)

Upload

Client(LFTP)

*.*.23.242

Base server at CERN

CERN SEU

Server(VSFTP)client

(LFTP)

download

*.*.23.241

*.*.23.240

•  Data transfer architecture –  SEUèCERN( MC Simulation Data / Reconstructed Data )

•  client：LFTP •  server：VSFTP •  Request: 1~2TB/day

24


•  Data transfer improvement by optimizing –  two parameters: the size of socket buffer at both sending and

receiving sides; the number of parallel connections –  Increasing the socket buffer size of the sending and receiving

side will help to improve the data transfer performance –  Increasing the number of parallel connections does not always

increase the transfer performance

25


Transfer bandwidth with different number of parallel connections

Transfer time and packet loss rate with different number of parallel connections

26


Data Transfer Status

AMS data transfer reaches 400Mbit/sec after optimization.

27


SEU is able to transfer more than 1 TB AMS data to CERN per day

Sep 10 2015 Aug 10 2015 Jul 10 2015 Jun 10 2015 May 10 2015 0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Dai

ly tr

ansf

erre

d da

ta s

ize

(TB

)

Date

Cloud supporting platform layer

Hardwares in DC

Virtualization Elastic Resource Management

Big data storage and management layer

Pre-processing

Data processing layerGraph

ProcessingStreaming data

processingData Query

AMS application layerData

ReconstructionData AnalysisMC Simulation

Placement Migration

N etw ork R esources

C om puting R esources

StorageR esources

28

Researches on AMS big data processing

29

Cloud based AMS big data processing platform

…… ……Hardwares

in DC

Raw data

Ser

OS

Ser

OS

Task Scheduler

Map/Reduce

AMS-SOC portalSEU usersAMS users

MC Simulation Data Reconstruction Data Analysis

AMS applications

SCSI/SATA

TAPE

Data Storage

……

Data Placement

Data Migration

SSD

Big data processing

Data Query Data Analysis

Streaming Data Processing

Elastic Resource

ManagementOpenstack Ceph

o

Big data processing platformsHadoop Yarn/Spark/Tez/Storm




Outline

30

Conclusion

31

q After these 13 years collaborations, we have obtained some fruitful achievements Ø Successfully construct AMS-C, AMS-AIS and

AMS-SOC at SEU Ø Process and transfer more than 700TB

scientific data Ø Breakthrough several key technologies Ø Exchange more than 40 teachers and students

Future Work

32

q AMS-02 will continuously produce massive data（~4PB in next 10 years of AMS operation）, including : Ø  raw data (~40TB/year) Ø  reconstructed data（~200TB/year） Ø MC data（~200TB/year）

q Big data processing techniques need to be further employed including: Ø Geo-distributed data analysis among multiple DCs Ø High performance data transfer between AMS-SOCs

Thank You !

[email protected]

Documents

Southeast University and AMS-02 Experiment · PDF fileSoutheast University and AMS-02 Experiment Fang Dong School of Computer Science and Engineering, Southeast University [email protected]