Networks, Grids and the Digital Divide inNetworks, Grids and the Digital Divide in HEP and Global e-Science HEP and Global e-Science

Harvey B. NewmanHarvey B. Newman ICFA ICFA Workshop on HEP Networking, Grids, Workshop on HEP Networking, Grids, and Digital Divide Issues for Global e-Scienceand Digital Divide Issues for Global e-Science

Daegu, May 23 2005Daegu, May 23 2005

TOTEM pp, general purpose; HI

pp, general purpose; HI

LHCb: B-physics

ALICE : HI

pp s =14 TeV L=1034 cm-2 s-1

27 km Tunnel in Switzerland & France

Large Hadron Collider CERN, Geneva: 2007 Start

Large Hadron Collider CERN, Geneva: 2007 Start

CMS

Atlas

Higgs, SUSY, Extra Dimensions, CP Violation, QG Plasma, … the Unexpected

5000+ Physicists 250+ Institutes 60+ Countries

LHC Data Grid Hierarchy:Developed at Caltech

LHC Data Grid Hierarchy:Developed at Caltech

Tier 1

Tier2 Center

Online System

CERN Center PBs of Disk;

Tape Robot

FNAL CenterIN2P3 Center INFN Center RAL Center

InstituteInstituteInstituteInstitute

Workstations

~150-1500 MBytes/sec

~10 Gbps

1 to 10 Gbps

Tens of Petabytes by 2007-8,at ~100 Sites.

An Exabyte ~5-7 Years later.Physics data cache

~PByte/sec

10 - 40 Gbps

Tier2 CenterTier2 CenterTier2 Center

~1-10 Gbps

Tier 0 +1

Tier 3

Tier 4

Tier2 Center Tier 2

Experiment

CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1

Emerging Vision: A Richly Structured, Global Dynamic System

ICFA and Global Networks ICFA and Global Networks for Collaborative Sciencefor Collaborative Science

ICFA and Global Networks ICFA and Global Networks for Collaborative Sciencefor Collaborative Science

National and International Networks, with rapidly National and International Networks, with rapidly increasing capacity and end-to-end capability increasing capacity and end-to-end capability are essential, forare essential, forThe daily conduct of collaborative work in bothThe daily conduct of collaborative work in both

experiment and theory experiment and theoryExperiment development & construction Experiment development & construction

on a global scale on a global scaleGrid systems supporting analysis involving Grid systems supporting analysis involving

physicists in all world regions physicists in all world regions The conception, design and implementation of The conception, design and implementation of

next generation facilities as “global networks” next generation facilities as “global networks” ““Collaborations on this scale would never have been Collaborations on this scale would never have been

attempted, if they could not rely on excellent networks”attempted, if they could not rely on excellent networks”

Challenges of Next Generation Science in the Information AgeChallenges of Next Generation Science in the Information Age

Flagship Applications Flagship Applications High Energy & Nuclear Physics, AstroPhysics Sky Surveys:High Energy & Nuclear Physics, AstroPhysics Sky Surveys:

TByte to PByte “block” transfers at 1-10+ Gbps TByte to PByte “block” transfers at 1-10+ Gbps Fusion Energy:Fusion Energy: Time Critical Burst-Data Distribution; Time Critical Burst-Data Distribution;

Distributed Plasma Simulations, Visualization, Analysis Distributed Plasma Simulations, Visualization, Analysis eVLBI:eVLBI: Many (quasi-) real time data streams at 1-10 Gbps Many (quasi-) real time data streams at 1-10 Gbps BioInformatics, Clinical Imaging:BioInformatics, Clinical Imaging: GByte images on demand GByte images on demand

Advanced integrated Grid applications rely on reliable, Advanced integrated Grid applications rely on reliable, high performance operation of our LANs and WANs high performance operation of our LANs and WANs

Analysis Challenge:Analysis Challenge: Provide results with rapid turnaround, Provide results with rapid turnaround, over networks of varying capability in different world regionsover networks of varying capability in different world regions

Petabytes of complex data explored and analyzed by 100s-1000s Petabytes of complex data explored and analyzed by 100s-1000s of globally dispersed scientists, in 10s-100s of teamsof globally dispersed scientists, in 10s-100s of teams

Huygens Space Probe Lands on Titan - Monitored by 17 telescopes in Au, Jp, CN, US

Huygens Space Probe Lands on Titan - Monitored by 17 telescopes in Au, Jp, CN, US

In October 1997, the Cassini spacecraft left Earth to In October 1997, the Cassini spacecraft left Earth to travel to Saturntravel to Saturn

On Christmas day 2004, the Huygens probe separated On Christmas day 2004, the Huygens probe separated from Cassini from Cassini

On 14 January 2005 it started its descent through the On 14 January 2005 it started its descent through the dense (methane, nitrogen) atmosphere of Titan dense (methane, nitrogen) atmosphere of Titan (speculated to be similar to that of Earth (speculated to be similar to that of Earth billions of years ago) billions of years ago)

The signals sent back from Huygens to Cassini were The signals sent back from Huygens to Cassini were

monitored by 17 telescopes in Australia, China, Japan monitored by 17 telescopes in Australia, China, Japan and the US to accurately position the probe to within a and the US to accurately position the probe to within a kilometre (Titan is ~1.5 billion kilometres from Earth)kilometre (Titan is ~1.5 billion kilometres from Earth)

Courtesy G. McLaughlin

Australian eVLBI data sent over high speed links to the Netherlands

Australian eVLBI data sent over high speed links to the Netherlands

The data from two of the Australian telescopes were The data from two of the Australian telescopes were transferred to the Netherlands over the transferred to the Netherlands over the SXTransport SXTransport and and IEEAF IEEAF links, and links, and CA*net4 CA*net4 using using UCLPUCLP, and were the first to , and were the first to be received by JIVE (Joint Institute for VLBI in Europe), the be received by JIVE (Joint Institute for VLBI in Europe), the correlator sitecorrelator site

The data was transferred at an average rate of 400MbpsThe data was transferred at an average rate of 400Mbps (note 1Gbps was available) (note 1Gbps was available)

The data from these two telescopes were reformatted and The data from these two telescopes were reformatted and correlated within hours of the end of the landingcorrelated within hours of the end of the landing

This early correlation allowed calibration of the data This early correlation allowed calibration of the data processor at JIVE, ready for the data from other telescopes processor at JIVE, ready for the data from other telescopes to be addedto be added

Significant int’l collaborative effort: 9 OrganizationsSignificant int’l collaborative effort: 9 OrganizationsG. McLaughlin, D.

Riley

ICFA Standing Committee on Interregional Connectivity (SCIC)

ICFA Standing Committee on Interregional Connectivity (SCIC)

Created in July 1998 in Vancouver ; Following ICFA-NTFCreated in July 1998 in Vancouver ; Following ICFA-NTFCHARGE: CHARGE: Make recommendations to ICFA concerning the connectivityMake recommendations to ICFA concerning the connectivity

between between the Americasthe Americas, Asia and Europe , Asia and Europe As part of the process of developing theseAs part of the process of developing these

recommendations, the committee should recommendations, the committee should Monitor traffic on the world’s networks Monitor traffic on the world’s networks Keep track of technology developmentsKeep track of technology developments Periodically review forecasts of future Periodically review forecasts of future

bandwidth needs, and bandwidth needs, and Provide early warning of potential problemsProvide early warning of potential problems

Create subcommittees as needed to meet the chargeCreate subcommittees as needed to meet the charge Representatives: Major labs, ECFA, ACFA, North and Representatives: Major labs, ECFA, ACFA, North and

South American Users South American Users Chair of the committee reports to ICFA twice per yearChair of the committee reports to ICFA twice per year

SCIC in 2004-2005 http://cern.ch/icfa-scic SCIC in 2004-2005 http://cern.ch/icfa-scic

Three 2005 Reports, Presented to ICFA Today Three 2005 Reports, Presented to ICFA Today Main Report: “Networking for HENP”Main Report: “Networking for HENP” [H. Newman et al.] [H. Newman et al.]

Includes Updates on the Digital Divide, WorldIncludes Updates on the Digital Divide, WorldNetwork Status; Brief updates on Monitoring and Network Status; Brief updates on Monitoring and Advanced Technologies [*]Advanced Technologies [*]

18 Appendices: A World Network Overview 18 Appendices: A World Network Overview Status and Plans for the Next Few Years of Nat’l & Status and Plans for the Next Few Years of Nat’l & Regional Networks, and Optical Network InitiativesRegional Networks, and Optical Network Initiatives

Monitoring Working Group ReportMonitoring Working Group Report [L. Cottrell] [L. Cottrell]

Also See: Also See: SCIC Digital Divide ReportSCIC Digital Divide Report [A. Santoro et [A. Santoro et

al.]al.] SCIC 2004 Digital Divide in Russia Report [V. Ilyin]SCIC 2004 Digital Divide in Russia Report [V. Ilyin] TERENA (TERENA (www.terena.nlwww.terena.nl) 2004 Compendium ) 2004 Compendium

SCIC Main Conclusion for 2002-5SCIC Main Conclusion for 2002-5

The disparity among regions in HENP could increase The disparity among regions in HENP could increase even more sharply, as we learn to use advanced even more sharply, as we learn to use advanced networks effectively, and we develop dynamic Grid networks effectively, and we develop dynamic Grid systems in the “most favored” regions systems in the “most favored” regions

We must therefore take action, and We must therefore take action, and work work to Close the Digital Divideto Close the Digital Divide To make Scientists in All World Regions Full To make Scientists in All World Regions Full

Partners in the Process of Frontier Discoveries Partners in the Process of Frontier Discoveries This is essential for the health of our global This is essential for the health of our global

experimental collaborations, for our field, experimental collaborations, for our field, and for international collaboration in many and for international collaboration in many fields of science.fields of science.

NEWS: Bulletin: ONE TWOWELCOME BULLETIN General InformationRegistrationTravel Information Hotel RegistrationParticipant List How to Get UERJ/Hotel Computer AccountsUseful Phone Numbers ProgramContact us: Secretariat Chairmen

 CLAF  CNPQ FAPERJ      UERJ

SPONSORS

HEPGRID and Digital Divide Workshop

UERJ, Rio de Janeiro, Feb. 16-20 2004Theme: Global Collaborations, Grids and

Their Relationship to the Digital DivideFor the past three years the SCIC has focused on understanding and seeking the means of reducing or eliminating the Digital Divide. It proposed to ICFA that these issues, as they

affect our field, be brought to our community for discussion. This led to ICFA’s approval, in July 2003, of the Digital Divide and HEP Grid

Workshop. Review of R&E Networks; Major Grid

Projects Perspectives on Digital Divide Issues by

Major HEP Experiments, Regional Representatives Focus on Digital Divide Issues in Latin

America; Relate to Problems in Other Regions

See http://www.lishep.uerj.br

TutorialsC++Grid TechnologiesGrid-Enabled

AnalysisNetworksCollaborative Systems

A. Santoro

International ICFA Workshop onInternational ICFA Workshop on HEP Networking, Grids, and Digital HEP Networking, Grids, and Digital Divide Issues for Global e-ScienceDivide Issues for Global e-Science

May 23-27, 2005Daegu, Korea

Dongchul SonCenter for High Energy Physics

Harvey NewmanCalifornia Institute of

Technology Focus on Asia-Pacific Focus on Asia-Pacific Also Latin America, Middle East, Also Latin America, Middle East,

AfricaAfrica

Approved by ICFA Approved by ICFA August 2004August 2004

고에너지물리연구센터 CENTER FOR HIGH ENERGY PHYSICS

International ICFA Workshop on HEP International ICFA Workshop on HEP Networking, Grids Networking, Grids and Digital Divide Issues for Global e-Scienceand Digital Divide Issues for Global e-Science

Workshop Goals Review the current status, progress and barriers to effective

use of major national, continental and transoceanic networks

Review progress, strengthen opportunities for collaboration, and explore the means to deal with key issues in Grid computing and Grid-enabled data analysis, for high energy physics and other fields of data intensive science

Exchange information and ideas, and formulate plans to develop solutions to specific problems related to the Digital Divide, with a focus on the Asia Pacific region, as well as Latin America, Russia and Africa

Continue to advance a broad program of work on reducing or eliminating the Digital Divide, and ensuring global collaboration, as related to all of the above aspects.

PingER: World View from SLAC, CERN

S.E. Europe, Russia: Catching upLatin Am., China: Keeping upIndia, Mid-East, Africa: Falling Behind

C. Asia, Russia, SE Europe, L. America, M. East, China: 4-7 yrs behindIndia, Africa: 7-8 yrs behind

Latin AmericaLatin America

R. Cottrell

Connectivity to AfricaConnectivity to Africa

Internet Access: More than an order of magnitude lower than the Internet Access: More than an order of magnitude lower than the corresponding percentages in Europe (33%) & N. America (70%). corresponding percentages in Europe (33%) & N. America (70%).

Digital Divide: Lack of Infrastructure, especially interior, high prices Digital Divide: Lack of Infrastructure, especially interior, high prices (e.g. $ 4-10/kbps/mo.); “Gray” satellite bandwidth market(e.g. $ 4-10/kbps/mo.); “Gray” satellite bandwidth market

Intiatives: Intiatives: EUMEDCONNECT (EU-North Africa); GEANT: 155 Mbps to EUMEDCONNECT (EU-North Africa); GEANT: 155 Mbps to S. Africa; Nectarnet (Ga. Tech); IEEAF/I2 NSF-Sponsored InitiativeS. Africa; Nectarnet (Ga. Tech); IEEAF/I2 NSF-Sponsored Initiative

INTERNET USERS AND POPULATION STATISTICS FOR AFRICA

Population( 2004 Est. )

Pop. Pct.

World

Internet Users

Usage Growth(2000-4)

Penetration:Percent of Population

Users: Percentin World

TOTAL for AFRICA

893 M 14.0 % 12.9 M 186.6 % 1.4 % 1.6 %

REST of WORLD

5,497 M 86.0 % 800 M 124.4 % 14.6 % 98.4 %

Sample Bandwidth Costs for African Universities

$0.27

$3.00

$6.77

$9.84

$11.03

$20.00

$0.00 $5.00 $10.00 $15.00 $20.00 $25.00

USA

IBAUD Target

Ghana

Uganda

Average

Nigeria

$/kbps/month

Sample size of 26 universitiesAverage Cost for VSAT service: Quality, CIR, Rx, Tx not distinguished

Bandwidth prices in Africa vary dramatically; are in general many times what they could be if universities purchase in volume

Roy Steiner Internet2 2004 Workshop

Avg. Unit Cost is 40X US Avg.; Cost is Several Hundred Times, Compared to Leading Countries

Asia Pacific Academic Network ConnectivityAsia Pacific Academic Network Connectivity

Access PointExchange PointCurrent Status2004 (plan)

US

Europe

ID

VN

LK

KR

TH

MY

SG

RU

PH

CN

HK

JP2G

12M

34M

45M

20.9G

155M

777M

1.2G

TW

200M

310M

AU

45M`722M

IN

16M

1.5M

1.5M

7.5M2M

2.5M

2M

90M

155M

622M

155M

9.1G

932M

Better Better North/South Linkages within Asia North/South Linkages within Asia NeededNeeded JP- TH link: 2Mbps JP- TH link: 2Mbps 45Mbps in 2004. 45Mbps in 2004.

Connectivity Connectivity to US from JP, to US from JP, KO, AU is KO, AU is Advancing Advancing Rapidly.Rapidly.

Progress in Progress in the Region, the Region, and to Europe and to Europe is Much is Much Slower Slower

APAN Status APAN Status 7/20047/2004

D. Y. Kim

CountriesCountries NetworkNetwork Bandwidth(Mbps)Bandwidth(Mbps) AUP/RemarkAUP/RemarkID-IDID-ID AI3 (ITB-UNIBRAW)AI3 (ITB-UNIBRAW) 0.128/0.1280.128/0.128 R&E + CommodityR&E + CommodityIN-US/UKIN-US/UK ERNETERNET 7070 R&ER&EJP-CNJP-CN NICT-CERNETNICT-CERNET 1 Gbps1 Gbps R&ER&EJP-CNJP-CN NICT-CSTNETNICT-CSTNET 1 Gbps1 Gbps R&ER&EJP-IDJP-ID AI3 (ITB)AI3 (ITB) 0.5/1.5 (From/To JP)0.5/1.5 (From/To JP) R&E + CommodityR&E + CommodityJP-KRJP-KR APIIAPII 2 Gbps2 Gbps R&E R&E JP-LAJP-LA AI3 (NUOL)AI3 (NUOL) 0.1280.128 R&E + CommodityR&E + CommodityJP-LKJP-LK AI3 (IOIT)AI3 (IOIT) 0.5/0.5 (From/To JP)0.5/0.5 (From/To JP) R&E + CommodityR&E + CommodityJP-MYJP-MY AI3 (USM)AI3 (USM) 0.5/0.5 (From/To JP)0.5/0.5 (From/To JP) R&E + CommodityR&E + CommodityJP-PHJP-PH AI3 (ASTI)AI3 (ASTI) 0.5/0.5 (From/To JP)0.5/0.5 (From/To JP) R&E + CommodityR&E + CommodityJP-PHJP-PH MAFFIN (ASTI)MAFFIN (ASTI) 66 R&E R&E JP-SGJP-SG AI3 (TP)AI3 (TP) 0.5/0.5 (From/To JP)0.5/0.5 (From/To JP) R&E + CommodityR&E + CommodityJP-THJP-TH AI3 (AIT)AI3 (AIT) 0.5/1.5 (From/To JP)0.5/1.5 (From/To JP) R&E + CommodityR&E + CommodityJP-THJP-TH SINET (ThaiSarn)SINET (ThaiSarn) 4545 R&E R&E JP-USJP-US IEEAFIEEAF 10 Gbps+62210 Gbps+622 R&E R&E JP-USJP-US JGN2JGN2 10 Gbps10 Gbps R&E R&E JP-USJP-US SINETSINET 10 Gbps10 Gbps R&E R&E

Some APAN Links Some APAN Links

G. McLaughlin

Digital Divide Illustrated by Network Infrastructures: TERENA NREN Core Capacity

Digital Divide Illustrated by Network Infrastructures: TERENA NREN Core Capacity

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Ukraine

Moldova

Azerbaijan

Jordan

Morocco

Iran

Bulgaria

Algeria

Croatia

Romania

Turkey

Serbia/Montenegro

Cyprus

Estonia

Latvia

Malta

Slovenia

Lithuania

Austria

Denmark

Iceland

Ireland

Slovakia

Switzerland

Portugal

Czech Republic

Finland

France

Greece

Hungary

Italy

Norway

Spain

Belgium

Germany

Netherlands

Poland

Sweden

United Kingdom

Core Capacity in Gbps

Current Core Capacity

Expected Increase in two years

Core capacity goes up in Large Steps: 10 to 20 Gbps; 2.5 to 10 Gbps; 0.6-1 to 2.5 Gbps

Current In Two Years

SE Europe, Medit., FSU, Middle East:Less Progress Based on Older

Technologies (Below 0.15, 1.0 Gbps): Digital Divide Will Not Be Closed

Source: TERENA

Long Term Trends in Network Traffic Volumes: 300-1000X/10Yrs

Long Term Trends in Network Traffic Volumes: 300-1000X/10Yrs

SLAC Traffic ~400 Mbps; Growth in Steps (ESNet Limit): ~ 10X/4 Years.

Projected: ~2 Terabits/s by ~2014 July 2005: 2x10 Gbps links: one

for production and one for research

ESnet Accepted Traffic 1990 – 2004Exponential Growth Since ’92;

Annual Rate Increased from 1.7 to 2.0X Per Year In the Last 5 Years

W. Johnston

L. Cottrell

Progress

in Steps

ESnet Monthly Accepted Traffic ThroughNov, 2004

0

50

100

150

200

250

300

350

400

Jan,

94Ju

n, 94

Nov,

94Ap

r, 95

Sep,

95Fe

b, 96

Jul, 9

6De

c, 96

May,

97Oc

t, 97

Mar, 9

8Au

g, 98

Jan,

99Ju

n, 99

Nov,

99Ap

r, 00

Sep,

00Fe

b, 01

Jul,

01De

c, 01

May,0

2Oc

t, 02

Mar, 0

3Au

g, 03

Jan,

04Ju

n, 04

Nov,

04

TByte

/Mon

th

W. Johnston

10 Gbit/s

FNAL: 10 to 20 (+40) Gbps by Fall 2005

CANARIE (Canada) Utilization Trends and UCLPv2

0

5

10

15

20

25

30LightpathsIP PeakIP Average

GbpsNetwork Capacity Limit

Jan 0

5

“Demand for Customer Empowered Nets (CENs) is exceeding our wildest expectations

New version of UCLP will allow easier integration of CENs into E2E nets for specific communities &/or disciplines

UCLPv2 will be based on SOA, web services and workflow to allow easy integration into cyber-infrastructure projects

The Network is no longer a static fixed facility – but can be ‘orchestrated’ with different topologies, routing etc to meet specific needs of end users”

W. St. Arnaud

National Lambda Rail (NLR): www.nlr.netNational Lambda Rail (NLR): www.nlr.net

NLR Initially 4-8 10G

Wavelengths To 40 10G

Waves in Future Ultralight, Internet2

HOPI, Cisco Research & UltraScience NetInitiatives w/HEP

Atlantic & Pacific Wave

Transition beginning now to optical, multi-wavelength Community owned or leased

“dark fiber” networks for R&E

Initiatives in: nl, ca, jp, uk, kr; pl, cz, sk, pt, ei, gr, sb/mn … + 30 US States (Ca, Il, Fl, In, …)

GEANT2 Hybrid ArchitectureGEANT2 Hybrid Architecture

Cooperation of 26 NRENs Implementation on dark fiber, IRU

Asset, Transmission & Switching Equipment Layer 1 & 2 switching, “the Light Path” Point to Point (E2E) Wavelength services LHC in Europe: N X 10G T0-T1 Overlay Net

Global Connectivity

10 Gbps + 3x2.5 Gbps to North America

2.5 Gbps to Japan

622 Mbps to South America

45 Mbps to Mediterranean countries

155 Mbps to South Africa

Will be Improved in GEANT2

H. Doebbling

SXTransport: Au-US 2 X 10GSXTransport: Au-US 2 X 10G

AARNet has dual 10Gbps circuits to the US via Hawaii, dual 622Mbps commodity links G. McLaughlin

JGN2: Japan Gigabit Network (4/04 – 3/08)20 Gbps Backbone, 6 Optical Cross-Connects

Kanazawa Sendai

Sapporo

Nagano

Kochi Nagoya

Fukuoka

Okinawa

Okayama

< １ G>　・ Teleport Okayama (Okayama) 　・ Hiroshima University (Higashi

Hiroshima) < １００ M>　・ Tottori University of

Environmental Studies (Tottori) 　・ Techno Ark Shimane (Matsue) 　・ New Media Plaza Yamaguchi

(Yamaguchi)

< １０ G>　・ Kyoto University (Kyoto) 　・ Osaka University (Ibaraki) < １ G>　 ・ NICT Kansai Advanced Research Center (Kobe) < １００ M>　・ Lake Biwa Data Highway AP * (Ohtsu)　・ Nara Prefectural Institute of Industrial

Technology (Nara) 　・ Wakayama University (Wakayama) 　・ Hyogo Prefecture Nishiharima Technopolis

(Kamigori-cho, Hyogo Prefecture) < １０ G> 　 ・ Kyushu University (Fukuoka) < １００ M>　・ NetCom Saga (Saga) 　・ Nagasaki University

(Nagasaki) 　・ Kumamoto Prefectural Office

(Kumamoto) 　・ Toyonokuni Hyper Network

AP *(Oita)　・ Miyazaki University (Miyazaki) 　・ Kagoshima University

(Kagoshima)

< １００ M>　・ Kagawa Prefecture Industry Promotion Center (Takamatsu) 　・ Tokushima University (Tokushima) 　・ Ehime University (Matsuyama) 　・ Kochi University of Technology

(Tosayamada-cho, Kochi Prefecture)

< １００ M>　・ Nagoya University (Nagoya) 　・ University of Shizuoka (Shizuoka) 　・ Softopia Japan (Ogaki, Gifu Prefecture) 　・ Mie Prefectural College of Nursing (Tsu)

< １０ G>　・ Ishikawa Hi-tech Exchange Center

(Tatsunokuchi-machi, Ishikawa Prefecture) < １００ M>　・ Toyama Institute of Information Systems (Toyama) 　・ Fukui Prefecture Data Super Highway AP * (Fukui) 　

< １００ M>　・ Niigata University

(Niigata) 　・ Matsumoto Information Creation Center

(Matsumoto, Nagano Prefecture)

< １０ G>　・ Tokyo University

(Bunkyo Ward, Tokyo) 　・ NICT Kashima Space Research Center

(Kashima, Ibaraki Prefecture) <1G>　・ Yokosuka Telecom Research Park

(Yokosuka, Kanagawa Prefecture) <100M>　・ Utsunomiya University (Utsunomiya) 　・ Gunma Industrial Technology Center

(Maebashi) 　・ Reitaku University

(Kashiwa, Chiba Prefecture) 　・ NICT Honjo Information and Communications Open Laboratory

(Honjo, Saitama Prefecture) 　・ Yamanashi Prefecture Open R&D Center

(Nakakoma-gun, Yamanashi Prefecture)

< １ G>　・ Tohoku University (Sendai) 　・ NICT Iwate IT Open Laboratory

(Takizawa-mura, Iwate Prefecture) < １００ M>　・ Hachinohe Institute of Technology

(Hachinohe, Aomori Prefecture) 　・ Akita Regional IX * (Akita) 　・ Keio University Tsuruoka Campus

(Tsuruoka, Yamagata Prefecture) 　・ Aizu University (Aizu Wakamatsu)

< １００ M>　 ・ Hokkaido Regional Network Association AP * (Sapporo)　　　　　　　　　　　　　　　　

NICT Tsukuba Research Center

20Gbps10Gbps1GbpsOptical testbeds

Core network nodes

Access points

Osaka

Otemachi

USA

NICT Keihannna Human Info-Communications

Research Center

NICT Kita Kyushu IT Open Laboratory

NICT Koganei Headquarters

[Legends ]

*IX:Internet eXchange

AP:Access Point

JGN2 Connection services at

the optical level: 1 GbE and 10GbE

Optical testbeds: e.g.GMPLS Interop. Tests

Y. Karita

APAN-KR : KREONET/KREONet2 IIAPAN-KR : KREONET/KREONet2 II

KREONET 11 Regions, 12 POP Centers Optical 2.5-10G Backbone;

SONET/SDH, POS, ATM National IX Connection

KREONET2 Support for Next Gen. Apps: IPv6, QoS, Multicast;

Bandwidth Alloc. Services StarLight/Abilene Connection

International Links GLORIAD Link to 10G to

Seattle Aug. 1 (MOST) US: 2 X 622 Mbps via

CA*Net; GbE via TransPAC Japan: 2 Gbps TEIN to GEANT: 155 Mbps

SuperSIREN (7 Res. Institutes) Optical 10-40G Backbone Collaborative Environment

Support High Speed Wireless: 1.25 G

D. Son

The Global Lambda Integrated Facility for Research and Education (GLIF)

The Global Lambda Integrated Facility for Research and Education (GLIF)

Architecting an international LambdaGrid infrastructure Architecting an international LambdaGrid infrastructure Virtual organizationVirtual organization supports persistent data-intensive scientific supports persistent data-intensive scientific

research and middleware development on “LambdaGrids”research and middleware development on “LambdaGrids”

Many 2.5 - 10G Links Across the Atlantic and PacificPeerings: Pacific & Atlantic Wave; Seattle, LA, Chicago, NYC, HK

Product of transfer speed and distance Product of transfer speed and distance using standard using standard Internet (TCP/IP) protocols.Internet (TCP/IP) protocols.

Single Stream Single Stream 7.5 Gbps X 16 kkm7.5 Gbps X 16 kkm with with Linux: July 2004Linux: July 2004

IPv4 Multi-stream record with FAST IPv4 Multi-stream record with FAST TCP: TCP: 6.86 Gbps X 27kkm:6.86 Gbps X 27kkm: Nov 2004 Nov 2004

IPv6 record: IPv6 record: 5.11 Gbps5.11 Gbps between between Geneva and Starlight: Jan. 2005Geneva and Starlight: Jan. 2005

Concentrate now on reliable Concentrate now on reliable Terabyte-scale file transfersTerabyte-scale file transfers Disk-to-disk Marks: Disk-to-disk Marks:

536 Mbytes/sec (Windows); 536 Mbytes/sec (Windows); 500 Mbytes/sec (Linux) 500 Mbytes/sec (Linux)

Note Note System Issues:System Issues: PCI-X Bus, PCI-X Bus, Network Interface, Disk I/O Network Interface, Disk I/O Controllers, CPU, DriversControllers, CPU, Drivers

Internet 2 Land Speed Record (LSR)

NB: Computing Manuf.’s Roadmaps for 2006: One Server Pair to One 10G Link

Nov. 2004 Record Network

6.6 Gbps16500km

4.2 Gbps16343km5.6 Gbps

10949km

5.4 Gbps7067km2.5 Gbps

10037km0.9 Gbps10978km0.4 Gbps

12272km

0

20

40

60

80

100

120

140

160

Thro

ughp

ut (G

bps)

Internet2 LSR - Single IPv4 TCP stream 7.21 Gbps20675 kmInternet2 LSRs:

Blue = HEP

7.2G X 20.7 kkm

Th

rou

hg

pu

t (P

eta

bit

-m/s

ec)

S. Ravot

SC2004 Bandwidth Record by HEP: High Speed TeraByte Transfers for Physics

SC2004 Bandwidth Record by HEP: High Speed TeraByte Transfers for Physics

Caltech, CERN Caltech, CERN SLAC, FNAL, UFl, SLAC, FNAL, UFl, FIU, ESNet, UK, FIU, ESNet, UK, Brazil, Korea;Brazil, Korea;NLR, Abilene, NLR, Abilene, LHCNet, TeraGrid;LHCNet, TeraGrid;DOE, NSF, EU, …; DOE, NSF, EU, …; Cisco, Neterion, Cisco, Neterion, HP, NewiSys, …HP, NewiSys, …

Ten 10G Waves, Ten 10G Waves, 80 10GbE Ports, 80 10GbE Ports, 50 10GbE NICs50 10GbE NICs

Aggregate Rate of Aggregate Rate of 101 Gbps101 Gbps

1.6 Gbps to/from 1.6 Gbps to/from KoreaKorea

2.93 Gbps to/from 2.93 Gbps to/from BrazilBrazil

UERJ, USP

Monitoring NLR, Abilene, LHCNet, SCINet, UERJ, USP, Int’l R&E Nets and 9000+ Grid Nodes Simultaneously

I. Legrand

SC2004 KNU Traffic: 1.6 Gbps to/From Pittsburgh Via Transpac (LA) and NLR SC2004 KNU Traffic: 1.6 Gbps to/From Pittsburgh Via Transpac (LA) and NLR

Monitoring in Daegu

Courtesy K. Kwon

SC2004: 2.93 (1.95 + 0.98) Gbps Sao Paulo – Miami – Pittsburgh

(Via Abilene)

Brazilian T2+T3 HEPGrid: Rio + Sao

Paolo

Also 500 Mbps Via Red CLARA, GEANT (Madrid)

Madrid &

GEANT

GEANT (SURFNet

)

J. Ibarra

HENP Bandwidth Roadmap for Major Links (in Gbps)

HENP Bandwidth Roadmap for Major Links (in Gbps)

Year Production Experimental Remarks

2001 0.155 0.622-2.5 SONET/SDH

2002 0.622 2.5 SONET/SDH DWDM; GigE Integ.

2003 2.5 10 DWDM; 1 + 10 GigE Integration

2005 10 2-4 X 10 Switch; Provisioning

2007 2-4 X 10 ~10 X 10; 40 Gbps

1st Gen. Grids

2009 ~10 X 10 or 1-2 X 40

~5 X 40 or ~20-50 X 10

40 Gbps Switching

2011 ~5 X 40 or

~20 X 10

~25 X 40 or ~100 X 10

2nd Gen Grids Terabit Networks

2013 ~Terabit ~MultiTbps ~Fill One Fiber

Continuing Trend: ~1000 Times Bandwidth Growth Per Decade;HEP: Co-Developer as well as Application Driver of Global Nets

Evolving Quantitative Science Requirements for Networks (DOE High Perf. Network Workshop)

Evolving Quantitative Science Requirements for Networks (DOE High Perf. Network Workshop)

Science AreasScience AreasToday Today

End2EndEnd2End ThroughputThroughput

5 years 5 years End2End End2End

ThroughputThroughput

5-10 Years 5-10 Years End2End End2End

ThroughputThroughput

RemarksRemarks

High Energy High Energy PhysicsPhysics

0.5 Gb/s0.5 Gb/s 100 Gb/s100 Gb/s 1000 Gb/s1000 Gb/s High bulk High bulk throughputthroughput

Climate (Data & Climate (Data & Computation)Computation)

0.5 Gb/s0.5 Gb/s 160-200 Gb/s160-200 Gb/s N x 1000 Gb/sN x 1000 Gb/s High bulk High bulk throughputthroughput

SNS SNS NanoScienceNanoScience

Not yetNot yet started started

1 Gb/s1 Gb/s 1000 Gb/s + QoS 1000 Gb/s + QoS for Control for Control

ChannelChannel

Remote Remote control and control and time critical time critical throughputthroughput

Fusion EnergyFusion Energy 0.066 Gb/s0.066 Gb/s(500 MB/s (500 MB/s

burst)burst)

0.198 Gb/s0.198 Gb/s(500MB/(500MB/

20 sec. burst)20 sec. burst)

N x 1000 Gb/sN x 1000 Gb/s Time critical Time critical throughputthroughput

AstrophysicsAstrophysics 0.013 Gb/s0.013 Gb/s(1 TByte/week)(1 TByte/week)

N*N multicastN*N multicast 1000 Gb/s1000 Gb/s Computat’l Computat’l steering and steering and

collaborationscollaborations

Genomics Data Genomics Data & Computation& Computation

0.091 Gb/s0.091 Gb/s(1 TBy/day)(1 TBy/day)

100s of users100s of users 1000 Gb/s + QoS 1000 Gb/s + QoS for Control for Control

ChannelChannel

High High throughput throughput

and steeringand steering

See http://www.doecollaboratory.org/meetings/hpnpw/ W. Johnston

LHCNet , ESnet Plan 2007/2008:40Gbps US-CERN, ESnet MANs, IRNC

LHCNet , ESnet Plan 2007/2008:40Gbps US-CERN, ESnet MANs, IRNC

DENDEN

ELPELP

ALBALBATLATL

Metropolitan Area Rings

Aus.

Europe

SDGSDG

AsiaPacSEASEA

Major DOE Office of Science SitesHigh-speed cross connects with Internet2/Abilene

New ESnet hubsESnet hubs

SNVSNV

Europe

Japan

Science Data Network core, 40-60 Gbps circuit transportLab suppliedMajor international

Production IP ESnet core, 10 Gbps enterprise IP traffic

Japan

Aus.

Metro Rings

ESnet2nd Core:

30-50G

ESnet IP Core(≥10 Gbps)

10Gb/s10Gb/s

30Gb/s2 x 10Gb/s

NYCNYCCHICHI

LHCNetData Network(4 x 10 Gbps US-CERN)

LHCNet Data Network

DCDCGEANT2SURFNetIN2P3

NSF/IRNC circuit; GVA-AMS connection via Surfnet or Geant2

CERN

FNAL

BNL

LHCNet US-CERN: 9/05: 10G CHI + 10G NY

2007: 20G + 20G 2009: ~40G + 40G

S. Ravot

We Need to Work on the Digital Dividefrom Several Perspectives

We Need to Work on the Digital Dividefrom Several Perspectives

Workshops and Tutorials/Training SessionsWorkshops and Tutorials/Training Sessions For Example: ICFA DD Workshops, Rio 2/04, HONET (Pakistan) For Example: ICFA DD Workshops, Rio 2/04, HONET (Pakistan)

12/04; 12/04; Daegu May 2005Daegu May 2005 Share Information: Monitoring, BW Progress; Dark Fiber Projects;Share Information: Monitoring, BW Progress; Dark Fiber Projects;

Prices in different markets Prices in different markets Use Model Cases: Poland, Slovakia, Czech Rep., China, Brazil,…Use Model Cases: Poland, Slovakia, Czech Rep., China, Brazil,…

Encourage, and Work on Inter-Regional Projects Encourage, and Work on Inter-Regional Projects GLORIAD, Russia-China-Korea US Optical RingGLORIAD, Russia-China-Korea US Optical Ring Latin America: CHEPREO/WHREN (US-Brazil); RedCLARALatin America: CHEPREO/WHREN (US-Brazil); RedCLARA

Help with Modernizing the Infrastructure Help with Modernizing the Infrastructure Design, Commissioning, Development Design, Commissioning, Development Provide Tools for Effective Use: Monitoring, CollaborationProvide Tools for Effective Use: Monitoring, Collaboration

Systems; Advanced TCP Stacks, Grid System SoftwareSystems; Advanced TCP Stacks, Grid System Software Work on Policies and/or Pricing: pk, br, cn, SE Europe, in, …Work on Policies and/or Pricing: pk, br, cn, SE Europe, in, …

Encourage Access to Dark FiberEncourage Access to Dark Fiber Raise World Awareness of Problems, & Opportunities for SolutionsRaise World Awareness of Problems, & Opportunities for Solutions

UERJ T2 HEPGRID Inauguration: Dec. 2004: The Team (Santoro et al.)

UERJ Tier2 Now On Grid3 and Open Science Grid

(5/13)

100 Dual Nodes;

UpgradesPlanned

Also Tier3 in Sao Paulo

(Novaes)

Grid3, the Open Science Grid and DISUN

Grid3, the Open Science Grid and DISUN

Grid3: A National Grid Infrastructure

35 sites, 3500 CPUs: Univ. + 4 Nat’l labs

Part of LHC Grid Running since

October 2003 HEP, LIGO,

SDSS, Biology, Computer Sci.

Transition to Open Science Grid (www.openscience.org)

+Brazil (UERJ, USP)

7 US CMS Tier2s; Caltech, Florida, UCSD, UWisc Form DISUN

P. Avery

Science-Driven: HEPGRID (CMS) in BrazilHEPGRID-CMS/BRAZIL is a project to build a Grid thatAt Regional Level will include CBPF,UFRJ,UFRGS,UFBA, UERJ & UNESPAt International Level will be integrated with CMS Grid based at CERN; focal points include Grid3/OSG and bilateral projects with Caltech Group

France Italy USAGermany BRAZIL622 Mbps

UFRGS

UERJUFRJ

T1

IndividualMachines

On line systems

Brazilian HEPGRID

CBPF

UNESP/USPSPRACE-Working

Gigabit

CERN 2.5 - 10 Gbps

UFBA

UERJ RegionalTier2 Ctr

T4

T0 +T1

T2 T1

T3 T2

UERJ: T2T1,

100500 Nodes;

Plus T2s to

100 Nodes

ICFA DD Workshop 2/04; T2 Inauguration

+ GIGA/RNP Agree 12/04

Rio Tier2-SPRACE (Sao Paolo)-Ampath Direct Link at 1 GbpsRio Tier2-SPRACE (Sao Paolo)-Ampath Direct Link at 1 Gbps

AMPATHNAP of Brazil

TerremarkEletropaulo Fiber leased to ANSP

1 Gbps

Iqar

a Fib

er

leas

ed to

ANSP1

Gbps

SPRACE

T2 Rio

CC-USP

Jump CC

Giga Fiber

1 Gbps

Giga Fiber

1 Gbps

1 Gb

ps

1 Gb

ps

GigaRouter

ANSPRouters

UERJ

Caltech/CiscoRoutersL. Lopez

Highest Bandwidth Link in NREN’s Infrastructure, EU & EFTA Countries, & Dark Fiber

Highest Bandwidth Link in NREN’s Infrastructure, EU & EFTA Countries, & Dark Fiber

1

10

100

1000

10000

Hig

hest

Lin

k S

peed

(M

bp

s)

Legend

has dark fiber

will have dark fiber

no dark fiber

Owning (or leasing) dark fiber is an interesting option Owning (or leasing) dark fiber is an interesting option for an NREN; Depends on the national situation. for an NREN; Depends on the national situation.

NRENs that own dark fiber can decide for themselves NRENs that own dark fiber can decide for themselves which technology and what speeds to use on it which technology and what speeds to use on it

10.0G

1.0G

0.1G

0.01G

Source: TERENA

Europe: Core Network Bandwidth Increase for Years 2001-2004 and 2004-2006

Europe: Core Network Bandwidth Increase for Years 2001-2004 and 2004-2006

1

10

100

1000

Incre

ase f

acto

r

Expected increase in two years

Increase factor for 2001-2004

Countries With No Increase Already Had 1-2.5G Backbone in 2001 These are all going to 10G backbones by 2006-7

Countries Showing the Largest Increase Are: PIONIER (Poland) from 155 Mbps to 10 Gbps capacity (64X) SANET (Slovakia) from 4 Mbps to 1 Gbps (250X).

Source: TERENA

1660 km of Dark Fiber CWDM Links, 1 to 4 Gbps (GbE) August 2002: Dark Fiber Link, to Austria April 2003: Dark Fiber Link to Czech Republic 2004: Dark Fiber Link to Poland Planning 10 Gbps Backbone

> 250X: 2002-2005

120km CBDF Cost 4 k

Per Month 1 GE Now;

10G Planned

T. Weis

Dark Fiber in Eastern Europe Poland: PIONIER (10-20G) Network

Dark Fiber in Eastern Europe Poland: PIONIER (10-20G) Network

2763 km Lit Fiber 2763 km Lit Fiber Connects 22 MANs; Connects 22 MANs;

+1286 km (9/05) +1286 km (9/05) + 1159 km (4Q/06)+ 1159 km (4Q/06)

Vision: Support - Vision: Support - Computational Grids;Computational Grids;

Domain-Specific Grids Domain-Specific Grids Digital LibrariesDigital Libraries Interactive TVInteractive TV

Add’l Fibers for Add’l Fibers for e-Regional Initiatives e-Regional Initiatives

10 Gb/s

MetropolitanAreaNetworks

1 Gb/s

10 Gb/s(2 lambdas)

GDAŃSK

POZNAŃ

ZIELONA GÓRA

KATOWICE

KRAKÓW

LUBLIN

WARSZAWA

BYDGOSZCZTORUŃ

CZĘSTOCHOWA

BIAŁYSTOK

OLSZTYN

RZESZÓW

BIELSKO-BIAŁA

GÉANT

KOSZALIN

SZCZECIN

WROCŁAW

ŁÓDŹ

KIELCE PUŁAWYOPOLE

RADOM

BASNET 34 Mb/s

CESNET, SANET

4Q05 Plan: Courtesy 4Q05 Plan: Courtesy M. PrzybylskiM. Przybylski

PIONIER Cross Border Dark Fiber Plan LocationsPIONIER Cross Border Dark Fiber Plan Locations

Single Single GEANT PoPGEANT PoPin Poznanin Poznan

CESNET2 (Czech Republic) Network Topology, Dec. 2004

2500+ km 2500+ km Leased Fibers Leased Fibers (Since 1999) (Since 1999)

2005:2005: 10GE Link Praha-Brno (300km) in Service; 10GE Link Praha-Brno (300km) in Service; Plan to go to 4 X 10G and higher as needed;Plan to go to 4 X 10G and higher as needed;

More 10GE links planned More 10GE links planned J. Gruntorad

APAN China ConsortiumAPAN China Consortium Established in 1999.  The China Education and Research Established in 1999.  The China Education and Research

Network (Network (CERNETCERNET) and the China Science and Technology ) and the China Science and Technology Network (Network (CSTNETCSTNET) are the main advanced networks. ) are the main advanced networks.

CERNet

2.5 Gbps

CSTnet

CERNET 2000: Own dark fiber crossing 30+

major cities and 30,000 kilometers 2003: 1300+ universities and

institutes, over 15 million users

CERNET 2: Next Generation R&E Net Backbone connects 15-20 Giga-

POPs at 2.5G-10Gbps (I2-like) Connects to 200 Universities and

100+ Research Institutes at 1 Gbps-10 Gbps

Native IPv6 and Lambda Networking

From 6 to 78 Million Internet Users in China from Jan. – July 2004J. P. Wu, H.

Chen

Brazil (RNP2): Rapid Backbone Progressand the GIGA Project

Brazil (RNP2): Rapid Backbone Progressand the GIGA Project

RNPRNP Connects the regional Connects the regional networks in all 26 states of Brazilnetworks in all 26 states of Brazil

Backbone on major links to 155 Backbone on major links to 155 Mbps; 622 Mbps Rio – Sao Paulo.Mbps; 622 Mbps Rio – Sao Paulo.

2.5G to 10G Core in 2005 2.5G to 10G Core in 2005 (300X Improvement in 2 Years)(300X Improvement in 2 Years)

The GIGA ProjectThe GIGA Project – Dark Fiber – Dark Fiber experimental networkexperimental network

700 km of fiber, 7 cities 700 km of fiber, 7 cities and 20 institutions in and 20 institutions in Sao Paolo and Rio Sao Paolo and Rio

GbE to Rio Tier-2, Sao GbE to Rio Tier-2, Sao Paulo Tier-3Paulo Tier-3

RNP & GIGA: RNP & GIGA: Extend GIGA Extend GIGA to the Northeast, with 4000 to the Northeast, with 4000 km of dark fiber by 2008km of dark fiber by 2008

L. Lopez

DFN (Germany): X-WiN-Fiber Network

Faser KPNFaser GLFaser GCFaser vorhanden

GAR

ERL

BAY

MUE

FZJ

AAC BIR

DES

HAM

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JENWEI

ESF

HUB

ADH

Most of the Most of the X-WiNX-WiN core core will be a fibre network, will be a fibre network, (see map), the rest will (see map), the rest will be provided by be provided by wavelengthswavelengths

Several fibre and Several fibre and wavelengths providerswavelengths providers

Fibre is relatively cheap Fibre is relatively cheap – in most cases more – in most cases more economic than (one) economic than (one) wavelengthwavelength

X-Win creates many new X-Win creates many new options besides being options besides being cheaper than the current cheaper than the current G-WiN coreG-WiN core

13.04.2005

K. Schauerhammer

Romania: Inter-City Links were 2 to 6 Mbps in 2002; Improved to 155 Mbps in 2003-2004;

GEANT-Bucharest Link: 155 to 622 Mbps

Romania: Inter-City Links were 2 to 6 Mbps in 2002; Improved to 155 Mbps in 2003-2004;

GEANT-Bucharest Link: 155 to 622 Mbps

RoEduNetJanuary 2005

Plan: 3-4 Centers at 2.5 Gbps; Dark Fiber InterCity Backbone

Compare Pk: 56 univ. share 155 Mbps Internationally

N. Tapus T. Ul Haq

ICFA Report: Networks for HENPICFA Report: Networks for HENPGeneral ConclusionsGeneral Conclusions

ICFA Report: Networks for HENPICFA Report: Networks for HENPGeneral ConclusionsGeneral Conclusions

Reliable high Reliable high End-to-end PerformanceEnd-to-end Performance of networked applications such of networked applications such as Data Grids is required. Achieving this requires:as Data Grids is required. Achieving this requires:A coherent approach to End-to-end monitoring in all regions A coherent approach to End-to-end monitoring in all regions

that allows scientists throughout the world to extract clear that allows scientists throughout the world to extract clear information information

Upgrading campus infrastructures.Upgrading campus infrastructures. To support Gbps flows to HEP centers. To support Gbps flows to HEP centers.

Removing local, last mile, and nat’l and int’l bottlenecks Removing local, last mile, and nat’l and int’l bottlenecks end-to-end, whether technical or political in origin.end-to-end, whether technical or political in origin.

Bandwidth Bandwidth across borders, the countryside or the city isacross borders, the countryside or the city isoften much less than on national backbones and int’l linksoften much less than on national backbones and int’l links

This problem is very widespread in our community: This problem is very widespread in our community: Examples stretching from the Asia Pacific to Latin Examples stretching from the Asia Pacific to Latin America America to the Northeastern US. Root causes vary, from to the Northeastern US. Root causes vary, from lack of lack of local infrastructure, to unfavorable policies and local infrastructure, to unfavorable policies and pricing pricing

Situation of Local Access in Belém in Brazil in 2004

Situation of Local Access in Belém in Brazil in 2004InstitutionInstitution Summary of local network connectionsSummary of local network connections Annual Annual

Cost (US$)Cost (US$)

CEFET CEFET Access to provider at 512 kbpsAccess to provider at 512 kbps 22,20022,200

CESUPA CESUPA (4 campi)(4 campi)

Internal + access to provider at 6 MbpsInternal + access to provider at 6 Mbps 57,80057,800

IEC/MS IEC/MS (2 campi)(2 campi)

Internal at 512 kbps + Access to provider at Internal at 512 kbps + Access to provider at 512 kbps512 kbps

13,30013,300

MPEG MPEG (2 campi)(2 campi)

Internal at 256 kbps; Access at 34 Mbps Internal at 256 kbps; Access at 34 Mbps (radio link)(radio link)

7,6007,600

UEPA UEPA (5 campi)(5 campi)

Internal at 128 kbps; Access at 512 kbpsInternal at 128 kbps; Access at 512 kbps 18,50018,500

UFPA UFPA (4 campi)(4 campi)

Internal at 128 kbps; Provider PoPInternal at 128 kbps; Provider PoP 16,70016,700

UFRAUFRA Access to provider at 1 MbpsAccess to provider at 1 Mbps 16,00016,000

UNAMA UNAMA (4 campi)(4 campi)

Internal wireless links, access at 6 MbpsInternal wireless links, access at 6 Mbps 88,90088,900

Annual telco charges for POOR local access = US$ 241,000

Belém: a Possible Topology(30 km ring)

Belém: a Possible Topology(30 km ring)

Alternative Approach in Brazil – Do It Yourself (DIY) Networking (M. Stanton, RNP)

Alternative Approach in Brazil – Do It Yourself (DIY) Networking (M. Stanton, RNP)

1.1. Form a consortium for joint network provisionForm a consortium for joint network provision

2.2. Build your own optical fibre network to reach ALL the campi Build your own optical fibre network to reach ALL the campi of ALL consortium members of ALL consortium members

3.3. Light it up and go!Light it up and go!

Costs involved:Costs involved: Building out the fibre: using utility poles of electric companyBuilding out the fibre: using utility poles of electric company

US$ 7,000 per kmUS$ 7,000 per km Monthly rental of US $1 per pole (~25 poles per km)Monthly rental of US $1 per pole (~25 poles per km)

Equipment costs: mostly use cheap 2 port GbE switchesEquipment costs: mostly use cheap 2 port GbE switches Operation and maintenanceOperation and maintenance

In Belém for 11 institutions using In Belém for 11 institutions using All GigEAll GigE connections: connections: Capital costs around US $ 500,000Capital costs around US $ 500,000 Running costs around US $ 40,000 p.a.Running costs around US $ 40,000 p.a. Compare with current US $ 240,000 p.a. for traditional telco Compare with current US $ 240,000 p.a. for traditional telco

solution solution [for 0.128 to 6 Mbps: ~1000X less bandwidth[for 0.128 to 6 Mbps: ~1000X less bandwidth]]

Brazil: RNP Nat’l Plan for Optical Metro Nets in 2005-6

Brazil: RNP Nat’l Plan for Optical Metro Nets in 2005-6

In December 2004, RNP signed contracts with In December 2004, RNP signed contracts with

“Finep” (the agency of the Ministry of Science and “Finep” (the agency of the Ministry of Science and

Technology) to Technology) to build optical metro networks in all 27 build optical metro networks in all 27

capital cities in Brazil capital cities in Brazil

Total value of more than US$15 millionsTotal value of more than US$15 millions

Most of this money will be spent in 2005Most of this money will be spent in 2005

M. Stanton

G. Cole

SegmentSegment CurrentCurrent Year 1Year 1 Year 2Year 2 Year 3Year 3 Year 4Year 4 Year Year 55

1 - Trans-1 - Trans-AsiaAsia

155 Mbps155 Mbps 2.5 Gbps (US-2.5 Gbps (US-China),10 Gbps China),10 Gbps (US-Korea-(US-Korea-China)China)

2 x 10 Gbps 2 x 10 Gbps US-China; US-China; US-Korea US-Korea -China -China

2 x 10 2 x 10 GbpsGbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

2 - Trans-2 - Trans-ChinaChina

2.5 Gbps (155 2.5 Gbps (155 Mbps, Beijing-Mbps, Beijing-Khabarovsk)Khabarovsk)

2.5 Gbps2.5 Gbps 1 x 10 Gbps1 x 10 Gbps 2 x 10 2 x 10 GbpsGbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

3 - Trans-3 - Trans-RussiaRussia

155 Mbps155 Mbps 622 Mbps622 Mbps 2.5 Gbps2.5 Gbps 1 x 10 1 x 10 Gbps Gbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

4 - Trans-4 - Trans-EuropeEurope

622 Mbps622 Mbps 622 Mbps622 Mbps 622 Mbps622 Mbps 2 x 10 2 x 10 GbpsGbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

5 - Trans-5 - Trans-AtlanticAtlantic

622 Mbps622 Mbps 1 Gbps1 Gbps 1 x 10 Gbps1 x 10 Gbps 2 x 10 2 x 10 GbpsGbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

6 - Trans-6 - Trans-North North AmericaAmerica

2.5G (Asia- 2.5G (Asia- Chicago), GbE Chicago), GbE NYC-Chicago NYC-Chicago (via CANARIE)(via CANARIE)

10 Gbps, Seattle-10 Gbps, Seattle-Chicago-NYCChicago-NYC

10 Gbps, 10 Gbps, Seattle-Seattle-Chicago-NYCChicago-NYC

2 x 10 2 x 10 GbpsGbps

N x 10 N x 10 GbpsGbps

N x N x 10 10 GbpsGbps

SeattleChicago

NYC

Amsterdam

Moscow

Novosibirsk Khabarovsk

Beijing

Hong Kong

GLORIAD Topology – Current, Plans for Years 1-5

Pusan

Closing the Digital Divide: R&E Networks in/to Latin America

RNP2 and the GIGA Project RNP2 and the GIGA Project (ANSP)(ANSP)

AmPathAmPath: 45 Mbps Links : 45 Mbps Links to US (2002-5)to US (2002-5)

CHEPREO (NSF, from 2004):CHEPREO (NSF, from 2004): 622 Mbps Sao Paulo – Miami622 Mbps Sao Paulo – Miami

WHREN/LILA (NSF, from 2005)WHREN/LILA (NSF, from 2005)0.6 to 2.5 G Ring by 20060.6 to 2.5 G Ring by 2006Connections to Connections to

Pacific & Atlantic Wave Pacific & Atlantic WaveRedCLARA (EU):RedCLARA (EU): Connects Connects

18 Latin Am. NRENs, Cuba; 18 Latin Am. NRENs, Cuba; 622 Mbps Link to Europe622 Mbps Link to Europe

PacificWave

Sao PauloANSP

AtlanticWave

To GEANT622 Mbps

GOAL: An GOAL: An Information Society:Information Society:“… One in which highly developed “… One in which highly developed networks, equitable and ubiquitous networks, equitable and ubiquitous access to information, appropriate access to information, appropriate content in accessible formats, content in accessible formats, and effective communication can and effective communication can help people achieve their potential”help people achieve their potential”

Role of Science in the Information Society; WSIS 2003-2005

Role of Science in the Information Society; WSIS 2003-2005

HEP Active in WSIS I, GenevaHEP Active in WSIS I, GenevaTheme: Theme: “Creating a “Creating a

Sustainable Process of Sustainable Process of Innovation” Innovation”

CERN RSIS EventCERN RSIS EventSIS Forum & CERN/Caltech SIS Forum & CERN/Caltech

Online Stand at WSIS I (12/03) Online Stand at WSIS I (12/03) > 50 Demos: Advanced Nets > 50 Demos: Advanced Nets & Grids, Global VideoConf., & Grids, Global VideoConf., Telesurgery, “Music Grids”…Telesurgery, “Music Grids”…

Visitors at WSIS I: Visitors at WSIS I: Kofi Annan, UN Sec’y Gen’l Kofi Annan, UN Sec’y Gen’l John H. Marburger, Science John H. Marburger, Science

Adviser to US PresidentAdviser to US PresidentIon Iliescu, President of Ion Iliescu, President of Romania; Romania;

and Dan Nica, and Dan Nica, Minister of ICTMinister of ICT……

World Conference on PhysicsWorld Conference on Physics and Sustainable Developmentand Sustainable Development

Durban, South Africa 10/31-11/2/05Durban, South Africa 10/31-11/2/05

““The World Conference will serve as the first The World Conference will serve as the first global forum to focus the physics global forum to focus the physics

community toward development goals and to community toward development goals and to create new mechanisms of cooperation create new mechanisms of cooperation

toward their achievement.”toward their achievement.”www.saip.org.za/physics2005/WCPSD2005.htmlwww.saip.org.za/physics2005/WCPSD2005.html

WSIS II: TUNIS WSIS II: TUNIS 11/16-11/18/200511/16-11/18/2005www.itu.int/wsiswww.itu.int/wsis

Networks and Grids for HEP and Data Intensive Global Science

Networks and Grids for HEP and Data Intensive Global Science

Networks used by HEP and other fields of DIS are advancing rapidlyNetworks used by HEP and other fields of DIS are advancing rapidly To the 10 G range and now N X 10G; much faster than Moore’s LawTo the 10 G range and now N X 10G; much faster than Moore’s Law New HENP and DOE Roadmaps: Factor ~1000 BW Growth/DecadeNew HENP and DOE Roadmaps: Factor ~1000 BW Growth/Decade

HEP & CS are learning to use long range 10 Gbps networks effectively HEP & CS are learning to use long range 10 Gbps networks effectively 2004-5: 7+ Gbps TCP flows over 20+ kkm; 101 Gbps Record2004-5: 7+ Gbps TCP flows over 20+ kkm; 101 Gbps Record

Transition to community-operated/owned optical R&E networks Transition to community-operated/owned optical R&E networks (us, ca, nl, jp, kr; (us, ca, nl, jp, kr; pl, cz, br, sk, pt, ei, gr, …pl, cz, br, sk, pt, ei, gr, … ) is underway ) is underway

A new era of “hybrid” optical networks & Grid systems is emergingA new era of “hybrid” optical networks & Grid systems is emerging We Must Work to Close to Digital Divide, from Several PerspectivesWe Must Work to Close to Digital Divide, from Several Perspectives

To Allow Scientists in All World Regions to Take Part in DiscoveriesTo Allow Scientists in All World Regions to Take Part in Discoveries Removing Regional, Last Mile, Local Bottlenecks and Removing Regional, Last Mile, Local Bottlenecks and

Compromises in Network Quality are Compromises in Network Quality are On the Critical PathOn the Critical Path Important ExamplesImportant Examples on the Road to Closing the Digital Divide on the Road to Closing the Digital Divide

GLORIAD (US-Russia-China-Korea) Global Optical RingGLORIAD (US-Russia-China-Korea) Global Optical Ring IEEAF “Global Quilt”; NSF: IRNC and IEEAF “Global Quilt”; NSF: IRNC and New Initiative on AfricaNew Initiative on Africa CHEPREO, WHREN and the Brazil HEPGrid in Latin AmericaCHEPREO, WHREN and the Brazil HEPGrid in Latin America Leadership & Outreach: HEP Groups in US, EU, Japan, Korea,Leadership & Outreach: HEP Groups in US, EU, Japan, Korea,

Latin America Latin America

AcknowledgementsAcknowledgements R. AikenR. Aiken A. AliA. Ali S. AltmanovaS. Altmanova P. AveryP. Avery J. BoroumandJ. Boroumand J. BakkenJ. Bakken L. BauerdickL. Bauerdick J. BunnJ. Bunn R. CavanaughR. Cavanaugh H. S. ChenH. S. Chen K. ChoK. Cho G. ColeG. Cole L. CottrellL. Cottrell D. DavidsD. Davids H. DoebblingH. Doebbling J. DolgonasJ. Dolgonas E. FantegrossiE. Fantegrossi D. FosterD. Foster I. FosterI. Foster P. GalvezP. Galvez

J. GruntoradJ. Gruntorad J. IbarraJ. Ibarra V. IlyinV. Ilyin W. JohnstonW. Johnston Y. KaritaY. Karita D. Y. KimD. Y. Kim Y. KimY. Kim K. KwonK. Kwon I. LegrandI. Legrand M. LivnyM. Livny S. LowS. Low O. MartinO. Martin R. MountR. Mount S. McKeeS. McKee G. McLaughlinG. McLaughlin D. Nae D. Nae K. NeggersK. Neggers S. NovaesS. Novaes J. PoolJ. Pool D. PetravickD. Petravick S. RavotS. Ravot

D. ReeseD. Reese D. RileyD. Riley A. SantoroA. Santoro K. SchauerhammerK. Schauerhammer C. SmithC. Smith D. SonD. Son M. StantonM. Stanton C. SteenbergC. Steenberg X. SuX. Su R. SummerhillR. Summerhill M. ThomasM. Thomas F. Van LingenF. Van Lingen E. VelikhovE. Velikhov D. WalstenD. Walsten T. WeisT. Weis T. WestT. West D. WilliamsD. Williams V. WhiteV. White J. P. WuJ. P. Wu F. WuerthweinF. Wuerthwein Y. XiaY. Xia

US DOEUS DOE US NSFUS NSF ESnetESnet EuropeanEuropean

CommissionCommission CERNCERN SLACSLAC FermilabFermilab CACRCACR NLRNLR CENICCENIC Internet2Internet2

/HOPI /HOPI FLRFLR UltraIightUltraIight StarlightStarlight KISTI, KAISTKISTI, KAIST RNP, ANSPRNP, ANSP CiscoCisco NeterionNeterion

Some Extra Slides

Follow

Some Extra Slides

Follow

Int’l Networks BW on Major Links for HENP: US-CERN Example

Int’l Networks BW on Major Links for HENP: US-CERN Example

Rate of Progress >> Moore’s Law (US-CERN Example)Rate of Progress >> Moore’s Law (US-CERN Example) 9.6 kbps Analog9.6 kbps Analog 1985) 1985) 64-256 kbps Digital 64-256 kbps Digital 1989 - 1994) 1989 - 1994) [X 7 – 27][X 7 – 27] 1.5 Mbps Shared 1.5 Mbps Shared 1990-3; IBM) 1990-3; IBM) [X 160][X 160] 2 -4 Mbps2 -4 Mbps 1996-1998) 1996-1998) [X 200-400][X 200-400] 12-20 Mbps 12-20 Mbps 1999-2000) 1999-2000) [X 1.2k-2k][X 1.2k-2k] 155-310 Mbps155-310 Mbps 2001-2) 2001-2) [X 16k – [X 16k –

32k]32k] 622 Mbps622 Mbps 2002-3) 2002-3) [X 65k][X 65k] 2.5 Gbps (2.5 Gbps () ) 2003-4 2003-4 [X 250k][X 250k] 10 Gbps 10 Gbps 2004-5 2004-5 [X 1M] [X 1M] 2 x 10 Gbps 2 x 10 Gbps 2005-6 2005-6 [X 2M][X 2M] 4 4 x 10 Gbps x 10 Gbps ~2007-8~2007-8 [X 4M][X 4M] 8 x 10 Gbps or 2x40Gbps 8 x 10 Gbps or 2x40Gbps ~2009-10~2009-10 [X 8M][X 8M]

A factor of ~1M Bandwidth Improvement over A factor of ~1M Bandwidth Improvement over 1985-2005; A factor of ~5k during 1995-20051985-2005; A factor of ~5k during 1995-2005

HENP has become a leading applications driver, HENP has become a leading applications driver, and also a co-developer of global networks and also a co-developer of global networks

Internet Growth in the World At Large

Internet Growth in the World At Large

Amsterdam Internet Exchange Point Example

20 Gbps

60 Gbps

Average

5 Minute

Max

Some Annual Growth Spurts;

Typically In Summer-Fall

“Acceleration” Last Summer The Rate of HENP Network Usage Growth

(> 100% Per Year) is Not Unlike the World at Large

66 G

bp

s

40 Gbps

Brazil: National backbone in 2003Brazil: National backbone in 2003

RNP2 – May 2003

( 30 Mbps)

M. Stanton

BRAZIL: Nat’l Backbone May 2005BRAZIL: Nat’l Backbone May 2005

RNP – May 2005( 622 Mbps)

BRAZIL: RNPng – 10G & 2.5G Core Network (3Q2005)

BRAZIL: RNPng – 10G & 2.5G Core Network (3Q2005)

Fortaleza

Recife

Salvador

Rio de Janeiro

Belo Horizonte

Brasília

São PauloCuritiba

Florianópolis

Porto Alegre

2.5 Gbps10 Gbps

Core Network Tender

Completed May 12th-13th

The classical Grid architecture had a number of implicit The classical Grid architecture had a number of implicit assumptionsassumptions

The ability to locate and schedule suitable resources,The ability to locate and schedule suitable resources, within a tolerably short time (i.e. resource richness) within a tolerably short time (i.e. resource richness)

Short transactions with relatively simple failure modesShort transactions with relatively simple failure modesHENP Grids are HENP Grids are Data Intensive & Resource-ConstrainedData Intensive & Resource-Constrained

Resource usage governed by local and global policiesResource usage governed by local and global policies Long transactions; some long queuesLong transactions; some long queues

Grid Analysis:Grid Analysis: 1000s of users compete for resources 1000s of users compete for resources at dozens of sites: Complex scheduling; management at dozens of sites: Complex scheduling; management

HENP HENP Stateful, End-to-end Monitored and Tracked ParadigmStateful, End-to-end Monitored and Tracked Paradigm Adopted in OGSA, Now WS Resource FrameworkAdopted in OGSA, Now WS Resource Framework

HENP Data Grids, and Now HENP Data Grids, and Now Services-Oriented GridsServices-Oriented Grids

Grid Analysis: A Real Time SOA Enabling Global Communities

Grid Analysis: A Real Time SOA Enabling Global Communities

Analysis Clients talk standard protocols to the Clarens data/services portal; hides complexity.

Simple Web service API allows diverse Analysis Clients (simple or complex)

Clarens Servers Autodiscover, Auto-connect to form a “Services Fabric”

Key features: Global Scheduler, Catalogs, Monitoring, and Strategic Grid-wide Execution service.

SchedulerCatalogs

Analysis Client

Grid ServicesWeb Server

ExecutionPriority

Manager

Grid WideExecutionService

DataManagement

Fully-ConcretePlanner

Analysis Client

AnalysisClient

Virtual

Data

Replica

ApplicationsMonitoring

Partially-AbstractPlanner

Metadata

HTTP, SOAP, XML-RPC

MonAL

ISA

Clar

ens

ROOT-

Clar

ens/

Cojac/

IGUAN

A

BOSS

RefD

B/MOPD

B

POOL

ORCA

ROOT

FAMOS

Fully-AbstractPlanner

VDT-

Serv

er

Chim

era

MCRun

jobSp

hinx

F. Van Lingen,

M. Thomas

Next generation Information System, with the network as an integrated, Next generation Information System, with the network as an integrated, actively managed subsystem in a global Gridactively managed subsystem in a global Grid

Hybrid network infrastructure: packet-switched + dynamic optical paths Hybrid network infrastructure: packet-switched + dynamic optical paths End-to-end monitoring; Realtime tracking and optimization; End-to-end monitoring; Realtime tracking and optimization;

Dynamic bandwidth provisioning; Dynamic bandwidth provisioning; Agent-based services spanning all layersAgent-based services spanning all layers

http://http://ultralight.caltech.eduultralight.caltech.edu

UERJ, USP

CHEPREO

KNU10

Gbps

Caltech, UF, UMich, FNAL, SLAC, CERN,

UERJ (Rio), USP (Sao Paulo), FIU,KNU (Korea), KEK

NLR, CENIC, UCAID, Translight, UKLight, Netherlight, UvA; UCLondon, Taiwan

Cisco

S. McKee; G. Karmous-Edwards

LISA- Localhost Information Service Agent End To End Monitoring Tool

LISA- Localhost Information Service Agent End To End Monitoring Tool

Easy to deploy & install with any browser; Easy to deploy & install with any browser; user friendly GUI user friendly GUI

Detects system architecture, OS Detects system architecture, OS For all versions of Windows, Linux, Mac.For all versions of Windows, Linux, Mac.

Complete system monitoring of the hostComplete system monitoring of the host CPU, memory, IO, disk, …CPU, memory, IO, disk, …

Hardware detection including Audio, Video Hardware detection including Audio, Video equipment; drivers installed in the system equipment; drivers installed in the system

Provides embedded clients for IPERF Provides embedded clients for IPERF (or other net monitoring tools, e.g. Web 100 ) (or other net monitoring tools, e.g. Web 100 )

LISA and ApMon2:LISA and ApMon2: A basis for strategic, A basis for strategic, end-to-end managed Gridsend-to-end managed Grids

Complete, lightweight monitoring of end user systems & network connectivity. Uses MonALISA framework

to optimize client applications.

Try it: http://monalisa.caltech.edu/lisa/lisa.jnlpTry it: http://monalisa.caltech.edu/lisa/lisa.jnlp I. Legrand

ApMon Grid Analysis Demo: Distributed Processing and Monitoring at SC2004

ApMon Grid Analysis Demo: Distributed Processing and Monitoring at SC2004

Demonstrated Demonstrated how CMS analysis jobs can be submitted to how CMS analysis jobs can be submitted to multiple sites, and monitored from anywhere else on the multiple sites, and monitored from anywhere else on the

gridgrid Three sites: Caltech, Three sites: Caltech, UERJ and USPUERJ and USP Using the “BOSS” job submission and tracking tool,Using the “BOSS” job submission and tracking tool,

running as a “Clarens” Grid service running as a “Clarens” Grid service Job status is monitored by MonALISA, and visualized usingJob status is monitored by MonALISA, and visualized using

(netlogger-style) “lifelines” in real time(netlogger-style) “lifelines” in real time

M. Thomas