Embed Size (px)
Citation preview
International Computing e-Infrastructures: Past, Present and
Future...
International Computing e-Infrastructures: Past, Present and
Future...
Fabrizio GagliardiEMEA DirectorTechnical ComputingMicrosoft Corporation
Fabrizio GagliardiEMEA DirectorTechnical ComputingMicrosoft Corporation
CGW'05 2
Outline of the talkOutline of the talk
Some personal introductory remarks
Definition of e-Infrastructures
Need for e-Infrastructures
Recent past history
Current situation, accomplishments and challenges
Outlook for the future
Some personal introductory remarks
Definition of e-Infrastructures
Need for e-Infrastructures
Recent past history
Current situation, accomplishments and challenges
Outlook for the future
CGW'05 3
Definition of e-InfrastructuresDefinition of e-Infrastructures
Infrastructures to support wide geographically distributed communities which share problems and resources to work towards common goals
Leveraging international network interconnectivity
Based on safe AAA architecture
Need persistent software middleware (S/W is integral part of the infrastructure)
Infrastructures to support wide geographically distributed communities which share problems and resources to work towards common goals
Leveraging international network interconnectivity
Based on safe AAA architecture
Need persistent software middleware (S/W is integral part of the infrastructure)
CGW'05 4
‘Grids’: A Catch-All Marketing Term‘Grids’: A Catch-All Marketing Term
‘Grids’ mean many different things to many different people/companies:
P2P desktop cycle-stealing
Linked Supercomputer Centers
Managed virtual distributed clusters
Internet access to giant, distributed repositories
Virtualization of data center IT resources
Out-sourcing to “utility compute centers”
Sharing resources distributed among different administrative domains (Ian Foster)
For Microsoft, Grids are about Data Management as much as Compute Cycles
‘Grids’ mean many different things to many different people/companies:
P2P desktop cycle-stealing
Linked Supercomputer Centers
Managed virtual distributed clusters
Internet access to giant, distributed repositories
Virtualization of data center IT resources
Out-sourcing to “utility compute centers”
Sharing resources distributed among different administrative domains (Ian Foster)
For Microsoft, Grids are about Data Management as much as Compute Cycles
CGW'05 5
Need for e-InfrastructuresNeed for e-Infrastructures
Science, industry and commerce are more and more digital, process vast amounts of data and need massive computing power
We live in a “flat” world:Science is more and more an international collaboration and often requires a multidisciplinary approach
Need to use technology for the good causeFight Digital/Divide
Industrial uptake has become essential
Science, industry and commerce are more and more digital, process vast amounts of data and need massive computing power
We live in a “flat” world:Science is more and more an international collaboration and often requires a multidisciplinary approach
Need to use technology for the good causeFight Digital/Divide
Industrial uptake has become essential
CGW'05 6
Recent past historyRecent past history
Meta-computing and distributed computing early examples in the 80’ and 90’ (CASA, I-Way, Unicore, Condor etc.)
EU-US workshop in Annapolis in 1999 on large scientific data bases: http://www.cacr.caltech.edu/euus/
EU FP5 and US Trillium and national Grids
EU FP6, US OSG, NAREGI/Japan…
Meta-computing and distributed computing early examples in the 80’ and 90’ (CASA, I-Way, Unicore, Condor etc.)
EU-US workshop in Annapolis in 1999 on large scientific data bases: http://www.cacr.caltech.edu/euus/
EU FP5 and US Trillium and national Grids
EU FP6, US OSG, NAREGI/Japan…
CGW'05 7
ChronologyChronology
EGEE-XX 2008?EGEE start 2004
EDG start 2001
CHEP2000
EGEE-II start 2006
We are here
IGO-----ETICS/EGO----
CGW'05 8
“Unlimited” bandwidth, breaking the frontiers of computing: the path in Europe from FP5 to FP6.
ARCADE 2002
Barcelona, 14th February 2002
"The views expressed in this presentation are those of the author and do not necessarily reflect the views of the European Commission"
Antonella KarlsonResearch Networks Unit, DG INFSO, EC
CGW'05 9
GRIDs - IST projects (~36m Euro)An integrated approach
ScienceIndustry / business
ApplicationsApplications
MiddlewareMiddleware& Tools& Tools
Underlying Underlying InfrastructuresInfrastructures
CROSSGRID
DATAGRID
DATATAG
GRIDLAB
EGSO
GRIA
GRIP EUROGRID
DAMIENG
RID
STA
RT
CGW'05 10
DATAGRID, CROSSGRID DATAGRID, CROSSGRID DATAGRID, CROSSGRID DATAGRID, CROSSGRID
GRIDs:Examples of large testbeds
• 17 European countries
• Collaboration of more than 2000 scientists
Application requirements:• Computing > 20 TFlops/s• Downloads > 0.5PBytes• Network speeds at 10
Gbps
GEANT
INFRASTRUCTURE
CGW'05 11
DATATAG (cross-Atlantic testbed)DATATAG (cross-Atlantic testbed) DATATAG (cross-Atlantic testbed)DATATAG (cross-Atlantic testbed)
GRIDs:Examples of large testbeds
(2Gbps)
Links with US projects
(GriPhyN, PPDG, iVDGL,…)
CGW'05 12
Current situation: accomplishments and challengesCurrent situation: accomplishments and challenges
Many Grids around the world, very few maintained as a persistent infrastructure (best example is the “secret” Google Grid)
Need for public and open Grids (OSG, EGEE and related projects, NAREGI, and TERAGRID, DEISA good prototypes)
Persistence, support, sustainability, long term funding, easy access are the major challenges
Many Grids around the world, very few maintained as a persistent infrastructure (best example is the “secret” Google Grid)
Need for public and open Grids (OSG, EGEE and related projects, NAREGI, and TERAGRID, DEISA good prototypes)
Persistence, support, sustainability, long term funding, easy access are the major challenges
CGW'05 13
Enabling Grids for E-sciencE
INFSO-RI-508833
Projects in Europe (I)
• Access to IT-resources (connectivity, computing, data, instrumentation…) for scientists: – Providing e-Infrastructure
Géant2 EGEE DEISA SEE-GRID
– Benefiting from e-Infrastructure DILIGENT SIMDAT GRIDCC CoreGRID GridLab
– Concertation: GRIDSTART, GridCoord
– Grid mobility: Akogrimo
CGW'05 14
Enabling Grids for E-sciencE
INFSO-RI-508833
Projects in Europe (II)
• Sample of National Grid projects: – Austrian Grid Initiative– DutchGrid – France:
e-Toile ACI Grid
– Germany D-Grid Unicore
– Grid Ireland – Italy
INFNGrid GRID.IT
– NorduGrid – UK e-Science
National Grid Service OMII GridPP project
D-GRID
CGW'05 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Policy Forums
• The e-Infrastructures Reflection Group (eIRG)– Mission: study and promote policies for easy and cost-effective shared
use of electronic resources in Europe– 25 countries (government-appointed representatives), EU: 2 members– White Papers
• European Strategy Forum on Research Infrastructures (ESFRI)– Role: to support a coherent approach to policy-making on research
infrastructures in Europe, and to act as an incubator for international negotiations about concrete initiatives
– representatives of the 25 EU Member States, appointed by Research Ministers and a representative of the European Commission
• ESFRI + eIRG: European roadmap for new research infrastructures of pan-European interest (10-20 years)
CGW'05 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Géant2
• GÉANT2 is the 7th generation of the pan-European research and education network, successor to the multi-gigabit research network GÉANT. – Official start: 1 September 2004, Duration: 4 years– Funding: EC, national research, education networks– Managed by DANTE
• Goal: – To connect 34 countries through 30 national research and
education networks (NRENs) – using multiple 10Gbps wavelengths
• Status: – Equipment and services currently in operations (officially
inaugurated by Commissioner Reding last June)– Transition from GÉANT network to GÉANT2 gradually
completing, started in the first quarter of 2005
CGW'05 17
Enabling Grids for E-sciencE
INFSO-RI-508833
EU Grid technology & infrastructure (I)
New Grid Research Projects in FP6EU Funding:52 MILLION - Start: SUMMER 2004EU Funding:52 MILLION - Start: SUMMER 2004
inteliGRIDSemantic Grid based virtual organisations
ProvenanceTrust and provenance
for Grids
DataminingGridDatamining
tools & services
UniGridSExtended OGSA
Implementation based on UNICORE
K-WF GridKnowledge based
workflow & collaboration
GRIDCOORDBuilding the ERA in Grid research
OntoGridKnowledge Services for the semantic Grid
HPC4UFault tolerance,dependability
for Grid
Grid-based generic enabling application technologies to facilitate
solution of industrial problemsSIMDAT
EU-driven Grid services architecture for businesS
and industry NextGRID
Mobile Grid architecture and services for dynamic
virtual organisations Akogrimo
European-wide virtual laboratory for longer term Grid research-creating the foundation for next generation Grids
CoreGRID
Specific support action Integrated project Network of excellence Specific targeted research project
From a talk by Ulf Dahlsten, Den Haag, Nov 2004
CGW'05 18
Enabling Grids for E-sciencE
INFSO-RI-508833
EU Grid technology & infrastructure (II)
Building the European eInfrastructure for research20002000 20012001 20022002 20032003 20042004 20052005 20062006 20072007 20082008
FP5FP5FP6FP6
Complementary to National infrastructuresComplementary to National infrastructures
TEN 155 network
GÉANT network (FP6)
Grid enabled Infrastructures (EGEE, DEISA, SEE-GRID,…)
(other) testbeds
IPv6 actionsIPv6 testbeds
GÉANT network
Grid testbeds
(other) testbeds
IPv6 testbeds
TEN 155 network
FP7
From a talk by Ulf Dahlsten, Den Haag, Nov 2004
CGW'05 19
Enabling Grids for E-sciencE
INFSO-RI-508833
EU Grid technology & infrastructure (III)
eInfrastructure – Testbeds
LOBSTER
Traffic monitoring
EUROLABS
Experimental testbeds
IPv6TF SC
IPv6 Task Force
supportSpecific Support Actions
User involvement… …technology validation
eInfrastru
ctureEGEE
DEISA
S
EE-GRID
DILIGENT
New
use
r co
mm
un
itie
s u
sin
g G
rid
s –
Dig
ital
L
ibra
ries
GRIDCC
Rea
l ti
me
Gri
d f
or
rem
ote
co
ntr
ol
of
inst
rum
ents
MUPPET
Op
tica
l so
luti
on
s fo
r G
rid
in
fras
tru
ct.
EUQoS
Fle
xib
le
Qu
alit
y o
f S
ervi
ce
Ass
ura
nce
Under
negotiation
Courtesy of K. Baxevanidis, EU
CGW'05 20
Enabling Grids for E-sciencE
INFSO-RI-508833
EU Grid technology & infrastructure (V)
• eInfrastructure – achievements• Connectivity service
• Computing, storage service
• Testbeds
• International links
• GÉANT network: 10Gbit/s, IPv6 enabled, 3900 Research Centres connected
• EGEE: production quality, >10000 CPUs, >5PB storage, training, coverage of 27 countries
• DEISA: Supercomputer network, reaching 40 Tflop/s
• Rich set of technologies tested/ verified (IPv6, Grids, Optical, End-to-End QoS, Security, Mobility…) and communities involved (scientific, industry)
• USA, Russia, Mediterranean, Asia, Latin America...
From a talk by Ulf Dahlsten, Den Haag, Nov 2004
CGW'05 21
Outlook for the futureOutlook for the future
Outlook for the futureOutlook for the future
CGW'05 22
Supercomputing Goes PersonalSupercomputing Goes Personal
1991 1998 2005System Cray Y-MP C916 Sun HPC10000 Shuttle @ NewEgg.com
Architecture 16 x Vector4GB, Bus
24 x 333MHz Ultra-SPARCII, 24GB, SBus
4 x 2.2GHz x644GB, GigE
OS UNICOS Solaris 2.5.1 Windows Server 2003 SP1
GFlops ~10 ~10 ~10
Top500 # 1 500 N/A
Price $40,000,000 $1,000,000 (40x drop) < $4,000 (250x drop)
Customers Government Labs Large Enterprises Every Engineer & Scientist
Applications Classified, Climate, Physics Research
Manufacturing, Energy, Finance, Telecom
Bioinformatics, Materials Sciences, Digital Media
CGW'05 23
The Continuing Trend Towards Decentralized, Networked ResourcesThe Continuing Trend Towards Decentralized, Networked Resources
Grids of personal &
departmental clusters
Personal workstations &
departmental servers
Minicomputers
Mainframes
CGW'05 24
Leverage IT Industry’s Existing R&D Leverage IT Industry’s Existing R&DParallel applications development
High-productivity IDEsIntegrated debugging/profiling/tracing/analysisCode designer wizards
Concurrent programming frameworks
Platform optimizationsDynamic, profile-guided optimizationNew programming abstractions
Distributed systems issuesWeb Services & HPC grids
SecurityInteroperabilityScalability
Dynamic Systems ManagementSelf (re)configuration & tuningReliability & availability
RDMS + data miningEase-of-useAdvanced indexing & query processingAdvanced data mining algorithms
Parallel applications developmentHigh-productivity IDEs
Integrated debugging/profiling/tracing/analysisCode designer wizards
Concurrent programming frameworks
Platform optimizationsDynamic, profile-guided optimizationNew programming abstractions
Distributed systems issuesWeb Services & HPC grids
SecurityInteroperabilityScalability
Dynamic Systems ManagementSelf (re)configuration & tuningReliability & availability
RDMS + data miningEase-of-useAdvanced indexing & query processingAdvanced data mining algorithms
Digital experimentationCollaboration-enhanced Office productivity tools
Structure experiment data and derived results in a manner appropriate for human reading/reasoning (as opposed to optimizing for query processing and/or storage efficiency)Enable collaboration among colleagues
(Scientific) workflow environments
Automated orchestrationVisual scriptingProvenance
Digital experimentationCollaboration-enhanced Office productivity tools
Structure experiment data and derived results in a manner appropriate for human reading/reasoning (as opposed to optimizing for query processing and/or storage efficiency)Enable collaboration among colleagues
(Scientific) workflow environments
Automated orchestrationVisual scriptingProvenance
CGW'05 25
Scientific Information Worker:Past and FutureScientific Information Worker:Past and Future
PastBuy lab equipmentKeep lab notebookRun experiments by handAssemble & analyze data (using stat pkg)Collaborate by phone/email; write up results with Latex
Metaphor:Physical experimentation“Do it yourself”Lots of disparate systems/pieces
PastBuy lab equipmentKeep lab notebookRun experiments by handAssemble & analyze data (using stat pkg)Collaborate by phone/email; write up results with Latex
Metaphor:Physical experimentation“Do it yourself”Lots of disparate systems/pieces
FutureBuy hardware & softwareAutomatic provenanceWorkflow with 3rd party domain packagesExcel & Access/SQL-ServerOffice tool suite with collaboration support
Metaphor:Digital experimentationTurn-key desktop supercomputerSingle integrated system
FutureBuy hardware & softwareAutomatic provenanceWorkflow with 3rd party domain packagesExcel & Access/SQL-ServerOffice tool suite with collaboration support
Metaphor:Digital experimentationTurn-key desktop supercomputerSingle integrated system
CGW'05 26
Where Grids will be in 5 years?Where Grids will be in 5 years?
Like in the past ES, AI, networking, OS they will disappear from the hot research (and hype) space and become mainstream technologyMajor Grids already work in production (EGEE: 18’000 computers, Google: 100’000 computers?...)Major IT vendors will integrate Grid middleware in their standard products (industrial uptake)Computing and data resources will become commodities on the InternetISPs will offer a wide range of services Grid based, a full mature market will develop for these servicesThe result will be a tremendous computing and data processing power which will enable a new set of scientific applications and generate large revenues for business applicationsA potential leveler for a worldwide science and economy => digital Divide could be moderated
Like in the past ES, AI, networking, OS they will disappear from the hot research (and hype) space and become mainstream technologyMajor Grids already work in production (EGEE: 18’000 computers, Google: 100’000 computers?...)Major IT vendors will integrate Grid middleware in their standard products (industrial uptake)Computing and data resources will become commodities on the InternetISPs will offer a wide range of services Grid based, a full mature market will develop for these servicesThe result will be a tremendous computing and data processing power which will enable a new set of scientific applications and generate large revenues for business applicationsA potential leveler for a worldwide science and economy => digital Divide could be moderated
CGW'05 27
… And time will tell how wrong we are in our predictions now
See you back here next year!
… And time will tell how wrong we are in our predictions now
See you back here next year!
