Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004

GridLab 2003/4 „Steady leadership in changing

times!”Jarek Nabrzyski

Project Coordinator

Poznan Supercomputing and Networking Center

GridLab Project

Funded by the EU (5+ M€), January 2002 – December 2004Application and Testbed oriented

Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled

Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools:

Resource management (GRMS), Data management (GDMS),Monitoring (Mercury) and information services, Adaptive components (Pythia), Mobile user support and remote visualization, Security services (GAS),Portals (GridSphere),

... and test them on a real testbed with real applications

GridLab Members

PSNC (Poznan) - coordination AEI (Potsdam) ZIB (Berlin) Univ. of Lecce Cardiff University Vrije Univ. (Amsterdam) SZTAKI (Budapest) Masaryk Univ. (Brno) NTUA (Athens)

Sun Microsystems GmbHHP France

ANL (Chicago, I. Foster) ISI (LA, C.Kesselman) UoWisconsin (M. Livny)

collaborating with:Users!

EU Astrophysics Network,DFN TiKSL/GriKSLNSF ASC Project

other Grid projectsGlobus, Condor,GrADS,PROGRESS, ClusterixGriPhyn/iVDGL,European Grid Projects (GRIDSTART)KISTI, Canadian Grid, LSU…

Review Issues and Concerns

It is GridLab’s first direct contact with the new PO and 2 out of 3 reviewers are new to the project,Difficult to review a project of this size and complexity in several hours,Decision to present the project from the integration angle,Problems with synchronization of deliverables with the last-minute amendment (details in the WP presentations)Presentations: Short time per workpackage, short time for the demo,Innovative GAT, the portal and services presented in a practical demo during SC2003 in PhoenixThe more features we add (to every single WP out of 12) the more complex the integration gets

GridLab Aims

Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community, ...),Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizationsDream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

What GridLab isn’t

We are not developing low level Grid infrastructure - we do more actually, we want to be infrastructure independent,We do not want to repeat work which has already been done (want to incorporate and assimilate it …)

Globus APIs, OGSA/OGSI/…, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn,...

Project Goals for 2003

Integration of all the GridLab software,“Zakopane Scenario” driving all the developments and integration,First release of GridLab software scheduled for December 2003, Make the testbed more stable and ready for any demo action and/or development,SC2003 was chosen to demo the GridLab software,Strong dissemination of the project in order to get attention of interesting Grid applications,Start the GAT and other GridLab standardization work under GGF,Strengthen management of the project, with clear responsibilities to ensure better control of resources, fast identification of the risks, ensure progress, GridLab Open Source License, Prepare the plan for exploitation of the results,Adjust the Technical Description of Work to the current needs of the project

What have we done during last 8 months?

Response to the reviewers report sent to allOpen Source LicenseGT2 vs. GT3 issues (finally it was wise decision)Strategy for dissemination in 2004Deliverables, some delays here (see details in the WP presentations)Contract amendment is on the way

GridLab meeting in Eger (April) and Olomouc (October)More developments and integrationMore and more WPs brought to the scenarioToday most of the WPs were integrated with each other

SC2003 in Phoenix (2 big „demos”, many „small” demos)Painful demo, but successful: GRMS + portal + GAT application + Testbed + WP5 information service + Adaptive with some last minute problems with other services„The devil is in the details” - the most true sentence.

Major achievements

Integration efforts and devotion of project staff,Meeting in Eger in March/April, meeting in Olomouc in October,Many small integration meetings (Poznan-Berlin-Potsdam, Potsdam-Budapest, Potsdam-Prague, Cardiff, GGF meetings and at other confs.),

Getting a major attention towards the GAT of many application communities,

GRIDSTART, GGF, application communities and 6FPP (EGEE, DEISA, HPC-Europa, …)Expectations are high!

Many successful implementations of servicesSingle GridLab services being used by other projects,Most of the services are fully integrated with each other, but can be configured as stand aloneGAT can connect to most of the services. All will be available in a month.

Major achievement (cont.)

GridLab middleware being exploited by other projects and production testbeds

GridSphere: tens of projects (Grid Infrastructure Project, GEON, HPC-Europa, Physics Portal Development Project, …)GRMS (Clusterix, SGI Grid, Progress, Canadian Grid)Mercury Monitor (DataGrid, P-GRADE)WP8 Visualization - GriKSL

Open Source LicenseGridSphere has it’s own Open Source license

Major achievementsStandards, collaboration

Global Grid ForumPositions in GGF

Alexander Reinefeld - GGF Advisory BoardJarek Nabrzyski – step down from the SC and AD for Apps and now take care of the GSA RG), Ed Seidel, Thilo Kielman chairing the Apps RGThilo Kileman chairing Grid CPR WGAndre Merzky chairing the Persistent Archive WG and secretary of Data Area and Persistent Environments

GAT architecture put at the Apps and APM groupsMany WG talks about GridLabPutting the GAT to the BOF in October 2003

Very successfulParticipation in Gridstart!

GridLab chairs at least half of the TWGs

OtherProjects: GriPhyN, GRIKSL, CrossGrid, Reality Grid (and some others) have a certain level of collaboration with GridLabCompanies: Grid Systems, Platform Computing, IBM, HP, Sun

Major achievementsDissemination

Improved the www.gridlab.orgSeveral new papers accepted for books, journals and conferencesMany talks have been given at major conferences and eventsEvents organized

GridLab Conference and Workshop in Eger, March-April 2003 Grid Portals Conference, August 2003, Edinburgh Grid Application Workshop in Czestochowa, September 2003GridLab Conference and Workshop in Olomouc, October 2003Grid Resource Management Workshop, Poznan, October 2003Workshops at international conferences and symposia

Tutorials on major conferences, also via AccessGrid2 GridLab brochuresIn 2004 we will produce 14 different GridLab brochures featuring all the services, tools and advanced scenarios (see first three issues)

Book on Grid Resource Management

4 GridLab chapters:Application requirements for resource brokeringMulticriteria resource management (GRMS)Triana peer-to-peer RMWorkflow management

ISI/Argonne/Wicsonsin contributionsPublished in November by Kluwer Academic Publishers

Technical Progress

GAT API specification and GAT adaptor developer’s guideGAT tested by Cactus and a GAT subset implementation for Triana (GAP)Strenghten the collaboration with ISI, ANL, WisconsinNew Portal Framework (GridSphere) - 1st release in 2003 (2nd in 2004) - being used by tens of projects! Participation in the Grid Technology Repository (Jarek Nabrzyski serves as the Int. Advisory Board member)Strong participation in the GRIDSTART project (GridLab was one of the main contributors to the Technical Roadmap)Services ready and interoperable in spite of changing requirementsGAT ready for standardization process, already under the community comment process

Technical Progress (cont.)

Extending the GridLab scenario Successful meeting in Eger (new application users involved, new developments) and in Olomouc (integration)Problems/Issues

GAT development delayed - underestimated efforts needed for the GAT development

Should be fully ready in a monthWill be revisited and refactored! User’s input is needed. Good docs is needed.Hiring a new person, experienced with Cactus development to speed up the CGAT integration.

GAT documentation is behind! We are shifting efforts a bit to speed up the process.Some requirements and scenarios have changed dramatically: new ones appeared (WP7)Some unexpected problems with hackers on one of the test machines

Issues

GAT development delayed - underestimated efforts needed for the GAT development

Should be fully ready in a monthWill be revisited and refactored! User’s input is needed. Good docs is needed.Hiring a new person, experienced with Cactus development to speed up the CGAT integration.

GAT documentation is behind! We are shifting efforts a bit to speed up the process.Some deliverables are delayed

New Annex1 caused some problemsGAT development delayed (CGAT, GAT documentation)

Some requirements and scenarios have changed dramatically: new ones appeared (WP7)Some unexpected problems with hackers on one of the test machines

Plans for 2004Continue the development of GAT and services

more info in the following presentationsMore complicated scenarios

collaborative environments,submitting, controling and steering jobs from mobile devices,more dynamic behavior of applications

Two more GridLab Workshop Meetings (Lecce: 16-22 May, Zakopane: early December)Organizing the GridLab/GT3.2/WSRF integration meeting withUS partners (to ensure GAT compatibility)Preparing for the Supercomputing demosGGF BOF (GAT) and finally GGF WGGGF Scheduling Architecture Working Group now started (GRMS)Exploitation of the project results

Close work with GridLab’s commercial partnersGlobal Grid Application Alliance with GridLab’s leadershipGGF activities (long term)GridSuit (based on GridLab plus Gridstart Open Source, PSNC plus partners)

Open Source + commercial supportEuropean Grid Support Centre (PSNC)

Summary

GAT is importantWe will put yet more effort into the GATGAT will be revisited after finalizing the 1st release (in one month)RefactoringLecce Meeting to bring application people, TWG8, tutorial, exploitationDocumentation!GGF/Gridstart

There was a great interest and we are talking to many groupsThe project works better than ever!

Project is unique in this sense (this might be an impression of ours only)

Integration was much more difficult than expectedWe had to deal with many (too many) unexpected problems in 3rd party software.

Collaboration with ISI

Collaboration with Argonne

OGSA C bindingsGridLab/GT3.2 Inegration Workshop (4 Argonne GT3 developers/experts are coming to help us with the integration)Help with developing the GT3.2 adaptorsGAT review

Collaboration with Condor Group

Condor cont.

Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004

Exploitation Plan

Jarek Nabrzyski

Pillars of exploitation

Close work with GridLab’s commercial partnersGlobal Grid Application Alliance with GridLab’s leadershipGGF activities (long term)GridSuite (based on GridLab plus Gridstart Open Source, PSNC plus partners)European Grid Support Centre (PSNC, HP)European Grid Technology Broker

Exploitation of the results through commercial partners

HPGrid Resource Toolkit. HP investigates which GL software can be adopted.Distribution through server/software distribution

SunThis meeting!

Actions to be taken:GridLab will work closely with the companies to ensure that the companies evaluate the software and make right decisions on which software is supposed to be part of their distributionsResponsible: WP managers will provide software plus documentation to a contact person at the vendor side.

Global Grid Application Alliance

GlobalEurope - GridLabUS - Ed SeidelAsia Pacific - both SatoshisKorea - KISTI

TasksBring the GAT and services to the communityTrain people to make then use the GATPeople will write adaptors for their favorite servicesWork on the new scenarios that the Alliance will bringWork on portable applications on a global scale using the GridLab technologies

Actions:The initial meeting of the founders took already placeSC will be responsible for the actions: meetings soon to be scheduled

GGF, GRIDSTART activities

SAGAGSAand more will be continued beyond the timeframe of the projectGridstart TWGs will be used to work with other Grid projects in order to deploy the GAT and GridLab tools in these projectsTWG chairs (from GridLab) are responsible to make it happen (GridLab WP managers, SC members)

GridSuite at PSNC

GridSuite (based on GridLab plus Gridstart open source, PSNC plus partners) - Grid out-of-the-box complete solution

Open Source + commercial supportPSNC has declared to hire a team of 6-8 persons to build the GridSuite this yearPSNC will work with GridLab partners to provide an expertise to build the GridSuite and to design a business plan for the effort.Release roadmap is under way. This might be a background for the commercial company selling the Grid services for European industryActions to be taken:

PSNC will discuss this with the GridLab Steering Committee soon,This will be more open to other GridLab partners (not limited to PSNC)Strategy is being developed by PSNC nowInitial money from the European infrastructure funds (under way) and PSNC’s own funds.

European Grid Support Centre

European Grid Support Centre (HP, PSNC plus UK eScience, CERN, PDC)Institutions’ own fundsThe task is to support different user groups (currently DataGrid, EGEE, GridLab, NorduGrid)The tools and the procedures are now being established. The ground for GridLab’s support for its users and infrastructure builders,Currently PSNC and HP from GridLab only. More will be involved with increase of the number of users.Actions:

Build the procedures and assure that GridLab user’s know about the Center and use its services,To be discussed by the SC in more details in 6 weeks time,Extension of the group with a time and load (currently one person half-time at PSNC, similarly at HP, 3-4 persons at PSNC soon).

Exploitation by partners - general strategy

Every GL partner will work with the users that the SC agrees to work with closely,In such a case particular institution will make sure that appropriate resources are assigned to work with the user in order to deploy GridLab software in the user’s environment,The appropriate resolutions will be worked out in details and decided by the Steering Committee by the existence of the GAT release, no later than by the end of April. At the beginning we will focus on strategic users to ensure the added value to the project.We will work closely with the GridLab application communities first of all (Cactus and Triana) on the GAT usage.

Beyond the project

Seek for local funding by every partnerSurvive the gap after the GridLabContinue work, bring more users, get new requirements, analyze scenarios, work with the GGAA (Global Grid Application Alliance)