MSA2.2 - Continuous Integration and Certification …MSA2.2 - Continuous Integration and Certification Testbeds in Place Doc. Identifier: EMI_MS23_v1.0.doc Date: 31/07/2010 MILESTONE

MSA2.2 - Continuous Integration and Certification Testbeds in Place Doc. Identifier: EMI_MS23_v1.0.doc Date: 31/07/2010

EUROPEAN MIDDLEWARE

INITIATIVE

MSA2.2 - CONTINUOUS INTEGR ATION AND CERTIF ICATION TESTBEDS IN PLACE

EC MILESTONE: MS23

Document identifier: EMI_MS23_v1.0.doc

Date: 31/07/2010

Activity: SA2

Lead Partner: INFN

Document status: Final

Document link: http://cdsweb.cern.ch/record/1277551?ln=en

http://cdsweb.cern.ch/record/1277551?ln=en�


Copyright notice:

Copyright (c) Members of the EMI Collaboration. 2010.

See http://www.eu-emi.eu/about/Partners/ for details on the copyright holders.

EMI (“European Middleware Initiative”) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see http://www.eu-emi.eu.

This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) 2010. Members of the EMI Collaboration. http://www.eu-emi.eu ".

The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date.

EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT.


Delivery Slip

Name Partner / Activity Date Signature

From Danilo N.Dongiovanni INFN 20/11/2010

Approved by PEB 26/11/2010

Document Log

Issue Date Comment Author / Partner 1 08/05/2010 First version available for revision within SA2 Danilo Dongiovanni

2 25/09/2010 Internal Review Alberto Aimar

3 10/08/2010 Added Unicore Resources Danilo Dongiovanni

4 10/28/2010 Added EMI Testbed GGUS Support Unit Danilo Dongiovanni

5 11/02/2010 Changed document structure: milestone report on first page and technical details in following sections

Danilo Dongiovanni

6 11/19/2010 Changed document structure: milestone report on first page. Summary of technical details reported in the documentation url in Annex.

Danilo Dongiovanni

Document Change Record

Issue Item Reason for Change 1

2

3


MILESTONE REPORT EMI continuous integration and certification testbed has been put in place and it is currently available for Product Team developers to perform integration tests.

The infrastructural and operational resources in place consist of:

• HW/SW resources provided (currently ≈ 90 server instances) with:

• OS installed (one among agreed platforms for EMI, i.e. SL5 x32 or x86_64), network connection and utilities.

• Server host certificates (required in ARC and gLite, containers certificate in UNICORE not related to host). Also a tool for testing certificates generation was made available (ARC).

• Server level monitoring tool to collect statistics about server availability/reliability. • Software Product installed and configured. Full coverage of already available products

mentioned in EMI Technical plan was granted. dCache middleware testing instances were provided by DESY partner and integrated in the testbed.

• A set of configurable information system service instances publishing resources available in the testbed.

• User Interface accounts provided to PT testers on demand.

• Operational resources provided:

• A Virtual Organization for testing has been created: testers.emi-eu.eu • EMI Testbed Documentation • Instances logbook reporting details on installed software version and configuration. • A GGUS Support Unit has been created for support requests handling and tracking • Communication channel and task tracking tools for SA2.6 task members activity

coordination

Technical details and full documentation about the infrastructural and operational resources together with a description of testing scenarios and use cases supported can be found at the following public url: https://twiki.cern.ch/twiki/bin/view/EMI/TestBed

A summary of contents there available are also attached in the Annex of this document.


ANNEX: Technical Details on implemented testbed infrastructure

1.1. SUPPORTED TESTING SCENARIOS The testbed model definition moved from EMI Dow and task participants experience and resources and took advantage of discussion within SA2 and with developers’ representatives (JRA1, SA1, Release and SW Area Manager). The information was collected through surveys or meeting and summarized in minutes or documents available at SA2.6 task homepage1

A detailed presentation of EMI integration testbed models is object of the DSA2.4 deliverable (ref. SA2.6 task homepage); therefore here we focus on the testing scenarios supported in the testbed infrastructure put in place:

.

1. EMI Internal Testing scenario A: Integration testing within a minor release (no backward compatibility broken), so that a Release Candidate2

2. EMI Internal Testing scenario B: Integration testing for a major release (where it is allowed to have new features or backward compatibility broken for many services). This implies a testbed of RCSV available for each middleware stack, so basically this means providing hardware with platform installed for Product Teams (PT) to install needed RCSV and allow them for previewing other's PT RCSV. Key performance indicators KSA2.1, KSA2.2 will apply to this testbed.

Service Version (RCSV in the following) can be tested VERSUS other Production Version Services (PVS in the following). This implies a distributed testbed of production services available for each middleware stack, with possible multiple instances for central services. This could also imply cases of RCSV vs. other RCSV or RCSV vs. (RCSV + PVS): imagine the case of two interacting products in which a common bug is fixed contemporaneously, that would imply RCSV for both of them to be tested together with all other services at production version. Key performance indicators KSA2.1, KSA2.2 will apply to this testbed.

1.2. DEFAULT USE CASES SUPPORTED Use Case A: developer John needs to test the correct interaction between service X (Release Candidate version) and services Y (Production Version), Z (Release Candidate Version). Solution: service X is configured to see resources published in the chosen EMI Testbed central information system instance. Depending on the test performed John may need some configuration effort on services Y or Z, to ensure they can interact with X. John sends a support request to EMI Testbed group (see section 1.5). Use Case B: developer John needs to test the correct interaction between his service X (version X.X.X installed on some instance of his PT) and services Y (Production Version), Z (Release Candidate Version). Solution: service X is configured to see resources published in chosen the EMI Testbed central information system instance. He can also setup a new information system merging information from both the mentioned central information system and a local information system publishing some development resources, building a custom testbed view. Notice that service Y and Z will not be configured to see resources out of EMI Testbed. Use Case C: developer John needs to test the correct interaction between his service X (Release Candidate Version) and services Y (Production Version), Z (Release Candidate Version) not currently in the testbed, through a User Interface (ex. Job submission from UI involving broker, information system, storage element, compute element). Solution: John requests (see line 3 in this table) an account on one of the User Interfaces provided in the testbed, which is configured to see resources published in the chosen EMI Testbed central information system instance. Depending on the 1 https://twiki.cern.ch/twiki/bin/view/EMI/TSA26

2 Here we assume that for each service a single Release Candidate version per Release exists.


test performed John may need some configuration effort on services Y or Z, to ensure they can interact with X. Moreover John needs service Z to be installed in the Testbed. John sends a support request to EMI Testbed group (see section 1.5).

1.3. TESTBED INFRASTRUCTURE INVENTORY

1.3.1 ARC ARC3

The following services (multiple instances for some services) were made available:

middleware currently deploys 11 products on Fedora, Debian, RedHat, Ubuntu, Windows and MacOSX platforms.

Product Name Version Platform Partner Site

GIIS service (ARC LDAP-Infosys)

Release 0.8.2 Production

CentOS5.5 i386 Kosice

Nagios Release 3.2.1 Production


Instant CA Release 0.9 Production


Classic ARC Grid Monitor Release 0.8.2 Production


WS-ARC Grid Monitor Release Candidate CentOS5.5 i386 Kosice

ARC ISIS service (4 Instances) Release Candidate CentOS5.5 i386; SLC5.3/x86; Debian Lenny /x86

Kosice, NIIF

Classic ARC CE Release 0.8.2 Production


CE1 type Release 1.1 Production


A-REX Release 1.1 Production

SLC5.3/x86; Debian Lenny /x8 Kosice, NIIF

Bartender service (2 instances) Release 1.1 Production

Debian Lenny /x86; CentOS5.5/x86_64

Kosice, NIIF

AHash service (2 instances) Release 1.1 Production

Debian Lenny /x86 NIIF

Classic ARC clients Release 0.8.2 Production

Ubuntu Hardy /i386 Kosice

WS-ARC clients Release 1.1 Production


ARC data clients Release 0.8.2 Production


3 http://www.knowarc.eu


Librarian service (3 instances) Release 1.1 Production

CentOS5.5/x86_64 ; Debian Lenny /x86

Kosice, NIIF

Echo service Release 1.1 Production

SLC5.3/x86 ; Debian Lenny /x86

Kosice, NIIF

Shepherd service (2 instances) Release 1.1 Production

Debian Lenny /x86 NIIF

1.3.2 gLite GLite4

The following services (multiple instances for some services) were made available:

middleware currently deploys 19 products in Release 3.1 SL4 OS and 17 products in Release 3.2 SL5 (when service implementation changes without affecting API services are counted as one, ex. gliteVOMS mysql/oracle database implementation) on SL4 and SL5 platforms.


gliteWMS (3 instances) gLite 3.1 Production; RC SL4 INFN, CERN

dGas ig_HLR Production Version ig48_sl4 SL4 INFN

gLite-Cream 3.2 Production Version(LSF); 3.1.24 Production

SL5/x86_64;SLC4.8/x86

INFN

gLite UI (3 instances) 3.1 Production; 3.2 Production, RC SLC4.8; SL5/x86_64

INFN, CERN

GliteBDII (site; Top) Production 3.1.23 SLC4.8/x86 CERN

gLite-PX Production 3.1.29 SLC4.8/x86 CERN

gLite-lcgCE Production 3.1.40 SLC4.8/x86 CERN

gLite WN Production 3.1.11; 3.1.30 ; 3.2.7 SL4.8/x86 , SL4.8/x86_64; SL5.5/x86_64

CERN

gLite-LFC_mysql Production 3.1.29 SL4.8/x86_64 CERN

gLite-VOMS Production 3.1.27; Production 3.2 SLC4.8/x86; SL5.5/x86_64

CERN

gLite-FTS_oracle Production 3.1.22 SLC4.8/x86 CERN

gLite-VOBOX Production 3.1.42 SLC4.8/x86 CERN

gLite-SE_dpm_mysql Production 3.2.5 SL5.5/x86_64 CERN

4 http://www.glite.eu/


gLite-DPM_pool Production 3.1.32 SLC4.8/x86 CERN

gLite-SE_dpm_mysql Production 3.1.35 / disk 3.1.29 SLC4.8/x86 CERN

Nagios Production 3.2.1-1; 3.2.0-1 SLC5.5/x86_64 ; SLC4.8/x86

CERN

gLite (9 instances) Production 2.1.7-1, RC SLC5.5/x86_64 ; SLC4.8/x86

CESNET

STORM INFN grid Release 3.1.0-0_ig50_sl4 Production Version SL4 INFN

1.3.3 UNICORE UNICORE5

middleware currently deploys 11 products without particular dependencies making it executable on Linux, Windows and MacOSX platforms, generally deployed on openSuSE 11.2 for certification purposes.


Gateway 6.3.1 Production; openSuSE 11.3 JUELICH

Registry 6.3.1 Production; openSuSE 11.3 JUELICH

X incl. XNJS 6.3.1 Production; openSuSE 11.3 JUELICH

OGSA-BES interfaces 6.3.1 Production; openSuSE 11.3 JUELICH

HiLA 2.1 Production; openSuSE 11.3 JUELICH

XUUDB 6.3.1 Production; openSuSE 11.3 JUELICH

UVOS 6.3.1 Production; openSuSE 11.3 JUELICH

Command line Client (UCC) 6.3.1 Production; openSuSE 11.3 JUELICH

1.3.4 dCache Also dCache6

certification resources were kindly made available for integration testing purposes from dCache EMI partners. In particular resources below can be accessed through CERN gLiteUI instance.

Product Name Version Platform Partner

Site

5 http://www.unicore.eu/

6 http://www.dcache.org/


BDII SL4 DESY

dCache A SL5;

SL4 32bit PNFS;

SL4 64bit PNFS

DESY

dCache B Sl5;

SL4 32bit Chimera;

SL4 64bit Chimera;

DESY

1.4. MONITORING AND KPI Key performance indicators KSA2.1 and KSA2.2 reported in table below, imply automatic monitoring solutions for resources able to produce statistics on server's availability and reliability. Each middleware currently has a monitoring solution deployed: ARC (Ganglia, GridMonitor, Nagios), gLite (Nagios), UNICORE (Nagios).

The Key Performance Indicators.

CODE KPI Description Method to Measure

Estimated Targets

KSA2.1 SA2 Services Reliability

% uptime dependent only on the SA2 services themselves (individual KPIs for test beds, repository, etc)

Participating sites monitoring tools

99.00%

KSA2.2 SA2 Services Availability

Total % uptime including the underlying suppliers (individual KPIs for test beds, repository, etc)

Participating sites monitoring tools

97.00%

Availability and reliability statistics are currently provided just on testbed server instances, not on services given the fact that not all services have availability/reliability metrics defined and tools to measure them. Concerning the adopted tool to monitor instances and produce statistics, all middlewares plan to converge on Nagios7

• Evolution of Nagios into grid monitoring

solution for two suitable features for our task purposes: 8

• Solution for geographical distribution: a second level Nagios can implement a central instance, republishing and aggregating data coming from local sites Nagios instances.

Nagios, which is expected to provide metrics for service monitoring

Initially, availability and reliability statistics periodically produced by local sites Nagios instances will be made available in the testbed public documentation center for SA2.6 described in section 1.5.

7 http://www.nagios.org/

8 https://twiki.cern.ch/twiki/bin/view/EGEE/OAT_EGEE_III


1.5. DOCUMENTATION For public documentation of EMI Internal Testbed resources the following web page was put in place https://twiki.cern.ch/twiki/bin/view/EMI/TestBed, reporting:

◦ Description of supported testing scenarios ◦ Role and duties of SA2.6 task and PT contribution ◦ Procedures for testbed update requests ◦ Testbed Monitoring solutions ◦ Procedure to enable testers.emi-eu.eu VO ◦ EMI Testbed Coverage of EMI components ◦ Testbed Inventory with a list of provided instances specifying: Middleware Suite, Service

Deployed, Platform, Server Hostname, Site Location, reference Product Team, Status Logbook

◦ Status Logbook field in previous table is a link to a instance-specific web page describing the hardware details of instance, software version installed, configuration information and history of updates. The maintenance of this page is in charge of people performing installation; configuration or updates (can be PT members).

1.6. USER SUPPORT, COORDINATION AND COMMUNICATION Both coordination and installation/configuration activities concerning testbeds require clear channels of communication and a way to track the effort of people involved.

As mentioned in DS2.4 deliverable document, a distributed effort model for testbed setup and maintenance was adopted, with the possible involvement of Product Teams members as support effort for service installation and configuration. To coordinate and track all the distributed effort the following solution was adopted:

• User Support Requests Handling: An emi-support-testbed support unit has been created in GGUS, for testbed support requests reception. Representatives from all partners contributing to testbed are members of the support unit, and agreed on a 2 working days response time on best effort. The Support Unit will be part of next GGUS release. The adoption of GGUS will give a common framework to handle both requests coming from EMI developers or users and those coming from external users (e.g. the users of large scale testbed involving other projects or partners contribution).

• Communication:

◦ SA2.6: an [email protected] was created both for task internal communication and for testbed requests reception.

• Activity Tracking: SA2.6 and Product Teams activities on testbed will be tracked through Savannah9

◦ An emi-sa2-testbed Savannah squad has been created to submit requests.

tasks.

◦ Product Team squads have been created to track PT activities on testbeds

1.7. TESTBED UPDATE Resources made available for integration testing described in section 1.2 form a first nucleus of EMI testbed, putting together services currently used for certification from all middlewares converging into EMI.

9 https://savannah.cern.ch/


EMI testbed evolution is then strictly connected to its actual usage by product teams’ members performing integration testing. In fact depending on the specific integration test to be performed different coordination, installation or configuration activities can be required. So we expect EMI internal testbed customers (PTs, SW Area Manager, SA1, and JRA1) to submit support requests for the following expected cases, treated as described below:

• Requests for configuration support of existing services. These requests may include enabling VO/users, making services to talk to each other, custom bdii setup etc. Procedure:

• Open a GGUS ticket assigned to EMI-Testbeds Support Unit explaining your testing and configuration needs. The request will be then evaluated and tracked into a savannah task on the testbed squad.

• If needed a PT members of services involved in the test will be contacted and their contribute will be tracked by savannah

• Requests for new services setup (or particular service RC versions setup): • Open a GGUS ticket assigned to EMI-Testbeds Support Unit explaining:

• Your testing needs and the type and version of services you need to be installed and the PT producing that service

• Please also specify if you need the service to be included in the permanent EMI testbed.

• The request will be then evaluated and tracked into a savannah task on the testbed squad. If needed, the involved PT members will be then contacted and their contribute tracked by savannah tasks.

• Requests for specific testbed (in this category: performance tests, security tests, data management tests, etc.):

• Open a GGUS ticket assigned to EMI-Testbeds Support Unit explaining: • Your testing needs and an estimate of HW and SW requirements for your test • PTs involved in the setup and suggestions on possible sites/PT/NGI that may help

in the setup. The request will be then evaluated and tracked into a savannah task on the testbed squad.

• The period of time you expect to have the testbed on for • The involved PT members will be then contacted and their contribute tracked by

savannah tasks.

1.8. TESTBED ACCESSIBILITY User Interface Service Instances: As default use case we assume testbed users to have direct access just to user interface instances, that is just to grid middleware access point services. To request an account on a EMI Testbed User Interfaces instance, every user with a valid certificate from a trusted Certification Authority, should send a user support request following the procedure and tools described at point 4 in this table. Notice that it is also possible to install the set of clients directly on personal machine (ex. usual use case in UNICORE).

Other Service Instances: Root access on other services can be granted on request, depending on the local sites security policy (which generally is also subjected to national laws about traceability of access on servers). If the access is required for debugging or logs exploration purposes, logs sharing solutions will be implemented on demand (publishing of logs on public AFS area, GridFTP downloads, https access) .

1.9. TESTBED RESOURCES DISCOVERY Information Systems Configuration: Each of the middleware has a service for resource discovery and publication (ARC, gLite BDII, Unicore Registry). A central information system instance was configured for each middleware publishing the resources in the testbed. Cross-middleware


compatibility among existing information system services is in EMI plans, and EMI Testbed will reflect that integration once it will be technically available.

Implications for testbed usage: the set of resources visible to the end users (developers) depends on the configuration of their access point (the information system instance configured in the User Interface instance user is logged on). In practice user can build a custom testbed by selecting needed resources from the pool of those published in the central information system or merging them with other resources published on other information system (ex. Product Team internal development testbed).

Documents

MSA2.2 - Continuous Integration and Certification …MSA2.2 - Continuous Integration and Certification Testbeds in Place Doc. Identifier: EMI_MS23_v1.0.doc Date: 31/07/2010 MILESTONE