Glite I/O Storm Testing in EDG-LCG Framework

Elena Slabospitskaya, Vadim Petukhov, (IHEP, Russia)

Gilbert Grosdidier, (CNRC, France)

NEC'2005, Sept 16

Outline

● Framework’ Contributed Authors

● TSTG Suite Dedication

● Design of TSTG Framework S/W

● Up to gLite- port TSTG framework to gLite environment

● Example of gLite plug-in - I/O Stress Test Suite

NEC’05 Elena Slabospitskaya (IHEP) et al.

Framework’Contributed Authors

The mainly contributed authors are● Miguel Barcelo● Frederique Choilett● Gilbert Grosdidier● Andrey Kiryanov● Charles Loomis● Gonzalo Merrino● Rene Metery● Danila Oleynik● Andrea Sciaba● Elena Slabospitskaya● Many other people contributed to the design and ideas leading to the

current suites, among them the INFN Testing Team

NEC’05 Elena Slabospitskaya (IHEP) et al.

New accent of MW testing

"The development of full featured test suites that for the certification of the new MW components"

"The development of testing procedures focused on special details of various software features" 

"Porting the exiting test suites and procedures to the TSTG framework"

The coordination of this task will be done in close collaboration with EGEE-II SA3 to address the shortcomings of available tests for gLite middleware. This involves agreement on schedule, provision of access to development test-bed(s), support on issues hindering the development process (e.g. missing or failing test-bed functionality).

NEC’05 Elena Slabospitskaya (IHEP) et al.

Levels of testing required

1. Installation and configuration testing

Targets each machine/service individually

2. Unit testing

Targets each basic functionality of a given service independently from the rest of the TB

3. Functionality testing

For the all TB exercises full functionalities

4. Stress testing

Huge number of jobs launched one by one or in parallel

NEC’05 Elena Slabospitskaya (IHEP) et al.

TSTG suite dedication

1. Site certification• Initial check of the install• Check than changes or misconfiguration occur • Regular daily or weekly re-checking (under cron)2. TB Certification• Basic functional tests of components (services)• Basic grid functionalities with individual tests• Full grid functionalities on a full TB including remote sites with

global/group tests3. MW validation (Functionalities, Robustness, Stress Testing)

NEC’05 Elena Slabospitskaya (IHEP) et al.

Framework S/W design

Configure StepTest LoopPlug-insDefinition FilesPresenter Step

It is split into 2 main parts, dedicated to 3 tasks-the top level driver, written in Perl

-the plug-ins, one of each specific test (Perl, bash, C++)The top level driver is responsible for these 3 tasks----Configure (build environment)----Test Loop (run one of several selected tests)----Presenter : merges the results and builds the web enabled summary page NEC’05 Elena Slabospitskaya (IHEP) et al.

Examples of LCG-2 Plug-ins

Job StormCopy StormReplica StormData StormGFAL StormMatchmaking StromRLSSesMDSChecksum Storm

NEC’05 Elena Slabospitskaya (IHEP) et al.

Structure of CTB SL3

NEC’05 Elena Slabospitskaya (IHEP) et al.

LCG-2 Testing under TSTG Framework

To change: View -> Header and Footer 19

Enabling Grids for E-sciencE

INFSO-RI -508833

Storage Element Interfaces

• SRM interface– Management and control

– SRM (with possible evolution)

• Posix-like File I/O– File Access

– Open, read, write

– Not real posix (like rfio) SRM interface

rfio dcap chirp aio

Castor dCache NeST Disk

POSIXAPIFile I/O

Control

User

•The gLite-I/O server allows posix-like access to files stored in the SRM

gLite Data Access via I/O server

NEC’05 Elena Slabospitskaya (IHEP) et al.

Input/OutputServer

UserInterface

WorkerNodes

WorkerNodes

StorageElement 1

StorageElement 2

Storage Element 3

StorageElement 4

gLite I/O plug-in

User launches multiple (parallel or one by one) jobs from User Interface to Worker Node.Each job on Worker Node launches a script which compiles and linkages C++ program and creates a jdl file (this program is a part of job in input sandbox) The C++ program via API creates files (size is optional) and transfers (read/write/seek) these files between WN and I/O server (there and back). The result returns to user (to User Interface). Sources and results of tests are arranged at web.

NEC’05 Elena Slabospitskaya (IHEP) et al.

Results of I/O Stress Tests

NEC’05 Elena Slabospitskaya (IHEP) et al.

I/O test. XML output

Conclusions

1. Reuse as much as possible existing suites/frameworks/ can be easily incorporated from the LCG-2 certification to gLite•Minimize duplication of effort when creating new tests•Make sure new tests can be easily incorporated into the existing framework

2. The next most urgent is to write new tests or the functionality being released and /or different from LCG-2•Most must one from EGEE JRA1 team•But not exclusively, LCG team will contribute