Applications Area Status and Plans CMS CPT Week 18 April 2002 Torre Wenaus, BNL & CERN/EP LCG Applications Project Manager [email protected]

Applications Area Status and Plans

CMS CPT Week18 April 2002

Torre Wenaus, BNL & CERN/EPLCG Applications Project Manager

[email protected]://cern.ch/lcg/peb/applications

CMS CPT Week, Apr 18, 2002 Slide 2 Torre Wenaus, BNL & CERN/EP

News

Dirk Duellmann has accepted to be Project Leader for Persistency Framework project

• Dirk leads the CT (Client Technology) section and the Physics Data Management activity in IT/DB

• Extremely well qualified for this position• Positive feedback from all experiments

Architects Forum will be meeting for the first time next week

• Renamed with less unappealing name (formerly Architects Committee)

Rene Brun has accepted an invitation to join the Architects Forum

• Invitation unanimously extended by forum members• ROOT will have major role in many of the projects we expect

to undertake (beginning with the first one)• Rene’s participation will be essential


Persistency Workshop

As suggested in Launch Week, we plan a workshop soon in the persistency project

2-3 days Tentative window: week of June 10

• Need feedback on the date; have to settle it quickly• Only other possible week in the vicinity (late May

through end June) is week of June 3• Provide input to Dirk and me


Applications Area Meetings

As suggested in recent email, a first weekly Applications Area meeting will be this week

• Friday April 19, 16:30-18:00• IT conference room in B.31• VRVS room VENUS• Sorry for lateness of official announcement

Agenda• Organizational business• Status and plans; software process (Torre Wenaus)• Persistency – RTAG outcome and project plans (Dirk

Duellmann)• A lot of similarity to presentations today – but I hope

you come anyway!


Collaboration with experiments

Direct technical collaboration between experiment participants, IT, ROOT, new LCG personnel; as far as the technical work goes, project lines should be more important than line management categories

‘Hosting’ of projects in experiments, where appropriate

• I do not see persistency as such a case; close involvement by all experiments is expected, from the beginning, given its importance

Completely open activities and communication Issues at all levels dealt with in regular open

meetings as much as possible Mechanism accepted and almost in place for

formally involving experiment architects, and dealing with contentious issues


Participation of Architects(Accepted Proposal)

Monthly open meeting (expanded weekly meeting)• Accumulated issues to be taken up with architects• Architects in attendance; coordinators invited

Information has gone out beforehand, so architects are ‘primed’

Meeting is informational, and decision-making (for the easier decisions)

• An issue is either Resolved (the easy ones) Flagged for addressing in the ‘Architects Forum’


Participation of Architects

Architects Forum:• Members: experiment architects + applications manager

(chair)• Invited: computing coordinators, LCG project manager and CTO• Others invited at discretion of members

e.g. project leader of project at issue Meets shortly after the open meeting (also bi-weekly?) Decides the difficult issues

• Most of the time, committee will converge on a decision• If not, try harder• If still not, applications manager takes decision

Such decisions can be accepted or challenged Challenged decisions go to full PEB, then if necessary to

SC2• PEB role of raising issues to be taken up by SC2• We all abide happily by an SC2 decision

Committee meetings also cover general current issues and exchange of views

Committee decisions, actions documented in public minutes


Participation of Architects

Architecture discussions of the Persistency RTAG will surely carry over into discussions in the Architects Forum – beginning with next week’s meeting, I would expect

I hope that discussions in the concrete context of a (major) project, persistency, will aid in mutual understanding and convergence

We’ll see! I am confident we will deliver a persistency

framework in a timely way, and concretely address many architectural issues in the process


Active Project Areas

Software Process• Launched by SC2 following the RTAG recommendations as

presented during Launch Week• Starting from these recommendations, I’ve prepared the

thoughts and plans on software process and infrastructure I will present today

Persistency• Launched by SC2 following the RTAG report that was

recently completed and is now under review by SC2• Released to PEB to initiate activity and work plan

development, subject to any adjustments arising from SC2 deliberations

• SC2 will finalize the requirements it delivers to the PEB in the May SC2 meeting

• Project activity is effectively being launched this week: c.f. Dirk’s talk, the discussions today, and the ATLAS discussions earlier this week (RD’s talk)

In both these areas, project must develop work plan for submission to SC2, for their review and approval


New Active Areas in the Near Term?

New RTAGs under discussion by SC2…• Detector geometry and materials description• Simulation tools

Both were top priority in my candidate RTAG suggestions last month

Won’t say more about them here (and won’t use the term ‘RTAG’ again in this talk!)


Setting Up Infrastructure

LCG Applications server now up and running• lcgapp.cern.ch• Provided by CERN IT• Initial host for web, CVS, and other (MySQL?) services• Agreement with CERN IT is to migrate the services to official

CERN IT platforms and services when it is practical to do so e.g. IT CVS services being planned Input given on requirements (well developed

requirements list already exists)• But, we are stuck on backup arrangements at the moment• Thanks to David Stickland and CMS for help in getting this

going LCG Applications AFS area is set up

• /afs/cern.ch/project/lcg/app• For web and other documents, software, etc.


Setting Up Tools

Full time process/infrastructure support person from existing IT staff still in negotiation

• Experienced initial person to get things off to a fast start and transfer expertise to new LCG staff

• Assignments from new LCG staff still to be settled as well

Plans being developed with IT/API and Geant4 to share tool experts (maybe also installations)

• e.g. Bonsai/Tinderbox, possibly• Again, transferring expertise to new LCG staff

And an interesting possibility in the Sourceforge area…


Sourceforge & Savannah

One of many good recommendations from process RTAG was to investigate Sourceforge as a possible all-in-one collaborative software development tool

• With CVS interface, bug reporting, mailing list, monitoring and reporting, searching,

• This we will do, but also I’ve learned of another similar possibility I think we should examine

Savannah, http://savannah.gnu.org is an open source Sourceforge variant that branched from Sourceforge when it became commercial

• Used by GNU projects One of the developers and Savannah site maintainers is

an ATLAS collaborator from Portugal• Jaime Villate• Able and willing to devote substantial effort to an LCG

installation and evaluation if we decide to do so• Plan an initial visit to CERN to explore this May 2-3


Planning

Project breakdown and schedule implemented in the planning tool and in early development in terms of content

• Accessible from the applications web page High level milestones for LHCC proposed and

passed through PEB• June 2003: General release of hybrid event store• Nov 2003: Distributed production environment using

grid services• May 2004: Distributed end-user interactive analysis

from a Tier 3• March 2005: Full function release of persistency

framework


Personnel

Still a molehill, but people are trickling in New hiring round in UK is well advanced

• Shortlist assembled• Interviews later this month for ~15 positions in total• Many (~8?) expected to be in applications

Also getting or expecting contributions at lower levels from several other countries


Software Process

The rest of my talk will focus on software process and infrastructure, based on the material announced recently that is found on the applications area web site

Based around SC2 input, and also drawing on various discussions and material from the experiments, IT, and ROOT

• And thanks to Gabriele Cosmo for feedback! These are thoughts for discussion


Policy

All projects will adopt the same set of tools, standards and procedures

These, together with the software itself, will be centrally installed, maintained and supported

Commonly used open source or commercial software will be adopted in preference to 'do it yourself' solutions, where conditions (e.g. cost) permit. Effort cost should be factored in, as should the fact that in the LCG project, personnel are easier to come by than CHF.

Testing and documentation will be integral parts of the software development process. Successful acceptance testing and provision of documentation will be required for inclusion of code in releases. Mechanisms will be put in place to minimize these burdens for developers.

There will be no distinction between the communication and/or problem reporting tools and fora used internally by developers and those used by end users and the community; they will be the same.

Web based information and tools will be given a high priority, with comprehensive and uniform access to materials for all projects and the common infrastructure


Methodology

I address• Extreme Programming• Rational Unified Process (RUP, an extension of USDP)

For useful elements of a software process meeting our needs.

• Should be disciplined but not weighted down with heavy formal procedures and documents

• It should aid, not impede, the successful execution of the project.


What should we take from XP?




What should we take from RUP?

We should follow all the suggested "best practices" • Develop iteratively. • Manage requirements. The requirements from SC2

that kick off a project will be the core of this. • Use component architectures. • Model visually. Documentation should include

visual models of the software. • Verify quality. Testing and validation will be an

integral part of the software development and release process.

• Control changes. But in a more lightweight way than RUP suggests. Use the design and release planning process to plan changes and their integration into the development process, not formal 'change requests'.



Adapting the 'ten essentials of RUP' (Ref: Probasco): • Vision: Develop a Vision (Requirements)

Captures the high level requirements and design constraints, which we receive from the SC2/RTAG, and which provide the basis for defining the system and (as necessary) developing more detailed requirements.

Should define terms, goals, requirements. Will consist of a document whose first version is the SC2/RTAG report that

triggers the project. • Plan: Manage to the Plan

in RUP, the software development plan (SDP) gathers all information required to manage the project. Addresses roject organization, schedule, resources, and software process.

The latter in RUP is covered by a myriad of Plans: requirements management, configuration management, problem resolution, QA, …

In our case, in constitutes the software process we develop that will be common across all our projects.

Will consist of the project's work plan document delivered from the PEB to the SC2. The common software process will be covered by a single software process document.

• Risks: Identify and Mitigate RisksAccording to RUP, risks should be identified early and mitigation plans established. Risky areas should be attacked first.

These considerations will be applied in project prioritization, design and planning.



• Issues: Assign and Track IssuesUse regular status assessments to address, communicate and resolve management issues, technical issues, and project risks. This will be done via the planned open meetings, with escalation to the 'Architects Forum' and above as required

• Business case: Examine the Business CaseProvides the justification for the project and establishes its constraints and viability in terms of resources.

A project will not have reached the PEB unless it is felt to be needed The project workplan allows SC2 assessment of the viability of the plan.

• Architecture: Design a Component ArchitectureThe analysis and design workflow in RUP: defining a candidate architecture, refining it, analyzing behavior, and designing system components.

Defining a candidate architecture will be an up-front task Refining it will be done incrementally and iteratively as the project develops Designing (and implementing) system components will be done by the concerned project

teams under the leadership of the project leader The candidate high level architecture will be described in an architecture document

which will evolve as the architecture is refined and the component suite is developed. • Product: Incrementally build and test the product

Will do. • Evaluation: Regularly assess results

Will do.• Change requests: Manage and control changes

Will do it in a more lightweight way than RUP suggests. Use the design and release planning process to plan changes and their integration into the development process, not formal 'change requests'.

• User support.Should include documentation for users, developers, installers. Releases should be accompanied by release notes. All code should be browsable, by version and release. Open communication tools between users and developers for questions and discussion and bug reporting/tracking should be available. Training materials and tutorials should be provided. A central support and distribution point (CERN) should offer software packaged for easy distribution and remote installation.


Developing, Committing, Tagging Code

Developers should be committing their code 'constantly'; i.e. on timescales of hours/days, not weeks

If multiple developers are working on a package, they should all have rights to commit to CVS and to apply internal (not release) tags to the package

An internal tag is a declaration on the part of the developer that the package is tested and working, and should pass the nightly build/test

A release tag is a declaration on the part of the package coordinator that the package is tested and ready for release integration

Only the package coordinator should have the right to apply release tags to a package, and has the responsibility to deliver a tested, working release tag for release integration

The CVS head, all tags, the active set of releases, and major past releases should all be browsable on the web


Release Process

Minor (development) releases scheduled every 2 (or 3) weeks• The same pre-release acceptance testing as major releases.• the real 'heartbeat' of the software development process• Frequent enough to ensure that we are operating in a 'release early,

release often' mode with a tight incremental development loop. Major (user) releases occur ~3 times a year, with target deliverables

specified in advance• Not subjected to serious delays in order to incorporate late deliverables.

New, pro and old release version aliases will indicate to users the latest-and-greatest, production, and old production releases at any given time.

Release integration and management will be supported by web-based release integration tool

• by which release content in terms of package versions is controlled and openly consulted

Release integration also supported by a dedicated automated build+test system

• same system used for nightly builds, but a distinct build set• will build package versions declared for release inclusion in the release

integration tool. Routine automated nightly builds based on (by default) the most

recent tag of a given package, whether that tag is an internal or a release tag.


Major/Minor Releases – Common Features

Functionality in a release is agreed on in advance, through the weekly project meetings, monthly planning/design meetings, and (for the high level plan of major releases) workshops

Comprehensive, covering all project software (but not all packages necessarily have new versions in any given release)

Preceded by comprehensive acceptance testing, automated as much as possible

Accompanied by updated documentation Packages included based on 'release tags' named with a standard-

format version name (x.y.z), and declared by the package coordinator as the correct version for inclusion in the release via a web-based release integration tool

Distinct (in format) from 'release tags' will be package internal tags, used for internal coordination among developers and for controlling the code included in nightly builds

A global release version tag is applied to all the software in the release Releases are packaged for remote distribution and made available at

CERN. Source, binaries, documentation, and acceptance test results will be available in a standard distribution format.

Releases are installed for direct use at CERN using AFS Releases are added to a release archive/browser on the web Releases will not interfere with the development process by

requiring freezing (denial or restriction of commits) of the repository during pre-release integration and testing


Automated Builds

Automated builds will • always be based on package tags, never on the CVS head • build both optimized and debug versions of programs and

libraries • build autogenerated documentation and other ancillary

materials in addition to code • apply the full hierarchy of available tests (unit tests,

integrated system-level tests, code checking, etc.) • build on the range of supported platforms and compilers, to

continuously test portability • present comprehensive, user-friendly information on build

and test results on the web • send email to developers reporting on build and test

problems • provide access to a range of recent builds (e.g. the nightlies

for the last week) • be accessible for use as functional software builds with a

simple environment configuration If and when the whole hierarchy of tests becomes too

much for the nightlies, it can be cut back (eg. Unit tests)


Testing

Tests should be written together with the code. Unit tests should validate expected functionality at the level of individual classes or small groups of collaborating classes.

Regression test suite should be built from tests involving package and component interactions and system level tests

• problems found in the pre-release QA process • user reported problems • system level tests based on requirements • test cases that have failed in the past

Validation tests with standard inputs and comparisons to reference output Coding standards compliance Dependency analysis Leak checking in 'predictable' places and/or where leak problems have been

observed in the past Examples and example code should be incorporated in the test suites, so that user

examples are continuously validated for correctness. All levels of testing should be run as part of automated nightly (and pre-release)

builds New tests should be added to a steadily growing acceptance test suite for releases,

used in pre-release testing and usable by end users to validate releases on their own. Performance testing should be included initially for measurement/monitoring

purposes. Once performance requirements must be met on delivered software, performance testing should be included in acceptance tests. This should include testing of scalability requirements.

Test failures on platforms or compilers being used only for portability assurance and bug finding should not automatically invalidate a candidate release


Software Distribution & Central Support

The full software base and associated web-based materials for both in-house and external software will be hosted in a uniform way at CERN, for the use of the LHC experiments and the CERN and general HEP communities

• A 'CERN Program Library' like service Software will be packaged and distributed so it is

eas for remote users to install, configure and use it• Any physicist should be able to do it!• The project cannot control the packaging of external

software, but a higher level packaging/distribution will be used such that all software, in-house and external, is distributed and installed in a uniform manner

A help desk function should also be hosted at CERN, providing first level support to users


Platforms

Software builds and test suites will be performed on multiple platforms and OS/compiler configurations in order to maintain portability of the software to future platforms and compilers, and to find more bugs

The suggested set is • Pentium, Sparc/64, IA-64 • Linux, Solaris, Windows • gcc, CC, VC++, icc (Intel), ecc (IA-64 version of icc)

Covering this full set is likely to be contingent upon substantial help coming from experiments making use of the less universally used platforms/compilers.


Coding standards and conventions

A concise set of coding standards (rules and guidelines) and naming conventions will be defined based on those already established (drawing on some or all of the four experiments, ROOT, and IT projects)

The standards will be enforced to the extent possible using a rule checking tool (IRST RuleChecker, Codewizard?) and rule set.

(CMS’s has the virtue of being concise!)


Software Metrics

Suggested package and component metrics: • test coverage (number of implemented tests) • test results • validation results • nightly build performance (failure rate) • bug counts in released code (from bug reports) • time to bug resolution • milestone performance • amount of invested effort • documentation assessment • coding standards compliance • compile performance (warnings, etc.) • number of dependencies • kLOC • commit count, frequency • date of last commit • release version count, frequency • date of last new release version • developer count (cvs committers) • metrics from dedicated software metric tools


Documentation

Standard pieces of documentation for a project: • Installation guide • User and reference manual • Design documentation • Code examples (tested as part of release acceptance) • Autogenerated web-based documentation and code

browsing • Release notes

Documentation should be updated in responses to changes and extended functionality, before the corresponding code enters a release

• For this to be realizable, it must be extremely easy for a developer to update the documentation

• Assistance must be provided, and mechanisms for 'encouragement’


Roles

Release manager • Manages release scheduling, definition, integration, acceptance testing, and creation• Works closely with package coordinators and project leaders to drive the timely completion of code,

fixing of bugs and successful integration of the software • Provides status reports and problem warnings on the progress of release integration • Coordinates integration of external and internal packages • Responsible for the completeness, functionality and consistency of deliverables included in the

release • Responsible for both major and minor releases • Generates and publishes the release notes

Librarian • Builds and tests releases on all reference platforms • Installs, packages and distributes the releases • Keeps external software up to date, installed, packaged, and available for distribution • Responsible for the operation of the automated build system providing nightly builds and release

integration builds • Installs, configures, maintains the tools supporting this process. May also participate in their

development and/or refinement. Toolsmith

• Installs and maintains all products and tools needed to support the development process. May also participate in the development and/or refinement of tools.

QA coordinator • Builds up comprehensive test suites and software metrics to evaluate and track software

functionality, correctness, performance and quality. • Proactively develops and incorporates software tests, particularly regression test cases and cross-

component system integration tests • Works with the release manager to establish acceptance test suites for releases • Develops web-based QA materials. Works with the librarian to develop and support comprehensive

software testing and metric results on the web • Configures and provides help and support for QA tools used in the project • Maintains and develops QA documentation (e.g. software standards and guidelines)

Technical writer


Tools

A long list, following the RTAG. See the web page.


Immediate Process/Infrastructure Tasks

Establish project AFS area, HEPix group/environment, general environment configuration

Establish strawman repository, with ancillary services Sourceforge investigation. Trial? Code browsing tools: LXR, cvsweb, Bonsai(?) Coding standards: assemble rules, guidelines and rule set from

existing experiments/projects Web server with Apache + tomcat + php/mysql MySQL service Infrastructure for uniform web based access/support/doc for third

party tools Needed tools and packages: ROOT, Python, Perl, ... Documentation infrastructure - e.g. doxygen Bug reporting system - Remedy or other? survey and implement SCRAM/CMT assessment soon Packaging assessment soon UML diagramming tool


Workplan priorities

Considerations for developing workplan and priorities:• Experiment need and priority• Suitability to a common project• Is it a key component of the architecture

E.g. object dictionary• Timing: when will the conditions be right to initiate a

common project Do established solutions exist in the experiments Are they open to review or are they entrenched

• Availability of resources• Allocation of effort

Is there existing effort which would be better spent doing something else

• Availability, maturity of associated third party software E.g. grid software

• Pragmatism and seizing opportunity. A workplan derived from a grand design does not fit the reality of this project


Top Priorities

1. Establish process and infrastructure• Nicely covered by software process RTAG

2. Address core areas essential to building a coherent architecture, and develop the architecture

• Object dictionary• Persistency• Interactive frameworks, analysis tools (also driven by

assigning personnel optimally)

3. Address priority common project opportunities• Driven opportunistically by a combination of experiment

need, appropriateness to common project, and ‘the right moment’ (existing but not entrenched solutions in some experiments)

Detector description and geometry model• Driven by need and available manpower

Simulation tools

Initiate: First half 2002


Near term priorities

Build outward from the core top tier components• Conditions database• Framework services, class libraries

Address common project areas of less immediate priority

• Math libraries• Physics packages

Extend and elaborate the support infrastructure• Software testing and distribution

Initiate: Second half 2002

Deliver: fall/late 2002: basic working persistency framework


Medium term

The core components have been addressed, architecture and component breakdown laid out, work begun. Grid products have had another year to develop and mature. Now explicitly address physics applications integration into the grid applications layer.

• Distributed production systems. End-to-end grid application/framework for production.

• Distributed analysis interfaces. Grid-aware analysis environment and grid-enabled tools.

Some common software components are now available. Build on them.

• Lightweight persistency, based on persistency framework

• Release LCG benchmarking suite

Initiate: First half 2003


Long term

Longer term items waiting for their moment (cf. considerations):

• ‘Hard’ ones, perhaps made easier by a growing common software architecture

Event processing framework• Address evolution of how we write software

OO language usage• Longer term needs; capabilities emerging from R&D

Advanced grid tools, online notebooks, …

Initiate: Second half 2003 and later


Conclusions

We’re now active in some important areas:• Process and infrastructure needed to do real work• A major project in persistency, with participation

planned from all four experiments Participation beginning to be quantified (e.g.

ATLAS (and CMS?) this week); need to complete this

Need to combine experiment needs with realistic goals to produce plan and schedule: e.g. a first usable version when?

• Regular meetings and activities are beginning Need to settle a permanent slot for the weekly

meeting Ready to engage participation by the experiments Need to actively engage the wider community in

participation, collaboration

Documents

Applications Area Status and Plans CMS CPT Week 18 April 2002 Torre Wenaus, BNL & CERN/EP LCG Applications Project Manager [email protected]