February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL1 How to progress fast towards a new OO HEP library? Some ideas on prerequisites and impediments

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL

1

How to progress fast towards a new OO HEP library?

Some ideas on prerequisitesand impediments


2

Outline

Disclaimer

Some thoughts

Summary and Outlook


3

Disclaimer

This is not a talk about prerequisites: software project management software engineering software architecture data models

I have no new ‘silver bullet’ to offer! go and read (one of many good books):

“How to run successful projects II” by Fergus O’Connel


4

Disclaimer(2)

This is only my biased, imperfect view.

The whole talk is meant to trigger discussion.


5

A few observations on software projects

Fraunhofer Institute for Software and Systems Engineering (Berlin, Germany):

33% of software projects fail

Of those which don’t fail: 80% last substantially longer than planned and cost

more than estimated

Track record in HENP is of the same order of magnitude!!


6

Software projects: A comparison

If you want to build a summer house it is sufficient to employ a bricklayer who will do it using his experience.

If you want to build a skyscraper or a bridge you need foremen, engineers, architects and an agreed process. Nobody would do it with a 1000 bricklayers only!!! (except physicists)

A bridge lasts 100 years, very good software lasts 10-20 years at most, much software in HEP experiments survives < 5 years.


7

How to progress fast towards a new

OO HEP library?

Using OO in HENP for now is a fact, and a requirement.

We should careful evaluate which parts are HENP specific.

CERNLIB style libraries: Quite some code and algorithms are

‘reinvented’ over and over => incompatible


8

Fergus O’Connel: The silver bullet (1)

Planning the [software, mka] project1: Visualize the goal; set your eyes on the prize

2: Make a list of jobs to be done

3: There must be one leader

4: Assign people to jobs

5: Manage expectations, allow a margin error, have a fallback position


9

Fergus O’Connel: The silver bullet (2)

Implementing the plan/Achieving the goal6: Use an appropriate leadership style

7: Know what is going on

8: Tell people what is going on

9: Repeat 1-8 until 10

10: The prize (“oh what fun, my job is done”)


10

Planning - 1: Visualize the goal

A new OO HEP library?!! Contains all experiment-non-specific but HEP-

specific algorithms Elements for common solutions for HEP problems

Robust, debugged, available for different OS’s compatible to all or most commonly used non-HEP

packages Leverage knowledge from public domain or IT industry

allows HEP to concentrate on HEP-specific problems May need adaptation for HEP


11

Example: CMS Software Tasks

This talk

Why not have a HEP framework?


12

Issues for Core Software Types

Non - HEP software Version evolution on multi-platforms Distribution and possible licensing Cost Long term availability of commercial product

HEP Software Manpower costs for development and maintenance Technical requirements for “common items”

Layered Architecture Standard Application Program Interfaces (APIs) Phasing: early linkage with key experiment-specific

architecture decisions


13

Example: LHC ++ LHC++ is a HEP-wide effort to provide a suite of OO software covering roughly the functionality of CERNLIB

Use public domain or commercial SW when appropriate Develop own HEP software if necessary Collect and distribute SW; take care of possible licensing

Develop a well-engineered toolkit of software components Build interface layers for HEP usage Develop/acquire, maintain and replace components separately Build different applications from components

Use standards and standard solutions Use of standards may help in maintaining and developing

software over the long lifetime of the experiment

LHC++ has ‘one big’ problem: already the name indicates that is it not meant to be a HEP-wide solution ;-)


14

LHC++ components

Standard components C++ libraries, STL: Object Space Basic mathematics: NAG C library ODBMG database: Objectivity/DB Standard graphics: OpenGL, OpenInventor, Qt, ...

HEP components Simulation tools: Pythia7, Geant4 Data analysis, graphics, visualisation: HEPVis, HEPTags,… Mathematical libraries: Gemini, HEPfitting Histogramming: HTL Foundation level class libraries: CLHEP, HepODBMS


15

Planning - 3: There must be one leader

A leader is required who can lead who is leading who is identified with the project and wants to be successful

One and only one leader is required, not a committee.

Bridges are not build by 1000 bricklayers who do what they want.


16

Implementation - 7: Know what is going on

Use the plan to guide the project.


17

Implementation - 8: Tell people what is going on

Progress reviews, updated ‘work-plans’ or WBS’s.

Good and bad surprises should be published instantly in order to adapt.


18

Implementation - 9: Repeat 1-8 until 10

Be prepared to take long term commitment to finish project there is no shortcut to finish the project

without cutting back in requirements.need to compromise on short-term goals


19

Implementation - 10: The prize

Now the maintenance starts: needs resources and commitment, again: conflict of interest for scientists who want to operate

experiment and do HEP science.

We have to get serious: Learn to involve professionals from IT disciplines (like

engineers in detector building). HEP software designers needs a career path like detector

designers and builders. We should assign a higher status to those who release their

scientific code with open source licenses.


20

How to progress fast towards a new OO HEP library?

My preferred model is: an idea/goal is connected to a person (leader, PI) an approval process happens resources must be allocated

at universities, IT-divisions, in experiments progress is tracked feedback and early buy-in by customers operations and maintenance phase must be covered.

Use ‘silver bullet’ steps 1-10.

This is how experiments are build, also applied to successful software projects.


21

How to convince experiments to use a new OO HEP library?

Experience shows:

It is hard to impossible to coordinate or enforce usage of common products.

Different time-scales and decisions in experiments make common development difficult.

Chance for Success:

Persuade with good quality software and functionality.

As systems become more complex, calculations that are verifiable in principle are not verifiable in practice without public access to the code.

Authors of open scientific code should be recognized in the same way as they are recognized for scientific papers.


22

Summary and Outlook

We have to develop and grow the understanding of the benefit of a common library to achieve a common long-term goal.

Good ideas and leadership must be acknowledged and supported with resources.

We want the best algorithms and modules available to HEP, this needs the best people!

The academic career path for HEP software builders must be better developed.

Publishing code should be equivalent to publishing a paper!


23

Planning - 2: Make a list of jobs to be done


24

Planning - 4: Assign people to jobs

Each job has a name


25

Planning - 5: Manage expectations, allow a margin error, have a fallback position


26

Implementation - 6: Use an appropriate leadership style

Documents

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL1 How to progress fast towards a new OO HEP library? Some ideas on prerequisites and impediments