OpenScientist (13.0)

CHEP'04 G.Barrand, permanent debugger of CERN software at LAL.

1

OpenScientist (13.0)

An integration to do

scientific visualization and data analysis

http://www.lal.in2p3.fr/OpenScientist


2

What is OpenScientist ?

• It is an INTEGRATION of products working together to do scientific visualization and data analysis.• It is NOT one million lines of home made code. It is not a reinvention of everything and then cannot be considered as another “// The

ROOT of EVERYTHING”.• This INTEGRATION is driven by the following rules :

• Follow some “software least action principle” : reach a maximum of functionalities by minimizing the number of home made code and number of packages in general.

• For GUI and graphic, use what desktop providers offers (to have full speed).• For the rest, use a maximum of dedicated open source software. • Target the local desktop machines (since now most of people have their own (powerful) machine). Then targets are locals Linux(es),

MacOSX and Windows.• Have a flexible architecture to be able to EVOLVE. This is done by having “hub” packages for doing the coarse graining couplings

and by using, among other things, pure abstract interfaces to minimize these couplings.

• “Use open source” does not mean that we have nothing to do : the work is displaced in the tuning of things to bring to users a consistent environment.


3

Vis and GUI products

• Language : stick to C++. Because data frameworks (event model managers) are in C++ but also because C and its derivative (C++, ObjectiveC) are still the native languages of desktop providers.

• Rendering layer : more than ever : OpenGL.• Scene manager : Open Inventor. By using the Coin3d implementation of System

In Motion (Norway).• Do all graphic with the uppers (including 2D (for example histogram plotting))• GUI : The mess. No standard API emerged. Too much choices. GUI is still the

horse battle of major desktop actors : Bill Gates (Blue Screen Company) with Windows. Steve Jobs (Apple) with Cocoa (in fact NextStep), Richard Stallman (GNU) with GNOME / gtk and some cute unknown for KDE / Qt.

• All of them put together make our life a nightmare concerning GUI question :

WE NEED SOME STANDARD API coming from them (some XML ?) .


4

• An home made GUI toolkit ? No, it will bring us back to the age of stone. • Choose a cross platform toolkit ? “frustrating” solution. Someone fond of

[Windows,Cocoa,KDE,GNOME] wants to use [Windows,Cocoa,KDE,GNOME] on its machine.

• Then try to use what desktop providers offers :– Use Windows on a Windows– NextStep on a Mac– Gtk on a Linux/GNOME– Qt on a Linux/KDE– Motif on other UNIXes (but in fact, Motif still the fastest solution on a Linux…)

This way offers (obviously) the best performances on local machines. GUI described in XML….

GUI : three stratégies


5

• OnX : the integration place for GUI, graphic and scripting. Massive usage of abstract interfaces here.• Light home-made solution over existing GUI toolkits. • It should be seen as some omni-interface-builder (from an XML description of the GUI). • “Drivers” exist for Xt/Motif, Windows, gtk, Qt, NextStep. OnX could be seen also as a factory for these GUI toolkit.• Integration of Inventor viewers for the upper drivers done (SoXt, SoWin, SoQt, SoGtk, SC21 from SIM)• Scripted callback :

• Dld of “extern C” functions : <widget class=“PushButton”> <activate exec=“C++”>OnX ui_echo hello</activate> <activate exec=“C++”>MyDLL my_cbk arg</activate> </widget>• Python (native code wrapped with SWIG) : <activate exec=“Python”>from import OnX *;ui.echo(‘hello’)</activate>• KUIP : <activate exec=“kuip”> h/plot the_famous_10</activate>• CINT : <activate exec=“CINT”>{AIDA::IAnalysisFactory* aida = ….}</activate>

OpenScientist / OnX package

extern “C” {void my_cbk(IUI&,std::vector<std::string>&) { /*my code*/ }}


6

OnX

OnX

Xt, Win, Qt, gtk+

Dld, Python, CINT

GL,Inventor

XML


7

GUI paradigm

“A la PowerPoint” : a compact GUIs organized around a document area.

This is 100% in opposition to a terminal prompt spawning viewers around the screen


8

LHCb / Panoramix on a MacOSX by using native Cocoa (NextStep) / OpenGL.


9

• Integration place for statistical tools.• AIDA-3.2.1 compliant. It works (sorry Rene Brun (alias TRex) and T-all). It is simple, which is not the same than being “naive” (sorry Pere

Mato). It does not use ROOT and stick to AIDA (sorry Vincenzo Innocente and Torre Wenaus)• HCL for histograms : 20 % faster than THs (sorry T-all, you are behind). Why faster ? Because a std::vector is much faster (in -O) than a

reinvented TArray. In particular it means that LCG / PI::Histogram that uses TH is already the slowest solution of the known universe. HCL::Histogram is a truly multidimensional histogram class over std::vector. In particular in which a 1D object does not have forever

dummies TAxis fYaxis, fZaxis fields irrelevant for a 1D object ; in which there is no inheritance of graphical attributes (done differently) ; etc….

• Rio (see poster) : light and clean package for doing IO at the ROOT format (What is a file at the ROOT format ?). With that OpenScientist is ROOT FREE (a huge relief).

• Fitting : migrate from Midnight to C++/Minuit done at CERN. This shows that the author is not systematically against CERN software…• HEPVis / SoPlotter for the plotting. More and more efficient, more features. (Is a plotter ever complete ?). See OpenPAW poster for views.• NEW : OpenPAW… (also in poster).

OpenScientist / Lab


10

• opaw program : it is an OnX application to emulate PAW. OS> opaw [my.kumac] OS> opaw -gui [my.kumac]• Could be seen as a PAW interactive front end to AIDA. (Or AIDA could be seen as the

C++ API to OpenPAW).• C part of KUIP taken (for long) from old CERNLIB.• Pawcdf.cdf taken from old CERNLIB. Then SAME syntax as PAW.• pawex1 to 24 emulated with, in general, better performances on most aspects.• COMIS replaced by on the fly compilation and loading. It works with f77 but also with

C and C++ without having the burden of interpreters.• Vectors and SIGMA commands are here too.• See poster at the Thursday morning session.• Obviously not all command (and options) recovered but things are under way…

OpenPAW


11

OpenPAW

OS> opaw pawex10

OS> opaw pawex1

OS> opaw -gui -Xt pawex9


12

• G4Lab : visualization for Geant4. It needs that Geant4 be dynamically loadable (problems on MacOSX and Windows with latest release of G4).

• G3Lab : visualization for Geant3 (heavily used by some at LAL).• Mangrove : visualization for ROOT seen as a data framework. We first get rid of any GUI and

graphic coming with ROOT and come with an embryo of TVirtualPad over Inventor. It permits to visualize a TGeo with the OpenScientist tools (then OnX and Inventor).

See Panoramix talk for issues.• Hippo : to integrate hippodraw core. See Panoramix talk for issues.• OnXSvc : a Gaudi service for OnX exists. See Panoramix talk.

Other modules


13

Packages

Who can show something so clean ?


14

Pure abstract interfaces (PAI) to minimize coupling

• Principle : • virtual IA::toBeUsed() = 0;• Class A : public virtual IA {}. Code put in some libA.• B::use(IA& a) {a.toBeUsed();}. Code put in some libB.

No need to link libB to libA !!!! It simplifies a lot. (Relationship established at compilation time through a “#include <IA.h>” in B.cxx and at run time through dynamic

loading)• Geant4, ROOT, hippodraw, etc… have “base classes” only virtual BaseA::toBeUsed(){} that induces to link libBs to libA. They (we) would gain a lot by pushing one step further : BaseA -> IA (for exa if working with ROOT, no need to link with any libCore, etc…)• (HCL/AIDA vs TH shows that it is not an issue at run time)• Moreover, PAIs permit to discuss “what are things” and fix user APIs without caring of the concrete implementation :

{ PAI } = “the bosonic virtual software bus” to tied fermionic developers !!!


15

New, some manpower !

• LAL finally recognize that the author may need some help… A young engineer, Laurent Garnier, had got a job at LAL to help on the graphical questions (and then on the OpenScientist tools).

• Already contributions around SoPlotter (hatching) and OnX / gtk-2.• But also some help to create logos.


16

More than ever : = Anti OpenScientist

• TClass::Draw() : Sorry people, I still do not believe that a software that has a draw method on its basic introspection system is a breakthrough of humankind. (*).

• In front of all good quality dedicated things available for free now, why, in 2004, do we have to bear forever knotty cranky and secretly-garage-made classes for storage, GUI, scene manager, string, math, “Virtual Monte Carlo”, etc… ?

• That’s right that ROOT comes with an IO system ; but at which cost ! (200 klines of code to store an histogram !). Rio demonstrates that we can do it with 20 klines of dedicated code only. WE NEED AN APPEALING OODB, DOT.

• Already slugish : TH vs HCL demonstrates that. TGUI and TGraphic will be forever slower than anything based on the unavoidable native desktop providers products.

• Hard to swallow : the bright borrowing of CERN name (**) whilst software engineers of institutes that participate to the CERN program for years had not been consulted on the overall design and had seen their request for a major revision for the LHC put aside. In fact we are in the same position than in front of a private company product…

• When are we going to have on our (blue) screen something like the below ? *** Error : your ROOT license had expired. Contact RootSoftInc (previously CERN) to renew your payment. *** (and please, don’t laugh, remember CMZ from CodeME).• What are the priorities ? create private companies (on our back) or run collaborative experiments ? It is not the same.

PHYSICISTS, WHEN ARE YOU GOING TO OPEN YOUR EYES ?

* And I am not the only one : among others, creators of C++, java, C#, Python, Inventor, Qt, NextStep.** See Linux Journal of July 1998 : “ROOT an OO data analysis framework. A data analysis tool developed and used by CERN”


17

• CERN / LCG did not order a unification of engineering researches done around « data-frameworks » in various experiments. This was definitely an historical mistake.

It was « only » a question to compel five to six men to put aside their personal ambitions and to work for

six to twelve months on converging (*) their things. In particular, It was the last hope to compel a major revision of ROOT in order to federate engineers (and not only physicists) for the LHC around one appealing common basement. CERN HAD NOT BEEN ABLE TO DO THAT.

• Today the « LCG uses ROOT » leads already to something like :

in which someone can find : two dictionaries (that will never be merged), three plug in managers, three build systems, etc…. All this though with different logics. AN ASTOUNDING PAIN.

• A sociological and engineering shame : LHC does not deserve that.

“LCG uses ROOT” : an already disaster

gaudi/sealpool pi root/thcint

Gaudi Some CMS basementRoot Some Susy Framework

root/iolcgdict python

* Converging and NOT choose one by eliminating others.


18

What is CERN ?• What is CERN ? For me (*), CERN had been created, first of all, to federate scientific and engineering forces

of a set of countries to run (big) experiments to do physics. Part of accelerators and detectors, after agreements between members, are build in home institutes or private companies and assembled at CERN, where this place offers the infrastructure and organization to assemble and operate all that.

• SOFTWARE is clearly an engineering deliverable and what have to be provided and maintained is clearly out of scope of one man alone. Then software must follow the same paradigm than machine and detectors.

• It means, in particular, that this system can’t work by putting engineers and technicians in pure concurrency from begin to end. At some point a convergence on technologies must be done in order to put in sync ALL PARTIES ; mainly because the gifted manpower (**) is in fact spare and that everybody is needed.

A set of fermionic developers must become at some moment a bosonic group.• “LCG uses ROOT” (another way to say “take it and shut up”) shows that some people, part of the program,

are out of engineering control. By violating the idea of federation, it signs the death of CERN as an engineering federating place for doing scientific software. By giving engineering immunity to some, it opens, once more, the way to another era of poor engineering and overall mediocrity concerning software questions.

And anyway happy birthday CERN (at least the idea of it was great)

* I am not the only one, see “point c” in CERN Courier March 2004. Robert Aymar “CERN’s role on the European stage”** How much people know the internal of Gaudi & Geant4 & POOL & ROOT ? It could be counted on one hand.


19

• What else to say ? The OpenScientist integration works and is used.• To the question of Fons Rademaker (the second TRex) posed some time ago at an

Anaphe meeting : “Do you have something to show ?” the author answers now : “Yes, I have something to show, and I show it in a better way than you !”

• Due to the overall context :

Conclusions

More than ever, OpenScientist should and will continue…

Documents

OpenScientist (13.0)