Java Root IO Part of the FreeHEP Java Library Tony Johnson (tony_johnson@slac.stanford.edu)...

Java Root IOJava Root IOPart of the FreeHEP Java Part of the FreeHEP Java

LibraryLibraryTony Johnson (Tony Johnson (tony_johnson@slac.stanford.edutony_johnson@slac.stanford.edu))Mark Dönszelmann (Mark Dönszelmann (duns@slac.stanford.eduduns@slac.stanford.edu))

Victor Serbo (Victor Serbo (serbo@slac.stanford.eduserbo@slac.stanford.edu))Max Turri (Max Turri (turri@slac.stanford.eduturri@slac.stanford.edu))

Presented at Computing in High Energy PhysicsPresented at Computing in High Energy PhysicsInterlaken, Switzerland, September 2004Interlaken, Switzerland, September 2004

ContentsContents GoalsGoals Java IO library for RootJava IO library for Root

– ImplementationImplementation– Test SuiteTest Suite– BenchmarksBenchmarks– Networked Root IO (rootd/xrootd)Networked Root IO (rootd/xrootd)– Demo toolsDemo tools– Root Plugin for AIDARoot Plugin for AIDA

Examples of UseExamples of Use– AIDATLD Web InterfaceAIDATLD Web Interface– JAS3JAS3– WIRED4WIRED4

Future plansFuture plans Links for more informationLinks for more information

Java Root IO: GoalsJava Root IO: Goals GoalsGoals

– Pure Java package for reading Root FilesPure Java package for reading Root Files Will be extended to Will be extended to writingwriting later later

– High PerformanceHigh Performance We want to do data analysis so we need very good We want to do data analysis so we need very good

performanceperformance– Should not need to know about objects ahead Should not need to know about objects ahead

of timeof time(no need for dll’s, .so files (no need for dll’s, .so files etc.etc.))

But should provide full support for reading user-But should provide full support for reading user-defined objectsdefined objects

– Easy to useEasy to use– Provide access to Root Provide access to Root datadata, not C++ code, not C++ code

ImplementationImplementation

Use “Use “TStreamerInfoTStreamerInfo” objects inside Root files ” objects inside Root files (since Root 3.0) to decode contents of file(since Root 3.0) to decode contents of file

Dynamically generate Java proxies for each Root Dynamically generate Java proxies for each Root object readobject read

Each proxy has custom “streamer” Each proxy has custom “streamer” – Build as Java Build as Java bytecodebytecode using BCEL ( using BCEL (http://http://

jakarta.apache.org/bceljakarta.apache.org/bcel) ) – Converted at runtime to machine code by HotSpot VMConverted at runtime to machine code by HotSpot VM

Uses java.nio package (new since Java 1.4)Uses java.nio package (new since Java 1.4)– Standard Java package for high-performance binary IOStandard Java package for high-performance binary IO

Implementation DetailsImplementation Details

TStreamerInfo

Java InterfaceAccessor method for each data member .java file

Java ProxyCustom version based on how object is stored in file

Implem

ents

InterfaceGenerator

Example of UseExample of Use

Test SuiteTest Suite Most classes are dynamically generated from info Most classes are dynamically generated from info

in Root file, so are fairly robust against changesin Root file, so are fairly robust against changes Need bootstrap process, in particularly need Need bootstrap process, in particularly need

enough custom Java classes to read the streamer enough custom Java classes to read the streamer info, and interpret itinfo, and interpret it– Relatively few classes, butRelatively few classes, but

This tends to change from one Root release to anotherThis tends to change from one Root release to another Typically have to make small changes in the bootstrap code Typically have to make small changes in the bootstrap code

for each new Root releasefor each new Root release We have test suite that creates a set of test files We have test suite that creates a set of test files

from each new root release, and verifies that our from each new root release, and verifies that our library can read the files.library can read the files.– Currently test 3.00.06, 3.01.06, 3.02.07, 3.03.07, 3.05.07, Currently test 3.00.06, 3.01.06, 3.02.07, 3.03.07, 3.05.07,

3.10.02, 4.00.083.10.02, 4.00.08 In addition we have a set of Root files collected In addition we have a set of Root files collected

from users, and the test suite verifies that we can from users, and the test suite verifies that we can (continue) to read these files with each release.(continue) to read these files with each release.

Java Root IO - BenchmarkJava Root IO - Benchmark

1.4GHz AMD Athlon, Windows 1.4GHz AMD Athlon, Windows XP, 512MByte DDR, ATA100 XP, 512MByte DDR, ATA100 disk..disk..

SS= Split level = 0,1,2= Split level = 0,1,2 CC= Compression = 0,1= Compression = 0,1 Each test run 5 times in Each test run 5 times in

succession, lines show range in succession, lines show range in results results – in later runs file is typically in later runs file is typically

cached in memory == faster if cached in memory == faster if task IO limitedtask IO limited

Task: Loop over all events, loop over all tracks, sum Task: Loop over all events, loop over all tracks, sum px for all tracks. 5000 event files created with “event” px for all tracks. 5000 event files created with “event” test program distributed with Root (200-400 MByte).test program distributed with Root (200-400 MByte).

C S ms/Event

0 0 12.3 - 4.23

0 1 10.7 -- 2.75

0 2 1.70 -- 0.20

1 0 10.5 -- 5.12

1 1 9.6 -- 4.77

1 2 2.7 -- 0.25

Network Root IO (x)rootdNetwork Root IO (x)rootd Recently added ability for Java library to Recently added ability for Java library to

talk directly to rootd or xrootd servertalk directly to rootd or xrootd server– Defined “root:” Defined “root:” protocol handlerprotocol handler for URL class. for URL class.– Defined Defined AuthenticatorAuthenticator for authenticating to for authenticating to

rootdrootd– Extended Extended RootFileReaderRootFileReader to accept (any) URL to accept (any) URL

argument to constructorargument to constructor If URL corresponds to If URL corresponds to file:file: read the file directly read the file directly If URL corresponds to If URL corresponds to root:root: protocol use xrootd/rootd protocol use xrootd/rootd

(random access)(random access) If URL corresponds to If URL corresponds to http: http: and web server supports and web server supports

http 1.1 “ranges”, use extended httphttp 1.1 “ranges”, use extended http Otherwise buffer URL contents into memory and read Otherwise buffer URL contents into memory and read

from there (suitable for small files only).from there (suitable for small files only).

Authentification IssuesAuthentification Issues Authentification can be set programmatically, or via Authentification can be set programmatically, or via

GUI AuthenticatorGUI Authenticator Rootd supports many authentification schemesRootd supports many authentification schemes

1.1. Anonymous (c.f. anonymous ftp)Anonymous (c.f. anonymous ftp)2.2. User/Password (clear text)User/Password (clear text)3.3. User/Password (RSA encrypted)User/Password (RSA encrypted)4.4. SCPSCP5.5. SSH likeSSH like6.6. GSI certificateGSI certificate7.7. UID/GID UID/GID

• Currently FreeHep library supports only 1 and 2Currently FreeHep library supports only 1 and 2• Work underway on 3 (but some compatibility problems Work underway on 3 (but some compatibility problems

between Root and Java RSA implementations)between Root and Java RSA implementations)• Java libraries exist for all other methods, so can be Java libraries exist for all other methods, so can be

supported in future.supported in future.

Network Root IO: Example of UseNetwork Root IO: Example of Use

Demo Apps: Root Object BrowserDemo Apps: Root Object Browser

Demo Apps: Root Histogram Demo Apps: Root Histogram BrowserBrowser

Root Plug-in for JAIDARoot Plug-in for JAIDA JAIDAJAIDA = Java implementation of AIDA = Java implementation of AIDA AIDAAIDA = Abstract Interfaces for Data Analysis = = Abstract Interfaces for Data Analysis =

Histograms + Tuples + Fitting + Plotting + IO data Histograms + Tuples + Fitting + Plotting + IO data analysis package analysis package – (See Monday’s talk by Victor Serbo)(See Monday’s talk by Victor Serbo)

Root plugin for JAIDA makes it possible to read root Root plugin for JAIDA makes it possible to read root files via the AIDA interfaces and treat the objects files via the AIDA interfaces and treat the objects contained in the root file as AIDA objectscontained in the root file as AIDA objects– Useful if:Useful if:

You like AIDA interfaces better than root interfacesYou like AIDA interfaces better than root interfaces You want to use language not supported by RootYou want to use language not supported by Root You want to analyze data in mixed formatsYou want to analyze data in mixed formats You want to work in J2EE, J2ME, Applet You want to work in J2EE, J2ME, Applet etc.etc.

Root Plug-in for JAIDA: ExampleRoot Plug-in for JAIDA: Example

Root Java IO: Root Java IO: Examples of UseExamples of Use

AIDATLD Web InterfaceAIDATLD Web Interface

JAS3JAS3

WIRED4WIRED4

AIDATLD + RootAIDATLD + Root AIDATLD = AIDA Tag AIDATLD = AIDA Tag

Library for use in JSP Library for use in JSP

pagespages – Trivial to put “live” Trivial to put “live”

histograms on webhistograms on web– Directly supports Root Directly supports Root

filesfiles

AIDATLD Example: GLAST System AIDATLD Example: GLAST System Tests Web InterfaceTests Web Interface

JAS3 + RootJAS3 + Root

JAS3 = Modular, Extensible, Data Format JAS3 = Modular, Extensible, Data Format Independent data analysis system.Independent data analysis system.

Among many other formats JAS3 supports Among many other formats JAS3 supports RootRoot– Contents of Root files can be accessed directly, Contents of Root files can be accessed directly,

or via Root->AIDA interfaceor via Root->AIDA interface Full access to AIDA functionality for fitting, projections, Full access to AIDA functionality for fitting, projections,

plotting etc.plotting etc.

– Can operate on Root data Can operate on Root data using GUIusing GUI using scripting (Python, Pnuts) using scripting (Python, Pnuts) using compiled Java codeusing compiled Java code

JAS3 Access to Root FilesJAS3 Access to Root Files

JAS3 + Root: ScriptingJAS3 + Root: Scripting

Root + WIRED4Root + WIRED4 Build yourself a super cool event display for your custom Build yourself a super cool event display for your custom

Root file in under 1 hour!Root file in under 1 hour!

Future PlansFuture Plans Continue to track new Root releasesContinue to track new Root releases

– Root network protocolRoot network protocol More work on authentificationMore work on authentification

– Root file writingRoot file writing Most of work already done by Petr Vokac/Julius Most of work already done by Petr Vokac/Julius

HrivnacHrivnac Easier than reading because only need to support Easier than reading because only need to support

one root protocol versionone root protocol version Support Root Support Root XML formatXML format

– Will be much slower, but perhaps more robustWill be much slower, but perhaps more robust Use Use LCGDICT systemLCGDICT system to allow Java access to allow Java access

to C++ methods as well as data?to C++ methods as well as data?– Probably also slower, but more flexibleProbably also slower, but more flexible

More InfoMore Info

Freehep Java LibraryFreehep Java Library– http://java.freehep.org/http://java.freehep.org/

Root IO component of libraryRoot IO component of library– http://java.freehep.org/lib/freehep/doc/root/http://java.freehep.org/lib/freehep/doc/root/

AIDATLDAIDATLD– http://http://aidatld.freehep.orgaidatld.freehep.org//

JAIDAJAIDA– http://http://java.freehep.org/jaida/index.htmljava.freehep.org/jaida/index.html

JAS3JAS3– http://jas.freehep.org/jas3http://jas.freehep.org/jas3