The HERA-B detector The database problem The Architecture The Berkeley-DB DBMS

A. Amorim

1

The HERA-B database servicesdetector configuration, calibration,

alignment, slow control, data classification

• The HERA-B detector• The database problem• The Architecture• The Berkeley-DB DBMS• The client/server integration• The domains and solutions• Conclusions and Outlook

A. Amorim, Vasco Amaral, Umberto Marconi, Tome Pessegueiro, Stefan Steinbeck, Antonio Tome, Vicenzo Vagnoni and Helmut Wolters

A. Amorim

2

HERA-B Experiment B/B-tagging B/B-tagging

B0/B0 J/KS B0/B0 J/KS

Vertex Detector Si strip 12 m resolution

RICH (/K)multianode PMT

TRD (eh)straw tubes+thin fibers

ECAL(+eh)W/Pb scintillator

shashlik

MUON (h)tube,pad and gaspixel chambers

Tracking: - ITR(<20cm): MSGC-GEM - OTR(>20cm): 5+10mm drift cells

Magnet: 2 Tm

C4F10

HiPt triggerpad/gas pixel

A. Amorim

3

Magnet

Muon 1,3,4

SVD(not visible)

The HERA-B detector

ITR- OTR chambers

TRDOTR

ECALRICH

A. Amorim

4

The main challenge: Selecting

A. Amorim

5

How do we select them?

Time scale

Input rate

10s

10MHz

5ms

50 kHz

200ms 4 s

500 Hz 50 Hz 20 Hz

Pretrigger: ECAL, System, p T padsL1: e/“Tracking” in4 SL, p T cut, mass cut

Pretrigger: ECAL, System, p T padsL1: e/“Tracking” in4 SL, p T cut, mass cut

1/200

L2: + drift times, magnettraversal, vertexing

L2: + drift times, magnettraversal, vertexing

L3: full track & vertexfit, +SVD tracks, p. id

L3: full track & vertexfit, +SVD tracks, p. id1/100

L4: + full reconstruction,physics selection

L4: + full reconstruction,physics selection

TAPETAPE

1/2.51/10

A. Amorim

6

HERA-B DAQ

L2-farm: 240 PC’s L4-farm 200 PC’s

1000 SHARC (DSP)

Detector Front End Electronics

FCS

DSP SWITCH DSP SWITCHEvent Control

TriggerPC

TriggerPC

TriggerPC

INTERNET SWITCH

4LTPC

4LTPC

LoggerPC

SLT/TLT

4LT

A. Amorim

7

The HERA-B database problem

To provide persistence services (including online-offline replication) to:

• Detector configuration

• Common accepted schema

• Calibration and alignment • Distributing information to the reconstruction and trigger farms

• Associate each event with the corresponding database information

•Slow control

• Manage updates without data redundancy

•Data set and event classification

•Online Bookkeeping

• Detector Configuration

• Calibration and Alignment

• Slow Control

• Data Set and Event Classification

• Online Bookkeeping

A. Amorim

8

Characterizing the context

Inter-processdata-flow rate

Complexity

Persistent Data-Volume

ARTE in memory structures:· Accessed very often by 1 process· Transient data limited to memory· No versioning, high complexity

Event Stream:· Accessed 2-3 times by 1 process· Large integrated data volume· Relatively simple structure

Database Services:· Accessed every data-set or slow control

update by Several hundred processes· Persistent data always available for retrieval· Time and version, high complexity

Analysis

20 Terabytes /year

Gigabyte/update

Large number of classesMerging data and updates

Complex queries

A. Amorim

9

Querying on time intervals

one can select objects by the values of their attributes

Most of our requests want to query on time or version

exception: Event Tag Database: select on (Part Type, E, Pt, etc).

Query on time => Object(Time) or Object(t1,t2)

Our simple database layer on top of Berkeley DB provides that

otherwise one has to specialize the DBMS

( example the conditions database of BaBar and R&D45)t

A. Amorim

10

Keys, objects and client/server

SDB/local thin layerMIZZI

C- formatter/client

Datastore Server

SDB/client thin layer

HERA-B client server infrastructure (DAQ software)

C- formatter/localMIZZI

Berkeley DB / Sleepycat embedded database managerOpen Source

client/server at the SDB level +RPM -> an UDP based communication package.

/PM/ Descrip. field1 ; field 2; ...

.2

.5-.1

56892

...Db: /RICH/HV/

Db: /RICH/HV/

versions

Key= name+ version

Machine independent blub of DATA

A. Amorim

11

The Berkeley DB

See http://www.sleepycat.com/

• Embedded transactional store with: logging, locking, commit and roll back, disaster recovery.

•Intended for: high-concurrency read-write workloads, transactions and recoverability. • Cursors to speed access from many clients..

• Open Source policy The license is free for non-commercial purposes - rather nice support

•No client/server support is provided

A. Amorim

12


C- formatter/client

Datastore Server





Slow Control Interface

Slow Control interface Client/server

Metadata Object Data Object Update

time

Pmt1000Pmt1003Pmt2000

Update

1 ...1.5 ...2 ...

1.2 ...

1.6 ...2.3 ...

slctemp

nfields : intcolumsize : intfieldtype : int*xbuf : void *ids : int *columdata : void *

LocCopy2Table( )LocCopyTable( )LocMergeSlcTable( )

MIZZItable

SlcDatabase

x : Tablesdbname : char *

SlcBookDatabase( )

db

1..1

.sch tupd.dat

tdat

SlcTable

nused : int = 0nmax : intcarray : char **tabledes : char *tabledat : char *tableupd : char *maxnupd : intslctemp : slctempdat *nchardef : int

SlcBookTable( )SlcUnbookTable( )SlcGetDatabase( )SlcOpenTables( )SlcCloseTables( )SlcAdd2Table( )SlcFindInTable( )SlcGetDescTable( )SlcPutDescTable( )SlcStoreUpdTable( )SlcStoreDataTable( )SlcMemTable( )SlcTableValues( )SlcTableGetpt( )SlcTableGetSpt( )

1..1

(trough the tabledes name )

.upd

Optimized Queries

A. Amorim

13

Associations to Events

Index Objects (referenced by events) Client/server

Active server interface


C- formatter/client

Datastore Server






Index Obj. Index Obj.

... ...

Index Obj.

...

Revision 0 - online

Index Obj.

Calibrating

1 - offline

Dynamic Associations

Index Obj. Created in active

Servers

A. Amorim

14

The index API design

servers

transtabs

nend

nstartkey_server

name : char *vnew : int

key

n_servers : intmax_servers : intn_tables : intmax_tables : intn_transtabs : intmax_transtabs : int

add2servers( )add2tables( )add2transtabs( )marknotused( )cleannotused( )

tables

transtab

key_transtab

x : Tableskeyt : intmajor : intminor : intvnew : intline : intdbname : char *tablename : char *

key_table

name : char *major : intminor : intvnew : int

transtab == line

Index objects can associate also to transaction objects which are not data but are associated to sets of data objects that must be considered together.

Tools were also developed

A. Amorim

15

The Parameter Distribution (cont.)

Calibration/Alignment Process Calibration/Alignment Process

C&A Manager

These are the first servers to startup

startup in second place startup in

third place

Db_Server

Db_Server

Db_Server

KeyDb_server

5: StoreTable(Table)

1: SUBSCRIBE(Table)

6: UPDATE(Table)

2: SUBSCRIBE(Table)

3: SUBSCRIBE(Table)

4: SUBSCRIBE(KeyTable)

7: UPDATE(KeyTable)

A. Amorim

16

A. Amorim

17

Basic n-n associations (LEDA)

LEDA - Object Manager (hash table implemented associations)

Key objects (referenced by events) Client/server

Active server interface


C- formatter/client

Datastore Server






Key key’

Key key’

•Associations are navigated with iterators•Using hash tables.•Keys as OID’s with the scope of classes. •Explicitly loaded or saved (as containers)

A. Amorim

18

GUI for editing and drawing

From R&D: JAVA, TCL/TK, gtkReusing and extending widget.

Data hidden from TCL/TK

ROOTdatabaseBindingSocket:

Client/ Server

A. Amorim

19

General Architecture


C- formatter/client

Datastore ServerSlow Control interface

Key objects (referenced by events)

Replication

Tcl/TkOpen Source


ROOTOpen Source



LEDA - Object Manager(hash table implemented associations)


Database GUI Spreadsheet

Db GUIPlotting

Root ObjectPersistency

Client/server

Client/server

Socket

Active serverinterface

MySql, RDBMSOpen Source

Tag and Data Quality

Tag and D/Q replication

109 Evt./y

A. Amorim

20

The Cache Server

Userclient

Userclient

Userclient

Cache

Cache

Db- server

Db- server

Cache

Memory

WAN TCP/IP gateway

A. Amorim

21

The Replication Mechanism

DB server

ONLINE OFFLINEfirewall

DB server

Incremental

dump files

Offline DB server

imported

Send to tape Offline

DB server

A. Amorim

22

DAQ configuration

Software Components

Logical Hardware

Physical General Hardware

Fast Control System

FLT

Slow Control Hardware

Subdetector FED

A. Amorim

23

DAQ (Software

Components)

DAQ_ProcTemp

type : const char[64]name : char[64]path : char[64]exec : char[64]env : char[256]args : char[256]restart : unsigned int

(from DAQ_CONF)

Software Component:Run Configuration

Software ComponentSMC Configuration

Software ComponentProcess Templates

Software ComponentProcess Configuration Trees

ProcCfg

RunCfg

DAQ_RunCfg

name : char [64](from DAQ_CONF)

ProcCfg

SMCCfg

DAQ_SMCCfg

name : char [64]sst : unsigned intgsdt : char [256]sdt : char [256]

(from DAQ_CONF)

up

down

NodeGrp

ProcCfg

ProcTemp

ProcCfg

DAQ_ProcCfg

grname : char[64]domain : char[64]name : char[64]range : unsigned int [2]env : char[256]args : char[256]init : char[64]init_args : char[256]external : unsigned intcritical : unsigned intrestart : unsigned intsynch : unsigned intfini : char[64]fini_args : char[256]

(from DAQ_CONF)

DAQ_R_RunProc

DAQ_R_SMCCfg

DAQ_R_ProcCfg

IUdownIUup

NodeGrp

DAQ_NodeGrp

name : char[32]type : char[32]

(from DAQ_CONF)

DAQ_R_ProcNgrp

ProcTemp

Software Components

DAQ_R_ProcTemp

DAQ_R_UITemp

DAQ_R_TempNgrp

A. Amorim

24

VDS databases

A. Amorim

25

Maintaining the system

• A slow control process is permanently checking the state of the database servers.

• It issues alarms for the detector shift crew

• Tools to start and stop the dynamic configuration of database servers are to be used by a set of experts.

• The configuration and startup of the distributed database server system is performed using a special configuration database for this system.

A. Amorim

26

ConclusionsONLINE:• Large number of Clients => Gigabytes per Update

•broadcast simultaneously to SLT• tree of cache database servers to the 4LT

• Correlates (dynamically) each event with the databases objects• 600 k SLC parameters using data and update objects

•parameter history is re-clustered on the database servers • The online database system has been successfully commissioned

OFFLINE:• Replication mechanism decouples online from offline

•also provides incremental backup of the data • TCP/IP gateways and proxies

• “data warehousing” for data-set classification -> MySQL • Relation to event tag under evaluation• Also providing persistency to ROOT objects

Using Open Source external packages has been extremely useful.

A. Amorim

27

Future directions ...

CORBA

Persistent State ServicePSS

Berkeley DBPersistent State Service

PSS

MySQL

Open Communications Interface (OCI)

IIOP UDP/ based

Client Client Client

(ORBacus)Client

Client

One must have a plan even if it is a wrong one ...

FarmsWAN

Documents

The HERA-B detector The database problem The Architecture The Berkeley-DB DBMS