1 Archiving Workshop (Soleil, May 2010) Archiving System Status

Archiving Workshop (Soleil, May 2010) 1

Archiving System Status


Plan

Archiving Systems Overview HDB / TDB / SNAP / ADB

Soleil Deployment Status Oracle DB infrastructure description Latest modifications Production Missing features


Archiving Systems Overview


HDB/TDB Archiving: General Description

Operators Requirements Storage of a large number of data produced by the Control System Data exploitation at anytime

• Watch the vacuum temperature and the pressure evolution• Look for an explanation of a beam loss

The tools Historical DataBase (HDB)

• Data are kept for lifetime• The minimal archiving period is 10 seconds• On-line extraction available during x years (hardware capacity limitation)

Temporary DataBase (TDB)• Data are kept x days (hardware capacity limitation)• The minimal archiving period is 100 milliseconds

The two are managed via the graphical tool named Mambo


HDB/TDB Archiving: Mambo

MAMBO : Configuration and Extraction application

Web access also available via JBOSS


HDB/TDB Archiving: Devices

Devices Hdb/TdbArchiver : Collect of the attributes values and storage in

the DB

ArchivingManager : Upper HDB/TDBArchiver, contains Start/Stop archiving commands

HDB/TDBExtractor : various extraction commands as getAttDataBetweenDate, getArchivingMode …

HDB/TDBArchivingWatcher : Supervision Device

• Diagnosis tool : detecting abnormal archiving interruption

• Recovery : reactivate archiving on the failed attributes


SNAP Archiving: General Description

Operators Requirements Capability to take a picture of a set of attributes at t time

• Motors positions before a planned electric halt Compare quickly and easily the attributes values

• Before and after an experience to analyse the beamline parameters evolution

Send instructions easily to several equipments • Set the beamline in a configuration reference

The tool Snap DataBase (SNAP)

• Allows to take a snapshot of a set of attributes• Allows to apply values on a set of attributes• Allow to compare attributes values to each other

It is managed via the graphical tool named Bensikin


SNAP Archiving: Bensikin

BENSIKIN : Configuration and Exploitation


SNAP Archiving: Devices

Devices SnapArchiver : takes the snapshot and stores it in the DB

(Hibernate java technology) SnapManager : triggers the launch snapshot to an available

archiver SnapExtractor : various extraction commands as getSnap


ADB Archiving: General Description

Operators Requirements Supervise attribute quality and device state changes

The tool Alarms DataBase (ADB)

• Monitors the attributes alarm state

It is managed via the graphical tool named Macarena


ADB Archiving: MacarenaMacarena : Current and Deleted Alarms


ADB Archiving: Devices

Devices AdbArchiver : monitors the attributes quality and stores the

values in the DB

AdbManager :

• dispatches alarm supervision between the existing AdbArchiver devices

• Used by the GUI for each database transaction.

AlarmDiary : contains the available alarms list, access by the GUI only


Common Points

SGAD: GUI tool generates a MySql DB creation script for Unix or Windows for HDB, TDB or/and SNAP DB

Common point : Archiving systems are provided in Java 1.6


Soleil Infrastructure


Machine Deployment status

All the archiving systems use an Oracle Database

SNAP Oracle engine updated Following some Bensikin updates and a little ICA pressure the

operators plan to work more with it during the next weeks• Currently important data are stored in Matlab files• Still now, the Vacuum group makes some manual comparison at

every run startup

ADB Before the data were stored into files System recently migrated to be built as HDB/TDB system GUI consolidation is ongoing and strongly awaiting by the operators to

help them in the control system monitoring


Machine Deployment status

HDB : 8900 attributes 8750 scalars + 150 spectrums

• 1800 at 10 s• 4000 at [30 – 60] s• 1800 at [200 – 900] s• 1000 at [1000 – 3600] s• The rest is upper 3600 s

25 DServers * 5 devices, so 125 devices hold by One Linux Server• Each device supports about 70/75 attributes (initial estimation was 50

attributes / device) 5 years of online data

TDB : 6500 attributes 6470 scalars + 30 spectrums

• 10 at 100 ms• 1900 at 500 ms• 3100 at [1 – 5] s• 650 at [10 – 60]• The rest is upper 60 s

60 DServers * 5 devices, so 300 devices hold by Three Linux Servers• Each device supports about 20/25 attributes (estimation is about 30

attributes / device) Data are available 30 days


Beamline Deployment status

All the archiving systems use a MySql Database

SNAP Really often used to store motor positions, to set beamline in reference

position ...

TDB: at most 250 attributes Mainly scalars 1 DServers * 5 devices Data are available 20 days (easily configurable per beamline) Minimal archiving period equals to 1 s due to a MySql limitation

HDB: Not available Oracle installation is in progress All the hardware infrastructure is available Will be deployed progressively beamline per beamline


Oracle DB Solution


Oracle Solution

Refer to the Jérémy’s presentation


Latest Modifications


Latest Modifications: Mambo

Mambo

Better .lock file management

• At Mambo startup this file is stored in the directory linked to the connexion profile

• This file is still available when the application is brutally interrupted

The profile is no more usable

• Now .lock can be easily suppressed after confirmation



DB connection parameters location

Either in command line with • … -DHDB_HOST=srv1 -DHDB_NAME=hdb

-DTDB_HOST=srv1 -DTDB_NAME=tdb fr.soleil.mambo.Mambo …

• Useful to start 2 Mambo in front of 2 differents DB

Or in Hdb/Tdb Archiver class properties

• DbHost, DbName

• Properties also used by the archivers devices



Mambo

Display Multiple VC Some parts has been transformed in java bean :

• Table Mode• Connection Window• VC (not completly done)

Start archiving error description in the log part


Archivage HDB/TDB : Mambo

Archiving modes: something more in the Absolute and Relative modes Definition

• Attribute value is archived when the variation between two successive values (in absolute value or in percentage) is upper configured bounds.

Default behavior • the variation is computed with the attribute value previously read on the device and the current

value Slow drift behavior

• the variation is computed with the attribute value previously stored in the DB and the current value


Latest Modifications: Mambo Mambo

New Graphical tool usage named Comete various impacts :

• Spectrum viewer • X Axis configuration

management

Indice

Time

Time Stack



Mambo

Set All Colors in the New/Modify VC


Latest Modifications: HDB/TDB Devices

HdbArchiver Some attributes can be archived under 10 s by configuration (not

fully tested)

Hdb/TdbArchivingWatcher Device state modification Number of KO attributes ≥ MinNumberOfKoForAlarm (new

property) then State = ALARM otherwise State = ON

Database Api refactoring Done but Not completed


Latest modifications: Bensikin

Context description Number of attributes

SnapShot description Contexte number Number of attributes

SetEquipment With Command Send a command with the

Read or Write value as argin

At startup: last open snapshot is loaded (quickly opened)



SnapShots comparaison String and boolean attributs types can be also compared

Table bean and Connection window bean usage



Application options (proper to the profile) : Date Filter criteria Snapshot expanded

Prefilled Time field



Columns order definition in the comparison window

Configuration of the Data included in the Snapshot export

Application options (proper to the profile)


Latest modifications: SNAP Devices

SnapArchiver / Manager LaunchSnapshot method has been moved to its right place: it

concerns the SnapManager and call the TriggerLaunchSnapshot command of the SnapArchiver


Production


Maven production

Now .classpath project is empty pom.xml usage

Pom.xml contains the main jars files see Vincent Hardion presentation on

Thursday

Archiving packaging ongoing Either one zip file with the complete archiving systems distribution Or one zip file per archiving system Zip file will be stored in the Tango Pink site

<dependency> <groupId>fr.soleil.lib</groupId> <artifactId>hdbtdbArchivingApi</artifactId></dependency>

<dependency> <groupId>fr.soleil.lib</groupId><artifactId>commonArchivingApi</artifactId><version>1.4.4</version></dependency>

<dependency><groupId>fr.esrf.tango</groupId><artifactId>TangORB</artifactId></dependency>


To be improved


Missing features

No archiving system manage the Tango::Image data_format Archiving system monitoring must be improved for computer

team and operators Long term TDB: some data mustn’t be deleted Logging information improvement with log 4 j usage Database api code refactoring not fully finalized MySql doesn’t support the milliseconds Tdb/Mysql DB upgrade (versus Oracle) to ensure a heavy load

(eq. SqlLoader, jobs, partitions, backup mechanism …)


Summary


To sum-up

Archiving Systems are functionally fully operational many interesting features are missing

Deployment is a big part of the archiving system activity we are lacking tools to ease this task

Oracle DB infrastructure management is a huge task How to share the know-how for MySql management ?

Archiving Systems are always in evolution Sharing modifications, with the community, is crucial to avoid

forking

Last but not least : the Archiving system philosophy Give Users the ability to be autonomous

Documents

1 Archiving Workshop (Soleil, May 2010) Archiving System Status