Managed by UT-Battelle for the Department of Energy Best Ever Archive Utility, Yet Creating a BEAST was easy. BEAUtY seems harder. [email protected] April

Managed by UT-Battellefor the Department of Energy

Best Ever Archive Utility, Yet

Creating a BEAST was easy.

BEAUtY seems harder.

[email protected]

April 2009

2 Managed by UT-Battellefor the Department of Energy

CSS-based OPICSS-based OPI

ArchiveArchive EngineArchiveArchive Engineconfig.xmlconfig.xml

IOCIOC

Binary Data Files

Data ServerData Server

“ASCII” Config.“ASCII” Config.Channel AccessChannel Access

XML-RPCXML-RPC

Previous Setup

History~2000:

Started byBob Dalesio

~2003:Index Tools,Data Server

~2007:CSS Client


Details Older Java “archive viewer” also available.

Archive Engines running on “sampling” computer, data server on “serving” machine.

Scripts to restart engines, copy data, update indices.

Headaches with data maintenance:– Index time grows with data

– Stuck when index files reach 2GB

– SNS Users faced with ~80 sub-archives

– No clue what needs fixing after network/power problems

– No idea who contributes how many samples

– No way to remove selected channels or time ranges

– Improving on this means implementing an RDB


CSS-based OPICSS-based OPI

ArchiveEngineArchiveEngine

config.xmlconfig.xml

IOCIOC

RDB(Oracle/MySQL)

RDB(Oracle/MySQL)

Channel AccessChannel Access

New Setup

SamplesSamplesConfig.Config.

EngineConfig-Import

Other tools for config & samples


CSS Data Browser Handles Both New URL

Just one‘RDB’sub-archive

Old and newdata can becombined inone plot


Configure Sample Engines

Configuration is in RDB– Directly use SQL

– EngineConfigImport for legacy config files

– View/Edit via web

Hierarchical (as before)– Sampling engine (name, where to run, …)

Groups– Channels

No more duplicate channels!


Web Interface to Engine Config in RDB

Tomcat/JSP/Servlets to view and edit


Web Config View: Channel Stats


Web Config View: Sample Stats

OK

??


Stats 34 sample engines, 79400 channels

Host that runs sampling engine:– CPU load 45%, zero disk I/O wait, very responsive

Oracle performance– 8000 samples/sec peak in write tests

– Operationally maybe ¼ of that

Better configurationwould likely havefewer samples/sec


Report Example, SNS Beam Instr. Group

May be SNS-specific, but shows usefulness of data in RDB


Summary

Still testing, many Oracle setup issues– Ran since Oct 08, then..

– Oracle cluster setup since ~March

– Updated partitioning in April

Plan for about a year now:Make operational in maybe 2 months

Thanks go to– Oracle: Jeff Patton

– JSP: Katia Danilova

– Beam Instr. Reports: Mariano Padilla, Jeff, Douglas Edwardson

– IT: Greg Lawson


Stuff


Hurdles Months: Get new Oracle server configured

– Interface cards for storage array, fiber switches

– Firewall holes for office access, backup, admin

Changes in 10g– SELECT MIN(stamp), MAX(stamp) -> NULL, NULL

– No “range” partitioning on Index-Organized-Tables

Configuration issues– ORA-01654: unable to extend …

– ORA-00257: archiver error

What used to be impossible is now “easy”, but still expensive– DELETE FROM SAMPLES WHERE …


Other Ideas

Jlab’s MyA– Operational, but

Wrapper code around MySQL to create ‘cluster’ Handles less meta info (units, limits, …) Viewer is one-of C++/TCL/Tk

Gabriele Carcassi mentioned RDDTool– Toolset for logging data with data-aging

– Command-line RPN tools, web viewer

– May be faster than general-purpose RDB but lacks advantage of gpp RDB


SNS Setup for EngineConfigImport Tool

– Imports existing XML config files into RDB

– On srv02 in …/imported_config *.xml files copied from archive1, includes

some editing to remove duplicate channels Review *.log files

– 'chmk/chmkhebt/PPS_HEBT:PLC_B:Chmk100AlarmContactA’already in 'pps/pps_hebt’

Maybe edit “your” *.xml file, refer to import.sh to re-import


Getting Sources, Building binaries

Get sources from DESY CVS repository– Maybe binaries “real soon” from ChannelArchiver web page

Build binaries– Find somebody who understands “Eclipse RCP”, how to use

org.csstudio.archive.rdb/EngineConfigImport.product org.csstudio.archive.engine/ArchiveEngine.product org.csstudio.sns.product/SNS_CSS.product (or DESY CSS.product)

– Settings Compile-time defaults usually in “preferences.ini” and

“plugin_customization.ini” files. Either change, or prepare file for

-pluginCustomization command-line option


RDB Setup

Currently supported: MySQL or Oracle– Main table difference: Oracle TIMESTAMP has

nanosecs, MySQL has separate nanosecs col.

– All tools auto-configure based on JDBC URL

Table Schema– ERD, MySQL DBD in org.csstudio.archive.rdb/dbd

– Setup for Oracle (table types, partitioning) not final


Run Archive Engine

Command-line tool similar to previous engine– More settings (RDB, Logging, JCA/CAJ, …)

Compiled-in defaults from “preferences.ini” and “plugin_customization.ini”

Can specify overrides via-pluginCustomization my_plugin_customization.ini

Mandatory command-line options for instance– engine_name to locate config in RDB– port for consistency, must match settings in RDB

Built-in web server for status, restart, stop: http://host:port/: Empty page (used to be ‘main’) …./main: Main page …./reset: Reset statistics …./restart: Stop, re-read configuration, start again …./stop: Stop

Documents

Managed by UT-Battelle for the Department of Energy Best Ever Archive Utility, Yet Creating a BEAST was easy. BEAUtY seems harder. [email protected] April