Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

1

CO deliverables for HWC –lessons learnt and the proposed way ahead

Markus Zerlauth, AB-CO-MI

for the CO group

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

2

Outline

Post Mortem analysis and test automation

Circuit Synoptic

Logging

CCC environment for HWC

E-logbook

Conclusion

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

3

Post Mortem Analysis

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

4

3. PNO.2 auto analysis

D2Q5

Q4

1 s

Current PMA packages

Courtesy of PM team

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

5

Current Architecture of PMA system v5Individual system analysis

Logical system analysis

Magnetsystem

SDDS

logging

additionalresults

Raw data

Result data 1

Analysis sequencer

Result data 2

General system analysis

Cryo circuits

PC

PIC

QPS

Cryo

Vac

BLM

Circuit600 A

PIC1, 2

Circuit

Circuit

Circuit

Electrical circuits

Equip

Equip

Equip

DFB, SCLinkStand. cell

GSA

Analysis sequencer

ready 29-06-2007

Circuit60, 80, 120A

Courtesy of A.Rijllart

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

6

Post Mortem Analysis – further automation

No fully automatic analysis feasible yet, but further automated hand-shake between sequencer and PMA possible

Complete Logical system analysis + with further dedicated analysis packages of type PNO.2

Priority must be on circuit types <6kA (largest quantity) PO, PMA and sequencer teams are investigating possibilities to further automate PO

related tests for 60A and 600A circuits (PCC, crow-bar, etc..) Issues:

– In-homogeneous test plan & interest of involved parties requires global coordination of priorities for

• Final sequences (need to agree on last changes & freeze as much as possible)

• Related automated analysis

– Development priorities need to be urgently agreed upon in between all involved parties in the next weeks to provide operational & tested packages already for sector 45

– Automated test plan versus manual (MTF reporting, parameter update,..) Series of SACEC meetings to address these issues

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

7

Sequences versus ‘automated’ PMA

60-120A

x 1036

PIC1.x

PCC

PIC2.x

PNO.1

PNO.2

PNO.3

PNO.4

PAC

600A

X 410

PIC1.x

PCC

PIC2.x

PCS.1-4

PLI1.x

PLI2.x

PLI3.x

PNO.1

PNO.6

PNO.10

IPD

X 16

PIC1.x

PCC

PIC2.x

PLI1.x

PLI2.x

PLI3.x

PNO.5

PNO.6

PNO.10

PAC

IPQ

X 78

PIC1.x

PCC

PIC2.x

PLI1.x

PLI2.x

PLI3.x

PNO.5

PNO.6

PNO.7

PNO.10

PSQ

PAC

13kA

X 32

PIC1.x

PCC

PIC2.x

PCS.1-4

PLI1.x

PLI2.x

PLI3.x

PNO.1

PNO.6

PNO.10

PAC

Sequences ready

PM analysis automated

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

8

Post Mortem Analysis – ongoing improvements

PM Data Browser

– Automated loading of correlated PM data from QPS/PIC/EE, etc. based on the FGC event

– Event search based on circuit name and/or time window

– possibility to manage the list of the signals to display in the HWC view (via a text editor)

– user can save the jpeg of the CHART or the excel sheet from the TABLE in a local folder

– Link with cryogenic signals

• link between Electrical circuits & cryogenic equipment foreseen to be established in the Layout DB

• used for PM Data browser, Cryo SCADA, CIET and Circuit Synoptics

• scripts generating a set of signals / circuit

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

9

Post Mortem System – scalability

Dedicated scalability tests + HWC experience have not shown any bottlenecks in the available PM infrastructure (data collection on PM server and data conversion)

– in case it does in the future, further servers will be deployed

Current limitations mostly lie with the gathering of data on the client side

PM scalability @ example of AUG test 09th July at 9:50:28

– 49 ext FGC events, 77 self FGC events, 17 DQAMS events

– The PM data was received on the server in two batches, each ~1min long: at 9:51 and 9:56. No errors on the server side, no client disconnections detected during the test.

– Conversion time to pmd format ~50ms for 600A converter file, ~500ms for 60A converter file, ~40ms for DQAMS file

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

10

Circuit Synoptic

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

11

Circuit Synoptic

Circuit Synoptic an important tool during HWC, gathering necessary information related to circuit powering on a single screen

2 major improvements proposed for next sectors

– Inclusion of 60A converters (60A SW power permit and FGC status)

– Link of circuit synoptics with related cryogenic signals, e.g. CL temperatures, liquid levels in related DFBs, CRYO_START, CRYO_MAINTAIN, etc...

Requires considerable work on configuration DB for mapping of signals (together with ACR) + data publishing from CRYO system (ACR & CO), but could be exploited by circuit synoptic, CIET, CRYO SCADA, etc..

Would allow for ‚real-time‘ trends of e.g. circuit current vs. current lead temperature within PVSS, etc...

Courtesy of F.Bernard

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

12

Circuit Synoptic v2

Courtesy of F.Bernard

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

13

Logging

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

14

LHC Logging Service

The LHC Logging Services - Architecture

<Data Source>

<Data Source>

Logging DB

Measurement DB

Technical Services DB<Data Source>

WRITEAPI

XML

HTTP

HTTP

WRITEAPI

READAPI

JDBC

JDBC

JDBC

JDBC

PL/SQL

PL/SQL

<Data User>Generic Usage

Measurement Database Contain raw time series data, at up to 2Hz rate

Store data for 7 days

Derive new values based on measured values

Sends filtered data of interest to Logging DB

Generate statistics on accelerator performance

10g AS

Oracle RAC Database Host 2 x SUN Fire V240

Dual 1 GHz CPU

4 GB RAM

Dual internal disks

Dual power supply

SUN StorEdge 3510FC disk array

2-Gb fiber channel bus Dual RAID controllers 1 GB cache 12 x 146 GB 10k rpm FC disks 24 x 300 GB 10k rpm FC disks (2007) SUN Solaris 9 Veritas Volume Manager 3.5 Oracle9i Enterprise Edition Release 9.2.0.7.0

60,000,000 records / day

129,000,000 records / day

147,000 records / day

6,800,000 records / day

Courtesy of C.Roderick

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

15

Logging - Infrastructure + Interface

HW infrastructure– current Logging service has to cope with rapidly increasing demands– initially intended to deal purely with LHC machine/beam related data; yet today, data

comes from PS Complex, SPS, SPS-EA, CNGS, ATLAS, LHCb, CMS, CIET…– Worry about interdependency with LSA (currently on the same database server) +

scalability for 8 parallel sectors – DM has launched a study and will present a proposal for the backend upgrades at a

TC on August 16.  HW upgrades are tough to be in place for October 2007...

Data extraction interface– technology behind the current data extraction tool allows for an easy web-access from

inside and outside CERN, but is for this reason also specific and by now outdated – Already planned to be replaced new implementation, thus addressing the open issues

such as compatibility, performance and flexibility– implementation has partially started but the increasing workload for projects such as

LSA, hardware commissioning,.. has made it impossible to further progress up to now (estimated to 2man-months, due to specific knowledge no easy 'outsourcing‘ possible)

– Currently on hold….

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

16

Logging & Data providers – example of CRYO data

The present Logging system is used by most players of HWC, which store their system

data at rates and accuracies which are primarily defined by system experts and inherent

limitations of their systems (while CO will discuss with every client to minimise data

volumes for long-term storage if possible)

For the global event analysis, additional data of in this case the cryogenic system has

been retrieved, but first attempts showed limited time resolution and accuracy

First meeting with experts revealed several possibilities of improvement in the controls

infrastructure that deserve follow-up, but also inherent limitations of the acquisition

electronics -> see also summery paper of Mike (to be released)

Requirements for data storage of system designers (doing process control & system

setup) and HWC & MPP experts (doing event analysis) can be very different and should

be discussed by the latter two in case the provided accuracy and data rates are not

sufficient for HWC activities

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

17

The CIET system

LT data

0.1 Hz

FEC

5 HzWFIP Signal

Processing(median calculation)Delay 30 s

SCADA

Threshold(0.5%)

db

LT data

0.1 Hz

FEC

5 HzWFIP

PLC

The CRYO SCADA system

1HzTime stamp

PLC SCADA

Threshold(0.08%)

Threshold(0.1%)

SCADAArchive db

Dead time(10s)

Signal Processing(median calculation)

Logging – CRYO data

Courtesy of M.Koratzinos

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

18

CCC Environment + elogbook

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

19

Comments on working environment…

Generally, CCC environment very much appreciated by teams

Passwords, account-timeouts, CMF updates and limited access to GPN perceived as disturbing

during HWC, but need to follow new CERN security policy

Unique way to start programs from Linux and Windows consoles not easily possible as many clients

only run on Windows (PVSS) or Linux, but to be followed up

Printing from the different consoles perceived difficult

Installation of LabView 8.2 on Windows consoles to run PM application directly

Elogbook

– Vital tool for tracking and re-construction of HWC activities (EICs can confirm, especially when multiple tests are ongoing and time is limited)

– Operators have the good habit to accurately fill these logbooks

– Especially for parallel commissioning, changing shifts and if tests are carried out manually (no LSA tracking) an asset

Do we need further μCCC during HWC?

Har

dwar

e C

omm

issi

onin

g R

evie

w, M

arku

s Ze

rlaut

h, 1

1th J

uly

2007

20

Conclusions and Outlook

Many CO services involved during sector 78 commissioning, which all gathered valuable input for further evolvement and improvements that will be digested and implemented during the next weeks

(timing, infrastructure, PIC, WIC, CRYO SCADA, CIET, QPS SCADA, PMA, applications, WorldFIP, databases, Logging, etc...)

Further input needed from HWC and MPP to have asap a common view on priorities for further improvements, test automation & global analysis -> SACEC

Issues as databases, timing, etc not treated here but will be adressed with concerned teams -> SACEC

Feedback from equipment groups & this HWC review will be addressed and followed up within the CO group during a dedicated technical commitee next week