Software Tools for MPS Kajetan Fuchsberger MPP Workshop, 2013, Annecy Many thanks for input from M. Zerlauth, J. Wenninger, R. Schmidt, G. Kruk, V. Baggiolini,

Software Tools for MPSKajetan Fuchsberger

MPP Workshop, 2013, Annecy

Many thanks for input fromM. Zerlauth, J. Wenninger, R. Schmidt,

G. Kruk, V. Baggiolini, G. Papotti, D. Jacquet and the TE-MPE-MS Software Team

Software Tools for MPS

The solution for everything ;-)

3Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS

Content

Tests & Procedures

Early detection of failures

Aperture Meter


Content

Tests & Procedures


Aperture Meter


Tests and Procedures Currently: Sharepoint Sites

Disadvantages: Order not enforced

(scheduling done ‘on the fly’).

Nothing enforces the tests to be really done.

No real overview, what was done, what still has to be done?


The AccTesting Framework Designed from the experience

of Hardware Commissioning.

Successfully Used in HWC Campaigns 2012, 2013

Enforces Correct Order of Tests

Many additional features: Automatic Scheduling Constraints Statistics


AccTesting Concept

Migrate of MPS Commissioning Procedures: 1) Transform all the MPS comissioning steps to

AccTesting (simple sign-only tests). 2) Replace one by one with automated tests.

Test-Phases.

Tests


Barriers When the commissioning of a system

reaches a Barrier Point, it has to wait until all other systems (which have the same barrier) reach it also.

QPS IST1

BLM IST2 IST3 IST4 INJ1 INJ2

BIS IST5 IST6 IST7 INJ3

LBDS IST8 INJ4

RQTL… IST10 IST11 POW1 POW2

Ready for Powering

Ready for Injection

Knowledge of System-Dependencies required!


Interlocks based on Test Plan? Examples:

Prohibit, that circuit can be powered, before all the tests are done. Do not allow injection of beam before all necessary MPS tests are

done.

Possibly: Enforce Tests after changes: E.g. Which tests have to be executed, after a change of a QPS – card? Pre-defined test plans or (partially) auto-generated? Knowledge about System-Dependencies required!

What to interlock? E.g. Additional Circuit lock on PIC level which allows tests? Injection interlock?


Automation? Automate whatever can be automated!

(Avoid Errors, Reproducibility)

First steps: Concentrate on systems, where many interlocks of same type: BIC/PIC: Already dedicated tool, in the pipeline to be integrated with

AccTesting Collimators

Integrate BIS connection Test? Do we really have to look ‘by eye’ on lossmaps? Could there be a Hierarchy-check without a reference?

Vacuum BLMs:

E.g.: Test Of Latency: Close collimators: Done in pt3, pt7: Could be done for more, if automatic.

Others?


System Dependencies & Information

Key Concept: Use existing Sources!

Prototype in place

Possible Cooperation between TE-MPE-MS software team and BE-CO-DO (S. Jensen, D. Csikos).

System Information

Server

System Providers:• Lsa• Layout Db• …

Dependency Providers:• Layout Db• Runtime info (JMS)• Java code analysis• …

Information Providers:• Status• Issues (Jira)• Deployments • Faults? …

Clients (e.g. AccTesting, Diamon, ….)


Test Analysis Vision:

Simple description of test-expectations (assertions) on signals resulting from tests (Java).

Universal GUI components to visualize problems.

First Implementation: Cooperation between TE-MPE-MS software team and BE-CO-DA (R.

Gorbonosov, A. Jalal) Used in AccTesting could be used e.g. in future PM modules. (Maybe also in sequencer

tasks and other checks?)


Data Analysis TE-MPE-MS Vision: More general Data Analysis Framework

Key Concept: Perform analysis as close as possible to the data!

(Potential) Collaboration with BE-CO-DA to optimize resources, avoid duplication of efforts

First Implementation in place.

Analysis Framework (Cluster?)

Logging DB

Hadoop Cluster?

Post MortemClients (e.g. AccTesting, Timber, Custom Analysis….)

Others?


Content

Tests & Procedures


Aperture Meter


Early detection of FailuresYASP

LSA Server

trim

PcInterlockGUI

PCPC

PC

trim subs

crib

e

SIS

monitor

Dump!

?

?Not Allowed!

PcInterlock Server


LSA: The natural place to check

LSA Server

Applications: YASP Lumi-Scan

application Others…

Checks: Power Converter

Currents Collimator Settings BPM readings (actual +

predictions) online modeling?

Others?


LSA checks - proposal Vision:

Have an API like ‚isAllowed(TrimRequest request)‘

First implementation: Use ‚TrimPostProcessor‘ mechanism of LSA. Executed after trim saved in DB, on error LSA will do

a rollback. Incorporation = Trim Same mechanism to avoid

propagating trims, which would otherwise e.g. fail during ramp.

Override for MDs.


Further securing LSA LSA Method protection

Currently: Only basic protection (Sending of timing events, Trim, Drive)

Every method in LSA should be reviewed, if it can do harm or not and protected appropriately

Protect reference Cycles for PcInterlock/SIS: First approach: TrimPostProcessor which evaluates

RBAC roles


Content

Tests & Procedures


Aperture Meter


Aperture Meter


Aperture Meter - Status Prototype Operational

Proved to be very useful during MDs Follows the operational cycle (optics, BP, time in BP) Listens to (some) LSA trims

What is missing? Improvements of User Interface Performance Improvements Tighter integration with Operational changes

(Collimator offsets after alignment, BPM usage from Yasp) Could be used as another source for LSA trim-verification

(e.g. large collimator movements, Collimator Hierarchy)


Summary MPS Commissioning:

Tracking in Acctesting Framework. Automate! (Step by Step) Candidates?

Preview: Two new frameworks: Tracking of system dependencies (Test-) Data Analysis

Early detection of potential problems could be done in by additional trim checks in LSA.

Aperture Meter: Useful prototype, but needs improvements.


Tools, Tools, Tools, Tools, Tools, …?

…Tools do not solve everything! Culture & communication

(High Quality) Software is not for free!

Challenges: Large part of Software manpower goes already into maintenance. A lot of ‘grown’ projects, partly written by un-experienced programmers

(E.g. Students).

How to improve? Reliability = Maintainability = Quality

(Self explaining code + Automated Testing, Testing, Tes…. ) Supervision of any SW projects from two sides: System Expert + SW expert

(‘Mentoring’) Who has the overview and can judge what is already out there and into

which framework a certain requirement would fit best? Communication!

Who is authorized to define priorities? Resources?


Thank you for your Attention!

Questions?

Small Steps-

Big Visions!


Fail Fast! Actual BP:

Would a certain trim exceed some limits?

Check before functional BP: Collimator-limits prediction Corrector Interlocks vs. Lsa settings Check at incorporation time


Single Source for „Operational State“

For certain applications BeamMode is not enough!

Combination of: Beam Mode, Beam Process, Time within BP, (Optics)

Currently (at least) 2 implementations: Combination of Beam Modes and PC state

changes(Aperture Meter)

Dedicated timing events (PcInterlock) Both not optimal. Reference Implementation

required.


Fail Fast!

„A fail-fast system is designed to immediately report […] any failure or condition that is likely to lead to failure. ”

[Wikipedia]


Data Analysis Mandate in BE-CO-DA (C. Roderick) to provide Data Analysis Solutions

Currently in Use Case Analysis Phase

Aim to be as easy as possible for users to specify: What to produce for an analysis case (e.g. signal names, units, description) What are the inputs (signals and sources) Which algorithms should be applied When analysis should be performed (event driven e.g. fill / beam mode, time

interval driven e.g. every 15 minutes)

Without user needing to: Configure a specific development environment Care about compilation and deployment Look elsewhere for signal names etc. i.e. provide directory of signals and sources

Key concept: Perform data analysis as close as possible to the data!


MPS Commissioning

What is missing? Dependencies between Systems (e.g. ELQA for a circuit

is ready, when the test are done for all magnets within the circuit)

Global Phases?

QPS IST1

BLM IST2 IST3 IST4 INJ1 INJ2

BIS IST5 IST6 IST7 INJ3

LBDS IST8 INJ4

RQTL… IST10 IST11 POW1 POW2

IST Powering Injection


System Dependencies Vision: Central Instance (Server?) to answer questions like:

Which magnets are connected to a certain power converter? (AccTesting – ELQA) What is the common point for a set of crates? (Diamon) What systems have to be potentially re-tested if e.g. a QPS card is changed?

(AccTesting) What systems are potentially affected by a OS upgrade on a certain host? (SUWG)


Interlocks during functional BPs

In place for orbit correctors. Ready to be extended to other circuits.

Potentially useful for other systems, e.g. Collimator movements?

Documents

Software Tools for MPS Kajetan Fuchsberger MPP Workshop, 2013, Annecy Many thanks for input from M. Zerlauth, J. Wenninger, R. Schmidt, G. Kruk, V. Baggiolini,