Acctesting Framework - Motivation, Overview and First Experience

1

Acctesting Framework - Motivation, Overview and First Experience

Kajetan FuchsbergerTE-MPE-TM, 2012-05-10

Thanks to:M. Galetzka, V.Baggiolini, R.Gorbonosov, M. Pojer,

M. Solfaroli Camillocci, M. Zerlauth

K. Fuchsberger 2TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Content

1. Introduction/Motivation

2. System Overview

3. First Experience

K. Fuchsberger 3TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Content

1. Introduction/Motivation

2. System Overview

3. First Experience

K. Fuchsberger 4TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

LHC Hardware Comissioning

• About 7000 Tests on Magnet Circuits• Must be executed every year after Christmas

Stop (Potentially more after LS1)• Workflow:

1. ‘Execution’: Test Sequence is Executed on the Hardware Commissioning Sequencer.

2. ‘Analysis’: Test-Data (Measured Signals) is analysed (manually or partly automatic).

3. ‘Signing’: Experts have to ‘sign’ the test result (If manually analysed).

K. Fuchsberger 5TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Motivation I

Initial Project Description: “Create replacement for the p2n Web-Page

aka Alvaro’s pages”

• PHP, grown over time Hard to maintain

K. Fuchsberger 6TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Motivation II

• Some tests should be allowed to be executed even if previous analysis was not completed.

• Avoid starting of test if system is not ready.(e.g. locked)

• Avoid starting of tests on Circuits on same QPS controller.

• Force starting of tests on QF, QD at the same time

• ….. and some more …

New Requirements:

K. Fuchsberger 7TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Motivation III

The HWC Legacy System:

• Many interdependent Systems:• Sequencer• DB (Central Point)• web page• Daemon• Fesa Class• LabView• …

Decision: Design a new system to orchestrate the whole process (and can replace several parts).

K. Fuchsberger 8TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Content

1. Introduction/Motivation

2. System Overview

3. First Experience

K. Fuchsberger 9TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Overview

e.g.: LabView, Powering Server …

e.g.: HWC Sequencer,

LHC Sequencer orCustom Components

e.g.: Magnet Circuits, BIC/PIC …

K. Fuchsberger 10TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Acctesting Server

• Orchestrates the whole process:• Test Execution• Test Analysis

• Exclusively reads/persists data in the database.

• Notifies all the GUIs about changes.• Robust Design:

• Continuously persists relevant data to be able to recover in case of a crash.

• Gracefully handles unexpected behaviour of Execution- and Analysis Components.

K. Fuchsberger 11TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Workflow I

1. User “expresses his wish” to execute one (or many) tests.

K. Fuchsberger 12TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Workflow II

2. The Tests go to the Execution Basket (Server!)

3. The Scheduler (on the Server) will decide when to start which test(s).

K. Fuchsberger 13TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Why so complicated?

• Central Scheduling can respect all the conditions, even if requests come from different GUIs.

• When conditions are fulfilled later, the tests are started automatically (No delays).

• No Need for reservation of circuits anymore. More dynamic

K. Fuchsberger 14TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Test Phases

Tests within a test phase can be executed in arbitrary order.

TestPhase contains one or more Tests

K. Fuchsberger 15TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Preconditions

• Before: Only email exchange, that e.g. “Cryo is ready for a certain group of circuits”.• Hard to track (History)• No real constraint on test execution ( Many

wrongly started tests in previous years)• Now: ‘SignOnlyTest’ for each precondition.

Automated Tests

K. Fuchsberger 16TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Designed for Extension

• Server:• TestStepHandler (handle certain types of

SystemTests)• Constraints (Restrict Test Execution)• LockProvider (PIC, Db, …)• SystemInformationProvider (e.g. Issues)

• GUI:• TestResultsViewer (E.g. Powering Server)

Extension Points:

K. Fuchsberger 17TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Test Step Handlers

• Responsible for executing a specific TestStep (execution, analysis) for a certain type of tests.

• Is itself responsible for communicating with other systems, if required.

• Examples:• HwcTestExecutionHandler: Communicates with HWC

Sequencer to execute the tests• DaemoneAnalysisHandler: Communicates with

LabView system to retrieve analysis results• … Future: PicBicTestExecutionHandler?

K. Fuchsberger 18TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Constraints

• Simple Extension Point, which has to decide if one Test is allowed to be run together with another one. (Simple Yes/No decision)

• Used to formulate requirements like:“Only start one test on one of the four circuits on the same QPS controller.”

• Checked by the Scheduler, to decide if a certain test-configuration is allowed or not. See Michael’s presentation.

K. Fuchsberger 19TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Soft Migration

• Working System had to be in place for start up 2012, but:• Old System should still work.• The switch between the two Systems should be easy.

• Achieved by:• Using the old Db-Schema for new System (some

restrictions!) + some (careful) extensions• WebPage stayed operational.

K. Fuchsberger 20TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Content

1. Introduction/Motivation

2. System Overview

3. First Experience

K. Fuchsberger 21TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Some Statistics

4 big Campaigns:

Min 5 Tests launched.

K. Fuchsberger 22TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Test Statistics

Test failure rate did not decrease much !?

Average number of Tests per shift

Percentage of failed tests that timed out.

(Preconditions!)Efficiency increased.

K. Fuchsberger 23TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Time between Tests?

• Expectation: Average time between test executions on one circuit should decrease!? Not evident !?

In the shadow of Analysis Time!

Tim

e [h

]

(Data: X-Mas 11/12)

K. Fuchsberger 24TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Analysis Time

PIC2 and PNO.a1 automated (if successful)

K. Fuchsberger 25TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

HWC Campaign 2012

• System was operational from the beginning. No big problems. Never stopped progress.

• GUI was very well accepted (Lot of positive feedback).• Preconditions were useful, at least for tracking. Users

Point of view?• Constraints turned out to be very useful although

sometimes puzzling. Prevented many mistakes. Many additional constraints were added during the campaign.

• After some doubt, the automated scheduling was found to be very convenient.

K. Fuchsberger 26TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

To Improve

• Missing: Editing of Test Plans! (A lot of Db hacks were necessary )

• Constraints should be more dynamic (Currently Java Classes)… dynamic loading for plugins?

• Old page still used for statistics and quick lookups. Missing GUI features.

K. Fuchsberger 27TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Next Steps

• Database Migration (Old web page will die)• Improvements of GUI statistic features

(Summer Student)• Add test-plan editor to GUI.• Improve Scheduling Algorithm ( Michael)• Integrate other systems:

• Beam Commissioning as SignOnly tests. Migrate smoothly to partly automatized tests?

• BIC/PIC tests?• Others?

• More Automated Analysis (Powering Server)

K. Fuchsberger 28TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Conclusion

• System worked very well in 2011.• Some improvements (test plan editing, more

dynamic constraints, new GUI features)• A lot to improve in analysis part!• Next Steps important:

• Db migration.• Improvement of scheduling.• Improvement of automated analysis.• Integrate more systems.

K. Fuchsberger 29TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience

Thank you for your attention!