byZinour Charifoulline, Scott Rowan

Automation of Hardware Commissioning - Offline Analysis of 60A circuits

• Test Procedures/Offline Analysis• Automated Analysis Walkthrough• Automated Analysis Results

• PCC.1• PNO.d1

• Inconsistencies• EPC Feedback• Summary/What Next?

Outline

S. Rowan - MPE-TM - 27/02/2013

Test Procedures/Offline Analysis

S. Rowan - MPE-TM - 27/02/2013

• Tests Procedures exist for all circuit types by current level

• Defines all powering cycles to fully commission circuits

• Defines offline analysis criteria to be checked and signed by experts

• Pass Criteria is seemingly simple enough to automate

Test Procedures

S. Rowan - MPE-TM - 27/02/2013

• Using the eDSL Java Interface pass criteria can be written in an easy to read/understand sentence form

Offline Analysis ±PCC.1

//Verify converter current to be +3.7A/+-0.6A 60seconds after the 1st crowbar activation (2nd PM)assertThat(I_MEAS).isEqualTo(+3.7, AMPERE).withinAbs(0.6, AMPERE)

.at(60, SECOND).after(Occurrence.SECOND, PM_EVENT_TRIGGER); //Verify converter current & voltage decay is an exponential function with an absolute tolerance of 0.1A/0.05V starting 20ms after the 1st crowbar activation (2nd PM)assertThat(I_MEAS).isEqualTo(EXP).withinAbs(0.1, AMPERE)

.starting(20, MILLI(SECOND)).after(Occurrence.SECOND, PM_EVENT_TRIGGER)

.ending(150, SECOND).after(Occurrence.SECOND, PM_EVENT_TRIGGER);assertThat(V_MEAS).isEqualTo(EXP).withinAbs(0.1, VOLT)…

…

S. Rowan - MPE-TM - 27/02/2013

• Using the eDSL Java Interface pass criteria can be written in an easy to read/understand sentence form

Offline Analysis ±PCC.1

//Verify 1st PM-trigger exist and offset is zero assertThat(I_MEAS).isEqualTo(0.0, AMPERE).withinAbs(0.1, AMPERE)

.at(0, MILLI(SECOND)).after(Occurrence.FIRST, PM_EVENT_TRIGGER); //Verify start point at I_PCC=10AassertThat(I_MEAS).isEqualTo(10.0, AMPERE).withinRel(1.0, PERCENT)

.at(1, SECOND).before(Occurrence.SECOND, PM_EVENT_TRIGGER);

Several additional checks are added to improve robustness

S. Rowan - MPE-TM - 27/02/2013

Question!

• Can this be used by a non-java expert??

Automated Analysis Walkthrough

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Walkthrough

Offline Analysis ±PCC.1

Verify 1st PM-trigger exists and offset is zero

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Walkthrough

Offline Analysis ±PCC.1

Verify plateau current I_PCC=10A

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Walkthrough

Offline Analysis ±PCC.1

Verify converter current to be +3.7A/+-0.6A 60sec after the 1st crowbar activation

S. Rowan - TE-MPE-PE - 27/12/2013

• PCC.1 Automated Analysis Walkthrough

Offline Analysis ±PCC.1

Verify current decay is an exponential function with an absolute tolerance of 0.1A after the 1st crowbar activation

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Walkthrough

Offline Analysis ±PCC.1

Verify voltage decay is an exponential function with an absolute tolerance of 0.05V after the 1st crowbar activation

Then simply repeat for tests for negative cycle!!!

S. Rowan - MPE-TM - 27/02/2013

Automated Analysis Results

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Results 2011

Automated Analysis Results ±PCC.1

2011 HWC campaign• 233 PCC.1 tests on 225 60A-circuits• 17 consistency fails • 10 failed due to I_ERR_PCC which is not in

offline analysis• 3 cases due “just” failing decay criteria• 92.7% of consistency.

S. Rowan - MPE-TM - 27/02/2013

• PCC.1 Automated Analysis Results 2012-2013

Automated Analysis Results ±PCC.1

2012-2013 HWC campaign• 398 PCC.1 tests on 350 60A-circuits• 10 consistency fails • 1 was “successful” but “tuning to be done”?• 8 cases due to “just” failing decay criteria• 1 anomaly accepted manually???• 97.5% of consistency!

Should have failed!Circuit: RCBV19.L5B1

S. Rowan - MPE-TM - 27/02/2013

2011 HWC campaign• 1022 PNO.d1 tests on 750 60A-circuits• 69 consistency fails • 58 cases due to “just” failing decay criteria• 8 of which due to crowbar voltage • 2 current plateau checks failed• 93.2% of consistency!

• PNO.d1 Automated Analysis Results 2011

Automated Analysis Results ±PNO.d1

S. Rowan - MPE-TM - 27/02/2013

• PNO.d1 Automated Analysis Results 2012-2013

Automated Analysis Results ±PNO.d1

2012-2013 HWC campaign• 997 PNO.d1 tests on 750 60A-circuits• 97 consistency fails • 64 cases due to “just” failing decay criteria• 20 due to failing crowbar voltage check• 1 significantly bad circuit (same as PCC.1)• 12 based on data errors (passed manually)• 90.3% of consistency!

S. Rowan - MPE-TM - 27/02/2013

Should have failed!Circuit: RCBV19.L5B1

Inconsistencies

S. Rowan - MPE-TM - 27/02/2013

• ~60% of inconsistencies (4% of all tests) due to high circuit resistance, resulting in decay being outwith the defined criteria

Inconsistencies - Current decay

Thanks to LabView+RADE!

3.7±0.6A@60s

All should pass the criteria

27 circuits should failwith current criteriaR_TOTAL>0.0675Ω

Possible solutions:1. Refine thresholds: 3.7±0.6A => 3.4±0.5A2. Use Layout DB to compare decay constant

S. Rowan - MPE-TM - 27/02/2013

3.4±0.5A@60s

Layout Database values

• ~30% of inconsistencies (2% of all tests) due to voltage limits across crowbar• Most failures due to strict criteria and/or noise in signal.

Inconsistencies – Crowbar Voltage limits

Possible solutions:1. Relax thresholds: 1.1±0.1A => 1.1±0.2A2. Check average of plateau lies within limits

S. Rowan - MPE-TM - 27/02/2013

EPC Feedback

S. Rowan - MPE-TM – 27/02/2013

• Consistency checks «correspondent vraiment» to what is expected

• Consistency check over previous campaigns resulted in discovering a faulty module (RCBV19.L5B1) that EPC were unaware of!

• Regarding pass criteria - EPC accept all proposed solutions• With existing pass criteria the automated analysis has >90% consistency• With relaxed pass criteria 60 A circuit analysis has >99% consistency

• Regarding possibly more robust assertions• Comparison to a modelled reference curve? – difficult to implement• Plateau averages? – possibility, though accuracy/necessity questionable• Time constant check on decay? – to be discussed, will require Layout DB

• Ability to use parameters from Layout Database? – “soon™”

EPC Feedback

S. Rowan - MPE-TM - 27/02/2013

• A simple qualification check was requested for 3 known faults. • Note: All had already been flagged by consistency check.

EPC Feedback

S. Rowan - MPE-TM - 27/02/2013

RCBH13.L6B1:PNO.d1

Fails, even with relaxed criteria

• A simple qualification check was requested for 3 known faults. • Note: All had already been flagged by consistency check.

EPC Feedback

S. Rowan - MPE-TM - 27/02/2013

Fails, even with relaxed criteria

RCBV30.L4B1:PCC.1

RCBV24.L4B1:PNO.d1

• A simple qualification check was requested for 3 known faults. • Note: All had already been flagged by consistency check.

EPC Feedback

S. Rowan - MPE-TM - 27/02/2013

Interesting curve• Passes 25A check point but fails 8A• Must be taken into account when

defining new criteria

Summary/What Next?

S. Rowan - MPE-TM - 27/02/2013

• Framework and language clearly usable by non-java experts!

• 60 A circuit offline analysis can now be fully automated (PCC1, PNO.d1)• Will be ready for upcoming HWC campaign• With relaxed pass criteria 60 A circuit analysis has >99% consistency• Pass criteria to be finalised/documented in test procedures

• All test procedures currently being updated by MP3

• Due to the discovery of an unknown fault, we propose to continue and carry out consistency checks for all remaining numerous circuit types (80-120 A & 600 A).• May find further unknown system faults• Will assist in refining pass criteria, useful for manual analysis• Analysis not likely to be automated during coming HWC, but if the

algorithms are written and qualified, future implementation is a possibility.

Summary

S. Rowan - TE-MPE-PE - 11/12/2013

Thanks for listening!

S. Rowan - MPE-TM - 27/02/2013