Post-mortem diagnostics for the Taiwan Light Source - CERN · EPAC 2008, Genoa, Italy, June 23-27, 2008 2 Outline ¾NSRRC accelerator facility Taiwan Light Source (TLS) and Taiwan

EPAC 2008, Genoa, Italy, June 23-27, 2008

Post-mortem diagnostics for the Taiwan Light Source

K.H. Hu, Jenny Chen, C.Y. Wu, Demi Lee, C.H. Kuo, P.C. Chiu, S.Y. Hsu, C.J. Wang, K.T. Hsu, M.S. Yeh, Ch. Wang

Presented by K.T. Hsu

NSRRC, Hsinchu 30076, Taiwan

2EPAC 2008, Genoa, Italy, June 23-27, 2008

Outline

NSRRC accelerator facility

Taiwan Light Source (TLS) and Taiwan Photon Source (TPS)

Operation status of the TLS (poster WEPC058)

Tools to capture data for post-mortem analysis

Some trip scenarios analysis

Summary

3EPAC 2008, Genoa, Italy, June 23-27, 2008 Hsinchu Science Park

NSRRC accelerator facility

Hsinchu

Taiwan

(Courtesy of Google Earth)


NSRRC accelerator facility – cont.

11

2233

44

55

6677

88

99

國家同步輻射研究中心

中心設施增能環館機電館

鄰近單位 99

11

1010

大門 22 行政大樓 33 研光大樓 44 儀光大樓 55 儲存環館 77 88 招待所國家高速網路與計算中心交通大學

66

Taiwan Photon Source

Taiwan Light Source

Chiao-Tung UniversityTsing-Hua University

Hsinchu Science Park


TPS officeActivity center & guest house &

Utility building no.3

Taiwan Photon Source project

Taiwan Photon Source (TPS)

3 GeV, C=518.4 mCommissioning : 2013+

Taiwan Light Source(TLS)

1.5 GeV, C=120 mDedication in 1993

Aerial view

North

TPS related poster:

TUPC054TUPC120TUPC127TUPD007TUPP025WEPC004WEPC005WEPC006WEPC017WEPC082WEPC104WEPC113WEPC114WEPC142THPC005THPC041THPC120THPP118THPP143


TLS beamline status

Soft X-ray Hard X-ray

20

19

1817

1615 1413

12

11

1009

08

07

0605

04030201

2122

23

24

SW

U5

EPU5.6

W20

U9

SWLS

01A 01B X-Ray Microscopy - SWLS01C

White X-ray - SWLS

EXAFS, Powder Diffraction - SWLS

03A Gas Phase (HF-CGM) - BM

04A 04B

Beam Diagnosis - BM

05A Spin-Polarized PES, PEEM - EPU 05B

Soft X-ray Scattering - EPU

07AX-ray Scattering - IASW

08B

XPS, UPS (L-SGM) - BM 08A

Beam Diagnosis - BM

09ASPEM, XPS - U5

11AMCD, XAS (Dragon) - BM11B

13B13C

13A SW6 - X-ray ScatteringSW6 - Protein CrystallographySW6 - Protein Crystallography

14A BM - IR Microscopy

15A

16A

IASW

17B W20 - X-ray Scattering

17C W20 - EXAFS

18A BM - Beam Diagnosis18B BM - LIGA

19A BM - X-ray Lithography19B BM - Photo Stimulated Desorption

20A20B

BM - (H-SGM) XAS, XPSBM - X-ray Instrumentation

21A21B

U9 - (White Light) Chemical Dynamics

23A IASW - SAXS

24A BM - (WR-SGM) XPS, UPS

Gas Phase (Seya) - BM

Beam Diagnosis - BM

U9 - (CGM) Angle-Resolved UPS, Gas Phase

Under construction

: 9: 10

NSRRC Beamlines

IR, VUV : 5

or planning

IASW

IAS

W

IASW

SP12U Inelastic X-ray Scattering

SP12B X-ray Absorption, Diffraction

12NSRRC Beamlines

Hard X-ray : 2

at SPring-8, JapanProtein Crystallography,

( )

( )

( )

BM - Tender X-ray Absorption, Diffraction

( )

( )

at Hsinchu, Taiwan

17A W20 - X-ray Diffraction

: Participating Research Group ( )

SWLS: Superconducting Wavelength Shifter

EPU5.6: Elliptically Polarized UndulatorIASW: In Archomatic Superconducting Wiggler

U5: Undulator (5 cm, 76 Periods)

W20: Wiggler (1.8 T, 27 Poles)U9: Undulator (9 cm, 48 Periods)

SW: Superconducting Wiggler (3.2 T, 32 Poles)

( )

Beam Diagnosis : 6

10B Beam Diagnosis - BM

( )( )

Insertion Devices

XAS, XPS ( M-AGM) - BM 08B

SP12D HE Photoemission

June 2006

ID Status

IDs:W20U5U9EPU5.6EPU4.6 (2009)

Superconducting IDs:SWLSSW6IASW6-R6IASW6-R2 (2009)IASW6-R4 (2009)


301 mA Top-Up operation ~ 0.05% photon flux constancy were achieved routinely> 98 % availability achieved

Keeping high stability and high availability is our commitment to the users

How to achieve this?

TLS operation status

@ 2008


Bad experience – one of an example

Problem of master oscillator related problem

0

5

10

15

20

25

30

35

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8

Num

ber

of B

eam

Los

s

2000 2001 2002 2003

What happened ?

Shutdown


Trilogy of the abnormal high trip rate in (February - December, 2000)

LLRFRF Low Level Electronics

RFTransmitter

Power distribution panel,bad contact

(Voltage drop to about 70 ~80 V in a few power line cycle)

Control board malfunction ,(always in by-pass mode,

no indication)

Output coaxial switch was open when line

voltage dropped to less than 90 V

RF Loss(< 0.1 sec)

Beam Trip

Rule of Thumb : Don’t Trust “Common Sense”

Bad experience – one of an example - cont.

UPSCircuit Breaker Master Oscillator

115 V60 Hz

If some trip diagnostic tools existed, it might be solved soon!


久病成良醫 (三折肱成良醫 )Failure is the Mother of Success (失敗是成功之母)Experience is the Mother of Wisdom (經驗是智慧之母)Fall into a moat and you will gain wisdom from the experience (吃一塹長一智)Fall Down Seven Times, Get Up Eight (七転八起)Success comes from experience; but experience comes from failure (Mark Twain)

Proverb

Strategies to improve or to keep high availability

Preventative maintenanceIdentify weaknesses of the system and remove themOperation automationAnalyze each trip event to identify where is the problem

=> Fault diagnosticCapture data before and after specific event happened

=> Post-mortem analysisMany measures were done after identify the real reasons for further improvement!


High time resolution

100 msec 10 msec 400 nsec

Oscilloscope, Fast digitizer

Beam intensityBeam positionBeam instabilityBeam trip triggerKicker waveform

…….

Control system archiverBeam currentBeam positionBeam loss distributionSuperconducting insertion device related signals….…

Vision XP,High density cPCIdigitizer

Beam currentBeam trip triggerSRF interlockSuperconducting insertion device related signals…….

BPM electronicsTurn-by-turn beam positionbeam phasebeam intensity …….

Low time resolution~ nsec

Various tools for beam trip data acquisition

All tools are trigged by the sum signal: beam trip + superconducting IDs quench + SRF trip + predefined trigger conditions


AND

100 μsecMonostable

50 μsecMonostable

50 μsecMonostable

Beam trip trigger

Ref IcBeam intensity50 mA/V

How to detect beam loss?

Beam

DCCT(Beam current) Ref Ia

Ref Ib

Beam tripNon beam trip0 mA

300 mA


Superconducting insertion device quench detection

MainPower Supply

+/- 350 A

TrimmingPower Supply

+/- 20 A

TrimmingPower Supply

+/- 20 A

Comparator Latch

Comparator Latch

WindowDetector Latch

1

4

7

Superconductingcoils

HTC lead

HTC lead

Diff. Amp.


Recorder, oscilloscope, and b-by-b feedback processor, t-by-t BPM

Recorder

Data memory in bunch-by-bunch feedback processor Post-mortem buffer of BPM electronics

Oscilloscope


Control Ethernet

cPCI based high density digitizer

Beam trip trigger

SRF related signals

SWLS related signals

SW6 related signals

IASW-R6 related signals



PC/LinuxFile Server

1. Record data before and after the trigger event for several seconds

2. The server saves all data for further analysis

3. Automatically identify faulty signals

96 channels/board

Sampling rate: up to 500 kS/secRecord length: msec to 10 sec (depending on sampling rate)

Embedded EPICS IOC

Power Lines


Maximum Sampling Rate:

100 MS/sec500 MS/sec

…

8 Ch14 Bit

100 Ms/secADC

Trigger

Kicker waveform

Gating (optional)

8 Ch14 Bit

100 Ms/secADC

Trigger

DataGating

Up to 8 boards (Maximum 64 Channels/Link)

PC/LinuxConfiguration,Data archive,

Analysis,User interface

Control Ethernet

VME based fast digitizer

Kicker trigger

Fast beam current detector


Kicker trigger

VME based fast digitizer – cont.

Beam trip

Triggeroutput

Trigger output

Beamcurrent

Kickerwaveform

Beam loss due to kicker misfired, …

Beam loss due to kicker misfired, SID trip, SRF trip, Transmitter trip, …

Kickertrigger

Live display for every top-up injection cycle

Store data if beam loss occurred

Injection time Non injection time


Klystron Anode Current

Klystron Arcing

Linac klystron stand arcing diagnostics

Simple pinhole CCD camera is also very useful!

This symptom is just the precursor of isolation breakdown!

10 μsec

100 msec


Diagnostic tools for the SRF system

- Try to understand the reasons of every RF/SRF fault

• What we experience a lot…– SRF “quench” due to fast change of beam behaviors…

• Fast beam dump or partial beam loss due to various reasons;• Fast beam dump due to kicker misfired;• Orbit change;• …

– LLRF’s feedback loop becomes unstable due to heavy beam loading…– False alarm by window arcing…– We experience with a “new” kind of trip event every few months and

usually solve it by reducing various feedback loop gains.– Modified many LLRF electronics to avoid false alarm.

• What we never or seldom experience…– Multipacting in the cold surface waveguide (Pf < 70 kW);– Real quench or field emission (rf gap voltage < 1.6 MV);– Vacuum burst due to hydrogen desorption (vacuum better than 0.7 nTorr);


Beam currentTuning angle

Pr

Pf

Pt phase

Beam phase

Interlock faultKlystron phase

SRF trip due to mismatch of beam loaded cavity


SRF trip due to mismatch of beam loaded cavity – kicker misfired

Spontaneous kicker fired(bad grounding contact)


SRF Trip due to mismatch of beam loaded cavity – SW6 fault


Feedback loop oscillations due to heavy beam loading

Amp loop

RF Gap voltage

Full scale = +/- 3.6o

Tuner loop

287 mA


Kicker #2 no fire

Kicker #2 miss fire during Top-up Injection.

Beam tripBeam Current

Kicker Trigger

SRF Interlock Sum

Tuning Angle

Phase Angle

Beam Phase Detector Output

The beam trip caused by kicker misfired


SWLS main coil PS failed

T1 (8.5oK)

T7 (6.5oK)T2 (6.5oK)

030 10/19/2006 17:52

Trip caused by the SWLS main power supply failed

10 Hz archived data, SWLS MPS drop lead beam current drop later slightly!


1 2 3 4 5 6 7 8 9 10024

Vol

tSWLS Trip Event on 19-Oct-2006 17:52:41

1 2 3 4 5 6 7 8 9 10024

Vol

t

1 2 3 4 5 6 7 8 9 1005

10

Vol

t

1 2 3 4 5 6 7 8 9 10-0.2

00.2

Vol

t

1 2 3 4 5 6 7 8 9 100

102030

Vol

t

1 2 3 4 5 6 7 8 9 100

102030

Vol

t

Sec

Beam Trip

Beam Current

SWLS MPSI

SWLS Vdiff

SWLS QD Out

SWLS PLC Int

302 mA0 mA

Trip caused by the SWLS main power supply failed – cont.

SWLS MPS tripped firstly (Vision XP data)。Reason: SWLS main power supply failed – control firmware problem!

Time (sec)

SWLS Trip Event on 19-Oct-2006 17:52;41


SW6 and IASW-R6 quench - I

SW6 Quench@ 14:03:51

IASW-R6 Quench@ 14:04:34

Partial Beam Loss

SID Trip Event on 25-Sep-2007


SW6 and IASW-R6 quench – II (SW6 quench)

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9

300

302

SID Trip Event on 25-Sep-2007 14:03:51m

A

Beam Current

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 LS218 Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 Coil Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 Pressure Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

100200300

A

SW6 MPSI RD

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9

-2-10

Vol

t

SW6 Vdiff

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

SW6 QD OUT

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Sec

Vol

t

SW6 PLC INT

Partial beam lossSID Trip Event on 25-Sep-2007 14:03:52

Time (sec)


SW6 and IASW-R6 quench – III (SW6 quench)

4.4 4.5 4.6 4.7 4.8 4.92.5

3

3.5SID Trip Event on 25-Sep-2007 14:03:51

Vol

t

IASW-R6 He Pressure

4.4 4.5 4.6 4.7 4.8 4.9298

300

302

304

mA

Beam Current

4.4 4.5 4.6 4.7 4.8 4.9

0

2

4

Vol

t

Inj Sync

4.4 4.5 4.6 4.7 4.8 4.90

2

4

6

Vol

t

IASW-R6 N2 Pressure

4.4 4.5 4.6 4.7 4.8 4.90

100

200

300

A

SW6 MPSI RD

4.4 4.5 4.6 4.7 4.8 4.9

-2

-1

0

1

Vol

t

SW6 Vdiff

4.4 4.5 4.6 4.7 4.8 4.90

10

20

Vol

t

SW6 QD OUT

4.4 4.5 4.6 4.7 4.8 4.90

10

20

Sec

Vol

t

SW6 PLC INT

Interlock active

Coil quench

Non injection time

Partial beam loss

SID Trip Event on 25-Sep-2007 14:03:52


SW6 and IASW-R6 quench – IV (IASW-R6 quench)

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9

260

280

SID Trip Event on 25-Sep-2007 14:04:34m

A

Beam Current

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 LS218 Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 Coil Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW-R6 Pressure Trip

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

100200

A

IASW MPSI RD

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9-505

Vol

t

IASW Vdiff

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Vol

t

IASW QD OUT

4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90

1020

Sec

Vol

t

IASW PLC INT


Time (sec)


SW6 and IASW-R6 quench – V (IASW-R6 quench)

4.4 4.5 4.6 4.7 4.8 4.9

3.3

3.4

3.5

3.6SID Trip Event on 25-Sep-2007 14:04:34

Vol

t

IASW-R6 He Pressure

4.4 4.5 4.6 4.7 4.8 4.9

260

270

280

290

mA

Beam Current

4.4 4.5 4.6 4.7 4.8 4.90

2

4

Vol

t

Inj Sync

4.4 4.5 4.6 4.7 4.8 4.9

4.974.984.99

55.01

Vol

t

IASW-R6 N2 Pressure

4.4 4.5 4.6 4.7 4.8 4.90

100

200

A

IASW MPSI RD

4.4 4.5 4.6 4.7 4.8 4.9-4-20246

Vol

t

IASW Vdiff

4.4 4.5 4.6 4.7 4.8 4.90

10

20

Vol

t

IASW QD OUT

4.4 4.5 4.6 4.7 4.8 4.90

10

20

Sec

Vol

t

IASW PLC INT

Partial beam loss

Coil quench

Trip MPS

Interlock active

He Pressure increase


Time (sec)

Injection


25-Sep-2007 14:03:51 SW6 quench. * SW6 quench happened.* MPS output dropped to 0 (before PLC interlock tripped MPS).

25-Sep-2007 14:04:34 IASW-R6 quench.* Partial beam loss happened during injection instance.* IASW-R6 quench.* PLC interlock tripped MPS.

Unknown reason caused SW6 quench, then partial beam loss at every injection instance, then IASW-R6 quench.

Many improvements were done: coil winding (poster WEPC114), liquid He filling procedure, interlock logic modification…

SW6 and IASW-R6 quench – VI


Beam loss induced IASW-R6 trip (2008/04/15 06:57)

IASW-R6

Partial beam loss

RF system trip

Coil quench

Summary

SWLS

SW6

Reason: one vertical corrector PS failed


Kicker diagnostic

Kicker diagnostic (VME digitizer working in segmented capture mode)

Beam Loss

Normal top-up injectionMaintenance test


BPM post-mortem data for large kicker jitters

Libera Brilliance post-mortem data (turn-by-turn)Diagnostics node (10 KHz rate for ~ 10 sec) is in implementation

K1 trigger -150ns

SRF Interlock Sum Active

Beam Loss

R1BPM3 Spectrogram

Short-time Fourier Analysis


BPM post-mortem data for IASW-R6 trip

Vertical tune jumps due to IASW-R6 trip


What we have achieved?

Several different tools are available.Some occasions have helped to identify some peculiar events.However no all events can explanation.

=> further efforts are essential !Simple automatic signal analysis are supported.

Our current efforts

Better integration of various diagnostic tools with the control system.Archive the events.Develop automatic analysis tools to search various signals.Develop better user interface.Automatic report generation.


Summary

Taiwan Light Source (TLS) is current operated at 301 mAtop-up operation. Top-up operation at 360 mA was commissioning.=> Poster WEPC043 (stable beam up to 400 mA has been tested)

Photon flux constancy can be kept around ~ 0.05%.

Availability is achieved 97 ~ 99% routinely.

To provide user better service, reduce trip rate is an important routine job.

Improving reliability, availability and serviceability is still on going.

All of the experiences will help the reliability and availability design for the newly proposed Taiwan Photon Source.

=> Reliability design for various subsystem.

=> Post-mortem buffer are expected embedded in BPM electronics, superconducting insertion devices’ controller, LLRF system, power supply controller, FOFB diagnostic node, … etc.


We learn and get helps from accelerator community a lot.

Thank You

Hope we can contribute something!

Documents

Post-mortem diagnostics for the Taiwan Light Source - CERN · EPAC 2008, Genoa, Italy, June 23-27, 2008 2 Outline ¾NSRRC accelerator facility Taiwan Light Source (TLS) and Taiwan