Introduction to DECam ICS/Alarms

• DECam, being a complex instrument, implements a series of alarms and interlocks in case of problems

• There are two sources of alarms: ICS and SISPI– ICS: instrument control system, these are the alarms closer

to the hardware instrument, and have relation with the actual instrument housekeeping

– SISPI: this is the high level software that runs the instrument. Provides the user interface. The alarms generated at these level are related to operational alarms, or derived directly from the ICS operation (lack of status coming from the ICS, or corrupted data, etc)

1

2

Overview

• In the present document we will talk about the ICS alarms only

• We will NOT describe how to handle each alarms, we will just talk about the alarm mechanisms themselves

• We will also give an overview on the ICS itself

3

Introduction to DECam Alarms

4

DECam Alarms: ICS architecture overview

• Based on National instrument compactRio (cRio) and compactFieldPoint (cFP) technologies

• cFP: Focal Plane temperature control (14.92), on H frame• LN2 cRio: LN2 and cryogenics control (14.86), on old

console floor• Camera cRio: monsoon crates, vacuum, alarms (14.87),

on H frame

Compact Rio (cRIO)

• Has a processor running a Real Time OS (VX works) that allows precise timing (usecs)• Has a main CPU unit that has several I/O lines, including a TCP/IP connection• Has embedded a programmable FPGA that allow for extremely fast and precise timing (nsecs)• Besides the main unit (Real Time CPU + FPGA) it has several slots where different I/OModules can be installed • They are programmed directly in Labview –including the FPGA –no VHDL or any other language.• The Real Time program runs in parallel to the FPGA program• Due to the parallel nature of labview, each one of these (Real Time OS and FPGA) can run as many parallel tasks as desired, using any I/O line

Compact FieldPoint (cFP)

• It is basically the same as the cRio, but it does not have a FPGA, just the Real Time part.

• Allows for several I/O modules also• Also programmed in labview directly

• To see the running processes there are two options:– Starts front Panels in a windows machine where the

cRio/fieldpoint where “deployed”. These panels connect to the running process and the user can see/modify anything

– Use a web browser to connect. It can be set so the user can even control the panel using the web browser.

• They can be programmed so everything starts automatically at boot time (both the Real Time program and/or the FPGA program)

• No PC is really required for them to run, only to watch what is going on

For both

8

DECam Alarms: ICS architecture overview

cFP

ethernet

FPA temp. control

Monsoon electronics, shutterVacuum readoutAlarms (includes heartbeats)

Cryogenic system

AUTODIALER

DECam Alarms: ICS architecture overview

Camera cRio

LN2 cRio

cFP

SISPI machines (Linux)ftp log data (sensors)

Direct tcp/ip:• regular status stream (per device)• alarm email message• overall flag (something is wrong)• Heartbeats

Database(history, web graphs, GUI, etc)Alarms if no logging or bad dataAlarms if no status streamAlarms if acquisition Software says so

ethernet

emails

Emails,twitter

AUTODIALER

Telephone network

Windows ICS

machine

• cRio and cFP Fron panels(vnc and/or browser)• Status diagram (exportedIn html, visible in sispi GUI)

Monsoonacquisition

Direct tcp/ip:• telemetry and alarms on:• ccds biases• Power supplies• crate temperatures

• Alarms are not the same as Interlocks. An interlock is a mechanism to prevent damage or to avoid some event automatically. An alarm is an action generated in order to call for external (human) action on the system

• Interlocks: (described in document #6601)– Hardware: the hardware makes an automatic action -no

software in between– Software: the software running on the ICS makes an

automatic action

• Alarms:– Emails: sends an email to a pre-defined list of addresses– Autodialer: makes some phone calls. This is reserved for

critical alarms10

DECam Alarms: alarms versus interlocks

• ICS (direct) alarms: AlarmSend vi– The camera cRio has a tasks that monitors several variables

from the other cRio and the cFP. If some variable gets out of range, it will:

• Send a direct email to a predefined list. This email has the alarmID (table at the end) and some brief explanation on the problem

• Sends the same message through a direct tcp/ip connection to the SISPI scripts running on a Linux machine (ics1)

- Regularly (alarm or not) it sends two status streams to the SISPI scripts running on ics1:

- A components message, that has the status of some detailed devices (DHE, vacuum, etc)

- A overall status message that has 1 or 0, meaning “all ok” or “there is something wrong”

- A quick look in this vi tells you if there is any alarm11

DECam Alarms: alarm mechanisms

Alarm mechanisms: AlarmSend

Components message

Overal status

Alarm ID

Email recipients

LN2 HB

• ICS (direct) alarms: LN2 vi- The LN2 cRio is connected (physically) to the autodialer. If

some critical problem appears, it will directly activate it for the phone call, no dependency on any other susbsystem

- This cRio is also looking at the status of the Alarms tasks on the camera cRio. If it founds the Alarms task not responding, it will send an email and also an autodial.

- The autodialer is connected to the CTIO telephone system using a converter from analog to VOP device. It will make a phone call if:

- Receives the signal to do so from the LN2 cRio (electrical signal)

- The power fails

- The backup battery is low

- The Alarms task on the camera cRio is not responding13

DECam Alarms: alarm mechanisms

• ICS (direct) alarms: LN2 vi- The critical alarms are the one that uses the autodialer

- In the LN2 vi, Interlocks tab:- The autodialer can be disabled, so be careful- Some of the autodialer alarms can be individually disabled- The relevant sensors values as its limits can be seen- If any out of range, it should turn red and blink- When an autodialer alarm is out of limit and the autodialer calls,

That alarms gets “latched”, meaning that it won’t go away until it is

Explicitaly reset , this for safety- A quick look into the Interlocks tab should tell if there is or there was

An autodialer alarm

14

DECam Alarms: alarm mechanisms

15

Alarm mechanisms: LN2 vi, Interlocks Tab

Autodialer diable/enable

Disable/enable individual alarms

Fault ocurred?

AlarmSend running?

Sensor values

Limits

• ICS (direct) alarms:HeartBeats:

FieldPoint

DECam Alarms: alarm mechanisms

AlarmSend

ICC VacuumShutter

CCD Temperatures

Ion Pump

LN2

Camera cRioLN2 cRio

PFFloor

• SISPI alarms related to ICS:– Receives the components and status stream from the camera

cRio. If something is wrong according to these messages, it will send some alarm (email, GUI errors, etc) and will store it in the database

– Retrieves log files with data values (sensors) from the LN2 cRio and the cFP. If there is no log data after X seconds, or if the data is corrupted, it will generate an alarm (email, etc). This serves as a way to check if everything is propertly running on the ICS subsystems

17

DECam Alarms: alarm mechanisms

18

DECam Alarms: alarms and email classification

• From the ICS point of view, there are two Levels of alarms:– Critical: needs to act immediately. Usually these alarms are

associated with the autodialer, and require attention in less than 1 hour. This are alarms that prevent an actual damage to the instrument unless they are acted quickly.

– Alert: needs to act as soon as possible, but no damage to the instrument involved

19

DECam Alarms: alarms and email classification

• ICS Emails – The emails that are sent by the ICS have a well-defined structure:

Subject: CTIO_<subsystem> <Alert | Critical>

Body:Alarm ID <ID> :<Alert | Critical> - <explanation/instructions….>

– The AlarmID stated on the email is a unique ID for each alarm

– Using this ID the responder can go to the master excell spreadsheet with the alarms IDs, and also to the “signals map” diagram where it is possible to follow what may have caused each alarm ID

• For a more detailed explanation on the conditions that produces each alarm, please see document #6601

20

Alarm ID Type Reason Level Interlock Phone

CCD Temperature 1510 criog >-10C Critical Mnsn Vicor turned off

CCD Temp alert 1520 criog >-85C or <-120C Critical Mnsn Vicor turned off (dial)

CCD Crate Temp 1560 elec >30 C Critical If >38Mnsn Vicor turned off

(dial)

CCD Crate DewPoint

1565 elec >14 Critical If >=15Mnsn Vicor turned off

(dial)

Mnsn Vicor Fault 1200 elec Overvoltage, etc. Alert Mnsn Vicor turned off

Mnsn Vicor Off 1210 elec Other alarms, user error

Alert -

LN2 pump off 1300 criog/elec Four LN2 interlocks Critical - (dial)

Shutter 1100 elec Shutter box error Alert -

CCD Photodiode 1500 elec Voltage>0.04 Alert Mnsn Vicor off shutter forced close

Vacuum critical 1430 crio >1.0E-5 Torr Alert

Vacuum alert 1400 criog >2.2E-4 Torr Critical LN2 pump stops (dial)

LN2 Level 1310 criog <45% or >95% Critical LN2 pump stops (dial)

DECam Alarms: alarms table (1)

21

Alarm ID Type Reason Level Interlock Phone

LN2 pressure 1320 criog <50psi or >130psi Critical LN2 pump stops (dial)

LN2 diff. pressure 1330 criog >20psi Critical LN2 pump stops (dial)

LN2 Heater 1340 criog Heater off Critical - (dial)

Oxygen sensor 1350 criog <15% Critical Shuts off LN2 valve and eventually pump

(dial)

Imager cRIO Heartbeat

2000 Ele/soft Loss of Heartbeat Alert - -

LN2 cRIO Heartbeat 1360 Elec/soft Loss of Heartbeat Alert - -

Fieldpoint Heartbeat 1440 Loss of Heartbeat Alert - -

Vacuum Heartbeat 1410 Elec/soft Loss of Heartbeat Alert - -

Glycol overtemp 1220 cryog Glycol source temperature too high

Critical Hardware FEE off (dial)

Valvebox pressure 1550 cryog <80psi Alert If <60Turn off valvebox

Facility gas pressure 1555 cryog <80psi Critical If <60Turn off valvebox

Windows PC Heartbeat

1999 Ele/soft Loss of Heartbeat Alert - -

Telemetry scripts 1998 soft Alert - -

Fake 1000 test none - -

DECam Alarms: alarms table (2)

• Under normal, regular operations:– All the ICS starts up automatically when the power to the cRios

and cFP is given

– To visualize the panels, open a VNC viewer to the ics2 or ics3 machines, startup the internet explorer brower, and click on the iterms under “favorite”. IT will show the running panel, which can be operated too.

The windows machines are not required to run the system, just to watch

- The favorite list points to:- 139.229.14.87:8000/ICC.html ICC (camera)- 139.229.14.87:8000/Vacuum.html vacuum and photodiode- 139.229.14.87:8000/Shutter.html shutter- 139.229.14.87:8000/Alarms.html alarm/status handler- 139.229.14.86:8000/LN2.html LN2 system- 139.229.14.92:8000/cFP.html camera temp. control

22

DECam Alarms: ICS startup

• Under development stages:- Open a vncviewer to ics2 or ics3.

- Click on the “decam” icon

- This will open the complete “Project” for all the ICS code.

- Open up the VI that needs to be modified (if any)

- Make the mod and save.

- To deploy the modification, you need to deploy manually. You will need to deploy all vis for both cRios and cFP manually, and you will have running the panels now on labview directly. For development faces this is the recomendation, as now you can stop/start vis manually. See next slide.

- Under this circumstances, you still can see the panels using the Internet Explorer browser, besides the direct panels here.

23

DECam Alarms: ICS startup

1

1: LN2 cRio (LN2 floor)2: cFP (focal plan temps)3: Vacuum (camera cRio)4: Shutter (camera cRio)5: ICC (camera cRio)6: Alarms (camera cRio)

2

3

4

5

6 Number 1 and 2 should be first,And Alarms should be last. The Order for the others does notmatters

Manual deploy

If for some reason the panels are lost,It is very likely the code in the cRio isStill running. One option to recober the panels is toClose the complete project (or kill),Then open it again. Now you mayDeploy it manually as explained before,But you may also try to simply “reconnect” the panels to the code in the cRio.For this select the target(LN2 cRio, camera cRio or cFP), Right click and select “connect”.If they successfully reconnect, the panelsMay open imediately. If that does not workThen proceed to manually deploy all of them

If panels are lost

26

If panels are lost and there is no recovery,LAST RESOURCE

In the worst of case, you can reboot both cRios and the cFP using the MAC.After a while you should be able to see the code running in the web browser.Now you can manually deploy as described before, if you want to.

REBOOT using the Measurement and Automation Exlorer:

-Click on the MAC icon (on ics2 or ics3 machines)- Click on “remote systems” in the left side- Select the cRio you wish to reboot- Press “Restart” button.