Introduction Software architecture Data flow

THE@INT Software Systems

John Young

April 26, 2019

Introduction Software architecture Data flow

Outline

Introduction

Software architecture

Data flow

2/21

Introduction Software architecture Data flow

Overview

Introduction

Software architecture

Data flow

3/21

Introduction Software architecture Data flow

Overall status of Software Systems

I Preliminary Design Review in May 2017I Updated documents needed

I Prototyping partially doneI Final Design Review planned for December 2019

I Telescope refurb tender on critical path

4/21

Introduction Software architecture Data flow

Mission

I Design and implement a fully robotic control system forthe INT and HARPS3I To carry out the Terra Hunting Experiment and open time

observationsI Must accommodate a possible companion instrumentI Must accommodate a possible solar telescope

I Science requirements – see HARPS3-EXE-0004I Operations model and use cases – see HARPS3-CAM-0007

5/21

Introduction Software architecture Data flow

Overview

Introduction

Software architecture

Data flow

6/21

Introduction Software architecture Data flow

Control system architecture

7/21

Introduction Software architecture Data flow

Instrument control software

I Telescope Control System (TCS)I Integrates mount, dome and environment sensing

I Cass Instrument Control Software (CICS)I Cass Unit

I ADCI Cassegrain Fibre AdapterI Polarimeter

I Acquisition and guiding (2 fields)I Calibration Unit

I Spectrograph Control Software (SCS)I Detector Control Software (DCS)

8/21

Introduction Software architecture Data flow

Supervisory software

I Automated schedulerI Schedules daytime calibrations, nighttime observations,

focus, pointing testI Observing Control Software (OCS)

I Observation Sequencer, Fault Manager, FITS BuilderI Data handling

I Engineering data collector and databaseI Local database and automatic transfer to Exeter mirror

I Science data archiverI Local storage, automatic transfer to Exeter, load into database

I Data Reduction SoftwareI Based on ESPRESSO DRS

9/21

Introduction Software architecture Data flow

Overview

Introduction

Software architecture

Data flow

10/21

Introduction Software architecture Data flow

Actors

Open time observer A scientist who wishes to carry outobservations with the INT, in service mode, andmake use of the resulting data

Consortium observer A scientist from the HARPS3consortium working on the Terra-HuntingExperiment (THE)

Consortium engineer An engineer from the HARPS3consortium, responsible for ensuring that bothopen time and THE observations are carried outsuccessfully

WHT operator Responsible for ensuring safe startup andshutdown of the INT. MoU states that the ING willprovide safety override for robotic operation

11/21

Introduction Software architecture Data flow

Service mode observing and data reduction

12/21

Introduction Software architecture Data flow

Data handling overview

13/21

Introduction Software architecture Data flow

Observing dashboard

14/21

Introduction Software architecture Data flow

Detail of telescope/instrument control

15/21

Introduction Software architecture Data flow

Automated Scheduler

Phase 2 - Add observation group

THE Scheduler

Observation Pool

Sequencer

Scheduler

Scheduler gets observation block

for 'best' target from pool

Request for target sent across network Observation block

returned back across network to sequencer

Observation blocks added to pool

Phase 1 - New observation group requested

Yes

WEB APPLICATION MESSAGING PROTOCOL (WAMP)

Accepted?

A

A

A A

B

B

BB

B

C

C

C

DD

D

D

D

D

D

16/21

Introduction Software architecture Data flow

Detail of post-collection data handling

17/21

Introduction Software architecture Data flow

Engineering data collection

I INT/HARPS3 sub-systems will publish engineering datacontinuously

I Published data will automatically be collected and storedin the on-site engineering databaseI Data collection must be robust to sub-systems going

offline/onlineI Query facility used by OCS FITS Builder

I Database mirrored in Exeter, updated nightly (TBD)I High-bandwidth data only kept for N days, available for

manual transferI Guide camera videoI Very low-level debug messages

18/21

Introduction Software architecture Data flow

Science/calibration data collectionI FITS files will be assembled from a raw detector frame and

ancillary data/metadata provided by the robotic controlsystemI Science metadata: required by DRS (timestamp, exposure,

target and telescope coordinates, . . . )I Engineering metadata: summarise instrument status

during exposureI Exposure meter time seriesI Acquisition and integrated guide images

I Temporary storage on siteI Transfer to main Archive in Exeter

I Automatic pipeline reduction on importI Retain multiple, traceable reductions of each observation

I Edit metadata (e.g. spectral type) and re-reduceI Upgrade DRS (occasionally) and re-reduce

19/21

Backup slides

Data Reduction Software

I ESPRESSO DRS statusI Validated and operational on ParanalI Able to read HARPS dataI New web-based "trigger" front-end for CPL recipes

I New work for HARPS3I Adapt ESPRESSO DRS for HARPS-X data and test

(Geneva)I FITS Builder (Cambridge)I Integrate/adapt Trigger (Cambridge)

I Details to be discussed with GenevaI Interfaces with HARPS3 Archive for pipeline input/output

files (Exeter)I Science QC tools (includes alerts) for consortium observer

(Exeter)

20/21

Alerts

I Alerts can be generated fromI Engineering dataI DRS-processed data

I FlexibilityI Use any monitored variablesI Complex alert criteria?

I Combinations of variablesI Linked to events

21/21

User interfaces (1)

I Operations UIs (consortium use)I Observing dashboardI Scheduler “queue” UII Engineering archive access

User interfaces (2)

I Engineering GUIs (view only during robotic operation)I Telescope systemI Cass instrumentI SpectrographI Detector

I Observer tools (community/consortium use)I Exposure Time CalculatorI Scheduler simulation tool

I For repeated observation or surveyI Science archive access

Communications framework

I Robotic control of telescope + instrument requires adistributed event-driven software system

I Components communicate over network using WAMPprotocolI Open standard WebSocket subprotocolI Data serialization using JSON or MsgPack

I Use open-source Crossbar.io router and Autobahn clientlibs (many languages)

WAMPWAMP implements two communications patterns:1. Remote Procedure Call (command/response)2. Publish Subscribe

Realistic WAMP example

Python asyncio event loop

WAMP

translate

pexpect

Receive remoteprocedure call

Convert into messagefor target instrument

Dispatch message to instrument and

yield

WAMP

translate

pexpect

Return from remote procedure

call

Convert into return value

Receive message from instrument

Periodic timer

translate

pexpect

Initiate poll of instrument data

Convert into messagefor target instrument

Dispatch message to instrument and

yield

WAMP

translate

pexpect

Publish value

Convert into return value

Receive message from instrument