1 BROOKHAVEN SCIENCE ASSOCIATES Control System Overview Control System Group – Bob Dalesio presenting NSLS-II ASAC Review March 26 – March 27, 2009

1 BROOKHAVEN SCIENCE ASSOCIATES

Control System Overview

Control System Group – Bob Dalesio presentingNSLS-II ASAC Review

March 26 – March 27, 2009


Outline

• Design Requirements• Technical Requirements & Specifications• Cost & Schedule Baseline

• Status• Staffing• Standards• Technical Development

• Recent Accomplishments• Near Term Plans• Conclusions


Control System Requirements – 1 of 2

• Bunch Length 1-40 psecs• 2.6 usec ring revolution• Top off every 1 minute• Top off bunch train 140-300 nsec• Top off damping time 10-50 msecs (no extraction)

• Slow control of orbit trims, quadrupoles, sextupoles, and insertion devices are slow but can be set synchronous to the orbit reference.

• 5 Hz updates to operators of up to 1000 chosen parameters • Archive up to 20,000 parameters at a rate of 1 Hz continually

• Must scale to support 150,000 physical I/O connections and 400,000 computed variables

• 99.7% availability during operation


Control System Requirements – 2 of 2

• Transient Recording• Take coherent turn by turn orbit data for up to 800 channels for 1024 turns• Latch the last 10 seconds of data from all parameters in the storage ring • Beam line needs 1 msec archiving over 1 minute for temperatures and positions

• Provide data for all control aspects (no hidden parameters)• 5 KHz RF Feedback on beam phase• 10 kHz orbit feedback, (100 usec loop time)

• 360 BPMs (12 per cell)• 120 Corrector PS (4 per cell)• All data available to system with revolution identifier for turn by turn data correlation.

• 20 msec equipment protection mitigation• 1 Hz model based control • 10 kHz power supply read backs triggered from timing sys• 10’s of Hz Data Collection for RF loop correction.• 80 psecs pulse to pulse timing jitter.

• During top off, some beam lines will need 1.1 - 1.8 psecs of timing jitter• 2 nsec timing resolution

• Provide electron detector as event for beam line


Cost & Schedule Baseline• Baseline

• Manpower by FY09• 4 High Level Application Engineers• 2 RDB Architects• 9 Project Engineers• 1 EPICS Expert• 1 System Administrator

• Hardware• 156 IOCs w/ timing hardware• $1M network hardware & 360K timing distribution• $400K Servers and Control Room Consoles

• Software Licenses $300K• Schedule

• Build synchronous distributed device controllers by end 2009• Install Physics Environment for Application development 2009• Install First release of component. lattice, and wiring RDB tools 2009• Prototype subsystems and detailed design by end 2009• Procure and implement subsystems in 2010• Install and test Subsystems in 2011/2012• Improve and manage RDB tools over the project to support design, installation, test, and

maintenance.

• Extend Physics Environment to support installation, test, commissioning and operation.


Staffing

• High Level Applications• Nikolay Malitsky, Guobao Shen, and Jayesh Shah• Contract work or new hire

• Relational Database (IRMIS)• Don Dohan and Gabriele Carcassi• Contracts are being used to enhance our capabilities early in the project

• Project Engineers• Yuke Tian, TBD – power supplies• Huijuan Xu – vacuum• Rob Petkus - network• Yong Hu, Kiman Ha – diagnostics• Michael Davidsaver - RF• David Dudley – facility and EPS control• TBD – timing• Daron Chabot, TBD – beam line control• Contracts with other groups: Larry Doolittle et. al. LBL, Joe Meade BNL• Contract employees for support (Sheng Peng, Steve Lewis, Steve Hunt, Cosylab, Observatory Sciences)

• EPICS• Ralph Lange (BESSY), Mauro Gianchini (INFN) temporary for 12/6 months• Position to fill


Control System Standards - 1 of 5

Psy:PI-Ssy:SI-Tsy:TI<Dev:DI>Sg:SgI-SD

Control System Naming Examples

S:C30-VA:G1<SGV:A>Pos:1-Opn

S:C20-FE:B<FV>Pos:1-Cls

S:C20-VA:G1<IP:A>E:1-RB

S:C20-VA:G4<CHMA:B>

S:C20-VA:G4<BLWR>

S:C24-RF:G1<TMP:B>

S:C24-RF:G1<CCG:B>P

S:C24-VA:G1<CCG:D>P

System Device Signal

Comment

cell 30, vacuum, girder 1, sector gate valve A (upstream), open

cell 20 BM front end fast valve

1st sputter ion pump at C20 girder #1

2nd aluminum chamber at C20 girder #4

rf shielded bellows at C20 girder #4

TMP station at 2nd SC cavity insulating vacuum

cold cathode gauge at 2nd SC cavity insulating vacuum

cold cathode gauge at 2nd SC cavity beam vacuum

Nomenclature Standard is in place


Channel Access Client (CAC)Connection Data Transfers

WAN/LAN/Local

Connection Data TransfersChannel Access Server (CAS)

DB Engine

Device Support

Driver SupportI/O Controller

Archiving

Archive Access

Archive Viewing Archive AnalysisUser ToolsDM/MEDM/DM2K/EDM/JDMAlarm Handler / stripTool/knobManager

Types

Connection Server

Program InterfacesC, C++, Fortran, JAVAIDL / Matlab / MathematicaActiveX / DDE / VisBasicSDDS / SAD / tcl / PERL / PythonState Notation Lang / FSQT

Control System Standards 2 of 5Control System Standards 2 of 5

Configuration tools provide:

data acquisition

supervisory control

steady state control

operator displays

alarm management

Interfaces exist to add:

drivers for new hardware

state control

model based control

EPICS Tools Reduce Programming


Control System Standards – 3 of 5

• IRMIS tools are used for all configuration data: lattice, components, wiring etc.• Embedded Real-Time Operating System choice: RTEMS (LCLS, Spear, CLS) or

VxWorks (APS, SNS, Diamond, SLS)• Linux Workstations running Debian• Evaluate the possibility of participating in the CSS development• Visual Database Configuration Tool w/ Modifications for Table Entry• Use physics applications: Matlab Middle Layer Toolkit – MMLT (many light

sources), eXtensible Accelerator Language - XAL (SNS/SLAC), Elegant (APS), Tracy online simulation (Diamond)

• Evaluate Use of Experiment Control Tools: SPEC (ubiquitous), BluIce (SLAC), IDL, SynApps (APS/EPICS), GDA (Diamond), CBass (BNL), Scientific Studio

Software standards are being evaluated



Shared Memory – Ethernet Hardware w/ Synchronous Protocol

Ethernet – EPICS Channel Access Protocol

Events/ / Timing Data

PLC

CNET

I /O

Field I/O Field I/O…..

Instrumentation Ethernet

ALS

NET

Field I/O Field I/O…..PLCS ,Slow I/O, High Reliability, Low Accuracy, High Density Vacuum, PPS, MPS, Non FOFB PS, Cryo., Facility control

CPU

EVR

I /O

I /O

MTRS

Read Remote Ethernet and serial devices. High density IO, Motor Control Position Control

Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility

CPU

EVG

CEL L

Timing Master

Fast Equipment Protection Signal

Architecture for instrumentation is being prototyped

CPU

EVR

CEL L

PS IOCs DCCT

PrMon

Non FOFB

Diags.

CPU

CEL L

FOFB IOC

FC1

FC2

FC3

SC1

SC2

SC6

Correctors

RFBPM 1

XRay BPM1

RFBPM 2

RFBPM 8

XRay BPM2

XRay BPM4



• Dell Linux development workstations• PLC Solutions – having pricing, checking reliability, prototyping test stands.

SiemensAllen Bradley Control Logix

• Building AutomationALS Needs fast Ethernet based interface board

• VME cratesRittal 7 slot VME64x-4U7S-PS900C-SM 4,100.00Weiner 9 slot VME195xPO 4,100.00V-Ross uSystem uTCA w/MCH & Intel CPU 5,000.00

• CPU BoardsMotorola MVME 3100 1,400.00

• Motor ControllersOregon MicroSystems OMS MaxV 1,600.00

Hardware Components are being selected


Development Opportunities• A Synchronous Device Interface that implements an open

standard for high speed, deterministic functions provides a modular platform on which to develop a Fast Orbit Feedback system.

• High Level Applications currently tie together functions through data or file structures. To make the components of High Level Applications modular and distributed (and therefore reusable), a client/server architecture is needed.

• Implement asset management in the relational database. Development of adequate tools to enter, report, and track equipment and document its installation saves significant money over the life of the project.


Fast Feedback (1 of 4)





Latency calculations:

Data per BPM Data per cell Total data to be distributed

GTP 1.25Gbps 8-bit datawidth



10 bytes(X:4; Y: 4; Status:2)

120 Byte(8 BPM4 XBPM)

120 * 30 = 3.6Kbyte

3.6Kbye/1Gbps= 28.8 us

FPGA: 125MHZ

3.6Kbye/2Gbps= 14.4 us

FPGA: 250MHZ

3.6Kbye/2Gbps=14.4 us

FPGA: 125MHZ


SDI core design: LBL BNL: BPM measurement, GTP interface test, Integration

GTP interface test1.fiber loop back two SFT ports2.Data is internal counter


17 BROOKHAVEN SCIENCE ASSOCIATESDesigned lattice & installed hardware seq

Design Goal

Distributed IOC Process Databases

Channel Access

Application / Family

Mid Level Data Client/Server

PhysicsApplications(Thick Client)

Lattice

Name Mapping

ConfigurationParameters

Conversions

Under development

Existing

Need to port

Computed Data

Dipole Quad Sext. Corr. BPM RF

Measured Orbit

Orbit Differences

Gradient Errs & Corrections

Beam R. M. Diff’sData Server

Optics Deviations

OpticsResp. Matrix (S)Model Server

PhysicsApplications(Thin Client)

High Level Applications – 1 of 4

EPICS Client/Server

Need to develop



Real Machine

Distributed IOC process databases

Dipole Quad Sext.

Corr.

BPM RF

VIOC

EPICS Client/Server

Tracy Simulation Engine

High level

Low level

MMLTMMLT XAL/TracyXAL/Tracy Python/TracyPython/Tracy

EPICS Client/Server

Online simulation and physics tools are operational

Dipole Quad Sext.

Corr.

BPM rf

Elegant Simulation Engine

Dipole Quad Sext.

Corr.

BPM rf

Model APIVIOC

EPICS Client/ServerEPICS Client/Server

Model API

ATTracy

Elegant

EPICS Client/ServerEPICS Client/Server


Kick beam: 1e-5 for 1st H&V


MMLT Setorbit – orbit correction (against VIOC)

Run setorbitgui (with 5 iterations)

Orbit reset to 0 after correction



• High level application development environment is available for developing commissioning tools based on MMLT, XAL, or Python.

• Online simulations of Tracy 3 and Elegant provide a solid EPICS environment for testing these tools.

• The API for model servers is being formulated while these online simulations are brought online.

• The DDS API is being prototyped as the interface for these applications.

• Channel Access is being overloaded in version 3 and extended in version 4 to provide a client/server environment for these applications.


XML protocol (REST style WS)

IRMIS – 1 of 4

Web applicationsWeb applications

JavaScript bridgeJavaScript bridge Applets and WidgetsApplets and Widgets

Java Client APIJava Client API

Data Service layerData Service layer

Database layerDatabase layer

Application Architecture Supports Independent Development


Database Status – 2 of 4

Component Type

Component

Lattice

EPICS Database

Name Mapping

Wiring

Develop Use Cases

Configuration Tools

Component Type

Component

Lattice

EPICS Database

Name Mapping

Wiring

Scripts

Web Based Reports

Component Type

Component

Lattice

EPICS Database

Name Mapping

Wiring

Files for control

Lattice

EPICS Database

Name Mapping Complete Release 1

Complete FY 09

Tools develop in subsequent releases as users are added.

Component TypeComponentLatticeEPICS DatabaseName MappingWiring

Complete FY 10


IRMIS – 3 of 4


IRMIS 4 of 4Application prototype

Lattice Deck manager using IRMIS

IRMISIRMIS

IRMIS Data IRMIS Data ServiceService

HLA AppHLA AppDeck GeneratorDeck Generator

Tracy Elegant

Other LatticeOther LatticeInformationInformation

Simulation Server(Tracy)

Simulation Server(Elegant)

IRMIS to Deck Object Mapping

HLA AppHLA AppDeck InputDeck Input

Deck to IRMISObject Mapping

Deck ParserTracy/Elegant

ElegantDeckTracyDeck


Recent Accomplishments

• Prototype subsystems started for slow applications • Vacuum, beam line and personnel protection.

• Prototype subsystems started for beam correlated applications • Diagnostics, power supplies, and LLRF

• Prototype protocols running for High Level Applications architecture

• DDS deployed as an interface to some channel access mechanisms.

• Standard interface for orbit control running.• IRMIS API for lattice entry and report is running and integrated

into the online model environment.• Preliminary design documents for vacuum, power supply,

diagnostics, and network started. (*)


Near Term Plans

• Prototype subsystem test stands for facility control and personnel protection.

• Prototype subsystem test stands for diagnostics, timing and LLRF.

• Expand DDS interface to all channel access mechanisms.• Prototype HLA communication requirements on CA V4• Prototype the Twiss Server.• Develop tools for lattice and wiring for entry and browsing.• Complete preliminary design documents for vacuum, power

supply, diagnostics, and network (*).• Write the preliminary design documents for timing, RF, PPS,

and facility control (*).


Concluding Remarks

• We have had good success in staffing. We are putting staff augmentation capability in place through the use of contract labor and temporary employees.

• Choosing EPICS reduces the programming needed to accomplish the engineering control tasks.

• The control team is developing the preliminary design documents and interface control documents for subsystems that have project engineers on board. The others will follow shortly(*)

• Hardware tests will be in place for crates and processors in the next 2 months. (*)• All subsystems will be prototyped by early FY10.• Development in the relational database is keeping pace with the needs of the users of the

relational database.• Development of the fast orbit feedback hardware now has reasonable momentum. A full

working prototype based on the Avnet board should be operational in August.• The high level application environment already supports several models and sets of

commissioning tools. The development of a client/server architecture is progressing well.

(*) Comments from the previous review that are being addressed. Also, coverage of wiring costs was confirmed.


Backup – High Level Applications

• DDS open standard as an API for the high level applications client and server.


DDS-Based High Level Application EnvironmentConceptual Solution: Start the implementation of the DDS specification in the form of the EPICS extension based on the Channel Access protocol

EPICS-DDS


Client/Server Architecture for High Level Apps

High Level Application Environment – Matlab Middle Layer Toolkit, XAL

Client API – CAV3: what is used now for channel data not enough

TINE: is used for channel data – large interface

DDS: superset of what is needed. Evaluate for packing/unpacking data

CAV4: will be applied to the interface definition

Protocol – CAV3: could overload array packet for early tests

TINE: could also overload array packet for early tests

Open-DDS: evaluated and deemed too slow – based on Corba

ICE: evaluated and deemed to complicated to deploy

CAV4: will be applied to the payloads we determine are needed for HLA

Server API – CAV3: what is used now for channel data not enough

TINE: is used for channel data – large interface

DDS: superset of what is needed. Evaluate for packing/unpacking data

CAV4: will be applied to the interface definition


Status

• Middle-layer server [ caExampleApp ]• Application-specific fixed structures based on the EPICS waveform record, array of characters [ TwissApp ]• Synchronous access [ caExampleApp ]• Asynchronous access [ caMonitorApp ]• Benchmark use case [ caTime ]

http://sourceforge.net/projects/epics-dds

Prototype / working solutions of the following aspects:


Backup – Subsystem Interface Design

• Network• Power Supply• Conventional Facilities• Vacuum• RF• Beam Lines• Diagnostics


Subsystem I/F - Network


Subsystem I/F - Network

• Pre-spliced = >$ material, <$ installation• On-site splicing = <$ material, >$installation

• Little margin for error• (2) 60-fibre bundles in each direction = 120x2• (1) tap/cell, (4) fibres/tap, Optitip -> LC harness• (2) spare fibres/cell = security, growth

AnyLAN Pre-tapped, redundant STAR topology


Fast Orbit Feedback Synchronous Bus




CPU

CEL L

FOFB IOC

Feedback Interface

……….

FC1

FC2

FC3

SC1

SC2

SC6

30 Storage Ring2 Injection

Correctors

DCPS 01

DC Power Supplies

RFBPM 1

XRay BPM1

RFBPM 2

RFBPM 8

XRay BPM2

XRay BPM4

DCPS 02

DCPS 39

Subsystem I/F – Fast Orbit Feedback




ALC

NET

Field I/O Field I/O…..

CPU

ME-LGR Multi Equipment - LAN Gate

Router. .


5 1 per SR Equipment Building1 for the injection Equipment Bldg.1 for the cryo building

Subsystem I/F – Conventional Facilities



PLC

CNET

I /O

Field I/O…..


CPU

I /O


Field I/O

24 Port Digi

5 1 per SR Equipment Building1 for the injection Equipment Bldg.1 for the cryo building

Vacuum Device Controllers

24 Port Digi

Vacuum Device Controllers

Per Cell

30x

Subsystem I/F – Vacuum


Subsystem I/F - Vacuum

• Super period Vacuum - Even Cell

IP TSP NEG-C

NEG-S

TMP

CCG

TCG

RGA

GV

S1-LS

2 2 2 1 2 1 1 2

S2 1 1 1

S3 1 1 1 1 1

S4 2 2 1 1 1

S5 1 1 1 1 1

S6 1 1 1

Total

8 8 1 7 2 4 2 2 2



• Super period Odd Cell

IP TSP NEG-C

NEG-S

TMP

CCG

TCG

RGA

GV

S1-SS

1 1 1 1 2 1 1 2

S2 1 1 1

S3 1 1 1 1 1

S4 2 2 1 1 1

S5 1 1 1 1 1

S6 1 1 1

Total 7 7 1 6 2 4 2 2 2



• Injection device vacuum - Even Cell

• Injection device vacuum - Odd Cell

IP TSP NEG-C

NEG-S

TMP

CCG

TCG

RGA

GV FV

ID-FE

4 4 4 1 4 2 2 2 1

IP TSP NEG-C

NEG-S

TMP

CCG

TCG

RGA

GV FV

ID-FE

4 4 1 4 2 2 2 1



• Beam line front end Vacuum

• Note:• IP, TSP will be installed as a unit• NEG-C and NEG-S will be activated during maintenance period using portable

power supply• RV will be operated manually and will not be monitored by control system

IP TSP

NEG-C

NEG-S

TMP

CCG

TCG

RGA

GV FV

BM-FE

4 4 1 4 2 2 2 1


Phase Control Synchronous Bus




CPU

CEL L

RF IOC

……….

CFC

RFP

4 Storage Ring1 Booster Ring7 LINAC

HPPLC

CFC

RFP

DSP

Subsystem I/F - RF





CPU

EVR

MTR

BL IOC

……….6 -1 per BeamlineMTR

PS IOC

MPSPLC

MTR

LVDT

CPU

EVR

MTR

MTR

MPSPLC

MTR

LVDT

Subsystem I/F – Beam line





CPU

EVR

CPU

EVR

F I R E

Camera

DCCT

FCT

Camera

Camera

Camera

Camera

Camera

CPU

EVR

BLM 00

BLM 01

BLM 02

BLM 59

Subsystem I/F – SR Diagnostics





CPU

EVR

CPU

EVR

F I R E

Camera

MTR

MTR

Camera

CPU

EVR

TUNE

OSCILLATOR

Subsystem I/F – SR Diagnostics


Design of Diagnostics Controls for Injector (Linac & LtB)

Monitor type Quantity Sensor Controller/DAQ EPICS IOC platform Wall Current Monitor(WCM) 3 Resistive PCIe ADC(100KS/s) PCIe / Linux Fast Current Transformer (FCT) 1 Bergoz FCT PCIe ADC(500MS/s) PCIe / Linux Fluorescent Screen 3 Firewire-based CCD Camera? microIOC CosyEye microIOC / Linux

Linac:

Monitor Quantity Sensor Controller/DAQ IOC platform Beam Position Monitor (BPM) 3 RF Four-button Libera Brilliance PC/Linux Fluorescent Screen 4 Firewire-based CCD Camera? microIOC CosyEye microIOC /Linux Fast Current Transformer (FCT) 1 Bergoz FCT PCIe ADC(500MS/s) PCIe/ Linux

LtB transport line :

• IOCs for Linac & LtB diagnostics (2 in total)• One IOC (PCIe/Linux) for 3 WCMs + 2 FCTs

• One IOC (microIOC/Linux) for 7 Screens via firewire & repeaters (longer distances than 4.5m)


Design of Diagnostics Controls for Injector (Booster & BtS)

Booster:

BtS transport line :

• IOCs for Booster & BtS diagnostics (3 in total)• One IOC (PCIe/Linux) for 1 DCCT + 2 FCTs

• One IOC (microIOC/Linux) for 12 Fluorescent screens via firewire & repeaters

• One IOC (PC/Linux) for 29 BPMs in injector

Monitor Quantity Sensor Controller/DAQ IOC platform Beam Position Monitor (BPM)

20 RF Four-button Libera Brilliance PC/Linux

Fast Current Transformer (FCT)

1 Bergoz FCT PCIe ADC(500MS/s)

PCIe/ Linux

DC Current Transformer (DCCT)

1 Bergoz NPCT PCIe ADC(100KS/s) PCIe/ Linux

Streak Camera 1 Optronis IOV-10 & SC-10 & ANIMA-PX 25 & Ethernet

Included PC/ Linux

Firewire Camera 1 IEEE-1394b interface microIOC CosyEye microIOC /Linux

Fluorescent Screen 6 Firewire-based CCD camera? microIOC CosyEye microIOC /Linux

Monitor Quantity Sensor Controller/DAQ IOC platform Beam Position Monitor (BPM) 6 RF Four-button Libera Brilliance PC/Linux Fluorescent Screen 6 Firewire-based CCD camera? microIOC CosyEye microIOC /Linux Fast Current Transformer (FCT) 1 Bergoz FCT PCIe ADC(500MS/s) PCIe/ Linux


Design of Diagnostics Controls for Storage ring

Monitor Quantity Sensor Controller/DAQ IOC platform Beam Position Monitor (BPM)

240 RF Four-button Libera Brilliance MicroTCA/Linux

Photon BPM (PhBPM)

Number of undulators

FMB-Berlin Four-blade with motorized stages

Libera module & Motor Controller

MicroTCA/Linux VME/RTEMS

Fluorescent Screen 1 Firewire-based CCD camera? microIOC CosyEye microIOC /Linux Fast Current Transformer (FCT)

1 Bergoz FCT PCIe ADC(500MS/s) PCIe/ Linux

DC Current Transformer (DCCT)

1 Bergoz NPCT PCIe ADC(100KS/s) PCIe/ Linux

Beam Loss Monitor (BLM)

70 Bergoz p-i-n diodes CosyLab microIOC BLM microIOC /Linux

Streak Camera 1 Optronis IOV-10 & SC-10 & ANIMA-PX 25 & Ethernet

Included; via Ethernet PC / Linux

Firewire Camera 1 IEEE-1394b interface microIOC CosyEye microIOC /Linux Pinhole Camera 1 Point Grey Flea2 Included; via firewire microIOC /Linux Emittance Monitor 1 X-ray CCD camera Included; via firewire? microIOC /Linux Tune Monitor 1 Pick-up electrodes & RF-drive

stripline & amplifier Spectrum analyzer with Ethernet

PC / Linux

Beam Oscillations Monitor

1 Pick-up electrodes Spectrum analyzer with Ethernet

PC / Linux

X and Y Beam Scrapers

2 Step motor Motor Controller VME/RTEMS

Documents

1 BROOKHAVEN SCIENCE ASSOCIATES Control System Overview Control System Group – Bob Dalesio presenting NSLS-II ASAC Review March 26 – March 27, 2009