LLRF Embedded Development · 2018. 9. 28. · –DAQ analysis and storage –High-speed means what for each system? •Data lifetime? •Scope capabilities •Event identification

LLRF Embedded Development

Brian Chase, Joshua Einstein-Curtis

Controls Modernization Group

28 September 2018

Introduction

• LLRF is a high-speed, high-precision real time system.

• For the FNAL accelerator infrastructure, have relatively high

data rates

– Similar to beam instrumentation

– Necessitated builds of a Labview client for commissioning,

debug, and diagnostics (Backdoor/custom)

9/27/2018LLRF Embedded Development2

Example:CMTS has 4 ADC and 2 DAC channels per cavity. In addition to the raw data, there are also the baseband IQ and amplitude/phase signals, error, and controller intermediate stages

2 * (4*2 + 2 + 10) * 65 MSPS * 32-bit = 82 Gbps max82/65 = 1.26 Gbps if 1 MSPS sampling

1.26 Gbps / 8 * 360 = 56 TB/hour

0.02 * 56 = 1.12 TB/hourif only 20 ms of 1MSPS data every second

Real-time Needs

• Hard Realtime LLRF (timescale-dependent, of course)

– Failures not allowed

• Bucket-resolution timestamping on data

• Necessary to do extensive hardware testing or simulation

(Qemu, ARM simulator)

– Good debugging tools a necessity

• Utilization and overloading

– Need for optimizations and good compilers


vxWorks

• MVME2400, MVME5500 with Kinetic Systems backplane

• VxWorks 5.5, 6.1, 6.4, 6.9

– 5.5 running on MI, FAST, 6.4 on Recycler, 6.9 dev system provided to controls

• Power PC 603e, 750 or 7457

• Standardized communication between front ends enables faster

development. Shared libraries, etc.

– https://www-bd.fnal.gov/controls/micro_p/mooc_project/welcome.html

• Protocol Compiler

– Custom for each device? Possibly leads to complexity and versioning

issues

– Each device has own receive stack, and each group develops their

own


https://www-bd.fnal.gov/controls/micro_p/mooc_project/welcome.html

Current Model


Spreadsheet LLRF Frontendsys_init.cpp

PythonScripts-to-dabbel

Modification Scripts

Dennis’ Frontends

Acnet and Front-end independently record and manage variables

SoC MFC


LLRF Control and Timing

• VUCD

– Developed in mid-nineties

– No replacement for current systems available

– Timing and data decoding using backplane triggers

– Multi-timer system

– No planned replacement for current/deployed systems

• Bucket resolution and accurate timestamping needed for

future diagnostics


Build Environment

• nova build environment (esd tools) well developed

– LLRF projects are all there, including configuration files for new

SoC systems (spreadsheet register maps and defaults)

• Need for a well-developed upgrade plan across departments

• Need for supporting both legacy systems and new systems

using same tools?

• Significant increases in performance with each controller, but

processor-specific programming can make porting difficult


Revision Control

• Currently part of build toolchains on nova

• One of the largest issues we’ve run in to is revision and

versioning

– Separate versions for firmware, software, libraries

– Storing git commits in f/w and software works well

– Xilinx allows for same bitfile from same files

• routing/placement hash/seed is based on files included in project

• Enforcing this across all project levels is a challenge

• PLC versioning? Software versioning? HDL versioning?


Revision Control

• LBL/LCLS-II and CERN have self-describing

firmware/software

– Register maps in control system and front-ends match

• gzip’d JSON in memory on devices

• CERN’s FESA and cheby

• Acnet database

• TFTP/remote image loading is crucial

– Limit wear on flash (limited write) devices

• Remote/central logging is crucial

– See above

• Verification and Continuous Integration are valuable tools


What is plan for future?

• Responsibility of each department?

• High-speed data and diagnostics

– artDAQ? FTP? Something new?

– DAQ analysis and storage

– High-speed means what for each system?

• Data lifetime?

• Scope capabilities

• Event identification

• Design lifetime

– 20 years? 5 years? 1 year?

• Tool availability and automation

– Build system for non-experts

– Nova replacement? Command-line functionality?


Looking Forward

• What will be available in 2 years? 5 years? 10 years?

• Need a long-term plan

• Due to few resources at developer level, really need process

and procedure directed from managment

• Hardware - should there be a standard developed across

groups?

– Shared FPGA/Processor boards?

– Crate standards?

– Physical interfaces?


• Designed for cell base

stations, radar, military

• Single-chip includes all

necessary for an RF system,

but might not meet

performance requirements of

modern accelerators.

• Separate application and

real-time processors

• Shared memory makes for

very powerful, high-

bandwidth toolset

Xilinx Zynq Ultrascale+ RFSoC

9/27/201813 LLRF Embedded Development

https://www.xilinx.com/products/silicon-devices/soc/rfsoc.html

Conclusion

• Concerns for future

– DAQ and timing/precision

– Data retention and high-speed data management

– Revision Control and Build Environment are significant

concerns

– Ease of integrating in to control system: shared protocols,

development architectures, etc

• Need a multi-year, division-wide path forward

– While still supporting legacy systems

• Thanks!


Extra Slides


Switching to Fiber

• Fiber benefits outweigh slightly increased cost

• Decreased power

• Higher bandwidth

• Low cost per bandwidth

• Relative ease of implementation given available tools


Communication Proposals

• 9.027 MHz RF synchronous frame rate

– Local clock source needed on each device

• 1300 MHz harmonic as data rate

• Need known frame size (ie 128-bit)

• Previous State, Events, Beam pattern for next group

• Single-source star

• Send as timing/events -- Return line as MPS

– Data concentrators?

• Throw out proposal

• 3 links – DAQ, timing/MPS, control



Intel Cyclone V and Arria 10


SoC MFC Test DAQ System


Courtesy Ed Cullerton

Arm Product Line


https://www.silabs.com/documents/public/white-papers/Which-ARM-Cortex-Core-Is-Right-for-Your-

Application.pdf

Microsemi SmartFusion2


Single Event Upset Immune design

LPC17xx

Kinetis K60

NXP

9/27/201823 LLRF Embedded Development

i.MX

Why not Pis?

• The question is really: what does real-time mean?

– Bucket-level resolution and repeatability?

– TCLK decoding accuracy? Event decoding accuracy?

• What runs at each layer?

– Firmware/embedded RTOS

– Software/application layer

• Lifetime and long-term support

– Open Hardware

• Interconnect standards

– SPI + I2C support? (1 SPI, 1 I2C, 1 UART)

– Standard interconnects?

– Bus/backplane protocols?


DSP

• LLRF requires higher-performance DSP and DAQ

• Newer FPGAs are offering dedicated floating-point blocks

– Require use of special tools or overloading standard operators

• Integrated simulation and verification environment being

more important

• Collaboration requires the ability to transfer IP easily


SoM Benefits


https://www.intel.com/content/www/us/en/programmable/products/soc/ecosystem/system-on-modules.html

Linux (and variants)

• Linux w/ preemptRT

• RTAI (Linux as task)

• NI Real-Time Linux (dual-schedulers)


RTOS

• FreeRTOS -- https://www.freertos.org/

• Contiki (IoT) -- http://www.contiki-os.org/

• MicroC/OS-IIx

• Mbed (IoT)

• RTEMS

• vxWorks

– Now a Linux variant

• MQX


https://www.freertos.org/

http://www.contiki-os.org/

MPC603e


MPC750


MPC7457


Processing Models

• IOC+Epics

– “Dumb” front-end with managing IOC

– Subscriber model

• Crate processors

– Slot 0 manages all cards in crate. Advanced data processing

happens ???

• NADs

– Multiple models available

• Acnet

• Requirements on who does each portion is not defined

– Controls communication (MOOC(AN), libacnet, erlang, PC)

– Debug communication (Labview, Python)

– Front-end intelligence


Documents

LLRF Embedded Development · 2018. 9. 28. · –DAQ analysis and storage –High-speed means what for each system? •Data lifetime? •Scope capabilities •Event identification