Consorzio RFX – Padova, February 7 th 2011 RFX-mod Programme Workshop 2011 – Padova, February 7 th 2011 RFX-mod Programme Workshop 2011 RFX-mod Feedback

Consorzio RFX – Padova, February 7th 2011

RFX-mod Programme Workshop 2011 – Padova, February 7th 2011

RFX-modProgramme Workshop 2011

RFX-modFeedback Control System

Upgrade

G. Manduchi, A. Barbalace


Current Control System Architecture

MHD Mode Control

Plasma Position Control

Toroidal Field Control


Main Features of the Current System

Distributed• To account for different location of sensors

and actuators;• to distribute computing load.

• Advantages: computing load can be spread among computers. Sensor and actuators can be dislocated around the experiment.

• Drawbacks: network connections introduce delays in system response. The system must be pipelined to achieves higher throughput.

Modular• The same basic software used on all

nodes;• only control algorithms are changed.

• Advantages: modularity proved to be a successful approach.

• Drawbacks: a more comprehensive software Framework is required to handle complex systems

Configurable• Algorithm updates and IO upgrades can be

fulfilled by editing a C header file (recompilation is required).

• Advantages: easy configuration proved to be a key factor in reducing development time

• Drawbacks: reconfiguration need recompilation.


Critical Factors

Latency• Current overall latency is around 1.5 ms

(between 3 ÷ 4 sampling periods).• This represents a critical factor in quality of

control sometimes leading to instabilities.

Sampling frequency• Current sampling frequency is 2.5 kHz. • A higher sampling rate improves the quality

of integration/derivation.

Computing Power• Currently only most significant modes are

considered.• Operations such as sideband correction and

sensor radius extrapolation are highly computing-intensive and require more power

Simulation• Currently it is not possible to inject a

dummy signal to test the control system.• The possibility of dry runs of the system,

automatically performed before the shot, allows the detection of possible malfunctions which may affect plasma performance and machine integrity.


New Control System Design

from Distributed processing..

switch

..to Centralized processing• To account for different location of sensors

and actuators.• Sensors and actuators are still displaced

around the system;• PCIe Bus Extenders (optical fiber or copper

cable) will provide direct connection to the central processing (no Reflective Memory).

• To distribute computing load. • Multicore/processor technology provides all the required computing power;

• Modularity is retained: the work carried out by the former embedded computer it is now executed by a dedicated processing core.

processor

switch

There is no latency due todata distribution over a network


New Control System Architecture


Evaluated Hardware Details

• We have evaluated National Instruments PXI-6723; a 13bit multi Digital to Analog converter with 32 analog outputs.

• Each reference generation node is made up with a PXI rack hosting 3 PXI-6723 boards. Every output is used.

• Data generation completes after 60us (Intel QPI bus) or after 120us (Intel FSB bus).

• Data transfer is governed by the processing core via Programmed I/O.

• We have evaluated National Instruments PXI-6255; a non simultaneous sampling multiplexed 16bit ADC with 80 analog inputs.

• Each acquisition node is made up with a PXI rack hosting 3 PXI-6255 boards. Every board is setup to scan 64 channels periodically.

• A scan completes in 90us and data is ready to the central processing after few microseconds.

• Data transfer is mastered by the devices via Direct Memory Access (DMA).

Data Acquisition Waveform Generation


Non Simultaneous Sampling

In Real-Time• The reconstruction will lead to a quasi

exact low-frequency signal and a less precise high frequency signal.

• The acquisition downgrade to 14bit. • Data is filtered via a Fractional Delay FIR

implementing Lagrangian interpolation

Is it possible to realign the samples?

After the Pulse (offline)• The reconstruction will lead to a

theoretically exact signal.

• All 16bit could be restored.• Data is convolved with a sinc function.

Yes, because channels are sampled at known instants.

Frequency spectrum of a sample DFLU signal

..but multiplexing introduces alsosettling time issues

due to switching between different channels


Simultaneous Sampling

Solutions alternative to evaluated hardwareNational Instruments

• Data bus PXI Express (PCIe) in PXI rack;• PXIe-6358 16 input channel boards, 16bit;• Each 192-channel acquisition node may be

implemented with a PXIe-1075 chassis with 12 (18 available) PXIe-6348 boards.

• Data bus PCI Express in ATCA rack;• Custom devel 32 input channelboards, 24bit;• Each 192-channel acquisition node may be

housed in an ATCA shelf with 6 custom developed ATCA-MIMO-ISOL boards.

IST-IPFN

Both focus on high data throughputIPFN solution is designed for high availability

(five nines)


New Software Framework

What about the previous Framework?• The software framework developed and currently used in RFX did a good job.• However, it has been developed with a well-defined architecture in mind, so it is not so easily portable to new architectures

and platforms.

Joint efforts in fusion community gave rise to..• General frameworks for real-time control, notably PCS and MARTe.• PCS has been developed at DIII-D and is used in a few machines (e.g. DIII-D , MAST, EAST).• MARTe has been developed at JET (by the PPCC group in collaboration also with RFX) and is also

used in few machines (e.g. JET, FTU, ISTTOK, COMPASS).

MARTe has been chosen!


New Operating System

The current system uses Wind River VxWorks

• Rugged real-time operating system;• Some important bugs in TCP/IP

communication;• Very expensive.

Linux is becoming an interesting candidate for the new system

• Soft real-time system;• Real-time extensions of Linux make

it a hard real-time low-latency operating system;

• Linux’s official RT patches are good enough to make it hard real-time OS.

http://www.xenomai.orghttp://www.kernel.org

http://rt.wiki.kernel.org

We have chosen Linux with itsofficially released RT patches

http://www.rtai.org

http://www.windriver.com


Pipelined Multicore Execution

Real time computation for MHD control is carried out in a pipelined organization with three stages: Data Acquisition, Control Computation, and Reference Waveform Generation.

New cores can be added to provide additional computation in the case it can be performed in parallel.

acqClock 0

CPUcore 0

CPUcore 1

CPUcore 2

CPUcore 3

CPUcore 4

CPUcore 5

time

...

...

...

ADCstage

algorithmicstage

DACstage

SoftwarePIPELINE

data out 0 data out 1

acqClock 1 acqClock 2 acqClock 3


Realtime Performance

Prototype running:• A simple two stage pipelined I/O cycle

developed in MARTe;• RedHat Enterprise Linux 5.5 with RT

patches from CERN (Scientific Linux Distribution) kernel 2.6.24.7-149.el5rt;

• Multicore Intel Xeon processor E5500 @ 2GHz (4 HT cores x86 64bit) QPI bus;

8kHz data acquisition Clock Frequency

Prototype setup:• A PXI input rack hosting two NI PXI-6255

acquiring 64*2 input channels;• A PXI output rack hosting two NI PXI-

6723 generating 32*2 output references.

MeasuredOverall latency150us (average)200us (WCET)


racks Qty Unit Price Total network Qty Unit Price Total IO Qty Unit Price Total cabling Qty Unit Price Total ComputingQty Unit Price TotalPXI-1033 4 1110 4440 PXI-PCI8336 1 3079 3079 PXI-6255 9 2271 20439 SHC68-68-EPM9 130 1170 server 1 4336 4336PXI-1031 1 989 989 0 PXI-6723 6 1362 8172 SHC68-68 9 90 810 0rack mount 5 161 805 0 0 SH68-c68-s 12 100 1200 0

0 0 0 0 06234 3079 28611 3180 4336

racksnetworkIOcablingcomputing

total

racks Qty Unit Price Total network Qty Unit Price Total IO Qty Unit Price Total cabling Qty Unit Price Total ComputingQty Unit Price Total0 budgetary 1 5000 5000 VGX 9 8700 78300 SHC68-68-EPM9 130 1170 CPU 5 4000 200000 0 MPV914 6 4000 24000 SHC68-68 9 90 810 0

VME 5 4500 22500 0 FPDP 5 4000 20000 SH68-c68-s 12 100 1200 00 0 MPV922 5 1000 5000 0 0

22500 5000 127300 3180 20000

racksnetworkIOcablingcomputingRTOS

total

1273003180

2000020000

197.980

62343079

2861131804336

45.440

225005000

IO62%

cabling7%

computing10%

racks14%

netw ork7%

IO64%

cabling2%

RTOS10%

computing10%

netw ork3%

racks11%

Costs Comparison

NI Prototype Cost

Old System Cost


Commissioning Plan (1 of 2)

Mailstone 1 - Deadline 30 March 2011 -Test of NI Prototype in RFX Experimental Campaign

Objectives• This installation will assess the usability of the evaluated hardware for data

acquisition (checking cross-talk and cross-channel switching problems).• Investigate the possibility of numerical integration (both integrated and non

integrated signals will be acquired).

Procedure• A first prototype system is being installed for acquiring poloidal field values

Requirements• Front-end duplication

Costs• All the computing and acquisition components are available in house.

– The support computer is recycled (there are no real-time computing to implement on)

– The IO boards were used in the prototype• Front end boards (i.e. signal splitters) must be designed and made.


Commissioning Plan (2 of 2)

Mailstone 2 - Deadline TO BE DEFINED -MHD Feedback System Commissioning

Objectives• Run the new MHD Control System in parallel with the current System.• The new system will not interfere with RFX operations.

Procedure• The new system will be installed in parallel to the current one using signal

splitters.

Requirements• Multicore server;• IO boards, cabling and racks;• Development of few software components.

Costs• Multicore server;• IO boards, cabling and racks;• Front end boards (i.e. signal splitters).


Conclusions

• The new Feedback Control System of RFX proposes an innovative hardware architecture in Fusion Devices; distributed nodes communicating via a real-time network (ATM, Ethernet, Reflective Memory) moved to multiprocessor cores and communication is achieved on the same RAM via shared memory.

• COTS solutions for computing (multicore servers) ADC and DAC (PXI-based solutions with bus extenders) proved cost-effective and reliable.

• Linux is becoming a very good candidate for real-time feedback control systems in fusion devices.

• MARTe proved to be an extremely flexible environment to develop Hard Real-Time Control Systems.

processor

switch

switch

Documents

Consorzio RFX – Padova, February 7 th 2011 RFX-mod Programme Workshop 2011 – Padova, February 7 th 2011 RFX-mod Programme Workshop 2011 RFX-mod Feedback