SLAC 1st April 2010

Paul Scherrer InstitutErnst Johansen

The „Pie in the Sky“

Agenda

Unified Processing Architecture

MotivationHistoryTechnologyFeasibilityProposalCostDiscussion

ma.a3uUs

Motivation

PSI Motivation• We will soon operate four Machines (2x proton, 2x electron)• Maintenance will become a huge Challenge• Focus on synergies between all machines• We need very long lifetime for the control solutions

Personal Motivation• Bring in +15 years of experience in designing Industrial Control Systems• I like do design systems on a “decent” platform

Unified Processing Architecture ?

GPACPPC440, OS = ?

IOCPPC, Intel

VxWorks, Linux

VPCPPC405, DSP

No OS, Xilkernel

UPACommon CPU, OS

Common FW

SLS, HIPADSP, uC

Use Cases

PLCSiemens, Rockwell,

Wago,Beckhoff

Platforms

Religion….They somehow do it all… don’t they ?

Its very expensive to change themYou need alliances to be strong…

TechnologyAre there any new technologies around ?

How to apply these to the VME64x platform ?

Significant ImprovementsPCI Express

FMC (VITA57.1-2008)Larger, low-power FPGA

How to apply PCI Express, FMC and larger FPGA’s to VME64x ?

Design – VME Bottleneck

CPU

FPGA FPGA FPGA

≈250MB/s(2Gbps)

VME64x

CPU

DSP

Design – Additional Bandwidth at Low-Cost

CPU

FPGA

PCIe 4x8GbpsDuplex

GEthernet 2x2GbpsDuplex

SFP 4x15GbpsDuplex

VM

E64

x C

ard

Design – Scalable Bandwidth

CPU

FPGA

CPU

FPGA

CPU

FPGA

N x 8Gbps

10G Ethernet Switch

PCI Express, Fiber Optics

N x 2Gbps

N x 15Gbps

COTS

N = 1 .. 21

VME

Improved P0 – VME64x Compatible

Improved P0• 2mm Legacy Compatible• High-Speed Links• SFP 7Gbps

SPF

PCIe

VersaLink

FMC – High-Speed I/O

Defined PSI standard, but…

VITA57.1-2008 FPGA Mezzanine Cards is here• Supports the PSI Use Cases• Broad industrial acceptance• 69 x 76.5 mm2 (small)

Examples• 4DSP

–8 channel ADC 250Msps @ 14-bit–4 channel ADC 125Msps @ 16-bit–1 channel ADC 5Gsps @ 8-bit–4 channel DAC 1000Msps @ 16-bit

• Curtiss-Wright–4 channel DAC 500Msps @ 16-bit

Proposal• VITA57.1-2008 compatibility• Support for extended PCB size where required

Hardware Design

Are there any reusable existing products on the market? What has to be extended ?

IOxOS

IOxOS• Swiss Company Gland by Geneva• Experts VME, PCIe, Board Design

Products• VME64x High-Performance VME64x Bridge• Design Kits Virtex-5, Virtex-6

Status• NDA Signed• Offer Design UPA• Sell Into research community

PSI, 01.04.10

CPU

Intel Atom• No ECC

AMCC APM83290• ~21W

Marvell MV78200• ARM, 6W, immature Linux

Freescale P2020• Dual 1.2GHz PowerPC , <8W

Freescale P4040 • Quad 1.5GHz PowerPC, ~20W

Freescale P4080• Octal 1.5GHz PowerPC, ~ 30WBroadcom, NetLogic, Cavium…

P2020RDB

Design

IOxOS TOSCA II Design Kit• Based on mature TOSCA product• Designed for Reuse: VME64x, PCIe Bridge, IDMA, DDR3• Innovative PCIexpress 2.0 Network-on-Chip Architecture• Supports Virtex-6 CXT – Cost efficient 40nm process• VHDL Simulation Framework• Linux Drivers and Test Framework• Excellent documentation• All VHDL source and hardware design files available !!

Freescale P2020RDB Design Kit• P2020 Dual Core CPU• ECC on memory• 2x Ethernet, USB 2.0, UART, NOR and NAND Flash• Long term availability• Low power 45nm process

Unified Processing Architecture – IOxOS

S YS M E M (+ E C C )D D R 3 x8

1 /2 /4 G B y te s

F la s h E P R O MN A N D 8 G BN O R 128M B

D D R 3x1 6

D D R 3x1 6

V M E _ P 03 M U H M

V M E _ P 13 /5 R o w s

V M E _ P 23 /5 R o w s

S p a rta n-6 F P G AL X 4 5

F F 4 8 4

V M E 6 4 XE le c tric a l I/F

L V TTL

E th e rn e t P H YR G M II

R S 2 3 2 U S B 2 .0

PCIeEX T

P C I E xp re s s G e n2S w i tc h

P E X _ 86 1 8

P 2 0 2 0D ua l C o re P P C 1 .2 G H z

F M C C o n n e c to r # 1

XM C J n 1 5

P M C J n 1 1

P M C J n 1 2

PCIe - PCIP EX _ 8 1 1 2

P I3 P C IE2 2 1 5

P M C J n 1 4

F M C C o n n e c to r # 2

XM C J n 2 5

P M C J n 2 1

P M C J n 2 2

PCIe - PCIP EX _ 8 1 1 2

P I3 P C IE2 2 1 5

P M C J n 2 4

[x4 ]

[x4 ]

[x4 ]

[x4]

[x1 ]

[x1 ]

[x1 ]

[x1 ]

[x1 ]

[x1 ] [x1 ]

[x1 ]

[x1 ] [x1 ]

[x1 6 0 ]

[x6 4 ]

[x6 4 ]

[x6 4 ] [x4 0 ]

[x6 4 ][x4 0 ]

[x1 0 0 ]

[x 8]

[x 8]

[x4 ][x4 ]

X i linxV irte x-6 F P G A

C X T 1 30 TF F 1 1 5 6

5* Q uad_G TX

[x4 ]

[x4]

G PIO

S TA TIC

[x1 6 0 ]

TO

SC

A I

IH

W+

FW

reu

seP

2020

RD

BH

W+

BS

P r

euse

Unified Processing Architecture – HW Benefits

S YS M E M (+ E C C )D D R 3 x8

1 /2 /4 G B y te s

F la s h E P R O MN A N D 8 G BN O R 128M B

D D R 3x1 6

D D R 3x1 6

V M E _ P 03 M U H M

V M E _ P 13 /5 R o w s

V M E _ P 23 /5 R o w s

S p a rta n-6 F P G AL X 4 5

F F 4 8 4

V M E 6 4 XE le c tric a l I/F

L V TTL

E th e rn e t P H YR G M II

R S 2 3 2 U S B 2 .0

PCIeEX T

P C I E xp re s s G e n2S w i tc h

P E X _ 86 1 8

P 2 0 2 0D ua l C o re P P C 1 .2 G H z

F M C C o n n e c to r # 1

XM C J n 1 5

P M C J n 1 1

P M C J n 1 2

PCIe - PCIP EX _ 8 1 1 2

P I3 P C IE2 2 1 5

P M C J n 1 4

F M C C o n n e c to r # 2

XM C J n 2 5

P M C J n 2 1

P M C J n 2 2

PCIe - PCIP EX _ 8 1 1 2

P I3 P C IE2 2 1 5

P M C J n 2 4

[x4 ]

[x4 ]

[x4 ]

[x4]

[x1 ]

[x1 ]

[x1 ]

[x1 ]

[x1 ]

[x1 ] [x1 ]

[x1 ]

[x1 ] [x1 ]

[x1 6 0 ]

[x6 4 ]

[x6 4 ]

[x6 4 ] [x4 0 ]

[x6 4 ][x4 0 ]

[x1 0 0 ]

[x 8]

[x 8]

[x4 ][x4 ]

X i linxV irte x-6 F P G A

C X T 1 30 TF F 1 1 5 6

5* Q uad_G TX

[x4 ]

[x4]

G PIO

S TA TIC

[x1 6 0 ]

High-ReliabilityDual Core IOC ≈250USD

Fast VME64x ≈100USD

FMC (Digitizers)XMC (PCIe)PMC (PCI)

PCI Express (P0) High-End Workstation

Large and fastFPGA ≈300USD

EthernetRemote debugging& download

P2 Legacy Support3.3V I/O FPGA

IOxOS Thermal Design

RearIO

PMC / XMC / FMC

PMC / XMC / FMC

Cooler

Software Design

Are there any reusable existing products on the market? What has to be extended ?

3S - CoDeSys SP V3 (OEM)

CoDeSys IEC61131-3• De-facto Industrial Standard• Used by several hundred companies• Supports all IEC61131-3 Languages• Ported to Linux • Build-In EtherCAT Stack

Status• Offer Porting to UPA

EtherCAT – Medium-Speed I/O

Bus Master• Based on standard Ethernet• Very low protocol overhead

Key Data• Performance

–1.000 Digital-I/Os in 30 µs –200 Analog-I/Os (16 Bit) in 50 µs –100 Servo-axis in 100 µs

• Suitable PCI / PMC replacement• Suitable IP-module replacement

Technology• http://www.beckhoff.de

Design

Open Source Real Time• Linux with PREEMPT_RT patch

P2020RDB• Linux BSP, SMP configuration

IOxOS TOSCA II• Linux Drivers• Test Framework

EPICS• Linux Port

CoDeSys• IEC61131-3 with EtherCAT Stack• Linux PREEMPT_RT

Capabilities - Linux RT on P2020

Similar to Emerson MVME4100 performance, but has additional• Second CPU core for real-time DSP

Real-Time PREEMPT_RT on P2020 in SMP-Mode (One operating system – two cores)• Patch is easy to apply and well supported• Moved all Processes to PPC0• Moved all Interrupts to PPC0• 100% PPC0 load (cyclictest)• Executed EPICS on PPC0• Executed cylictest on PPC1

PPC1 Latency Results• Min 5us Avg 5us Max 40 us

ConclusionsGuarding one CPU enables hard real-time behavior running EPICS in parallel.Preliminary results shows that 10kHz @ 50% CPU load is feasible.

Unified Processing Architecture – Software

EPICSChannelAccess

EPICSIOC

CoDeSysIEC61131

MatlabMC

IOxOSTOSCA II

EtherCATI/O

RF FMC

VME64x

IDMADRAM

LegacyIO

Potential for Cost Reductions

Cooperate with Industry to find a Common PlatformSell Platform into different Markets

PECTRON

PSI IOxOS

SwissFEL RF Applications

Process Automation

PECTRON – Power Electronics Control

Status• Swiss start-up company• Share NRE• Sell into Power Electronic Market

Crate• ELMA 2-Slots

PLC• CoDeSys SoftPLC• IEC61131-3 Programming

EtherCAT• Beckhoff

VITA57.1-2008 FMC – Partner

Strategy• Cooperate with Industry• Utilize latest available technology

Curtiss-Wright• NDA• Next Generation FMC• ADC 250Msps @ 16-bit ???

4DSP• Next Generation FMC• 6-channel ADC 250Msps @ 16-bit ???

Conclusions

We need a long term strategy for our control systems

UPA enables one single framework for PLC, IOC , Medium-Speed- and RF applications

VME64x is the most stable specification and compatible with SLS and HIPA Retrofit

Higher integration we can considerably reduce the cost of VME64x

The UPA has low component count and ECC – suitable for high-reliability applications

With UPA we can achieve near to zero long-term royalties

With strategic sourcing a lifetime of 20 years is feasible (if we want to)

We get complete design in source code – including hardware and VHDL!!!

There are potential partners to get into a win-win situation!!!

The „Pie in the Sky“