Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Chris ShentonJodrell Bank ObservatoryUniversity of Manchester
STaN CoDR June 2011
Signal Transport And Networks
1
Custom or Proprietary Solution
What does that really mean?
• Actually a custom implementation built from commercial standard building blocks
• No new esoteric technology involved
• Aiming for the removal of unnecessary cost
• Full engagement with Industrial Partners
2
“Well of course we all know that a custom solution is always the most expensive...”
System Topology
• In SKA Phase 1 the majority of elements are within AA lo stations; 250 dish Rx's vs 560,000 AA RX Elements
• Majority of elements within 5Km of centre
• Majority of elements separated by only a few metres
• Maximum of 100Km reach required in Phase 1
• Dishes grouped in small 'clumps' with AA station close
• Phase 1 deployment 2016
3
System Characteristics
• Asymmetric data flows
• Tolerant of bit errors within data
• Synchronous data transfer required
• Tolerant of high latency
• Many 'end points' in the networks
• Fine Grained at element level (10G)
4
What Kind of Networks Terminate at an End Point
• Data transmission (Digital, Proprietary)
• Reference Phase transfer (Analogue, Proprietary)
• Synchronisation (Digital, Proprietary)
• Monitoring & Control (Digital, COTS, Ethernet)
5
Physical Realisation Concept
7
• High Volume Manufacture > 0.5 Million units required...
• Design for low Cost
• LRU for simple maintenance and installation
• Reducing number of connectors improves reliability
• Not high tech
• Good enough... not as good as possible. It’s a Mini not a Rolls Royce
• Same design works for dishes
Integrated Digital Receiver Concept
Xilinx ARTIX Class FPGA
PHY Interface
PCS Interface
ADC Interface
LOGIC Core Interface
Interfaces
ADC 1:4 R
4x 4b
Deserialiser
1:6
n
n+1
I IF in
dc - 1GHz
800mV
Flogic = 100MHzFdeser = 600MHz
4x 4b
LVDS @
Fsample/4
9.6Gb/s Ch
DSP
4x 24b @
Fdeser/6
96b @
Flogic
FramerPCS
64b66b10.3125Gb/s
96b @
Flogic 64b @
156.25MHz
XGMII
66b @
156.25MHz
10Gb/s 10.3125Gb/s
Q IF in
dc - 1GHz
800mV
1x 1b CML @
10.3125GHz
n+2
n+3
ADC 1:4 R
n
n+1
9.6Gb/s Ch
DSPFramer
n+2
n+3
Fsample = 2.4GHz
4 bit sample word
4 Channel
Parallel
Optic
Transceiver
VCSEL
Array
850nm
300m
Timing Offset
Subsystem
M&C
Subsystem
uC 10/100 Eth
MPO
Fibre
Cable
4 x
OM3
Data
Aggregation
Frame
Sync
TxRx 50MHz
AnaloguePLL
Clock Synthesis
Subsystem
Digital ReceiverBoad
Low Cost
XTAL
PCS
64b66b10.3125Gb/s
10.3125Gb/sPCS
64b66b
Metadata
8
How to Reduce Costs
• Electrical & Logical Aggregation maximises the utilisation of available bandwidth
• Sharing of high cost Physical Resources such as interconnect and FPGA/ASIC platform
• Optical Aggregation via WDM maximises utilisation of physical fibre
• Standards based technology allows interception of commoditised technology
• Reduce the number of high cost terminations
• Reduce data volume by 'upstream' processing (this also has the happy side effect of simplifying the station and core processing)
• Reduce number of cables at Station and Core processing (1120 4x,600 12x)
9
System With Aggregation
MPO 12x
Station
Processing
~ 1:10
Data Reduction
Digital
Receiver
MPO 12x
12 x 100Gb/s
1.2Tb/s1,120 x 10Gb/s
10G-BASE-LR
(10Km 1550 SMF)
MPO 12x
760Ms/s
4 bits per sample
x 2 polarisations
6.06Gb/s
Aggregating
or 'Tile'
Processing
~ 1:10
Data Reduction
MPO 4x
Digital
Receiver
MPO 4x
10G-BASE-SR
(10m 850nm OM3)
Digital
ReceiverAggregating
or 'Tile'
Processing
~ 1:10
Data ReductionDigital
Receiver
MPO 4x
Digital
ReceiverAggregating
or 'Tile'
Processing
~ 1:10
Data ReductionDigital
Receiver
MPO 4x
x 1,120
x 10
x 10
x 10
MPO 4x
MPO 4x
MPO 4x
MPO 4x
MPO 4xMPO 4x
MPO 4x
MPO 4x MPO 4x
MPO 4xMPO 4x
MPO 4x
MPO 4x
MPO 4x
10G-BASE-SR
(100m 850nm OM3)
Core Processing
MPO 12x
MPO 12x
MPO 12x
12 x 100Gb/s
1.2Tb/s
x 50
MPO 12x
MPO 12x
MPO 12x
12 x 100Gb/s
1.2Tb/s
From other
Station
Processors
10
System Without Aggregation
MPO 12x
Station
Processing
~ 1:100
Data Reduction
Digital
Receiver
MPO 12x
12 x 100Gb/s
1.2Tb/s
10G-BASE-LR
(10Km 1550 SMF)
MPO 12x
760Ms/s
4 bits per sample
x 2 polarisations
6.06Gb/s
Digital
Receiver10G-BASE-SR
(100m 850nm OM3)
Digital
Receiver
10G x 11200
MPO 4x
MPO 4x
MPO 4x
MPO 4x
MPO 4x
MPO 4x
Core Processing
MPO 12x
MPO 12x
MPO 12x
12 x 100Gb/s
1.2Tb/s
x 50
MPO 12x
MPO 12x
MPO 12x
12 x 100Gb/s
1.2Tb/s
From other
Station
Processors
x 11200
11
Strategy for getting to a Cost Effective System Solution
• Opportunities for resource sharing across multiple logical networks
• Use of appropriate technology for the application
• Obtain reduced costs by economies of scale
• Highest tech solution not necessarily appropriate
• Targeted use of high cost resources
12
Aggregation Processor Concept
Xilinx VIRTEX 7 Class FPGA
Aggregation Module Board
Analogue ConditioningBoard + Optical Modules
Signal
Processing
&
Beamforming
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
SPF
Rx
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
Agg
Block
MPO
Fibre
Cable
4 Way
4 Channel
Parallel
Optic
Transceiver
VCSEL/PIN
Array
M&C
10/100 Ethernet
Hub
M&C
Embedded
Proc
Digital
Offset
+
GPS
timecode
GPSModule
Analogue
Phase Signal
Conditioning
Board
REF RTN
50MHz Analogue
Phase Return
Digital Phase
Processor
Analogue Phase Lock
and 1:10 Fanout
REF RTN
1KHz x 10
Digital Phase
Pairs
50MHz Analogue
Phase Ref
High Quality
LO
13
Electrical Aggregation and Tile Processing
• Majority of terminations short hop (560,000 connections less than 10m)
• Creation of sub groups or tiles allows electrical aggregation using low cost silicon ($10 class FPGA not $1000 class FPGA)
• Element Volumes make a simple ASIC a further cost reduction opportunity (cheap and cheerful 90nm technology will do this job... no need for esoteric 22nm...)
• Distribution of the processing into the end points and aggregators enables the maximum utilisation of logic resources.
• Major cost driver is IO in the form of the network termination
• Appropriate but simple processing at the element or tile level becomes effectively 'for free' as part of FPGA or ASIC logic
• Channelisation and forming of partial beams reduces upstream data rate
14
Physical Layer• 4x/12x/24x 10G MPO Structured Cabling System
• VCSEL Driver Array
• 40G/120G/240G Capacity over fibre pairs
• 25Gb/s VCSEL TXCR’s in development pushing capacity to 100G/300G/600G
• 850nm MMF (1310nm and 1550nm SMF products in development)
• OM3 Cable for SR Links
• IEEE 802.3 10G-BASE-SR/LR PHY specification
• Driven by data centre patch interconnect market
• Target $0.25 per gigabit per termination ($20 per 4x)
• Power Consumption 700mW per end 40% Tx/60% Rx per 4x (4 x 10G)
• MSA (SFP+) plug-able interfaces high cost and not getting cheaper quickly
15
Transport Layer
• Proprietary
• Simple Synchronous (Plesiochronous) framing required
• Unidirectional
• Minimal transport overhead required probably just a CRC
• No retransmission, use it or lose it
• Half as many driven Tx nodes in the data backhaul
17
• ‘Ethernet’ UDP/TCP/IP
• Could be used for framing...
• No real benefit
• Costs Tx power
• ‘Only really required for M&C network.
• M&C Network over TCP/IP can easily be carried within synchronous frames as a VC
• Could implement partial Layer 2-4 stack (ARP, Ping etc) at the endpoint... but why bother when it isn’t a switched network?
Can We Use 100Gb DP-QPSK Technology to Reduce Costs
• Depends on cost of transceiver technology - 10G SFP+ still expensive and cost reduction curve looks shallow
• Could reduce number of connections from Station to Core processing
• Still immature technology
• Probably required in SKA 2 for dense AA
18
Use of DWDM
• Optical Aggregation sub assemblies are expensive
• Optical MUX’s and DeMUX’s at 50GHz channel spacing very expensive
• Relationship between number of EDFA’s and no of channels per fibre needs analysing for best cost tradeoff
• Point to point many core fibre is becoming more widespread
• Driven by ‘Fibre to the Home’
• Use of DWDM could be cost effective in long reach links with many stations
• Deploy where physical fibre costs become significant proportion of system cost
• Cost Tradeoff analysis needs to be done to find optimum solution
• Potentially more significant as the number of remote stations increases in Phase 2
19
Interface Standardisation
• ADC ➡ Logic
• LVDS (or low power derivative)
• Lower power solution required for dense arrays
• Electrical TXCVR ➡ Optical Module
• XGMII (or CGMII if 100G arrives)
• Plug-able Module?
• SFP+ is still very expensive
• SFP+ Cost includes high percentage of housing and mechanical assembly
• Integrate the E/O and O/E into the Digital Receiver subassembly
• Reduces cost
• Improves Reliability
20
Cost Breakdown Aggregated
21
Based on 10:1 Aggregation
A 10 Element Group Consists of;
10 x DR FPGA/ASIC ($10)10 x 10m Terminated Optical 4x MPO Cable Assembly ($30)1 x AGG FPGA/ASIC ($100)1 x 100m Terminated Optical 4x MPO Cable Assembly ($120)
Total Cost per Group
100+300+100+120 = $620 per group of 10 elements
A 11200 Element Station Consists of;
1120 x 10 Element Groups ($620)12 x 100m Terminated Optical 4x MPO Cable Assembly ($120)
Total Cost per Station (at station inputs)
1120 x 620 = $ 694400 per station
Total SKA 1 Cost
50 x $700K + 600 x $4K = $37.4M
Station to Core Connectivity Cost12x MPO Termination $60Fibre 5Km SMF in 192 cores @$5 per metre24 Cores $3125 per 12x ConnectionFibre termination (Shared Fibre Cable) $800
Total is;
$60 + $3125 + $800 = ~$4000 per connection
600 connections at the core; (12 x 50) 600 x $4000 = $2.4M
Cost Breakdown Direct Connection to Station
22
No Aggregation
A Single Element DR Consists of;
1 x DR FPGA ($10)1 x 100m Terminated Optical 4x MPO Cable Assembly ($120)
Total Cost per Element
10 + 120 = $130 per element
A 11200 Element Station Consists of;
11200 x Single Elements ($130)
Total Cost per Station (at station inputs)
11200 x 130 = $1.456M per station
Total SKA 1 Cost
50 x $1.456M + 600 x $4K = $75.2M
Station to Core Connectivity Cost12x MPO Termination $60Fibre 5Km SMF in 192 cores @$5 per metre24 Cores $3125 per 12x ConnectionFibre termination (Shared Fibre Cable) $800
Total is;
$60 + $3125 + $800 = ~$4000 per connection
600 connections at the core; (12 x 50) 600 x $4000 = $2.4M
Power Requirements
23
Aggregated
11200 x (1.4W + 5W) = 72KW1120 x (1.4W +20W) = 31KW
Total 104KW per station
Non Aggregated
11200 x (1.4W + 5W) = 72KW
Total 72KW per station
Risks• Compromises are required in order to simplify and cost reduce the system
• Hard to pin down spec from scientists, let alone impose constraints
• Cost constraint now hard to ignore
• Data Rate estimates are questionable for AA technology
• LOFAR has data growth in beamforming to maintain S/N performance
• is the required station data rate actually bigger than current spec states...
• Proposal based on emerging technologies - pricing is fluid
• We really need a cheap 10G-BASE-LR (10Km) option, 1550nm VCSEL based technology not currently mature
• Work with technology providers, use their expertise to get real world solutions
• Subject to wider market forces
• Fallback implementation schemes look expensive
• At the mercy of interconnect technology ‘futures market’
24
Scaling up For Phase 2
25
• All of the above... but much worse!
• Apply same principles to dense array and focal plane array solutions
• Its a scalability and affordability problem not a technical one
• Dense Array Tile processor for AA-Mid or PAF will look something like this ➔
• Need to find the right level of integration to remove cost and maintain adequate performance
Multi Channel Integrated Digital Receiver Concept aka ‘Tile Processor’
26
28nm class ASIC
16 Element DP DigitalReceiver Module
PCA or MCM
Fsample = 2.4GHz
4 bit sample word
Clock SynthesisSubsystem
Local Clock Subsystem
LO
Timing Offset
Subsystem
M&C
Subsystem
uC 10/100 Eth
Frame
Sync
Subsystem
TxRx 1.25Gb/s
TxRx 50MHz
Analogue
TxRx 100Mb/s
TxRx 100Mb/s
PLLLogic Clock Distribution
8 x 32bit busses
@ 300MHz
8 samples every
3.33ns
ADC
8 channels
2.4Gs/s
4bit word
8 x I IF in
dc - 1GHz
800mV
Ag
gre
ga
tor
4 x
19
.2G
b/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
4x 66b
Fra
me
r
4 x
25
Gb
/s
4x 6
4b
PCS
4x 6
4b
8 x 9.6Gb/s
8 x I IF in
dc - 1GHz
800mV
8 x Q IF in
dc - 1GHz
800mV
8 x Q IF in
dc - 1GHz
800mV
4x 6
4b
Ag
gre
ga
tor
4 x
19
.2G
b/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
4x 66b
Fra
me
r
4 x
25
Gb
/s
4x 6
4b
PCS
4x 6
4b
8 x 9.6Gb/s 4x 6
4b
Ag
gre
ga
tor
4 x
19
.2G
b/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
4x 66b
Fra
me
r
4 x
25
Gb
/s
4x 6
4b
PCS
4x 6
4b
8 x 9.6Gb/s 4x 6
4b
Ag
gre
ga
tor
4 x
19
.2G
b/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
Tx 28Gb/s
4x 66b
Fra
me
r
4 x
25
Gb
/s
4x 6
4b
PCS
4x 6
4b
8 x 9.6Gb/s 4x 6
4b
DSP
8 x 9.6Gb/s
8
x
1:32
Gearbox
Rx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6G
8 x 9.6Gb/s
8
x
1:32
Gearbox
Rx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6G
8 x 9.6Gb/s
8
x
1:32
Gearbox
Rx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6G
8 x 9.6Gb/s
8
x
1:32
Gearbox
Rx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6GRx9.6G
ADC
8 channels
2.4Gs/s
4bit word
ADC
8 channels
2.4Gs/s
4bit word
ADC
8 channels
2.4Gs/s
4bit word
Sampling Clock Distribution
24 Channel
VCSEL Array
28Gb/s
1550
SMF
MPO
Fibre Cable
24 x
SMF
What Happens Next...• Continue Design work on integrated solution
based system reference design
• Architectural proving using 2-PAD and other Array based systems
• Update documentation
• Update the design concept with Station and Core processor Line Interfaces
• Concepts understood and partially developed but additional work required
• Costing of these interfaces
• Establish a real world reference design and a component level design database under way
• We are way past the stage where generic costings are adequate
27
• Continue work on enclosure designs and RFI containment
• Explore cost implications of Analogue Transport to Integrated Tile Processor
• Conventional
• Analogue over Fibre
• Continue to build relationships with key technology suppliers and potential delivery partners;
• Interconnect Vendors
• current work is supported by TE (Formerly Tyco)
• FPGA Vendors (Altera, Xilinx)
Conclusions
• We need to put constraints on the system performance in order to meet the cost targets
• Moving raw data samples over expensive links is expensive and unnecessary
• This is not a datacomms network, we don’t have to deliver unprocessed data
• We need to establish detailed reference designs for major options
• The scientists need to support these engineering activities by analysing the impact of the constrained design and buying into it...
• Working at a theoretical level is now not adequate
• Can we build phase 1?
• Yes, but not without compromises
• Can we build phase 2?
• Yes, but we probably need to move to a high level of integration to make it affordable
• We need to continue to develop real engineering relationships with technology partners
28