PanSTARRS Internal Spec Review 8/5/2003 PanSTARRS Gigapixel Array Controller System Overview Present Status –CPU board –FPGA board –DAQ board Staffing

PanSTARRS Internal Spec Review 8/5/2003

PanSTARRS Gigapixel Array Controller System

• Overview

• Present Status– CPU board– FPGA board– DAQ board

• Staffing

• Infrastructure

• Schedule

Outline


Illustration of Bar Mounting Scheme - 3

OTA flexcables attach to internalmotherboard with amplifiers. A single controller module is shown for reference (1 controller for 4 OTAs). Note: hermetic feedthrough deleted for clarity.


Gigapixel Camera Controller Electronics

• Controllers are mounted behind cryostat.

• 16 controllers per Gigapixel camera.


Baseline Controller Design

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA OTA OTA OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA OTA OTA OTA

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

IOT

A3U

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA OTA OTA OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA

OTA OTA OTA OTA

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

• IOTA3U controller operates 4 OTAs• For a 8 x 8 OTA focal plane, 2 rows of 1 x 8 OTA’s • Each row has 8 IOTA3U controllers.

1Gb ethernet

Power Supply

LAN Switch

Fiber isolated

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

Pixel Server

1Gb ethernet

2-row OTA Controller “Rack”


Data Storage Advances

• Storage Area Networks and Network Attached Storage– Cost effective high performance– Connectivity to “second tier” processors if needed

• Pixel Servers also have built in RAIDs– Lowest cost but performance not benchmarked

Back-end

LAN+SAN

SAN

SWITCH

Second Tier

Processing

Fiber Broadcast


1 x 4 IOTA3U Assembly

HDC

FP

GA

CO

TS

CP

U board

OTA

CPCI backplane

CO

TS

1G ethernet

Flexprint P

CB

Cryostat wall

Signal chains and A

DC

s

Interface bd • Green areas are custom designed

• Blue areas are COTS• 3U hardware could include

connectors, front panels, guide rails, cooling components, chassis parts, etc.

Chassis

Front Panel

Cooling Cooling

Interface Board

Signal chains and A

DC

sS

ignal chains and AD

Cs

Signal chains and A

DC

sS

ignal chains and AD

Cs

FP

GA

FP

GA

CO

TS

CP

U

CO

TS

CP

U

Flexprint P

CB

OTA OTA OTA

CPCI backplane

OTA

Signal chains and A

DC

s

HDC

Module w

all


Mechanical Check 1 x 4

Green crosshatch shows 1 x 4 OTA shadow4 sets of 2mm metric connectors shown

2 are CompactPCI bus standard2 for OTA signal processing

Standard 3U front panel shown in maroon.


IOTA3U Boards

• COTS CPU board – Baseline LINUX based PowerPC– 1 or 2 channel 1G Ethernet fiber PMC

daughterboard– CPCI bus interface

• FPGA3U board– CPCI interface– FPGA +1Ge??– Buffer memory– Data interface to DAQ3U

• DAQ3U board– Data interface– FPGA– Buffer memory– Signal Chain and ADCs

• Backplanes– CPCI backplane– Interface backplane

Signal fanout Filtering Connectors

FP

GA

3U

CO

TS

CP

U board

CPCI backplane

CO

TS

1G ethernet

DA

Q3U

Signal chains and A

DC

s

Interface bd

DA

Q3U

Signal chains and A

DC

s


COTS CPU – Status

• Colleagues at CFHT have evaluated an SBS RL4

– PowerPC 7400/750 CPU at 400 to 500MHz

– Fast 10/100Mbit Ethernet– PMC extension slot– Conduction cooled option

• LINUX support not good

• IfA has purchased and received a Menmicro F1N

– PowerPC MPC8245/300MHz– 1 Gbit Ethernet daughterboard– LINUX ElinOS support (RTAI also)

• STATUS – Under Test


FPGA3U Board Design

Pow

er

sect

ion

To DAQ3U

128Mx32 SDRAM

1Gb/100BTPHY

Fiber coupled 100BT* /100BT /10 BT

32Kx32 Dual Port RAM Buffer

Compact PCI Bus

LDV

S

Loca

lX

tal

Optocoupled Trigger I/O

Trigger signals

Tes

t C

onn

Support CPLD

JTAG

FP

GA

P

RO

M

Xilinx Virtex 2Pro

FPGA

405 PPC

• Virtex2pro FPGA possible centroid/shift calculator


Xilinx ML300 Development Board

Pow

er

sect

ion

To DAQ3U

128MB DDR SDRAM

1GPHY

Fiber coupled 1G ethernet

•PMC PCI Bus

LDV

S

Loca

lX

tal

System ACE

JTAG

FP

GA

P

RO

M

Xilinx Virtex 2Pro

FPGA

405 PPC

1394

PH

Y

PCMCIA RS

232

LCD

Tou

ch

scre

en

Dis

play

Par

alle

l po

rtK

eybo

ard

port


DAQ3U Board Design

Analog Devices

5379 Analog switch

Analog switchAnalog

switchAnalog switchAnalog

switchesAnalog switches

Analog switches

Analog switches

Voltage range buffers

TBD RAM

Analog Devices

9826

3 CCD outputs

Analog Devices

9826

3 CCD outputs

Analog Devices

9826

3 CCD outputs

LDV

S

Analog Devices

9826

3 CCD outputs

Analog Devices

9826

3 CCD outputs

Analog Devices

9826

3 CCD outputs

32 bit pixel read

bus

Optocoupled Trigger I/O

Trigger signals

Tes

t C

onn

Xilinx Virtex 2Pro

FPGA405 PPC

Pow

er

sect

ion

Support CPLD

JTAG

FP

GA

P

RO

M100BTPHY

Fiber coupled 100BT

• Virtex2pro FPGA clocking and pixel data aggregation (possible to incorporate into FPGA board Virtex2pro


Mechanical Check DAQ3U

999999-14XYZ 04/19/23

ADC selection

Baseline AD9826 due to small size and speed but need to test for INL and ENOB (&/or oversample)

16BIT ADC Freq Ana Input Input R Tconvert S/N 100khz S/N 900khz THD 100Khz ref stab Integral nonlin Diff nonIin ENOB Size PowerAnalogic ADC4322 2Mhz +/-2.5/5/10 750/1.5k 300ns 86dBmin 75dBmin -86dBmax 15ppm/C +/- .003% +/- .75LSB 14.29s/n 2.4"x1.5" 2.3WDatel ADS-932MC 2Mhz +/-2.75 500 pipeline 81dBmin 80dBmin -88dBmax +/-30ppm/C +/- 1LSB +1LSB 13.45s/n 2.1"x1.11" 1.85WDatel ADS-935MC 5Mhz +/-2.75 400 3pipeline 83dBmin 75dBmin -86dBmax +/-30ppm/C +/- 1LSB +1LSB 13.78s/n 2.1"x1.11" 2.85WLinear Tech LT1604 333Khz +/-2.55 ? ? 88dBtyp -100dBmax +/-15ppm/C 36 SSOP 0.2WLinear Tech LT1620 500Khz +/-2.5 ? 380ns 90dBmin -88dBtyp +/-15ppm/C +/- 2LSB 36 SSOP 0.42WAnalog Devices 7671 1-0.8Msps +/-2.5/5/10 948/1.63K no pipe 90dB typ TBD external +/- 1LSB external 48 LQFP .15WAnalog Devices 7677 1-0.8Msps -.1 to 3VDE ? no pipe 94dB 20Khz n/a -110dB external +/- 1LSB +/- 1LSB external 48 LQFP .13WAnalog Devices 7676 500Ksps +/-2.5 ? no pipe 94dB 20Khz n/a 94dB 20Hz external +/- 1LSB external 48 LQFP .15WAnalog Devices 9260Analog Devices 9826 15MSPS 3CHNL 0 t0 4V* 3pipeline TOTAL 3LSB int? +/- 16LSB +/- .5LSB 13s/n 28 SSOP 400mWAKM semi SPT8100 (13ENOB) 5Mhz 0 to +5V 5.5K ohms 200ns X6 cyc 78dBmin 80dB typ -84max ? +/- 1.25LSB +/- .5LSB ? 44 LQFP 0.465mWMaxim MAX1200 1Mhz 0 t0 4V* 55K ohms 4us(pipeline) 83min -82max external +/- 3.5LSB +/- .6LSB external 44MQFP


AD9826 ADC Preliminary Testing 1

• Fabricated small test PCB with same pinout as Analogic AD4322 (2MHz 16 bit) .

• 2 PCBs tested in earlier version array controller.

• Quick tests showed ~X3 gain vs AD4322s, but 11-12 ADUs Standard Deviation (range 9-12) in 1 channel Sample and Hold mode.

– Data sheet says “3LSB” PGA=1– PGA set =1, sampling freq ~5.6Mhz– Running Raytheon 206 multiplexer (3 ADUs STD

nominal)

• Tested with all 8 channels and image with Raytheon 206 mux to confirm signal gain and S/N.


AD9826 ADC Preliminary Testing 2

• Bench tested with precision voltage source, logic analyzer

• 3 channel CDS mode has 0-1 ADUs STD.– PGA set =1, sampling freq ~1.9Mhz, AC coupled.– Device only wants Vreset more positive than Vsignal.– Test not valid if grounded input = 0 volts into ADC.– First/few? pixels bad.

• NOAO results:– CDS and S&H modes have 7-8 ADUs noise @ 1.6usec– Removed 2nd stage premap stage = removed “capacitive charge effect”.

None seen at IfA– “Note, apparently the AFE devices were designed to be used in a free

running mode so that the capacitive effects relating to charging the AC coupling cap to the bias clamp voltage are eliminated.” Will investigate, IfA setup had different offset/clamp circuit.

• Will retest next week to confirm results.

• Samples of TI ADCs received.


Clocking

Will build on previous (Redstar3) system VHDL design

• 16,000 gate Altera FPGA EP6016

• 128kx16 Sync SRAM

• Clocking FPGA– patterns– 7 x 1/48Mhz = 145.83nsec min pattern time– 1/48Mhz = 20.8333nsec pattern time extension– 10 bit pattern run length extension = 21.313usec – rows and frames

10 bit rep count = 1024 times

– integration timer– 25usec time increments– 28 bit timer = 3.728 hrs max ITIME


Pixel server = Thin Server 1U rackmount PCs

• Compaq Proliant DL360 G3 under test

• Supermicro dual Xeon under test

•Pixel servers in use on previous design with good results (faster than dual 32 bit 40MHz DSP system).


OTA Controller Power Supplies

• HP 66000 System Power Supply, 8 modules/chassis

• Each module has over voltage and over current protection– Simultaneous shutdown

• Configured for simultaneous turn on and off

• Proven low noise and safety performance on Mauna Kea (IRTF, SUBARU) – UPS run down and HELCO black/brownouts

• Mechanical Specifications 19 inch rackmount

• Width: 426 mm (16.75 in)

• Height: 178 mm (7 in)

• Depth: 678 mm (26.7 in)


Staffing Estimates

• Needed: 5 FTEs estimated, plus use of Contract Manufacturers 2 Electronic engineers (Onaka,TBD) 2 Software engineers (Lockhart, Isani *) 1 Electronic Tech (Ching)

• Presently: 3+ FTEs on staff– 1 new EEI causal hire => direct hire position hire started– + 1 SE interviewing*– 1 EEII (with data comm, ISCSI ASIC/IC experience) in discussion

• Contract Manufacturers (CM) for board assembly– Ching site visited 3 CMs.– IfA H85 AO 10 board run being sent through one CM as first run.


Production Plan

• Two phase plan– Prototype phase

Use Contract Manufacturer (CM) for high density parts (i.e. BGA). But obtain in house ability to rework - requires some infrastructure investment

at IfA. Contract out parts of electronics design that is “industry standard” (i.e. FPGA

CompactPCI VHDL).

– Production Phase Use Contract Manufacturer (CM) to get quantity discounts. Contract out design of automated test electronics. JTAG for digital? Build multiple copies of low cost test station. Have a conservative (high) spares count.


Lab Support/Infrastructure

• In-house capability to do BGA rework.– BGA inspection microscope received training done.– BGA Xray microscope received training 8/8/03.

• Previous Redstar2 and Redline controllers available for comparative measurements.

• Switching CAD software to leverage new hire experience.– Protel to Cadence (for SPECCTRA router)

Cadence university donation request sent (with UH Physics). Software download done, waiting for license.

• Xilinx FPGA requires additional tools– Symplicity VHDL synthesizer received.– Mentor Modelsim simulation models on order (not received).


Development Task Breakdown

Prototype Phase

FPGA3U

Basic Clock FPGA code

Backplane design Trigger Clock Down

load

ADC read code

Proto Chassis

Buy Dev Xilinx Board

Proto FPGA3U

Centroid and shift

Single OTA test

Proto Interface Bd

Analog Testing

DAQ3U

Proto DAQ3UBackplane interface

Interface to FPGA3U

Trigger design

Dual OTA test

COTS CPU + 1G ethernet

Dev environment

Multi OTA code

Centroid and shift

Proto User Interface

1 G ethernet

testsData Export testing

Interface to DAQ3U

Pixel Server Workstation

1 G ethernet

tests

Analog testing

Centroid and shift

Data Storage testing

Component selection


Schedule

Phase I End Goal: 4X1 OTA controller finished by Dec 2003Hardware

– Components selected Mid Aug.– PCB layout done = Oct.– Flex cabling finished Nov.– Controller prototype Dec-Jan.

Software– 1 G ethernet throughput tested – Aug. – CompactPCI throughput tested – Aug.– Centroid/shift/load – Aug/Sept.– Pixelserver code – Nov.– OTA control development during Integration Oct-Dec.


Draft Workpackage Description

Prototype Phase 1.0:Predesign 1.0.X

• Workpkg 1.0.1– Title: Location of OTA centroid/clocking generation.– Type: Study+test+decision– Description: – Combination study and test code generation for embedded

PowerPC on FPGA vs. individual Pixelserver vs. top level Pixelserver.

– Metrics: Appropriate centroid algorithm, speed of calculation vs collection, clocking pattern FPGA download and swtch time.

– Estimate: 60hrs Software Engineer, 30hrs EEII.– Dependency: FPGA clocking VHDL code at testable level on

ML300.


Draft Workpackage Description 2

Prototype Design workpackages: 1.1.X

• Workpkg 1.1.1– Title: Basic OTCCD clocking control– Type: Design+code+test– Description:

Software code of clocking, test of download. Hardware design on ML300, schematic and VHDL for CAD input for PCB.

– Metrics: OTCCD Sequenced readout functionality, speed.– Estimate: 60Hrs SE, 120hrs EEII, 80hrs SE*.– Dependency: none.


Workpackages

Prototype Phase 1.0:Predesign 1.0.X

Workpkg 1.0.1Title: Location of OTA centroid/clocking generation.Type: Study+test+decisionDescription: Combination study and test code generation for embedded PowerPCon FPGA vs. individual Pixelserver vs. top level Pixelserver.Metrics: Appropriate centroid algorithm, speed of calculation vscollection, clocking pattern FPGA download and swtch time.Estimate: 60hrs Software Engineer, 30hrs EEII.Dependency: FPGA clocking VHDL code at testable level onML300.

Workpkg 1.0.2Title: Number of FPGAs per controller.Type: Study+decisionDescription: Study of #FPGA I/Os + interboard connector vs. added powercomplexity of 2nd FPGA per controller. Metrics: Power consumption, PCB area available, connectorsavailable, analog signal chain integrity.Estimate: 40hrs EEII.Dependency: none.

Workpkg 1.0.3Title: 1G ethernet from FPGA or embedded COTS CPU?Type: code+test+decisionDescription: Combination lab test of ML300 dev platform 1Ge throughputvs. PCI+COTS CPU (MENmicro). Metrics: Sustained throughput > = 64MBytes/secEstimate: 60hrs Software Engineer,40hrs EEII.Dependency: none.


Workpackages 2

Workpkg 1.0.4Title: Real CCD test of AD9826 ADC on Leach systemType: Design+fabrication+testDescription: Design, fab and test AD9826 ADC on actual CCD to determinenoise and speed performance acceptability. Metrics: Read noise spec = CCD and = data sheet at 1MSPSEstimate: 20hrs EEII remaining (ET hrs used also).Dependency: none.

Workpkg 1.0.5Title: Release 1.0 of requirements for controllerType: DesignDescription: Hardware and software requirements document.Metrics: Completeness, science and control goals.Estimate: 20Hrs SE, 20hrs EEII.Dependency: LAN/WAN interface, Telescope scheduler reqs.-------------------------------------------------------------------Prototype Design workpackages: 1.1.X

Workpkg 1.1.1Title: Basic OTCCD clocking controlType: Design+code+testDescription: Software code of clocking, test of download. Hardware design on ML300, schematic and VHDL for CAD input for PCB.Metrics: OTCCD Sequenced readout functionality, speed.Estimate: 60Hrs SE, 120hrs EEII, 80hrs SE*.Dependency: none.


Workpackages 3 see draft plan

Workpkg 1.1.2Title: FPGA PCB Type: Design+code+CAD+fabricate+testDescription: Production of FPGA printed circuit board.Metrics: Power dissipation, size, speed of interfaces.Estimate: 40Hrs EEI, 60hrs EEII, ET estimate TBD*.Dependency: Predesign phase.

Workpkg 1.1.3Title: DAQ3U PCB Type: Design+code+CAD+fabricate+testDescription: Production of DAQ3U printed circuit board.Metrics: noise performance at speed, size, power dissipation.Estimate: 40Hrs EEI, 60hrs EEII, ET estimate TBD*.Dependency: Workpkg 1.0.4.

Workpkg 1.1.4Title: Cryo flex/PCB Type: Design+CAD+fabricate+testDescription: Production of rigid-flex cable assembly. Cyrotest integrity.Metrics: noise performance,size, vacuum performance, durability.Estimate: 40Hrs EEI, 30hrs EEII, ET estimate TBD*.Dependency: OTCCD pinouts final.

Documents

PanSTARRS Internal Spec Review 8/5/2003 PanSTARRS Gigapixel Array Controller System Overview Present Status –CPU board –FPGA board –DAQ board Staffing