11
0-7803-7367-7/02/$17.00 © 2002 IEEE 9.B.5-1 BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI STANDARD FOR SPACECRAFT AVIONICS Richard F. Conde, Joseph W. Haber, Richard W. Webbert, Russell J. Redman, Joanna D. Mellert, Joseph F. Bogdanski, and Sharon X. Ling The Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723-6099 David M. Hutcheson, BAE Systems, Manassas, VA 20110 Abstract MESSENGER (MErcury Surface, Space ENvironment, GEochemistry, and Ranging) is a mission to orbit and explore the planet Mercury. MESSENGER will carry out comprehensive measurements for one Earth-year. The subsystem that controls the MESSENGER spacecraft is called the Integrated Electronics Module (IEM). The IEM will operate the MESSENGER spacecraft, store data, and autonomously detect and mitigate onboard faults. In addition to meeting challenging requirements, the IEM must be available for integration on the spacecraft only 18 months after the start of full engineering development. The adoption of the 6U compact PCI (cPCI) standard has simplified the IEM development effort, reducing the time and cost that would otherwise be required. The 6U cPCI standard dictates board dimensions and the backplane electrical interface. The use of the standard has allowed some of the IEM boards to be specified and procured with a competitive selection process in a minimal amount of time. Prototype IEM systems have been assembled using commercially available backplanes, card racks, and Ethernet cards. Low-cost off-the-shelf cPCI tools such as board extenders and bus analyzers have been used to aid development. These components and tools would have to be developed at a significant cost if a proprietary or non-commercial board format had been adopted. The 6U cPCI specification itself does not address board level mechanical and thermal requirements, but features were added to the boards to meet these requirements while retaining compatibility with the cPCI standard. This paper details the benefits and lessons learned that have resulted from the use of the 6U cPCI standard in the development of spacecraft avionics. Introduction MESSENGER (MErcury Surface, Space ENvironment, GEochemistry, and Ranging) is a mission to orbit and explore the planet Mercury [1]. The mission plan is to launch in March 2004 and use two flybys each of Venus (June 2004 and March 2006) and Mercury (July 2007 and April 2008) to arrive at Mercury in April 2009 [2]. MESSENGER will orbit Mercury and carry out comprehensive measurements for one Earth-year. The subsystem that will control the MESSENGER spacecraft is called the Integrated Electronics Module (IEM). The IEM will operate the spacecraft, store data, and autonomously detect and mitigate onboard faults. IEM Design The Johns Hopkins University Applied Physics Laboratory (JHU/APL) started preliminary design of the MESSENGER mission in early 2000. The preliminary design of the IEM was completed by the end of that year. Design considerations included: The schedule was short; only 18 months were available from the start of full engineering development to delivery of first flight IEM. Previous JHU/APL IEM designs did not offer sufficient computational throughput or data storage, so new designs were necessary. Based on these considerations, it was decided that the IEM design should leverage as much as possible off work already performed in industry instead of pursuing in-house custom designs to meet the program requirements. In order to

BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

0-7803-7367-7/02/$17.00 © 2002 IEEE9.B.5-1

BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACTPCI STANDARD FOR SPACECRAFT AVIONICS

Richard F. Conde, Joseph W. Haber, Richard W. Webbert, Russell J. Redman, Joanna D. Mellert,Joseph F. Bogdanski, and Sharon X. Ling

The Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723-6099

David M. Hutcheson, BAE Systems, Manassas, VA 20110

AbstractMESSENGER (MErcury Surface, Space

ENvironment, GEochemistry, and Ranging) is amission to orbit and explore the planet Mercury.MESSENGER will carry out comprehensivemeasurements for one Earth-year. The subsystemthat controls the MESSENGER spacecraft is calledthe Integrated Electronics Module (IEM).

The IEM will operate the MESSENGERspacecraft, store data, and autonomously detect andmitigate onboard faults. In addition to meetingchallenging requirements, the IEM must beavailable for integration on the spacecraft only 18months after the start of full engineeringdevelopment. The adoption of the 6U compact PCI(cPCI) standard has simplified the IEMdevelopment effort, reducing the time and cost thatwould otherwise be required. The 6U cPCIstandard dictates board dimensions and thebackplane electrical interface. The use of thestandard has allowed some of the IEM boards to bespecified and procured with a competitive selectionprocess in a minimal amount of time. PrototypeIEM systems have been assembled usingcommercially available backplanes, card racks, andEthernet cards. Low-cost off-the-shelf cPCI toolssuch as board extenders and bus analyzers havebeen used to aid development. These componentsand tools would have to be developed at asignificant cost if a proprietary or non-commercialboard format had been adopted. The 6U cPCIspecification itself does not address board levelmechanical and thermal requirements, but featureswere added to the boards to meet theserequirements while retaining compatibility with thecPCI standard. This paper details the benefits andlessons learned that have resulted from the use ofthe 6U cPCI standard in the development ofspacecraft avionics.

IntroductionMESSENGER (MErcury Surface, Space

ENvironment, GEochemistry, and Ranging) is amission to orbit and explore the planet Mercury [1].The mission plan is to launch in March 2004 anduse two flybys each of Venus (June 2004 andMarch 2006) and Mercury (July 2007 and April2008) to arrive at Mercury in April 2009 [2].MESSENGER will orbit Mercury and carry outcomprehensive measurements for one Earth-year.The subsystem that will control the MESSENGERspacecraft is called the Integrated ElectronicsModule (IEM). The IEM will operate thespacecraft, store data, and autonomously detect andmitigate onboard faults.

IEM DesignThe Johns Hopkins University Applied Physics

Laboratory (JHU/APL) started preliminary designof the MESSENGER mission in early 2000. Thepreliminary design of the IEM was completed bythe end of that year. Design considerationsincluded:

• The schedule was short; only 18 monthswere available from the start of fullengineering development to delivery offirst flight IEM.

• Previous JHU/APL IEM designs did notoffer sufficient computational throughputor data storage, so new designs werenecessary.

Based on these considerations, it was decidedthat the IEM design should leverage as much aspossible off work already performed in industryinstead of pursuing in-house custom designs tomeet the program requirements. In order to

Page 2: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-2

accelerate the schedule and minimize costs, it wasdecided to base the design on the 6U compactPeripheral Component Interconnect (cPCI)standard. This choice greatly aided the preparationof board level specifications and maximized the useof commercial off-the-shelf (COTS) components.The use of COTS components is advantageousbecause the number of custom items that must bedeveloped (at a significant cost) is reduced. It wasrecognized that the 6U cPCI board format was notoptimal from a mechanical and thermal standpointfor spacecraft use, but it was believed to be “goodenough.” Conductive heat sinks and boardstiffeners could be used to satisfy mechanical andthermal requirements and retain 6U cPCIcompatibility.

At the conclusion of the preliminary designphase, the IEM design was partitioned into fivedaughter cards, a backplane, and a cast aluminumchassis. A block diagram of the IEM is shown inFigure 1. Three of the five daughter cardscommunicate over a PCI bus. Three of the cards(Main Processor, Fault Protection Processor, andSolid State Recorder) were designed andmanufactured by BAE SYSTEMS to JHU/APL

specifications. These boards were specified to be asgeneric as possible, incorporating few features thatwould be mission unique so that they could beeasily specified and used on multiple spacecraftprograms. The Main Processor (MP) and FaultProtection Processor (FPP) boards are nearlyidentical. The Interface Board and Converter Boardwere designed and built by JHU/APL. Theseboards capture the MESSENGER-uniquerequirements. The chassis and motherboard weredesigned by JHU/APL and fabricated out of house.

The three breadboard IEM systems were builtto maximize the amount of COTS components sothey could be produced as rapidly as possible. Thetwo flight IEM systems have mechanical andthermal requirements that necessitate custom built(not COTS) boards and chassis.

Board Format SelectionConsiderations

Three card formats were initially consideredfor the IEM: Extended SEM-E (formally known asIEEE-1101.7-1995), 3U cPCI, and 6U cPCI. TheExtended SEM-E format was considered because

ConverterBoard

+3.3V Unswitched+5V Unswitched+3.3V Switched+5V Switched

+15V Switched

FaultProtectionProcessor5 MHz RAD60004 MBytes RAM

4 MBytes EEPROM

MainProcessor25 or 6.25 MHz

RAD60008 MBytes RAM

4 MBytes EEPROM

Solid StateRecorder

Interface Board

PCI Bus

BC or RT BM and RT

MIL-STD-1553B Bus

SpacecraftInterfaces

IEMUnswitched

Power

IEMSwitched

Power

DiscretesDiscretesSecondary

PowerReset

Signals

Ground TestInterfaces

CriticalCmd

Decoder

TurboCoder

MissionElapsed

TIme

DnLinkBuffer

8 Gbits

UplinkBuffer

SpecCmdGen

OCXO

Discretes

Switched Power

Switched Power Unswitched Power

Unswitched Power

Integrated Electronics Module

Figure 1. MESSENGER Integrated Electronics Module Block Diagram

Page 3: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-3

JHU/APL had used it for the previous generation ofIEM, and it has good mechanical and thermaldesign features for space avionics use. The cPCIstandards are beginning to be used in spaceapplications; the Jet Propulsion Laboratory X2000program is using the 3U cPCI format, and theLangley Research Center-managed GIFTSspacecraft is using the 6U cPCI format. The 3U and6U cPCI cards have the same depth (160 mm), butthe 6U format is taller (233.35 mm) than the 3Uformat (100 mm).

The 6U card is almost identical to theExtended SEM-E in overall board area (lengthtimes width). Effective board area is the areaavailable for application electronics; it is the arealeft after subtracting out area needed for backplaneconnectors, stiffeners, card locks, and backplaneinterface circuitry. As seen in Table 1, the 6Uformat is slightly more efficient than the ExtendedSEM-E format in terms of effective board area andboard area per board weight. The implication is thatmore circuitry can fit on a 6U board than an

Extended SEM-E board, at a lower weight. The 6Uformat, with some of the modifications forspacecraft avionics use, is shown in Figure 2. Twoof the backplane connectors are shown removed(discussed in a later section), and wedgelocks areadded to clamp the board into the flight chassis.

Figure 2. 6U cPCI Board Format

Table 1. IEM Board Format Selection Criteria

Parameter ExtendedSEM-E

3U cPCI 6U cPCI

Area (one side, cm2) 364 157 366

Effective Board Area = Total Board area – card locks– backplane connectors - backplane I/F circuitry (cm2)

594 235 677

Effective board compared to desired functional blocks Close Match Too small Close match

Board edge length available for I/O connectors (cm) 13.6 8.1 21.3

Weight (g) 467 to 953(TIMED S/Cactual weights)

No data 526 – 998(MESS.estimates)

Effective board area per lb (cm2/g) 0.62 – 1.27 No data 0.68 – 1.29

Thermal Conductivity (rank, 1=best) 1 2 3

COTS equipment readily available No Yes Yes

Number of boards on PCI Bus (fewer is better) 3 9 3

Overall Development Cost (rank, 1= best) 3 (all custom) 3 (need 2x–3x thenumber ofcards)

1 (minimize #of boards,maximize use ofCOTS)

Page 4: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-4

Test equipment (cPCI backplane analyzers andcard extenders), prototyping equipment (chassis andbackplanes), and boards useful during development(Ethernet cards and bridge boards) are significantlycheaper with the 6U approach than with anExtended SEM-E approach. They are available offthe shelf instead of needing to go through a design-fabricate-assemble-test cycle.

The basic stand-alone building blocks in theIEM are the five boards: (1) MP, (2) FPP, (3) SolidState Recorder (SSR), (4) Interface, and (5) DC/DCConverter boards. All boards are tightly packed;the MP, FPP, and SSR make use of stackedcomponents in order to fit all desired functions onthe board. There is little wasted space. Photos ofMP and SSR engineering model boards are shownin Figures 3 and 4. The ratio of effective board areabetween the 6U and 3U formats is 677/235 = 2.9.This means that each densely packed 6U boardwould need to be replaced with three 3U cards. Ifthe boards were less densely packed, so that onlyone or two 3U boards were required to replace asingle 6U card, the 3U format would be moreattractive. But based on the board layouts, the five6U boards in the IEM would have to be replaced byabout fifteen 3U boards. An approach using 3Uboards would be more costly, since designing andfabricating three 3U boards would be moreexpensive than a single 6U board.

Figure 3. IEM Main Processor Board

Figure 4. IEM Solid State Recorder Board

Another consideration in selecting the 6Uformat over the 3U format is the number of IEMboards connected to the PCI backplane bus. If the3U format were used, nine boards would have to beconnected to the PCI bus. However, the cPCIspecification only allows a maximum of eightboards to operate on a single backplane. For thisreason there would be a significant technical risk inachieving proper function with a system havingnine boards on the PCI bus. It might be necessary tobreak the PCI backplane bus into two or moresegments. This would require the addition of PCIbridge chips to the design, a significantcomplication.

For a variety of reasons the decision was madeto contract out the MP, FPP, and SSR boards.Radiation-hardened, high-performance processorswere only obtainable from processor vendors asboards, not chips. There were no existing processoror SSR boards available that could meet theMESSENGER requirements; specifications had tobe written and a competitive bid process followed.By using commercial standards as much aspossible, the time required to write thespecifications was minimized, because most of thebackplane bus and card dimension sections couldsimply refer to the cPCI specification. The cPCIstandard was thus preferred to the Extended SEM-Eformat, which is not as widely used and does notinclude a backplane bus definition.

The major drawback of the 6U format is that itis more difficult to conduct heat off the board thanthe other standards. This is because the boardmakes contact with the chassis along the shorter

Page 5: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-5

160 mm board edge rather than the longer 233.35mm edge. Spacecraft avionics rely primarily onconduction to remove heat, so the lesser the contactarea between the board and chassis, the moredifficult it is to conduct heat off the board.However, thermal analyses have shown that thermalde-rating criteria will be met even for the hottestboard.

In summary, the 6U format was selected overthe Extended SEM-E format because it is somewhatmore efficient in effective board area and lower inweight per board area. It is lower in overall cost dueto the availability of low-cost COTS equipment,and saves time because the equipment can bebought off the shelf instead of being developed witha design-fabricate-assemble-test cycle. Also,specifications for 6U boards can be written morequickly than those using the Extended SEM-Eformat. The 6U format was selected over the 3Uformat because as many as fifteen 3U cards wouldbe required in place of the five 6U cards. Thisnumber of cards would add cost and technical risk.

PCI Chip SelectionWithin the MESSENGER IEM there are three

boards connected to the PCI bus. Of these, theRAD6000 chipset on the Main Processor board andthe Power PCI Bridge chip on Solid State Recorderboard have inherent PCI capability. The JHU/APLdesigned Interface board is the third board on thePCI bus. Several PCI interface implementationswere considered: an Actel PCI core in an Actelradiation-tolerant Field Programmable Gate Array(FPGA), and several chips with PCI interfacesavailable from BAE SYSTEMS. It was decidedthat the lowest-risk approach was to select the mostappropriate PCI chip from BAE SYSTEMS.

This approach had several advantages. First,since an existing chip has a fully defined backend,the development of the Interface board could beginalmost immediately. Second, it greatly increased thelikelihood that all of the components on the PCI buswould operate together properly, since they wouldall come from a common source (BAE SYSTEMS).After reviewing the available chip designs, thePHASOR (Power Handling, with Access toSummit and Optical bus Resources) chip wasselected. This chip has a 16-bit back end bus, whichminimized the amount of hardware needed on the

board, compared to a 32-bit bus. The datathroughput of the chip was low but adequate for thebandwidth required by the MESSENGER mission.

The PHASOR chip also has the advantage ofhaving Input/Output (I/O) that is tolerant ofvoltages higher than the 3.3 V that powers the chip.The PHASOR chip is on a switched 3.3 V DC/DCconverter, and the rest of the board is on anunswitched 3.3 V DC/DC converter. The high-voltage-tolerant I/O on the PHASOR chip made itpossible to connect together chips powered fromdifferent supplies.

Lessons Learned with COTSEquipment used for IEM BreadboardDevelopment

Backplanes and EnclosuresStandard cPCI backplanes and enclosures were

used for the IEM breadboards, but in a uniqueconfiguration three independent backplanes wereinstalled in a single enclosure. Bustronic wasselected to supply these backplanes and enclosures.Two four-slot and one six-slot backplanes wereused. Separate backplanes were required becausesome of the IEM boards operate on switched powerwhile others operate on unswitched power. An IEMbreadboard is shown in Figure 5.

Figure 5. Front of IEM Breadboard

Backplanes compliant to PICMG 2.0, Rev. 3.0were desired, because these have independent clocklines from the system slot board to each peripheralboard. The older cPCI specification is less robustbecause two peripheral slots each share a singleclock line. The MP board functions as the cPCIsystem slot board. System slot boards are required

Page 6: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-6

to have seven PCI clock signal outputs. Because theMP supports only five clock outputs, the backplanemanufacturer had to be contacted to determine howthe clock outputs are routed to the peripheral slotsinasmuch as this is not specified by cPCI. Only theperipheral slots that have clock lines routed to themfrom the MP can be used.

Extender CardsExtender cards are needed during the

development of cPCI boards for debuggingpurposes. While not guaranteed to work (theyviolate cPCI track-length requirements), it wasfound that a cPCI passive extender worked, up tothe 25-MHz PCI clock frequency used in the IEMdesign. Numerous cPCI passive extenders wereavailable; an extender from Catalyst Enterprises(P/N 6U-CPCI) was selected because it had thesimplest electrical design. Many extenders includedswitches that permitted each cPCI signal to beindividually disconnected from the board. It wasfelt that the simplest design, with the fewestdisturbances to the PCI signals, offered the bestchance of success.

Since the J3 and J5 backplane connectors arenot used in the IEM design, the extenders wereordered with those connectors depopulated in orderto minimize the insertion and extraction forces ofthe extender in the chassis. A downside of theextender is that there is no easy way to extract it.The extender used is shown in Figure 6.

Figure 6. 6U cPCI Extender Card

Ethernet CardCP610/1 Ethernet cards from RAMIX were

used in the IEM breadboards to support softwaredevelopment. The RAD6000 processors aredesigned so that the PCI clock rate equals theprocessor clock rate, so that when the processor isat a low clock rate, the PCI clock is at the same lowrate. A problem was found when the RAD6000processors were operated at 5 MHz (FPP) and 6.25MHz (MP in one mode). The Ethernet cards wouldnot operate properly when connected to theEthernet network switch. The problem was found tobe that the Ethernet cards would auto-negotiate withthe switch to 100-Mb/s operation but could notactually operate at 100-Mb/s at low PCI clock rates.The solution was to place a fixed rate 10 Mb/s hubbetween the Ethernet cards in the IEM and networkswitch. Another potential solution would be tochange the driver for the Ethernet card so that itforced operation at 10-Mb/s. The Ethernet cardsalso required a modification to operate without +/-12 V supplies, since the IEM operates from +3.3 Vand +5 V only.

Bus AnalyzerA cPCI bus analyzer from VMETRO, Inc.,

was used during the development of the IEM. Itoperated successfully with RAD6000 processorboards.

Rear Transition Card6U cPCI has five backplane connectors

designated J1-J5. The use of the J1 and J2connectors is reserved by the cPCI specification.The J3-J5 connectors are undesignated andavailable for use. On the IEM, J3 and J5 aredepopulated to minimize insertion and extractionforces, and all unique signals on the backplane arerouted on the J4 connector. The cPCI specificationdefines rear transition cards that can plug into theback of standard cPCI chassis to make connectionsto the J3-J5 connectors of each cPCI card.However, no COTS rear transition cards were foundthat would allow connections to be made from onerear transition card to another, so a custom reartransition card was designed and fabricated. Thecards are used to make all of the inter-boardconnections that are unique to the IEM. Figure 7shows an individual rear transition card and

Page 7: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-7

Figure 8 shows the back of a breadboard IEM withthe wiring harness used to make these connections.Because all of the signals are low speed, controlledimpedance connections were not needed.

Figure 7. Rear Transition Card

Figure 8. Rear of IEM Breadboard ShowingRear Transition Card Wiring Harness

Lessons Learned with COTSEquipment Used for IEM InterfaceBoard Development

An advantage of adhering to a commercialstandard for the IEM boards is that COTSequipment can be used to test the boards. AnInterface Board Test System (IBTS) was built totest the IEM Interface board. It employed an open-ended cPCI chassis from Tracewell Systems(Figure 9.). To reduce cost and development time,a standard desktop PC with a PCI-to-cPCI bridgewas used as a system slot board rather than using astandalone cPCI processor board (RAD6000 or

otherwise). A PC to cPCI board from SBSTechnologies was selected. Microsoft Visual C++and Venturcom Real Time Extensions (RTX) wereutilized to develop the software to control theInterface Board.

Figure 9. IEM Interface Breadboard in TestChassis

One difference encountered by using a PC witha bridge board rather than a RAD6000 processorboard is that a PC has a fixed PCI clock rate of 33MHz. This is faster than the PCI clock rate in theIEM (25 MHz). So, if a PC is used to drive thesystem slot in the cPCI backplane, the developershould verify that the PCI chip on the board undertest can support operation at 33 MHz even if it isoperated at a lower rate in the final configuration.

PHASOR Chip PCI Configuration ProblemTrouble arose when it was found that the

PHASOR chip used on the Interface board, whilefully PCI compatible, was not fully compatible withthe Basic Input/Output System (BIOS) found inWintel PCs. The difficulty arose because thePHASOR chip maps its Optical Bus Interface (OBI)into I/O space and requests 32 MB of space. Whilethis is well within the allowed limits of the PCIspecification (which allows up to 2 GB of IOspace), it far exceeds the maximum IO size allowedby the BIOS in a Wintel PC. The maximumavailable on a Wintel PC is only 64 kB. Theamount available to non-system devices is evenless, since all addresses below 0x3FF are reserved.

Page 8: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-8

This problem presents itself immediately uponboot-up of the host computer. During boot-up, theWintel BIOS configures the PCI devices byquerying them for their memory and IO spaceneeds, and then setting their Base Address Registers(BARs) accordingly. When the BIOS attempts toconfigure the PHASOR chip, it realizes that BAR1(the OBI IO Base Address Register) is requesting32 MB of IO space. Since this amount is notavailable, BIOS detects a failed configuration, andreports the following error:

Plug and Play Configuration Error

Strike the F1 key to continue, F2 to run thesetup utility.

By striking the F1 key, the computer continuesbooting up. Unfortunately, the PHASOR chip isleft in a state where both PCI IO and memoryaccesses are turned off.

Although this problem was difficult todiagnose, the solution is rather straightforward. Byperforming a configuration cycle on the PHASORchip, the BAR1 can be moved out of the PC’s IOspace and into an innocuous location. The softwareproduct Real Time Extensions (RTX) was used toperform the configuration read and writeoperations. RTX “provides an essential set of real-time programming interfaces in the Win32environment.” In this circumstance, RTX isparticularly useful because it provides a functioncall that will do configuration accesses directly overthe PCI bus under application software control.

Big Endian Versus Little EndianData stored in a byte addressable memory can

be implemented as Big Endian or Little Endian.The PCI bus specifies Little Endian byte ordering inthat the least significant byte is contained in bitsAD[7:0] and the most significant byte is containedin bits AD[31:24]. Only the first 64 bytes of thePCI configuration space for a given PCI device isspecified as Little Endian. RAD6000 devices aregenerally Big Endian, which reverses the bytescompared to Little Endian. This became an issuewhen accessing the PHASOR chip using a PC.While the PCI configuration space was as expected(Little Endian), byte swapping must be performedfor the other registers, relative to how they aredefined in the PHASOR specification. This will be

the case for many of the PCI chips developed byBAE SYSTEMS when connected to a non-RAD6000 (Little Endian) processor.

Interrupt ProcessingThe PHASOR chip interrupt processing was

not as straightforward as hoped, although theconfiguration cycle problem was certainly moredifficult to solve. The PHASOR did not implementthe Interrupt Pin and Interrupt Line registers in thePCI configuration header. The software thereforecould not query the configuration space and find outthe interrupt vector from the PHASOR itself. ThecPCI standard assigns interrupts based on logicalboard slots. The same board will drive one of fourdifferent interrupt pins based on its position in thebackplane. It was not clear what interrupt thePHASOR was driving in the cPCI chassis. Thiswas solved by using the VMETRO cPCI BusAnalyzer to discover which interrupt was driven bythe PHASOR for each slot and then attaching andenabling interrupts based on that knowledge. Thetest software needed access to the HardwareAbstraction Layer (HAL) interrupt API to attachinterrupts manually. The solution used RTX, whichallowed interrupts to be attached and enabled withinWindows.

The interrupt processing itself wasstraightforward. Windows uses the thread-interruptmodel instead of traditional interrupts. RTXprovided several ways of installing interrupts andalso generating a process to respond to them. Thesimplest and slowest method was used. A worst-case interrupt latency of less than 2 ms wasobserved using Windows NT 4.0. With additionaleffort this performance could be improved but thiswas sufficient for this application.

Deviations from the cPCISpecification

In some cases, requirements in the cPCIspecification could not be followed or wereintentionally not followed. The first class of thesedeviations is due to limitations of the RAD6000processor. The RAD6000 was designed before thecPCI specification was established. In some areasthe RAD6000 deviates from the cPCI specification,

Page 9: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-9

and modifications must be made to COTSequipment in order to work with the RAD6000.

Number of PCI Clocks on System Slot BoardThe RAD6000 chipset supports seven PCI

clock outputs. Since two are used on the processorboard, only five are available for use off card. ThecPCI specification requires the system slot board tohave seven clock outputs (one per peripheral slot).Because only five are available, the user must checkwith the backplane vendor to determine whichperipheral slots will be connected to the availableclock outputs, since the assignment of PCI clockoutputs to peripheral boards is not made in thespecification.

Number of Bus REQ/GNT Signals on SystemSlot Board

The RAD6000 chipset supports five PCI BusRequest/Bus Grant signal pairs. Two of the pairsare used on the processor board. Three areavailable for use off card. The cPCI specificationrequires the system slot board to have seven signalpairs (one per peripheral slot). Since only threesignal pairs are available, only three peripheralboards can be PCI bus masters, which make use ofthese signals to request use of the bus. The usermust assign peripherals cards with bus mastercapability to those slots, or modify the backplane sothat slots that need the signals have jumpers to slotsthat have them but don’t use them. Thesemodifications were made to the IEM breadboardbackplanes.

Different AD Signals Used for ID SelectThe cPCI specification calls out the use of ID

Select (IDSEL) signals to provide unique access toeach slot for configuration purposes. This is doneby tying a unique AD signal to the IDSEL pin ateach slot on the backplane. During configurationcycles, a device is selected for configuration bydriving its IDSEL signal active. The cPCIspecification specifies that AD[31:25] be used forthis purpose. However, the RAD6000 uses thesignals AD[18:11] instead. In order for PCIconfiguration cycles to work with a RAD6000,commercial backplanes must be modified. At eachperipheral slot, the IDSEL pin must be isolated, and

a jumper wire must be added between the IDSELpin and an AD pin that is driven by the RAD6000during PCI configuration.

Deletion of P3 and P5 Connectors6U cPCI requires the presence of five

backplane connectors, designated J1-J5. The use ofJ1 and J2 is reserved. J3, J4, and J5 are userdefined. J4 is used for all of the IEM-unique I/Osignals. J3 and J5 are not installed on the boards inorder to reduce insertion and extraction forces.Testing has shown that the J3 and J5 are not neededto provide mechanical support when the IEM isvibrated to simulate launch conditions.

Board Envelope and SpacingThe IEM boards are cPCI compliant for card

length and width, but they are not compliant inregards to card thickness. cPCI systems have aboard spacing of 20.32 mm inches. In addition,cPCI boards are defined as having componentsinstalled on one side only. The IEM boards violatethe thickness standard because they havecomponents on both sides. In order to usecommercial backplanes in the breadboard IEM, theslots on either side of some IEM breadboards areleft empty. The flight IEM is designed with a boardspacing of 26.82 mm to accommodate the thickerboards.

Box Cover Instead of Board Front PanelsThe cPCI specification requires the use of a

front panel with handles on each board. The IEMboards were designed without individual frontpanels. Instead, the flight model IEM has a singlefront cover with cutouts for the connectors on eachboard. This was done for several reasons. First, itwas assumed that the cPCI specified board spacingof 20.32 mm could not be adhered to. So, when thespecifications were written for the boards, the boardspacing was not known and the front panel widthcould not be specified at that point. Instead, thecover was designed after the board spacing wasdetermined. Second, the flight chassis would not bedesigned like a commercial cPCI enclosure due tothe additional mechanical and thermal constraints itmust satisfy. Until the flight chassis was designed,it was not clear how the front panels (if used) would

Page 10: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-10

be attached to the chassis. So, the front paneldesign could not be included in the boardspecifications. Third, a single integral coverprovides better mechanical support than individualfront panels. Fourth, a single front cover will weighless than individual front panels with handles. Theflight IEM design with attached cover is shown inFigure 10.

Figure 10. Flight IEM Design

Instead of using the ejectors normally found onthe front panel of cPCI boards, holes were includedin the top and bottom front edge of the cards. Anejector tool is used to engage the holes and removethe boards from the chassis (both the commercialbreadboard chassis and custom flight chassis). Theejector was modified to accommodate the thickerboards.

Length of AD Traces on System Slot BoardThe final deviation from the cPCI specification

is the length of the PCI bus signal traces on the MPand FPP boards. The actual PCI signal stub lengthsexceed the 63.50 mm length called out in thespecification. This is mitigated by reducing themaximum PCI clock to 25 MHz and by restrictingthe PCI signal loading in the flight configuration tothree boards.

ConclusionThree IEM breadboard systems have been

assembled using 6U cPCI COTS components, oneyear after the start of full engineering developmenton the MESSENGER program. The breadboardIEMs were designed, fabricated, and assembledapproximately six to twelve months faster than if afull custom, non-COTS approach had been taken.In addition, this design approach had other benefitsthat could not be easily achieved in a custom non-COTS design, such as the availability of Ethernetcards and bus analyzers. The schedule and costbenefits of a COTS-based design more than offsetthe non-optimal mechanical and thermal aspects of6U cPCI that required mitigation.

There are a couple of cautionary notes. Simplybecause a COTS-based design is used does notmean that the design does not have to be understoodin depth. System design decisions and tradeoffsstill have to be made. Also, while it may be mosteffective to procure boards in some cases, visibilityof board-level operations can be lost. This loss canbe significant, since design decisions made at theboard level can have significant system-levelimplications.

AcknowledgementsThe authors sincerely appreciate the support

from the following individuals and organizations:

Jennifer A. Davis (JHU/APL), who as TeamLeader is coordinating the design activities inAPL’s Technical Services Department.

Harold R. White, Jr. (JHU/APL), for leadingthe flight IEM mechanical design effort.

Laura B. Burcin (BAE SYSTEMS), foreffectively managing the development of the MP,FPP, and SSR boards at BAE SYSTEMS, and forworking closely with APL to resolve the inevitableissues that came up.

David D. Stott (JHU/APL), for designing theIEM and Interface board test systems.

Timothy S. Wescott (JHU/APL), for designingthe cPCI rear transition cards.

Robert C. Moore (JHU/APL) and Larry J.Frank (JHU/APL), for preparing the IEM Processorand Solid State Recorder specifications.

Page 11: BENEFITS AND LESSONS LEARNED FROM THE USE OF THE …messenger.jhuapl.edu/Resources/Publications/Conde.et.al.2002.pdf · BENEFITS AND LESSONS LEARNED FROM THE USE OF THE COMPACT PCI

9.B.5-11

References[1] Solomon, S. C., R. L. McNutt, Jr., R. E. Gold,M. H. Acuña, D. N. Baker, W. V. Boynton, C. R.Chapman, A. F. Cheng, G. Gloeckler, J. W. Head,III, S. M. Krimigis, W. E. McClintock, S. L.Murchie, S. J. Peale, R. J. Phillips, M. S. Robinson,J. A. Slavin, D. E. Smith, R. G. Strom, J. I.Trombka, and M. T. Zuber, 2001, TheMESSENGER mission to Mercury: Scientificobjectives and implementation, 49, Planet. SpaceSci., pp. 1445-1465.

[2] Santo, A. G., R. E. Gold, R. L. McNutt, Jr., S.C. Solomon, C. J. Ercol, R. W. Farquhar, T. J.Hartka, J. E. Jenkins, J. V. McAdams, L. E.Mosher, D. F. Persons, D. A. Artis, R. S. Bokulic,R. F. Conde, G. Dakermanji, M. E. Goss, Jr., D. R.Haley, K. J. Heeres, R. H. Maurer, R. C. Moore, E.H. Rodberg, T. G. Stern, S. R. Wiley, B. G.Williams, C. L. Yen, and M. R. Peterson, TheMESSENGER mission to Mercury: Spacecraft andmission design, Planet. Space Sci., 49, 1481-1500,2001.