PCI Express Verification using Reference Modeling

PCI Express Verificationusing

Reference Modeling

PCI Express Verificationusing

Reference Modeling

Asad Khan and Scott Morrison

January 23, 2007

Texas Instruments IncorporatedASIC / Backplane IP Development

2

DUT

PurposePurpose

CPUCPU

Root ComplexRoot Complex

PCIe™ EndpointPCIe™ Endpoint

PCI Express to PCI Bridge

PCI Express to PCI Bridge

PCIe EndpointPCIe Endpoint

4-port PCI Express

Switch

• We will present– Modeling techniques for the

complete verification of a PCI Express® Switch

– Our use of Specman eRM– Block-level to chip-level re-use

3

OutlineOutline

• The Device Under Test (DUT) • Reference Model Example: Ingress Port Logic• Reference Model Example: Router• Integration of Reference Models at chip-level• Conclusions

4

What is a Reference Model?What is a Reference Model?

• A model that is independent of DUT implementation• Coded in high-level, human-readable language• Cycle-accurate prediction where necessary

• Because of maintenance overhead

• Able to be co-simulated with the DUT to predict and check the runtime behavior

• The auto-checking RFM+DUT simulation environment makes use of directed-random stimulus• “Let the machine do the work”

5

PCI Express SwitchPCI Express Switch

PIPE

RXTX

PIPE

TXRX

PIPE

RXTX

PIPE

RXTX

Packet Crossbar

Router Crossbar

De-queue Crossbar

PCIe PHY

PCIe PHY

PCIe SwitchGlobalControl

PCIe PHY

PCIeDL / MAC

PCIe SwitchDownstream

TL

PCIe PHY

PCIeDL / MAC


TLSEPL

R

PCIeDL / MAC

S

RIPL

PCIe SwitchUpstream

Port


Port

IPL

EPL

PCIeDL / MAC

S

R

IPL Ingress Port Logic

Scheduler

Router

Egress Port Logic

Legend

EPL

6

The DUTThe DUT

• The DUT (or DUTs, depending how we slice it up)• Customer deliverable: 4-port Switch ASIC• Large building blocks (verified at module level)

• Data Link Layer• Ingress Port Memory• Router• Scheduler

• Small building blocks• Power management• GPIO• Hot plug• Advanced Error Reporting• And more…

7

Design for Verification (DFV)Design for Verification (DFV)

• Minimize side band controls• Use a standard bus for module interfaces

• Standard BFM needed for module simulations

• Add DFV signals to reduce verification complexity• Limited variability of pipe-line timing• Limit the number of building blocks

• Re-use some blocks even though it is overkill

8

Module and Chip Verification StrategyModule and Chip Verification Strategy

• Bottom-up methodology• Reusable• Constrained-random stimulus• Reference Models (RFMs) for automated checking

• Hybrid approach• Control paths: cycle-accurate modeling• Data paths: packet-accurate modeling• Integration of models for chip-level prediction• Rigorous testing of linked list management, data link layer,

routing, and arbitration logic• Some directed testing required

9

A Look at Chip-level Prediction ModelA Look at Chip-level Prediction Model

Down Port 0

Down Port 1

Down Port 2

Up Port

TIPCIe eVCEndpoint

TIPCIe eVCEndpoint

TIPCIe eVCEndpoint

TIPCIe eVC

Root Complex

PCIe Switch DUT

Down Port 0RFM

Down Port 1RFM

Down Port 2RFM

Up PortRFM

Switch Prediction ModelCycle-accurate and

Packet-accurate

DUT stimulus Switch reference model

SupportingChip-level Scoreboard

Scoreboards

10

Chip-level Prediction Model: Closer lookChip-level Prediction Model: Closer look

Down Port 0RFM

e

IPL RFM

DL RFM

EPL RFM

Scheduler RFM

Router RFM

Egre

ss T

LP

TX TLP

PCIe Switch PortReference Model

Predictor

Cycle accurate control path

Packet-accurate data path

RX TLP

11

IPL: Verification ChallengesIPL: Verification Challenges

• All incoming packets buffered at input• Infinite memory space challenge

• Dynamic link list for en-queue/de-queue of packets

• Dual mode support with dynamic behaviors • Cut-through & Store-and-Forward

• Aggregation of traffic without back-pressure• Support for normal as well as error packets• Scalability for up to 9 simultaneous de-queues• Support for all permutations of throughputs

• Among available ports (x1, x2 and x4)

12

IPL DUT: How to verify?IPL DUT: How to verify?

DLL TLP Processor

Ingress Access Port (IAP)

TLP Crossbar

DLL TLP Interface

RouteMaster

Route Crossbar

TLP Status

Memory

Internal TLP Processor

TLP Status List

CreditCounters

Internal TLP Interface

DLL Credit Interface

TLP Processor

Data Buffer

De-QueueCrossbar

Memory Slot Controller

DLL SYNC FIFO

TLP Arbiter

BroadcastCount

Memory

TLP MemoryTLP

Memory List

9 egress portsMixed trafficDual mode

13

IPL Reference Model ArchitectureIPL Reference Model Architecture

u_irxINTPKTRX

u_erxEXTPKTRX

u_epreprocEXT-PACKET

PREPROCESSOR(CYCLE-ACCURATE)

+ERROR DETECTOR

u_ipreprocINT-PKT

PREPROCESSOR

u_ellmExternal-Packet

Link List Manager

u_illmInternal-Packet

Link List Manager

u_dq_mgrDQ

Manager

PORT1

PORT2

PORT9

PORT3

u_pkt_sorterPacket

SORTER

ECRC MALF MISCERROR HEADER SCOREBOARDS u_rthdr_drv

ROUTE HEADERDRIVER

u_rthdr_scbROUTE HEADER CYCLE

ACCURATE SCOREBOARD

u_rabRT

HDRARB

e-Code High Level RFM Data Flow

u_cfrCTRLCFRI/F

DATA SCOREBOARDS

e Link-List Model

F G E M N PORT1

F G E M N PORT2

F G E M N PORT9

REG

14

Reference Model FeaturesReference Model Features

• Cycle and Packet Accuracy (Hybrid Modeling)• Cycle accurate route token modeling for chip-level DV• Packet accurate modeling for packet transmission path

• Modular Verification Architecture• Scalability for switch derivatives

• Pointer Management Scheme• Independent of Hardware Implementation• Re-usable

15

IPL Reference Model Architecture (cont.)IPL Reference Model Architecture (cont.)

u_irxINTPKTRX

u_erxEXTPKTRX

u_epreprocEXT-PACKET

PREPROCESSOR(CYCLE-ACCURATE)

+ERROR DETECTOR

u_ipreprocINT-PKT

PREPROCESSOR

u_ellmExternal-Packet

Link List Manager

u_illmInternal-Packet

Link List Manager

u_dq_mgrDQ

Manager

PORT1

PORT2

PORT9

PORT3

u_pkt_sorterPacket

SORTER

ECRC MALF MISCERROR HEADER SCOREBOARDS u_rthdr_drv

ROUTE HEADERDRIVER

u_rthdr_scbROUTE HEADER CYCLE

ACCURATE SCOREBOARD

u_rabRT

HDRARB

e-Code High Level RFM

u_cfrCTRLCFRI/F

REG

DATA SCOREBOARDS

e Link-List Model

F G E M N PORT1

F G E M N PORT2

F G E M N PORT9

16

Reference Modeling TechniquesReference Modeling Techniques

• Sub-function blocks coded as a high level units

• Communication between units done through ports

• A top-level unit binds all the sub-units through ports

• Data communicated using structs through ports

• Simulator callbacks reduced by using single event from top-level unit fanned out to sub-units

• Cycle-accurate information driven through HDL signals between RFMs

17

Reference Modeling Techniques (cont.)Reference Modeling Techniques (cont.)

• Generic data collection monitors for all speeds

• Units coded under eRM guidelines

• Design for Verification signals to avoid redundant modeling

• Hybrid modeling to reduce maintenance overhead

18

Coverage and Debug MessagesCoverage and Debug Messages

IPL RFM IPL RFM functional coverage functional coverage

19

Coverage and Debug Messages (cont.)Coverage and Debug Messages (cont.)

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: =================================

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: DUT TLP : MWr DW4_WD

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: RFM TLP : MWr DW4_WD

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: =================================

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: =============================

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: RFM POINTER 2


[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: IAP4 FULL LIST SCOREBOARD


[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: RFM DUT

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: byte byte


[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: 60 60


[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: b0 b0

[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: a a




[2712 ns] IPL_RFM_IAP_GEMN_SCOREBOARD: ff ff





IPL RFMIPL RFMdebug messages debug messages

20

• Verification Planning• Preparation

• Analyze all the relevant documents• Make sure you have the right people invited to the

meeting• Brainstorming session

• Capture non-specification coverage• Clarify specification ambiguities/issues

DUTDUT

Functional / DesignSpecifications

Cadence vPlan used for IPL VerificationCadence vPlan used for IPL Verification

21

Upstream

Ingress

Port

Upstream

Egress

Port

Scheduler

RS0 RS2 RS2 RS2 RS1 RS1

Upstream Port and Global Control Logic (GCL)

Ingress

Port

Egress

Port

Scheduler

RS3A

Internal Endpoint FunctionOHCI USB

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6

DownstreamPCIe Port 0

RR0 RR1A

RR2

GCL

Egress

Port

GCL

Ingress

Port

RS7 RS8

Ingress

Port

Egress

Port

Scheduler

RS3B

Internal Endpoint FunctionEHCI USB

RR1B RS8

RS9

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6


RR2 RS9

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6


RR2 RS9

Distributed Routers: “SuperRouter”Distributed Routers: “SuperRouter”

TLP

RouteToken

Schedulertoken

22

Router RFM: Why?Router RFM: Why?

PCI Express LiteraturePCI Express Base SpecificationPCI Express to PCI Bridge Specification

PCI LiteraturePCI Local Bus SpecificationPCI-to-PCI Bridge SpecificationPCI Bus Power Management Specification

TI Functional SpecificationsFunctional Spec for PCIe SwitchFunctional Spec for Multifunction PCIe Device

Directed test overload!

RTL SpecificationsPCI Express Switch Implementation SpecificationRouter Implementation Specification

1724 pages of specifications

+

+

+

+

23

Router RFM: Why? (cont.)Router RFM: Why? (cont.)

Specifications(1724 pages)

Verilog coding

e coding

Co-Simulation

RTL DUT

High level RFM

Interpretation Compare

Cycle accurate comparison

24

Router eRM ArchitectureRouter eRM Architecture

Upstream

Ingress

Port

Upstream

Egress

Port

Scheduler

RS0 RS2 RS2 RS2 RS1 RS1

Upstream Port and Global Control Logic (GCL)

Ingress

Port

Egress

Port

Scheduler

RS3A

Internal Endpoint FunctionOHCI USB

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6


RR0 RR1A

RR2

GCL

Egress

Port

GCL

Ingress

Port

RS7 RS8

Ingress

Port

Egress

Port

Scheduler

RS3B

Internal Endpoint FunctionEHCI USB

RR1B RS8

RS9

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6


RR2 RS9

Ingress

Port

Egress

Port

Scheduler

RS4 RS5 RS6 RS6


RR2 RS9

25

TYPE1’hdr_type

MULTIFUNCTION’modepexrtr_env_u

DOWNSTREAM’mode pexrtr_env_u

UPSTREAM’modepexrtr_env_u

TYPE0_EHCI’hdr_type

TYPE1’hdr_type TYPE0_OHCI’hdr_type

Router eRM Architecture (cont.)Router eRM Architecture (cont.)

RS0 RS2 RS2 RS2 RS1 RS1 RS3A

RS4 RS5 RS6 RS6

RR0 RR1A

RR2

RS7 RS8 RS3ARR1A

RS8

RS9 RS4 RS5 RS6 RS6RR2 RS9 RS4 RS5 RS6 RS6RR2 RS9

26

Router eRM EnvironmentRouter eRM Environment

Downstream Port Router “e” Environment

Verilog DUT

SlaveD

N Self

SlaveD

N Peer 0

SlaveD

N Peer 1

SlaveU

P Peer

Registers

e RFM dn_to_self’

slave_type

dn_to_dn’

slave_type

dn_to_dn’

slave_type

up_to_dn’

slave_type

TYPE1’

hdr_type

reg_u

TokenB

FM

TokenB

FM

TokenB

FM

TokenB

FM

TokenSB

TokenSB

TokenSB

TokenSB

ConfigBFM

Sideband SignalBFMs

DOWNSTREAM’kind TYPE1’hdr_typeTRUE’has_reference_model pexrtr_env_u

27

Router RFM Comparison LogicRouter RFM Comparison Logic

“e” RFM Token Comparison Unitpexrtr_rfm_compare_u

in_token : pexrtr_token_s;event in_token_done_e;

rfm_token : pexrtr_sch_token_s;event rfm_token_done_e;

env : pexrtr_env_u;bus_name : pexrtr_bus_name_t;slave_type : pexrtr_rfm_slave_t;

From input monitor

1) Daisy chain pre-processing sequential logic.

2) Call compare_<type>_token() based on Memory, IO, etc.

3) Daisy chain post-processing sequential logic.

To output scoreboard

Plug-in

rule sets

28

Techniques of Router RFMTechniques of Router RFM

• Cycle accurate• when inheritance plug-in rule sets• Simulation of pseudo chip-level SuperRouter• Detailed log files of each transaction• RFM includes text comments explaining expected

behavior:Bus mastering disabled, so ignore all Memory TLPs.Previously deemed Malformed. Ignore.In IO window. Blocked.Previously deemed UR.Completion is outside sec-to-sub window, and sec!=0, so claim it.Ignore all internally generated Vendor_Type1 MsgD messages.

29

Moving to chip levelMoving to chip level

• Building blocks• Multiple RFMs• 4 instances of TI PCIe eVC• Top-level register model• Top-level predictor• Top-level scoreboard

• Connect the dots• Using TLP ID• Uniquely identify every packet in the entire system for the

entire simulation

30

PCIe Port Reference ModelPCIe Port Reference Model

RTL e

IPL RFM

DL RFM

EPL RFM

IPL DUT

DL DUT

EPL DUT

Scheduler RFMScheduler DUT

RX Data

Route Token

Scheduling Token

EPL TokenD

Q T

oken

RX TLP

TLP

ID

Egre

ss T

LPTX TLP

Router RFMRouter DUT

TLP

ID

TLP

IDTL

PID

Egre

ss T

LP

TX TLPExample: Downstream Port

31

Finished Chip-Level Prediction ModelFinished Chip-Level Prediction Model

Down Port 1

Down Port 2

TIPCIe eVCEndpoint

TIPCIe eVCEndpoint

TIPCIe eVCEndpoint

Ingr

ess

TLPs

Down Port 2TLP Ingr List



Up PortTLP Ingr List

Ingr

ess

Lis

tse

arch

alg

orith

m

Up PortTLP Pred FIFO

Down Port 0TLP Pred FIFO



TLPCheck

TLPCheck

TLPCheck

TLPCheck

TLP ID from each Port Scheduler RFM

Up Ingress TLP

PCIe Switch DUT

Down Port 1RFM

Down Port 2RFM

Switch Prediction ModelCycle-accurate and

packet-accurate

Egre

ss T

LPs

ConfigurationSpace Register Model

AndCompletion Predictor

1) TLP Ingress

2) TLP Switching

3) TLP Egress

Down Port 0

Up Port

TIPCIe eVC

Root Complex

Down Port 0RFM

Up PortRFM

32

ConclusionsConclusions

• Architecture definition to facilitate Design for Verification

• RFM methodology requires dedicated and specialized Verification Engineer resources

• Rigorous block-level simulation permitted 2 days from integration to first chip-level simulation

• 2-weeks of chip-level simulations put robust FPGA build in the lab for emulation

• “SuperRouter” simulations eliminated lost/misdirected packets at chip-level

• Silicon had outstanding performance at Plug Fests and has shown no bugs in RFM features

33

Conclusions (cont.)Conclusions (cont.)

• RFM will provide dual, independent interpretation of specification

• RFM styles: Choose wisely• cycle accurate, packet accurate, hybrid

• Muscle of Specman random generation is only as good as the auto-checking features of testbench

• RFM hookup at chip-level:• Identify hookup issues, interface violations• Check for chip-level stimulus that was missed at module-

level• Chip-level debug acceleration

34

Let’s talkLet’s talk

Questions? Thoughts?

35

About the AuthorsAbout the Authors

• Asad Khan ([email protected])• Lead Design Verification Engineer for 1394 and PCI Express

products

• Scott Morrison ([email protected])• Lead Design Verification Engineer for Mixed Signal IP, including

high-speed SERDES, USB 2.0 PHY, and 1394 PHY

We would appreciate feedback. Feel free to contact us.

Technology

PCI Express Verification using Reference Modeling