PCI Express EZ IP Module - Twin Industries · 6.4.2 Power Management ... The Getting Started guide provides information to enable ... PCI Express EZ IP Module Reference

Version 1.1 October 2005Copyright © PLDApplications 1996-2005

PCI Express EZ IP ModuleReference Manual

2

PCI Express EZ IP Module Reference Manual

PCI Express EZ ModuleTechnical Reference Manual

Documentation Change History

Proprietary NoticeWords and logos marked with ® or ™ are registered trademarks or trademarks owned by PLDApplications SA.Other brands and names mentioned herein may be the trademarks of their respective owners.

Neither the whole nor any part of the information contained in, or the product described in, this document may beadapted or reproduced in any material form except with the prior written permission of the copyright holder.

The product described in this document is subject to continuous developments and improvements. All particularsof the product and its use contained in this document are given by PLDApplications (PLDA) in good faith. Thisdocument is provided “as is” with no warranties whatsoever, including any warranty of merchantability, noninfringement, fitness for any particular purpose, or any warranty otherwise arising out of any proposal,specification, or sample.

This document is intended only to assist the reader in the use of the product. PLDA shall not be liable for any lossor damage arising from the use of any information in this document, or any error or omission in such information,or any incorrect use of the product. Nor shall PLDA be liable for infringement of proprietary rights relating to use ofinformation in this document. No license, express or implied, by estoppel or otherwise, to any intellectual propertyrights is granted herein.

Product StatusThe information in this document is final content pertaining to the PLDA PCI Express EZ Module.

Web Addresshttp://www.plda.com

Date Version Number Change

October 1.1 ● Minor changes

April 2005 1.0 rev 1 ● Hot Plug support removed

April 2005 1.0 ● First publication


3

Table of Contents

Chapter 1 Introduction to the PCI Express Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

1.1 Placing PCI Express in an Historical Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.2 Understanding PCI Express Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.2.1 Lanes and Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.2.2 The PCI Express Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151.2.2.1 Root Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151.2.2.2 Endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161.2.2.3 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161.2.2.4 Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

1.2.3 Types of transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.2.4 Routing rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181.2.4.1 Routing by Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .181.2.4.2 Routing by ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .181.2.4.3 Implicit routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

1.2.5 Routing Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.2.5.1 Configuration Write transaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191.2.5.2 Memory Read transaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

1.2.6 The PCI Express protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.2.6.1 PCI Express layer architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211.2.6.2 Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211.2.6.3 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

Chapter 2 Introduction to the EZ Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

2.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.2 High-level functionality of the four-layer Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.3 EZ Layer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.4 Configuration Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5 Lane initialization and Lane reversal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Chapter 3 Architecture of the Slave Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.2 Slave Module Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Chapter 4 Architecture of the Master Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.2 Behavior of the Master Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3 Request & Completion Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3.1 Master Request Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35


4

4.3.2 Behavior of Master Completion Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3.3 Completion resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Chapter 5 DMA Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37

5.1 DMA Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.2 DMA Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.3 DMA Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.4 DMA Channel Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.5 Address / data interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.6 DMA Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.6.1 Connecting RAM devices to DMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.6.2 Connecting FIFO devices to DMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.6.3 Setting DMA Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.6.3.1 Local Address Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .415.6.3.2 Maximum DMA Transfer Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .415.6.3.3 Number of Completion Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41

Chapter 6 Core Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6.2 Slave Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.2.1 Slave Module Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.2.2 Transaction Examples using slave signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6.3 Master Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.3.1 Master Module Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.3.2 Transaction Examples using Master signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

6.4 Miscellaneous Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.4.1 EZ Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.4.2 Power Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.4.3 Miscellaneous signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Chapter 7 Physical Layer Interface Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

7.1 PIPE interface signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 8 Optimizing system performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

8.1 Measuring system performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.1.1 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.1.2 Maximum Effective Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.1.3 Actual Link Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62


5

8.2 System performance illustrated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.2.1 One outstanding request, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.2.2 Two outstanding requests, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.2.3 Four outstanding requests, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.2.4 Two outstanding requests, large packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.2.5 High latency system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Register content of TLPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66

Register content of the Configuration Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

EZ Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

6


List of Tables

PCI Express Features supported by the EZ IP Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12EZ Module-Specific features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Configuration of Switch ports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16PCI Express transaction types and characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Completion transaction characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Configuration Write transaction steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Memory Read transaction steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Example of an Endpoint’s advertised credits at and after

Link initialization and the effect on Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Lane initialization possibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29DMA Channel Register Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37DMA Channel Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38DMA Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Description of Completion Resource examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Slave Module Signals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Master Application Layer Interface Signals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49Configuration Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Power Management Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Miscellaneous interface signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58PIPE interface signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60Typical latency values of Read Request transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Maximum Effective Bandwidths for various packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Memory Read Request 32-bit addressing descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Memory Read Request-Locked 32-bit addressing descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Memory Read Request 64-bit addressing descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Memory Read Request-Locked 64-bit addressing descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . 67I/O Read Request descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Type 0 Configuration Read Request descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Type 1 Configuration Read Request descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Message (without data) descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Memory Write Request 32-bit addressing descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Memory Write Request 64-bit addressing descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Completion (without data) descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Completion Locked (without data) descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69I/O Write Request descriptor format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Type 0 Configuration Write Request descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Type 1 Configuration Write Request descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Completion (with data) descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Completion Locked (with data) descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Message (with data) descriptor format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Common Configuration Space Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Type 0 Configuration Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Message Signaled Interrupt (MSI) Capability Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Power Management Capability Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75PCI Express Capability Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Virtual Channel Capability Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75EZ Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77


7

List of Figures

A PCI Express Lane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14PCI Express x4 Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14A Typical PCI Express Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Tracing a Write Transaction through the fabric (numbers correspond to numbered steps in Table 6.) 19Tracing a Read Transaction through the fabric (numbers correspond to numbered steps in Table 7.) 20Flow Control through Virtual Channels (VCs) and Traffic Classes (TCs) . . . . . . . . . . . . . . . . . . . . . . . 22Flow Control through a single Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Receive buffers for a Virtual Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22PCI Express EZ Module Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26The four layers of the EZ Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Block diagram of the PCI Express EZ layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Slave Module Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Waveform illustrating immediate programming of the DMA registers . . . . . . . . . . . . . . . . . . . . . . . . . 32Waveform illustrating delayed programming of the DMA registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Architecture of the Master Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Behavior of Master Request Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Behavior of Master Completion Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36DMA registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37DMA Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38DMA Channel control behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39RAM connection to the DMA interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40FIFO connection to the DMA interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Potential Completion resource / Channel configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41I/O diagram of the EZ Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Waveform illustrating a typical Write request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Waveform illustrating a typical Read request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Waveform illustrating an aborted Write request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Waveform illustrating an aborted Read request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49Waveform illustrating a typical transfer of a Read Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52Waveform illustrating 4 DW DMA Write transaction with 2 clock cycles latency . . . . . . . . . . . . . . . . . 52Waveform illustrating 4 DW DMA Write transaction with 2 clock cycles latency . . . . . . . . . . . . . . . . . 53Waveform illustrating Writing to and Reading from the DMA registers . . . . . . . . . . . . . . . . . . . . . . . . 54Power Management Capabilities Register . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57One outstanding request, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Two outstanding requests, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Four outstanding requests, small packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Two outstanding requests, large packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64High latency system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65PLDA Core Status Register . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

8


Preface

The preface describes the contents of this document and suggests additional reading. It contains the followingsections:

● About this document● Additional Reading● Feedback and contact information


9

About this document

Intended audienceThis document has been written for design managers, system engineers, and designers of ASICs and FPGAs whoare evaluating or using the PLDA PCI Express EZ Module.

ScopeThis document provides the complete functional description of the PLDA PCI Express EZ Module.

Using this manualThis document includes the following chapters:

Chapter 1: Introduction to the PCI Express Protocol

Chapter one provides an introduction to the PCI Express protocol.

Chapter 2: Introduction to the EZ Module

Chapter two provides an introduction to the PCI Express EZ Module.

Chapter 3: Architecture of the Slave Module

Chapter three describes the architecture and behavior of the Slave Module.

Chapter 4: Architecture of the Master Module

Chapter four describes the architecture and behavior of the Master Module.

Chapter 5: DMA Channels

Chapter five describes the DMA channels included in the Core.

Chapter 6: Core Signals

Chapter six describes all Core signals and offers wave diagrams to illustrate their behavior.

Chapter 7: Physical Layer Interface Signals

Chapter seven describes Physical Layer signals.

Chapter 8: Optimizing system performance

Chapter eight provides hints, and concrete examples, for optimizing system performance.

Typographical conventions

italic Highlights important notes or publications

bold Highlights interface elements.

Times New Roman Denotes text used in a code example.

COURIER NEW DENOTES A SIGNAL.

10


Additional ReadingThis section lists additional resources from PLDA and third-parties.

PLDA periodically updates its documentation. Please contact PLDA at [email protected] or check the Web site athttp://www.plda.com for current versions.

PLDA publicationsPlease refer to the following documents for further information:

● Build History: The Build History lists changes made in each version and build of the Core.● Getting Started: The Getting Started guide provides information to enable designers to integrate the PLDA

PCI Express EZ Module into their design flow as quickly as possible (installing, customizing, integrating, andsimulating the Core).

Other publicationsPlease refer to the following documents for further information:

● PCI Express™ Base Specification Revision 1.0a


11

Feedback and contact information

Feedback about this documentPLDA welcomes comments and suggestions pertaining to this documentation. Please contact PLDA [email protected] and provide the following information:

● the title of the document● the page number to which your comments refer● a description of your comments

Contact informationPLDApplications Inc.

2750 North First St.San Jose, CA 95131

Telephone: 408-273-4530

Corporate Headquarters

PLDApplicationsEuroparc Pichaury A2 - 1330, rue Guillibert 13856 Aix-en-Provence Cedex 3 - France

Telephone: +33 442 393 600Fax: +33 442 394 902

http://www.plda.com

Sales

For sales questions, please contact [email protected]

Technical Support

For technical support questions, please contact [email protected]

General

For all other questions, please contact [email protected].

12


Table 1 describes PCI Express features supported by the EZ Module.

Table 2 describes additional features unique to the EZ Module.

Table 1: PCI Express Features supported by the EZ IP Module

General ● Core type: x1 or x4 legacy Endpoint● Maximum Payload: 256 bytes● Backend data path: 64-bit or 32-bit● Virtual Channels: 1

Configuration ● BARs, Expansion ROM: User-defined, set by the EZ Wizard● PCI ID: User-defined, set by the EZ Wizard● Legacy power management: Minimal or full, set by the EZ Wizard

Message SignaledInterrupt (MSI)

● Message count: 1● 64-bit address: Yes● Per-vector masking: No

PCI ExpressCapabilitiesStructure

● Phantom function: not supported● Extended tag: not supported● Advanced Error Reporting (AER): not supported

Table 2: EZ Module-Specific features

General ● DMA channels: up to 8 (set with the EZ Wizard)● Number of outstanding requests: 1/2/4/8 simultaneous requests (set with EZ Wizard)● Local Address Size: 13 to 32 bits (set with the EZ Wizard)● Maximum DMA transfer size: 8 KB (13 bits) to 4 GB (32 bits)

Introduction to the PCI Express Protocol PCI Express EZ IP Module Reference Manual

13

Chapter 1 Introduction to the PCI Express Protocol

1.1 Placing PCI Express in an Historical ContextAs the name implies, PCI Express owes much to its predecessors, PCI and PCI-X. However, PCI Expressrepresents an architectural leap beyond its sister technologies rather than simply an extension of the existingnorms.

PCI Express is a third generation, high-bandwidth, low voltage, differential serial interconnect technology thatmaintains compatibility with existing PCI infrastructures.

In order to place PCI Express within a historical context, the three technologies are briefly summarized below.

PCIIntroduced in the early 1990s, the PCI Bus (Peripheral Component Interconnect) offered a faster and unifiedalternative to fragmented ISA technologies, and included the following features:

● A processor-agnostic I/O standard, which allowed for adoption of a single standard across a wide range ofapplications

● Plug and Play operation, so that hardware and devices could be automatically detected and configured● A Parallel Interface● Parallel bus signaling at 33 megahertz (more recently 66 megahertz)

PCI-XPCI-X was introduced in 1999 with an increased Bus frequency of 66 megahertz and higher. PCI-X is a truederivative of PCI. But despite the similarities with its predecessor, only the server market adopted the technology,which is less sensitive to physical space constraints (constraints imposed by connector size and signal routingspace) than the mobile or desktop markets.

PCI ExpressWhile processor and memory frequencies have increased dramatically in the last decade, I/O technologies havenot. Despite the introduction of PCI-X and improvements within PCI itself, the technology is simply inadequate tomeet today’s I/O bandwidth needs. PCI Express is designed to close the gap and offer the I/O infrastructure uponwhich the majority of future systems will be based.

PCI Express includes the following features:● Increased bandwidth: high-speed links can currently transfer up to 2.5 gigabits per second, and future

implementations of PCI Express will support as much as 10 gigabits per second● Ιsochronous transactions: With the introduction of Virtual Channels (VCs) and Traffic Classes (TCs), the

designer can prioritize traffic flow● Serial point-to-point interface● High bandwidth per pin● Scalability● Support for differentiated services, i.e., different Qualities of Service (QoS)● Power Management and budgeting● Hot-Plug and Hot-Swap support● Ability to maintain Link-level and end-to-end data integrity● Advanced Error handling

14

PCI Express EZ IP Module Reference Manual Introduction to the PCI Express Protocol

1.2 Understanding PCI Express Fundamentals

1.2.1 Lanes and LinksFigure 1 illustrates a single lane, the fundamental connection between two devices.

Figure 1: A PCI Express LaneEach lane is defined by a pair of differential transmit signals and a pair of differential receive signals. Note thatunlike PCI, there is no arbitration for a shared Bus, as the Transmitting device is always directly connected with aReceiving device and each device is always both a transmitting device and a receiving device.

A PCI Express Link is defined as the connection between two devices and includes one or more lanes.

The PCI Express™ Base Specification Revision 1.0a defines the following configuration of serial Links: x1, x2, x4,x8, x12, x16, and x32. Figure 2 illustrates a x4 Link.

Figure 2: PCI Express x4 LinkPCI Express maintains compatibility with PCI software by simulating a PCI bus across each Link. PCI ExpressLinks connect components within the fabric, as explained in the next section.

Device A Device B

TxT+T-

T-T+

Rx

R+

R-

R-

R+

RxTx

Lane

Device BDevice ALane

Link


15

1.2.2 The PCI Express FabricFigure 3 illustrates a typical PCI Express fabric (or topology).

Figure 3: A Typical PCI Express Fabric

1.2.2.1 Root ComplexIn the example (Figure 3), the Root Complex is composed of one Root Complex Register Block (RCRB)associated with the sink unit (entry to the PCI Express fabric hierarchy) and two Root Ports. The sink unit and thetwo Root Ports constitute a virtual PCI bus segment that has Bus Number 0. This means that the Root Compleximplements a virtual embedded switch between the Root Ports and transfers are allowed directly between them(this is an optional feature of the Root Complex).

The PCI configuration software programs each Root Port as a Type 1 Configuration space. In the example, the firstRoot Port is Bus 0, Device 1 and the second Bus 0, Device 2.

The sink unit represents the start of the PCI Express fabric and is equivalent to a virtual PCI host port that cangenerate all transaction types (Memory, I/O, Completion, and Message) and also relay Request and Completiontransactions used to access the central memory.

Root Port Device 1 is connected to an Endpoint through virtual PCI Bus Segment 1 and has the following Type 1Configuration Space parameters:

● Primary Bus Number 0● Secondary Bus Number 1● Subordinate Bus Number 1 (only one bus number is located downstream of the root port)

Root Port device 2 is connected to a PCI Express Switch and has the following configuration parameters:● Primary Bus Number 0 ● Secondary Bus Number 2● Subordinate Bus Number 7 (bus segments 2 through 7 are located behind the Root Port)

CPU

Root Complex

MemoryRCRB

Virtual PCI Bus 0

Root Port

Virtual PCI Bus 1 Virtual PCI Bus 2

Virtual PCI Bus 3 Virtual PCI Bus 4

Virtual PCI Bus 5

PCI/PCI-X

Brid

geRoot Port

sink

Sw

PSwP

SwP

Sw

P

Switch

End-point

Real PCI Bus 7

Virtual PCI Bus 6

End-point

End-point

Switch Downstream PortType 1 Conf SpaceDev3 PRB3 SEB6 SUB7

PCI Express / PCI-X BridgeType 1 Conf SpaceDev0 PRB6 SEB7 SUB7

PCI or PCI-X BoardType 0 Conf SpaceDevX Bus 7

Endpoint (low latency)Type 0 Conf SpaceDev0 Bus1

Root Port (Downstream)Type 1 Conf SpaceDev1 PRB0 SEB1 SUB1

Root Complex Register BlockDev0 Bus0

Endpoint (medium latency)Type 0 Conf SpaceDev0 Bus5



Endpoint (medium latency)Type 0 Conf SpaceDev0 Bus4

Switch Upstream PortType 1 Conf SpaceDev0 PRB2 SEB3 SUB7

Root PortType 1 Conf SpaceDev2 PRB0 SEB2 SUB7

16


The Endpoint component located behind Root Port Device 1 is a low latency Endpoint, for example, a graphicboard where latency is critical for optimal performance. An Endpoint device number is always 0 and its Bus numberis programmed by the PCI configuration software (Bus 1, in this example).

1.2.2.2 EndpointAll PCI Express Endpoints necessarily implement a Type 0 Configuration Space and have Device number 0. In theexample (Figure 4), the two Endpoints located behind the Switch have Bus Numbers 4 and 5. These areconsidered medium latency components due to the fact that they are behind a Switch component and not directlyconnected to the Root Complex. In general, a PCI Express Endpoint component is considered to have high latencyif two or more Switch components lie between it and the Root Complex.

The PCI Endpoint components located behind the Bridge are also implemented as Type 0 Configuration spacecomponents and respond only to Type 0 Configuration access requests (if their PCI_IDSEL pin is asserted).

1.2.2.3 SwitchIn the example (Figure 3), the PCI Express Switch component implements four PCI Express ports, one upstreamand three downstream. Each port is equivalent to a PCI Bridge and implements a Type 1 Configuration space.Internal routing between ports is accomplished by a virtual PCI Bus segment.

Table 3 describes the configuration of each Switch port in Figure 3.

1.2.2.4 BridgeThe PCI Express / PCI Bridge component implements a common Type 1 Configuration space for both sides of thebridge: the PCI Express Core and the PCI component Core. The mechanism to handle transactions between thetwo is described in the PCI Express™ Base Specification Revision 1.0a. In Figure 3, the Bridge has thefollowing parameters:

● Device number 0 (mandatory)● Primary Bus Number 6 (PCI Express Core Link)● Secondary Bus Number 7 (PCI bus)● Subordinate Bus Number 7 (no other PCI / PCI Bridges are located on the PCI bus)

1.2.3 Types of transactionsTable 4 summarizes the different types of transactions possible in the fabric.

Table 4: PCI Express transaction types and characteristics

Table 3: Configuration of Switch ports

Device Number 0 (upstream) 1 (downstream) 2 (downstream) 3 (downstream)

Primary BusNumber

2 3 3 3

Secondary BusNumber

3 4 5 6

Subordinate BusNumber

7 4 5 7

Transaction Type

How request ishandled

Routing method Notes

Memory Write Posted (noCompletion isrequired)

Address Memory Write Requests use Posted TLPs andcan have data payloads of up to 4 Kbytes(depending on the Max_Payload_Sizeparameter of the Configuration Space).

Memory Write Requests can use 32-bit addressformatting or 64-bit address formatting.


17

Completion transactionsUnlike PCI, a component in the PCI Express fabric generates a distinct package, a Completion TLP, in response toa Read or Write Request. Like Configuration Requests, Completion transactions are routed by Transaction ID.

Completion transactions vary in nature depending on the reason for their generation. The following table describesdifferences in Completion transactions.

Memory Read Non-Posted(Completion isrequired)

Address Memory Read Requests use Non-Posted TLPsand have no payload. A dedicated register of theconfiguration space defines the maximum ReadRequest size. Please see the following sectionfor details concerning Completion transactions.Memory Read Requests can use 32-bit addressformatting or 64-bit address formatting.

I/O Write Non-Posted Address I/O Request transactions use Non-Posted TLPs.I/O Write Request transactions have a datapayload of 1DW. I/O Requests always require aCompletion packet, whether the request isreturned successful or aborted. Please see thefollowing section for details concerningCompletion transactions.

I/O Read Non-Posted Address

ConfigurationWrite

Non-Posted ID Configuration Requests use Non-Posted TLPs.Configuration Write requests have datapayloads of 1 DW. Each Configuration requestrequires a Completion regardless of whether therequest was accepted. Please see the followingsection for details concerning Completiontransactions.

ConfigurationRead

Non-Posted ID

Message Posted Address In addition to the three PCI transactions(Memory, I/O, and Configuration), PCI Expressintroduces a fourth type of transaction forMessages. Message transactions supportpower management requests and emulate PCIlegacy virtual pins (such as INT, PME, PERR,and SERR).

Table 5: Completion transaction characteristics

Completion transactions generated in response to a... have these characteristics...

Memory Request Completion transactions are generated for Memory Read Requests but not Memory Writerequests, since no completion is required for the latter.

Completions can have data payloads of up to 4 Kbytes, depending on the MaximumPayload Size. Note that large Read Request transactions (transactions of 4 Kbytes, forexample) might be divided into several smaller Completion Packets (64, 128, or 256 byteson a 64 or 128 address boundary) in order to reduce overall latency and to optimize dataflow within each Switch. Completion transactions may only be divided if they have an endaddress corresponding to the Read Completion Boundary (RCB) parameter, which is always128 bytes for Endpoints and 64 or 128 bytes for the Root Complex.

Note that Completion packets can only be divided into multiples of 64 bytes.

Note also that the Core does not check for violations of the Read Completion Boundary(RCB).

Transaction Type

How request ishandled

Routing method Notes

18


1.2.4 Routing rulesThe PCI Express fabric uses the same type of routing functionality as the PCI fabric.

● All Memory and I/O transactions are routed by address (BAR decoding for Endpoint Type 0 configurationspaces, and Memory Mapped I/O for Root Port, Switch, and Bridge Type 1 Configuration spaces).

● Configuration, Message, and Completion transactions are routed by the Transaction ID of the requestingcomponent (Bus, Device, and Function number) and comply with Type 1 Configuration Space routingspecifications.

Message transactions, introduced with PCI Express, are independent of the PCI configuration software and havedifferent types of routing. Please see Section 1.2.4.3 for details.

The header of a TLP includes information about the type of the transaction and whether the transaction is routedby address or ID.

1.2.4.1 Routing by AddressTransactions that are routed by address compare the destination address (included in the TLP header) with theregisters of the component.

Type 0 components (Endpoint)If the address of the transmitted TLP lies within one of the component’s implemented Base Address Registers(BARs), the component will accept the TLP. If the address of the TLP does not lie within the component’s BARs,the TLP is rejected and an error is generated.

Type 1 components (Root Port, Switch, or Bridge)● If a device receives a request on its Primary Bus and the destination address is located within one of the

defined address windows (as defined by its registers), the request is forwarded to the Secondary Bus.● If a device receives a request on its Secondary Bus and the destination address is located outside of the

defined address window, the request is forwarded to the Primary Bus.

1.2.4.2 Routing by IDRegardless of the type of PCI Express component, when a Type 1 Configuration Space port receives a TLP on itsPrimary Bus, the following rules apply:

● If the selected Bus Number is equal to its Secondary Bus number, the Type 1 Configuration request istransformed into a Type 0 Configuration request according to the standard PCI rules.

● In all other cases, the Type 1 Configuration request remains a Type 1 Configuration request and istransferred to the Secondary Bus (internal or external).

● If the selected Bus Number does not fall between the Secondary Bus number and the Subordinate Busnumber, the transaction is discarded.

I/O Request Completion transactions are generated for both Read and Write I/O requests.

Completion packets generated in response to I/O Write requests have no data payload,whereas a completion packet in response to an I/O Read request has a data payload of 1DW. I/O Requests always require a Completion transaction regardless of return status(accepted, successful, unsupported, aborted, or retry).

Configuration Request Completion transactions are generated for both Read and Write Configuration requests.

Completion packets have data payloads of 1 DW if the request is accepted. ConfigurationRequests always require a Completion transaction regardless of return status (accepted,successful, unsupported, aborted, or retry).

Table 5: Completion transaction characteristics

Completion transactions generated in response to a... have these characteristics...


19

1.2.4.3 Implicit routingMessages do not use Address or ID routing, but rely on other mechanisms referred to as “implicit” because thedestination is implied by the routing type (please see section 2.2.8 of the PCI Express™ Base SpecificationRevision 1.0a for more information). Implicit routing depends upon the inherent knowledge a PCI Expresscomponent has relating to upstream and downstream transmissions within the fabric.“Implicit” Routing types include the following:

● Routed to Root Complex● Broadcast from Root Complex● Local (Terminate at Receiver)● Gathered and routed to Root Complex● Reserved (Terminate at Receiver)

1.2.5 Routing ExamplesThe following section outlines two hypothetical transactions that correspond to Figure 3.

1.2.5.1 Configuration Write transactionConfiguration transactions use ID routing (routing based on the Bus, Device, and Function numbers of thedestination component).

Table 6: Configuration Write transaction steps

Step Component ID Description

1 Sink Unit /CPU

Dev0 Bus0 The CPU instructs the sink unit to generate aConfiguration Write transaction directed to BusNumber 3, Device 2.

Figure 4: Tracing a Write Transaction through the

fabric (numbers correspond to numbered

steps in Table 6.)

2 Root Port Dev2 PRB0SEB2 SUB7

The Root Port receives the transaction anddetermines that the destination Bus (3) is locatedwithin its Bus Address window (the range defined byits Secondary Bus number and Subordinate Busnumber, in this case 2-7). It accepts the transactionin order to forward it to its Secondary Bus. The targetBus (3) is not equal to the Secondary Bus (2), andthe transaction remains a Configuration Type 1transaction.

3 SwitchUpstream Port

Dev0 PRB2SEB3 SUB7

The Switch’s Upstream Port receives the transactionand determines that the destination Bus (3) islocated within its Bus Address window and acceptsthe transaction in order to forward it to its SecondaryBus (the Switch’s internal virtual PCI bus). The targetBus (3) is equal to the Secondary Bus (3), and theType 1 Configuration transaction is transformed intoa Type 0 Configuration transaction.

4 SwitchDownstreamPort

Dev2 PRB3SEB5 SUB5

Each downstream port in the Switch receives the Type 0 Configuration transactionon virtual Bus 3 and decodes the Device number (2). Port Device 2 recognizes itselfas the Completer, accepts the transaction, and generates a Completion TLP using aTransaction ID (Bus 0, Device 0).


Dev0 PRB2SEB3 SUB7

The Completion TLP is accepted by the Secondary Bus of the Switch’s upstreamport, which determines that the target Bus Number (0) does not lie within its BusAddress window, and the transaction is forwarded to its Primary Bus.


Likewise, the Root Port forwards the transaction back to Virtual Bus 0.

7 Sink Unit /CPU

Dev0 Bus0 Finally, the sink unit accepts the packet and passes it back to the CPU.

1 / 7

2 / 6

4

3 / 5

20


1.2.5.2 Memory Read transactionSteps 1-6 of the following example describe a Memory Read request, which uses address routing (routing basedon either a 64-bit format associated with 4 DW headers or a 32-bit format associated with 3 DW headers). For thisexample, it is assumed that address windows are correctly defined for each Type 1 Configuration Spacecomponent and that the Completer Endpoint Base Address Register (BAR) is correctly defined.

Steps 7-14 describe routing of the Completion transaction generated in response to the Memory Read request. Asin the first example (Configuration transaction), Completion transactions use ID routing (routing based on the Bus,Device, and Function numbers of the destination component).

Table 7: Memory Read transaction steps

Step Component ID Description

1 Endpoint Dev0 Bus1 The Endpoint component located behind Root PortDevice 1 generates a Memory Read request directedto the Endpoint located behind the Switch’s Portdevice 2.

Figure 5: Tracing a Read Transaction through the

fabric (numbers correspond to numbered

steps in Table 7.)


The Secondary Bus of the Root Port (Device 1)receives the transactions, determines that theaddress lies outside of its address window, andforwards the transaction through its Primary Bus tothe Virtual PCI Bus 0.


The Primary Bus of the Root Port (Device 2)determines that the destination address lies within itsprefetchable address window and forwards thetransactions through its Secondary Bus to the Switch.


Dev0 PRB2SEB3 SUB7

The Primary Bus of the upstream port of the Switchdetermines that the destination address lies within itsaddress window and forwards the transactionsthrough its Secondary Bus to the virtual PCI Bus 3.


Dev2 PRB3SEB5 SUB5

The Primary Bus of downstream port Device 2 of the Switch determines that thedestination address lies within its address window and forwards the transactionsthrough its Secondary Bus to the virtual PCI Bus 5.

6 Endpoint Dev0 PRB5 The Endpoint (the Completer, in this example) receives the transaction anddetermines that the address corresponds to one of its BARs and accepts therequest. The Endpoint responds to the request and generates one or moreCompletion transactions to send data back to the requester.

Steps 7-14 describe routing of the Completion transaction generated in response to the Memory Read request.


Dev2 PRB3SEB5 SUB5

Completion transactions use ID routing, and in this example target the requestingcomponent (Endpoint Bus1 Dev0). The Secondary Bus of the downstream portDevice 2 determines that the destination Bus (1) lies outside of its Bus window (5-5)and forwards the transaction through its Primary Bus to the virtual PCI Bus 3.


Dev0 PRB2SEB3 SUB7

The Secondary Bus of the upstream port determines that the destination Bus (1) liesoutside of its Bus window (3-7) and forwards the transaction through its Primary Busto the Root Port.


The Secondary Bus of the Root Port determines that the destination Bus (1) liesoutside of its Bus window (2-7) and forwards the transaction through its Primary Busto the virtual PCI bus 0.


The Primary Bus of the Root Port determines that the destination Bus (1) lies withinits Bus window (1-1) and forwards the transaction through its Secondary Bus to theEndpoint.

11 Endpoint Dev0 Bus1 The Endpoint recognizes itself as the destination component, verifies the tag of thetransaction, and processes the transmitted information.

1 / 11

2 / 10 3 / 9

4 / 8

5 / 7

6


21

1.2.6 The PCI Express protocolThe following section provides an overview of the PCI Express protocol, including:

● PCI Express Layer architecture● Flow Control

● Traffic Classes (TCs)● Virtual Channels (VCs)● TCs and VCs● Receive buffer● Flow Control credits● Deadlock avoidance

● Error Handling● Error Recovery● End-to-End Cyclic Redundancy Check (ECRC)● Retry buffer size● Completion time-out

1.2.6.1 PCI Express layer architectureThe PCI Express protocol is based on three layers:

● Transaction layer: The Transaction Layer is primarily responsible for the assembly and disassembly ofTransaction Layer packets (TLPs) and for the storage of configuration information. It also converts receivedCompletion packets into data payloads, updates status information, and is responsible for flow controlservices, ordering rules, and power management services.

● Data Link Layer: The Data Link Layer of a component (along with its counterpart on the other side of a PCIExpress Link) is responsible for Link management, including: TLP acknowledgement, a retry mechanism incase of a non-acknowledged packet, flow control across the Link (transmission and reception), powermanagement, CRC generation and CRC checking, error reporting, and logging.

● Physical Layer: The Physical Layer is responsible for power management, width and lane negotiation,reset/hot-plug control, 8-bit/10-bit encoding/decoding, scrambling/de-scrambling, embedded clock tuningand alignment, transmission and reception circuit, and elastic buffer and multi-lane de-skew on the receivingside. The frequency is scalable up to 10 Gbps per Lane, the maximum frequency of a Printed Circuit Board(PCB), however, only 2.5 Gbps per Lane is currently defined.

1.2.6.2 Flow Control

Traffic Classes (TCs)TCs allow for differentiated services, permitting you to prioritize the flow of certain data through the fabric. A Linkmust implement at least one TC (TC0) and can implement up to eight TCs, depending on your design.

Virtual Channels (VCs) VCs allow for multiple independent paths of data flow over a single Link. A Link must implement at least one VC(VC0) and can implement up to eight VCs, according to your design.

The number of VCs initialized across a Link has no relation to the number of lanes implemented by the Link. Forexample, a x1 component could have eight VCs, and a x4 component could have one VC. If two components on either side of a Link implement a different number of VCs, only the number of VCs the twocomponents have in common are initialized (see Figure 7).

TCs and VCsThe concept of TCs associated with VCs facilitate data flow in the fabric, allowing you to determine whatpercentage of a particular Link should be devoted to a particular kind of data transfer. This, in turn, helps to avoidcongestion and permits isochronous traffic.

22


Figure 6 illustrates the function of Virtual Channels and Traffic Classes in the fabric.

Figure 6: Flow Control through Virtual Channels (VCs) and Traffic Classes (TCs)Note that Figure 6 does not specify how many VCs are implemented per component. Consider Link 2, enlarged inFigure 7.

Figure 7: Flow Control through a single LinkThe Endpoint component might also have VC1 and VC2 implemented, but the Link only initializes the number ofcommon VCs shared across a Link.

Receive bufferEach component commits a certain number of resources (determined by the user) to the Receive buffer. TheReceive buffer is located in the Transaction Layer and accepts incoming TLPs from the Link and then sends themto the Application Layer for processing. Receive buffer resources are either implemented per VC or per Link.Figure 8 enlarges VC3 from Figure 6 and illustrates the various buffers that, taken together, make up the Receivebuffer.

Figure 8: Receive buffers for a Virtual ChannelThe Receive buffer stores TLPs based on the type of a transaction, not the TC of a transaction. Types oftransactions include Posted transactions, Non-Posted transactions, and Completion transactions.

TC[0:1]VC0VC1VC2VC3

TC[2:4]TC[5:6]TC7

Root Complex

TC[0:1]TC[2:4]TC[5:6]

TC7

MultiplexingMapping

TC[0:1]TC7

TC[0:1]TC[2:4]TC[5:6]TC7

Switch

VC0VC1VC2VC3

TC[0:1]TC7

VC0VC1VC2VC3

VC0VC3

TC[0:1]TC[2:4]TC[5:6]

TC7

Endpoint

Endpoint

Link 1

Link 2

Link 3

TC[0:1]

TC7

Switch

TC[0:1]

TC7

VC0

VC3

Endpoint Link 2

VC1VC2

Link 2, VC3 (from the example above)

RxTx Rx

Tx

PH PD NPH NPD CPLH CPLDPH PD NPH NPD CPLH CPLD

Endpoint Switch

Endpoint Receive buffer Switch Receive buffer


23

A transaction always has a header but does not necessarily have data. The Receive buffer accounts for thisdistinction, maintaining separate resources for the header and data of each type of transaction. To summarize,distinct buffer resources are maintained per initialized VC for each of the following elements:

● Posted transactions, header (PH)● Posted transactions, data (PD)● Non-Posted transactions, header (NPH)● Non-Posted transactions, data (NPD)● Completion transactions, header (CPLH)● Completion transactions, data (CPLD)

Note that the Receive buffer levels on one side of a Link have no relation to the Receive buffer levels on the otherside of a Link.

The size of the Receive buffer has a significant impact on system performance. The smallest possible size is theMaximum Payload Size (Max_Payload_Size), but the Receive buffer is typically set to at least four times theMaximum Payload Size for the following reasons:

● so that it can store a maximum-sized TLP in the buffer and forward a second TLP to the Application Layer● so that it can handle multiple Completions of minimum payload size. For example, a component set to

handle 16 outstanding requests, each of which might have four corresponding Completion packets, wouldneed a minimum Receive buffer size of 16 X 4 = 64 DW.

Flow Control creditsA component advertises Buffer space availability with Flow Control credits. FC credits are maintained for each ofthe six Receive buffers and are transferred using FC Packets, a type of Data Link Layer Packet (DLLP).

The transmitting side of a Link will not send a transaction if the receiving side hasn’t advertised enough FC credits(header and data credits) for that particular type of transaction. Table 8 offers an example of advertised FC creditsfor an Endpoint component.

The unit of a single FC credit differs between header and data:

● Header (maximum-sized header + digest1)● 4 DWs for Completion transactions● 5 DWs for Request transactions

● Data: 4 DWs (16-bytes aligned)

Table 8: Example of an Endpoint’s advertised credits at and after Link initialization and the effect on Flow Control

Type of FC credit

Advertised Credits atLink initialization

Advertised Credits atLink initialization + n

clock cyclesTransmission permitted...

Posted Header 16 0 No: Sufficient credits for bothheader and data must beadvertised before a packet istransmitted.

Posted Data 128 96

Non-PostedHeader

16 16 Yes

Non-PostedData

16 16

CompletionHeader

infinite infinite Yes: Please see the followingsection, Deadlock avoidance, foran explanation of infinite credits.

Completion data infinite infinite

1. The TLP digest is the end-to-end CRC located just before the LCRC at the end of the TLP. It is signaled with the TD bit set to 1 in the header.

24


FC Credits are initialized for each VC with maximum credits and then updated periodically as TLPs are extractedfrom the Receive buffer of the receiving side of a Link. Note that one DLLP can update FC Credits associated withone or more TLPs.

Deadlock AvoidanceAs in all Switch and Bridge architectures, deadlock occurs when a component can only proceed when one of itsinternal resources is freed. PCI Express prevents deadlock with two complementary mechanisms:

● Reordering of TLPs: Component resources are freed when Posted Write requests are executed or when acomponent receives the final Completion TLP in response to a Read request. More specifically, Non-Postedrequests must give priority to Posted and Completion transactions.

● Infinite credits: Endpoints and Root ports must advertise infinite Completion credits in order to preventdeadlock. To do so, they are not allowed to initiate Read Request transactions if they do not have sufficientCompletion buffer space in order to store the maximum number of Completion transactions that might begenerated in response to the Read Request.

Note: Other reordering rules govern interactions between Posted requests, Non-Posted requests, and Completiontransactions depending on the Traffic Class (TC) and on the “relaxed ordering” bit of the TLP. These additionalreordering rules have no effect on deadlock avoidance but can facilitate the flow of global traffic within a fabric.

1.2.6.3 Error Handling

Error RecoveryThe Data Link Layer handles error recovery. It is based on 32-bit Cyclic Redundant Check (CRC) error detection,TLP sequence number, retry buffer, and ACK/NAK DLLP exchange. A typical error check might transpire asfollows:

● Step 1 Transmitter: On the transmit side of the Link, the Data Link Layer adds a sequence number to the TLPand generates a 32-bit CRC in order to protect the complete packet. The transmitter also stores the sentpacket in its retry buffer.

● Step 2 Receiver: On the receive side of the Link, the Data Link Layer verifies the CRC and the sequencenumber and generates an Acknowledgement / Negative Acknowledgement (ACK/NAK) DLLP in order toreport to the transmitter if the packet has been correctly received.

● Step 3 Transmitter: If the received ACK/NAK DLLP is negative (NAK), the transmitter must end its currenttransmission and re-send the TLP that caused the NAK response. If, on a second try, the packet issuccessfully transmitted (ACK DLLP), the TLP is purged from the Retry buffer. If the transmitter has notreceived an ACK reply after three retries, the Link is directed to recovery, i.e. retraining of the Link.

Note: DLLPs are also protected by a 16-bit CRC, but no acknowledge or error recovery mechanism exists for thischeck. Instead, DLLPs are periodically generated in order to ensure that the proper information is transmitted.

End-to-End Cyclic Redundancy Check (ECRC)PCI Express maintains data integrity across two Endpoints of the fabric with the TLP digest (also called ECRC).The TD bit (TLP Digest bit) of the TLP header indicates whether the TLP includes a TLP digest.

Switch devices change packet sequence numbers between ingress and egress ports and can introduce errors asthey recalculate the CRC. The TLP digest is an optional feature used in high reliability systems to check forintroduced errors.

Retry BufferThe Retry buffer, located in the Data Link Layer and common to all VCs, stores a copy of a transmitted TLP untilthe transmitted packet is acknowledged by the receiving side of the Link. Each stored TLP includes the Header, anoptional data payload (of which the maximum size is determined by the Maximum Payload Size parameter), anoptional ECRC, the sequence number, and the LCRC field.

The receiving side of the Link acknowledges reception of a TLP with transmission to the transmitting side of anACK DLLP. In the case of CRC error on the receiving side, a NAK DLLP is sent to the transmitting side, whichretrieves the TLP from the Retry buffer and sends it again.

The user is responsible for setting the size of the Retry buffer, which should be of a sufficient size so that TLPtransmission is not delayed because of a full buffer. Retry buffer resources are only freed upon reception of anACK DLLP, which means that Link latency (associated with DLLP transmission and/or implementation of the


25

Physical Layer) affects the ideal size of the Retry buffer. In general, the Retry buffer size should be at least twicethe maximum TLP size.

Completion TimeoutEndpoint and Root Port components use a timeout mechanism (which is design-specific) for failed Read Requesttransactions in order to report errors to the Root Port (using Error Messages) and to free Completion resources.

26

PCI Express EZ IP Module Reference Manual Introduction to the EZ Module

Chapter 2 Introduction to the EZ Module

The PCI Express EZ Module is tailored for designers who have little or no experience with the PCI ExpressProtocol or experienced designers looking for a robust yet simple PCI Express interface.

2.1 System ArchitectureThe EZ Module is PCI Express™ Base Specification Revision 1.0a Compliant and includes the Physical, DataLink, and Transactions Layers, and the EZ Layer, as illustrated in Figure 9.

Figure 9: PCI Express EZ Module StructureNote that the EZ Module is designed to minimize the need to understand the complexities of the PCI ExpressProtocol. To this end, only the EZ Layer is accessible to the designer and the three layers mandated by the PCIExpress Protocol (Physical, Data Link, and Transaction Layers) can be considered a black box within the system.

PCI ExpressLink

Application Layer

EZ Interface

PhysicalLayer

Data LinkLayer

TransactionLayer

EZLayer*

PCI Express EZ Module

* Only layer of Core accessible to the designer

Introduction to the EZ Module PCI Express EZ IP Module Reference Manual

27

2.2 High-level functionality of the four-layer CoreFigure 10 broadly describes the roles of each layer of the Core.

Figure 10: The four layers of the EZ Module

Physical Layer Data Link Layer Transaction Layer

DMA

Towards Application LayerTowards Link

With informationsent by theApplication Layer,the TransactionLayer generatesa TLP, whichincludes a headerand, optionally, adata payload.

The Data LinkLayer ensurespacket integrity,and adds asequencenumber and LinkCyclicRedundancyCheck (LCRC) tothe packet.

The PhysicalLayer encodesthe packet andtransmits it to thereceiving deviceon the other sideof the Link.

The TransactionLayerdisassembles thetransaction andtransfers data tothe ApplicationLayer in a formthat it recognizes.

The Data LinkLayer verifies thepacket’ssequencenumber andchecks for errors.

The PhysicalLayer decodesthe incomingpacket andtransfers it to theData Link Layer.

Core

The EZ Layerhandles all PCIExpresscomplexity andprovides a simpleSlave / DMAinterface to theApplication Layer● checks credits

and implements buffers required by the PCI Express specification

● Automatically Encodes & decodes all TLPs

● Implements DMA channels

EZ Layer*

Rx

Tx

* Only layer of Core accessible to the designer

Slave

28

PCI Express EZ IP Module Reference Manual Introduction to the EZ Module

2.3 EZ Layer ArchitectureFigure 11 illustrates the broad functionality of each component of the EZ Layer.

Figure 11: Block diagram of the PCI Express EZ layerNote that while all four components of the EZ Layer are broadly described in this manual, you only need tounderstand how the Slave and Master modules work in order to integrate the EZ Module into your own design.

2.4 Configuration SpaceThe Configuration Space, located within the Transaction Layer, implements all configuration registers andassociated functions. Please refer to Appendix B:: Register content of the Configuration Space or Chapter 7 of thePCI Express™ Base Specification Revision 1.0a for the complete content of these registers:

● Type 0 Configuration Space● PCI Power Management Capability Structure● Message Signaled Interrupt (MSI) Capability Structure● PCI Express Capability Structure● Virtual Channel Capabilities

The configuration space also generates all messages (PME#, INT, Error, Power Slot limit, etc.), MSI requests, andCompletion packets from Configuration requests that flow in the direction of the Root Complex, with the exceptionof Power Slot Limit Messages, which are generated by a downstream port in the direction of the PCI Express Link.All such transactions are dependent upon the content of the PCI Express Configuration Space as described in thePCI Express™ Base Specification Revision 1.0a.

Towards Transaction Layer Towards Application Layer

EZ Module

Receive Module

rx_...

tx_...

req

cpl cpl

req & cpl

slv_...

dma_...

● receives TLPs from the PCI Express Core

● Forwards requests to the Slave module

● Forwards Completions to Master module

● Filters unsupported requests and unexpected Completions

Slave Module*● handles I/O, Memory Read, and

Memory Write requests● Decodes TLPs and provides data

to Application Layer● Can automatically send

Completions to Transmit module when necessary

Master Module*● Performs DMA read / write

transfers● Sends Completions for slave

requests

Transmit Module● Sends requests and Completions

to the PCI Express Core

* Only an understanding of the Slave and Master modules are necessary for a designer to integrate the Core

Introduction to the EZ Module PCI Express EZ IP Module Reference Manual

29

2.5 Lane initialization and Lane reversalAs illustrated in Figure 7, connected components may not necessarily support the same number of lanes.Additionally, physical configuration constraints might oblige you to connect physical lane 0 of one component withphysical lane 7 of another.

The Core implements Lane reversal, which permits the logical reversal of lane numbers. This ensures that wiresaren’t crossed during configuration and provides maximum flexibility in terms of initialization possibilities.

Table 9 describes all lane initialization possibilities.

Table 9: Lane initialization possibilities

Initialized Lane Numbers

7a

a. Currently unsupported by the EZ Module.

6a 5a 4a 3 2 1 0 Physical Lane Numbers

Logical Lane Numbers

7 6 5 4 3 2 1 0 Lane assignments for non-reversed Lanes

X X X X 3 2 1 0

X X X X X X 1 0

X X X X X X X 0

0 1 2 3 4 5 6 7 Lane reversal starting on Lane 7

0 1 2 3 X X X X

0 1 X X X X X X

0 X X X X X X X

0 1 2 3 Lane reversal starting on Lane 3

0 1 X X

0 X X X

X X 0 1 Lane reversal starting on Lane 1

X X 0 X

30

PCI Express EZ IP Module Reference Manual Architecture of the Slave Module

Chapter 3 Architecture of the Slave Module

3.1 IntroductionThe Slave Module is responsible for:

● Handling I/O, Memory Read, and Memory Write requests● Decoding TLPs and providing data about the Slave Module● Sending Completer Abort Completions when necessary

3.2 Slave Module BehaviorFigure 12 illustrates the logical flow of the Slave Module.

Figure 12: Slave Module BehaviorThe Slave Module receives request transactions coming from the PCI Express Core and provides the ApplicationLayer with the start address, targeted BAR, and size of transfer of a given transaction.

Having received a packet from the Slave Module, the Application Layer determines whether to accept or abort thetransaction with the following consequences:

● Write transaction accepted: Data is immediately processed and insertion of a wait-state is not permitted.For I/O Write transactions, a successful Completion is automatically sent.

● Read transaction accepted: The request is immediately forwarded to the Application Layer, which isresponsible for storing necessary information and performing the Completion.

● Aborted transaction: Any data is discarded and an Abort Completion is sent automatically, if appropriate.

Note that:● Application logic is allowed any reasonable amount of time before accepting or aborting an incoming

transaction.

Idle

Receive

Waits for I/O orMemory Request

request received

transaction addressand size sent toApplication Layer

Application Layeraccepts Read request

Completion Abort

Transaction requestaborted and anyreceived data flushed

Application Layer aborts request

Send Data

Data sent toApplication Layer.Insertion of WaitStates is notpermitted.

Application Layer accepts Write request

Memory Writetransaction

Completion OKSuccessfulCompletion (SC)header sent toTransmit Module

I/O Writetransaction

Architecture of the Slave Module PCI Express EZ IP Module Reference Manual

31

● No other request or completion transactions can be processed by the Receive Module until the incomingtransaction is accepted or aborted. This can prevent the Master Module from receiving Completions, thusdelaying DMA transfers.

CompletionsI/O Write and aborted Read transactions: I/O Write transactions and aborted Read requests are automaticallycompleted by the Slave Module, which sends a completion packet with appropriate status to the Transmit Module.

Accepted I/O and Memory Read transactions: The Application Layer is responsible for sending Completiontransactions in response to I/O and Memory Read requests using a DMA channel, as described below:

1. The Application Layer accepts a transfer with assertion of SLAVE ACCEPT: SLV_ACCEPT and stores the values of SLAVE BYTECOUNT: SL_BYTECOUNT[], Slave COMPLETION ADDRESS: SLV_CPLADDR[], and SLAVE COMPLETION PARAMETER: SLV_CPLPARAM[].

2. The Application Layer programs a DMA channel to perform the Completion.3. The Application Layer finishes DMA transfer as with other types of transfers.

32

PCI Express EZ IP Module Reference Manual Architecture of the Slave Module

Example 1: Immediate programming of DMA RegistersIn this example, the Application Layer chooses to immediately program the DMA registers and DMA Parameters,thereby avoiding the need to temporarily store Byte Count, Completion Parameters, and Completion Addressvalues elsewhere.

Note in particular:● clock cycle 4: DMA REGISTERS: DMA_REGIN[] and DMA PARAMETERS: DMA_PARAM[] can be

programmed on the same clock cycle as assertion of SLAVE ACCEPT: SLV_ACCEPT.

Figure 13: Waveform illustrating immediate programming of the DMA registers

slv_bytecount

slv_accept

dma_regin[127:96]

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

slv_cplparam

slv_readreq

8 9

slv_cpladdr

dma_regin[95:64]

byte count XX

user XX

XX

dmacontrol3

cpl param XX

cpl addr XX

defined

bytecount

dma_regin[63:32] 0s XX

dma_regin[31:0]cpl XX addr

dmacontrol2

dmacontrol1

dmacontrol0

dma_param[15:11] XXcplparam

dma_param[10:8] XX100

dma_param[3:0] XXuserdefined

dmacontrol4

Architecture of the Slave Module PCI Express EZ IP Module Reference Manual

33

Example 2: Delayed programming of DMA RegistersThis example is identical to the one above except that the Application Layer chooses to delay programming of theDMA Registers and DMA Parameters by N clock cycles. In this case, the Application Layer is required totemporarily store Byte Count, Completion Parameters, and Completion Address values elsewhere.

Note in particular:● clock cycle N+1: The DMA REGISTERS: DMA_REGIN can be programmed simultaneously or any time after

assertion of SLAVE ACCEPT: SLV_ACCEPT.

● clock cycle N+4: The DMA PARAMETERS: DMA_PARAM can be programmed simultaneously or any time afterDMA REGISTERS: DMA_REGIN has been programmed.

Figure 14: Waveform illustrating delayed programming of the DMA registers

slv_bytecount

slv_accept

dma_regin[127:96]

1 2 3 4 5 6 7 N+2 N+3 N+4 N+5 N+6 N+7Clock Cycles

slv_cplparam

slv_readreq

N N+1

slv_cpladdr

dma_regin[95:64]

byte count XX

user XX

XX

dmacontrol3

cpl param XX

cpl addr XX

defined

bytecount

dma_regin[63:32] 0s XX

dma_regin[31:0]cpl XX addr

dmacontrol2

dmacontrol1

dmacontrol0

dma_param[15:11] X cplparam

dma_param[10:8] X 100

dma_param[3:0] X userdefined

dmacontrol4

X

X

X

34

PCI Express EZ IP Module Reference Manual Architecture of the Master Module

Chapter 4 Architecture of the Master Module

4.1 IntroductionThe EZ Layer can implement up to 8 independent DMA channels that may be used simultaneously to manage 8separate data flows, as illustrated in Figure 15.

Figure 15: Architecture of the Master ModulePer-channel control and status signals make it possible to program, monitor, and control each DMA channelindependently.

The Address/Data Interface is common to all channels and permits data to be read and written through thismemory-like interface.

4.2 Behavior of the Master Module

4.3 Request & Completion ControlThe Central Control is responsible for:

● Sending transfer requests issued by DMA channels to the Transmit interface according to availableCompletion resources

● Receiving Completions from the Receive interface and forwarding them to the appropriate channel

channel xcontrol & status

Request &Completion

Control

Address/DataInterface

Completion 0

Completion ...

Completion 7

Tx Buffer

Channel 0

Channel...

Channel 7

channel 0control & status

channel 7control & status

Architecture of the Master Module PCI Express EZ IP Module Reference Manual

35

4.3.1 Master Request ModuleFigure 16 illustrates the behavior of the Master Module.

Figure 16: Behavior of Master Request LogicThe DMA Request logic receives and transmits transfer requests to the Transmit Module. When one or morechannel request are detected, the Master Request Module masks requests for which there are no Completionresource or not enough credits. It then selects requests according to a round-robin priority scheme.

Data is read from the local interface without any DMA channel interaction.

One or more DMA transfer requestsreceived:● requests are masked if there are no

Completion resources or insufficient credits

● A specific request is chosen according to a round-robin priority scheme

Idle

Preparecomputetransactionparameters andprepare request

Requestrequests transfers toTransmit Module andwaits forAcknowledge

transfer initiated

DataSend Packetpayload

36

PCI Express EZ IP Module Reference Manual Architecture of the Master Module

4.3.2 Behavior of Master Completion LogicFigure 17 illustrates behavior of Master Completion logic.

Figure 17: Behavior of Master Completion LogicMaster Completion logic receives Completions from the Receive module. It identifies the appropriate completionresource, DMA channel, and local address for the Completion and informs the requesting channel.

Data is transmitted to the local interface without any DMA channel interaction. The Master Completion Modulereceives Completions from the Receive module and informs the requesting channel.

4.3.3 Completion resourcesUp to 8 Completion resources manage Non Posted requests and maintain necessary information to complete thetransfer.

A timeout error is signalled if no Completion is received after a given amount of time and the transfer is aborted.

Each completion resource is assigned a fixed tag, from 00000000 to 00000111. Completion resources areallocated dynamically and several handlers can serve the same DMA channel simultaneously.

Waits for Receivemodule to send aCompletion

Idle

IdentifyFinds correctCompletion resource,DMA channel, and localaddress

ReceiveInforms DMA channelthat Completion isreceived andprepares to receiveCompletion

DataPacket Payloadreceived andforwarded toApplication Layer

CompleteInforms requesterchannel thattransaction iscomplete

Completion received

DMA Channels PCI Express EZ IP Module Reference Manual

37

Chapter 5 DMA Channels

5.1 DMA RegistersThe DMA registers, programmed by the Application Layer, are illustrated in Figure 18.

Figure 18: DMA registersTable 10 describes each field of the DMA Channel Register.

Table 10: DMA Channel Register Fields

Name Description

PCI Address[63:0] The Application Layer programs the PCI start address of a DMA transfer in thisregister. The MSB indicates addressing as follows:

● MSB equal to 0: 32-bit addressing used● MSB not equal to 0: 64-bit addressing is used (if DMA is a memory transfer)

Note that a transfer must not cross a 4GB addressing boundary.

The address is automatically incremented and indicates the address of the lastrequested data + 1. Upon successful completion of a transfer, this register shouldcontain the value: pci_start_address + transfer_size.

Transfer Size[95:64] This register is used by the Application Layer to record the total transfer size (inbytes) for burst transfers. The value of this register must be programmed to 4 bytesfor DW transfers.

The size is automatically decremented and this register indicates the remaining datasize that can be requested. Upon successful completion of a DMA burst transfer, thevalue of this register should be 0.

Local address[127:96] This register is used by the Application Layer to record the local start address of aDMA transfer.

The local address is automatically incremented during transfers and indicates theaddress of the last transferred data + 1. Upon successful completion of a transfer,the value of this register should be local_start_address + transfer_size.

6495

Transfer Size (bytes)

96127

Local Address

3263

PCI address MSBdman

_reg

indm

an_r

egou

t

031

PCI address LSB

38

PCI Express EZ IP Module Reference Manual DMA Channels

5.2 DMA ParametersThe DMA Parameters, programmed by the Application Layer, are illustrated in Figure 19.

Figure 19: DMA ParametersTable 11 describes each field of the DMA Parameters.

5.3 DMA Commands● DW commands have the following characteristics:

● Restricted to a single data phase● The Transfer Size register is ignored after a transfer● Byte-enable bits specified in the Byte Enable register are used

● Burst commands have the following characteristics:

● Can be any size from 1 byte to 232 - 1 bytes● The Address register is automatically incremented after a transfer● The appropriate byte-enable bits are automatically computed by the DMA engine for all other commands

Table 11: DMA Channel Parameters

Name Description

DMA Mode[0] FIFO Mode (bit set to 0): In this mode, only one PCI Express request is permitted ata time. DMA FIFO COUNT: DMA_FIFOCNT[] determines whether a transfer ispossible. Data is always received in order and a FIFO can be safely used.

RAM Mode (bit set to 1): In this mode, multiple PCI Express requests are permittedat a time and the Application layer must be constantly ready to provide / accept alldata. Read DMA data may be received in any order. A FIFO may not be safely used.

Reserved[1] Bit 1 of the DMA register is reserved and must be tied to 0.

Read Latency[3:2] This register indicates the number of clock cycles the Application Layer needs toprovide data located at DMA ADDRESS: DMA_ADDR[] after a read is requested onthe local interface by DMA READ: DMA_READ.● 00: data is available on next clock cycle● 01: data is available 2 clock cycles later● 10: data is available 3 clock cycles later● 11: data is available 4 clock cycles later

Byte Enable[7:4] The Application Layer programs this register with the appropriate byte-enable forDW transfers. The value of this register is ignored for burst transfers.

Command[10:8] The Application Layer programs this register with a DMA command in order tospecify which PCI Express command should be used for a transfer.

TC[13:11] Traffic class for a transfer. Default is 0s.

Attributes[15:14] Bit 15 determines whether relaxed ordering for a transfer is permitted. Default is 0.Bit 14 determines whether No-Snoop for a transfer is permitted. Default is 0.

012471011131415command byte enable

DMA modeReserved

Read latency

Attributes

TC


39

Table 12 describes DMA Commands.

5.4 DMA Channel ControlFigure 20 illustrates behavior of the DMA Channel Control.

Figure 20: DMA Channel control behavior

Table 12: DMA Commands

Command Bytes Type Allowed Transfer Size Allowed Value for TC & Attributes

I/O read 000 DW 4 bytes 0s

I/O write 001 DW 4 bytes 0s

Memory read DW 010 DW 4 bytes any

Memory write DW 011 DW 4 bytes any

Completion with data 100 Burst same as request same as request

Reserved 101 - - -

Memory read burst 110 Burst any any

Memory write burst 111 Burst any any

Idle

Run

param register

Waits until newaction is possible

Wait Completion

Waits for alloutstanding requests

programmed

Data Request

request transfer

Complete

Signals end oftransfer to theApplication Layer

Data Wait

Waits until previouslyrequested transfer iscomplete (if DMA isin FIFO mode)

Completion Receivedcplabort

send Completionwith Completer Abortstatus

end and completion pending

end and no

all Completions received

acknowledge

Application stop and

acknowledge

request

Completion pending

command is completion

or RAM mode

40


DMA channels are controlled by a Module and behave according to the following rules:● Each channel can have up to 8 outstanding requests, however only one request is permitted at a time in

DMA Mode (in which case the channel waits for a Completion before making another request).● A channel must wait for all Completions before stopping if all requests have been issued, the Application

Layer stops DMA, or an error is detected.

Note that the DMA channels do not support scatter-gather.

5.5 Address / data interfaceData for DMA transfers is read from or written to the registers through a local interface based on a designer-defined 32-bit addressing scheme and simple read/write controls.

Write transfers● data and the data’s location address are presented simultaneously● the Application Layer can pipeline Write cycles if necessary

Read transfers: The start address is presented 0 to 3 clock cycles before data. This latency is programmed foreach DMA channel and permits maximum flexibility in order to interface with a wide range of peripherals.

Note that data for transfers programmed in RAM mode can be received in a non-linear order.

5.6 DMA Module

5.6.1 Connecting RAM devices to DMAConnection of a RAM or RAM-like device is illustrated in Figure 21:

Figure 21: RAM connection to the DMA interfaceWhen programming a DMA channel, the Read latency is adjusted depending on the number of register levelsbetween the Read Address port and the DMA Data Input port.

5.6.2 Connecting FIFO devices to DMAConnection of a FIFO device to DMA is illustrated in Figure 22.

Figure 22: FIFO connection to the DMA interface

Writ

e P

ort R

ead Port

RAMDevice

dma_rdaddr[]

dma_rddata[]

dma_rd

dma_wraddr[]

dma_wrdata[]

dma_wrbytevalid[]

dma_wr

dma_rddata[]

dma_rd

dma_wrdata[]

dma_rdchannel[n]

FIFO

dma_wr

dma_wrchannel[n]

dma_fifocnt[]


41

FIFO devices must not be used in DMA-RAM Mode since data can arrive in a non-linear order. You must adjustRead latency when programming DMA channels depending on the number of register levels between the ReadEnable port and the DMA Data Read port.

5.6.3 Setting DMA OptionsThe EZ Wizard permits you to set the Local Address size, the maximum DMA transfer size, and the number ofCompletion Resources, allowing you to choose between increased system performance or reduced Core logic.

5.6.3.1 Local Address SizeThe Local Address Size register can be set with the EZ Wizard to between 13 and 32 bits.

If, for example, the Application Layer only needs 23 bits to map all peripherals connected to the DMA channels,this signal can be set to 23. Doing so ties unused bits of the Local Address Size register to 0, thus saving logic.

Note that a design that uses DMA exclusively in FIFO mode shouldn’t need to use the Local Address register, andthis setting can be set to its minimum value.

5.6.3.2 Maximum DMA Transfer SizeThe Transfer Size register can be set with the EZ Wizard to between 8KB - 1 (13 bits) and 4GB - 1 (32 bits).

For example, if the Application Layer never performs DMA transfers of more than 1MB, this signal can be set to2MB - 1 (21 bits). Doing so ties unused bits of the DMA Transfer register to 0, thus saving logic.

5.6.3.3 Number of Completion ResourcesThe number of Completion Resources implemented in a design can be set with the EZ Wizard to between 1 and 8.Completion resources are only necessary for Non Posted (Memory Read and I/O) DMA transfers. The moreCompletion resources available in a design, the greater the potential performance, but at the price of increasedlogic.

Figure 23illustrates three example Completion Resource configurations.

Figure 23: Potential Completion resource / Channel configurationsTable 13 describes the examples illustrated in Figure 23.

Table 13: Description of Completion Resource examples

Example Description

1 You might choose to implement one single Completion resource if:● Your system only performs Posted (Memory Write) DMA transactions● System performance is not a high priority

2 In this example, DMA1 performs Write transfers and has no need for any Completionresources. DMA0 performs Read transfers and can have up to 4 outstanding requests at anygiven time.

CPL0 DMA1

DMA0

DMA2

CPL0

DMA1

DMA0

CPL2

CPL1

CPL3

CPL0

DMA1

DMA0

CPL2

CPL1

CPL3

Example 1 Example 2 Example 3

42


3 In this example, both DMA0 and DMA1 perform DMA Read and Write transfers. Each DMAchannel can have from 1 to 4 outstanding requests at any given time, depending on the numberof Completion resources already utilized by the other DMA channel.

Table 13: Description of Completion Resource examples

Example Description

Core Signals PCI Express EZ IP Module Reference Manual

43

Chapter 6 Core Signals

6.1 IntroductionFigure 24 includes all Core signals.

Figure 24: I/O diagram of the EZ Module

EZ Layer

Bus

Signal

Slave Signals

slv_dataout[63:0] or [31:0]slv_bytevalid[7:0] or [3:0]

slv_bytecount[13:0]slv_dwcount[11:0]

slv_addr[63:0]slv_bar[7:0]slv_readreq

slv_cpladdr[31:0]slv_cplparam[4:0]

slv_writereqslv_write

slv_acceptslv_abort

txdatan[15:0] (per lane)

txdatakn[1:0] (per lane)

txdetectrx txelecidlen (per lane)

txcompliancen (per lane) rxpolarityn (per lane)

powerdown[1:0]rxdatan[15:0] (per lane)

rxdatakn[1:0] (per lane)rxvalidn (per lane)

phystatus

rxelecidlen (per lane)rxstatusn[2:0] (per lane)

8/16-bit PIPE

Master Signals

(per channel) dman_regin[127:0](per channel) dman_regout[127:0]

(per channel) dman_param[15:0](per channel) dman_control[5:0](per channel) dman_status[3:0](per channel) dman_fifocnt[8:0]

dma_rdaddr [31:0]

dma_wr

k_bar[223:0]

k_aspm[20:0]

k_ez[15:0]

EZ Configuration Signals

pm_event

pm_auxpwr

pm_data[9:0]

cfg_pmcsr[31:0]

Power Management

int_requestcfg_prmcsr[31:0]cfg_devcsr[31:0]

Miscellaneous Signals

Application LayerTransaction Layer

clkrstn

nporsimulation_mode[7:0]

rstn_outnpor_out

slv_lastwrite

dma_rd

dma_rdchannel [7:0]dma_rddata[63:0] or [31:0]

dma_wraddr [31:0]dma_wrchannel [7:0]

dma_wrdata [63:0] or [31:0]dma_wrbytevalid [7:0] or [3:0]

k_pciid[95:0]k_pm[15:0]

cfg_ltssm[4:0]

44

PCI Express EZ IP Module Reference Manual Core Signals

6.2 Slave Module

6.2.1 Slave Module SignalsTable 14 describes signals used to communicate between the Application Layer and the Slave module of the EZLayer.

Note that some of the signals below may not appear in a particular implementation of the Core based onconfiguration choices made with the EZ Wizard.

Table 14: Slave Module Signals

Signal I/O Description

slv_dataout[63:0] or [31:0]

out Slave Data Out: This signals is the data bus for output in slave mode. It isused to read data received from the PCI Express bus during Slave Writetransactions.

slv_bytevalid[7:0] or [3:0]

out Slave Byte Valid: This signal indicates which bytes of SLAVE DATA OUT:SLV_DATAOUT[] are valid during a transaction. ● slv_bytevalid[0]: enables data bits 7:0● slv_bytevalid[7]: enables data bits 63:56

slv_bytecount[13:0]: out Slave Byte Count: This signals indicates the size of a transaction in bytes.Possible values range from 1 to 4096 bytes.

slv_dwcount[11:0] out Slave DW Count: This signal Indicates transaction DW count. Possiblevalues range from 1 to 1024 DW.

slv_addr[63:0] out Slave Address: This signal acts as a memory address counter. Address bitscan be directly connected to internal or external static memory devices. TheAddress is initialized with the PCI start address of a transaction and thenautomatically incremented whenever a DW is written.

slv_bar[6:0] out Slave BAR: This signal indicates which space is targeted during a slaveaccess:● slv_bar[5:0]: set when BAR0 ... BAR5 are targeted● slv_bar[6]: set when expansion ROM BAR is targeted

slv_readreq out Slave Read Request: This signal indicates when a read request is beingreceived. Application logic must store the address, target BAR and size (asnecessary), and SLAVE READ COMPLETION: SLV_READCPL[] in order tosubsequently complete the transaction.

slv_cpladdr[31:0] out Slave Completion Address: This signal indicates the DMA addressnecessary to perform a completion.● slv_cpladdr[6:0]: low address● slv_cpladdr[7]: reserved● slv_cpladdr[15:8]: requester tag● slv_cpladdr[31:16]: requester ID

slv_cplparam[4:0] out Slave Completion Parameter: This signal contains necessary DMAparameters to perform a completion:● slv_cplparam[4:3]: request attributes● slv_cplparam[2:0]: request traffic class

slv_writereq out Slave Write Request: This signal indicates when a write request is beingreceived.

slv_write out Slave Write: This signal indicates that data present on SLAVE DATA OUT:SLV_DATA_OUT must be written at the address specified by SLAVE ADDRESS:SLV_ADDR.


45

6.2.2 Transaction Examples using slave signals

slv_lastwrite out Slave Last Write: This signal is asserted for 1 clock cycle when the last DWof a Memory or I/O Write transfer is received.

slv_accept in Slave Accept: The Application asserts this signal when SLAVE READREQUEST: SLV_READREQ or SLAVE WRITE REQUEST: SLV_WRITEREQ areasserted in order to accept the corresponding transaction. If the transaction isan I/O Write, then a completion with Successful Completion (SC) status isissued.

slv_abort in Slave Abort: The Application asserts this signal while SLAVE WRITEREQUEST: SLV_WRITEREQ or SLAVE READ REQUEST: SLV_READREQ areasserted in order to abort the corresponding transaction. If the transaction isnot posted, then a Completion with Completion Abort (CA) status is issued.

Table 14: Slave Module Signals


46


Example 3: Typical transfer of a Write requestIn this example, the Core transmits a Write request. If the request is an I/O Write transaction, the Coreautomatically sends a Completion with Successful Completion (SC) status.

Note in particular:● clock cycle 5: Data transmission begins 2 clock cycles after assertion of SLAVE ACCEPT: SLV_ACCEPT.

Figure 25: Waveform illustrating a typical Write request

slv_bytecount

slv_accept

slv_dwcount

slv_bytevalid

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

slv_bar

slv_writereq

8 9

slv_addr

slv_dataout

BYTE COUNT X

X

X

X SELECTED BAR

ADDRESS COUNTER XX

DW COUNT XX

X

slv_write

X

XX

DW0 DW1 DW2 DW3

F F F F

slv_lastwrite


47

Example 4: Typical transfer of a Read requestIn this example, the Core transmits a Read request. The Application Layer is responsible for sending aCompletion.

Figure 26: Waveform illustrating a typical Read request

slv_bytecount

slv_accept

slv_dwcount

slv_cplparam

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

slv_bar

slv_readreq

8 9

slv_addr

slv_cpladdr

BYTE COUNT X

X

X

X BAR SELECTED

ADDRESS COUNTER XX

DW COUNT XX

DMA COMP ADDRESS XX

DMA PARAMETERS XX

48


Example 5: Aborting a Write requestIn this example, the Core transmits a Write request. Note that Application logic might need to abort a transaction incase of a permanent error.

Figure 27: Waveform illustrating an aborted Write requestWhen a I/O Write Request is aborted, the Core will automatically send a Completion with Completion Abort (CA)status (when required).

slv_abort

slv_accept

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

slv_write

slv_writereq

8 9

slv_lastwrite


49

Example 6: Aborting a Read requestIn this example, the Core transmits a Read request of 4 DW. A Completion with Completion Abort (CA) status isautomatically sent.

Note in particular:● clock cycle 5: SLAVE READ REQUEST: SLV_READREQ is deasserted 1 clock cycle after assertion of SLAVEABORT: SLV_ABORT.

Figure 28: Waveform illustrating an aborted Read request

6.3 Master Module

6.3.1 Master Module SignalsTable 15 describes signals used to communicate between the Application Layer and the Master module of the EZLayer, including status, control, and data signals.


Table 15: Master Application Layer Interface Signals


dman_regin[127:0] in DMA Register In: This signal indicates the values to be written to theAddress and Size DMA registers when DMA CONTROL:DMAN_CONTROL[3:0] are asserted.

dman_regout[127:0] out DMA Register Out: This signal reflects the state of the DMA register. It canbe read back in order to check the current channel address, remaining datacount, and other settings.

dman_param[15:0] in DMA Parameter: This signal indicates the value to write to the DMAParameters register when m_dman_control[] bit 4 is asserted.

slv_abort

slv_accept

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

slv_readreq

8 9

slv_lastwrite

50


dman_control[5:0] in DMA Control: This signal is used to program a DMA channel, as follows:● dman_control[0]: write enable for PCI address LSB register.● dman_control[1]: write enable for PCI address MSB register.● dman_control[2]: write enable for size register.● dman_control[3]: write enable for local address register.● dman_control[4]: write enable for DMA parameters register.● dman_control[5]: used to abort a currently running DMA transfer.

● If asserted and the channel still has outstanding requests, all requests are handled before transfer is aborted, otherwise transfer is immediately aborted.

● If asserted, and the current transfer is of a Completion, than a Completion with Completion Abort (CA) status is sent and the channel stops

dman_status[3:0] out DMA Channel Status: This signal reports details about the channel state.● 0000: (idle state): Last transfer ended successfully● 0001: (idle state): Last transfer was stopped by backend● 0010: (idle state): Last transfer ended because of CPL timeout● 0011: (idle state): Last transfer ended because of CPL UR error● 0100: (idle state): Last transfer ended because of CPL CA error● 0xxx: (idle state): reserved

● 1000: (busy state): Channel is busy processing: DMA channel computes next action

● 1001: (busy state): Requesting transfer: DMA channel attempts to send a request to the PCI Express Link

● 1010: (busy state): Waiting for Completion: Waiting for Completion(s): DMA channel waits for reception of Completion(s) that correspond to sent requests

● 1011: (busy state): Waiting for backend to provide/accept data: DMA is in FIFO mode and DMA FIFO COUNT: DMAN_FIFOCNT is not large enough to allow data transfer

● 1xxx: (busy state): reserved

dman_fifocnt[8:0] in DMA FIFO Count: The DMA channel refers to this signal to determine howmany bytes of data the Application Layer is currently able to provide / accept.This signal is used in FIFO mode only. Depending on the value of this signal,the DMA channel can pause data transfer until the FIFO can accept orprovide more data.

dma_rd out DMA Read: This signal is asserted when the Master module is reading datafrom the Application Layer, during which time the Application Layer providesdata located at the address specified by DMA ADDRESS: DMA_ADDR[] onDMA DATA IN: DMA_DATA_IN[].

dma_rdaddr[31:0] out DMA Read Address: This signal indicates the local address from which datashould be read. The specified data must be present on DMA READ DATA:DMA_RDDATA[].

dma_rdchannel[7:0] out DMA Read Channel: This signal indicates which DMA channel is currentlyreading data.

dma_rddata[31:0] or [63:0]

in DMA Read Data: This signal is the data bus for providing data for DMA Writetransactions.

Data must be naturally aligned on byte lanes in the following manner:● Bits[7:0]: Data for addresses similar to xx000● Bits[63:56]: Data for addresses similar to xx111



51

6.3.2 Transaction Examples using Master signals

dma_wr out DMA Write: This signal is asserted when the Master Module is writing data tothe Application Layer.

Application Layer logic stores current data on DMA WRITE DATA:DMA_WRDATA[] at the address specified by DMA WRITE ADDRESS:DMA_WRADDR[]. Valid bytes are indicated by DMA WRITE BYTE VALID:DMA_WRBYTEVALID[].

dma_wraddr[31:0] out DMA Write Address: This signals indicates the location at which datapresent on DMA WRITE DATA: DMA_WRDATA[] should be written.

dma_wrchannel[7:0] out DMA Write Channel: This signal indicates which DMA channel is currentlywriting data.

dma_wrdata[63:0]or [31:0]

out DMA Write Data: This signal is the data bus for receiving data for DMA Readtransactions.

Data must be naturally aligned on byte lanes. DMA WRITE BYTE VALID:DMA_WRBYTEVALID[] indicates which byte lanes are valid.

dma_wrbytevalid[7:0]or [3:0]

out DMA Write Byte Valid: This signal indicates which bytes of DMA WRITEDATA: DMA_WRDATA[] are valid during transmission.● dma_wrbytevalid[0] enables data bits 7:0● dma_wrbytevalid[7] enables data bits 63:56


52


Example 7: Typical transfer of a Read requestFigure 29 illustrates a typical transfer of a Read request.

Figure 29: Waveform illustrating a typical transfer of a Read RequestData read from the PCI Express Core is written to the local interface using dma_wr... signals.

The Application Layer may not insert wait states and must store data immediately.

Example 8: DMA Write transaction with 1 clock cycle latencyIn this example, the Core transmits a 4 DW DMA Write transaction.

Note in particular:● clock cycle 3: Data transfer doesn’t begin until clock cycle 4 due to system latency.

Figure 30: Waveform illustrating 4 DW DMA Write transaction with 2 clock cycles latencyIn order to prepare PCI Express Write transactions, the DMA Control logic reads data from the local interface usingdma_rd... signals.

In this example, the Application Logic has programmed DMA latency to 00b, causing the address to be presented1 clock cycle in advance.

dma_wraddr

dma_wr

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

dma_wrbytevalid

dma_rd

8 9

dma_wrdata

X

X

X

X

XX DW0 DW1 DW2 DW3

AD0 AD1 AD2 AD3

F F F F

dma_wrchannel XX SELECTED CHANNEL

dma_rdaddr

dma_wr

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

dma_rd

8 9

dma_rdchannel

XX

XX

AD0 AD1 AD2 AD3

dma_rddata XX DW0 DW1 DW2 DW3

SELECTED CHANNEL


53

Example 9: DMA Write transaction with 2 clock cycles latencyThis example is identical to the one above except that data transfer is delayed 2 clock cycles instead of 1.

In this example, the Core transmits a 4 DW DMA Write transaction.

Note in particular:● clock cycle 4: Data transfer doesn’t begin until clock cycle 4 due to system latency.

Figure 31: Waveform illustrating 4 DW DMA Write transaction with 2 clock cycles latencyIn order to prepare PCI Express Write logic, the DMA Control reads data from the local interface using dma_rd...signals.

In this example, the Application Logic has programmed DMA latency to 01b, causing the address to be presented2 clock cycle in advance.

dma_rdaddr

dma_wr

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

dma_rd

8 9

dma_rdchannel

XX

XX

AD0 AD1 AD2 AD3

dma_rddata XX DW0 DW1 DW2 DW3

SELECTED CHANNEL

54


Example 10: Writing to and Reading from the DMA registersThis example illustrates writing to and reading from the DMA registers.

Note in particular:● clock cycle 3: DMA CONTROL: DMA_CONTROL0 must be asserted in order for Address 1 to be written to the

DMA register. ● clock cycle 5: DMA REGISTER OUT: DMA_REGOUT[31:0] now contains Address 1.

Figure 32: Waveform illustrating Writing to and Reading from the DMA registers

1 2 3 4 5 6 7 10 11 12 13 14 15Clock Cycles

dma_regin[127:96]

8 9

dma_regin[195:64]

X

X

X

X

dma_regin[63:32]

X Xdma_regin[31:0]

dma_control1

dma_control2

dma_control3

X

X

X

dma_regout[127:96]

dma_regout[195:64]

dma_regout[63:32]

dma_regout[31:0]

X

dma_control0

ADDR1

ADDR 1

LOC ADDR4

LOC ADDR4

SIZE4

SIZE4

X


55

6.4 Miscellaneous Features

6.4.1 EZ ParametersTable 16 describes signals used to configure the EZ Module. Note that values for these signals are established bythe EZ Module Wizard.

Table 16: Configuration Signals


k_ez[15:0] in EZ-specific settings:● k_ez[0]

● 0: Native Endpoint● 1: Legacy Endpoint

● k_ez[3:1]: Maximum simultaneous requests count● 000: 1● 001: 2● 010: 3● 011: 4● 100: 5● 101: 6● 110: 7● 111: 8

● k_ez[5:4]: Lane width● 00: x1● 01: x2● 10: x4● 11: x8

● k_ez[6]: Implement Advanced Error Reporting (AER)● k_ez[7]: Clock Speed

● 0: 125 MHz● 1: 250 MHz

● k_ez[15:8]: DMA-implemented bit for each DMA channel● 01h: 1● 03h: 2● 07h: 3● 0Fh: 4● 1Fh: 5● 3Fh: 6● 7Fh: 7● FFh: 8

k_bar[223:0] in BAR settings: Please see the PCI Express Core Reference Manual fordetails.

k_pciid[95:0] in Core ID Settings:● k_pciid[15:0]: Vendor ID● k_pciid[31:16]: Device ID● k_pciid[39:32]: Revision ID● k_pciid[63:40]: Class code● k_pciid[79:64]: Sub-system vendor ID● k_pciid[95:80]: Sub-system device ID

56


6.4.2 Power ManagementTable 17 describes Power Management signals. Power Management for the EZ Module is maintained by the PCIExpress Core. Note that these signals do not appear if minimal Power Management is implemented.

k_pm[15:0] in Legacy Power Management Settings:● k_pm[4:0]: reserved = 00011● k_pm[5]: DSI● k_pm[8:6]: Aux current● k_pm[9]: D1● k_pm[10]: D2● k_pm[11]: PME D0● k_pm[12]: PME D1● k_pm[13]: PME D2● k_pm[14]: PME D3Hot● k_pm[15]: PME D3Cold

k_aspm[20:0] in ASPM Setting: ● k_aspm[0]: Device is an FPGA● k_aspm[1]: Device uses reference clock● k_aspm[4:2]: L0s exit latency● k_aspm[7:5]: Endpoint L0s acceptable latency● k_aspm[15:8]: NTFS● k_aspm[20:16]: L0s entry latency

Table 17: Power Management Signals


pm_event in Power Management Event: This signal initiates a Power Management Event Message(PM_PME) that gets sent to the Root Complex.If the Core is in low-power state L0S or L1, the Link exits from low-power state in order to sendthe Message. If the Link is in low-power state L2, a Beacon (or Wake#) is generated in order tore-initialize the Link before the Core generates the message.

This signal is edge sensitive.

pm_auxpwr in Power Management Auxiliary Power: This signal must be used to inform the Core that anauxiliary power has been detected.

Table 16: Configuration Signals



57

pm_data[9:0] in Power Management Data: This bus indicates power consumption of the component. This buscan only be implemented if all three bits of AUX_power (part of the Power Management Capa-bilities structure) are set to 0. This bus includes the following bits:● PM_DATA[9:2]: Data Register: This register is used to maintain a value associated with the

power consumed by the component (see the example below).● PM_DATA[1:0]: Data Scale: This register is used to maintain the scale used to find the

power consumed by a particular component (see the example below) and can include the following values:● 00: unknown● 01: 0.1 x● 10: 0.01 x● 11: 0.001 x

ExampleFor example, the two registers might have the following values:

● pm_data[9:2]: 1110010 = 114● pm_data[1:0]: 10, which encodes a factor of .01

To find the maximum power consumed by this component, multiply Data Value by Data Scale(114 X .01 = 1.14). 1.14 Watts is the maximum power permitted to this component in the powerstate selected by the data_select field.

cfg_pmcsr[31:0] out Power Management Capabilities Register: This register is read only and provides informa-tion related to power management for a specific function.

Figure 33: Power Management Capabilities Register● cfg_pmcsr[31:24]: Data Register: This field indicates which power states a function can

assert PME#.● cfg_pmcsr[23:16]: Reserved.● cfg_pmcsr[15]: PME_status: When this signal is set to 1 it indicates that the function would

normally assert the PME# signal independent of the state of the PME_en bit.● cfg_pmcsr[14:13]: Data_scale: This field indicates the scaling factor when interpreting the

value retrieved from the Data register. This field is read only.● cfg_pmcsr[12:9]: Data_select: This field indicates which data should be reported through the

Data register and the Data_scale field.● cfg_pmcsr[8]: PME_EN:

● 1: indicates that the function can assert PME#● 0: indicates that the function cannot assert PME#

● cfg_pmcsr[7:2]: Reserved● cfg_pmcsr[1:0]: PM_STATE

Table 17: Power Management Signals


031 24 23 1615 13 9 8 7 2 11214

Data Register

ReservedPME_statusData_scale

Data_selectPME_ENReserved

PM_state

58


6.4.3 Miscellaneous signalsTable 18 describes miscellaneous signals.


Table 18: Miscellaneous interface signals


clk in PCI Express Clock: This signal is the PCI Express clock (125 or 250 MHz).

rstn in PCI Express Clock Reset: This signal is the active-low reset signalassociated with PCI EXPRESS CLOCK: CLK. This signal, which is used toreset all Core registers which are not sticky, is triggered by a state transitionof the LTSSM (for example, exit from L2 or Hot Reset).

Note that assertion of this signal can be asynchronous with PCI EXPRESSCLOCK: CLK but deassertion must be synchronous.

This signal is active low.

npor in Power on Reset: This signal is the active low power-on reset signalassociated with PCI EXPRESS CLOCK: CLK. Note that assertion of thissignal can be asynchronous with PCI EXPRESS CLOCK: CLK butdeassertion must be synchronous. This reset signal is used to initialize allsticky registers and SERDES circuitry.


simulation_mode[7:0] in Simulation Mode: This signal is used for simulation and debuggingpurposes only. Default values are all 0s.● simulation_mode[0]: Setting this bit to 1 decreases PCI Express Link

initialisation time in order to simplify Core simulation.● simulation_mode[1]: Setting this bit to 1 forces the Receive Module to treat

all requests as Unsupported Requests. ● simulation_mode[2]: Setting this bit to 1 forces the Master module to

consider timeout expired for all pending Completions.● simulation_mode[4:3]: reserved● simulation_mode[5]: Disable PCI Express compliance mode.● simulation_mode[6]: Force PCI Express compliance mode.● simulation_mode[7]: Disable PCI Express low-power negotiation.

rstn_out out Reset Out: The Core asserts this signal to indicate that a hot reset isnecessary for Core and Application logic. This signal is typically used by thethe controller designated for the Application Layer in order to drive RESET:RSTN.


npor_out out Cold Reset Out: The Core asserts this signal to indicate that a cold reset isnecessary for Core and Application logic. This signal is typically used by thethe controller designated for the Application Layer in order to drive POWER-ONRESET: NPOR.


int_request in Interrupt Request: This signal is asserted by the Application Layer torequest an interrupt. This signal is level sensitive. Note that interrupt lines arenot asserted and the MSI mechanism is used instead to signal an interruptwhen MSI is enabled.

cfg_prmcsr[31:0] out Configuration Primary Control Status Register: content of register tocontrol status of pci

cfg_devcsr[31:0] out Configuration Dev Control Status Register: See PCI Expressspecifications for details.


59

cfg_linkcsr[31:0] out Configuration Link Control Status Register: See PCI Expressspecifications for details.

cfg_ltssm[4:0] out Configuration Link Training Module: Reports Link Training Module state asindicated below:● 00000: det_quiet● 00001: det_act● 00010: pol_act● 00011: pol_comp● 00100: pol_cfg● 00101: pol_spd● 00110: cfg_lkst● 00111: cfg_lkac● 01000: cfg_lnac● 01001: cfg_lnwt● 01010: cfg_cpl● 01011: cfg_idl● 01100: rec_rxlk● 01101: rec_rxcfg● 01110: rec_idl● 01111: l/0● 10000: disab● 10001: lpbk_ent● 10010: lpbk_act● 10011: lpbk_exit● 10100: hot_rst● 10101: l0s● 10110: l1_ent● 10111: l1_idl● 11000: l2_idl● 11001: l2_txwk

Table 18: Miscellaneous interface signals


60

PCI Express EZ IP Module Reference Manual Physical Layer Interface Signals

Chapter 7 Physical Layer Interface SignalsPlease refer to Figure 24 on page 43 for a diagram of all Core signals.

This chapter describes signals for the three possible types of physical interfaces (1-bit, 20-bit, or PIPE).

7.1 PIPE interface signalsThe Core is compliant with the 8-bit and 16-bit versions of the PIPE interface, enabling real-time FPGA prototyping(using exactly the same source code for FPGA and ASIC implementations).

Note: Signals that include lane number 0 also exist for lanes 1 - 7, as marked in the table.

Table 19: PIPE interface signals


txdata0[15:0](also 1 - 7)

out Transmit Data 0 (2 symbols on lane 0): This bus transmits data on lane 0.The first transmitted symbol is TRANSMIT DATA: TXDATA[7:0] and thesecond transmitted symbol is TRANSMIT DATA: TXDATA[15:8].

txdatak0[1:0](also 1 - 7)

out Transmit Data Control 0 (2 symbols on lane 0): This signal serves as thecontrol bit for TRANSMIT DATA 0: TXDATA0; TRANSMIT DATA CONTROL:TXDATAK[0] for the first transmitted symbol and TRANSMIT DATA CONTROL:TXDATAK[1] for the second (8b/10b encoding).

txdetectrx out Transmit Detect Receive: This signal is used to tell the PHY layer to start areceive detection operation or to begin loopback.

txelecidle0(also 1 - 7)

out Transmit Electrical Idle 0: This signal forces the transmit output to electricalidle.

txcompl0(also 1 - 7)

out Transmit Compliance 0: This signal forces the running disparity to negativein Compliance mode (negative COM character).

rxpolarity0(also 1 - 7)

out Receive Polarity 0: This signal instructs the PHY layer to do a polarity inver-sion on the 8b/10b receiver decoding block.

powerdown[1:0] out Power Down: This signal is responsible for determining the power phase of atransaction (L0, L0s, L1, or L2).

rxdata0[15:0](also 1 - 7)

in Receive Data 0 (2 symbols on lane 0): This bus receives data on lane 0. Thefirst received symbol is RECEIVE DATA: RXDATA[7:0] and the secondreceived symbol is RECEIVE DATA: RXDATA[15:8].

rxdatak0[1:0](also 1 - 7)

in Receive Data Control 0 (2 symbols on lane 0): This signal is used for sepa-rating control and data symbols. The first symbol received is aligned withRECEIVE DATA CONTROL: RXDATAK[0] and the second symbol received isaligned with RECEIVE DATA CONTROL: RXDATA[1].

rxvalid0(also 1 - 7)

in Receive Valid 0: This symbol indicates symbol lock and valid data onRECEIVE DATA 0: RXDATA0[15:0] and RECEIVE DATA CONTROL 0:RXDATAK0.

phystatus in PHY Status: This signal is used to communicate completion of several PHYrequests.

rxelecidle0(also 1 - 7)

in Receive Electrical Idle 0: This signal forces the receive output to electricalidle.

rxstatus0[2:0](also 1 - 7)

in Receive Status 0: This signal encodes receive status and error codes for thereceive data stream and receiver detection.

Optimizing system performance PCI Express EZ IP Module Reference Manual

61

Chapter 8 Optimizing system performanceThe EZ Module is customizable, and choices made during configuration have a strong impact on how the Corefunctions in your application. The following section introduces considerations, and concrete examples, foroptimizing the Core for your project

8.1 Measuring system performance

8.1.1 LatencyLatency is the delay a packet experiences when transferred between two points. Global latency, i.e., the totallatency within a fabric, is the sum of several contributing factors:

● Data payload size of a TLP● Core latency between the Application Layer and the Link (varies between Receive and Transmit

transactions)● Switch latency● Completer latency, like SDRAM read latency and SRAM mailbox read latency● PCI Express to PCI/PCI-X bridge latency and latency inherent to a PCI component

High Global latency can be avoided with a careful architectural design of your system (limiting the number ofSwitches, for example) and by modifying the width of critical Links.

Table 20 offers an example of a x4 lane of typical latency values for various types of Read Request transactionsoriginating from different points in the fabric and for different types of memory (mailbox or SDRAM).

Latency concerns are more or less important depending on the nature of a transaction:● Read transactions (high importance): Most systems generate more than one Completion transaction per

Read Request. ● Write transactions (low importance): Write operations are Posted and do not require completion.● Small transactions (high importance): Data payload transfer time contributes to the overall latency. Many

small packets will cause greater latency than one large packet.

8.1.2 Maximum Effective BandwidthBandwidth is a measure of the rate at which data are transferred at a specific point of a Link. Maximum EffectiveBandwidth is the rate at which “valuable” data are transferred at a particular point discounting overhead associatedwith transactions (overhead includes such things as a header, a sequence number, a CRC, an ECRC, and otherpackets like DLLPs and SKIP advanced set).

Table 20: Typical latency values of Read Request transactions

Fabric Environment Data payload size and Latency

8B 32B 256B

Link bandwidth usage 64 ns 88 ns 312 ns

Point-to-point, mailbox 240 ns 264 ns 464 ns

Point-to-point, SDRAM 272 ns 296 ns 496 ns

Point-to-point, PCI peripherals 448 ns 472 ns 904 ns

Through switch, mailbox 528 ns 552 ns 776 ns

Through switch, SDRAM 560 ns 584 ns 812 ns

Through switch, PCI peripherals 672 ns 720 ns 1144 ns

62

PCI Express EZ IP Module Reference Manual Optimizing system performance

Maximum Effective Bandwidth = data / (data + overhead)

In other words, Maximum Effective Bandwidth measures the ideal effective bandwidth possible given certainsystem conditions. Note that Maximum Effective Bandwidth is not necessarily close to 100%.

The table below lists Maximum Effective Bandwidths for Completion transactions sent in response to consecutiveRead or Write requests traversing a x4 Link. Note that for Read Requests, the Completion transaction size mightbe split into packets of 64 DW, 32 DW, or 16 DW. It is assumed that the overhead for each packet is 5 DW for theheader. An additional 2 DW must be added for the DLLP for Read Request transactions.

For example, to calculate the Maximum Effective Bandwidth for a Completion packet of 128 DW (divided into fourpackets each of 32 DW) generated in response to a Read Request:

4 = number of Completion Packets

5 = number of DW of overhead per packet

2 = number of DW per DLLP for the complete Read Request transaction

Overhead = (4 X 5) + 2 = 22

Maximum Effective Bandwidth = 128 / (128 + 22) = 85%

Note that a Maximum Payload Size of 64 DW or 128 DW provides between 92% and 96% of effective bandwidthfor Write transactions and practically the saturation limit for Read transactions. Maximum payloads in this rangecan be a good trade-off between latency, congestion, and size of the Core. Systems that require higher bandwidthmight need higher maximum data payloads in order to achieve effective bandwidth values of up to 99%.

8.1.3 Actual Link UsageActual Link Usage is a measure of how much a Link is actually used in a given period of time, and is defined by thefollowing equation:

Actual Link Usage = active time / (active time + idle time)

Table 21: Maximum Effective Bandwidths for various packets

Completion transaction size

(bytes / DW)Completion packet(s) in response to a...

Write request

Read Request packet(s) of 64

DW


DW


DW

4 / 1 16% 12% 12% 12%

8 / 2 28% 22% 22% 22%

32 / 8 61% 53% 53% 53%

64 / 16 76% 69% 69% 69%

128 / 32 86% 82% 82% 72%

256 / 64 92% 90% 84% 74%

512 / 128a

a. Currently unsupported by the EZ Module.

96% 91% 85% 75%

1024 / 256a 98% 92% 86% 75%

2048 / 512a 99% 92% 86% 75%

4096 / 1024a 99% 93% 86% 76%


63

8.2 System performance illustratedOptimizing System performance is based on defining your objectives (is maintaining a small Core size moreimportant than maximum throughput?) and balancing the following factors:

● The Maximum Packet Size● Global system latency● Setting the number of outstanding requests per component

In general, you should consider increasing the number of outstanding requests with small data payloads and inhigh-latency systems.

The following scenarios are meant to illustrate these considerations:

8.2.1 One outstanding request, small packet

Figure 34: One outstanding request, small packetIn this example, Tx1 represents a Read Request of 32 DW. Rx1 represents the corresponding Completion packet.One period (the latency of the transaction) is the time it takes for a Request transaction to be sent to a Completerand have a Completion packet returned. The Actual Link Usage is:

Actual Link Usage = Rx1 / (Rx1 + idle time)

Actual Link Usage = 3 / (3 + 7) = 30%

We know that for a Read Request (32 DW), the Maximum Effective Bandwidth is 86% (see Table 21: MaximumEffective Bandwidths for various packets). We just determined that the actual link usage is 30%. To calculateeffective (actual) bandwidth, we calculate 30% of the Maximum Effective Bandwidth:

Effective Bandwidth = .3 X .86 = 26%

The effective bandwidth of this Link is only 26%.

8.2.2 Two outstanding requests, small packet

Figure 35: Two outstanding requests, small packetTx1 and Tx2 represent Read Requests of 32 DW. By increasing the number of outstanding requests to two andleaving all other variables the same, our effective bandwidth becomes:



Note that a point exists after which increasing outstanding requests cannot improve Effective Bandwidth, asillustrated in Figure 36 below.

time (Tx)time (Tx)

time (Rx)Rx1 Rx1

Time necessary for Rx1 to pass a pointIdle time

Latency

Tx1Tx1 Tx1

time (Tx)

time (Rx)

Tx1

Rx1

Tx1 Tx1

Rx1

Tx2 Tx2 Tx2

Rx2 Rx1 Rx2

64


8.2.3 Four outstanding requests, small packet

Figure 36: Four outstanding requests, small packetTx1, Tx2, Tx3, and Tx4 represent Read Requests of 32 DW. Outstanding requests are set to four and the ActualLink Usage = 100%.


Effective Bandwidth = 1 X .86 = 86%

This Link is performing at Maximum Effective Bandwidth. Increasing the number of outstanding requests will notimprove system performance and will needlessly increase the size of the Core.

8.2.4 Two outstanding requests, large packet

Figure 37: Two outstanding requests, large packetTx1 and Tx2represent Read Requests of 32 DW. The number of outstanding requests is only two, however, weachieve Maximum Effective Bandwidth by increasing the Maximum Packet Size (Max_Payload_Size).


Effective Bandwidth = 1 X .86 = 86%

time (Tx)

time (Rx)

Tx1

Rx1

Tx1Tx2 Tx2

Rx2 Rx3 Rx4

Tx1 Tx1 Tx2Tx3 Tx4

Rx1 Rx1 Rx2 Rx3

Tx3Tx4

time (Tx)

time (Rx)

Tx1 Tx1Tx2 Tx2

Rx1 Rx1 Rx1Rx2Rx2

Tx1 Tx2


65

8.2.5 High latency systemThe last example simply illustrates the effects of a high latency system on Effective Bandwidth. Even with a highnumber of outstanding requests and large packets (Max_Payload_Size), this Link will never reach its MaximumEffective Bandwidth with this configuration because of high global latency.

Figure 38: High latency systemTx1, Tx2, Tx3, and Tx4 represent Read Requests of 32 DW. The time it takes for a Request to be sent and for thecomponent to receive the corresponding Completion TLP is relatively large, 22 in this case.



time (Tx)

time (Rx)

Tx2Tx3Tx4Tx1 Tx1 Tx2

Rx1 Rx2 Rx3

time (Tx)

time (Rx)

Tx3

Rx4Rx3Rx3 Rx4Rx1 Rx1

Tx1 Tx2Tx3 Tx4

Rx1

Note the gap

5 10 15 20 25 30

35 40 45 50 55 60

31

31

66


Appendix A: Register content of TLPs

The following tables describe register content for all types of descriptors (TLPs): ● Register content for TLPs without a data payload● Register content for TLPs with a data payload

This Appendix is provided in order to help you debug potential problems.


67

Register content for a TLP without a data payloadTable 22: Memory Read Request 32-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 0 0 0 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length

Byte 4 Requester ID Tag Last BE First BE

Byte 8 Address[31:2] 0 0

Byte12

Reserved

Table 23: Memory Read Request-Locked 32-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 0 0 0 1 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte12

Reserved

Table 24: Memory Read Request 64-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 1 0 0 0 0 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length


Byte 8 Address[63:32]

Byte12

Address[31:2] 0 0

Table 25: Memory Read Request-Locked 64-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 1 0 0 0 0 1 0 TC 0 0 0 0 T EP

Attr 0 0 Length



Byte12

Address[31:2] 0 0

68


Table 26: I/O Read Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1

Byte 4 Requester ID Tag 0 0 0 0 First BE


Byte12

R

Table 27: Type 0 Configuration Read Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1


Byte 8 Bus Number Device Nb. Func 0 0 0 0 Ext. Reg. Register Nb. 0 0

Byte12

R

Table 28: Type 1 Configuration Read Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1



Byte12

R

Table 29: Message (without data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 1 1 0 r2

r1

r0

0 TC 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 0

Byte 4 Requester ID Tag Message Code

Byte 8 Vendor defined or all zeros

Byte12

Vendor defined or all zeros


69

Register content for a TLP with a data payload

Table 30: Completion (without data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 1 0 1 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length

Byte 4 Completer ID Status B Byte Count

Byte 8 Requester ID Tag 0 Lower Address

Byte 12

R

Table 31: Completion Locked (without data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 0 0 0 1 0 1 1 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte 12

R

Table 32: Memory Write Request 32-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 0 0 0 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte 12

R

Table 33: Memory Write Request 64-bit addressing descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 1 0 0 0 0 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte 12

Address[31:2] 0 0

70


Table 34: I/O Write Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1



Byte 12

R

Table 35: Type 0 Configuration Write Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1



Byte 12

R

Table 36: Type 1 Configuration Write Request descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 TD

EP

0 0 0 0 0 0 0 0 0 0 0 0 0 1



Byte 12

R

Table 37: Completion (with data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 1 0 1 0 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte 12

R


71

Table 38: Completion Locked (with data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 0 0 1 0 1 1 0 TC 0 0 0 0 TD

EP

Attr 0 0 Length



Byte 12

Table 39: Message (with data) descriptor format

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

Byte 0 0 1 1 1 0 r2

r1

r0

0 TC 0 0 0 0 TD

EP

0 0 0 0 Length

Byte 4 Requester ID Tag Message Code

Byte 8 Vendor defined or all zeros for Slot Power Limit

Byte 12

Vendor defined or all zeros for Slots Power Limit

72


Appendix B: Register content of the Configuration Space

Appendix B describes the registers in the Configuration Space and in the extended Configuration Space. Pleasesee chapter 7 of the PCI Express Base Specification Revision 1.0a for more details.

The Common Configuration Space Header (the first table shown below), includes the following registers(described in subsequent tables):

● Type 0 Configuration Settings● MSI Capability Structure● Power Management Capability Structure● PCI Express Capability Structure● Virtual Channel Capability Structure


73

Common Configuration Space Header

Type 0 Configuration Settings

Table 40: Common Configuration Space Header

31:24 23:16 15:8 7:0 Byte Offset

TYPE 0 CONFIGURATION REGISTER (See below for details.)

000h..03Ch

Reserved 040h

PLDA CSR 044h

Reserved 048h..04Ch

MSI CAPABILITY STRUCTURE(See below for details.)

050..05Ch

Reserved 060h..074h

POWER MANAGEMENT CAPABILITY STRUCTURE(See below for details.)

078..07Ch

PCI EXPRESS CAPABILITY STRUCTURE(See below for details.)

080h..0A0h

Reserved 0A4h..0FCh

VIRTUAL CHANNEL CAPABILITY STRUCTURE(See below for details.)

100h..16Ch

Reserved 170h..17Ch

VC ARBITRATION TABLE 180h..1FCh

PORT VC0 ARBITRATION TABLE (Reserved) 200h..23Ch


PORT VC2 ARBITRATION TABLE (Reserved) 280h..2BCh

PORT VC3 ARBITRATION TABLE (Reserved) 2C0h..2FCh



PORT VC6 ARBITRATION TABLE (Reserved) 380h..3BCh

PORT VC7 ARBITRATION TABLE (Reserved) 3C0h..3FCh

Reserved 400h..FFFh

Table 41: Type 0 Configuration Settings

31:24 23:16 15:8 7:0 Byte Offset

Device ID Vendor ID 000h

Status Command 004h

Class Code Revision ID 008h

Cache Line Size 00Ch

Base Address 0 010h

74


PLDA Core Status RegisterFigure 39 illustrates the content of the PLDA Core Status Register.

Figure 39: PLDA Core Status Register

Base Address 1 014h

Base Address 2 018h

Base Address 3 01Ch

Base Address 4 020h

Base Address 5 024h

028h

Subsystem ID Subsystem Vendor ID 02Ch

Expansion ROM base address 030h

Capabilities PTR 034h

038h

Int. Pin Int. Line 03Ch

Table 41: Type 0 Configuration Settings

31:24 23:16 15:8 7:0 Byte Offset

03416 15272831reserved reservedCore version PLDA signature


75

MSI Capability Structure

Power Management Capability Structure

PCI Express Capability Structure

Virtual Channel Capability Structure

Table 42: Message Signaled Interrupt (MSI) Capability Structure

31:24 23:16 15:8 7:0 Byte Offset

Message Control Next Pointer Cap ID 050h

Message Address 054h

Message Upper Address 058h

Message Data 05Ch

Table 43: Power Management Capability Structure

31:24 23:16 15:8 7:0 Byte Offset

Capabilities Register Next Cap PTR Cap ID 078h

Data PM Control/Status Bridge Extensions

Power Management Status & Control 07Ch

Table 44: PCI Express Capability Structure

31:24 23:16 15:8 7:0 Byte Offset

Capabilities Register Next Cap PTR Capability ID 080h

Device capabilities 084h

Device Status Device control 088h

Link capabilities 08Ch

Link Status Link control 090h

Slot capabilities 094h

Slot Status Slot Control 098h

Root Control 09Ch

Root Status 0A0h

Table 45: Virtual Channel Capability Structure

31:24 23:16 15:8 7:0 Byte Offset

Next Cap PTR Vers. Extended Cap ID 100h

Port VC Cap 1 104h

VAT offset VC arbit. cap 108h

76


Port VC Status Port VC control 10Ch

PAT offset 0 (31:24) VC Resource Capability Register (0) 110h

VC Resource Control Register (0) 114h

VC Resource Status Register (0) RsvdP 118h

PAT offset 1 (31:24) VC Resource Capability Register (1) 11Ch


VC Resource Status Register (1) RsvdP 124h

...

PAT offset 7 (31:24) VC Resource Capability Register (7) 164h


VC Resource Status Register (7) RsvdP 16Ch

Table 45: Virtual Channel Capability Structure

31:24 23:16 15:8 7:0 Byte Offset


77

Appendix C: EZ Memory Usage

Appendix C describes EZ memory usage.

The size of some memory blocks is automatically adjusted according to the number of maximum simultaneousrequests (chosen with the EZ wizard).

Table 46: EZ Memory Usage

Memory block Design unit name Size (1,2a)

a. Number of maximum simultaneous requests

Size (3,4a) Size (5,6,7,8a)

Receive buffer pciez_rcvbuf 64-bit x 128 64-bit x 256 64-bit x 512

Replay buffer pciez_rplbuf 64-bit x 128 64-bit x 128 64-bit x 128

DMA buffer pciez_dmabuf 64-bit x 32 64-bit x 32 64-bit x 32

Receive FIFO pciexp_dcram_rxvc 9-bit x 64 10-bit x 64 11-bit x 64

Retry FIFO pciexp_dcram_rtry 6-bit x 128 6-bit x 128 6-bit x 128

Deskew buffer(PIPE 16-bit only)

pciexp_dcram 18-bit x 8(per lane)

18-bit x 8(per lane)

18-bit x 8(per lane)

Documents

PCI Express EZ IP Module - Twin Industries · 6.4.2 Power Management ... The Getting Started guide provides information to enable ... PCI Express EZ IP Module Reference