PCI Express Update forPCI Express Update forWindows LonghornWindows Longhorn

Tony PierceTechnical EvangelistWindows Hardware Evangelismtonypi @ microsoft.comMicrosoft Corporation

Session OutlineSession Outline

PCI Express UpdatePCI Express features supportedin Windows codenamed “Longhorn”

PCI Express features not supported in Longhorn

Resource considerations during design

PCI Express 1.1Overview of Delta from PCI Express1.0a

Functionality that should be considered in Windows Platform design

Session GoalsSession Goals

Attendees should leave this session with the following:

Know which PCI Express features are supported by Longhorn:

How feature is supported by Longhorn

Any hardware, firmware, or driver considerations to work well with Longhorn

Know which PCI Express features are not supported by Longhorn:

Why features are not supported

Any work to address feature for future support

Which of the new functionality in PCI Express 1.1 is important when designing for Longhorn

Enhanced Configuration Space: Support Enhanced Configuration Space: Support

Memory mapped access to enhanced configuration space

MCFG table indicates memory base to OS

No new interfacesExisting APIs updated to support 4K enhanced config space:BUS_INTERFACE_STANDARD

IRP_MN_READ_CONFIG, IRP_MN_WRITE_CONFIG

_OSC indicates support by OSASL config access PCIConfig operation region extended to 4K

Segments support

SAL support for Itanium

Enhanced Configuration Space: Enhanced Configuration Space: ConsiderationsConsiderations

Firmware considerationsUse MCFG table to communicate memory mapped config base to Longhorn

_OSC method indicates segment and ASL supportby OS

See PCI Firmware Specification V3.0 at http://www.pcisig.com/specifications/conventional/pci_bios

Driver considerations:Drivers must NOT use CFC/CF8

Driver must use documented interfaces for configuration space access

PCI Express RegistersPCI Express Registers

PCI Express Capability registersLonghorn will comprehend and program registers

Devices with capability version not understood by OS (1 is the only currently defined value) will be treated like a PCI device

Save/Restore of PCI Express Configuration Settings

Save and restore PCIe capability settings and registers across power events

BIOS ConfigurationsBIOS configurations should be maintained on Longhorn

Not always possible

Device Serial NumbersDevice Serial Numbers

Recommended for removable devices

Must be unique per device, same for functions okay

Allow device setting migration from port to port

Identifying new device in same port with a different serial number needs a remove/reinstall

Required for combo PCIe/USB ExpressCards

PCI Express HierarchyPCI Express Hierarchy

Hierarchies must follow PCIe specification – device type tells what it is

Violation will assert and not have PCIe features enabled

Violations involve root complex and switchesNo switches or endpoints in root complex

Virtual PCI-to-PCI bridges must be implemented in switches

No integrated devices appearing on internal bus of switches

Implementation ConsiderationsImplementation Considerations

Firmware considerations:Some base features, such as Max Payload Size/Max Read Request Size, should be initially set by firmware

OS may override firmware settings

Driver considerations:Drivers will not be allowed to modify base feature registers

Hardware considerations:Pay attention to switch implementation and PCIe hierarchy

Be sure to properly support TransactionsPending

Serial numbers required for certain classes of devices

Message Signaled Interrupt (MSI)Message Signaled Interrupt (MSI)

Kernel updates to improve interrupt handling and support

Both MSI/MSI-X fully supported:Reduces problems associated with interrupt sharing

MSI/MSI-X preferred over INTx emulation for PCIe

Class driver support:VideoPort

StorPort

NDIS

Others in future

Windows Driver Foundation fully supports MSI/MSI-X

MSI: ConsiderationsMSI: Considerations

Driver considerations:Driver changes required to use MSI/MSI-X

Whitepaper: http://www.microsoft.com/whdc/system/bus/pci/MSI.mspx

Hardware considerations:Single vs. multiple messages:

Most devices support single messageStill requires a read to interrupt status register

Multiple messagesDifferent interrupt status use different messages

MSI-X does not require contiguous IDT entries or identical target processor sets across messages (very flexible)

Hot PlugHot Plug

System interrupt signals Hot Plug eventRoot/switch ports must implement MSI asthe system interrupt

Legacy INTx support is allowed as well

State machine supports all combinations of hardware elements

Supports orderly or surprise removalExpressCard can be surprise removed

Edge-style cards will issue OS request for removal

Hot Plug: ConsiderationsHot Plug: Considerations

Firmware considerations:Firmware implements _OSC for OS to take native control

See PCI Firmware Specification V3.0

Driver considerations:No additional driver requirements but drivers must be able to start and stop properly – test thoroughly!

Hardware considerations:Special HW design considerations for compatibility with legacy OSs

PME PME

System interrupt signals PME eventRoot/switch ports must implement MSI as the system interrupt

Legacy INTx support is allowed as well

PME event handlingOS interrupt handler reads Requester ID

Notifies driver of waking device of the wakeup event

Driver interface remains the same – Wait/Wake DDK rules don’t change

Fully supported by Windows Driver Foundation

ACPI PME notification highly discouragedIncreases firmware support and development costs

Firmware-directed PME doesn’t work in many cases and causes crashes

PME: ConsiderationsPME: Considerations

Firmware considerations:Firmware implements _OSC for OS to take native control

See PCI Firmware Specification V3.0

Hardware considerations:Don’t share PME between PCI Express and PCI/PCI-X

Special HW design considerations for PME# routing for PCI Express to PCI/PCI-X bridges

Active State Power Management (ASPM)Active State Power Management (ASPM)

Enables ASPM based on user power scheme and performance preference

Plans are still under consideration

PolicySystem wide on/off

Per device on/off

Fixed per OS or user defined power scheme

Supports setting of ASPM on hot plugged devices

ASPM: ConsiderationsASPM: Considerations

Firmware considerations:Firmware can enable at POST if desired

OS may override firmware settings

Driver considerations:Drivers will not be allowed to modify ASPM

Hardware considerations:Devices must correctly indicate L0s/L1 exit and acceptable latencies

Advanced Error Logging and Reporting Advanced Error Logging and Reporting

Reporting and logging of correct and uncorrectable errors

I/O errors are big issues for server systems

Error logging and reporting a big part of diagnosis

Hardware vendors strongly encouraged to support this feature

Both chipsets and endpoint devices

Windows Longhorn Server will support AER via the Windows Hardware Error Architecture (WHEA)

Devices will be expected to implement support for AER

Windows expects to be able to “own” AER

PCI Express (via the PCI bus driver) will be a WHEA error source

Resource ConsiderationsResource Considerations

The BIOS provides resource information for each PCI root bus.

Windows attempts to respect BIOS configuration of devices from these resources. When not possible, Windows chooses an acceptable location for these devices.

There is a growing trend towards 64-bit systems with large memory configurations and large number of PCI devices on every system.

This puts constraints on the lower region of the physical address space.

Configure PCI Devices with memory resources above 4 GB

Memory Resource AssignmentMemory Resource Assignment

Windows Server 2003A PCI device with BIOS configuration above 4GB, is always assigned resources from below the 4GB regionIf no range below 4GB region is available, then the device is assigned a range above the 4GB boundary

This holds good even if the Windows OS cannot physically access the address range above 4GB

LonghornA memory address range above 4GB is available for PCI devices only if that range is physically accessible by the OSWithin this constraint, Windows will always attempt to respect the BIOS configuration on a PCI device.A PCI device with 64-bit BARs and no BIOS configuration is still assigned a memory address range above 4GB as available on the parent

Hardware Design RecommendationsHardware Design Recommendations

For Longhorn to assign memory resources to a PCI device above 4GB:

Entire device path from the PCI root bus to the device must support 64-bit memory BARs

A PCI device behind a PCI-PCI bridge can only have a Prefetchable memory window above the 4GB boundary

This constraint is imposed by the PCI-PCI bridge specification

PCI devices on root bus can have any memory type resources above the 4GB boundary

Include Prefetchable BARs on PCI devices behind a bridge

BIOS Design RecommendationsBIOS Design Recommendations

The ACPI BIOS should describe the memory range above 4GB in the _CRS and/or _PRS of the PCI root bus

This is described using the QWord Address Space descriptor as defined in the ACPI spec (Section 6.4.3.5.1).

Windows will evaluate the _SRS method with a buffer in the same format as the _CRS/_PRS

The memory range for the PCI root bus should not overlap with the physical RAM or some other range

The memory range is required to be physically accessible by the processor/chipset

The ACPI BIOS should configure the resources on the PCI devices after evaluating the _OSI method to account for the Server 2003 behavior

The ACPI BIOS should return an appropriate buffer in the evaluation of the _CRS/_PRS on a PCI root bus to account for the Server 2003 behavior

PCI Express Features Not Supported PCI Express Features Not Supported in Longhornin Longhorn

Extended VC/IsochronousNot supported and also a potentiallogo failure issue

Slot Power Budgeting

Extended VC/Isochronous Is Not Extended VC/Isochronous Is Not Supported: Potential Logo Failure!Supported: Potential Logo Failure!

Optional feature of PCI ExpressChipsets may or may not implement it

Devices cannot rely on chipset Isochronous support, and must be capable of using VC0!

No usage rules and latency guaranteesVCs are a shared resource

Different device assumptions may be incompatible

Designed For Windows Logo requirements in place:

WLP 2.2 and 3.0 requirements disallow usage of Extended VC/Isochronous

Slot Power BudgetingSlot Power Budgeting

Why Not Supported:No standard mechanism to convey overall system power budget to OS

Firmware can set slot power budget before booting OS

Future Direction:Currently being evaluated

PCI Express 1.1PCI Express 1.1

PCI Express 1.1 containsAll PCI Express ECNs for 1.0a

All errata against 1.0a

Learnings of Design community for past 2 years

PCI Express 1.1 scheduled to be released 1H05

PCI Express 1.0a ECNs—BasePCI Express 1.0a ECNs—Base

Root Complex Topology Discovery

Optional: enables software discovery of Root Complex internal components to program extended capabilities

Integrated Devices/Event Collector

Optional: allows integration of devices in a Root Complex; adds Root Complex Event Collector endpoint association capability

Reset Limit AdjustmentReduces time limit from end of Fundamental Reset to the entry to LTSSM Detect

from 80ms to 20ms; clarifies PERST#. Must validate components against new limit.

Assorted Error Clarifications Clarifies and removes inconsistencies in the description of error handling

Multi-Function Virtual Channel Optional: defines new MFVC capability structure to enable QoS management such as mapping of traffic classes to VCs, etc in Multi-Function Devices

MMConfig/Enhanced Configuration

Describes multiple enhancements to the way in which the memory-mapped configuration space is accessed in an improved and flexible manner

Training Error RemovalRemoves references to Training Error (chapters 4, 6-7) from the specification.

Remove or disable Training Error if implemented.

Bridge UpdatesIncorporates VDM from the PCI Express Bridge specification and shows usage of the Bridge Configuration Retry Enable bit

MSI-X Addition Requires devices that support interrupts to implement MSI or MSI-X or both

Request DependenciesClarifies the rules for forwarding packets by Root Complexes, Switches and End Points between different Virtual Channels

CRS Software VisibilityDescribes Root Complex behavior for configuration requests that return the Configuration Request Retry Status (CRS)

Mechanical

Data Link

Transaction

Software

Physical

Platform Reference Clock Power Management

Provides for platform clock power management reporting mechanism

VSECDefines Vendor-Specific Extended Capability for Extended Configuration Space in a standard and architected manner

Return LossRelaxes transmitter differential RL budget from 12dB to 10dB and receiver

differential RL budget from 15dB to 10dB. Phy signal validation must comprehend these changes.

PME Turn OffAll components are potentially required to participate in PME_Turn_Off protocol and must comprehend these changes even if they have no special power management functionality.

Surprise Down ErrorIntroduces optional error logging mechanism to report unexpected link failure (transition from Active to Inactive)

Flow Control InitializationUpdates the FC Init protocol with clarifications to the permitted intervals. All components must verify compliance with these changes.

Hot PlugIncludes significant revisions to all hot plug specifications. System software impacted. Improves ability to validate hardware for correct hot plug functionality. Hardware simplifications may be enabled for some components.

Posted Request Acceptance

Imposes an acceptance limit on posted requests (memory writes) to avoid or reduce PCIe fabric congestion

JitterIncorporates many changes including Tx & Rx eye measurements done with

‘clean’ clock. Phy signal validation must comprehend the changes.

Error Reporting

Modifies the required behavior for all devices when enabled to report detection of non-fatal errors.

PCI Express 1.0a ECNs—BasePCI Express 1.0a ECNs—Base

Mechanical

Data Link

Transaction

Software

Physical

ECNs that are Important to WindowsECNs that are Important to Windows

Integrated Devices - Event CollectorAllows chipset integrated devices that are Express-like but sit on an internal bus to take advantage of Express features

Reset Limit AdjustmentClarifications impact hot plug

MMCONFIG ECN Critical for correct system operation

Windows will rely on systems and firmware being built to this spec

ECNs that are Important to WindowsECNs that are Important to Windows

MSI-X ECN for PCI ExpressMSI-X offers great flexibility and benefits for many devices

Longhorn is planned to have full support

CRS Software VisibilityWindows my use this capability. More relevant to high end systems

Longhorn plan under investigation.

PR Acceptance Limit Not software visible but very important clarifications

PCI Express Hot PlugLonghorn hot plug implementation will be built to these updates

ECNs that are Important to WindowsECNs that are Important to Windows

PCI Express Base PCI Bus Power ManagementLonghorn will utilize the no device reset

Platform reference clock power managementNot software visible but very important for ExpressCard implementations

Surprise Down Error Critical piece of hot plug

All hot plug ports need to implement

Error Reporting Important to systems that implement Advanced Error Reporting capability

Call to ActionCall to Action

Ensure you are up to date on the PCI Express 1.1 and PCI Firmware 3.0 specifications

Implement _OSC in your firmware for proper handoff of PCI Express Features to Windows Longhorn

Ensure your devices meet Designed for Windows Logo requirements

Ensure you are signed up for the Windows Longhorn Beta program

Regularly test

Provide Feedback

Send your hardware to our test labs

Community ResourcesCommunity Resources

Windows Hardware & Driver Central (WHDC)www.microsoft.com/whdc/default.mspx

Technical Communitieswww.microsoft.com/communities/products/default.mspx

Non-Microsoft Community Siteswww.microsoft.com/communities/related/default.mspx

Microsoft Public Newsgroupswww.microsoft.com/communities/newsgroups

Technical Chats and Webcastswww.microsoft.com/communities/chats/default.mspx

www.microsoft.com/webcasts

Microsoft Blogswww.microsoft.com/communities/blogs

Additional ResourcesAdditional Resources

Community Sites

http://www.microsoft.com/communities/default.mspx

List of Newsgroupshttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspx

Attend a free chat or webcasthttp://www.microsoft.com/communities/chats/default.mspx

http://www.microsoft.com/seminar/events/webcasts/default.mspx

Locate a local user group(s)http://www.microsoft.com/communities/usergroups/default.mspx

Non-Microsoft Community Siteshttp://www.microsoft.com/communities/related/default.mspx

Additional ResourcesAdditional Resources

EmailPciesup @ microsoft.com

SpecsPCI Express Base Specification v1.0a http://www.pcisig.comPCI Firmware Specification v3.0 http://www.pcisig.com

Whitepapers http://www.microsoft.com/whdcPCI Express OverviewPCI Express Legacy TransitionPCI Express Longhorn Implementation PCI-PCI Bridge Resource IssuesPCI Hot-Plug PCI I/O Reduction

Related SessionsWindows Hardware Error Architecture Error Management Solutions Synergy with WHEAPCI Express Trusted Configuration Space

© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.