Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
©2017 Intel Corporation. All rights reserved. ALTERA, ARRIA, CYCLONE, HARDCOPY, INTEL, MAX, MEGACORE, NIOS, QUARTUS and STRATIX words and logos are trademarks of Intel Corporation and registered in the U.S. Patent and Trademark Office and in other countries. All other words and logos identified as trademarks or service marks are the property of their respective holders as described at www.altera.com/common/legal.html. Intel warrants performance of its semiconductor products to current specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Date: Aug 2017
Revision: 1.0
Arria 10 Low Latency 10G Ethernet MAC and
1G/2.5G/5G/10G Multi-Rate Ethernet PHY
Interoperability Hardware Demonstration with
Aquantia PHY Reference Design
2
Table of Contents Introduction .................................................................................................................................................. 3
Functional Flow For NBASE-T Ethernet Interoperability System .................................................................. 4
Design Components ...................................................................................................................................... 6
Hardware and Software Requirements ........................................................................................................ 8
Hardware Testing .......................................................................................................................................... 9
Speed Change Test .................................................................................................................................. 15
Aquantia PHY Loopback mode ................................................................................................................ 22
Reference Design Debug Features .......................................................................................................... 25
3
Introduction
This application note describes a design that demonstrates the interoperability between Intel Low
Latency 10G Ethernet IP solution and Aquantia PHY device. It uses Intel’s Arria 10 Low Latency Ethernet
10G Media Access Controller (MAC) and 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP cores in NBASE-T
mode on Arria 10 GX Transceiver Signal Integrity Development Kit together with a Aquantia Evaluation
Board which contains the AQR105 Ethernet PHY device.
This design offers the following features:
• Auto negotiation test between Intel Arria 10 Field Programmable Gate Array (FPGA) and
Aquantia AQR105.
• Transmission and reception of packets between Arria 10 FPGA and Aquantia AQR105.
• Sequential random bursts tests. You can configure the number of packets, payload-data type,
and payload size for each burst.
• Packet statistics for traffic generator, monitor, MAC transmitter (TX) and MAC receiver (RX).
• Packet classification for different frame sizes transmitted and received by the MAC.
• Throughput for the traffic received by the traffic monitor.
• System Console user interface. This tool command language (Tcl)-based user interface allows
you to dynamically configure and monitor any registers in the reference design.
4
Functional Flow For NBASE-T Ethernet Interoperability System
The following figure illustrates the overview of the design’s system architecture. In this system, the
Arria 10 FPGA will interact with the Aquantia AQR105.
Figure 1. System Architecture Overview
Arria 10 Transceiver Signal Integrity Development Kit
Arria 10 GX FPGA
NBASE-T Ethernet Subsystem (LL10GMAC
+ MGE PHY)
SFP+
10GbE SFP+ Direct Attach
Copper(System Side)
Aquantia Ethernet PHY Evaluation Board
Aquantia AQR105 PHY
Ethernet Cat6a cable (Line Side)
Aquantia Ethernet PHY Evaluation Board
Aquantia AQR105 PHY
Arria 10 Transceiver Signal Integrity Development Kit
Arria 10 GX FPGA
NBASE-T Ethernet Subsystem (LL10GMAC
+ MGE PHY)SFP+
10GbE SFP+ Direct Attach
Copper(System Side)
JTAG JTAG
PC and System Console
Subsystem
Aquantia GUI (PC)
1. Based on the Figure 1 setup, it contains 2 Ethernet stations (which each of it consist of the Arria
10 FPGA development kit and Aquantia AQR105 evaluation board). Both stations connected
with each other using the category (CAT) 6A cable.
2. The design used to load into both Intel Arria 10 Transceiver Signal Integrity (SI) development kit
will be based on the Intel Arria 10 10G Universal Serial Media Independent Interface (USXGMII)
Ethernet Design Example. Refer to Intel Arria 10 Low Latency Ethernet 10G MAC Design
Example User Guide, section 10G USXGMII Ethernet Design Example for details.
3. The PC running on a Linux or a Window operating system, preinstalled the Quartus Prime will be
used to configure and control the FPGA. Hardware testing on FPGA side will be demonstrated
using the Tcl-based System Console from Quartus Prime software. For Aquantia PHY side, the
Aquantia GUI tool will be used to provide the access to the Aquantia PHY’s Management Data
Input/Output (MDIO) registers to control and monitor the link performance.
5
4. Connection between the transceiver GXB side of FPGA to the Aquantia AQR105 will be using the
small form-factor pluggable transceiver (SFP+) interface. A 10GbE SFP+ direct attach copper
cable is used to connect between the SFP+ cage on Arria 10 Transceiver SI development kit to
the system side of Aquantia AQR105 evaluation board.
5. This design will include various hardware testing, such as auto negotiation test and basic traffic
check test. (The test setup details will be discussed on the following section)
Related Links
Intel Arria 10 Low Latency Ethernet 10G MAC Design Example User Guide
https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug-20016.pdf
Section 1 Quick Start Guide on page 5
Describes on how to generate the design example from Quartus Prime IP Parameter Editor.
Section 10G USXGMII Ethernet Design Example on page 71
Describes the design example features, function description, architecture and steps to perform the
hardware testing.
6
Design Components
The reference design consists of four main subsystems: Arria 10 FPGA NBASE-T Ethernet Subsystem,
Aquantia ARQ105, Aquantia Graphical User Interface (GUI) and System Console.
Table 1. Intel FPGA Subsystem Design Components
This table lists the components of the FPGA design’s subsystems.
Component Description
Low Latency Ethernet 10G MAC IP core This IP core handles the flow of data through
the Multi-rate Ethernet PHY IP core.
• On transmit path, the MAC accepts client
frames and constructs Ethernet frames before
forwarding them to the PHY layer.
•On the receive path, the MAC accepts
Ethernet frames through the PHY layer,
performs checks and removes the relevant
fields before forwarding the frames to the
client. For this design, the MAC uses the
memory-based statistics counters.
1G/2.5G/5G/10G Multi-rate Ethernet PHY IP
core
The PHY IP core will be using the
1G/2.5G/5G/10G (USXGMII) variant mode.
Transceiver Reset Controller The Intel Transceiver PHY Reset Controller IP
core. It helps to reset the transceiver.
ATX PLL Generate a TX serial clock for the Arria 10
Transceiver.
IO PLL Generates clocks for all design components.
Traffic Controller The traffic controller consists of:
• Traffic generator—injects client packet bursts
into the MAC TX core.
7
• Traffic monitor—receives packet bursts from
the MAC RX core.
The traffic controller connects to the Avalon-ST
single-clock FIFO in the Ethernet subsystem
through the Avalon Streaming (Avalon-ST)
interface.
Ethernet Packet Generator This module consists of Avalon Memory-
Mapped (Avalon-MM) registers, Ethernet
packet generation block, CRC generator, and
shift register.
Ethernet Packet Monitor This module verifies the payload of received
packets and collects information from the
statistics counters. This consists of Avalon-MM
registers and CRC checkers
JTAG to Avalon Master Bridge This IP core provides a connection between the
System Console and Qsys system through the
physical interfaces. The System Console
initiates Avalon-MM transactions by sending
encoded streams of bytes through the bridge’s
physical interfaces.
8
Hardware and Software Requirements
Intel uses the following hardware and software to test the design.
Hardware
• Intel Arria 10 GX Transceiver Signal Integrity (SI) development kit (Qty: 2)
• Aquantia ARQ105 evaluation board (Qty: 2)
• 10GbE SFP+ direct attach copper cable (Qty: 2)
• Ethernet CAT 6A cable
Software
• Quartus® Prime Standard version 17.0 (for hardware testing)
• Future Technology Device International (FTDI) driver*
• Aquantia USXGMII firmware*
• Aquantia PHY GUI*
*For Aquantia related firmware and software, please register on Aquantia Customer Portal on the link
below:
https://portal.aquantia.com/customer-portal
For a new user, please register on the website and create a new account. After the account is approved
by Aquantia, user may login into the portal and obtain all the required firmware and software. The
firmware and software will be update from time to time. Please consult Aquantia for the latest version
of firmware and software available. (email: [email protected])
9
Hardware Testing
Follow the steps below to perform the hardware test:
1. Install the FTDI driver from the link below. (It is required for the Aquantia ARQ105 evaluation
board micro-USB access port. Driver must be installed for the evaluation board’s connection
over the USB port. Connect the micro-USB port to the PC to install the FTDI driver.)
2. Next, connect the Aquantia board to PC. Program the Aquantia board with the USXGMII
firmware provided by Aquantia. (The firmware (cld file) and software can be downloaded from
the Aquantia Customer Portal. Detail steps can refer to section above. All the files were zipped
in the file named as AQGUI_USB_ShuttleTools.zip)
3. Extract the AQGUI_USB_ShuttleTools zip file and store the firmware together into a similar
directory in PC. Open a command prompt and change the directory to where the Aquantia
related files located. Next, type the command below in command prompt to flash the firmware
into the Aquantia AQR105 evaluation board.
Command: flashburn.exe <target_board_part_number> 0 <firmware_name>.cld
4. After the firmware is successfully flash into the board, open the Aquantia GUI
(AQGUI_USB_Shuttle.exe) and choose the devices option to see the PHY status. (The links are
expected to not link up for both PHYs as our FPGA have not go through any bring up process
yet.)
5. The Aquantia GUI is only applicable for user set to NBASE-T mode. It is not compatible when
user used the IEEE802.3bz mode. By default, user will be using the NBASE-T mode. NBASE‐T is a
new Ethernet signaling method that has been adopted as the basis of IEEE 802.3bz. Newer PHY
firmware may have IEEE802.3bz mode enabled. You can manually set the IEEE802.3bz registers
like the speed rate advertisements under the Register Access box at the bottom of the GUI.
10
Figure 2. Aquantia GUI
6. Generate the Arria 10 10G USXGMII Ethernet design example from Quartus Prime 17.0 Standard
edition IP Parameter Editor. (Refer to section 1 on Intel Arria 10 Low Latency Ethernet 10G MAC
Design Example User Guide for the detail steps on how to perform the generation.)
7. Open the Clock Control application, which is the part of development kit. You may download it
from the Arria 10 GX Transceiver SI development kit link below. From the directory
arria10GX_10ax115sf45_si_v15.1\examples\board_test_system \ClockController.exe, open
the tool and set Y5 to 644.53125 MHz and Y6 to 125 MHz. (Remember to perform this changes
to both dev kits. If you connected to 2 dev kits on one single PC, a pop up will prompt up for
choosing which path you would to modify after you open the ClockController.exe)
8. Open the project file (altera_eth_top.qpf) of Arria 10 10G USXGMII Ethernet design example in
the Quartus Prime software.
11
9. Click Processing ➤ Start Compilation to compile the design.
10. After the compilation completed, configure the FPGA using the generated configuration file
(altera_eth_top.sof).
11. Modification needed to be perform on the Tcl file for hwtesting. First, look for the
system_console folder from directory LL10G_10G_USXGMII/hwtesting/system_console. Then,
duplicate another version of this folder. For example, you will get 2 folders, rename it to
system_console_pod13_A and system_console_pod13_B.
12. Since one PC is used to connect with 2 Arria 10 GX SI development kits, both kits will be using
different paths for the System Console command. Maintain the system_console_pod13_A with
no changes is required. For system_console_pod13_B, open the basic.tcl from directory
system_console_pod13_B/basic and change the following:
Table 1. System Console basic.tcl file command name
From To
set port_id [lindex [get_service_paths master]
0];
set port_id [lindex [get_service_paths master]
1];
13. In the Quartus Prime software, select Tools ➤ System Debugging Tools ➤ System Console to
launch the system console.
14. Change the working directory to LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A.
15. Initialize the design command list by running this command: source main.tcl
16. Run the following command in the system console to initialize the channel and enable the auto-
negotiation test:
SET_CHANNEL_BASE_ADDR 1
SET_PHY_USXGMII_AN
17. Perform the similar command from steps 13 to 15 for different directory, changed to
LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B.
18. Next, we need to turn on the auto negotiation mode for the Aquantia PHY. Using the Aquantia
GUI, choose the device and go to Utilities > Register Guide and write register 4.C441 with value
0x8808 to enable the auto negotiation on the PHY. After you successfully bring up, you will
12
observe from the figure below that the link is up. (We need to enable for both Aquantia
evaluation board.)
Figure 3. Aquantia GUI (Auto negotiation enabled)
19. After both links are up on both Aquantia boards, we will continue the traffic test using the
system console.
20. Back to the system console, change the working directory to LL10G_10G_USXGMII /hwtesting/
system_console_pod13_A. Initialize the design command list by running this command: source
main.tcl
21. Run the following command in the system console to start the test. (The test must run at
channel 1 only as the design example is assigned the channel 1 of transceiver to the SFP+
interface.)
TEST_EXT_LB <channel> <speed> <burst_size>
Example: TEST_EXT_LB 1 10G 80000000
22. For a successfully run, you will observe the packet monitor results such as below for both FPGAs.
When the test completed on one path, the following diagrams show that the packet monitor
13
block received the same number of packets generated bt traffic generator without error,
together with the TX and RX statistics counters details.
Figure 4. Sample Output – Packet Monitor
14
Figure 5. Sample Output – Packet Monitor
23. Perform the similar command from steps 18 to 19 for different directory, changed to
LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B. Similar with previous results, you
need to observed no error packet observed from the packet monitor.
15
Speed Change Test
This test is basically covered the steps to perform the auto negotiation test between the Arria 10 FPGA
and Aquantia ARQ105 PHY. Auto negotiation is an Ethernet procedure protocol by which two devices
connecting with each other and choose to use the common transmission parameter. In the process, one
of the connected device will first share their capabilities and then choose the highest performance
transmission mode they both support. In our test case, we will use the speed as the transmission
parameter. The following steps will guide user on how to perform the speed change test. User need to
make sure the testing the design steps are completed before continue on the steps below.
1. After completed the Testing the design section on basic bring up on both Arria 10 SI kit and
Aquantia evaluation board, next user can continue with the more comprehensive speed
switching test. First, user may perform the first condition where speed change done on Aquantia
PHY side, Arria 10 FPGA will negotiate to the correct speed.
2. Next, open the Aquantia GUI, disable the unwanted speed and press the restart Autonegotiation
button. For example, if user want to enable 5G speed, need to untick the 10Gb speed option on
the auto negotiation section from the GUI. By performing this changes, the Aquantia PHY will
connect to 5G as this is the highest speed among others remaining. (It is expected based on the
auto negotiation behavior.) Perform the same changes on both Aquantia PHYs, using 2
separated GUI. Figure below shows the option of the speed selection.
16
Figure 6. Aquantia PHY GUI – Speed selections
3. The following table show all the combination that user can set to perform the speed change
test.
Table 2. Speed Mode
Aquantia PHY (Speed mode) Auto-negotiated speed
1G/2.5G/5G/10G 10G
1G/2.5G/5G 5G
1G/2.5G 2.5G
1G 1G
4. Next, user need to continue the configuration on FPGA side. In the Quartus Prime software,
select Tools ➤ System Debugging Tools ➤ System Console to launch the system console.
5. Change the working directory to LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A.
6. Initialize the design command list by running this command: source main.tcl
17
7. After the speed is set on the Aquantia PHY, user can restart the auto negotiation (AN) on Arria
10 FPGA (if the auto negotiation is enabled previously) on system console. Type the command
below on system console to restart the AN process on FPGA.
Command: SET_CHANNEL_BASE_ADDR 1
Command: SETPHY_USXGMII_AN_RESTART
8. If user do not enable the auto negotiation speed previous, run the following command in the
system console to initialize the channel and enable the auto-negotiation test:
Command: SET_CHANNEL_BASE_ADDR 1
Command: SET_PHY_USXGMII_AN
9. Next, key in the command below on system console to check the Multi-rate Ethernet PHY IP
core register settings.
Command: CHK_PHY_STATUS
10. Figure below shows the example result for the PHY status reading. Those PHY register
definitions are available on Arria 10 Transceiver PHY user guide.
Figure 7. PHY status register result
Table below shows the description on all the USXGMII register settings:
18
Table 3. USXGMII Registers detail
USXGMII Control Register Bit [0]
• 0: 10GBASE-R mode
• 1: USXGMII mode
Bit [1]
• 0: Disable USXGMII Auto-Negotiation
• 1: Enable USXGMII Auto-Negotiation
USXGMII Status Register Bit [2]
• 0: Link is up
• 1: Link is down
Bit [5]
• 1: Auto-negotiation process is
completed.
USXGMII Partner Ability Register Bit [11:9] Speed
• 3’b010 : 1G
• 3’b011: 10G
• 3’b100: 2.5G
• 3’b101: 5G
Bit [15] Link status
• 0: Link down
• 1:Link up
All the register mentioned above are important for user to track the status of auto negotiation
on FPGA. For full PHY register definitions, please refer to Arria 10 Transceiver PHY user guide.
11. Next, user need to repeat the steps 4 to 10 by changing to another FPGA on different path.
(The directory is LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B.)
12. Make sure both FPGAs can initiate to the respective speed set by the Aquantia PHY. Check
for the cable connectivity if the link is not able to bring up. User may follow the steps on the
section below, description on how to enable the loopback mode on Aquantia PHY to
perform any debugging test.
19
13. After the speed switch completed for the system, we will continue the traffic test using the
system console. Back to the system console, change the working directory to
LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A. Initialize the design command list
by running this command: source main.tcl
14. Run the following command in the system console to start the test. (The test must run at
channel 1 only as the design example is assigned the channel 1 of transceiver to the SFP+
interface.)
TEST_EXT_LB <channel> <speed> <burst_size>
Example: TEST_EXT_LB 1 10G 80000000
15. For a successfully run, you will observe the packet monitor results such as below for both FPGAs.
When the test completed on one path, the following diagrams show that the packet monitor
block received the same number of packets generated bt traffic generator without error,
together with the TX and RX statistics counters details.
20
Figure 4. Sample Output – Packet Monitor
21
Figure 5. Sample Output – Packet Monitor
16. Perform the similar command from steps 18 to 19 for different directory, changed to
LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B. Similar with previous results, you
need to observed no error packet observed from the packet monitor.
22
Aquantia PHY Loopback mode
Aquantia PHYs support several test modes in all the supported line-rates providing a flexible and
effective debug capability to narrow down the point of failure within the PHY and to eliminate any
possible cause of errors. This debug mode is useful for user when implementing Intel FPGA and Aquantia
PHY with external devices, such as Ethernet switch or router.
There are two main loopback modes supported by Aquantia PHYs:
1. Line Side Loopback mode
• Packets received from the line-side interface are sent back to the link-side interface.
• There are two type of line side loopback supported, which is the shallow and deep type.
• Figure 5 below shows this loopback mode is named as Network - System Loopback
(Deep system facing loopback).
2. System Side Loopback mode
• Packets received from the system-side interface and sent back to the system-side
interface.
• This mode is not tested in this design as not applicable to use as the debug mode.
When the loopback mode is enabled, auto negotiation function will be disabled. User need to manual
set the loopback rate. All the loopback enable and set the loopback rate can be done by using the
Aquantia PHY register map. When the loopback mode enabled, user must set the loopback rate in order
to make the link up. User need to turn on the loopback mode for the Aquantia PHY by using the
Aquantia GUI, choose the device and go to Utilities > Register Guide and write register 4.C444. Table
below shows the register values for different loopback types and rates.
23
Table 4. Aquantia PHY Register 4.C444 Loopback Type
Register 4.C444.[F:B] Loopback Type
0x00 No loopback
0x01 System - System Loopback (Shallow system facing loopback)
0x03 System - Network Loopback (Deep network facing loopback)
0x09 Network - System Loopback (Deep system facing loopback)
0x0B Network - Network Loopback (Shallow network facing loopback)
Table 5. Aquantia PHY Register 4.C444 Loopback Rate
Register 4.C444.[2:0] Loopback Rate
0x0 Reserved
0x1 100Mbps
0x2 1Gbps
0x3 10Gbps
0x4 2.5Gbps
0x5 5Gbps
24
Figure 5. Aquantia supported loopback paths
PCS PMAUSXGMII
System I/F
Line
System Interface – System Loopback (PCS loopback)
Network Interface – System Loopback (PMA loopback)
System Interface – Network Loopback (PHY-XS loopback)
Network Interface – Network Loopback
25
Reference Design Debug Features
The debug strategy below helps user to make a self-debug if you encounter system bring up issue. The
recommended steps below help make a high level isolation on which device cause the link down, either
the FPGA system or the Aquantia PHY side.
Refer to steps below on how to perform the FPGA only test:
1. Change the test setup with the reference to Figure 7.
Figure 6. NBASE-T Sub system (FPGA loopback mode, without Aquantia PHY)
Arria 10 Transceiver Signal Integrity Development Kit
Arria 10 GX FPGA
NBASE-T Ethernet Subsystem (LL10GMAC
+ MGE PHY)
SFP+
Arria 10 Transceiver Signal Integrity Development Kit
Arria 10 GX FPGA
NBASE-T Ethernet Subsystem (LL10GMAC
+ MGE PHY)SFP+
JTAG JTAG
PC and System Console
Subsystem
SFP+ loopback adapter module SFP+ loopback adapter
module
2. Next remove the two Aquantia evaluation boards, and connect the SFP+ cage of Arria 10
Transceiver SI kit with a SFP+ loopback adapter module which is used for 10GbE Ethernet
compliance testing and copper/optical cable emulation.
3. For user that have not completed the steps to bring up the test from testing the design section,
may refer to steps 5 – 14 to bring up both FPGAs, and continue with step 20 -22 to run the
traffic test on both FPGA kits and the traffic results from both TX and RX statistic counters.
26
Related Links
FTDI Driver
http://www.ftdichip.com/FTDrivers.htm
Aquantia Customer Portal
https://portal.aquantia.com/customer-portal
Intel Arria 10 Low Latency Ethernet 10G MAC Design Example User Guide
https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug-20016.pdf
Section 1 Quick Start Guide on page 5
Describes on how to generate the design example from Quartus IP Parameter Editor.
Arria 10 GX Transceiver SI Development Kit Downloadable Content
https://www.altera.com/products/boards_and_kits/dev-kits/altera/kit-a10-gx-si.html
Download the Arria 10 GX Transceiver Signal Integrity Installation Package (Production Edition)
Arria 10 Transceiver PHY User Guide
Section 2.6.5 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP Core page 198
Contain the PHY registers definition.