Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
CERN-IPMC DevKit: User guide
This document introduces the CERN-IPMC Development kit. This setup is a start-up platform that
can be used as a module tester or a debugging platform. This paper lists the features available
and how to use them.
CERN-IPMC DevKit – User guide v.4.0b
Sources: CERN-IPMC – https://cern-ipmc.web.cern.ch Support e-mail – [email protected]
CERN-IPMC DevKit – User guide EP-ESE-BE
2
Table of Contents 1. Introduction ......................................................................................................................................... 3
2. Hardware ............................................................................................................................................. 4
2.1 Power .................................................................................................................................................. 5
2.2 Management interface ........................................................................................................................ 5
2.3 UART interfaces ................................................................................................................................. 6
2.4 Ethernet .............................................................................................................................................. 6
2.5 I2Cs ..................................................................................................................................................... 6
2.6 IPMB Bus ........................................................................................................................................... 7
2.7 IPMC JTAG interface ......................................................................................................................... 7
2.8 JTAG Master Interface ....................................................................................................................... 8
2.9 Management controller reset .............................................................................................................. 8
3. Software ............................................................................................................................................... 9
3.1 Connection to Raspberry PI ................................................................................................................ 9
3.2 JTAG Player ....................................................................................................................................... 9
3.3 IPMB Analyser ................................................................................................................................. 10
3.4 DevKit Control tool .......................................................................................................................... 12
3.5 IPMC Test tool.................................................................................................................................. 13
CERN-IPMC DevKit – User guide EP-ESE-BE
1. Introduction
AdvancedTCA standard defines Intelligent Platform Management Controllers (IPMC) to
control the user blade. It is responsible for hardware management and monitoring with the hot
swap control and temperature/voltage checking. As the module is mandatory for all of the blades
and ATCA was selected for the electronics upgrade of the CERN experiments. The EP-ESE group
has designed an IPMC solution based on the Pigeon Point’s product. It was adapted to be
configurable and run on a common hardware providing all of the interfaces required to control
a blade. Therefore, the pinout provides connectivity to: 9 AMC ports, 51 User I/Os, 3 UART
interfaces, 2 I2C buses, one JTAG master interface as well as all of the specified pins required to
detect the blade position and status. Additional details can be found in the technical overview
document1.
In this context, a system was designed to test all of the interfaces after production. This
product was extended as a development kit for the user as it offers a set of tool that helps
debugging and learning how the module works.
1 CERN-IPMC Tecnical overview: https://cern-ipmc.web.cern.ch/doc
CERN-IPMC DevKit – User guide EP-ESE-BE
4
2. Hardware
The CERN-IPMC development kit is made of an electronic cards that can be used out or in
an AdvancedTCA crate. The block diagram below details the system.
According to scheme, all of the interfaces are connected. The system is supervised via the
uMGT controller, which receives UART commands on its dedicated USB port – named
Management. The “Management interface commands” annexe lists all of the instructions that
can be used. Because of the limited number of ports on the 8bit controller, a CPLD is used as a
multiplexer for the user I/Os. The same idea has been used to avoid duplicating the AMC
emulation bloc 9 times. Therefore, only one I/O or one AMC port can be tested at the time. The
master JTAG port is connected to a 10 pin header – named Master JTAG –, allowing programming
an external FPGA, while the slave JTAG to program the IPMC is either connected to a second 10
pin header – named IPMC JTAG – and the raspberry pi. All of the I2C ports are connected to
multiplexing mechanism, which can be used to redirect a port to a connector and GPIOs pins of
the raspberry pi. Additional components, which are a sensor temperature and an EEPROM
memory, are also connected to the sensor and management buses coming from the IPMC.
Finally, the IPMB-A/B buses are also connected to a 10 pin connector compatible with the shelf
manager development kit from Pigeon Point (KIT BTC 7200R ATCA).
CERN-IPMC DevKit – User guide EP-ESE-BE
5
2.1 Power
The CERN-IPMC devkit is designed to work in both standalone and in-crate mode.
Therefore, a power connector is located on the front side of the blade and can be used as
described below:
Figure 1:Power connector
In order to protect the board of input conflicts, a switch allows selecting the power source
from external or backplane. The 3 position switch selects:
Top position: external power
Middle position: not connected
Bottom position: backplane source
The picture bellows show the switch, which is located next to the external input connector.
Finally, the additional jumper SW1 connects the 5 volts to the input of the 3.3 volts
regulator, which supply the IPMC module and management controllers. Removing the jumper
turn-off the module under test, allowing the operator to change the IPMC without rebooting the
raspberry pi.
2.2 Management interface
The CERN-IPMC development kit control is made via the USB to Serial interface available
on the “management” connector located on the front of the card. This input is linked to the
CERN-IPMC DevKit – User guide EP-ESE-BE
6
ATMEGA128 controller through an FTDI chip, which takes care of the USB to Serial conversion.
Once the FTDI drivers are installed on the master computer, a new serial port is detected and
can be used using the following configuration:
9600 Bauds,
8bit,
No parity,
No control.
The implemented commands are listed into the “Management interface commands”
annexe, located at the end of the document.
2.3 UART interfaces
All of the UART interfaces available on the CERN-IPMC modules are routed to USB ports
using FTDI chip for translation. In this context, UART[0] is connected to pins 57/60, UART[1] to
pins 58/61 and UART[2] to pins 75/76 or 92/93 depending on the SW4/SW5 configuration. The
latest UART is the optional one from the CERN-IPMC. It is based on user I/Os pins and can be
routed to USER_IO[0:1] (pin 75/76) or USER_IO[24:26] depending on the SW4/SW5
configuration. The pictures below shows the options:
2.4 Ethernet
An Ethernet port is available on the IPMC mezzanine card. It can be used to send IPMI
commands (RMCP/RMCP+) directly to the controller, to open a Serial Over LAN (SOL) session
and/or to configure a Xilinx FPGA using the Xilinx Virtual Cable server. This port is directly
connected to the Ethernet connector located on the bottom of the front side. It can be directly
connected to the Raspberry pi.
2.5I2Cs
The CERN-IPMC implements 3 types of I2C buses: IPMB (A, B and L, which are connected
to the MMC or the IPMB connector), the sensor I2C and the Mgt I2C. In order to test these two
last buses, a temperature sensor (LM75) is connected to the Sensor I2C bus and an EEPROM
(AT24C64D) to the Mgt bus. Additionally, all of the I2C buses can be redirected to the I2C OUT
CERN-IPMC DevKit – User guide EP-ESE-BE
7
connector through I2C buffers, which are enabled depending the configuration set by the
management controller. The picture below shows the connector pinout:
Additionally, the IPMB interfaces, as well as the output selected on I2C OUT, are routed to the
Raspberry pi connector, allowing to spy the buses and analyse IPMI traces using Wireshark.
Additional details are introduced in the Software section of the document.
2.6 IPMB Bus
In order to allow using the CERN-IPMC on benches, the IPMB-A and IPMB-B buses are
routed to a 10 pins connector on the top of the IPMC development kit. The pin assignment was
designed in a way to be compatible with BTC 7200R ATCA kit from Pigeon Point. However, when
the cars is used within a crate, the IPMB buses are routed to the backplane connector and the
IPMB-A/B one should be left unconnected. The picture below shows the pinout of the 10 pins
connector:
2.7 IPMC JTAG interface
Even if the HPM.1 is recommended to configure the module with the latest version of the
software. The mezzanine card implements a JTAG connector allowing to flash the controller.
Therefore, this connectivity is reported to a 10 pin connector located on the top of the card –
IPMB JTAG. It connects the pins to be compliant with the Flashpro tool from Microsemi as
described below:
CERN-IPMC DevKit – User guide EP-ESE-BE
8
Additionally, the JTAG slave interface is also connected to the raspberry pi, allowing to
program the card via a JTAG player running on GPIOs. In case of Raspberry Pi use, the SW6 jumper
has to be set to VCC (RPI configuration is not supported yet). How to configure the controller via
a Raspberry pi is described in the Software section.
2.8 JTAG Master Interface
The CERN-IPMC JTAG master interface is directly routed from the DIMM-DDR3 VLP
connector to a 10 pin one, located on the top of the card – MASTER JTAG. It allows connecting
the interface to external devices, like Xilinx development kit. The module was tested, at CERN,
with a Basys 3 development kit from Diligent. The picture below describes the 10 pin connector:
2.9 Management controller reset
The management controller of the development kit can be set in reset by connecting the SW3 jumper.
CERN-IPMC DevKit – User guide EP-ESE-BE
9
3. Software
The CERN-IPMC development kit software runs on a Raspberry pi version 3B+, included in
the setup. The operating system, based on Raspbian – Linux –, is pre-installed on the SD card and
support a set of tools allowing to play with the setup.
In order to use the tools described below, one USB port of the Raspberry has to be
connected to the Management interface, as well as the Ethernet port.
3.1 Connection to Raspberry PI
On the first boot, the WLAN interface is not configured. Therefore, it is recommended to
connect the HDMI output to a monitor and directly control the raspberry from a local keyboard.
The graphical user interface can be used to configure the wireless network in order to get the
system ready. Once the module is connected to the network, it is possible to access via SSH with
the following credentials:
Username: pi
Password: raspberry
Note: It is strongly to keep the Ethernet interface configuration as they are, except for expert
person.
3.2 JTAG Player
The JTAG player tool allows reconfiguring the CERN-IPMC with the default binaries.
Therefore, the user customization will be loose. It is recommended to use the interface when
new firmware is available or to reset a module to manufactory default. The program can be
simply ran using the “jtag_player -reset” command.
CERN-IPMC DevKit – User guide EP-ESE-BE
10
When the configuration finishes, the program print a dictionary providing details of the
different programming stages. On success, the result shall be similar to the one presented in the
picture below:
The JTAG player tool can also be used to flash a custom firmware by replacing the -reset
instruction by a filename. However, this action is not recommended except in case of a specific
message from the support team.
3.3 IPMB Analyser
The IPMB analyser tool allows sniffing one of the IPMB bus connected to the CERN-IPMC:
IPMB-A, IPMB-B or IPMB-L. It captures all of the bytes exchanged on the I2C lines and store it into
PCAP files that can be read with Wireshark2. It can be executed using the following command:
> ipmb_sniffer -t <timeout> -f <output file> -p <port>
2 A specific version of wireshare, patched to analyse IPMB packet, can be downloaded here.
CERN-IPMC DevKit – User guide EP-ESE-BE
11
Where the parameters are:
Timeout: Acquisition time in second. Wait for Enter to be pressed when not specified.
Output file: Path to the output file where the data are saved.
Port: Select IPMB port to spy. The value can be ipmba, ipmbb or ipmbl.
The picture below shows an example of data capturing with a timeout set to 60 seconds,
on the IPMB-L bus.
When the timeout value is not set, the program wait for the touch Enter to be pressed to
stop. In this case, the acquisition delay is not based on time but on a user input.
Finally, the files can be extracted from the Raspberry using an SCP program or via
MobaXterm for example. Then, it can be opened in Wireshark, which can analyse the transaction
and make them verbose. The picture below shows the result of the IPMB-L acquisition.
CERN-IPMC DevKit – User guide EP-ESE-BE
12
3.4 DevKit Control tool
The DevKit Control tool allow playing with the IPMC interfaces connected to the uMGT
controller. It sends the instruction detailed in the “Management interface commands” annexe
depending on the action passed in parameter. The list of available commands can be get using
the -h argument:
> devkit_ctrl -h
Depending the command, additional parameter might be requested. Using the “-a <action>
-h” argument explains the usage of the specific command. Then, calling the program with the
right action and arguments display the result in the terminal. The picture below shows a call of
the get_power instruction.
Because it might be interesting to script a list of instruction, the tool also support JSON
output. It shall be specified using the “--json" argument when calling the program, as illustrated
below.
CERN-IPMC DevKit – User guide EP-ESE-BE
13
3.5IPMC Test tool
The IPMC Test tool perform a check of the CERN-IPMC hardware. It is based on the same
test that is performed on module reception after production. The tester print information during
the execution and can store all of the result in a pickle file when the output argument is defined
(optional). The tester can be executed using the following command:
> ipmc_test [-o <output.p>]
CERN-IPMC DevKit – User guide EP-ESE-BE
14
Annexe 1: Connectors description
The table below lists all of the connectors:
REF. DESCRIPTION
J1 Do not used (uMGT JTAG connector)
J2 Do not used (AMC JTAG connector)
J3 Do not used (CPLD debug connector)
MASTER JTAG Master JTAG from IPMC:
Pin 1: TCK
Pin 3: TDO
Pin 4: VCC (3.3V)
Pin 5: TMS
Pin 6: TRST
Pin 9: TDI
Pin 10: GND
IPMC JTAG Slave JTAG to CERN-IPMC:
Pin 1: TCK
Pin 2: GND
Pin 3: TDO
Pin 6: VCC (3.3V)
Pin 5: TRST (Payload_nrst)
Pin 10: TDI
IPMB-A/B IPMB-0 connector:
Pin 1: IPMB-B SCL
Pin 2: GND
Pin 3: IPMB-B SDA
Pin 4: GND
Pin 6: GND
Pin 7: IPMB-A SCL
Pin 8: GND
Pin 9: IPMB-A SDA
Pin 10: GND
I2C OUT I2C output from mux. selection:
Pin 1: SCL
Pin 2: GND
Pin 3: SDA
Pin 10: GND
CERN-IPMC DevKit – User guide EP-ESE-BE
15
Annexe 2: Jumpers description
The table below describes all of the jumpers:
SWITCH DESCRIPTION
SW1 IPMC and uMGT enable: connect IPMC and uMGT power to VCC.
SW2 -48V to 5V DC/DC enable: turn ON backplane power DC/DC
SW3 uMGT reset: reset uMGT controller when set.
SW4/SW5 UART[2] routing:
No jumpers: not connected
[0:1]: connected to user I/O[0:1]
[24:26]: connected to user I/O[24:26]
SW6 IPMC JTAG TRST signal (used as payload_nrst during runtime):
No jumper: recommended except for programming. Pin
used as payload_nrst.
VCC: Force TRST signal for module programming
RPI: Connected to a raspberry pi GPIO (automatic
control)
CERN-IPMC DevKit – User guide EP-ESE-BE
16
Annexe 3: Management interface commands
COMMAND DESCRIPTION
DEVKIT RELATED COMMANDS
GETDEVICE Returns a description string “IPMCTP_CTRLER_V1.0”
GETMGTVOLTAGE Get MGT voltage (must be around 3.3V)
GETIPMCVOLTAGE Get IPMC voltage (must be around 3.3V)
GETIPMCCURRENT Get IPMC drawn current
SELI2C 0x<id> Select I2C port to redirect on “beagle_ipmb_l1” connector:
<id> values:
00: IPMB-A
01: IPMB-B
02: IPMB-L
03: IPMC Sensor
04: IPMC Mgt
ATCA RELATED COMMANDS
SETHA 0x<addr> Set the IPMC Hardware address (addr must be an hex. value)
SETHSW Set the IPMC’s handle switch pin to VCC
CLRHSW Set the IPMC’s handle switch pin to GND
SETIO 0x<id> Set the selected IO to VCC (id must be an hex. value)
CLRIO 0x<id> Set the selected IO to GND (id must be an hex. value) *1
GETIO 0x<id> Get value of the selected IO (id must be an hex. value) *1
GETBLUELED Get blue led status *2
GET12VEN Get 12V Enable status
ASSERTPGOODA Assert PowerGood A signal
ASSERTPGOODB Assert PowerGood B signal
DEASSERTPGOODA De-assert PowerGood A signal
DEASSERTPGOODB De-assert PowerGood B signal
AMC RELATED COMMANDS
SETAMCGA 0x<addr> Set AMC’s geographical address
SELAMC 0x<port> Select the IPMC port to be connected to the AMC
SETPS1 Emulate AMC insertion
CLRPS1 Emulate AMC extraction
CLOSEAMCHS Close AMC handle switch
OPENAMCHS Open AMC handle switch
GETAMCMPEN Get MP Enable signal from AMC emulator
GETAMCPPEN Get PP Enable signal from AMC emulator
ASSERTMPGOOD Assert MP good signal on AMC emulator
DEASSERTMPGOOD De-assert MP good signal on AMC emulator
ASSERTPPGOOD Assert PP good signal on AMC emulator
DEASSERTPPGOOD De-assert PP good signal on AMC emulator