An Interop Event for NVMe SSDs Thursday October 2, 2014 2PM
Eastern NVMe Plugfest Webinar Presented by: David Woolf - UNH-IOL
[email protected] 2014 University of New Hampshire InterOperability
Laboratory
Slide 2
2 UNH-IOL Industry leading 3 rd -party test facility for data,
telecom and storage networking technology & consumer
electronics 100% funded by commercial industry 150+ companies
32,000 sq. ft lab facility Boston, USA Metro Area 7,200 sq. ft
pre-wired space dedicated to Plugfests 2 2014 University of New
Hampshire InterOperability Laboratory
Slide 3
3 Member Involvement (150+ Companies) 3
Slide 4
4 UNH-IOL NVMe Consortium Benefits Access to UNH-IOL NVMe test
services. Attendance at NVMe Plugfest Access to UNH-IOL INTERACT PC
Edition and UNH-IOL INTERACT Teledyne-LeCroy Edition software for
performing NVMe Conformance Tests. Ability to list qualifying
products on the NVMe Integrators List: Place products in the
UNH-IOL NVMe Interop test bed for continuous interop testing.
Support for UNH-IOL Administration of open source tNVMe tool PCIe
Electrical testing available through UNH-IOL PCIe Consortium
Slide 5
5 NVMe Plugfest #3 UNH-IOL November 10-13, 2014 Durham, NH
USA
Slide 6
6 Agenda NVMe Test Program Goals and Overview Plugfest
Components Interop Tests Conformance Tests Plugfest Logistics
Registration Test Stations Walk-Thru Reporting Listing Products
Testing After the Plugfest
Slide 7
7 Agenda NVMe Test Program Goals and Overview Plugfest
Components Interop Tests Conformance Tests Plugfest Logistics
Registration Test Stations Walk-Thru Reporting Listing Products
Testing After the Plugfest
Slide 8
8 Growth of NVMe Enterprise SSD market is expected to reach $4
billion in revenues by 2016, nearly six times that of 2011, while
unit shipments are expected to increase by ten times during that
period, to approximately 4 million units. - Forbes.com July 2014
NVMe is the next gen interface for both client and enterprise
systems. NVMe products announcements are happening There is a
strong need to demonstrate the interoperability of NVMe products.
The NVMe plugfest at UNH-IOL is a key component in accomplishing
this.
Slide 9
9 Since 2012 NVMe Promoters Group collaborating with UNH-IOL to
manage the Integrators List based Test Program for NVMe. NVMe Test
Program Overview
Slide 10
10 NVMe Integrators List
https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php
NVMe Integrators List hosted by UNH-IOL as of Sept 30, 2014:
Slide 11
11 NVMe Integrators List NVMe Integrators List is the industry
resource for listing products proven to interoperable and
conformance according to industry accepted test practices. Listed
products demonstrate commitment to interoperability and open
standards conformance. OEMs viewing Integrators List qualification
as necessary for components.
Slide 12
12 NVMe Test Program Goals Prove the robustness and
interoperability of NVMe through continually improving interop and
conformance testing. Help implementers ensure the conformance and
interoperability of their products Provide recognition for products
meeting the necessary qualifications backed by test reports.
Slide 13
13 How do we populate the Integrators List? New Interop and
Conformance Testing Requirements approved by NVMe Promoters group
New Testing requirements launched at plugfests. Testing for
Integrators List performed throughout the year at UNH-IOL.
Slide 14
14 Agenda NVMe Test Program Goals and Overview Plugfest
Components Interop Tests Conformance Tests Plugfest Logistics
Registration Test Stations Walk-Thru Reporting Listing Products
Testing After the Plugfest
Slide 15
15 NVMe Plugfest Components NVMe Plugfest #3 November 10-13,
2014 will test according to version 1.1b of the NVMe Test Program.
First opportunity to test devices to the new requirements and have
them listed on IL. After the Plugfest, UNH-IOL members can request
1.1b Integrators List testing throughout the year.
Slide 16
16 Test Plans 2 Test plans will be executed at the plugfest.
1.UNH-IOL NVMe Interoperability Test Suite Document 1.1b 2.UNH-IOL
NVMe Conformance Test Suite Document v1.1b
Slide 17
17 Common Test Terms Mandatory DUT must pass test in order to
qualify for IL. Optional If DUT supports an optional NVMe feature,
it must pass test inorder to qualify for IL. If option is not
supported, the test is not performed and does not affect IL
qualification. FYI Test does not affect IL qualification, but may
become mandatory in the future.
Slide 18
18 Passing Requirements Overview Conformance Tests DUT must
pass all mandatory conformance tests supported by test tools If
feature is supported, DUT must also pass optional tests supported
by test tools
Slide 19
19 Passing Requirements Overview Interop Tests Plugfest #3
(Today) DUT must pass mandatory tests with 4/5 systems. FYI tests
do not affect IL qualification, are only performed with 1 system
Plugfest #4 (2015) DUT must pass mandatory tests with 5/6 systems.
FYI tests will become mandatory, are only performed with 1
system
Slide 20
20 Interop Test Plan Mandatory Tests (1.1, 1.2, 1.3, 1.4)
Install SSD, identify drive in OS Format drive, write/read/verify
for 3 minutes Reboot 3x/shut down 3x FYI Tests (1.5, 1.6) 1.5 -
Boot from NVMe device 1.6 - FYI Hotplug test (SFF-8639 only) Will
run all SSD and Host combinations possible at the event. Publically
available at:
https://www.iol.unh.edu/services/testing/NVMe/testsuites
Slide 21
21 Interop Test Tools VDBENCH 5.04 Command Line utility for
generating IO Workloads to disk. Java program works on Windows,
Solaris, Linux stations. Available at oracle.com
http://www.oracle.com/technetwork/server-
storage/vdbench-downloads-1901681.html Download vdbench Parameter
File for NVMe Interop Tests created by UNH- IOL.
https://www.iol.unh.edu/services/testing/
NVMe/unh_interop_1_1b_linux.txt
https://www.iol.unh.edu/services/testing/
NVMe/unh_interop_1_1b_windows.txt
Slide 22
22 UNH-IOL will have 2 SFF-8639 to CEM adapters on hand. If you
have a non CEM form factor, please bring it, and also please bring
at least 2 of your own adapters. This will facilitate interop
testing. SFF-8639 / M.2 Adapters 22 PCI-AD-x439-01 adapter from
Serial Cables (serialcables.com)
Slide 23
23 Conformance tests will only require 1 sample. Interop tests
require that 2 samples be used simultaneously. Please bring at
least 2 samples of your product, this will facilitate interop
testing. How many samples? 23
Slide 24
24 What Interop Tests are Required for Plugfest #3? Test
GroupMandatoryOptional/FYI Test 1.1 Storage Device Identified Pass
with 4/5 Hosts Test 1.2 Format Storage Device Pass with 4/5 Hosts
Test 1.3 Write Read Compare Pass with 4/5 Hosts Test 1.4 Multiple
Devices on Bus Pass with 4/5 Hosts Test 1.5 Boot from NVMe Device
Mandatory for Plugfest #4 Pass with 1 Host: Dell PowerEdge R720
Test 1.6 Hotplug NVMe Device Mandatory for Plugfest #4 Pass with 1
Host: TBD Test 1.7 Dual Port Device Not implemented
Slide 25
25 Interop Systems PlatformOSIntended Testing Dell PowerEdge
R720Windows Server 2012 (OFA Reference Driver) Write/Read/Verify
Intel Patsburg CRBWindows 8.1 (Microsoft Driver) Write/Read/Verify
Intel Patsburg CRBUbuntu 14.04 (Reference Driver) Write/Read/Verify
SUPERMICRO MBD-X10SAT-O ATX Server Motherboard RHEL 6.5 (Reference
Driver) Write/Read/Verify SUPERMICRO MBD-X10SAT-O ATX Server
Motherboard Windows Server 2012 R2 (Microsoft Driver)
Write/Read/Verify TBD Hotplug (FYI) Dell PowerEdge R720UEFI /
Windows 8.1Boot (FYI)
Slide 26
26 Conformance Test Plan Use Host Emulator to exercise and
stimulate SSD. Focus on completion of Admin Commands, NVM Commands,
Controller Registers Publically available at:
https://www.iol.unh.edu/services/testing/NVMe/ testsuites
Slide 27
27 Conformance Test Tools IOL INTERACT PC EDITION IOL INTERACT
TELEDYNE- LECROY EDITION
Slide 28
28 Test Tools IOL INTERACT PC EDITION Linux based NVMe
Conformance Test Station uses: tNVMe v.3.0 available from github
(UNH-IOL admin) IOL INTERACT 1.1 (available from UNH-IOL) RELEASED
July 30, 2014 Ubuntu 10.10, 12.04, 13.04, 14.04 Executes most
required conformance tests defined in UNH-IOL NVMe Conformance Test
Suite Document Installation walkthru:
http://www.flashmemorysummit.com/English/Collat
erals/Proceedings/2013/20130812_PreConfD_Mehta _Woolf.pdf Obtained
through UNH-IOL membership Contact: [email protected]
Slide 29
29 Test Tools IOL INTERACT TELEDYNE-LECROY EDITION Requires
Teledyne-LeCroy Summit Z3 Exerciser and Analyzer, Windows PC
Currently on LeCroy Software Build 6.75, aiming for 7.2 by plugfest
time. Executes required conformance tests defined in UNH-IOL NVMe
Conformance Test Suite Document For final IL Qualification UNH-IOL
will examine bus traces from IOL INTERACT TELEDYNE-LECROY EDITION
Obtained through UNH-IOL membership. Contact:
[email protected]
Slide 30
30 Why 2 Conformance Tools? Community needs a tool with a
relatively low cost barrier, to be able to determine conformance.
The PC Edition meets that need. Community also needs a tool with
deep debug and analysis capability. Teledyne- LeCroy edition meets
that need. Tradeoff: Cost vs. Debug Capability + Trace
Analysis
Slide 31
31 Test Support Test Group IOL INTERACT PC Edition v1.1 IOL
INTERACT Teledyne-LeCroy Edition v1.2 Group 1: Admin Command Set
All tests except 1.5 - Abort Command All tests supported Group 2:
NVM Command Set All tests except 2.7 - Write Zeros All tests
supported Group 3: NVM FeaturesTests 3.2 and 3.3 not implemented.
All Tests Supported Group 4: Controller Registers All Tests
Supported Group 5: System Memory Structure All Tests Supported
Group 6: Controller Architecture No Tests implementedAll tests
except 6.1 Group 7: ReservationsNot Implemented
Slide 32
32 What Conformance Tests are Required? Test GroupMandatory
Optional (if supported must be supported correctly) Group 1: Admin
Command Set All Tests MandatoryNone Group 2: NVM Command Set 2.3,
2.4, 2.62.1, 2.2, 2.5, 2.7 Group 3: NVM Features3.33.1, 3.2 Group
4: Controller Registers All tests except 4.6 Arbitration Mechanism
4.6 Arbitration Mechanism Group 5: System Memory Structure All
Tests MandatoryNone Group 6: Controller Architecture 6.1, 6.2,
6.36.4
Slide 33
33 Agenda NVMe Test Program Goals and Overview Plugfest
Components Interop Tests Conformance Tests Plugfest Logistics
Registration Test Stations Walk-Thru Reporting Listing Products
Testing After the Plugfest
Slide 34
34 Registration open to UNH-IOL members:
https://www.iol.unh.edu/servic es/testing/NVMe/plugfest.php
https://www.iol.unh.edu/servic es/testing/NVMe/plugfest.php
Membership https://www.iol.unh.edu/servic es/membership/form/
https://www.iol.unh.edu/servic es/membership/form/ Plugfest
Logistics - Registration
Slide 35
35 Companies developing or deploying NVMe products Companies
supporting NVMe products in their systems Products should be
functional to the point that they are identified as a storage
device in a Windows or Linux system with an NVMe driver. No value
in attending plugfest without a functional product. Plugfest
Logistics Who should attend?
Slide 36
36 Each SSD will be cycled through multiple conformance and
interop test stations and interop hosts. Conformance stations are
test tools. Interop Stations are NVMe Hosts with varying NVMe
drivers. Not all stations required for Integrators List
qualification Some stations are FYI, to experiment for future
tests. Plugfest Test Stations
Slide 37
37 Following PCI-SIG recommended test procedures SFF-8639 only
Help in tuning phy parameters in new form factors Tests are FYI
only, not required for Integrators List qualification FYI
Conformance Test Stations PCIe PHY Conformance Test Station
(FYI)
Slide 38
38 Additional Test equipment companies may attend to help
improve their test offerings. Any non UNH-IOL stations should be
considered FYI. FYI Conformance Test Stations Protocol Conformance
Test Station (FYI)
Slide 39
39 Hotplug Test Station Experimental test to help create
appropriate hot plug test cases for future. UEFI /Boot Test Station
Experimental test to exercise UEFI NVMe driver. Tests are FYI only,
not required for Integrators List qualification today. These tests
will be required at Plugfest #4 FYI Interop Test Stations Hotplug
Interop Test Station (FYI) UEFI/Boot Interop Test Station
(FYI)
Slide 40
40 Reporting Conformance Test Data will be collected directly
by UNH-IOL Staff Bus traces and logs Interop Data may be collected
by UNH-IOL staff, or engineers supporting their companies equipment
Summary and Error logs produced by VDBENCH
Slide 41
41 Reporting UNH-IOL will use collected test data to produce
Interop and Conformance Test Reports for all products at the
Plugfest Test Reports are necessary to list a product on the NVMe
Integrators List Reports are can be shared with partners and
customers
Slide 42
42 Agenda NVMe Test Program Goals and Overview Plugfest
Components Interop Tests Conformance Tests Plugfest Logistics
Registration Test Stations Walk-Thru Reporting Listing Products
Testing After the Plugfest
Slide 43
43 Listing Products NVMe Integrators List Policy:
https://www.iol.unh.edu/services/testing/NVMe/20131219_NVMe_Integrators_List_Policy_v1_1b.pdf
Test Reports are necessary to list a product on the NVMe
Integrators List UNH-IOL Report will indicate whether a product is
eligible or not Products are not automatically listed In order to
list a product, companies must make specific email request to
UNH-IOL through [email protected]
Slide 44
44 Listing Products Listed Information Includes: Product,
includes Manufacturer, Model Name and Family Name (i.e. different
capacities of one SSD are in a single family) Product type
(accepted products types are: NVMe SSD, SSD Controller, or SSD
Controller IP, server, server board, motherboard, add-in card, RAID
Controller, or IP device). Operating System and Driver (Host Only)
Firmware revision (Device Only) Interop Program Revision (i.e. 1.0,
1.1, 1.1b etc...) Date of listing and Test ID Further Information
(primary contact or website)
Slide 45
45 Listing Similar Products Products with differences that do
not affect NVMe operation (form factor, storage capacity) can be
listed under a single product listing, listed by similarity. The
products listed by similarity must have the same Firmware revision
and Model Name, and the listing company confirms that the products
are materially similar with no substantive changes to the NVMe
interface.
Slide 46
46 Agenda NVMe Test Program Goals and Overview Plugfest
Components Test Plans Test Tools Plugfest Logistics Registration
Test Stations Walk-Thru Reporting Listing Products Testing After
the Plugfest
Slide 47
47 Post-Plugfest Testing Products will continue to be tested at
UNH-IOL throughout the year. Test requests can be made through:
https://www.iol.unh.edu/ my/ https://www.iol.unh.edu/ my/ Passing
products can be listed on the NVMe Integrators List
Slide 48
48 Future Requirements Mandatory interop tests will increase
from 4/5 to 5/6 UEFI Boot test will be mandatory at Plugfest #4
Hotplug test will be mandatory at Plugfest #4
Slide 49
49 UNH-IOL NVMe Consortium Benefits: Access to UNH-IOL NVMe
test services. Attendance at NVMe Plugfest Access to UNH-IOL
INTERACT PC Edition and UNH-IOL INTERACT Teledyne-LeCroy Edition
software for performing NVMe Conformance Tests Ability to list
qualifying products on the NVMe Integrators List Place products in
the UNH-IOL NVMe Interop test bed for continuous interop testing
Support for UNH-IOL Administration of open source tNVMe tool PCI
testing available through UNH-IOL PCIe Consortium
Slide 50
50 Review NVMe Test Program focused on Integrators List to
showcase interoperable and conformant products. Test tools and
plans available from UNH-IOL. Plugfest will allow the first
opportunity to list products on the NVMe Integrators list v1.1b.
Plugfest will walk SSDs through multiple conformance and interop
stations. Testing can continue after the Plugfest for those unable
to attend.
Slide 51
51 Resources UNH-IOL Membership
https://www.iol.unh.edu/services/membership/form/ Plugfest
Registration
https://www.iol.unh.edu/services/testing/NVMe/plugfest.php NVMe
Conformance and Interop Test Suites
https://www.iol.unh.edu/services/testing/NVMe/testsuites/ NVMe
Integrators List and Policy Document
https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php
53 Q&A When will the 1.1b IOL-INTERACT Linux and Lecroy
tools be available for download? As far as test support, the
currently released tools support all the necessary tests. Future
releases prior to the plugfest will contain bug fixes and update
LeCroy software support.
Slide 54
54 Q&A FYI boot from NVMe will require an OS to be
installed. What if the NVMe device is too small for an OS to be
installed? Then the test cannot be performed. This is why we have
this as an FYI test today, so that you can properly prepare for the
next plugfest and have a device large enough for an OS then.
Slide 55
55 Q&A How does the CEM adapter support hot-plug testing as
the server motherboard PCIe slots also should support the hot plug
We will not be performing hotplug on CEM card systems or
devices.
Slide 56
56 Q&A What is the difference between github version and
iol-interact SW other than a gui interface? IOL INTERACT supports
test groups 3, 4, and 5, which tNVMe does not directly
support.
Slide 57
57 Q&A Slide mentions IOL-Interact 1.1, but not 1.1b
release In this case the 1.1 refers to the software version. The
1.1 software version supports most of the tests required for the
NVMe IL version 1.1b (see table on slide 31 ). Its just an
unfortunate coincidence that the numbers are close.
Slide 58
58 Q&A Is PCIe phy conformance available for non-CEM form
factors Yes Is UEFI testing available on any motherboard that can
be done locally? other platforms that support UEFI Yes, the test
can be performed on any platform that supports UEFI, but UNH-IOL
will only supply 1 platform at the plugfest.
Slide 59
59 Q&A Is there a deadline for registration? October 17,
2014 Do you use Solaris host for interop test? Currently we dont
have a Solaris host for this interop event. However, using a
Solaris host as one of the hosts for interop testing would still
help qualify an SSD for the IL.
Slide 60
60 Q&A We are interested in bringing a server instead of a
drive. Would you require 2 models the same as you want for drives?
Bringing a single server is fine. We only need two samples of
SSDs.