Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
INTEROPERABILITY TEST AND CERTIFICATION
MANAGEMENT ASSISTANCE ANALYSIS
Prepared for:The UCAIug OpenADE Task Force, UCAIug SG
Prepared by:The UCAIug OpenADE Task Force and QualityLogic, Inc
Managed by:UCAIug OpenADE Task Force
Version0.01
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 iUCA International Users Group December 12, 2011
1
2
Revision History
Rev Date Summary Marked0.01 2011-21-12 Initial draft for team review N
Open Editorial Items and Issues LogAs open items and issues are addressed in new versions of this document, they are removed from this list.
Item No.
Date Provided By
Summary of the Issue Status / Disposition
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 2UCA International Users Group December 12, 2011
3
4
56
78
Executive SummarySmart Grid Technical Standards are critically important to development of interoperable products, increasing competition, reducing costs of Smart Grid components, and speeding implementation of new technologies. These key principles enable mainstream market adoption of Smart Grid products and services. This is important for utilities and other market participants to be able to make viable infrastructure investments, and reduce project costs and product development investments required to leverage the opportunities that Smart Grid technologies make available.
The UCA International OpenADE Task Group developed the requirements that led to the North American Energy Standards Board, (NAESB) REQ18/WEQ19, PAP10 Energy Usage Information and NAESB REQ21 Energy Services Provider Interface (ESPI) standards. These are specifically referenced in the November 8, 2011, US Department of Energy Funding Opportunity to demonstrate or pilot innovative applications based on standardized energy usage availability. OpenADE/ESPI is a key standard in the NIST V2.0 Smart Grid Standards Roadmap and is the most important standard addressing technology standardization for enabling consumer access to energy usage information.
Two of the important tasks that the OpenADE Task Group plans to undertake are:
- Develop a conformance, certification, and testing process and program for OpenADE, coordinated with entities such as standard development organizations (SDOs), user groups, and Smart Grid activities.
- Develop programs to allow vendors to develop, test, and demonstrate their ability to integrate with OpenADE communications protocols.
Creating a test and certification program is a challenging activity, especially in light of the rigorous requirements embodied in V1.0 and Draft Version 2.0 of the Smart Grid Interoperability Panel (SGIP) Test and Certification Committee’s (TCC) Interoperability Process Reference Manual (IPRM)1.
This UCA International OpenADE Task Group is aiming to establish an accelerated test and certification program for OpenADE/ESPI (NAESB REQ 21) that can:
1. Accelerate the development and implementation of a successful interoperability certification program for OpenADE that meets or exceeds the applicable SGIP TCC IPRM requirements
1 See SGIP TCC TWIKI at http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/SGTCCIPRM, V1.0, November 18, 2010.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 3UCA International Users Group December 12, 2011
9
10111213141516
1718192021222324
25
262728
2930
31323334
3536
373839
12
2. Create a standard test specification, test cases, test scripts, test harness, and other tools that can be used by vendors and others to test interoperable OpenADE products prior to certification
3. Establish a maintenance and update process and program to e ensure currency of the certification program and pre-certification tools
The balance of this requirements document will review the requirements of the SGIP TCC IPRM; provide commentary on the challenges in achieving the IPRM goals; outline the tasks required to achieve the goals of OpenADE for the interoperability test and certification; and suggest a set of next steps for the Task Group.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 4UCA International Users Group December 12, 2011
404142
4344
45464748
49
Table of ContentsSGIP IPRM REQUIREMENTS.................................................................................................................... 7
TRADE ALLIANCE FUNCTIONS............................................................................................................... 16
1.1. ITCA Test and Certification Tasks Based on the IPRM......................................................171.1.1. Organize the ITCA (Implied Tasks)......................................................................................181.1.2. Manage and Promote the Standard (Implied Tasks).................................................................181.1.3. Organize the Certification Program....................................................................................191.1.4. Define Certification Program (Explicit Tasks)........................................................................191.1.5. Establish Vendor Partnerships (Implied Tasks).......................................................................201.1.6. Implement Certification Program (Explicit Tasks)...................................................................211.1.7. Improvements in the Standard and the Certification Program (Explicit Tasks)................................221.1.8. Cyber-Security (Explicit Tasks)..........................................................................................231.1.9. Governance (Explicit Tasks)..............................................................................................23
NEXT STEPS.......................................................................................................................................... 24
SCHEDULE............................................................................................................................................ 26
SUMMARY........................................................................................................................................... 27
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 5UCA International Users Group December 12, 2011
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
6768
69
AuthorsJames Mater, QualityLogic
Steve Van Ausdall, Xtensible / SCE
Edited by: Dave Jollota, QualityLogic
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 6UCA International Users Group December 12, 2011
70
71
72
73
74
75
SGIP IPRM RequirementsThe OpenADE Task Group is fortunate to have access to a ground-breaking guide to developing and managing a world-class test and certification program. Indeed, until the issuance of Version 1.0 of the SGIP IPRM, nothing like it existed for the Smart Grid (or any other industry) that we know of. Every trade alliance like UCA International OpenADE Task Group has had to create its own program, which has resulted in a great deal of variation in how such programs have evolved. Having a roadmap such as the IPRM can greatly accelerate achieving the goals of product interoperability based on a specific standard.
It is also critical to understand that the IPRM defines the standard against which the SGIP will be assessing the quality and maturity of certification programs for Smart Grid standards. Although NIST2 is not directly bound to accept the conclusions and recommendations of the SGIP (in terms of which standards to adopt), it is clear that the SGIP process is closely watched by NIST managers, and SGIP assessments of certification program maturity are expected to influence the decisions that NIST makes.
The IPRM Version 1 identifies some 86 formal requirements in five distinct areas that serve as a specification for what a good test and certification program for a Smart Grid technology standard should look like. In addition, a number of guidelines are discussed
2 NIST is the National Institute of Standards and Technology, US Department of Commerce. NIST established the Smart Grid Interoperability Panel specifically to engage a broad range of stakeholders in assisting with their mission to identify Smart Grid technology standards.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 7UCA International Users Group December 12, 2011
76
7778798081828384
858687888990
919293
345
that further clarify how such a program can achieve its goals of interoperable and conformant products based on the specification.
Version 2 of the IPRM is in draft form but contains significant changes from Version 1. These are noted in this analysis, but it should be understood that until Version 2 is approved by the SGIP these changes are subject to further modification.
The following is a high-level overview of the key requirements that the SGIP has embodied in the IPRM V1.0 and Draft Version 2. Not all may apply to the OpenADE Task Group, and those that do not will be noted as such.
The IPRM defines the structure and functions of an organization that takes industry responsibility for a test and certification program for a standard. The organization is identified as an ITCA, or Interoperability Test and Certification Authority, and is characterized by its authority and competence to design and implement a program.
1. Section 5.1: General Test Policies provides a high-level overview of key policies an ITCA should consider and adopt. These include:
a. The level and types of information provided to vendors for certifications
b. The information that should be included in the final test report
c. Any conditions or expiration limits placed on certifications
d. Conformance versus interoperability certifications. Conformance does not necessarily imply interoperability between products.
e. Trade-off between certification testing and economic/business considerations with attention to safety and Cybersecurity related issues
f. Establishing policies to ensure adequacy of test tools used in certifications
2. Section 5.2 details the requirements for a Test Suite Specification (TSS) and includes details. Section 5.3 is related and deals with attributes of a Test Profile in lieu of a complete TSS. A TSS consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. A test profile evaluates a subset of a TSS and is used to target specific areas of product interoperability.
a. A common Test Suite Specification (TSS)3 shall be established when multiple test labs are deployed to test the same standard and/or profile. If common unique test procedures are required to support this test suite, then
3 A Test Suite Specification (TSS) consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. Test cases would include both valid and invalid behavior tests. Each test case is further described step-by-step with test procedures and well defined pass / fail / indeterminate criteria, along with references.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 8UCA International Users Group December 12, 2011
9495
969798
99100101
102103104105
106107
108
109
110
111112
113114
115
116117118119120121122
123124125
6789
10
they shall also be defined. The TSS should be test tool agnostic. Test Suite Specifications (TSS) used for interoperability or conformance testing shall be managed in a well-defined, open and formal manner with change control.
b. The TSS shall be subject to revision control, including revision history, revision numbering, and a defect and expansion management process. The TSS should clearly identify the test purpose, references, resource requirements, test setup, procedures, observable results and possible problems / lessons learned with the test approach. Observables should clearly identify pass / fail / indeterminate requirements and informational elements.
c. A Test profile, which MUST be a subset of a TSS, specifies all mandatory and optional elements and restrictions of the standard specification and is treated as a companion to the technical standard, including submission to an SSO for formal standardization.
3. Section 5.4 enumerates the Technical Design of Test and Certification Programs. This section covers areas of general technical, inheritance, version control, general testing, conformance, testing, interoperability, performance, tools and test lead. There are 35 specific technical requirements for an ITCA. These are summarized below:
a. The ITCA MUST specify in the test program requirements those features that are mandatory, and those features that are optional. The ITCA shall require and enforce that vendors declare the optional features implemented in a product. (Tech-1 and -2)
b. The ITCA MUST require that implementations of optional features be tested and certified for conformance and interoperability. Furthermore, the ITCA shall define common test cases for that optional feature to be used by all test labs when testing for that optional feature. (Tech-3)
c. An ITCA MUST define the record handling and retention requirements to be followed by the TL and CB functions, consistent with requirements of ISO 17025 and ISO Guide 65. (Techn-4)
d. The ITCA SHALL specify conditions under which the use of components that have been certified by other programs can be used in products to be certified by the ITCA program. Basically, OpenADE could decide that specific components or classes of components are “pre-certified” but the ITCA remains responsible for ensuring conformance and interoperability of the OpenADE Task Group certified products. The ITCA SHALL implement a Compliant Portion Description (CPD)4 to be used as a guide
4 See Glossary of Terms in the IPRM for definition and further explanation of CPD
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 9UCA International Users Group December 12, 2011
126127128129
130131132133134135136
137138139140
141142143144145
146147148149
150151152153
154155156
157158159160161162163
11
for assembling a product based on compatible sub-components. (Tech-5 to Tech-8)
e. Another section deals with differing versions of certified products and how the certification program should handle them. ITCAs need to manage re-certifications and the identification of current versions that are certified. (Tech-9 to Tech-12)
f. There shall be a defined correlation between implementations and required testing, commonly called a Proforma Implementation Conformance Statement (PICS). (Tech-14)
g. The testing and certification program shall maintain a current and upcoming list of applicable test cases to be called a Test Case Reference List. These test cases should be defined in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA. There shall be a Test Plan derived from the Test Case Reference List and used by all authorized test labs. Tests shall be identified using the test plan. (Tech-13, Tech-15-16
h. The testing and certification program SHALL require that a static conformance review5 take place prior to testing a product. (Tech-17)
i. The testing and certification program shall first validate the tests, and implement them utilizing validated test tools. Golden reference test equipment may be utilized where appropriate. (Tech-18)
j. The testing and certification program shall assure that defined product test cases cover application profiles for specific feature sets and functions defined by the specific application profile, and implement interoperability evaluation within that application profile. Where practical, the testing and certification program shall assure that defined product test cases cover all feature sets and functions. (Tech-20 to Tech-22)
k. The ITCA SHALL classify common or major market products according to their application profiles and conduct certification tests based on those applications. The testing and certification program SHALL assure that defined product use cases are covered in application profiles. Interoperability testing and evaluation SHALL be implemented within those application profiles. (Tech-23)
l. A section deals with interoperability testing. The section deals with what are commonly called “plugfests” but also addresses the selection and use
5 A review of designed feature sets versus the specified PICS to determine the extent to which the features are supported by the IUT. This is the first step when a product enters a testing program. Generally the test lab requests that the implementer declare all supported feature sets in a product. This information is used to create the test plan for that product.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 10UCA International Users Group December 12, 2011
164165
166167168169
170171172
173174175176177178179
180181
182183184
185186187188189190
191192193194195196
197198
12131415
of “golden” reference units. Plugfests are optional for application interface standards. (Tech-24 and Tech-25)
m. ITCAs SHALL use reference test tools6 where appropriate to the technology under test (hardware and/or software) to provide a consistent and replicable approach in generating test results across ITCA test labs. Successful testing programs assure that there is a known reference or constant, to which the system is evaluated against the desired metrics to determine conformance. ITCA program tests that are performed across multiple test facilities SHALL implement processes to assure they are each measuring against a common known reference and achieving repeatable results regardless of location. (Tech-26 and Tech-27)
n. When used, a minimum of two golden units are to be selected by a defined process (Tech-28 to Tech-30)
o. If an ITCA Certification Program involves multiple Smart Grid systems, then the Program Requirements SHALL support end-to-end testing of Smart Grid systems involving multiple product implementations to the fullest extent possible. An ITCA SHALL involve all relevant parties to define various business logic models for the end-to-end system testing, and make scenarios and test harness systems available for testing. (Tech-30-31)
p. The testing and certification program shall ensure that when functional performance requirements are defined in an application profile, the performance test profile(s) shall be designed to implement test cases for evaluating these requirements. (Tech-32)
q. The ITCA SHALL ensure that test tools have a complete mandatory feature-set coverage of a standard. In cases where two or more implementations of optional features are available, the ITCA shall incorporate those feature-sets in the test tool. The ITCA shall define procedures and processes to validate the use of test tools and reference implementations. (Tech-33 to Tech-34)
r. An ITCA shall develop criteria for surveillance to insure that certified products and vendors fulfill the certification agreement(s). The surveillance is to be carried out by its certification body(ies). (Tech-35)
6 A number of terms are used in describing reference test tools such as “common test harness”, “golden reference test equipment”, and “golden reference test products”. Generally, these each represent test tools available to a test lab or end user to provide a consistent baseline test either as a standalone implementation or in concert with the many other types of test tools available.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 11UCA International Users Group December 12, 2011
199200
201202203204205206207208209
210211
212213214215216217218
219220221222
223224225226227228
229230231
16171819
4. Section 5.5 addresses Program and Field Experience Feedback and points to the importance of end-user and product vendor experience with the standard and the certification program.
5. Section 6 of Draft 4, Version 2 of the IPRM is new and addresses Cybersecurity Testing. ITCAs are responsible for coordinating and overseeing the Cybersecurity criteria as applicable to the testing and certification programs that they operate. Section 6.7 includes specific requirements of an ITCA for Cybersecurity testing but these are still in discussion in terms of the ITCA responsibility in this area. This section includes proposed requirements that:
a. The ITCA SHALL define the procedures and processes that will be used to validate interoperability Cybersecurity requirements. Such tests are usually initiated by each vendor, to verify that a vendor product meets an industry established level of security, and tested by independent third-party labs using the same testing procedures that are pass/fail in nature. (Sec-1)
b. The testing and certification program shall ensure that Cybersecurity functional performance requirements are defined, and test cases designed to evaluate the requirements. Further, ITCAs are responsible to qualify testing personnel for Cybersecurity training and experience. (Sec-2 and Sec-5)
c. If applicable, ITCAs are responsible for Digital Certificate programs. (Sec-3 and Sec 4)
d. ITCAs are responsible for requiring widely-accepted security stress testing, including static analysis and penetration testing; assuring security policy models drive testing; ensuring that vendors submit threat analyses as part of certification process; documenting programs and standards used for security testing; and incorporating component-based Cybersecurity concepts in the testing program.
One interpretation of this proposed section is that an ITCA needs to establish a distinct certification program just for Cybersecurity issues and include this testing in the certification program. The IPRM is not yet clear on the qualifications for a lab to do Cybersecurity testing.
6. Section 7.2, titled Governance, provides a structural prescription for an ITCA. The OpenADE Task Group intends to establish the ITCA for the OpenADE/ESPI and related standards, although one purpose of this document is to assist in determining what organization will actually operate the ITCA for OpenADE/ESPI. Key governance requirements are:
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 12UCA International Users Group December 12, 2011
232233234
235236237238239240
241242243244245246
247248249250251
252253
254255256257258259
260261262263
264265266267268
a. The ITCA defines and documents the interoperability test program for the standard and oversees its implementation, including roles, responsibilities and resources. The ITCA SHALL provide oversight to provide confidence that implementations of Standard(s) in certified products are indeed interoperable. This will be the primary task of the OpenADE Task Group. (Gov-1 and Gov-5 in D3Rev4)
b. The ITCA determines whether first-party testing, third-party testing or both are allowed and defines the circumstances and process for submission to a certification body (CB) as well as the CB responsibilities. (Gov-2 and Gov-3 in D3Rev4)
c. The ITCA SHALL define a corrective process for resolving reported interoperability problems (e.g., in the field or as part of the test) for products for which they are responsible.7 Further, it SHALL implement preventive processes to avoid recurrence of such problems. A problem may be associated with the specification, the test processes and procedures or the test data. (Gov-4 in D3Rev4)
d. A key function is to ensure that issues that arise through the certification test process are fed back to appropriate parties for clarification or inclusion in subsequent versions of the standard. (Gov-6 in D3Rev4)
e. The ITCA SHALL maintain a publicly available certified product and systems list. (Gov 7 in D3Rev4)
f. The ITCA shall maintain a test case reference and modification history list8. This is a current master list of all tests that are to be included in a product certification test plan. This helps a product implementer in preparing fully conforming and interoperable products for an upcoming certification and launch. (Gov-8 in D3Rev4)
g. A common TSS SHALL be established when multiple test labs are deployed to test the same standard and/or profile. If common unique test procedures are required to support this test suite, then they SHALL also be defined. The TSS should be test tool agnostic. Test Suite Specifications (TSS)9 used for interoperability or conformance testing SHALL be managed in a well-defined, open and formal manner with change control. (Gov 9 and Gov 10 in D3Rev4)
h. The ITCA shall minimize divergence of interoperability requirements interpretations. If an ITCA has multiple testing laboratories and certifying
7 The ITCA should use best efforts in contacting a standards body with respect to a specification; however, it is not their responsibility to resolve issues with the specification.8 See Glossary of Terms in the IPRM for definition and explanation of the test case reference list. 9 See Glossary of Terms in the IPRM for definition and explanation of the TSS.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 13UCA International Users Group December 12, 2011
269270271272273274
275276277278
279280281282283284
285286287
288289
290291292293294
295296297298299300301
302303
20212223
bodies, processes shall be in place to avoid quality differences and assure repeatable testing between the laboratories. One way to minimize divergence of interpretations is to limit the number of labs to only one. Another option for minimizing divergence is to have a technical lead (also known as a lead lab) responsible for properly interpreting conformance and interoperability issues. (Gov-12 in D3Rev4)
i. Any organization that certifies the actual test labs and processes needs to conform to ISO/IEC 65 Guidelines and SHALL include the International Classification for Standards (ICS) Codes applicable to the technologies for which certification activities are performed. Further, Accreditation SHALL be by an accreditation body that is signatory, in good standing, to the International Accreditation Forum (IAF) multilateral agreement for “Product.” This set of guidelines generally specifies the formal documentation and processes required of such a body as well as criteria for eliminating or minimizing potential conflict of interest issues. A brief discussion of these Guidelines is contained in the IPRM Annex on page 66. (Gov-11 in D3Rev4)
j. The IPRM REQUIRES that product certification be issued by an ISO/IEC 65 accredited third party independent of the testing organization. (Section 1.4: Intended Audience in Draft 4 of Version 2)
k. The proposed IPRM Version 2 (Draft 4) includes an additional REQUIREMENT over and above ISO/IEC Guide 65 – the independent trusted third-party certification authority MUST only allow the statement that products are interoperable only if the products actually demonstrated interoperability during testing. They MUST not assume interoperability from conformance testing. They MUST demonstrate it before it may be part of the products certification statement. (Section 2.2: Overview of ISO/IEC Guide 65, Draft 4 of Version 2)
7. Section 7.3 is titled Lab Qualification and basically requires that lab selection SHALL be done in a uniform and transparent procedure and that any labs performing certification of products for the ITCA be ISO 17025 certified. Further, the ITCA SHALL define requirements to qualify the personnel involved in the certification and testing process for its standard. A discussion of ISO 17025 is contained in the IPRM Annex. In summary, ISO 17025 focuses on two major areas of laboratory operations: 1) management requirements; and 2) technical requirements. The management requirements address issues such as a lab’s documented practices (i.e., both administrative and technical), impartiality of the lab in its operations, responsibilities for continuous improvement and issues resolution, and the active support and involvement of lab management in assuring commitment to complying with these criteria. The technical requirements focus on areas such as ensuring that lab staff are competent in performing their testing
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 14UCA International Users Group December 12, 2011
304305306307308309
310311312313314315316317318319320
321322323
324325326327328329330331
332333334335336337338339340341342343344
duties, assuring that the lab environment is adequate for services performed, assuring that test plans and other necessary operating instructions are documented and available, and that necessary equipment and software used for testing is calibrated, maintained and appropriate for its intended usage.
8. Section 4 deals with improvements to the overall process, the standard documentation itself, test labs, the test and certification program, etc. Reference is made to the preference that an ITCA solicit direct feedback from customers of the certified products to assess that they meet customer interoperability needs.
The IPRM includes additional recommendations for best practices for interoperability and conformance testing certification programs. Much of the material is covered in the requirements themselves but there are a number of useful recommendations that the OpenADE Task Group could utilize in designing their own programs.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 15UCA International Users Group December 12, 2011
345346347348
349350351352
353354355356
Trade Alliance FunctionsLike the UCA International IEC 61850 and CIM User Groups, the OpenSG Committee, OpenADE Task Group of UCA, is taking on the functions of an industry trade alliance when it assumes the responsibility for developing and operating an interoperability certification program for OpenADE/ESPI. In doing so, it is accepting responsibility for pioneering an activity within the traditional OpenSG and UCA International charter. The key characteristics of an interoperability certification program that distinguish it from prior UCA International efforts are:
1. The OpenADE Task Group is a typical OpenSG “requirements” specification activity that assumes other organizations will adopt and formalize an actual national or international technical standard to implement the requirements defined by OpenADE. Taking on the certification functions is perhaps a logical extension of the Task Group activities but is without precedent in OpenSG. Implementing a certification program implies a series of new activities that OpenADE will be responsible for including:
a. Defining the conformance certification profile
b. Making major decisions concerning implementation of the certification program as outlined in the IPRM and including additional decisions concerning resources, the certification business model, and developing and managing relations with test labs, product developers and others that are new and different from normal OpenSG relationships
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 16UCA International Users Group December 12, 2011
357
358359360361362363364
365366367368369370371
372
373374375376377
c. Ensuring quality control and maintenance of the certification program
d. Managing the issuance and currency of certifications and the awareness of them in the user community
e. Marketing the technology standard and the certification program to ensure participation and customer awareness
2. The last standard that evolved from OpenSG Committee work was OpenADR Version 2. The functions of a trade alliance were taken on by a separate industry organization designed just for this purpose, rather than UCA International.
3. UCA International has two precedent activities in the IEC 61850 and CIM interoperability test programs. Both are conducted on a volunteer basis but neither of them meets the requirements set in the IPRM. To do so will require significant new investment by UCA International, as well as taking on additional functions.
OpenADE/ESPI could model itself after the IEC 61850 and CIM User Group activities and may choose to do so (ignoring SGTCC IPRM conformance). The concerns would be in terms of timeframe and available volunteer labor to develop and manage a successful ITCA program.
In setting up and operating an Interoperability Test and Certification Authority (ITCA) (for lack of a better term to describe the functions incumbent on UCA and OpenADE/ESPI), there are a series of activities and responsibilities that are addressed specifically or implied in the SGTCC IPRM, most of them enumerated in a separate section. These will need to be addressed whether the UCA takes on this role, or some other entity.
The following is an attempt to organize the IPRM explicit and implicit requirements and suggested best practices for an ITCA into an actual task list as summarized below:
1.1. ITCA Test and Certification Tasks Based on the IPRM
According to the IPRM proposed Version 2, Draft 4:
An Interoperability Testing and Certification Authority (ITCA) is the program management organization, providing oversight for testing and certification activities associated with one or more standards or specifications, that takes responsibility to insure that interoperable products within the scope of the specific ITCA program are brought to market. The ITCA coordinates the participation of certification bodies and test labs for its program.
The following are the tasks that OpenADE/ESPI and/or UCA International will need to complete as part of its ITCA goals under the IPRM Version 1 and proposed Version 2.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 17UCA International Users Group December 12, 2011
378
379380
381382
383384385
386387388389
390391392393
394395396397398399
400401
402403
404
405406407408409410
411412
1.1.1. Organize the ITCA (Implied Tasks) Develop a business plan for the ITCA
o Determine the level of conformance to the IPRM that the ITCA will aspire to implementing, including independent ISO/IEC 65 and ISO 17025 certification requirements
o Establish charter, scope and legal framework, including legal documents as needed
o Establish Governance policies and procedures
o Determine and organize ITCA budget and Treasury functions
o Determine staff support requirements and resources available
o Determine overall budget and schedule for a prudent period of time (3-5 years)
IPR Policy development and implementation
Recruit members who can contribute both financial and staff resources
Set sponsorship and dues levels or acquire other sources of funding
Select leadership
Establish meeting and working protocols
1.1.2. Manage and Promote the Standard (Implied Tasks) Recruit vendors and customers to adopt and use the OpenADE/ESPI standard
Continue to support development of the standard (working with the appropriate SSO10)
Conduct conferences, meetings, trade show exhibits and plugfests
Represent the interest of the ITCA members at appropriate events and organizations to promote the standard
Develop and maintain a web site for the standard as part of promotion, for listing certified products and for acknowledging sponsors, members and contributors
Put in place intellectual property protections – copyrights, trademarks, etc.
10 SSO = Standards Setting Organization. This may be an international standards organization such as ISO, IEC, OASIS, IEEE, etc., or in some cases it may be a trade association such as the ZigBee Alliance.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 18UCA International Users Group December 12, 2011
413
414
415416417
418419
420
421
422
423424
425
426
427
428
429
430
431
432433
434
435436
437438
439
2425
1.1.3. Organize the Certification Program Organize the ITCA certifications and determine what organization will act as the
Certification Body. Actual certifications must be issued by independent third parties which are not the test labs and are accredited to ISO/IEC 65 Guidelines. The ITCA could become an ISO/IEC 65 certified body and issue certifications of conformance and interoperability itself or contract with such an organization to do so.
o Certification bodies (CBs) should be accredited to ISO Guide 65, General Requirements for Bodies Operating Product Certification Systems
o Test laboratories should be accredited to ISO 17025, General Requirements for the Competence of Testing and Calibration Laboratories
o The ITCA should have an agreement with an accrediting organization(s) to assure that Certification Body and Test Lab accreditation is being performed in accordance with the ITCA program scheme.
An ITCA should have a strong relationship with the SSO associated with the standard for the purpose of feedback towards standard improvement and clarification where there may be ambiguities
1.1.4. Define Certification Program (Explicit Tasks) Define the certification program for OpenADE/ESPI products
The IPRM REQUIRES that product certification be issued by an ISO/IEC 65 accredited third party independent of the testing organization
The proposed IPRM Version 2 (Draft 3) includes an additional REQUIREMENT over and above ISO/IEC Guide 65 – the independent trusted third-party certification authority MUST only allow the statement that products are interoperable only if the products actually demonstrated interoperability during testing. They MUST not assume interoperability from conformance testing. They MUST demonstrate it before it may be part of the products certification statement.
Establish a detailed work program with schedule and resources
Determine the business model(s) to be used
o Volunteer labor for all activities
o Combination volunteer and funded staff: which activities for which
o Contractors and/or vendors and labs to implement aspects of the ITCA management and programs
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 19UCA International Users Group December 12, 2011
440
441442443444445446
447448
449450451
452453454
455456457
458
459
460461
462463464465466467
468
469
470
471
472473
Agree on one or more Proforma Implementation Conformance Statement (PICS) or profiles that represent the most likely use cases for products based on the standard
Develop the high-level certification test specification based on the PICS
Determine how detailed test cases, scripts and test harness(es) will be developed and maintained and who will develop and maintain them
Develop a program overview and applicant preparation guide
Define defect tracking and issue tracking requirements for both the technical program and tools and the business functions
1.1.5. Establish Vendor Partnerships (Implied Tasks) Determine philosophy of ITCA management (volunteer or professional)
o Develop RFP if professional management is determined
o Identify potential ITCA managers and solicit proposals in response to the RFP
o Select and develop contract with manager
o Manage contract manager activities and set policy
Determine if one or more test labs will be required and whether or not the structure needs to be developed to add test labs in the future
o Develop RFP for test lab(s)
o Identify potential test labs and solicit proposals in response to the RFP
o Select and develop contract with test lab
o Manage contract activities with test lab
o Follow guidelines in ISO Guide 65 to ensure that Labs are ISO 17025 accredited and maintain accreditation
Determine if a separate test tool vendor will be required and whether or not the structure needs to be developed to add vendors in the future
o Develop RFP for test tool vendor
o Identify potential vendors and solicit proposals in response to the RFP
o Select and develop contract with test tool vendor
o Manage contract activities with test tool vendor
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 20UCA International Users Group December 12, 2011
474475476
477
478479
480
481482
483
484
485
486487
488
489
490491
492
493
494
495
496497
498499
500
501
502
503
Determine if a separate marketing communications vendor will be required
o Develop RFP for marketing communications vendor
o Identify potential vendors and solicit proposals in response to the RFP
o Select and develop contract with marketing communications vendor
o Manage contract activities with marketing communications vendor
1.1.6. Implement Certification Program (Explicit Tasks) Define the General Test Policies for the certification program for OpenADE/ESPI
products (Section 5.1 of the IPRM D3V2)
Establish test report template, contents and example
Develop and maintain a test case reference and modification history list
Establish a Common Test Suite Specification (TSS11 – Section 5.2/3 of the IPRM D3V2) if multiple test labs are deployed to test the same standard and/or profile
If required, define common unique test procedures to support the TSS –these should be test tool agnostic
Manage the TSS in a well-defined, open and formal manner with change control
If there are multiple testing laboratories, put in place processes to avoid quality differences and assure repeatable testing between the laboratories
Specify in the test program requirements for those features that are mandatory, and those features that are optional (Section 5.4 of the IPRM D3V2)
o Require and enforce that vendors declare the optional features implemented in a product
o If more than one vendor implements the same optional feature in a product, require that future implementations of that optional feature be tested and certified for conformance and interoperability
o Define common test cases for optional features that need to be tested as part of the certification program
Establish certification programs, terms and conditions of award and re-certification
11 A Test Suite Specification (TSS) consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. Test cases would include both valid and invalid behavior tests. Each test case is further described step-by-step with test procedures and well defined pass / fail / indeterminate criteria, along with references.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 21UCA International Users Group December 12, 2011
504
505
506
507
508
509
510511
512
513
514515
516517
518
519520
521522
523524
525526527
528529
530531
2627282930
o Maintain a published list of certified products
o If a logo is part of the program, create the logo and licensing agreements
Determine which components, if any, certified by other industry programs can be “inherited” in product certifications
o Develop the procedures for validating that pre-certified components included in OpenADE/ESPI products do not impact interoperability and conformance to the OpenADE/ESPI specification
Develop common, well-defined standardized test cases in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA
o Validate the tests and implement them utilizing validated test tools. Golden reference test equipment may be utilized where appropriate. Define procedures and processes to validate the use of test tools and reference implementations.
o Ensure that test tools have a complete mandatory feature-set coverage of a standard. In cases where optional features are included in vendor products, incorporate those feature-sets in the test tool.
Maintain a current and upcoming list of applicable test cases to be called a Test Case Reference List
o Work with authorized labs to derive a Test Plan from the Test Case Reference List. Tests shall be identified using the test plan
Establish and maintain a revision control system, including revision history, revision numbering, and a defect and expansion management process for all tests in the TSS
Assure that defined product test cases cover application profiles for specific feature sets and functions defined by the specific application profile, and implement interoperability evaluation within that application profile
Define interoperability-specific testing procedures such as “plugfests” but also the selection and use of “golden” reference units. A minimum of two golden units are to be selected.
1.1.7. Improvements in the Standard and the Certification Program (Explicit Tasks)
Develop and maintain an improvement program for the overall process, the standard documentation itself, test labs, the test and certification program, etc. (Section 5.5 in the IPRM D3V2) If possible solicit direct feedback from
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 22UCA International Users Group December 12, 2011
532
533
534535
536537538
539540
541542543544
545546547
548549
550551
552553554
555556557
558559560
561
562
563564565
customers of the certified products to assess that they meet customer interoperability needs.
1.1.8. Cyber-Security (Explicit Tasks) The testing and certification program shall ensure that Cybersecurity functional
performance requirements are defined, and test cases designed and used to evaluate the requirements
The ITCA needs to work with NIST and the SGIP to ensure that the technical specification for OpenADE/ESPI standards is reviewed for cyber-security issues
The ITCA may need to establish a Digital Certificate Program if applicable
Determine and implement appropriate security stress testing including static analysis and penetration testing; assure security policy models drive testing; ensure that vendors submit threat analyses as part of certification process; document programs and standards used for security testing and incorporate component-based Cybersecurity concepts in the testing program
One interpretation of this proposed section is that an ITCA needs to establish a distinct certification program just for Cybersecurity issues and include this testing in the certification program. The IPRM is not yet clear on the qualifications for a lab to do Cybersecurity testing.
1.1.9. Governance (Explicit Tasks) Determine whether first party testing, third party testing or both are allowed
and define the circumstances and process for submission to a certification body (CB) as well as the CB responsibilities
Define a corrective process for resolving reported interoperability problems and implement preventative processes to avoid recurrence of such problems.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 23UCA International Users Group December 12, 2011
566567
568
569570571
572573
574
575576577578579
580581582583
584
585586587
588589
Next StepsThe UCA International OpenADE Task Group can move forward in different ways, but a process could be the following:
Review the above Task List and make key decisions on how to develop a formal ITCA and manage it. The initial key decisions are probably
o Whether or not to develop a formal ITCA
o Whether or not to set up formal membership and sponsorships in a legal structure for the standard
o Whether or not to raise funding to contract with a manager
o Whether or not to develop all of the artifacts with volunteer labor or raise funding to contract with appropriate vendors
o Whether or not to operate a formal third-party certification program or use a self-certification program
o Whether or not to contract with an independent test lab for certifications (depends on above)
o Whether or not to take on the marketing aspects of promoting the standard and certified vendor products
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 24UCA International Users Group December 12, 2011
590
591592
593594
595
596597
598
599600
601602
603604
605606
o What are the organizational requirements that would be needed to implement the key decisions
o How much funding would be required and how best to pursue it
Based on the key decisions above, develop a formal or informal request for proposal to solicit proposals for operating an OpenADE/ESPI ITCA as envisioned by the OpenADE Task Group.
Based on the key decisions, conduct a specific discussion with UCA International to understand the tasks that it can take on and the funding requirements to do so. Does UCA International have (or can it develop) the needed organizational structure to accomplish the goals of the OpenADE Task Group?
Determine if the UCA can meet the requirements of the OpenADE Task Group, including the schedule, funding and marketing activities (if any), RFP and contracting activities (if any), etc.
Depending on the above determination, solicit additional responses to the OpenADE/ESPI ITCA RFP.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 25UCA International Users Group December 12, 2011
607608
609
610611612
613614615616
617618619
620621
Schedule
Task Completion DateTo Be Determined
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 26UCA International Users Group December 12, 2011
622
623
SummaryIn summary, this initial requirements document has the potential to greatly accelerate the schedule and quality of both the OpenADE test and certification program and the overall OpenADE industry interoperability.
Interoperability Test and Certification Management Assistance Analysis Version – 0.01 27UCA International Users Group December 12, 2011
624
625626627