CMS Test Case Document - OpenSGosgug.ucaiug.org/sgsystems/OpenADE/Shared Docume… · Web viewDocument Overview This document provides a set of guidelines for developing the steps

SGIP TCC End-to-End Test Template

Energy Services Provider Interface Testing

Version: Draft v0.1

Release Date: 2011/08/11

Page 1 of 19

1

2

3

4


Document History

The following individuals and their companies are members of the OpenADE task force and have contributed and/or provided support to the work of the End-to-End Test Working Group End-to-End Test Case template document:

Page 2 of 19

1

2345

6


Document History

Revision History

Revision Number

Revision Date RevisionBy

Comments

.1 20111018 Marty Burns, Phil Beecher Initial seed draft

Page 3 of 19

1

2

3

4


Contents1 Introduction............................................................................................................................ 6

1.1 Objective......................................................................................................................................................6

1.2 Scope............................................................................................................................................................7

1.3 Abbreviations and Definitions......................................................................................................................7

1.4 References....................................................................................................................................................8

2 Document Overview................................................................................................................9

3 Test Use Case........................................................................................................................103.1 Summary....................................................................................................................................................10

3.2 Business Rules............................................................................................................................................10

3.3 Assumptions...............................................................................................................................................10

4 Developing Test Requirements.............................................................................................114.1 Test Architecture........................................................................................................................................11

4.1.1 Test Component View........................................................................................................................11

4.1.2 Test Information View........................................................................................................................11

4.1.3 Test Security View..............................................................................................................................11

4.2 Interoperability Functional Statements (IFS) Proforma..............................................................................11

4.3 Test Groups................................................................................................................................................12

4.4 Test Purpose...............................................................................................................................................12

4.5 Additional Requirements............................................................................................................................12

4.6 Testing Context and Methods....................................................................................................................12

4.6.1 Testing Context and Methods............................................................................................................12

4.6.2 Test Steps...........................................................................................................................................12

4.6.3 Test Sequence Diagram......................................................................................................................13

5 Testing Process......................................................................................................................145.1 Overview....................................................................................................................................................14

5.2 Policies and Principles................................................................................................................................14

5.3 Test Assessment.........................................................................................................................................14

5.3.1 Application Submission......................................................................................................................14

5.3.2 Test Environment Submission............................................................................................................14

5.4 Test Preparation.........................................................................................................................................14

5.4.1 Test Plan.............................................................................................................................................14

5.4.2 Test Design.........................................................................................................................................14

5.4.3 Test Configuration..............................................................................................................................14

Page 4 of 19

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34


5.5 Testing........................................................................................................................................................14

5.6 Test Results................................................................................................................................................14

6 Feedback...............................................................................................................................156.1 ITCA............................................................................................................................................................15

6.2 User Group.................................................................................................................................................15

Page 5 of 19

1

2

3

4

5

6


1 IntroductionThis document provides a test plan for the NAESB Energy Services Provider Interface standard. Roles and responsibilities are included in this document.

1.1 ObjectiveThe purpose of this document is to provide a set of steps required to support how energy vendors and providers can test Smart Grid End-to-End business functions in order to achieve interoperability. The focus of this framework is testing the integration among systems and/or applications to enable Smart Grid business processes at the inter-systems level within the NIST Conceptual Model domains and sub-domains. The test requirements will be driven by business processes and the technical requirements will be driven by SGIP Smart Grid architectural standards. The end-state of this effort will contribute to the development of testing efforts required to support interoperable Smart Grid solutions as business functions, standards and technologies evolve.

The TCC End-to-End Testing Work Group has created a set of categories below for comprehensive coverage of End-to-End Testing. See figure 1.1

To help establish End-to-End interoperability, a current set of test use cases have been placed within the identified categories. Below is a current list of potential test use-cases that have a different focus, but provide a good context for establishing End-to-End Interoperability. This list below is not a complete set and will change based on additional use-cases.

Installation / Configurationo End Device installation (Meter, HAN, etc.) o End Device configuration and registration

Control / Commando Connect and disconnect (smart meter) o Demand Response Direct Load Control

Eventso Notification, text messageso Pricing signals

Page 6 of 19

Installation / Configuratio

n

Control

Events

Energy

Usage

Measureme

nt / MonitoringEnd -

to - End

Busi

ness

Fu

nctio

ns

Fig.1.1

1

23

4

56789

101112

1314

15

16

17181920

212223242526272829


o Events and alarms (i.e Outage Events) Measurement / Monitoring

o Batch meter reading (one way) o On-demand meter reading (two way)

Energy Usage

1.2 ScopeThe scope of this test case document is on <…>.

In scope: Out of scope:

1.3 Abbreviations and DefinitionsThis section provides a list of acronyms and abbreviations required to properly interpret the document.

Acronym NameAMI Advanced Metering InfrastructureCIM IEC TC57 Common Information ModelESI Energy Service InterfaceHAN Home Area NetworkPCT Programmable Communicating ThermostatWS-I Web Service Interoperability OrganizationIUT Implementation Under TestQI Qualified ImplementationMoC Means of CommunicationSUT System Under TestESPI Energy Services Provider InterfaceNAESB North American Energy Standards Board

Name DefinitionActor A generic name for devices, systems, or programs that make decisions and

exchange information necessary for performing applications: smart meters, solar generators, and control systems represent examples of devices and systems.

AMI Advanced Metering InfrastructureApplications Tasks performed by one or more actors within a domain.CIM IEC TC57 Common Information ModelCommon Web Portal

Web interface for Regional Transmission Operator, customers, retail electric providers and transmission distribution service provider to function as a clearing house for energy information. Commonly used in deregulated markets.

Energy Service Provides security and, often, coordination functions that enable secure

Page 7 of 19

123456

7

89

1011

12

13

1415


Interface interactions between relevant Home Area Network Devices and the Utility. Permits applications such as remote load control, monitoring and control of distributed generation, in-home display of customer usage, reading of non-energy meters, and integration with building management systems. Also provides auditing/logging functions that record transactions to and from Home Area Networking Devices.

Home Area Network

A network of energy management devices, digital consumer electronics, signal-controlled or enabled appliances, and applications within a home environment that is on the home side of the electric meter.

Programmable Communicating Thermostat

A device within the premise that has communication capabilities and controls heating, ventilation and cooling systems.

End-to-End Abstract Test Case

A collection of abstract test cases designed to prove the ability of two (or more) systems and standards to interoperate

System Under Test

A collection of either one or more QA and an AUT

WS-I Web Service Interoperability Organization

1.4 ReferencesReferences are listed below:

OpenSG - SGConformity ETSI EG 202 237 Generic approach to interoperability testing

Page 8 of 19

12

3

4567

89


2 Document OverviewThis document provides a set of guidelines for developing the steps associated with End-to-End interoperability testing. The intent of this template is to apply a framework to a set of common Smart Grid use cases and technologies that support the Smart Grid business functions from End-to-End. This document provides a set of testing guidelines to help ensure that the implemented Smart Grid systems of systems deliver the intended business functions and benefits.

Each main category to support End-to-End testing contains a set of sub steps and provides an explanation for how it can be used. Below are the four main categories used in this document.

Test Use Case: This section provides a Smart Grid common set of business requirements and processes for customers to adopt and for vendors to help implement

Test Requirements: This section is primarily aimed at developing the abstract test suites for testing devices and systems that span across multiple business services of the NIST Smart Grid Conceptual Model.

Testing Process: This section provides the principles, arrangement, planning, and execution of End-to-End testing

Feed Back: This section provides a common process and format to communicate back to the SDO’s ITCA and user groups the observed events for End-to-End testing.

Page 9 of 19

E2E TestingManagement

1.Test Use

Case

2.Test

Requirments

3.Testing Process

4.Feed Back

1

2345

67

8

91011

1213

14

1516

17


3 Test Use CaseA test use case is summarized in this section.

3.1 Summary < This section provides a narrative that provides the end-to-end business process without specifying a specific set of standards and technologies. The starting point in the development of a test case narrative should be a set ratified requirements and specifications that support Smart Grid business functions usually in a form of a use case that has been identified and accepted by the industry.>

3.2 Business Rules < This section provides.>

3.3 Assumptions< This section provides.>

Page 10 of 19

1

2

3

4567

8

9

10

11

12

13

14

15


4 Developing Test RequirementsThis section provides the steps involved for developing End-to-End abstract test cases to support this particular test use case and help ensure End-to-End Interoperability.

4.1 Test Architecture <This section specifies three main views to understand the relevant functions, information and considerations for the End-to-End solution that will be tested.>

The following bullets summarize the testing architecture to be assembled for verifying conformance to ESPI.

break the standard down into its basic services, data types, and Use Case scenarios (comes right out of the standard).

For each service, there is a set of test outcomes from success to various failures. I intend to enumerate potential failures and suggest a way to inject them.

Similar there are data structures that may be sound or not sound. Then the Use Cases provide a defined sequence of using the services exchanging the data structures. We would script the Use Cases using Selenium and the interface concept I designed with John Teeter.

We would design a test coverage set of exercising the Use Cases and permutations of the success and fault injections.

4.1.1 Test Component View< This section provides an abstract architecture view of logical business components, relevant interfaces, standards and supporting technologies that support the end-to-end processes documented in the test use case. >

The following table shows the services that are used to implement ESPI:

Actor Description Logical PhysicalData Custodian Ability to get service status ReadServiceStatus ServiceStatus

Data Custodian

Initiate signed request_token request per RFC 5849 CreateRequestToken request_token

Data Custodian Initiate signed authorize request per RFC 5849 Authorize authorize

Data Custodian

Initiate signed access_token request per RFC 5849 CreateAccessToken access_token

Data Custodian Update existing authorization NotifyUpdateAuthoriz

ation_ Authorization

Data Custodian

Revoke existing authorization (Retail Customer)

NotifyUpdateAuthorization_ Authorization

Data Custodian Terminate existing authorization via service UpdateAuthorization Authorization

Data Custodian Request subscription to authorized resource CreateSubscription Subscription (from config)

Data Custodian Request authorized data resource(s) RequestData (request_token scope)

Data Custodian Receive requested and subscribed EUI ReadData (dataCustodianDefaultBatchRe

source)

Page 11 of 19

1

23

4

56

7

89

1011121314151617

18

192021

22


Data Custodian Request and receive authorized EUI ReadData_ (request_token scope)

Third Party Initiate callback specified in request_token per RFC 5849 ProvideAuthorization (request_token callback)

Third Party Revoke existing authorization (Data Custodian)

NotifyUpdateAuthorization_ Authorization

Third Party Send requested and subscribed EUI UpdateData_ (thirdPartyDefaultBatchResource)

Third Party Notify requested and subscribed EUI is available NotifyData_ (thirdPartyDefaultNotifyResou

rce)

4.1.2 Test Information View< This section provides an information view of key information objects and interfaces that support the end-to-end processes documented in the test use case. >

ESPI Model Element Type

(OAuth) access_token

(OAuth 2.0) expires_in

UsagePoint entry.id URN

UsagePoint entry.title String

UsagePoint.status UInt8

ServiceCategory.kind ServiceKind

ServiceKind 0

ServiceKind 1

ServiceKind 2

MeterReading entry.id URN

MeterReading entry.title String

ReadingType entry.id URN

ReadingType entry.title String

ReadingType.defaultQuality QualityOfReading

ReadingType.flowDirection FlowDirectionType

ReadingType.intervalLength UInt32

ReadingType.kind KindType

ReadingType.powerOfTenMultiplier PowerOfTenMultiplierType

ReadingType.uom UomType

ReadingType.accumulationBehaviour AccumulationBehaviourType

ReadingType.dataQualifier DataQualifierType

ReadingType.tou TOUType

ReadingType.currency CurrencyCode

ReadingType.commodity CommodityType

ReadingType.consumptionTier ConsumptionTierType

Page 12 of 19

1

2

34

5


ReadingType.phase PhaseCode

IntervalBlock entry.id URN

IntervalBlock entry.title String

IntervalBlock.interval DateTimeInterval

Reading.cost UInt48

Reading.timePeriod DateTimeInterval

Reading.value UInt48

ReadingQuality.quality QualityOfReading

DateTimeInterval.start TimeType

DateTimeInterval.duration UInt32

4.1.2.1.1 Master Data Considerations< This section provides a list of relevant technical master data objects that are key linkages to support the end-to-end processes documented in the test use case. >

4.1.3 Test Security View< This section provides a security view of logical business components, relevant interfaces, standards and supporting technologies that support the end-to-end processes documented in the test use case. >

4.1.3.1 Security Considerations< Need some help…… >

4.2 Interoperability Functional Statements (IFS) Proforma

< This section provides a description of which features / functions of the SUT that are implemented. The starting point in the development of an IFS should be the Protocol Implementation Conformance Statement(PICS), which should identify the options and conditions which apply to the protocol to be tested. The Following are some characteristics of an IFS:

Format is typically a questionnaire Each function / feature of the specification(s) should be clearly described with reference to the

specific clause(s) in the specification(s) All features / functions should be clearly described as mandatory or optional, with dependencies

described

Determine mandatory and optional requirements needed for End-to-End Interoperability testing

We need template?????>

Page 13 of 19

1

2

3

45

6

78

9

10

11

12

13

14151617

1819202122

23

24


4.2.1 ConformityConformant implementations must support the following test groups.

Protocol Fundamental Batch and/or Online UsagePoint and one or more information classes and associated classes All attributes of implemented classes

4.2.2 Conformity statements from the NAESB Standardo Note that not all attributes are required to be sent, but are required to be accepted

Conformant Data Custodian implementations include the following: Subscriptions (Optional)

o Accept POST to Subscription, Batch resourceo Allow subscriptions to authorized resources

Delivery (Optional)o Accept GET to Batch resource, specific to each Authorized Third Party.o Support POST to Authorized Third Party Notification resourceo Support POST to Authorized Third Party Batch resourceo Support GET of resources directly

Conformant Third Party implementations include the following: Subscriptions (Optional)

o Submit POST requests to Subscription, Batch resourceo Sign requests with access tokens

Delivery (Optional)o Submit GET request to Batch resourceo Accept POST to Authorized Third Party Notification resourceo Accept POST to Authorized Third Party Batch resourceo GET resources directly

All conformant implementations include the following: Security

o Server certificates and mutually authenticated HTTPSo Send and/or accept requests to OAuth endpoints

Contento The information elements or subset thereof to be transferred are to be

negotiated between parties.o Information elements with the meaning defined herein to be transferred

use the format and structure defined herein.o Additional information elements not defined herein are placed in

extension elements as defined by the ESPI schema herein, use a namespace different from the ESPI schema herein, and are optional.

o It is recommended that any additional information elements included in an implementation be submitted for consideration in future versions of ESPI.

Page 14 of 19

1

2

34567

8

9

10111213141516171819202122232425262728293031323334353637383940414243


4.3 Test Groups<This section provides a grouping of logical functions from specifications and IFS that can be broken into a set of test groups >

4.3.1 Protocol HTTP HTTPS

4.3.2 Fundamental ServiceStatus Authorization

o Requesto Authorizeo Callbacko Access

4.3.3 Application ApplicationInformation

4.3.4 Batch Subscription Notification Deliver (Push) Receive (Pull)

4.3.5 Online Request/Reply

4.3.6 Information UsagePoint

o MeterReadingo ElectricPowerQualitySummaryo ElectricPowerUsageSummary

Page 15 of 19

1

23

4

56

7

89

10111213

14

15

16

17181920

21

22

23

24252627

28

29


4.4 Test Purpose< This section provides.>

4.5 Additional Requirements< This section provides an add-in where additional requirements are added to support a specific set of needs by an organization. An example might include an organization that has a set of specific business functions that where additional testing is needed due a regulatory requirement. The specified Additional Requirements should not impact the end-to-end test process and can be inserted with minimal impact on the specified test requirements and test process.>

4.6 Testing Context and Methods

4.6.1 Testing Context and Methods

Test Context and Methods Overview

Interaction Diagram x.291 Standard

4.6.2 Test Steps< This section provides.>

Step # Ref. Message # Logical Components Detailed Step Completed Pass/Fail Notes

4.6.3 Test Sequence Diagram< This section provides.>

Page 16 of 19

1

2

3

45678

9

10

11

12

13

14

15

16

17

18

19

20

21


5 Testing Process

5.1 Overview

5.2 Policies and Principles

5.3 Test Assessment< This section provides.>

5.3.1 Application Submission< This section provides.>

5.3.2 Test Environment Submission< This section provides.>

5.4 Test Preparation< This section provides.>

5.4.1 Test Plan< This section provides.>

5.4.2 Test Design< This section provides.>

5.4.3 Test Configuration< This section provides.>

5.5 Testing< This section provides.>

5.6 Test Results< This section provides.>

Page 17 of 19

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22


Page 18 of 19

1


6 Feedback

6.1 ITCA< This section provides.>

6.2 User Group< This section provides.>

Page 19 of 19

1

2

3

4

5

6

7

Documents

CMS Test Case Document - OpenSGosgug.ucaiug.org/sgsystems/OpenADE/Shared Docume… · Web viewDocument Overview This document provides a set of guidelines for developing the steps