Parallel Operations External MP Kick - Off Meeting 10/8/2013 SPP Parallel Operations Team

Parallel Operations External MP Kick - Off Meeting

10/8/2013

SPP Parallel Operations Team

Agenda

2

• Purpose and Objectives

• Purpose, Processes, Outcomes, Differences from SMT

• Role of SPP and MPs during Parallel Operations

• Parallel Operations Success Metrics/Readiness

• Purpose, Processes, Outcomes of Integrated

Deployment Tests

• IDT Success metrics

PURPOSE AND OBJECTIVES

3

Market Trials – Where Are We?

4

Purpose of Kickoff

5

Provide Market Participants with:

•Understanding of objectives of Parallel Operations and Integrated Deployment Testing

•How to Support the Objectives of Parallel Operations and IDT

•What are we doing?

•Why are we doing it?

•What are the expectations of the individual Market Participants?

•How will IDTs work?

OBJECTIVESSection 2

6

Parallel Operations Objectives

7

• Process – How is this different from SMT

• Explain Integrated Deployment Testing

• Expectations

• Timelines

• Metrics

• Roles

• Expected Outcome

• Communication Plan

PARALLEL OPERATIONS PURPOSE, PROCESS, OUTCOMES, CHANGES FROM SMT

8

The Goal of Parallel Operations• Exercise all SPP systems with full use of all policies and

procedures as developed and tested during earlier phases of Market Trials.

• During the Integrated Deployment Tests (IDTs) the Real-Time systems focus on verifying the ability of SPP to maintain Area Control Error (ACE), to meet CPS1 and CPS2 standards, with all MPs and all available Resources being dispatched by the Integrated Marketplace.

9

Parallel Operations Processes• Every day of the week is an “Official Operating Day”.

• DA, RUC, RTBM, RTGEN, & Settlements run daily.

• Seven scenarios not tested in SMT will be executed in Parallel Operations:– Scenario #7 - Must Offer

– Scenario #13 - DDR with Settlements

– Scenario #24 - Repricing (Part 2) with Study MCE

– Scenario #28 - NFT with CMS

– Scenario #29 - Reserve Sharing Group

– Scenario #31 - BDR with Settlements

– Grandfathered Agreements carve-out

10

Parallel Operations Processes• All critical systems will be live and running in parallel with EIS

systems.

• The EIS Market will continue to determine the Resources’ actual Dispatch Instructions and serve as the basis for all Settlements.

• Integrated Marketplace Setpoint Instructions will be sent to Market Participants for information only purposes.

• Offer Curves should continue to be consistent with EIS costs, and all submission items should continue to reflect EIS systems in such a way that same amounts of generation to be dispatched and same active transmission constraints are reflected in both systems.

11

Parallel Operations Outcomes/Exit Criteria• Successful completion of Parallel Operations must

demonstrate the robustness of the Integrated Marketplace software, processes, and systems, both from a technical and commercial perspective including the ability of SPP to reliably maintain ACE for the SPP Balancing Authority footprint within Compliance Standards.– Successful completion of Deployment Tests.

– Successful execution of Integrated Marketplace Bid toBill functionality consistently for 7 consecutive Operating Days according to protocol timelines.

12

Parallel Operations is not Integrated Deployment Testing• The requirements to start Parallel Operations is

decoupled from IDT– IDT may start without Parallel Operations

– Parallel Operations may start without IDT

• The current timeline includes IDTs as specific tests within the Parallel Operations Market Trials testing sub-stage

13

Changes from SMT to Parallel Operations• Communications

– Hotline will be used during the first 2 weeks of Parallel Operations (11/12 – 11/27).

– Hotline process ends on 11/27; no weekend or Holiday coverage.

• Additional Communications channels (Starting 11-12)– For Real-Time Operational issues.

Contact SPP Operations Desk (24x7).

– For Questions, Inquiries, Requests, etc.: Submit an RMS with your issue, inquiry or request (24x7).

Contact your SPP Customer Relations Representative by phone or email (8 AM – 5 PM CDT).

14

Changes from SMT to Parallel Operations• Logistics

– Parallel Operations will operate from Maumelle in November.

– Resource planning will look similar to SMT.

– First 2 IDTs will be performed from Maumelle and are tentatively scheduled to run from 9:30-11:30 am on 11/19 and 11/21.

– All appropriate access, systems should be tested and support staff engaged for the IDTs.

– Staffing to support all IDTs.

– Move to ODC beginning of December.

15

Changes from SMT to Parallel Operations • Tracking Mechanisms

– RMS reports will continue throughout Parallel Operations.

– Known Issues Log continue.

– Release Scorecard will continue to be updated until everything is operational for Go-Live.

– Scenario Tracker continues.

16

Changes from SMT to Parallel Operations• Weekly Meetings

– External Meetings – Additional meetings will be held for Deployment Tests. Current weekly schedule will remain in place for November/December, possibly phasing out in January.

– Phase out the weekly calls’ overview of the previous OD. Focus instead on anomalies, defects, IDTs, & Parallel Operations scenarios only. Single thorough weekly meeting.

– One registration for all meetings, continue using SMT bridge/ WebEx.

• Participation Report

– Readiness metrics will be the same list of SMT reports

17

Changes from SMT to Parallel Operations• Daily Calendar

– Current 3 month calendar will be supplemented with a 3 week look ahead to provide a thorough, single member-facing calendar.

– Plan for 1st 3 week release October 22 (3 weeks prior to November 12).

• RMS – Quick picks will be revised.

– New CPOCs will be defined and trained by 11/1/13.

• External email distribution lists – Phased out in December.

18

Changes from SMT to Parallel Operations• Processes for Analysis of Statistics

– Analysis for Deployment Tests.

– Reviewing current metrics lists.

• Integrated Deployment Tests (IDT)– 15 separate tests planned, each will have its own

checklist/plan.

– Communication line opens 1 hour before test. Requires two conference bridge lines. During IDT, we will have

the ‘walled’ MP-side and the ‘non-walled’ Reliability side in separate rooms.

19

20

October November December January February March

Structured Market Trials Parallel Operations Go-Live

Production Support Process

Parallel OperationsSupport Process

SMT Support Process

Interim Support Process

• MP’s contact SPP through RMS or Hotline.

• Customer Relations routes RMS tickets to Workstreams.

• Defects are tracked in SILK.

• MP’s contact Real-Time Operations or for Non Real-Time issues contact Customer Relations or log issues directly into RMS by selecting the appropriate CPOC.

• If Customer Relations is contacted they will route RMS tickets to appropriate CPOCs (Departments, Desks, IT, etc).

• System defects will continue to be tracked in SILKCentral. Remedy will continue to track Change Requests and emergency fixes.

• Same as Parallel Operations Support Processes.

• Market Trials Hotline will continue to be operational.

• Hotline will address issues or reroute as appropriate while transitioning to Parallel Operations Support.

Changes from SMT to Parallel Operations IT Support Processes

Transition from SMT -> Parallel Operations• How 24x7 operations will work:

– 18 operators available to run 24x7 – performing all key IM processes – Balancing Coordinator, Reliability Unit Commitment, & Real-time Balancing Market; Day-Ahead will be staffed at Go-Live levels.

– Weekday 8 AM to 5 PM window, operators and coordinators available to assist/interact with MPs to simulate production.

– Less than a full host of operators in place to cover all functions in the “off peak” hours/weekends (specifically scheduling, Tariff and Gen Dispatch). Issue resolution for these functions may require a slower response time during

these hours.

– The main focus during that time will be to maintain the systems and continue to test/work through our processes and procedures.

– During the IDTs, every function supported by multiple operators, engineers and other operations staff.

21

Parallel Operations Timeline

22

• 11/03/13 – Structured/Unstructured testing complete.• 11/4/13 - 11/11/13 – Systems are scheduled to be down for

data cleanup and updates.• Will ensure data is available to run Settlements on DST

OD.• ETSE data purge prior to Parallel Operations after DST OD.

• 11/12/2013 – Parallel Operations is scheduled to start on

Key Dates• 11/03/13 - The end of SMT will include this 25 hour Daylight

Savings Time day.

• 11/04/13 - 11/11/13: Test data purge for Settlements.

• 11/12/13 - Parallel Operations begins – Moving to 7 Operating days per week and the Hotline will be open 24x7.

• 11/19/13 - First Integrated Deployment Test.

• 12/02/13 - Start of production processes for all systems.

23

ROLE OF SPP AND MPS DURING PARALLEL OPERATIONS

24

SPP ActivitiesDuring Parallel Operations, SPP will:

•Execute Multi-day Reliability Assessment, DA Market, DA-RUC, and ID-RUC.

•Execute and monitor Real-Time system components, including RTBM/AGC, State Estimator and all Real-Time security applications.

•Execute all EMS 2.5 Production studies, RTNET, RTRFCALC, RTCA, Losses, etc.

•Execute RTBM every 5 minutes using Market Participant data from ICCP and Portal submissions.

•Execute all rolling Forecasting applications, Wind, Parallel Loop Flow, Mid-Term, & Short-Term Load Forecasts.

•Verify Market Participant submission activities.

25

SPP Activities cont’d• Work with Market Participants to resolve issues.

• Generate Setpoints, LMPs, MCPs and Shadow Prices.

• Execute AGC every 4 seconds.

• Execute Deployment Tests to deploy generation from the new Integrated Marketplace systems.

• Communicate Real-Time reports and outputs.

• Update Counterparty credit limits with self-selected credit limits submitted by Market Participants.

• Generate Awards, Day-Ahead LMPs/MCPs, and other reports.

• Produce Meter Agent Reports.

• Communicate DA Market and RUC reports and outputs to the market.

• Produce Settlement Statements, Invoices.

26

Market Participant ActivitiesDuring Parallel Operations, Market Participants will:

•Submit, update, and cancel DA Market submission items:

– Three-Part Supply Offers (including Start up and No Load Costs)

– OR Offer

– OR Self-Arrangement

– Energy-Only

– Demand Bids

– Virtual Offers and Bids

•Verify ability to receive Resource Setpoints via ICCP & XML.

27

Market Participant Activities cont’d• Report non-conforming Loads.

• Create and Submit Meter data for each day.

• Report any issues through the RMS Download and review pre-DA Market and post-DA Market Report postings.

• Download and view Meter Agent reports from Portal.

• Download and review RUC postings, as applicable.

• Download and review Real-Time postings.

• Submit self-selected credit limit to SPP, if desired, as often as weekly.

• Create and Submit Bilateral Settlement Schedules.

• Download and review Settlement Statements and invoices.28

Market Participant Activities cont’d

• During Parallel Operations RTBM submissions need to be in closer tolerance then during SMT.– Update a unit’s min/max capacity if it changes by >10% or 5

MW, whichever is greater (for SMT it was >20% or 10 MW).

– Update a unit’s offer curve if it changes by >10% or $2 per 5 MWs, whichever is greater (for SMT it was >20%).

29

INTEGRATED DEPLOYMENT TESTS PURPOSE, PROCESS, OUTCOMES

30

Deployment Test Purpose

• These tests will be the first time the entire footprint will follow SPP Integrated Marketplace Setpoint Instructions.

• EIS BA’s AGC system should still be able to calculate ACE, CPS1, and CPS2 during the Deployment Tests, but not adjust any Setpoints to control for the EIS BA’s ACE.

• All EIS BAs have received waivers from SPP RE and MRO. – CPS1/CPS2 violations waived with the requirement of the BA’s

to maintain logs of start/stop times.

• A mock Deployment Test will be planned for the week prior to the 1st scheduled IDT – Tentatively set for 11/14/13.

31

Deployment Test Purpose

32

• Verify that SPP can transmit proper AGC signals to Resources and that the EIS BAs within the SPP footprint can receive the new information.

• Verify SPP can calculate a single ACE and respond to it appropriately.

• Verify that Resources respond to AGC signals received from SPP.• Verify that Resources can respond to a Congestion Management

Event (CME/TLR).• Verify that AGC is correctly biased in response to a Reserve

Sharing System event (RSS/CRD).• Verify that SPP and Resources can respond to a Contingency

Reserve Deployment (CRD).

Deployment Test Processes

33

During the Integrated Deployment Test:

•EIS and Integrated Marketplace systems will continue to operate in parallel.

•Resources will be switched from EIS to Integrated Marketplace and back.

•RTBM will provide Base Points and manage congestion.

•AGC will maintain ACE, provide Regulation dispatch instructions and dispatch Contingency Reserves (including Supplemental) during RSS events.

•The Market will settle based on EIS dispatch.– Uninstructed deviation penalties will be waived for a period before,

during, and after the Deployment Tests.


34

During the Integrated Deployment Test:

•Schedules will be mirrored from EIS production and convert the LCA/GCA for internal Balancing Authorities to an SPP CBA for calculating NSI for use in RTBM and AGC.

•No tags should be input by Market Participants for the period during the Integrated Deployment Test.

•Integrated Deployment Tests will be scheduled to ramp in/out at the bottom of the hour to minimize the impacts of transactions ramping at the top of the hour.


• Timing of tests will be designed so SPP can test for operational control in various conditions throughout the day:– Demonstrate AGC operation during extended heavy load pick

up during morning period.

– Demonstrate AGC operation during an evening load pick up period.

– Demonstrate crossing midnight period and operational continuation between days.

– A daily calendar has been developed with specific dates and times for the tests.

35

36

01:15Begin

Cutover Window EIS-IM

01:30 03:30

3:45 Cutover

Complete

Timing of Integrated Deployment Test (2 Hr)

01:30Cutover

Complete

03:30Begin Cutover Window IM -

EIS

Integrated Marketplace DispatchIntegrated Marketplace Dispatch

00:30Initiate MP Conference

Call

04:30Close MP

Conference Call

Timing of Integrated Deployment Test (2 Hr)• 00:30 – One hour prior to the start of the IDT, SPP will begin the

open conference call.

• 01:15 – 15 minutes prior to the start of the IDT, SPP will notify the BAs that they should prepare to be cutover from EIS to IM by 01:30 – if the BA cutover process only takes 2 minutes then they should begin that process at 01:28.

• 01:30-03:30 – 2 Hour IDT – Resources respond to IM Setpoints and SPP manages the SPP BA ACE.

• 3:30 – SPP will notify the BAs that they should begin their cutover from IM to EIS and all BAs should be following EIS setpoints by no later than 03:45.

• 4:30 - One hour following the end of the IDT, SPP will end the open conference call

37

Integrated Deployment Test

38

Real-time

Tact Tini Ttar0 5 10 15

target

Every 5 minutes, both EIS and Marketplace snap where physical Resources are in time. The updated dispatch instruction is sent via unique ICCP Object IDs.

Marketplace Setpoint

EIS Setpoint

IDT Calendar

39

November 2013Sunday Monday Tuesday Wednesday Thursday Friday Saturday

1 2 Tentative release of

Settlements Update

Model Upload Window for November 1 3 4 5 6 7 8 9 Par. Ops Go/No Go SPP Test for Market-Wide Decision Mock IDT on 11/14 Conference Call ETSE Dbase Purge SERC Site Visit Trainer Forum

REOPs Classes 10 11 12 13 14 15 16

Tentative week

Parallel Ops Begins Run SMT Scenario #28a All days color-codedfor MOS v1.8 SPP to Confirm 1st IDT Run SMT Scenario #7 Mock IDT with MPs CMS Outage NFT like this are available

to be in Production AGC/CPS Must Offer Volt. Control/CROW Market-Wide for unstructured Net Conference Net Conference Conference Call testing by MPS

17 18 19 20 21 22 2 Hour IDT 2 Hour IDT Market-Wide 9:30 AM - 11:30 AM 9:30 AM - 11:30 AM Conference Call Regional Restoration Drill 24 25 26 27 28 29 SMT Hotline Closed Thanksgiving Holiday Thanksgiving Holiday

Transition Parallel Operations from Maumelle to Chenal ODC Model Upload Window for December 1

Network and Commercial Model Upload Window SPP Training SPP Holiday MP Unstructured Test Dates

IDT Calendar

40

December 2013Sunday Monday Tuesday Wednesday Thursday Friday Saturday

1 2 3 4 5 6 7 2 Hour IDT Run SMT Scenario #24 2 Hour IDT Market-Wide 6:30 AM - 8:30 AM Re-Pricing 7:30 AM - 9:30 AM Conference Call 8 9 10 11 12 13 14 Run SMT Scenario #13 Market-Wide 2 Hour IDT DDR w/Settlements 4 Hour IDT Conference Call 1:30 PM - 3:30 PM Run SMT Scenario #31 6:30 AM - 10:30 AM BDR w/Settlements 15 4 Hour IDT 16 17 18 19 20 21

12:30 PM - 4:30 PM Tentative week 4 Hour IDT Run SMT Scenario #29 2 Hour IDT Market-Wide

for MOS v1.9 9:30 PM - 1:30 AM RSG 6:30 AM - 8:30 AM Conference Call to be in Production (Crossing Midnight)

22 23 24 25 26 27 28 Christmas Holiday Christmas Holiday Market-Wide Conference Call Model Upload Window for January 129 30 31

Model Upload Window for January 1


IDT Calendar

41

January 2014Sunday Monday Tuesday Wednesday Thursday Friday Saturday

1 2 3 4 New Year's Day Market-Wide Holiday Conference Call 5 6 7 8 9 10 8 Hour IDT 11 8 Hour IDT 2 Hour IDT 6:30 AM - 2:30 PM 3:30 PM - 11:30 PM 9:30 AM - 11:30 AM Market-Wide Conference Call 12 13 14 15 16 17 18

Tentative week Market-Wide for MOS v1.10 Conference Call

to be in Production 19 20 21 22 23 24 25 2 Hour IDT Market-Wide 5:30 PM - 7:30 PM Conference Call Upload Window26 27 28 29 30 31 4 Hour IDT 2 Hour IDT Parallel Ops Ends 5:30 PM - 9:30 PM 9:30 AM - 11:30 AM

Model Upload Window for February 1


Integrated Deployment Test Go Criteria

• If at any point in the IDT communication plan, a Resource owning MP deems it is not capable of participating and that MP’s Resources are scheduled to be online, SPP shall consider cancelling the IDT.

– This will be case-by-case.

– EIS-BA MPs must 100% be ready for each IDT.

42

Deployment Tests: Unit Commitment

43

• Unit Commitment will continue to be managed by the existing Balancing Authorities. Important all normal EIS functions occur in parallel with Integrated Marketplace during Deployment Tests.

• RTBM does not require RUC systems to operate to produce Base Points. RTBM will use Resource’s metered MW, telemetered Control Mode, CROW outage information and RTBM offer parameters for dispatch during Deployment Tests.

• When Resources are online, coming online or going offline in EIS, MPs must update RTBM offer, telemetered Control Mode, and Commitment status to “Self” in the Integrated Marketplace. Otherwise, RTBM may not be able to dispatch them accordingly.

– All online Resources must have a valid commitment status in the Current Operating Plan (COP) in order to be eligible for dispatch by RTBM.

– MPs can offer a commitment status of “Self” for all online resources.

Deployment Tests: Unit Commitment

44

• Note that the Integrated Marketplace will NOT be the basis for Unit Commitment.

• The SPP Integrated Marketplace tariff doesn’t take effect until 03/01/14 and SPP cannot be responsible for Unit Commitment until then.

• Therefore, following the DA RUC, SPP will perform a COP True-Up to essentially substitute EIS BA’s unit commitment plans for the results of the DA RUC.

Deployment Tests: COP True-up

45

• Allows the RTBM to dispatch resources based on real-time (EIS) operating conditions.

• If Resources are online in EIS and not in the COP, they are unable to be dispatched economically by RTBM.

– Output will be echoed.– Not eligible to clear reserves.

• After the DA RUC run, SPP checks the EIS Resource Plan statuses for the OD to identify Resources with conflicting status with the COP.

• SPP uses the ID-RUC to commit or de-commit resources that have a conflicting EIS status with the COP.

– If the resource has a EIS status of “Unavailable” and is committed in the COP, it will be de-committed by the ID-RUC.

• Process repeats at least 2 hours a day from 8am-4pm during supported ODs.– During unsupported days, the COP True-Up runs once a day.

Deployment Tests: Energy Offers

46

In order to minimize differences between RTBM Dispatch Instructions and EIS Market Dispatch Instructions:

•Resource Offers should be consistent with operational parameters submitted in the EIS Market Resource Plans and the Energy Offer curves submitted for use in the EIS Market.

•Regulation Ramp Rate curves and Contingency Reserve Ramp Rate curves should be set equal to the Ramp Rate Curve submitted in for use in the Up direction in the EIS Market.

Deployment Tests: Regulation Deployment

47

• Regulation will be deployed by the Integrated Marketplace during Deployment Tests.

– All Regulation Resources should have been self verified or tested by SPP during Structured Market Trials.

• All Resources scheduled to provide Regulation during the Deployment Test window in EIS should self schedule those Resources in Integrated Marketplace for Regulation for both up and down regulation.

– Regulation dispatch status “Fixed”.

– Fixed amount should be equal to capability of the Resource.

– Offer price of $0/MW.

– All other Resources eligible to provide regulation should offer in with a status of “Market” with an appropriate $/MW offer.

Deployment Test: Contingency Reserve Deployment

48

• Contingency Reserve Deployment will be deployed by the Integrated Marketplace during Deployment Tests.

• All Resources scheduled to provide contingency reserves during the Deployment Test window in EIS Ancillary Services Resource plans should self schedule those Resources in Integrated Marketplace for contingency reserves.

– Spinning dispatch status “Fixed” at specified MW amount with $0/MW offer.– On-Line dispatch status “Fixed” at specified MW amount with $0/MW offer.– Off-line dispatch status “Fixed” at specified MW amount with $0/MW offer.– All other Resources eligible to provide spinning and/or supplemental

reserves should offer in with a status of “Market” with an appropriate $/MW offer.

• Any event that would trigger a reserve sharing event will be entered into the EIS RSS as well as the Integrated Marketplace RSS.

Regulation-Qualified Resources

49

For Integrated Deployment Tests (IDTs) for Reg-Qualified

Resources, ensure:

For Integrated Deployment Tests (IDTs) for Reg-Qualified

Resources, ensure:

Dispatch Status = FixedDispatch Status = Fixed

Regulation-Qualified Resources (cont’d.)

50

Commitment Status = SelfCommitment Status = Self


51

Min. and Max. Reg Limits = Eco Min. and Eco Max. in EIS Offer

Min. and Max. Reg Limits = Eco Min. and Eco Max. in EIS Offer

Why? Mitigate lost

opportunity costs

Why? Mitigate lost

opportunity costs


Reg. Up or Reg. Down Example

*Can clear more than fixed amount if ramp is available.52

Resource A—EIS Market

Min. Econ. Limit 150MW

Max. Econ. Limit 600MW

Dispatch Status Available

EIS Carried Reg. 10MW

Resource A—Integrated Marketplace



Min. Reg. Limit 150MW

Max. Reg. Limit 600MW

Fixed MW Amount 10MW

Offer (Reg. Up or Down) $0*

Commit Status Self

Dispatch Status (Reg.) Fixed

ICCP Control Mode 2

Contingency Reserves

Spinning Reserve Example













Offer $0*

Commit Status Self

Dispatch Status Fixed

ICCP Control Mode 1

Contingency Reserves (cont’d.)

On-line Supplemental Reserve Example













Offer $0*

Commit Status Self


ICCP Control Mode 1

Contingency Reserves (cont’d.)

Off-line Supplemental Reserve Example













Offer $0*

Commit Status Market


ICCP Control Mode 0

Telemetered Control Mode

56

It is critical the Resource submit correct Telemetered Control Mode.–All Resources on outage for off-line should submit a Telemetered Control Mode equal of “0”.

–All Resources capable of providing contingency reserves, online and are dispatchable should provide a telemetered Control Mode of “1” if they are not qualified to provide regulation.

–All Resources capable of providing regulation, online and are dispatchable should provide a telemetered Control Mode of “2”.

–All Resources that are coming on-line, going off-line or otherwise not able to follow dispatch instructions should provide a telemetered Control Mode of “3”.

What is a Resource Control Mode?

•Resource Control Mode is a signal sent to SPP via ICCP for each Resource Asset.

•Resource Control Mode is also referred to as ICCP System Control Mode (CMOD).

57

What are Valid Resource Control Status Values?

58

Importance of Correct Resource Control Status

59

Calculating Effective Control Status

Special Cases: Effective Control Status

60

NDVERs initial Effective Control Status

Pumped Storage Resources (Resource output (-))

Resources excluded from Main Island

Resources with Good Quality MW>0 & ICCP Control Status of 0

33

33

33

33

Deployment Test Communications

61

• Market Communication the Friday before the test is scheduled as a reminder to the upcoming test.

• SPP will host a one-hour conference call the day prior to each scheduled Integrated Deployment Test to review the sequence of events during the testing period and to address any Market Participant concerns.

• A conference call will established one hour prior to the Integrated Deployment Test and continue until all ramp out activities are complete.

Deployment Test Success Criteria

• No CPS 1, 2, or DCS violation (calculated at SPP BA level and is a ‘what if’ calculation since there are waivers).

• Minimum Eastern Interconnection hourly inadvertent; develop a number +/- based on that particular month (for example, EIS accumulate 100 MW a week in November for a consistent metric applied to the specified IDT timeframe).

• Reasonable congestion pricing, dispatch, & response.

• Reasonable ramping pricing.

• Zero IROL & SOL violations.

62

Deployment Test Success Criteria

• All Resources respond correctly to Integrated Marketplace Setpoint instructions.

• STLF accuracy carried over from SMT/Parallel Operations.

• Availability metrics for RTBM, CHMFC, NHMFC, & Pre-RTBM as applicable for the IDT duration.

• EMS metrics carried over from SMT/Parallel Operations (MW mismatch to be at production-levels; i.e. 25 MW for Market, 45 MW for external).

• All anomalies explained on a weekly basis (similar to SMT & Parallel Operations requirement).

63

Parallel Operations / Deployment TestsKey information for Market Participants to review on expectations for Parallel Operations and Deployment Tests has been posted on SPP.org.

The Parallel Operations Market Participant Guide is posted on SPP.org at:•http://www.spp.org/publications/SPP_IM_Market_Trials_Parallel_Operations_Test_Market_Participant_Guide_v2%201%20Clean.pdf

The Integrated Deployment Test Market Participant Guide is posted on SPP.org at:•http://www.spp.org/publications/SPP_IM_Market_Trials_Integrated_Deployment_Test_Market_Participant_Guide_v1%200.pdf

64

Deployment Test MP Cutover PlansSPP developed a list of key activities that should be addressed by the EIS BAs for the cutover to Deployment Tests on SPP.org at: •http://www.spp.org/publications/SPP_IM_BA_Cutover_Activities.pdf

– The initial drafts of the EIS BA cutover plans have been submitted to SPP and approved on September 16th

A three month Parallel Operations Calendar is posted on SPP.org at:•http://www.spp.org/section.asp?group=2702&pageID=27

A Communications plan for Parallel Operations is posted on SPP.org at:•http://www.spp.org/section.asp?group=2702&pageID=27

65

PARALLEL OPERATIONS READINESS METRICS & GO/NO-GO CRITERIA

66

PARALLEL OPERATIONS READINESS METRICS

67

Readiness Metrics

• Developed January 2012, refined throughout the project– Internal Approval: Functional and Readiness Business Owners

– External Review: Regular reviews with the Readiness Liaisons and CWG

• Measure, Monitor, and Report Overall Program Readiness– Measure SPP & MP People, Process, and Technology Readiness

• Filed with FERC

68

Parallel Operations Readiness Metrics

69

Metric Reporting Start Target Completion Current Status

TRL-01: Market Trials ParticipationCurrently Tracking

End of Market Trials Green

TRL-02: Structured/Unstructured Test Complete

Currently Tracking 11/1/2013 Yellow

TRL-03: Market Trials-Deployment Testing 11/1/2013 TBD

Reporting on 11/1/2013

• The following Readiness Metrics will be tracked during Parallel Operations – Metrics inventory and full description document available in the

Readiness workstream folder on SPP.org

– Metrics status is posted in iDashboard, under the Readiness dropdown

Market Trials Metrics

Parallel Operations Readiness Metrics -Settlements

70

Metric Reporting Start Target Completion Current Status

STL-01: Access & Download Statements

Currently Tracking Market Trials – SMT Red

STL-02: Access & Download Invoices

Currently Tracking Market Trials –SMT Red

STL-03: Submit Meter DataCurrently Tracking Market Trials –SMT Green

STL-05: Submit DisputeCurrently Tracking Market Trials –SMT Green

STL-06: Settlement Statement Timeliness

Currently Tracking Market Trials - Parallel Ops Green

STL-07: SPP Validation of Settlement Charge Types

Currently Tracking TBD Gray

Parallel Operations Readiness Metrics - Operations

71

Metric Target Completion Current Status

OPS-02: State Estimator Availability Market Trials – Parallel Ops YellowOPS-03: State Estimator Accuracy Market Trials – Parallel Ops GreenOPS-04: Short-Term Load Forecast Accuracy Market Trials – Parallel Ops Green

OPS-05a: Mid-Term Load Forecast Accuracy - 4AM ID RUC Market Trials – Parallel Ops Green

OPS-05b: Mid-Term Load Forecast Accuracy - 12PM ID RUC Market Trials – Parallel Ops GreenOPS-05c: Mid-Term Load Forecast Accuracy - 5PM DA RUC Market Trials – Parallel Ops YellowOPS-06a: Wind Forecast Accuracy - Pre RTBM Market Trials – Parallel Ops YellowOPS-06b: Wind Forecast Accuracy - RTBM Market Trials – Parallel Ops YellowOPS-06c: Wind Forecast Accuracy - DA RUC Market Trials – Parallel Ops Yellow

OPS-07: Market Flow Calculations Availability – Current Hour Market Trials – Parallel Ops Green

OPS-08: Market Flow Calculations Availability – Next Hour Market Trials – Parallel Ops Green

Parallel Operations Readiness Metrics - Operations

72

Metric Target Completion Current Status

OPS-09: DA Market Results Posted Market Trials – Parallel Ops GreenOPS-10: Generate DA LMPs and Settlement Point Prices Market Trials – Parallel Ops Green

OPS-11: RTBM Calculation Availability Market Trials – Parallel Ops Green

OPS-12: Operating Reserves Requirements Market Trials – Parallel Ops Green

OPS-13: Regulation Pre-Certification 10/31/2013 Green

OPS-14: ACE Performance Market Trials - Parallel Ops 10/1/2013

OPS-15: CROW Outage Submission Go-live TBD

OPS-16: MP Response to Deployment Testing Market Trials - Parallel Ops TBDOPS-17: ICCP Model Validation 10/25/2013 Yellow

OPS-18: ICCP Control Status Tracking Market Trials – Parallel Ops TBD

PARALLEL OPERATIONS GO/NO-GO CRITERIA

73

Parallel Operations Go/No-Go Criteria• Developed by Readiness & Metrics workstream

– Internal Approval: Applicable SMEs and program leadership

– External Review: Readiness Liaisons and the CWG

• Provide quantitative information and analysis to be used as input into the Go/No-Go decision process

– SPP Recommendation

– CWG Recommendation

– Go Live Team Decision

• A Go-Live Team formed with a purpose to review and analyze, with input from SPP and CWG to determine ‘Go/No-Go’ of major program phases (i.e. TCR, Parallel Operations or Market Go-Live)

• Details are in the Go/No-Go Folder on the Readiness Center

74

Parallel Operations Decision Timeline

75

Parallel Operations Go/No-Go Criteria

76

Identifier Criteria

PO-01 All known issues/risks that could impact Parallel Operations are published and assessed

PO-02 Target mass participates in Structured/Unstructured Market Trials

PO-03100% of Structured/Unstructured testing scenarios are successfully executed and completed or removed from scope and successful completion is validated through Readiness Metric TRL-02.

PO-04 SPP and member ICCP Models are up to date and the data exchange and values have been validated for all applicable entities and relationships

PO-05 All MP EMS related changes, including AGC (if applicable), have been completed

PO-06 Successfully execute Integrated Marketplace Bid to Bill functionality consistently for 3 consecutive Operating Days according to protocol timelines

PO-07 Execute Integrated Marketplace Bid to Bill functionality consistently for 15 supported Unstructured Operating Days according to protocol timelines.

PO-08SAT (Site Acceptance Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained (with the exception of Markets)

PO-09 SIT (System Integration Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained


77

Identifier Criteria

PO-10FIT (Functional Integration Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained (with the exception of Settlements)

PO-11All ICCP and XML inbound and outbound Marketplace data points have been validated with all applicable Market Participants as part of Structured Market Trials

PO-12 Impacts of Critical/High defects are deemed acceptable by SPP and Market Participants

PO-13 MCE Certification sign-off obtained from Markets Business Owners

PO-14All charge types for target mass have been successfully published and verified through Structured Market Trials and Business Owner approval is obtained as demonstrated through the Settlements tracker

PO-15 Performance Testing Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained

PO-16Non Functional Testing should be completed and reviewed by SPP Technical Architects validating the systems demonstrate sufficient stability to commence Parallel Operations

PO-17 Systems components that are in scope for the start of Parallel Operations have been installed and integrated in Marketplace Production


78

Identifier Criteria

PO-18 All systems supporting Parallel Operations have the latest Commercial/ Network Model

PO-19 Protocol Compliance has been validated

PO-20 CPS1 and CPS2 waivers complete (approved by NERC) or an alternative plan is implemented (approved by CBASC)

PO-21 The SPP Parallel Operations Go-Live Plan is approved internally

PO-22 The SPP Parallel Operations Roll-In/Roll-Out Integrated Deployment Plan is approved internally including IDT MP Guide and Communication Plan

PO-23 EIS BA procedures/cutover plans have been submitted and validated by SPP

PO-24 Physical setup of the SPP facilities supporting Parallel Operations is complete

PO-25 SPP Staff is ready to support Parallel Operations

APPENDIX

79

Acronyms

80

ACE Area Control ErrorAGC Automatic Generation ControlBA Balancing AuthorityBC Balancing CoordinatorBDR Block Demand ResourceBI Business IntelligenceCBA Consolidated Balancing AuthorityCDT Central Daylight TimeCHMFC Current Hour Market Flow CalculatorCME Congestion Management EventCMS Credit Management SystemCOP Current Operating PlanCPOC Central Point of ContactCPS Control Performance StandardCRD Contingency Reserve DeploymentCROW Control Room Operations WindowDA Day AheadDAMKT Day-Ahead MarketDCS Disturbance Control StandardDDR Dispatchable Demand ResourceDST Daylight Savings TimeEADS Enterprise Architecture Data ServicesEIS Energy Imbalance ServiceEMS Energy Management SystemETSE eTerra SettlementsGCA Generation Control AreaGFA Grandfathered AgreementICCP Inter-Control Center Communications Protocol

IDT Integrated Deployment TestIROL Inter-Regional Operating LimitLCA Local Control AreaMCE Market Clearing EngineMMU Market Monitoring UnitMP Market ParticipantMRO Midwest Reliability OrganizationMTLF Mid-Term Load ForecastNDVER Non-Dispatchable Variable Energy ResourcesNFT Non-Functional TestingNHMFC Next Hour Market Flow CalculatorNSI Net System InterchangeODC Operations Data CenterOR Operating ReservesPOPS Post Operating Pre-SettlementsRMS Request Management SystemRSS Reserve Sharing SystemRTBM Real-Time Balancing MarketRTGEN Real-Time Generation ApplicationRTNET Real-Time Network ApplicationRUC Reliability Unit CommitmentSCADA Supervisory Control and Data AcquisitionSERC SERC Reliability CoordinatorSLA Service Level AgreementsSMT Structured Market TrialsSOL System Operating LimitSTLF Short-Term Load ForecastTLR Transmission Loading Relief

Feedback

The information in this presentation is based on our hypothesis that we have provided you with the information and planning you need to continue with SMT, and move forward with Parallel Operations, and IDT.

– What’s your reaction to what we’re doing in terms of SMT, Parallel Operations, IDT?

– Is there something you think we should be doing in terms of SMT, Parallel Operations, IDT that we’re not?

– Can you see any reason why our current plans and metrics for SMT, Parallel Operations, IDT won’t work?

81

Documents

Parallel Operations External MP Kick - Off Meeting 10/8/2013 SPP Parallel Operations Team