Meeting Minutes Personnel Certification Governance Committee August 12-13, 2014 Grand Hyatt Seattle 721 Pine Street Seattle, WA 98101

Administrative

A meeting of the Personnel Certification Governance Committee (PCGC) was held on August 12-13, 2014 in Seattle, WA. The agenda and action item list are affixed as Attachments A and B, respectively. 1. Welcome and Quorum

PCGC Chair, Brett Hallborg welcomed the committee members and NERC staff. A quorum was determined at 8:05 a. m. The list of meeting attendees can be found at the end of the meeting minutes.

NERC Antitrust Compliance Guidelines The NERC Antitrust Compliance Guidelines were reviewed and all attendees agreed to abide by the NERC Antitrust Guidelines.

Public Meeting Notice Trion King reminded all participants the meeting is public and was posted on the NERC website. Safety Briefing Trion King provided a safety briefing for the hotel. Task List Review The PCGC reviewed and updated the Task List, Attachment B.

2. Consent Agenda/Approve Meeting Minutes

The PCGC reviewed, amended and approved the agenda for this meeting. A motion was made and second to approve the May 2014 meeting minutes. The motion carried unanimously.

3. System Operator Certification and Continuing Education (SOCCED) Program Updates System Operator Certification: A total of 298 exams have been taken during the 1st and 2nd quarter of 2014. The YTD overall pass rates remain higher than expected pass rates for all credentials except Balancing and Interchange (BI). There is a concern that some exam candidates have found a way to bypass the 42 day requirement between exams. NERC will compile additional data on this issue and provide an update during the November 2014 meeting. Also, the new exams, which will be published in 2015, should address the

Meeting Minutes – Personnel Certification Governance Committee Meeting – August 12-13, 2014 2

increasing pass rates. One of the primary reasons for changing test vendors is the new vendor provides Linear On the Fly Testing (LOFT) which allows a variety of forms for each credential. The Reliability Coordinator credential remains the primary exam taken by most candidates (69.6%). Credential renewals for 2nd quarter continue to be higher than projected. There were 571 credential renewals versus the projected 412 renewals per quarter. Active Credentials:

Failures per Attempt:

*YTD 2014 ^The total number of exams taken is 298. SOCCED does not account for those passing on > 2nd attempts. SOCCED only captures 1st attempt passes. It does not count passes after the 1st attempt; however, the system counts all failures.

Meeting Minutes – Personnel Certification Governance Committee Meeting – August 12-13, 2014 3

Auto Generated Emails from SOCCED: The auto generated emails from SOCCED notifying operators of certification expiration dates are confusing and not clear. Operators are confused by the language in the emails. NERC will work with the vendor to clean up the message in the emails. Continuing Education: During the 2nd quarter (2014), 594 Continuing Education (CE) courses were approved and 3,144 CE hours (CEHs) were entered onto system operator transcripts. A total of 62 audits were conducted during the 2nd quarter. There are 299 active providers registered in the CE Program. As more courses are entering their annual review cycle, several courses submitted for CEHs are out of scope for certified system operators and their roles to maintain the reliability of the bulk power system (BPS). Refunds: There has been a significant number of refund requests. Instead of selecting “Request for Renewal”, operators are selecting “Request for New Exam”. The renewal process navigation is not intuitive. SOCCED will have to be modified to change the renewal process. In the meantime, a notice has been added to SOCCED’s login page instructing operators on how to renew their certificate. Duplicate Accounts: NERC is investigating system operator duplicate accounts. Legal Name Changes: The legal name change process has been added to the System Operator Certification (SOC) Program manual. System operators will have the option of printing a new certificate from SOCCED at no cost or pay a processing fee for a new printed certificate which will be mailed out by NERC.

4. Closed Sessions 10:00 – 11:20 AM, Tuesday, August 12, 2014 and 9:00 AM – 10:20 AM, Wednesday, August 13, 2014 PCGC entered into closed session to review special requests. NERC will send correspondence of the decisions of the PCGC to the appropriate parties.

5. PSI Update PSI is working to secure a contract with a third party for Manitoba, Canada. A candidate has submitted feedback concerning a test center. The information will be sent to PSI for investigation and review.

6. SOCCED Update Melinda Nicius, SOCCED Database Administrator, provided the latest update on SOCCED Phase 2A upgrade. MCG has submitted a Statement of Work (SOW). Melinda is leading the review of the SOW and working with MCG on clarifying any questions. The project is on a very tight schedule. The requirements are being evaluated on needed features vs. nice to have features. System challenges continue with SOCCED. NERC is preparing a one year contract extension for 2015.

7. Task Force Working Sessions

Meeting Minutes – Personnel Certification Governance Committee Meeting – August 12-13, 2014 4

Exam Development and Delivery (EDD) Task Force: EDD evaluated the current exam development timeline. The timeline is expected to change significantly with the new exam development vendor, PSI. PSI offers dynamic item change out which provides item flexibility and increases exam integrity. Policy Oversight Task Force: The Policy Oversight TF reviewed all PCGC documents and System Operator Certification Program documents currently on NERC’s website for accuracy. An action item to update the NERC website has been added to the Task List.

8. Task Force Reports a. Nomination Task Force: Two members have submitted their resignation from the PCGC. Their

roles at work require support of other NERC committees/Task Forces. The Nomination TF would like to have a nomination package sent out to the industry as soon as possible. The deadline for interested parties is September 12, 2014. The TF would like to have replacement candidates ready for approval during the November 2014 Board of Trustee (Board) meeting. Packages for the Board must be submitted six weeks prior to the meeting.

b. Budget Task Force: The Budget TF reviewed the 2nd quarter budget for 2014. The committee at large accepted the budget report.

c. Record Retention, Storage and Transmittal Task Force: Training was delivered to NERC staff in Atlanta during 2nd quarter 2014. NERC staff was very receptive to the processes and procedures.

d. Implementation Task Force: NERC continues to receive feedback comparing PSI test facilities to

previous AMP 3rd party Sylvan facilities. PSI’s primary product is flight credential testing. Therefore, a large majority of test centers are in airport hangers. A recommendation was made to vett more of the PSI facilities. The challenge with labeling a particular test center as “less than preferred” is that removing the test center may cause a candidate to travel farther to take an exam. The situation must be evaluated from all perspectives. The TF will evaluate facility feedback surveys from candidates and create an action plan for resolving “bad” test centers.

e. Policy Oversight Task Force: The Policy Oversight TF submitted their scope document to the

committee at large. A suggestion was made to add a revision history to track document changes.

f. Exam Development and Delivery Task Force: The Exam Development and Delivery (EDD) TF chair, John Kerr, is very pleased with the test development vendor, PSI and psychometrician, Amin Saiar, PhD. The online reviews were very good. The Exam Working Group (EWG) is engaged in the process and expects to complete evaluation of the items for the upcoming set of exams. There will not be a transition period for the new 2015 exams. The current exams will be available through December 31, 2014 and the new exams will start in 2015.

The EWG provided an update that all scored items have been reviewed and checked for references. The remaining pre-test items should be completed before the end of the month. The EWG will check references and exam study guides for accuracy. References may include industry

Meeting Minutes – Personnel Certification Governance Committee Meeting – August 12-13, 2014 5

practices, publications, or operating experience. It has been challenging to validate the items for the Balancing and Interchange (BI) and the Balancing, Interchange and Transmission (BT) credentials. Those exams are taken the least out of the four credentials. The EWG is excited about the LOFT process. LOFT is Linear on the Fly Tool which allows items to change on the exam forms in real time (at the time the candidate sits for the exam). Based on the number of items in the Reliability Coordinator (RC) bank, LOFT may be an option for this exam in the very near future. The EWG recommends planning an Item Writing Workshop in the near future to assist the other three credentials in transitioning to the LOFT process. Targets: Beta Testing September – October 2014 Launch new exams January 2015 Job Task Analysis development January 2015 Complete exam specifications August 2015 Exam Development September – December 2015

g. SOCCED Joint Task Force: No update provided prior to meeting.

9. Document Review The committee at large reviewed the following documents: Nomination Task Force Scope, Hardship Task Force Scope and Appendix A from the System Operator Certification Program Manual. Feedback was received from one of the CE Program course reviewers to add more specificity to the course topics listed in Appendix A. The committee at large agreed to reevaluate Appendix A after the 2015 JTA.

10. Future PCGC Meetings Dates: Location: Comments: November 12-13, 2014 Atlanta, GA (Westin) Annual Board Update February 10-11, 2015 Tampa, FL Joint Meeting with PS May 5-6, 2015 San Antonio, TX August 11-12, 2015 San Francisco, CA November 3-4, 2015 Atlanta, GA Annual Board Update

11. Roundtable

12. Closing Comments and Adjourn – Brett Hallborg, Chair Meeting was adjourned at 1:03 p. m. Wednesday, August 13, 2014.

May 2014 Attendees: Brett Hallborg, Chair Jake Burger Dave Carlson Eric Dietzman Mike Anderson, Vice Chair Bill Ellard Mike Hoke John Kerr Trion King, NERC Staff, Secretary Don Oatman Mike Scott

Attachment A

Agenda Personnel Certification Governance Committee

August 12-13, 2014 | 8:00 a.m. – 5:00 p.m. Pacific Grand Hyatt Seattle 721 Pine Street Seattle, WA 98101 Introductions and Chair’s Remarks NERC Antitrust Compliance Guidelines and Public Announcement Agenda

1. Administrative

PCGC Member(s): Trion King

Objective: Review agenda, meeting minutes, action items and task list

2. Approve Meeting Minutes

PCGC Member(s): Trion King

Objective: Approve previous meeting minutes: May 5-6, 2014 St. Louis, MO

3. Status Reports

a. System Operator Certification Program Reports

PCGC Member(s): Trion King

Objective: Present statistics of System Operator Certification Program

Background: PowerPoint Presentation will be provided during meeting.

b. Continuing Education Program Reports

PCGC Member(s): Trion King

Objective: Present statistics of Continuing Education Program

Background: PowerPoint presentation will be provided during the meeting.

4. System Operator Certification Program

a. Refunds

PCGC Member(s): Trion King

Objective: Discuss increase number of mistakes in selecting Renewal or New exam.

Background: There have been a significant increase in the number of refunds because operators are selecting a new exam instead of renewal.

b. Duplicate Accounts

PCGC Member(s): Trion King

Agenda – Personnel Certification Governance Committee Meeting August 12-13, 2014 2

Objective: Duplicate operator accounts.

Background:

c. Legal Name Changes

PCGC Member(s): Trion King

Objective: Names were changed prior to policy.

Background: Document requirements.

5. PSI Updates

PCGC Member(s): Trion King

Objective: Update on test centers and timely data transfer.

Background: Continuing to get calls about test centers.

6. SOCCED Update

PCGC Member(s): Melinda Nicius

Objective: Provide SOCCED Update. Status of Phase 2A Requirements and SOW. Automated emails.

Background: PowerPoint presentation will be provided during the meeting.

7. EWG Update

PCGC Member(s): Trion King

Objective: Status of item review and Item Bank.

Background:

8. Task Force (TF) Reports

a. Nomination Task Force: PCGC

PCGC Member(s): Bill Ellard, TF Chair; John Kerr; Don Oatman; EWG Chair

Objective: Status of PCGC membership. Nomination TF Scope update.

Background:

b. Budget Task Force:

PCGC Member(s): Dave Carlson, TF Chair; Mike Anderson, Jake Burger, Brett Hallborg

Objective: Review 2014 YTD Budget

Background: Budget TF will provide an update during the meeting.

c. Record Retention, Storage and Transmittal Task Force:

PCGC Member(s): Dave Carlson, TF Chair; Don Oatman, Bill Ellard

Objective: Develop training for NERC staff and Implementation process.

Background: Dave Carlson to provide update on training process during the meeting.

d. Implementation Task Force:

PCGC Member(s): Jake Burger, TF Chair; Mike Anderson, Bill Ellard

Objective: Status of transition; PSI test center complaints.

Background: Implementation of New Exam Development and Testing Vendor: Website cleanup. TF to provide update during the meeting.

Agenda – Personnel Certification Governance Committee Meeting August 12-13, 2014 3

e. Policy Oversight Task Force:

PCGC Member(s): Jake Burger, TF Chair; Eric Dietzman; Mike Scott; Dave Carlson

Objective: Policy Oversight TF scope. SOC Program Manual Update

Background:

f. Exam Development and Delivery Task Force:

PCGC Member(s): John Kerr, TF Chair; Dave Carlson; Eric Dietzman; Mike Scott

Objective: Update of exam development cycle.

Background: Responsible for tracking exam development.

g. SOCCED Joint Task Force:

PCGC Member(s): Eric Dietzman; Dave Carlson

Objective: Provide support to Phase 2A Update with PS members

Background:

9. Task List Review –

PCGC Member(s): Trion King

Objective: Review task lists and ensure items are properly documented.

Background: Included in the meeting minutes.

10. Future PCGC Meetings:

Dates: Location: Comments:

November 12-13, 2014 Atlanta, GA Annual Board Update

February 10-11, 2015 Tampa, FL Joint Meeting with PS

May 5-6, 2015 San Antonio, TX

August 11-12, 2015 TBD

November 3-4, 2015 Atlanta, GA Annual Board Update

11. Roundtable

12. Closing Comments and Adjourn – Brett Hallborg, Chair

Attachment B

Action Items Personnel Certification Governance Committee August 12 - 13, 2014

Item# Task Assignment NERC Support Staff Due Task Force

2013-2.1 PCGC Secure Site to SharePoint Trion King Trion King; NERC IT Next Meeting NERC

2013-4 Mass Email Option (Ability to contact operators through the use of SOCCED) - Proceed. Melinda has developed an alternative way to send mass emails.

Trion King Melinda Nicius/NERC IT

PCGC

2013-32 Update EWG Scope: Renewal Fee Policy Trion King/Technical Publications

Next Meeting EDD

2013-38 Budget YTD Budget On-going Budget

2014-1 Early Renewal Email response Process Trion King Melinda Nicius Will be included in SOCCED Phase 2A

2014-8 PSI to provide quarterly report. Track comments associated with score of Poor.

Trion King Ongoing EDD

2014-9 Finalize Policy Oversight Scope

Add revision history

Policy Oversight

Next meeting Policy Oversight

2014-21 2015 Joint meeting: Feb. 10th – 11th Trion King Meeting Planner

2014-24 Send draft of BOT report to Chair Trion King Ongoing

2014-26 Verify Standards references are accurate for 2015 Exams and Glossary of Terms for pilot items (2012).

EWG Next Meeting EDD

2014-27 Kick Start of 2018 Exam Process EDD November 2014 EDD

Meeting Action Items – Personnel Certification Governance Committee – August 12-13, 2014 2

Item# Task Assignment NERC Support Staff Due Task Force

2014-28 Create fluid process of reviewing exam items with current Standards (2018)

EDD

Long term – for 2018 exams

2014-29 Create Hardship TF Scope Hardship TF Completed Hardship

2014-30 Verify Hardship cases are documented in SOCCED

Trion King Next Meeting

2014-31 Update SOC Program Manual to reflect current vendor information

Trion King Technical Publications Reviewed by Technical Publications. Under review by Policy TF.

Policy Oversight

2014-33 Update How to Apply for SOC exam. Email process has changed.

EDD Trion King Next Meeting

2014-34 Response to request waiver Trion King

2014-35 Response to request for Carry over extension period

Trion King

2014-36 List of duplicate System operator accounts (short cut 42 days)

Trion King

2014-37 Send SPP candidate comments to PSI for response

Trion King Implementation TF

2014-38 Nominations for PCGC (2) announcement

Trion King Sandy Shiflett Next meeting

2014-39 Nominations TF Scope revision Trion King

2014-40 Hardship TF Scope revision Trion King

2014-41 Update status from approved to active Trion King

2014-42 Letter to all Providers – Formal review of ILAs

Trion King Melinda Nicius/Brenda Boline