October 26, 2021

Barry Jones

Western Area Power Administration

NERC CIP, O&P Compliance and

Information Assurance

Western Area Power Administration

NERC Reliability Standards Internal

Compliance Mock Audit Program

Western Area Power Administration Mock Audit

Program

3

10/26/2021

WAPA NERC Compliance Organizations, Processes and Technologies

WAPA Mock Audit Program

WAPA has established a cost-effective mock audit program to assess the health of reliability standards (HOS) for WAPA’s internal NERC reliability compliance program (ICP).

The program enlists the help of volunteer industry personnel paired with WAPA regional internal employees to collaborate and assess the program.

• Barry Jones – WAPA Sierra Nevada Regional

Compliance Manager

WAPA Mock Audit Program

About and Benefits

• Allows WAPA to assess and report WAPA’s Health Of

Standards – across all 4 WAPA regions

• An assessment of the program and internal controls from

both an internal (WAPA) and external (participants)

perspective

• WAPA and peer personnel benefit from peer check

assessments, collaboration and training

• Peer assessments used for process improvements and risk

mitigations

WAPA Mock Audit Program

WAPA Mock Audit ProgramAbout and Benefits (cont.)

• Training, collaboration and alignment of internal roles and responsibilities to compliance tasks and obligations

• WAPA customers and other entity participation (collaboration)

• Program and tools can be used for self-assessment• Provides industry with a cost-effective program to model

NERC Reliability Standards• Can be virtual or in-person or a combination• Preparation for WECC and MRO audits

1. Type of Mock Audit

2. Scope

3. Schedule

4. Roles

5. Logistics

6. Training

7. Implementation

8. Post Mock Audit

Where to Begin? Plan Framework

Logistics

Scope

Type

Implementation

Training

Schedule

• Full Mock Audit vs Partial vs Spot Check

▪ Desired outcomes and reporting?

▪ Timing – plant shutdowns and vulnerability

assessments

▪ Virtual or In-person?

▪ Impact to internal teams and resources?

▪ Mimic real life – bookend dates

▪ Entity core values / Leadership

Type of Audit

Schedule

Scope

Type

• CIP and O&P Standards▪ What areas are strong? Not as strong?

▪ Assess Internal Controls?

▪ Auditee leads and SMEs respondents/sub teams

▪ Mock auditor leads and SME’s - sub teams stds/reqs

▪ Evaluate Internal Compliance Program?• Considerations and contingencies

▪ Impacts to departments and resources• Resource availability (plant shutdown)

• Operating Events and other

Scope

• 3 Week Virtual Audit Schedule – build timeline

▪ Implement at least 6 months before regular audit

▪ 2 weeks of data requests and reviews

✓ Allows participants time to review documentation,

collaborate and prepare questions

✓ Touchbase sub-team meetings

▪ 1 week virtual onsite

▪ Interviews

▪ Further data requests

▪ Begin audit reporting

Schedule

• Coordinator (critical)• Can be internal or external

• Help mock auditor leads• schedules, sub-teams, reviews, scoring tools, DRs,

interview submittals, final report out, etc…

• DR Manager – routes DRs • Lead Auditors• Lead Auditees• SMEs – auditor and auditee• Notetaker and Observers

Roles – Mock Audit Preparation

• Identifying Resources and Tools - Auditors

• How will we authorize mock auditors?

• How will mock auditors collaborate between

each other or peer check in different time

zones and schedules?

• Tools to collaborate (Webex, Teams)

• How will mock auditors receive and review

documentation (security and confidentiality)?

Logistics – Questions…

• Identifying Resources and Tools – Auditors

(cont.)

• How will mock auditors agree on balls and

strikes (tools)?

▪ Different backgrounds, entity practices, experience,

cultures, etc…

• How will mock auditors document and report

status to a meaningful report (tools and

analytics)?

Logistics – Questions…

• Coordinating – Internal Resources

• Same as a regular audit

• Identifying and notifying organizations and

SMEs of mock audit or spot check - coverage

• Prepare Documentation RSAWS, ERT, Policies,

procedure, evidence

• SME training

▪ Tools – using secure file transfer

Logistics – Mock Auditee Preparation

• RSAW Development ▪ 3 Pillars of Compliance

▪ Person,

▪ Administrative Control/Process, and

▪ Evidence

▪ Clear, Concise and relative

▪ ERT Tips and Tricks

• Data Requests▪ Do not re-write the RSAW as a response

▪ How to submit a DR response

• Interviews▪ ID whom will be interviewed

▪ Practice questions around 3 pillars

Training Preparation - Auditees

• Welcome Packet

• Audit scope

• Schedule

• Introductions and expectations

• NDAs

• 2 auditors per standard requirement minimum

• Peer check each other for efficiency and

accuracy

Training Preparation - Auditors

• Auditing Process and Tools• Auditing Criteria – 5-point qualitative basis review of

RSAWs, Administrative Controls, Evidence and Interviews (3 pillars)

• Well Supported

• Supported• Generally Supported

• Partially Supported

• Not Supported

• Auditor Worksheet Tool

• Scoring, ranking using quantitative analytics to report status and risk (similar basis as VRF/VSL)

Training Preparation - Auditors

• Contingencies and Other

• Emergencies and events

• Internet connectivity

• Illness

• Family

• Work

• Outages

• Other

Training Preparation - Auditors

• Weeks 1 & 2▪ RSAWs, ERT and Evidence from Scope placed on

Secure FTP site

▪ Auditors begin review, take notes and issue DRs

✓ Individual and sub-team collaboration

✓ Prepare for additional DRs and/or interviews

✓ Touchbase sub-team meetings

✓ Auditor Worksheets

▪ DR auditee responses are held until Monday of week 3 – then sent to audit team

Implementation - 3 weeks

Implementation

• Week 3 (Mimics on-site audit)

▪ Monday – Opening Presentation and introductions

▪ Auditors receive DR responses and interview requests

(Webex/Teams)

▪ Monday to Wednesday – Auditors, Interviewees,

notetakers attend interviews – follow typical audit

processes

▪ Daily audit status meeting and release of SMEs

▪ Audit leads and Coordinator begin to develop

report based on scored closed items

Implementation - 3 weeks

• Week 3 (Mimics on-site) (cont.)

▪ Wednesday - Auditors sub-teams issue final

DRs and tally scores for Auditor Worksheet

▪ Thursday – no further DRs or interviews –

mock audit leads review scores for each

std/req.

▪ Mock Audit Team finalizes report

▪ Friday – WAPA Closing Presentation and Report

Implementation - 3 weeks

• Sort issues identified by the MA teams and consolidate the assessed issues

• Identify risk issues

• Communicate status

• Remediate and mitigate – working with SMEs and POCs for continuous process improvement

• Culture of compliance/just culture

• Prepare for the WECC Notice of Audit

Post Mock Audit – Issues Identified

• Review lessons learned from MA process

• What can we do better?

• Solicit feedback from auditors and participants

• Send thank you letters to all participants and their management

• We learn from each other - share…

“Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin

Post Mock Audit – MA Process

Sample Audit Scope and Auditors

Priority Std Req

O&P Lead Auditor Name

CompanyEmailPhone

CIP Mock Auditor

CIP Mock Auditor

CIP Mock Auditor

(O) Observe

CIP Mock Auditor

(O) Observe

CIP Mock Audit Observer

(watch/listen)(takes notes for

lessons learned/end of

day status)

Mock Audit POC (WAPA employee)

Mock Audit Coordinator and Quality Reviewer

(coordinates/manages data requests between Mock Audit team and

WAPA DR Team to SMEs (and reverse)

(ensures observations are tallied for daily

status report end of day)

NDA & Training?

2CIP-002-5.1a R1.0

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Yes

2CIP-002-5.1a R1.1

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Yes

2CIP-002-5.1a R1.2

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Yes

2CIP-002-5.1a R1.3

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Name: Company: Phone:Email:

Yes

Sample Auditor Worksheet

ReviewDate

ReviewerName

Std Req Completed

Documentation Quality Review Type

(see Example Basis Types tab)

Observation(mock audit period (bookends)

06/01/2019 to 10/01/2020)

Data Request

Interview Request

Compliance Evaluation

Ranking (use compliance

ranking guide)

RecommendationBest

PracticeAdditional Notes

10/1/20 NameCOM-002-2

R2 No RSAW QualityRSAW did not include reference to a

Section of the administrative procedure

No NoGenerally Supported

List current evidence No

*NOTE: Value judgment about the RSAW quality. Keep in mind RSAW Quality is feedback regarding the RSAW and how it communicates the NERC Internal Compliance Program at audit - the SME's will not be able to update the RSAW based on your comments during this audit as it's a future revision.

10/1/20 Name CIP-004-6R3.1 No Admin Control QualityProcedure XYZ did not contain a

process to confirm identity as required Part R3.1

Yes YesPartially

Supported

Update procedure with a section for confirming identity.

This will also help put all requirements into a single

administrative procedure with controls listed to meet 3.1

No Note: is the Administrative Control (Policy, Order, Plan, Procedure, Process, manual, work instruction, SOP) provided accurate, up to date, etc..? See the Examples tab for some ideas

10/1/20 Name CIP-007-6R3.1 No Evidence Quality

Evidence provided was a screen capture from 2015. The mock audit book end dates are from 10/1/2022

to 10/1/2023. Will DR for recent evidence

Yes No Not Supported

Evidence did not support requirement to Deploy

method(s) to deter, detect, orprevent malicious code.

No Note: is the Evidence (list, screen capture, record, completed task, diagram, drawing, etc...) relevant, recent, applicable, meaningful, etc..? See the Examples tab for some ideas

10/1/20 NameTOP-001-4

R6 Yes Other

During interview, the SME for this requirement stated that WAPA does

not know who their Balancing Authority is for the region. The SME

then said that each operator assumes that the BA is only itself and WAPA has nothing to do with

BA's

Yes No Not Supported

Evidence did not support requirement to Deploy

method(s) to deter, detect, orprevent malicious code.

No

**NOTE: Interviews, observations about culture of compliance, silos, corrective actions programs, organization, human performance, internal controls, RCA/ACA process, leadership

• Review lessons learned from MA process

• What can we do better?

• Solicite feedback from auditors and participants

• Send thank you letters to all participants and their management

• We learn from each other - share…

Pros and Cons of the WAPA MA

WAPA Mock Audit Program

Questions?

Contact:

Barry Jones

Western Area Power Administration

NERC CIP, O&P Compliance and Information Assurance

30