2011 ReliabilityFirst 693 Compliance Audit Process for 6 Year Audit

Cycle Entities

Glenn Kaht

Senior Consultant - Compliance ReliabilityFirst Corporation

January, 2011

05/03/23 2

Presentation Goals

The goals of this presentation are to: Discuss Compliance Audit references and define

“Compliance Audit” Discuss the Reliability Standards that are or may be

within the scope of a 2011 Compliance Audit and the audit review period

Provide an overview of the audit process for entities that are on a 6 year audit cycle

Answer questions regarding the 2011 Compliance Audit process for registered entities that are on a 6 year audit cycle

05/03/23 3

Audit Process References

Some references used in the performance of Compliance Audits: ReliabilityFirst Compliance Monitoring and

Enforcement Program (CMEP) NERC Rules of Procedure 2011 NERC and ReliabilityFirst Implementation Plans NERC 2011 Actively Monitored Reliability Standards ReliabilityFirst 2011 Compliance Monitoring Schedule Questionnaire-Reliability Standard Audit Worksheets

(Q-RSAWs)

05/03/23 4

Compliance Audit - Definition

What is a Compliance Audit?

“A systematic, objective review and examination of records and activities to determine whether a Registered Entity meets the requirements of applicable Reliability Standards.”

05/03/23 5

6 Year Audit Cycle Basis

NERC Rules of Procedure section 403.11.1: “For an entity registered as a balancing authority, reliability coordinator, or transmission operator, the compliance audit will be performed at least once every three years. For other bulk power system owners, operators, and users on the NERC Compliance Registry, compliance audits shall be performed on a schedule established by NERC.”

At this time, there are no plans to audit PSEs in 2011 Compliance Audits for other entities are to be performed at least

once every six years.

Compliance Audits of registered entities subject to a compliance audit at least once every six years will be conducted off-site from the facilities of the audited entity (although ReliabilityFirst may conduct audit activities on-site if deemed necessary).

05/03/23 6

Reliability Standards Within Audit Scope

Which Reliability Standards are within the scope of 2011 Compliance Audits? All applicable NERC Reliability Standards/requirements

identified to be monitored via Audit in the NERC 2011 Actively Monitored Reliability Standards list (unless NERC approves the exclusion)

Additional NERC Reliability Standards/requirements selected by ReliabilityFirst to be included within the scope of the Compliance Audit

ReliabilityFirst Standards approved by NERC and FERC

Open and completed mitigation plans will be reviewed by the audit team

05/03/23 7

Audit Review Period

In general, the audit review period for 2011 Compliance Audits is as follows: Current and 3 previous calendar years through the end of the

audit (i.e., January 1, 2008 through the end of the audit) Caveats: The start of the audit review period for a particular function will

be no earlier than the date that an entity is placed on the NERC compliance registry for that particular function.

If an entity was subject to a compliance audit within the 3 previous years, then the start of the 2011 audit review period corresponds to the end of the previous audit.

05/03/23 8

Data Retention Requirements

Data retention requirements for 2011 Compliance Audits Reference NERC Compliance Process Bulletin #2009-005 (Current In-Force

Document Data Retention Requirements for Registered Entities) issued on June 29, 2009

Generally consistent with data retention requirements identified within a particular Reliability Standard

Data retention section of PRC-005 specifies: “…shall retain evidence of the implementation of its Protection System maintenance and testing program for three years.”

Since the registered entity may specify a M&T interval longer than three years, the registered entity is expected to provide evidence of implementation of its Protection System M&T program for the entire review period

2011 Audit Process Overview

High level overview of 2011 Compliance Audits: 90 day audit notification to entity 85 day conference call with entity Entity submittal of pre-audit survey and sampling data

from Attachment C 30 days after receipt of 90 day notification

Entity submittal of completed Q-RSAWs and evidence 40 days before scheduled start date of audit

Audit team pre-audit review of evidence Off-Site Reviews Audit Report Completion

05/03/23 9

05/03/23 10

90 Day Audit Notification

A Compliance Audit Notification will be sent approximately 90 days prior to the start of the audit. The notification will include: 90 Day Audit Notification General Instructions Work history and participant agreements of

ReliabilityFirst audit team members Pre-audit survey Attachment A - List of Standards/Requirements within

the initial scope of the audit

05/03/23 11

90 Day Audit Notification – Cont’d

Attachment B - Entity Certification Signature form Attachment C – Data Sampling Evidence Spreadsheets Q-RSAWs for the NERC Standards within the initial

scope of the audit

90 Day Audit Notification and General Instructions

The 90 Day Audit Notification and General Instructions provide information and instructions regarding the audit and audit process and discusses information contained in the 90 day notification package (pre-audit survey, Q-RSAWs, etc.)

05/03/23 12

05/03/23 13

Work Histories and Participation Agreements

Work histories and participation agreements (e.g., Code of Business Conduct and Ethics, Confidentiality/Non-Disclosure) of the ReliabilityFirst audit team are provided to the audited entity.

Section 1500 of the NERC Rules of Procedure governs NERC staff (and the ReliabilityFirst audit team) responsibilities and obligations regarding Confidentiality.

Members of the audit team will not sign an entity specific confidentiality agreement.

05/03/23 14

Audit Team Makeup

The audit team will typically consist of 2 or more members with experience in Planning and/or Operations.

Audit Team Lead (Typically a member of the ReliabilityFirst Compliance Staff)

Audit Team Co-lead (if the audit team has 2 or more sub-teams)

Other team members or observers NERC observers and/or participants (@ NERC’s

discretion) FERC observers and/or participants (@ FERC’s

discretion)

05/03/23 15

Objection to a Team Member

A Registered Entity can object to an audit team member’s participation on the audit team: Objection may be based on the grounds of conflict of

interest, or the existence of other circumstances that could interfere with the team’s impartial performance of their duties

Objection must be provided in writing to ReliabilityFirst no later than 15 days prior to the start of the audit

ReliabilityFirst will make the final determination regarding the team member’s participation in the audit

NERC and FERC staff cannot be limited in their participation on an audit

05/03/23 16

Compliance Pre-Audit Survey

The pre-audit survey must be completed by the audited entity in order to provide the audit team:

General information of the organization, including contact information, registration details, organization profile, neighboring entities, etc.

Information regarding the audited entity’s internal compliance program and culture

Attachment A

Attachment A is a worksheet that:

Identifies all Standards/Requirements that are within the initial scope of the audit.

Identifies the applicable function(s) for each Requirement within the initial scope of the audit.

Can be used by the audit team and the audited entity to manage/track the audit scope and progress.

The audited entity should provide responses/evidence for each entry in Attachment A.

The scope of the audit may be expanded beyond the initial scope of the audit identified in Attachment A!

05/03/23 17

05/03/23 18

Attachment B

Attachment B - Entity Certification Signature form is to be completed and signed by an individual authorized to execute the Certification.

The individual who signs Attachment B is attesting that

the statements and supporting documents included in the response and appended to the certification are true and correct as of the date of signing.

The completed and signed Attachment B should be submitted to ReliabilityFirst at the same time that the evidence and completed Q-RSAWs are submitted.

Attachment C

In early 2011, the 90 day audit notification will include Attachment C – Data Sampling. Attachment C will include evidence/information requests for specific requirements.

Examples of items that may be requested:

Operators logs, voice recordings, etc. for specific days Evidence of submittal of study information for specific days List of entity equipment (substations, transmission and

generation protective equipment, UFLS relays, SPS equipment, etc.)

05/03/23 19

Attachment C – Cont’d

Attachment C is a tool that will be used by the audit team and the audited entity to compile certain evidence.

The use of Attachment C is intended to make the audit process more systematic and increase audit efficiency

Attachment C is not an all-inclusive listing of evidence that will need to be provided by the audited entity

05/03/23 20

Evidence Spreadsheet

In early 2011, the 90 day audit notification will include an Evidence Spreadsheet. The Evidence Spreadsheet: Is a guidance tool to be used by audited entities in their

compilation of evidence. Using the Evidence Spreadsheet does not ensure compliance but assists the entity and may increase efficiency for the audited entity and the audit team.

Is a listing of Standards/Requirements and types of evidence (agreements, procedures, logs, voice recording, etc.) that the entity should submit as evidence as per the requirements

Is not an all inclusive listing

The audit team may request additional substantiating evidence to assist the audit team in a determination of compliance

05/03/23 21

05/03/23 22

Q-RSAWs

Q-RSAWs: Audit worksheets for the Reliability Standards Provide guidelines concerning the

requirements (Compliance Assessment Approach)

Do not add additional requirements Posted on NERC Website Entity sections of the Q-RSAWs must be fully

completed and returned (including supporting evidence) 40 days before the scheduled start date of the audit

85 Day Conference Call

Approximately 85 days prior to the start of the audit, the Audit Team Lead will contact the audited entity to discuss the audit. Topics may include:

The 90 day notification package The pre-audit survey The Q-RSAWs Particular details of the audited entity Guidance on evidence submittalsAdditional questions from the audited entity

05/03/23 23

30 Day Submittals

No later than 30 days after receipt of the 90 day notification, the audited entity is to submit the following to ReliabilityFirst:

The completed pre-audit survey

Sampling evidence/information as specified in Attachment C

05/03/23 24

40 Day Submittal of Evidence

No later than 40 days prior to the scheduled start date of the audit, entities are to submit:

Completed Q-RSAWs Evidence of compliance to the

Standards/Requirements within the initial scope of the audit (Attachment A)

Completed and signed Attachment B

05/03/23 25

Audit Team Pre-Audit Reviews

After the initial evidence has been submitted, and prior to the scheduled start date of the audit, the audit team may conduct pre-audit reviews in order to: Schedule the opening presentation Review/discuss the evidence and information

submitted Make preliminary compliance determinations Develop additional requests for evidence as

necessaryMay be sent to the entity prior to the audit

05/03/23 26

05/03/23 27

Off-Site Reviews

The off-site reviews are conducted at the ReliabilityFirst offices and are expected to be completed within the assigned audit period that has been scheduled, but may be extended if necessary. It is not expected that the audited entity be present at the ReliabilityFirst offices during the reviews. The off-site reviews include: An opening presentation conducted by the audit team A review of compliance to the Standards/Requirements within

the scope of the audit An exit presentation scheduled and conducted by the audit team

05/03/23 28

Audit Team Opening Presentation

The audit team will conduct an opening presentation which will: Introduce the audit team members Review the authority of ReliabilityFirst Review the objectives and scope of the audit Discuss confidentiality issues Provide an overview of the audit process Discuss the role of SMEs Discuss evidence and types of evidence Answer questions related to the audit process

Off-site Reviews

The audit team will complete reviews of the evidence submitted by the audited entity. The audit team may request the audited entity to provide clarification of submitted evidence.

The audited entity should have SMEs available during the scheduled audit period.

Additional evidence may be requested by the audit team.

05/03/23 29

On-Site Visits

On-Site visits to entity facilities may be conducted as deemed necessary by the audit team.

05/03/23 30

05/03/23 31

Exit Presentation

At the conclusion of the audit, the audit team will conduct an exit presentation to: Review the audit scope Discuss the terms used in the audit findings Present the preliminary findings of the audit team

Explain the basis of any possible violations identified Review possible outcomes/actions resulting from possible

violations identified by the audit team (dismissal, notifications, appeals, settlement negotiations, mitigation plans, etc)

Discuss “Areas of Concern” identified by the audit team Discuss “Items for Consideration” identified by the audit team Discuss the audit report process and timeline Discuss feedback that the audited entity may provide regarding

the audit team and the audit process

05/03/23 32

Audit Report Completion

After the completion of the Compliance Audit, the audit team will develop a Compliance Audit report. There are 2 versions of the Compliance Audit report: Non-public version Public version (confidential information is redacted)

The audited entity will be provided the opportunity to review and comment on the audit report.

The Audit Team Lead develops a

draft report

The Audit Team Lead receives

comments from the Audit team

Audit Team provides

comments

The Audit Team Lead transmits the report for audit team

review

20 Business days

The Audit Team conducts an exit briefing

with the Registered Entity with preliminary

findings

Audit Team Lead sends the draft

report to the Audit Team for their

review and comments

The Audit Team Lead sends the

draft report to the Registered Entity for their review and comments

Audit Team Lead revises the draft

compliance report

The draft report is edited upon receipt of

Registered Entity

comments

Audit Team Lead

revises the report upon receipt of

Audit Team’s

comments

Send final report to RFC VP and

Director of Compliance, NERC and

Registered Entity

Audit Report Process and Timeline

20 business days

10 business days

5 business days

5 business days

Registered Entity reviews and

provide comments

Revision of the draft report

Audit Team provides

comments

5 business days

Audit Team Lead

completes final

compliance report

5 business days

2011 Compliance Audit Process 6 Year Audit Cycle

Questions?

34