NERC Internal Controls Evaluations...Apr 11, 2017 · NERC Standard COM-002-4 4/12/2017 12042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711 | [email protected]

April 11, 2017

NERC Internal Controls Evaluations

Common Practices, Approaches, and Other Control Ideas

Introductions

Archer Energy Solutions acquires compliance division of Utility System Efficiencies Panelists

o Richard Shiflett

o Bob Dintelman

4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | [email protected]

Objectives

Identify what internal controls are and why they are needed

Discuss risk thresholds and risk mitigation Discuss the types and characteristics of controls Discuss key controls Discuss a defense-in-depth approach for controls Provide a controls evaluation example for COM-

002-4 R5


What is an Internal Control?

A process effected by an entity’s oversight body, management, and other personnel that provides reasonable assurance that the objectives of an entity will be achieved:• Operations – Effectiveness and efficiency of

operations• Reporting – Reliability of reporting for internal

and external use• Compliance – Compliance with applicable laws

and regulationsTaken from United States General Accounting Office Standards for Internal Control in the Federal Government


http://www.gao.gov/assets/670/665712.pdf

Benefits of Internal Controls

Why would an entity want quality internal controls?


https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwjEs5ja_d7JAhXIKWMKHbu_B8IQjRwIBw&url=https://cjess1.wordpress.com/2013/03/02/internal-control-4/&bvm=bv.110151844,d.cGc&psig=AFQjCNGEcFEUomMztDMCXdun2BFGyPkB5A&ust=1450307126793767

Risk and Internal Controls

Identify risks and determine risk acceptance levels or thresholds


Risk and Internal Controls Internal controls help mitigate risk exposure Risk profiles drive nature and complexity of internal controls


Nature of Internal Controls

Internal controls can range in nature and complexityo ID cards, fences, locks, Virtual Private Network (VPN), or fireproof fileso Independent verification of processes and deliverableso Authorization of employee time cards


Basic Types of Controls Preventive

o Aimed at preventing any errors or irregularities from occurring which may have negative effects

o Example: Documented process requiring development and maintenance of training schedule

Detectiveo Designed to discover any errors or irregularities which may have occurredo Example: Documented process requiring periodic review to identify any

required training not completed as scheduled, as well as training not completed per reliability standard requirements.

- Quarterly review of completed training records to identify individuals who have not completed training by the required deadline.

- Documentation and utilization of an event review and root cause analysis process to determine cause and affects surrounding an unwanted event


Basic Types of Controls

Correctiveo Corrective controls restore the system or process

back to the state prior to a harmful evento Example: An entity may implement its restoration

plan for a computer system from backup tapes after evidence is found that someone has improperly altered the data


Control Characteristics

Examples of how controls may be characterized:


Less Assurance More Assurance

Manual Automated

Can be overridden Cannot be overridden

No management oversight Has management oversight

Simple Complex

Performed by junior or inexperienced personnel

Performed by experienced personnel

Single control Multiple or Layered Controls

High level Detail or transactional level

Control tests a sample Control tests entire population

Occurs after the fact Occurs in real time

Key Controls

What is a key control? A key control is a control that, if it fails, means

there is at least a reasonable likelihood that a material error would not be prevented or detected in a timely basis. In other words, a key control is one that is required to provide reasonable assurance that material errors will be prevented or timely detected.


Key Controls Example An entity has a list of 25 controls that it feels addresses

a risk area identifiedo Five controls occur at the end of the entire process and

confirm that the other 20 controls have done their work and that there are indeed no remaining problems or other errors.

Without the 20 earlier controls there would be a huge number of errors coming through and the five final checks would be little comfort

Focusing on the five final controls, that usually found nothing requiring correction, might be enough for the key control review if the controls are designed and implemented correctly.


Key Control Factors

Factors that help uncover possible key controls Likely points of failure

How controls rely on each othero Look at interaction between controlso Individual controls may not address the risko Some controls prevent other control failures


Controls – Defense in Depth

Preventive, Detective, and Corrective Layered controls supporting and enhancing

key controls Control output visibility


Controls - Defense in Depth

What is the right amount of controls?


Controls Evaluation ExampleCOM-002-4 R5

Each Balancing Authority, Reliability Coordinator, and Transmission Operator that issues an oral two-party, person-to-person Operating Instruction during an Emergency, excluding written or oral single-party to multiple-party burst Operating Instructions, shall either: • Confirm the receiver’s response if the repeated information

is correct (in accordance with Requirement R6).• Reissue the Operating Instruction if the repeated

information is incorrect or if requested by the receiver, or• Take an alternative action if a response is not received or if

the Operating Instruction was not understood by the receiver.

NERC Standard COM-002-4


http://www.nerc.com/_layouts/PrintStandard.aspx?standardnumber=COM-002-4&title=Operating%20Personnel%20Communications%20Protocols&jurisdiction=United%20States


Entity provides its communications protocols document Operator consoles have visual reminder to use 3 part

communication The entity has implemented a detailed and technical initial

training program for system operators, and retrain periodically

Operators use 3 part communication for all information exchanges

All operator communications are recorded Shift supervisor regularly listens to the recordings to verify

3 part communication Feedback to operators on improving 3 part communication



Preventiveo Communications protocol documento Operator visual remindero Initial and continual training of operatorso Use of 3-part communications throughout

Detectiveo Review of audio recordings by supervisoro Communications protocol document may have

detective controls present Corrective

o Feedback to operators for improvement



Key controls identifiedo Communications protocol documentationo Review of audio communications by supervisor

Characteristicso A mix of automated and manual controls, but largely

manualo No indication of management oversighto Controls are relatively simple and performed by

experienced personnelo Not clear if supervisor review of audio recordings are

sampled or noto No mention made of communications during Emergencies



Evaluationo Request evidence that controls are presento Grade may range from Partially Implemented to

Largely Implementedo Recommend entity include some form of

management oversight and/or notifications based upon Operating Instructions issued either as part of Emergencies or otherwise.


Q & A

Please post your questions to the Q&A area of the webinar.

If you would like, you may email us directly at:Richard Shiflett –[email protected] Dintelman –[email protected]


mailto:[email protected]

mailto:[email protected]

Page 1 of 1

[email protected] | www.archerenergysolutions.com 800‐805‐7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015

Scenario Instructions

Prior to conducting this exercise, students should know what the three types of controls are, how to identify key controls, and evaluate controls according to the NERC guidance document. By performing a controls analysis in this scenario, the students should:

‐ Identify risks to the entity associated with the scenario ‐ Determine the controls that ABC employs ‐ Identify the types controls utilized ‐ Identify key controls ‐ Evaluate and justify the evaluation of the controls set

Documents included as part of the scenario are:

‐ ABC Controls Answer Sheet (Spreadsheet) ‐ ABC Controls Scenario ‐ ABC Controls Scenario Answer Key

The students should be given the scenario document and the spreadsheet to record the answers to the questions contained at the end of the scenario document. The answer key is used to assess the answers. Please note that a bonus question is provided that may or may not be used for open discussion on possible recommendations for controls implementation level improvements.

Risk Factor(1)

InternalControl

Identified(2)

Rationale(3)

Type of Control

(4)P, D, or C

Key?(5)

"Key" SelectionSupport

(6)

Level ofAssessment

FI, LI, PI, NI or M(7)

ImplementationRationale

(8)

[email protected] | www.archerenergysolutions.com503-482-9397 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015

Page 1 of 2

[email protected] | www.archerenergysolutions.com 800-805-7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015

Scenario – ABC Company

ABC Electric Company (ABC) is a medium sized integrated electric utility operating in the US with over 2.5 million customers. With approximately 10,000 employees, ABC has an installed net generation capacity of 4200 MW. While ABC is moderate in size when compared to many corporations, as an electric utility, the technology infrastructures are very complex incorporating real-time operator control systems, communication systems supporting the delivery of electricity to the customers. This environment, coupled with the key responsibility to operate and maintain the critical electric grid infrastructures, sets the stage for the need of a robust operating environments.

ABC Electric Company uses state of the art status indication in their control center staffed with well trained, certified operators to avoid the risk of an operator making a mistake.

ABC has an Automatic Voltage Regulator (AVR) status indication so that an alarm alerts its Transmission Operator’s Control Center indicating an AVR status change from Automatic to Manual of a particular generating unit, thus providing notification to the TOP of an AVR status change within 30 minutes as required by Reliability Standard VAR-002. However, the GOP alarm did not update appropriately for a 24-hour period. ABC, as a GOP, self-reported a possible violation of VAR-002-4 R3. Unfortunately, generator operator G.I. Jane was expecting the AVR status to be updated and it was not.

G.I. Jane, as the GOP, should have been aware that the AVR was not changing since it often changed during that particular season. This fact is covered in ABC’s Operator Training material. Further investigation into G.I. Jane’s training record revealed that she missed this training. Somehow ABC missed this during their quarterly review of completed training records to identify individuals that have not completed training by the required deadline. ABC has an automated tracking tool that notifies the individual of scheduled training, reminds individuals to complete the training, and notifies management that training has not taken place prior to the training deadline so management can take appropriate action, but G.I. Jane ignored this reminder.

Furthermore, ABC had a 3rd party rate their capabilities in a Management System to Minimize Human Factor Issues. The 3rd party rated this capability Fully Implemented.

ABC provided the following evidence of its controls:

- GOP training program that includes discussion of the quarterly review process - Screen capture of AVR alarm on SCADA - Procedure identifying the seasonal change in AVR status - Reports from automated tracking tool listing operator training and an example notification

email - Report from 3rd party rating capabilities of Management System to Minimize Human Factors

Page 2 of 2

[email protected] | www.archerenergysolutions.com 800-805-7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015

Questions:

1. What are the risk factors in the scenario? (Fill in column 1) 2. For the identified risk factors, what are the internal controls that you identified in the scenario?

(Fill in answer in column 2). 3. For each identified internal control, briefly describe the rationale for the control. Explain how

the internal control is meant to mitigate risk. (Fill in column 3) 4. For each identified internal control, determine whether the control is preventative (P), detective

(D), or corrective (C). (Fill in column 4) 5. Review each possible control identified and determine whether the control is a key control. (Fill

in columns 5) 6. For each key control identified, include a brief explanation on why you considered the control to

be key. (Fill in column 6). 7. For the family of controls associated with VAR-002-3, determine the level of implementation.

Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially implemented (PI), not implemented (NI), or missing (M). (Fill in column 7).

8. Briefly explain what factors you considered to determine level of implementation. (Fill in column 8).

Bonus Question:

What controls recommendations would you provide to ABC to improve the level of implementation?

Page 1 of 2


Scenario Answer Key

Questions:

1. What are the risk factors in the scenario? (Fill in column 1) The risk factors may be those taken from the NERC guidance or developed ad hoc. Risk factors may include human performance (error), training, voltage stability, and others.

2. For the identified risk factors, what are the internal controls that you identified in the scenario?

(Fill in answer in column 2).

3. For each identified internal control, briefly describe the rationale for the control. Explain how

the internal control is meant to mitigate risk. (Fill in column 3)

4. For each identified internal control, determine whether the control is preventative (P), detective

(D), or corrective (C). (Fill in column 4)

Below is a list of controls from the scenario, the rationale, and the type of control.

Internal Control Identified (2) Rationale (3) Type of Control (4) [P, D, and/or C]

ABC has an alarm generated for AVR status changes

Reduce the likelihood that an AVR status change notification to the TOP is missed. P and D

ABC has periodic system operator training

System operator personnel receive training on current procedures with regards to

voltage regulation status. P

Quarterly reviews of training records are performed

Ensure that system operators are receiving required training before deadlines are met. P, D, and possibly C

Automated tracking tool for scheduled training that

provides notifications and reminders

Ensure that system operators are receiving required training before deadlines are met. P and D

3rd party assessment Determine if existing controls are sufficient C

5. Review each possible control identified and determine whether the control is a key control. (Fill

in columns 5)

6. For each key control identified, include a brief explanation on why you considered the control to

be key. (Fill in column 6).

Internal Control Identified (2)

Key? (5)

"Key" Selection Support (6)

ABC has an alarm generated for AVR status changes

Y Without the presence of the alarm, TOP personnel would need to rely upon verbal notification from

the GOP which is apparently nonexistent.

ABC has periodic system operator training

Y Without system operator training, personnel

would likely not be aware the AVR status alarm and may likely go unnoticed.

Page 2 of 2


Internal Control Identified (2)

Key? (5)

"Key" Selection Support (6)

Quarterly reviews of training records are performed

Y Failure of performing the quarterly review may

result in operators not receiving required training.

Automated tracking tool for scheduled training that provides notifications and reminders

N The tool assists operators and management in the administration of the training, but without it does not raise the likelihood of failure significantly.

3rd party assessment N Controls that are solely corrective cannot be key

controls.

7. For the family of controls associated with VAR‐002‐3, determine the level of implementation.

Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially

implemented (PI), not implemented (NI), or missing (M). (Fill in column 7).

8. Briefly explain what factors you considered to determine level of implementation. (Fill in column

8).

Columns 7 and 8 of the spreadsheet should evaluate the controls as Partially Implemented (PI). ABC has several preventative and detective controls that were documented, namely the training program, the AVR status alarm, and the automated tracking tool. However, the quarterly records review was not documented and there is a lack of internal corrective controls. The 3rd party assessment appeared to be a one‐off control and did not provide anything substantial regarding the processes associated with the AVR status. As added training, the students may be queried on what recommendations they would provide to ABC in order to improve the finding.