View
1
Download
0
Category
Preview:
Citation preview
April 11, 2017
NERC Internal Controls Evaluations
Common Practices, Approaches, and Other Control Ideas
Introductions
Archer Energy Solutions acquires compliance division of Utility System Efficiencies Panelists
o Richard Shiflett
o Bob Dintelman
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com3
Objectives
Identify what internal controls are and why they are needed
Discuss risk thresholds and risk mitigation Discuss the types and characteristics of controls Discuss key controls Discuss a defense-in-depth approach for controls Provide a controls evaluation example for COM-
002-4 R5
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com4
What is an Internal Control?
A process effected by an entity’s oversight body, management, and other personnel that provides reasonable assurance that the objectives of an entity will be achieved:• Operations – Effectiveness and efficiency of
operations• Reporting – Reliability of reporting for internal
and external use• Compliance – Compliance with applicable laws
and regulationsTaken from United States General Accounting Office Standards for Internal Control in the Federal Government
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com5
Benefits of Internal Controls
Why would an entity want quality internal controls?
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com6
Risk and Internal Controls
Identify risks and determine risk acceptance levels or thresholds
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com7
Risk and Internal Controls Internal controls help mitigate risk exposure Risk profiles drive nature and complexity of internal controls
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com8
Nature of Internal Controls
Internal controls can range in nature and complexityo ID cards, fences, locks, Virtual Private Network (VPN), or fireproof fileso Independent verification of processes and deliverableso Authorization of employee time cards
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com9
Basic Types of Controls Preventive
o Aimed at preventing any errors or irregularities from occurring which may have negative effects
o Example: Documented process requiring development and maintenance of training schedule
Detectiveo Designed to discover any errors or irregularities which may have occurredo Example: Documented process requiring periodic review to identify any
required training not completed as scheduled, as well as training not completed per reliability standard requirements.
- Quarterly review of completed training records to identify individuals who have not completed training by the required deadline.
- Documentation and utilization of an event review and root cause analysis process to determine cause and affects surrounding an unwanted event
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com10
Basic Types of Controls
Correctiveo Corrective controls restore the system or process
back to the state prior to a harmful evento Example: An entity may implement its restoration
plan for a computer system from backup tapes after evidence is found that someone has improperly altered the data
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com11
Control Characteristics
Examples of how controls may be characterized:
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com12
Less Assurance More Assurance
Manual Automated
Can be overridden Cannot be overridden
No management oversight Has management oversight
Simple Complex
Performed by junior or inexperienced personnel
Performed by experienced personnel
Single control Multiple or Layered Controls
High level Detail or transactional level
Control tests a sample Control tests entire population
Occurs after the fact Occurs in real time
Key Controls
What is a key control? A key control is a control that, if it fails, means
there is at least a reasonable likelihood that a material error would not be prevented or detected in a timely basis. In other words, a key control is one that is required to provide reasonable assurance that material errors will be prevented or timely detected.
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com13
Key Controls Example An entity has a list of 25 controls that it feels addresses
a risk area identifiedo Five controls occur at the end of the entire process and
confirm that the other 20 controls have done their work and that there are indeed no remaining problems or other errors.
Without the 20 earlier controls there would be a huge number of errors coming through and the five final checks would be little comfort
Focusing on the five final controls, that usually found nothing requiring correction, might be enough for the key control review if the controls are designed and implemented correctly.
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com14
Key Control Factors
Factors that help uncover possible key controls Likely points of failure
How controls rely on each othero Look at interaction between controlso Individual controls may not address the risko Some controls prevent other control failures
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com15
Controls – Defense in Depth
Preventive, Detective, and Corrective Layered controls supporting and enhancing
key controls Control output visibility
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com16
Controls - Defense in Depth
What is the right amount of controls?
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com17
Controls Evaluation ExampleCOM-002-4 R5
Each Balancing Authority, Reliability Coordinator, and Transmission Operator that issues an oral two-party, person-to-person Operating Instruction during an Emergency, excluding written or oral single-party to multiple-party burst Operating Instructions, shall either: • Confirm the receiver’s response if the repeated information
is correct (in accordance with Requirement R6).• Reissue the Operating Instruction if the repeated
information is incorrect or if requested by the receiver, or• Take an alternative action if a response is not received or if
the Operating Instruction was not understood by the receiver.
NERC Standard COM-002-4
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com18
Controls Evaluation ExampleCOM-002-4 R5
Entity provides its communications protocols document Operator consoles have visual reminder to use 3 part
communication The entity has implemented a detailed and technical initial
training program for system operators, and retrain periodically
Operators use 3 part communication for all information exchanges
All operator communications are recorded Shift supervisor regularly listens to the recordings to verify
3 part communication Feedback to operators on improving 3 part communication
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com19
Controls Evaluation ExampleCOM-002-4 R5
Preventiveo Communications protocol documento Operator visual remindero Initial and continual training of operatorso Use of 3-part communications throughout
Detectiveo Review of audio recordings by supervisoro Communications protocol document may have
detective controls present Corrective
o Feedback to operators for improvement
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com20
Controls Evaluation ExampleCOM-002-4 R5
Key controls identifiedo Communications protocol documentationo Review of audio communications by supervisor
Characteristicso A mix of automated and manual controls, but largely
manualo No indication of management oversighto Controls are relatively simple and performed by
experienced personnelo Not clear if supervisor review of audio recordings are
sampled or noto No mention made of communications during Emergencies
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com21
Controls Evaluation ExampleCOM-002-4 R5
Evaluationo Request evidence that controls are presento Grade may range from Partially Implemented to
Largely Implementedo Recommend entity include some form of
management oversight and/or notifications based upon Operating Instructions issued either as part of Emergencies or otherwise.
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com22
Q & A
Please post your questions to the Q&A area of the webinar.
If you would like, you may email us directly at:Richard Shiflett –r.shiflett@archerenergysolutions.comBob Dintelman –b.dintelman@archerenergysolutions.com
4/12/201712042 SE Sunnyside Road #292 Clackamas OR 97015 | 800-805-4711www.archerenergysolutions.com | info@archerenergysolutions.com23
Page 1 of 1
info@archerenergysolutions.com | www.archerenergysolutions.com 800‐805‐7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Scenario Instructions
Prior to conducting this exercise, students should know what the three types of controls are, how to identify key controls, and evaluate controls according to the NERC guidance document. By performing a controls analysis in this scenario, the students should:
‐ Identify risks to the entity associated with the scenario ‐ Determine the controls that ABC employs ‐ Identify the types controls utilized ‐ Identify key controls ‐ Evaluate and justify the evaluation of the controls set
Documents included as part of the scenario are:
‐ ABC Controls Answer Sheet (Spreadsheet) ‐ ABC Controls Scenario ‐ ABC Controls Scenario Answer Key
The students should be given the scenario document and the spreadsheet to record the answers to the questions contained at the end of the scenario document. The answer key is used to assess the answers. Please note that a bonus question is provided that may or may not be used for open discussion on possible recommendations for controls implementation level improvements.
Risk Factor(1)
InternalControl
Identified(2)
Rationale(3)
Type of Control
(4)P, D, or C
Key?(5)
"Key" SelectionSupport
(6)
Level ofAssessment
FI, LI, PI, NI or M(7)
ImplementationRationale
(8)
info@archerenergysolutions.com | www.archerenergysolutions.com503-482-9397 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Page 1 of 2
info@archerenergysolutions.com | www.archerenergysolutions.com 800-805-7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Scenario – ABC Company
ABC Electric Company (ABC) is a medium sized integrated electric utility operating in the US with over 2.5 million customers. With approximately 10,000 employees, ABC has an installed net generation capacity of 4200 MW. While ABC is moderate in size when compared to many corporations, as an electric utility, the technology infrastructures are very complex incorporating real-time operator control systems, communication systems supporting the delivery of electricity to the customers. This environment, coupled with the key responsibility to operate and maintain the critical electric grid infrastructures, sets the stage for the need of a robust operating environments.
ABC Electric Company uses state of the art status indication in their control center staffed with well trained, certified operators to avoid the risk of an operator making a mistake.
ABC has an Automatic Voltage Regulator (AVR) status indication so that an alarm alerts its Transmission Operator’s Control Center indicating an AVR status change from Automatic to Manual of a particular generating unit, thus providing notification to the TOP of an AVR status change within 30 minutes as required by Reliability Standard VAR-002. However, the GOP alarm did not update appropriately for a 24-hour period. ABC, as a GOP, self-reported a possible violation of VAR-002-4 R3. Unfortunately, generator operator G.I. Jane was expecting the AVR status to be updated and it was not.
G.I. Jane, as the GOP, should have been aware that the AVR was not changing since it often changed during that particular season. This fact is covered in ABC’s Operator Training material. Further investigation into G.I. Jane’s training record revealed that she missed this training. Somehow ABC missed this during their quarterly review of completed training records to identify individuals that have not completed training by the required deadline. ABC has an automated tracking tool that notifies the individual of scheduled training, reminds individuals to complete the training, and notifies management that training has not taken place prior to the training deadline so management can take appropriate action, but G.I. Jane ignored this reminder.
Furthermore, ABC had a 3rd party rate their capabilities in a Management System to Minimize Human Factor Issues. The 3rd party rated this capability Fully Implemented.
ABC provided the following evidence of its controls:
- GOP training program that includes discussion of the quarterly review process - Screen capture of AVR alarm on SCADA - Procedure identifying the seasonal change in AVR status - Reports from automated tracking tool listing operator training and an example notification
email - Report from 3rd party rating capabilities of Management System to Minimize Human Factors
Page 2 of 2
info@archerenergysolutions.com | www.archerenergysolutions.com 800-805-7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Questions:
1. What are the risk factors in the scenario? (Fill in column 1) 2. For the identified risk factors, what are the internal controls that you identified in the scenario?
(Fill in answer in column 2). 3. For each identified internal control, briefly describe the rationale for the control. Explain how
the internal control is meant to mitigate risk. (Fill in column 3) 4. For each identified internal control, determine whether the control is preventative (P), detective
(D), or corrective (C). (Fill in column 4) 5. Review each possible control identified and determine whether the control is a key control. (Fill
in columns 5) 6. For each key control identified, include a brief explanation on why you considered the control to
be key. (Fill in column 6). 7. For the family of controls associated with VAR-002-3, determine the level of implementation.
Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially implemented (PI), not implemented (NI), or missing (M). (Fill in column 7).
8. Briefly explain what factors you considered to determine level of implementation. (Fill in column 8).
Bonus Question:
What controls recommendations would you provide to ABC to improve the level of implementation?
Page 1 of 2
info@archerenergysolutions.com | www.archerenergysolutions.com 800‐805‐7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Scenario Answer Key
Questions:
1. What are the risk factors in the scenario? (Fill in column 1) The risk factors may be those taken from the NERC guidance or developed ad hoc. Risk factors may include human performance (error), training, voltage stability, and others.
2. For the identified risk factors, what are the internal controls that you identified in the scenario?
(Fill in answer in column 2).
3. For each identified internal control, briefly describe the rationale for the control. Explain how
the internal control is meant to mitigate risk. (Fill in column 3)
4. For each identified internal control, determine whether the control is preventative (P), detective
(D), or corrective (C). (Fill in column 4)
Below is a list of controls from the scenario, the rationale, and the type of control.
Internal Control Identified (2) Rationale (3) Type of Control (4) [P, D, and/or C]
ABC has an alarm generated for AVR status changes
Reduce the likelihood that an AVR status change notification to the TOP is missed. P and D
ABC has periodic system operator training
System operator personnel receive training on current procedures with regards to
voltage regulation status. P
Quarterly reviews of training records are performed
Ensure that system operators are receiving required training before deadlines are met. P, D, and possibly C
Automated tracking tool for scheduled training that
provides notifications and reminders
Ensure that system operators are receiving required training before deadlines are met. P and D
3rd party assessment Determine if existing controls are sufficient C
5. Review each possible control identified and determine whether the control is a key control. (Fill
in columns 5)
6. For each key control identified, include a brief explanation on why you considered the control to
be key. (Fill in column 6).
Internal Control Identified (2)
Key? (5)
"Key" Selection Support (6)
ABC has an alarm generated for AVR status changes
Y Without the presence of the alarm, TOP personnel would need to rely upon verbal notification from
the GOP which is apparently nonexistent.
ABC has periodic system operator training
Y Without system operator training, personnel
would likely not be aware the AVR status alarm and may likely go unnoticed.
Page 2 of 2
info@archerenergysolutions.com | www.archerenergysolutions.com 800‐805‐7411 | 12042 SE Sunnyside Road Suite 292 Clackamas OR 97015
Internal Control Identified (2)
Key? (5)
"Key" Selection Support (6)
Quarterly reviews of training records are performed
Y Failure of performing the quarterly review may
result in operators not receiving required training.
Automated tracking tool for scheduled training that provides notifications and reminders
N The tool assists operators and management in the administration of the training, but without it does not raise the likelihood of failure significantly.
3rd party assessment N Controls that are solely corrective cannot be key
controls.
7. For the family of controls associated with VAR‐002‐3, determine the level of implementation.
Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially
implemented (PI), not implemented (NI), or missing (M). (Fill in column 7).
8. Briefly explain what factors you considered to determine level of implementation. (Fill in column
8).
Columns 7 and 8 of the spreadsheet should evaluate the controls as Partially Implemented (PI). ABC has several preventative and detective controls that were documented, namely the training program, the AVR status alarm, and the automated tracking tool. However, the quarterly records review was not documented and there is a lack of internal corrective controls. The 3rd party assessment appeared to be a one‐off control and did not provide anything substantial regarding the processes associated with the AVR status. As added training, the students may be queried on what recommendations they would provide to ABC in order to improve the finding.
Recommended