88
© 2016 High Water Advisors Confidential. Do not distribute. Data Analytics for IT Audit Data Analytics Overview ISACA, North Texas Chapter

Data Analytics for IT Audit - ISACA North Texas All Sessions - IT... · Data Analytics for IT Audit Data Analytics Overview ... ‒ 3 years with ACL Professional Services ... Fuzzy

Embed Size (px)

Citation preview

© 2016 High Water AdvisorsConfidential. Do not distribute.

Data Analytics for IT Audit

Data Analytics Overview

ISACA, North Texas Chapter

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 1

About Me

20+ years data analytics experience

‒ Current: Client Solutions Director, High Water Advisors

‒ 2 years developing SAS programs analyzing occupation

information for US Department of Labors’ O*NET system

‒ 8 years with Nortel Networks developing simulations,

using predictive analytics, & managing a DA function

‒ 5 years with federal contractor, RTI International, leading

and performing operations and IT audits

‒ 3 years with ACL Professional Services

» Supported large public & financial sector clients

» SME for SOX-related test automation

Instructor for “Successful Data Analytics” course by the MIS

Training Institute

Local instructor for ISACA CISA exam prep

Jim Tarantino ACDA, CISA, CRISC

[email protected]

+1 855 RISK WATCH

www.linkedin.com/in/jimtarantino

@JimTarantino

www. highwateradvisors.com

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 2

Most IA departments have barely scratched the surface on achieving

the benefits of audit analytics

The challenge is largely a change-management issue, not a

technology issue

‒ Initially, successful analytics are 90% design and 10% technology

» However, technology does increase in importance as the need for

collaboration and sustainability increases

‒ The challenge for most auditors is learning to think differently about audit testing

‒ Poor processes result in the use of analytics stagnating in many organizations

My Beliefs

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 3

What is “data analytics”?

Using technology to enable various stages of data analysis

Current landscape of common technologies used for audit analytics

Business case for data analytics and their application to IT audit

How data analytics can be applied to phases of the IT audit process,

from IT risk assessment, planning, fieldwork, reporting, and follow-up

Key Topics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 4

Simply put:

‒ Breaking relevant problems and questions down into manageable sizes

‒ Using factual evidence to solve problems and deduce answers to questions

An analysis has four key elements:

1. Data / information

2. Logical reasoning / argument

3. Finding / results

4. Lesson / conclusion

Data Analysis

It’s elementary! Auditors

are natural analysts.

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 5

Specially-designed, auditor-friendly, data analysis routines and visualizations, enabling

internal auditors to gather and analyze raw digital evidence from the organization’s

information systems in order to conduct more comprehensive, objective, repeatable

and efficient assessments of risks and controls

Audit Analytics

Digital Evidence + Algorithms + Technology(set of operations)(business data) (understood by computers)

Dear Watson! With this

contraption, auditors can:• Quickly view lots of digital evidence

• Find interesting patterns in the data

• Relate multiple information sources

• Automate their thinking/reasoning

• Make compelling, evidence-based

conclusions

• Route key findings to those who can

do something about it

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 6

Based on… Categories/Types Description

Degree of

standardization &

automation

• Ad hoc

• Repeatable

• Continuous

• Explorative and investigative in nature

• Periodic analysis of processes, often from multiple data sources

• “Always on” — automated, scheduled auditing & monitoring of key processes

Time-orientation of the

question

• Descriptive

• Diagnostic

• Predictive

• Prescriptive

• What happened?

• Why did it happen?

• What will likely happen next?

• What’s the best course of action?

Content focus• Risk (KRIs)

• Performance (KPIs)

• Focused on the likelihood, impact and/or control status of a risk

• Focused on business or operational performance

Data type• Quantitative

• Qualitative

• Analysis of numeric values (e.g., aging, Benford’s Law, stratification)

• Analysis of text values (e.g., keyword matching, sentiment analysis)

Role in the analysis

process

• Data Preparation

• Data Profiling

• Utility

• Outlier

• Exception

• Planned routines to clean and standardize data

• Routines to provide quick visual summaries of data

• Helper routines that are not audit specific (e.g., keyword match)

• Routines that look for extreme cases relative to a group

• Routines that detect violations of a business rule

‘Types’ of Audit Analytics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 7

We deal with big populations (hosts, assets, incidents, ID’s)

Rate of flux in infrastructure, applications, users, regulations, threats

Variety of data sources creates issues

‒ Require special tools and permissions (e.g., Active Directory)

‒ Unstructured data (e.g., event logs, .ini files)

Scanning/testing tools

‒ Create verbose output

‒ Output suited for manual viewing (xml)

‒ Often built for point-in-time (not continuous scans)

Audit Analytics for IT

© 2015 High Water AdvisorsConfidential. Do not distribute without permission. Page: 8

“…it is nearly impossible to

conduct an effective audit

without using technology”

GTAG 16, Page 6

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 9

Enabling an Analysis Methodology with Technology

Locate, Understand

& Obtain Source Data

Validate, Transform

& Prepare Data

Interpret

& Validate Results

Report Findings

& Conclusions

Run Modeling

Procedures & Routines

Define Problem/Question

& Plan Approach

What do you want to know? How can you answer the question?

Enabled by a central database of prior results and insights

What data is needed and in what format?

Enabled by systematic access to mapped data sources

Are source data complete, accurate & ready to analyze?

Enabled by standard validation & data prep routines

What routines, models & visualizations best fit our objective?

Enabled by standard commands, routines, and datasets

Are the results valid and what do they indicate?

Enabled by standard export to validate & verify

Who needs to know the results & by when?

Enabled by standardized, repeatable report production

FR

AM

ES

OLV

EA

CT

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 10

Example: Active Directory

Locate, Understand

& Obtain Source Data

Validate, Transform

& Prepare Data

Interpret

& Validate Results

Report Findings

& Conclusions

Run Modeling

Procedures & Routines

Define Problem/Question

& Plan Approach

Do we have contractors with account expirations later than contract close out? Any history of this issue?

Use PowerShell, ADFind, vbScript to retrieve AD user listing.

Pull contracts/subcontracts from Oracle Financials.

AD: Isolate “user” object class. Convert long dates to YYYYMMDD. Filter contracts database to active contracts. Fuzzy match AD name and contractor name on contract.

Filter for contractors where ‘accountExpires’ > policy date

Stratify by date

Profile by contract, by business unit

Verify results with SME.

Prepare exception spreadsheet and email to IT.

FR

AM

ES

OLV

EA

CT

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 11

Example: Help Desk

Locate, Understand

& Obtain Source Data

Validate, Transform

& Prepare Data

Interpret

& Validate Results

Report Findings

& Conclusions

Run Modeling

Procedures & Routines

Define Problem/Question

& Plan Approach

Do we have an unacceptable rate of aging and reissued high-severity tickets?

Extract ticket header/detail from service now.

Join ticket head/detail and format dates to YYYYMMDD.

Isolate high severity tickets.

Age tickets and flag those older than 5 days.

Profile by user, by help desk personnel, by location

Verify results with SME.

Prepare exception spreadsheet and email to IT.

FR

AM

ES

OLV

EA

CT

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 12

Key Activities on the ‘Data Value Map’

Prepare

Produce

Task / Event / Activity

Measure / Observe

Record

Clone /

Replicate

Backup /

ArchiveWarehouse /

Repository

Post to

another app

Discover

Locate

Understand

Map

Persist (Store)

Routines

ResultsSource

Expose

Alerts / Notifications

Trends

Integrated Analysis

Content for BI / GRC

What people, process, and technology enablers are in place to (1) avoid data value

loss and (2) ensure sustainable value creation using data analytics?

Acquire

Indirect via Extract

Direct via Connector

Integrate

Blend

Templates

Analyze

Model Problem

Analyze / Visualize

Define Problem

Interpret / Validate

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 13

TRADITIONAL BI

Audit Analytic Technology Landscape

VISUALIZATION

TeamMate Analytics

ANALYTICS

MS OFFICE EXCEL ADD-INS CLIENT/SERVER CAATS ADVANCED DA WORKFLOW

DATA MGT / ETL MONITORING UTILITY SCRIPTING

TIGHT ERP

INTEGRATION

LOOSE ERP

INTEGRATION

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 14

Haven't We Seen This All Before?

Decision Support Systems

Expert Systems

Data Warehouses / Marts

Business Intelligence

Business Analytics

Data Mining

Data Science

Digital & networked processes

IoT and (“hyper-”) quantified self

Self-service and social mindset

“Open” software, data, & knowledge

Commodity hardware & virtualization

DA evolved for desktop & cloud use

‒ Can handle 4 V’s: Volume, Variety, Velocity, Veracity

‒ Automation, scheduling, collaboration, visualization

Prior Practices Current Tech Trends

We’re processing a greater amount of integrated, timely, and relevant digital evidence,

leading to richer insights, better decisions, and more effective actions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 15

How Risk Lives in Your Systems

Configurable system controls may be:• Disabled (past, present, future)• Misconfigured or left to default settings• Outdated/obsolete

Internal system processes may override expected controls, for example:• Sundry invoices bypass 3-way match• Auto-PO or auto-goods receipt generation• Credit and replace practices override pricing

Failed or faulty processing routines may impact data integrity and availability• Gaps in time series• Mishandled NULL values• Changes in scale/units

• Incomplete or erroneous legacy data• Sample or test data left in the system• Faulty join and aggregation logic• Hardware/software constraints

Realistic transaction

Typos, workarounds, duplicates, unstandardized text, truncated/censored entries

Unauthorized C/R/U/D as elevated user

Managerial override

‘Dirty data’ from downstream system

Buffer overflows, transmission problems

Exception report ignored,

misinterpreted, or

unreliable

System warnings are soft, ignored, or misinterpreted

Enterprise Application

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 16

Standard Reports Often Go Only So Far

Example: Monitoring customer credit changes

Standard reports often provide information needed to effectively monitor processes and

risk, but they may not provide it in a way that easily highlight abnormal conditions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 17

Challenges w/ Traditional Audit Process

Sampling

Rotational/cyclical auditing

Audit process and data silos

Auditor turnover

Keeping abreast of process/organizational changes

New approaches must be adopted to evaluate

risks and controls in a sustainable way

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 18

Add Value

Better insight into areas of management concern (data profiling / trending)

Increased ability to quantify issues

Streamlined data correlation (spanning data and process silos)

Increased responsiveness to management requests

Increase Efficiency

Increased cyclical/rotational audit efficiency (often after year 1)

Decrease cycle time to get through audit universe

Increased breadth of audit coverage (more auditable entities)

Reduced travel (data accessed from anywhere)Standardized, repeatable procedures (less reliance on the individual)Decreased reliance on IT for data acquisitionIncreased confidence & efficiency in verifying data accuracy/completeness

Reduce Risk

•Increased responsiveness to key risks (current and emerging)•Identify problems closer to the first occurrence (more real-time detection)•100% transaction review (providing greater depth of coverage & assurance)•Stratifying the population, thereby honing risk assessment•Keep abreast of system, process and organizational changes

Benefits of Audit Analytics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 19

DA Recognized as a Standard Practice

IIA Quality Assessment

‒ Conducted by an external independent team of qualified audit professionals well-versed in the Standards, assessment methodology, and successful internal audit practices

‒ Report audit activity’s conformance or nonconformance with the Standards and any recommendations for improvement opportunities

‒ More commonly flagging the lack of a robust data analytics program as a reportable concern

Data analysis within audit has finally moved from being

a leading practice to a standard practice

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 20

The Typical Audit Process

Risk Assessment

Audit Planning

Fieldwork / Testing

ReportingFollow-up & Monitoring

Audit Planning

Fieldwork / Testing

ReportingFollow-up & Monitoring

Audit Planning

Fieldwork / Testing

ReportingFollow-up & Monitoring

Audit Planning

Fieldwork / Testing

ReportingFollow-up & Monitoring

** darker color = greater DA usage

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 21

Data Analysis and Audit Process Stages

12 3 45

Typical implementation order of audit analytics over time

• Increasing involvement of other lines of defense

• Where is your organization in this process?

• Could IA benefit from doing more at a specific stage?

Risk Assessment

Audit Planning

Fieldwork & Testing

ReportingFollow-up

and Monitoring

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 22

The “Data-Enabled” Audit Process

Risk Assessment

Audit Planning

Fieldwork & Testing

ReportingFollow-up

and Monitoring

Identify data-driven

risk indicators

(current and

emerging)

Continuous Risk

Assessment

Aggregated across

all auditable

entities

Audit entity-

specific

Analysis of survey

and other data-

gathering activities

for trends or

correlation

Focused

discussions with

management about

anomalous audit

entities

Review previous

findings / results

Team brainstorm &

prioritize potential

data analytics

Data profiling of

relevant data and

meta-data

Assess available

data / plan additional

data acquisition

Data trending and

ratio analysis

Re-validate risk

assessment priorities

and refine audit plan

Develop data-driven

audit program and

test routines

Record future

analytic ideas

Independently

access audit-

relevant data

Data exploration

100% population

testing for

exceptions /

attributes

Automated, risk-

based sampling

Ad-hoc data

analysis

Queries / routines

for batch analysis

Store results and

analytics centrally

for other auditors

and future re-use

Visual dashboards,

scorecards, reports

& storyboards with

drill-down capabilities

Generate alerts &

notifications based

on business rules

Data-driven risk

quantification,

including where risk

initially identified

through manual

processes

Evaluate need for

continuous audit

procedures

Assess potential

future analytic

improvements

Provide management

with analytic

prototypes

Automated re-

testing of

resolution

Quality

Timing

Visual issue

trending

“Entity

Correlation”

Employees /

Vendors

Locations /

Departments

How are data analytics being applied across the IT audit process?

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 23

Audit Analytics mixes digital evidence, algorithms, & technology

Use technology and algorithms to enable the stages of data analysis

Vast market of technologies that can be used for audit analytics

Traditional audit challenges make it difficult to address risky systems

Analytics can help IT auditors Data analytics across various phases of

the IT audit process

Concluding Thoughts

© 2016 High Water AdvisorsConfidential. Do not distribute.

Data Analytics for IT Audit

Potential & Common Tests for IT Audit

ISACA, North Texas Chapter

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 25

Logical and physical access

Configuration and change management

Data integrity and master data management

SIEM and cybersecurity

IT project management and SDLC

Key Topics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 26

Question: Are functionality/permissions/rights provision to

appropriate users in a timely, efficient manner?

Challenge: Usually need additional multiple data pieces to clarify if

legitimate employees (Employee Master), with access (User Listing),

have the correct access (Roles/Entitlements)

Approach:

Logical Access – Account Provisioning

Application Access

HR/Employee Masterfile

Unmatched Right/Secondary:

(Primary key is blank) Employee, no

application access ID, possible

phantom employee

Matched: (Primary key = Secondary key) Application Access ID, and Employee record are both present,

combine with the Profiles file to see if access is appropriate for the department the employee works within

Unmatched Left/Primary:

(Secondary key is blank) Application

Access ID, no Employee record,

possible phantom ID

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 27

Question: Are functionality/permissions/rights terminated in a timely,

efficient manner?

Challenge: Obtaining an HR dataset with chronological hire,

transfer, promotion, leave, and termination information

Approach:

Logical Access – Account Deactivation/Termination

Application Access

HR/Employee Terminations

Matched: (Primary key = Secondary key) Application Access ID, and Employee record are both present,

combine with other information like date, manager, job title/code to assess risk

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 28

Question: Do we have duty conflicts? Have they been exploited?

Challenge: SoD projects tend to languish trying to figure where to

start remediation.

Approach:

‒ Recurse through roles table to see which roles coexist.

‒ Correlate duty conflict table to select those duties where conflicts exist

Logical Access – Segregation of Duties

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 29

Question: Do all the employees with the same job title have the

same permissions?

Challenge: This type of analysis typically requires a script that can

iterate of user and possible permissions within an application.

Approach:

‒ Obtain each users permissions and job function

‒ Cluster users by function

‒ Identify users with permissions different than their peers

Logical Access – Reasonable Permissions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 30

Questions: Are passwords exposed in clear text? Are passwords

reset timely? Are users timed out?

Challenge: Obtaining an HR dataset with chronological hire,

transfer, promotion, leave, and termination information

Approach:

‒ Directory Command – password.txt files

‒ Active Directory

» Policy Compliance: (filters or computed fields)

o cutoffdate - pwdLastSet > policy pwd life

o cutoffdate – lastLogon (dormant accounts)

» Couldn’t happen:

o accountCreated < date of hire (above x days)

o lastLogoff < lastLogon (or they are still logged in)

Logical Access – Passwords & Account Management

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 31

Question: Are facilities being access by unauthorized persons?

Challenge: Many doors, many people, many access transactions

Approach(es):

‒ Ingress/Egress logs correlated with valid badge holder

‒ Sequence analysis: Interior door access missing exterior door access

» Profile incidents by building, person

Physical Access – Physical Security

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 32

Question: Are configurations applied correctly and consistently

across information assets?

Challenge: Direct access to configuration data. Sometimes in

unstructured files.

Approach(es):

‒ Webserver XML Configuration, parese for key values and compare to standard

‒ Extract SAP/Oracle config tables and correlate settings to external standard

‒ Validate that the settting cross affiliates and company codes

Configuration Baselining

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 33

Questions: Are change to systems authorized, tested, implemented,

and communicated by the appropriate person in a timely manner?

Challenge: Detecting changes and correlating them with something

else that should have changed.

Approach:

‒ Obtain directory listings (before & after)

‒ Correlate to something else that should have changed

» System Development or Project Management records

» Change management records

» Testing (QA) records

» Incident/problem management records

» Help Desk tickets (all of the above could/should be here)

‒ Who made the change and when?

Change Management

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 34

Detecting file changes on Windows

Windows PowerShell:‒ Change to the directory containing the scope of the files:

cd c:\dir\with\in-scope\files

‒ Files last written to (changed) after a certain date:

get-childitem –recurse | where-object {$_.lastwritetime -gt “5/1/2012”} | out-

file “changedfileslist.txt” –Encoding ASCII –width 240

‒ To find all the files in the current and sub directories written to

(changed) in the last 15 days:$DateToCompare = (Get-date).AddDays(-15)

Get-Childitem –recurse | where-object {$_.lastwritetime –gt

$DateToCompare} | out-file “changedfileslist.txt” –Encoding ASCII

–width 240

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 35

Detecting file changes on Linux

Linux files changed within 30 days:find /directory –mtime -30 –ls

(modified time) (detailed list)

add: > changedfiles.txt to redirect to a file

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 36

Detecting Changes – General Model

Most Recent Listing

Prior Listing

Additions

On both listings, further analysis

needed of date modified - Changes

Deletions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 37

Question: Have important items change on a host? Has a critical

event happened (or not) that may leave the host vulnerable?

Challenge: Obtaining log files. Parsing event information.

Approach(es):

‒ New events since last run

‒ Missing events

‒ Classify events

‒ Keyword search

‒ Sample events, then:

» Correlate with help desk tickets

» Correlate with change control documentation

System Event Logs

Current Log

Summed Prior Log History

New Event

Recurring

event

Missed Event

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 38

Question: have critical events occurred and not been reported?

Challenge: Mining verbose logs

Approach(es):

‒ Import event logs/Query event DB

‒ Parse event strings

‒ Classify by event, asset, date, text tag

System Event Logs

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 39

Question: Am I exposing configuration and network settings of hosts

that can be found externally? Which vulnerabilities are not getting

addressed in a timely manner?

Challenge: Budget friendly tools don’t scale well over time or across

multiple hosts.

Approach:

‒ Create 2 scans per host (prior / current)

‒ For every host, join prior and current to identify new vulnerabilities

InfoSec - Configuration & Vulnerability Scans

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 40

Nessus

The Host Properties table has data stacked , 7 rows (at most) per host

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 41

The machine, date header values can be captured

The consistent 3 values in an item, “issue” . “score”, “result” can be

captured

The “detail” section of an issue is variable, class challenge – your

homework assignment, figure this out

Microsoft Baseline Security Scanner

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 42

Question: Have important items change on a host? Has a critical

event happened (or not) that may leave the host vulnerable?

Challenge: Obtaining log files. Parsing event information.

Approach(es):

‒ Kismet (free), will poll access points whether or not they are broadcasting their

SSID

‒ Generates a summary report of what was found and packet level detail, the

summary tags are “wireless-network”

» Correlate valid AP listing

» Identify rogue Aps

» Identify missing APs

Wireless network scans

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 43

Audit and monitor timely maintenance of critical systems (Network,

OS, App, or Data)

Determine whether planned maintenance schedules aligned with

historical incident/downtime trends

Deploy advanced models to predict optimal maintenance schedules

(predictive maintenance)

System Maintenance

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 44

DA Completeness - Reconcile source data in DB with independent

record(s)

Data Quality – data elements critical in line with expected data

conditions (standardized formats)

Referential Integrity - Verify that relationships between core data

(transactions) accurately and map completely to referential data

Database

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 45

SLA metrics recalculation

Help desk data quality

System re-performance

Data conversion verification

Process-to-Application Analytics

© 2016 High Water AdvisorsConfidential. Do not distribute.

Data Analytics for IT Audit

Making analytics sustainable

ISACA, North Texas Chapter

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 47

Data analytic approaches for IT audit planning and risk assessment

Automation and moving towards continuous IT risk assessment and

auditing

Tips for making IT data analytics a sustainable practice

Best practices for data analytics program implementation within IT

audit

Key Topics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 48

IT Audit Planning

Consolidate & summarize daily

“high” report

Consolidate IT help desk tickets

‒ Profile high-risk events by

function (e.g., ops, DB,

network)

Query event database items

‒ Profile high risk events by

asset, user, time, location

Obtain “Project Priorities” list

Review versions of key plans

‒ Priority differences (add/drop)

‒ Resource drift

‒ Scope creep

‒ Project slippage

Compare current & prior lists

‒ New, dropped, changed

projects

ApplicationInfrastructure

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 49

Data Profiling

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 50

Visualizations != Analysis

‒ Visualizations are information that has to be interpreted and given context

Examples: Graphs, Scorecards, Dashboards, Storyboards

Risk Assessment: Visualization/Storyboards

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 51

Uptime: 92%

Performance/response time: 80%

Data loss: 56%

Number of open issues: 52%

Average time to fix: 51%

Security breaches: 49%

Mean time between failures: 38%

Types of Indicators

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 52

Visual Display of Risk Information

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 53

Tech: ACL GRC / Results Manager

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 54

Tech: IDEA – CaseWare Monitor

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 55

Tech: SAP GRC / Process Control

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 56

Tech: SAS Enterprise GRC

Sustainable Analytics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 58

Audit Analytics are Sophisticated AND Sustainable

Sophisticated

• Analysis is linked to business objective?

• Leverage larger data volumes and variety?

• Leverage state-of-the-art technologies?

• The analysis uses a novel approach?

• Calculations, routines, and/or number of

variables considered are beyond traditional

analytical approaches?

Sustainable

• Am I re-performing the analysis?

• Can I easily repeat the analytic steps?

• Can I perform the analysis independent of IT?

• Does the analytic minimize ‘Time-to-Support’?

• Does the analytic minimize ‘Time-to-Use’?

• Can I scale the analytic?

• Can I share the analytic?

• Can the analytic be self-service?Low

High

Low High

Sustainability

S

op

his

tica

tio

n You are here

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 59

Document Standards & Guidelines

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 60

Object Examples

Collections

Folders

FY2015 SOX Testing

AP – Account Payable

Analytics AP01_detectDuplicateVendor

ACL Projects SOX_AccountsPayableTests

Script

Subscripts

AP01_duplicateNameCheck

AP01_duplicateNameCheck2

Tables

Views

Indexes

T_temp P_prepared S_source AP01_duplicateName_D

Default_view Essential_Fields Approvals Key_Dates

i_BSAK

Parameters

Variables

Filters

Computed fields

p_cUserName

v_dStateDate

f_isValueGT1000

c_nInvoiceAmount

Output files ctl_controlReport.xlsx AP01_duplicateName.xlsx

Naming Conventions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 61

Start projects from a template

Include reusable scripts in the

template

Complexity can be added or

removed depending on the nature of

the project

Standard Project

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 62

Standard High Value Analytics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 63

Shared data files (source, results)

Shared scripts / Analytic apps

Shared resource files (e.g., keyword lists)

Analytic documentation

Job aids & Checklists

Analytic ideas (new ideas, enhancements)

Application / DB inventory (w/ points of contact)

Repositories

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 64

QA

Developer vs. Independent QA

Non-technical QA

Technical QA

Capturing test results and updating analytics

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 65

Goal to ensure that analytic:

‒ Works as intended

‒ Reports intended results

‒ Does not miss any desired results

Key decisions:

‒ Who will be testing?

‒ What data will be used?

‒ Positive testing? Negative testing? Both?

‒ Stress testing included?

Testing

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 66

Largely driven by your industry and internal policies

Data to consider

‒ Source data

‒ Temporary files created during analytic

‒ Results

May also need to consider:

‒ Data in transit

‒ Data in memory

Data Security

DA Program Implementation

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 68

IA See Benefits, but Lacks Planning

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 69

Data Analytics Program

Formal, sponsored organizational effort to use data, models,

and fact-based management to evaluate the state of the

business and drive decisions and actions

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 70

Data Analytics Maturity Model

• Increased assurance & value-added services at higher levels

• Increased maturity and sophisticated DA typically requires

increased time and budget for:

Initial

Repeatable

Defined

Managed

Optimizing

► No formal data analytics

approach, procedures, or

methodology

► Performed occasionally, ad-

hoc, and often unplanned

► Dependent on skills of

limited number of staff

► Purpose-built tools are not

readily available

► Often focused on quick

development over ease of

repeatability

► Recognized as value-add

to the audit

► Partially integrated into

target audit process

► Often covers both audit

planning and fieldwork

► Structured approach, but

not yet institutionalized

► Tests reused and added

in subsequent audits

► Tools are available, but

not applied consistently or

correctly

► Well-documented and

consistently followed DA

methodology

► Centralized, structured

knowledge management:

Data

Audit tests

Results

Supporting docs

► Collaboration

► Controlled, secure access

to data and routines

► Established data access

protocols

► Suites of tests available

to audit team

► Concurrent, ongoing

auditing of multiple areas

► Structured issue reporting

and tracking

► Formalized process for

changing / improving

analytic routines

► Management monitoring

of own process

► IA assesses management

monitoring activities

► Continuous management

& reduction of false-

positives

► IA conducts continuous

risk assessment

Level 1 Level 2 Level 3 Level 4 Level 5

o Technology

o Data Management

o System Administration &

Operations

o Queries/Models

o Collaboration

o Governance

o Enterprise Integration

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 71

Understand the key drivers for DA

Create a vision of DA use within the audit function

‒ Define short- and long-term goals within context of vision

Anticipate implementation barriers

Create initial project plan

Develop performance metrics & measurement system

Assign responsibilities

Put plan into action

Measure and track progress

Review and refine approach over time

Program Strategy and Approach

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 72

Increase the value and quality of audit findings? Where?

‒ Finding money for bottom-line impact?

‒ Identifying operational inefficiencies?

Reduce the risk of knowledge loss? Improve audit consistency?

Implement continuous auditing? What scope? How soon?

Assurance: Increased coverage? Sleep better at night?

Increased efficiencies within audit? By when?

‒ To increase the number of audits within a year?

‒ To do more comprehensive testing within the same population of audits?

‒ To reduce audit costs?

Offload manual testing and re-focus on more strategic tasks and analysis?

‒ Most time-consuming manual processes?

‒ Testing disliked by auditors?

Key Value Drivers

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 73

Direct and Inspire Through Vision

Ensures everyone working towards same end and helps reduce

unnecessary diversions

Drives consistency of decision-making

‒ Short- and longer-term priorities

‒ Investments in resources and technology

“Let’s take a

vacation!”

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 74

Create Confidence Before Accelerating

Set realistic short-term goals

Use all available resources—don’t reinvent the wheel

‒ Public domain

‒ Peer organizations

‒ User groups

‒ Trusted partners

Conduct pilot project(s)

Accelerate only after building experience and internal support

necessary for sustainable growth

‒ Controlled acceleration = proactive, not reactive

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 75

Anticipate Implementation Challenges

Adequate Budget

Technology

Training

Professional services

Effective Support & Direction

Getting management buy-in and understanding of

effort

Poorly-defined analytic scope

Supportive Policy

Competing DA initiatives

Incongruent processes / policies

International issues

Capable and Available Resources

Understanding source data and related business

processes

Strategies for locating, accessing and integrating

disparate systems and formats

Availability of internal expertise to evaluate

results/anomalies

Outsourced IT

Dealing with false positives

Data Access & Quality

Data preparation time to resolve issues with data

quality

Manually-maintained data and manual controls

External and cloud-based data

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 76

DA Capability Levels – Management Tasks

Level 1: Initial

Level 2: Repeatable

Level 3: Defined

Level 4: Managed

Level 5: Optimizing

Early-Maturity

• Set clear vision & direction

• Consider IA role in continuous

monitoring

• Encourage experimentation/creativity

• Address ‘prior budget’ challenge

• Create incentives, or dis-incentives,

depending on your style

• Prioritize data access & preparation first

• Lynchpin for analytics and harder to

outsource efficiently

Mid-Maturity

• Champion the vision

• Communicate KPIs (current & desired)

• Eliminate data access as a hurdle

• Negotiate access with the CIO, not every

DBA

• Stress process over individual preference

• Establish effective QA process

• Set foundation for working smarter, not

harder

• Knowledge management and

collaboration are critical at this stage

Late Maturity

• Champion and lead the change management

required for Continuous Risk Assessment

• Recognize and support the additional

investment required to move from ad-hoc or

repetitive to continuous

• Scheduled data feeds

• Script logic adjustments

• Managing (and reducing) false-positives

• Facilitate breakdown of organizational silos

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 77

Practice Effective Change Management

There are multiple actors involved‒ Audit team, Internal Stakeholders, and External Stakeholders

‒ Identify and make plans to proactively address influencers / detractors

Change will occur in many areas of the department‒ New roles and responsibilities will required new skills, knowledge and behaviors

‒ Generating new types of deliverables and integrating new value to the business

‒ Introduction of new technologies and dependencies on upstream and downstream IT activity

‒ Revisions to existing audit processes, procedures, & schedules

» Add time to audit planning for audits where data analytics have not been attempted before

» Team brainstorming during planning instead of relying on the AIC

» Adding analytics steps to audit planning and audit QA checklists

» Documenting when DA was used/not used and why

» Capturing DA ideas in repository

» Data profiling and using reports/results in management interviews

Consider incentives to help get through the pain of doing something new

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 78

Track Relevant KPIs / Metrics

Value

# or % of audit findings resulting from

analytics (both low-level & audit-committee-

level)

Hours saved (audit hours, IT hours)

Coverage

# or % of transactions reviewed as part of the

audit (vs. what was covered by a traditional

sample)

# or % of key systems where direct data

access has been established

Number of key business processes with at

least X analytics

Efficiency

Time to complete specific audit steps

Cost of Audit (incl. travel…useful when

analytics may reduce travel time)

Collaboration (within audit team)

# Analytic requests / # Analytic submissions /

# Analytic self-serve downloads

# of Auditors trained and/or certified on data

analytic technology

# of auditors certified as data analysts

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 79

Technology Planning & Maintenance

Establish a solid data foundation

Plan the appropriate mix of Excel, CAATs, Visualization technology

licenses across the team

Ensure technologies support activities across the various stage of

data analysis

The need for technology increases as you move towards continuous

operations (the required skills change, too!)

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 80

Expertise to Embed Within the Function

Data access & modeling

‒ Access, prepare, and make data

available

Data analysis

‒ Perform advanced analysis and create

analytics and results for sharing

Data literacy / consumption

‒ Review and interpret results

‒ Perform simple analysis independently

DA Management

‒ Oversight of analytic-enabled audit

activities

» Guide analytics planning

» Monitoring progress and setting

boundaries

» Compliance with DA guidelines &

standards

» Ensuring valid results

» “Future proofing” the work

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 81

Training and Development

Identify gaps in skills, knowledge and understanding

Personal and team learning goals and metrics

Progress review and coaching/mentoring

Relevant assignments and challenges

Resource availability

‒ Reference material

‒ Usage examples

‒ Guidance and support

Suitable learning strategies

‒ Just-in-Time and blended learning approaches

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 82

Staffing and Retention

Map required skills to existing competencies

Develop staffing, recruiting, and interviewing plans

Identify critical long-term skills and competencies

Key strategies to consider

‒ Job pathing through increasingly technical roles

‒ Employee engagement / retention

‒ Staff coverage & backups

‒ Succession planning

‒ Documentation

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 83

Sustainable success is driven from a sound vision

Initial success requires momentum, but sustainability requires

effective planning and design

Everyone in the department plays a role

Audit management must be willing to adapt and change the audit

process to support effective analytic usage

Don’t just expect to evolve over time, plan for it

Concluding Thoughts

© 2015 High Water AdvisorsConfidential. Do not distribute without permission. Page: 84

Questions / Thoughts?

[email protected]

1-855-RISK-WATCH

Wrap-up

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 86

Workbook for a Successful Audit Analytics Program

‒ Just launched this week at IIA GAM

‒ 50% off introductory rate through the end of March, 2016

Set of working documents to guide you

through decisions

Checklists for CAE and management

Get your copy at:

‒ www.highwateradvisors.com/workbook-for-a-successful-audit-analytics-program

Just released: Workbook for a successful program

© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 87

High Water Advisors Can Help

Analytic development, QA and/or

optimization

Strategy development & coaching

those responsible for

managing/implementing analytic

programs

Training for those developing data

analytic routines

‒ Strategic concepts

(i.e. dealing with false-positives)

‒ Sustainable Analytic Design

‒ Technology-specific programming

Configuration baselining

‒ Recommended settings

‒ Reasonableness

‒ Comparative to similar entities

Transactional analysis & profiling

Control circumvention assessment

(suspicious activity)

Security assessment & back-door

activity analysis

Cybersecurity

SAP-Centric Data AnalyticsGeneral Data Analytics