View
3
Download
0
Category
Preview:
Citation preview
1
Harnessing the Power of Innovation
Marc Sabino
Chief Auditor – Head of IA Innovation, Citigroup
Robert Mullen
Director – IA Consumer Analytics and Reporting, Citigroup
October 28th, 2019
2
Participating in the Polling SessionDownload the IIA Events App
How to Vote:
• Download the app on your device:
– “IIA EVENTS” in App Store /
Google Play
– Login with the email you
registered with
– Password is diiasc19 (case
sensitive)
• Open “VOTING” from the menu,
select this session
• Click “VOTE NOW”
3
Citi-Sponsored Social Media EngagementDownload the Snapchat App
SHARE
SNAP
SAVE
Remember to enable your location
settings to see our fun Citi IA filter
Hit the down arrow and save your snap
Email your photos to lucy.liow@citi.com
and tag us using the hashtag #LifeAtCiti for
a chance to be featured on @CitiCareers
#LifeatCiti@CitiCareers
4
IntroductionMarc Sabino & Robert Mullen
Robert MullenDirector, IA Consumer Analytics and Reporting, Citigroup
Marc SabinoChief Auditor, Innovation, Citigroup
5
Polling Question
What is the size of your audit organization?
Some sizes you could submit are:
• 0 – 50 employees
• 51 – 100 employees
• 101 – 500 employees
• 500+ employees
Join the conference poll in order to submit your answer!
6
Polling Question
My organization receives support from senior
management to expand data analytics capabilities.
Please select from the below:
• Strongly Agree
• Agree
• Slightly Disagree
• Disagree
Join the conference poll in order to submit your answer!
7
Polling Question
What technology do you use the most in
implementing your testing?
Please select from below:
• Excel, VBA, and Kutools
• SQL, SAS, and other DB Management Languages
• Off the shelf analytics tool (e.g. ACL, Alteryx, etc.)
• Robotic Process Automation, via Automation Anywhere or other languages
• Python for HTML interactions
• Machine Learning Via Python, Tensorflow, or other languages
• Artificial Intelligence Via Python, Prolog, Java, or other languages
Join the conference poll in order to submit your answer!
8
How is Citi responding to new
methods and structures with
strategic insights on risk and
the control environment?
9
May 7, 2019
Introduction to Audit InnovationOur Mission
Smarter Auditing – significantly improve the control environment and assurance through
large population testing, anomaly detection, and new techniques
Advanced
Analytics
Integrated
Solutions
Alerting
& Insights
Natural
Language
Processing
Machine
Learning
Augmented
Intelligence
Audit of the
FutureInsightful Risk Assessment
Smarter Testing
❑Deeper testing through automation and full population
testing
❑Use of multiple solutions to help identify thematic
patterns across countries and entities
Improving the Auditor Journey
❑Using technology and innovation to support the auditor
of the future
❑Sharing subject matter expertise, yielding insights
and popularizing a data driven mindset
❑Identifying emerging risks and issues to drive insights
and connect the dots
❑Using data and integrated platforms to uncover hidden
patterns and develop new hypotheses
10
How do the degrees of
innovation deliver greater
assurance, build a premier
function, and help auditors
identify hidden patterns and
emerging risks?
11
Impact of Innovation
When auditors spend their time on highly repetitive and mundane tasks, it reduces the amount of time that they can
dedicate to stakeholders or high value-add activities.
❑Reduction of error rates and added assurance by
testing full population
❑Auditors are more empowered from the utilization and
exploration of data, augmenting their decision-making
and creating the auditor of the future
❑Auditors can identify emerging risks and issues to
connect the dots and drive strategic conversations
with stakeholders
❑Real time escalation of thematic issues
❑Widespread use of data helps gain an understanding
and strengthening of entire control environment,
providing thematic insight
❑Timely identification of issue themes and root causes
enhances business controls
Benefits for the Auditor Benefits for the Stakeholder
12
Impact of Innovation
❑To demonstrate that Internal Audit is a positive change agent for their entire organization in regards to Innovation
❑That the new tools and techniques leveraged to enhance their core businesses are also being leveraged within Internal Audit to further enhance the control environment
❑Assurance that Internal Audit has the subject matter expertise to audit new and emerging analytical tools and technology
❑Reduced risk of non-compliance to regulatory
requirements
❑Increased depth of coverage using full populations
❑Increased frequency of coverage from continuous
auditing
Benefits for the Board Regulatory Agencies
13
What does the average
auditor’s journey look like for
adopting and utilizing
analytics in audit work?
14
Our Journey from Audit Analytics to Advanced Innovation
Ch
ara
cte
risti
cs
Ma
turi
ty
Innovative Approach
Advanced
Transformation
Development and Global
use of Repeatable Analytics
Leading
Self-Service / Web-enabled
Auditing Solutions
Established
Develop Audit Innovation
Vision and Strategy
Smarter Assurance Greater InsightsAdded Capacity
All Inclusive
Encompassing
Time
The
Innovation Journey
Overview
IA Innovation embraces
transformative change – moving
beyond our original focus on testing
to encompassing and transforming
many aspects of the full audit cycle.
Our Innovation journey
demonstrates our continuous
improvement process.
Full Audit Cycle
15
What leading tools and
techniques can be
utilized for analytics?
16
May 7, 2019
Degrees of Innovation
Levels of Innovation Sophistication
CAATTS and
Audit Analytics
a b c
a b c
a b c
a b 9w e
z6
Scripting of scripts
Linear taskOrchestrated
activities
Read
e-mailCopy-paste Enter data
Send
confirmation
Non-
standard
Machine Learning & Natural
Language Processing
Contextual
Inference
Process Dynamic
a
b ¿?
c
d
¿?
Understand
different process
Convert
images to text
Understand
and learns
Systems Self Aware
Cognitive / AI
PredictiveSelf
Learning
Self
Healing
Predicts future
behaviors
Virtual
Assistants
Auto
Feedback
❑ Process of automating repetitive tasks
❑ Implementing rules and logic
❑ Productionizing the scripts for repetition
❑ Analysis of past and present data sources
to make algorithmic predictions
❑ Using thematic analysis to identify emerging
risks and regulatory policy matching
❑Machine programmed to think, work,
evolve and react like humans
❑ Helping humans become faster and
smarter at the tasks they're performing
Innovation covers a spectrum of technologies, including VBA, SAS, RPA, NLP, and Artificial Intelligence
17
Analytical Tools used in Audits and Issue ValidtionIA Consumer – Highlights
IA Consumer leverages multiple tools in audits and issues validation to test the completeness and accuracy of a
control or data set back to source systems
Powerful and Flexible Tools
❑SAS Enterprise Guide, Microsoft SQL Server, Oracle SQL Developer are powerful tools that enable auditors to
perform both one time and repeatable validations of nearly any control or issue remediation that uses data
❑The tools are flexible, providing auditors the capability to analyze/test in both direct an indirect ways
❑Any value add analysis could become a continuous audit and placed on a scheduler
Impact
❑Entirely independent validation from source to final product
❑Provides the power and flexibility auditors need to effectively test the complex and data extensive
businesses we oversee
❑Increases the effectiveness of the assurance provided by Internal Audit
Traditional Approach
❑Manual spreadsheet review of data
populations provided by the business
(limited independence)
❑Non-statistical sample review to host
systems (e.g. document review)
❑You don’t know what you can’t see
Innovative Approach
❑Size doesn’t matter, the size of the data has no impact on the
scope of the review
❑Provides the capability to review the entire process/data flow
for gaps
❑Enhanced completeness and accuracy testing using
templates, scripts, and programs
18
Analytical Tools used in Audits and Issue ValidtionIA Consumer – Regulatory Issue Validation
IA Consumer works with many regulators across the globe. In the US, they include, among others, the Federal Reserve Bank (FRB),
Office of the Comptroller of Currency (OCC), Consumer Finance Protection Bureau (CFPB), and the Department of Justice (DOJ).
The Challenge
❑How to validate large customer remediation's where complex analytics and waterfalls we’re employed to satisfy
the requirements of an Enforcement Action/Consent Order or Matter Requiring Attention (MRA)
❑How to test deep enough to provide the company sufficient assurance that work performed by the business was
complete and accurate, thus minimizing the chances of a regulatory reopen
Impact
❑Initially, much of the effort resulted in retargets and IA reopens
❑Eventually, the efforts dramatically reduced the regulatory reopen rate, new business procedures, and an
improved control environment
❑Today, the use of analytics for customer remediation's is an expectation for all organizations involved
The Solution
❑Implement a validation model within Internal Audit that leverages analytics from source systems to final
remediation population to increase confidence levels
❑Provide the business new guidelines for documentation analytics
❑Employ an agile engagement model between Internal Audit and the business to increase the likelihood of
success
19
Analytical Tools used in Audits and Issue ValidtionIA Consumer – Regulatory Issue Validation (cont’d)
Internal Audit’s expectations of the business in regards to documenting analytics
1. Supporting documentation includes the full scope of the analysis, from original data source(s) to finished product. Including:
a. Succinct beginning-to-end documentation for practical review (layman’s terms). Target audience is executive management,
b. Internal Audit and regulators.
c. Comprehensive technical documentation used to perform the analysis.
2. Supporting artifacts (e.g. code/logic) and data repositories should be supplied to provide the capability for a third party to re- perform
the complete analysis.
3. Data sources are documented, including supporting evidence that they are classified as the most appropriate source, or equivalent, for
the intended purpose (e.g. gold source, master files, etc.). Data flow diagrams are preferred for moderate to complex environments.
4. A data dictionary is provided for tables and relevant data fields; especially those used to include or exclude records (e.g. active or
inactive accounts, product codes, etc.).
5. Data profiling or quality review is performed on all data sources. The review should be comprehensive and include supporting
documentation to evidence completeness (e.g. graphs displaying counts of records distributed by relevant time periods).
6. Any data limitations and known gaps are disclosed, including any management decisions to remediate, compensate, or exclude with
supporting rationale.
7. For funnel or waterfall analysis, clearly define each group or bucket categorization. Including:
a. Supporting documentation for calculations.
b. Evidence that all records have been accounted for in aggregate.
8. A qualified checker(s) or reviewer(s) provides credible challenge for each phase of the analysis. Their efforts should be
documented, including any observations and subsequent revisions or versions that were performed.
20
Robotics in AuditFINRA License Testing Bot
Advanced robotic techniques automate manually intensive tests which are repeatable to increase efficiency
while enhancing audit assurance
Traditional Approach
❑15 minutes to manually check one broker
would take 1 year to test 5500 brokers
❑ 5% error rate and sample size of 25
Robotics
❑Programmable device that can perform tasks and interact with its environment, without the aid
of human interaction
❑Implementation without altering existing tech infrastructure
❑Utilizes automation, machine learning, and cognitive / artificial intelligence
Impact
❑Enhanced assurance through increased sample size
❑100% of broker populations tested in 3 business areas
❑Time Savings through the automation of repetitive manual tasks
Innovative Approach
❑1 minute to check one broker
❑0 Hours used by auditor for testing, allowing
auditor to focus on exceptions
❑100% population testing
21
Machine Learning and Natural Language Processing in AuditCustomer Complaints
Monitoring of complaints and social media data to identify trends and emerging themes to improve risk
assessment and predictive risk capabilities
Traditional Approach
❑Limited to sampling of internal complaints
❑Reactive vs predictive – utilizing social media
as a leading indicator
❑Requires time consuming manual reading
and categorization of applicable complaints
Machine Learning & Natural Language Processing
❑Sourcing Complaints from social media to enable early risk identification and intervention
❑Differentiate between comments and complaints
❑Categorize complaints into appropriate category
❑Visualize data to identify outlier activity and emerging risks
Impact
❑Leveraging solution to identify areas of increased risk based on customer experience
❑New data points to identify emerging risks and hidden patterns
❑Lending insight that can be used for audit scoping and business monitoring
Innovative Approach
❑Millions of messages analyzed in < 3 hours
❑Consistency in categorization without individual bias
❑Potential leading indicator of formal complaints and other
risk factors
22
Machine Learning and Natural Language Processing in AuditCustomer Complaints
1 2
1• Look at trends in the number of social media
complaints over time, compared to other channels
such as internal or CFPB
2• Identify complaint categories and easily detect
patterns and anomalies
3• Filter for and select the complaints of interest
• Customized filters possible for specific teams
3 4
4• Perform a thematic analysis of complaints within a
specific business or process
• Customized thematic views possible for specific
teams
565
• Easy, visual filtering capability enabled via word
cloud
6• View and export filtered complaints to Excel for
additional slicing and dicing
23
Polling Question
When using audit analytics and innovative techniques, where do
you think the strongest impact is to an organization?
Some Areas to consider for your answer are:
• Planning
• Fieldwork
• Testing
• Reporting
Join the conference poll in order to submit your answer!
24
Robotics, Machine Learning and Natural Language ProcessingEnterprise Platform
Web-based solution execution platform which centralizes Innovation tools for every phase of the audit
life cycle, enabling consistent & globally accessible monitoring, testing, & reporting
Traditional Approach
❑Use of Analytics limited to auditor skillset
❑Analytics performed in siloes
❑Solution execution performed on an ad-hoc
basis
Enterprise Platform
❑Web based solution execution enables global on-demand testing and monitoring
❑One stop shop for all Innovation offerings including Online Solutions, Packaged Solution and
Robotics requests
❑Allows the audit team to leverage analytics and innovation throughout all phases of the audit
life cycle
Impact
❑Cross-functional and utilized by auditors and business leaders
❑Empowers auditors to embrace data driven mindset
❑Reduction of manual documentation and mobile compatibility, facilitating auditor of the future
Innovative Approach
❑Solutions & Bots at the click of a button
❑Customized threshold monitoring & alerting
❑Consistent use of control tests
25
Enterprise Analytcal ToolsEnd User Computing (EUC) Solution
Enterprise solution that includes prebuilt tests and the capability to run custom reports from data
sourced from the host system
Traditional Approach
❑Individual requests for data from the business
for each audit
❑Single use testing approach
❑Manual review
Enterprise Analytics
❑Highlights overdue reviews, changes to existing EUCs and ratings, and other analysis aligned to
Citi policy and EUC testing plans
❑Common global platform
Impact
❑Uniform testing opportunity across the entire audit universe
❑Time savings using standardized metrics, documentation and reporting sources
Innovative Approach
❑Uniform default tests available to all from a reliable data
source
❑Centralized platform to add/enhance tests
26
Enterprise Analytical ToolsEnd User Computing (EUC) Solution
Host System Enterprise Analytical Platform
Direct Data Feed
Test Test Name Test Description
EUC01 Required Fields Identify missing (blank/null) fields within EUC portal data.
EUC02 Overdue Reviews This analytic identifies overdue reviews.
EUC03
Upgraded /
Downgraded Risk
Assessments
Identify all risk assessments that have been upgraded or
downgraded the last year.
EUC04 Missing EUCs Identify EUCs removed during the last year.
EUC05 EUC Complexity Alignment of EUC Characteristics with complexity values.
EUC06 Retirement Plan Verify that all High risk EUCs have a retirement plan.
Test Examples
27
Optical Character Recognition (OCR)Algorithmic Solution
Algorithmic solution that compares affidavits from various sources and detects anomalies across the
various files, leading to efficiencies over traditional sampling approach.
Traditional Approach
❑Reviews are done on a manual basis
❑Comparisons done as sample of 40
❑1 document comparison can take as much as
15 minutes to review
Enterprise Platform
❑Algorithmic solution enables efficient on-demand testing and monitoring of affidavits
❑Individual characters are compared one another and assessed for changes
❑Allows the audit team to leverage analytics to assess larger populations of documents
Impact
❑Cross-functional and ready to expand and scale to other types of documents
❑Enhances auditors ability to only consider true exceptions and leverage continuous monitoring
❑Adds value to audit work, as well as having applicability in regulatory and business uses
Innovative Approach
❑Solutions run quickly and are less prone to error
❑Comparisons can take place across entire document
population
❑Consistent use of control tests
28
Optical Character Recognition (OCR)Algorithmic Solution
1
21
2
3
3 4
4
56
5
6
Status: If the documents were compared
successfully, the Status will be ‘Completed’.
However, if there was some issue with the
comparison, such as a failure to compare due to
poor quality of the original documents, the status
will be something other than Completed
Difference Detected: If there is one or more
differences found in the documents, the value will
be Yes. Otherwise, if the two documents are
exactly alike, the value will be No.
Content Dissimilarity: Illustrates the magnitude of
the differences, i.e. the more difference, the more
bars
Amount Match: If a change is detected in an
Amount field
Date Match: If a change is detected in a Date
field
View Differences: View the differences in the two
documents side by side (see slide 7 for more
details)
29
Polling Question
Are there heighted expectations for enhanced IA coverage using analytics
and innovations in your organization?
Please select from the below:
• Strongly Agree
• Agree
• Slightly Disagree
• Disagree
Join the conference poll in order to submit your answer!
30
What’s Citi approach to
increase the use of
Analytics and Innovation
in Internal Audit?
31
Innovations Team
Auditors
Auditors
Citi’s Multifaceted Approach to Expand Analytics and Innovation
The multifaceted approach includes both top down and bottom up efforts to leverage the best of both
worlds to increase the development and use of analytics and innovations
Top Down
❑Dedicated Innovation Team
❑Staffed with innovation and analytics skill sets
❑Integrated engagement with audit teams
❑Focused on enterprise or large scale solutions
Impact
❑Leverages the best ideas from all viewpoints and backgrounds, from
technologist/data scientist to auditor
❑Ensures analytical capabilities are not limited to the few, empowering the
organization to increase their capacity at the fastest rate possible
❑Fosters practical and value add solutions that work beyond the test lab
Bottom Up
❑Staffed with audit and analytics skill sets
❑Leverage most appropriate solution or tool depending on audit scope
❑Provide recommendations for future solutions
32
What are the skills and
considerations for the People,
Processes, and Technology of a
Audit Function of the Future?
33
Considerations for People, Process, and Techology
Citi is expanding and enhancing all three elements of People, Process, and Technology to harness the
power of innovation, including:
Process
❑Innovation opportunities are identified continuously (culture)
❑Analytics demand for audits and issue validation is identified in advance
❑Ongoing engagement between Innovations and Audit Teams
People
❑Hiring technically skilled innovation staff
❑Providing analytical training to current staff
❑Modifying our new hire profiles to include analytical skills
❑Consider the technology being leveraged by those you audit to streamline future
technical engagements with stakeholders
❑For enterprise solutions, what platform(s) will best serve both current and emerging
technologies
❑Consider the user base of the proposed technology to ensure sufficient long term
support
People Process
Technology
Technology
34
Questions and Answers
Thank you for joining us today! We will now take questions.
Thank You!
Recommended