Presentation to ISACA New England Chapter March 20, 2008 Examination of IT Audit

Presentation to ISACA New England Chapter

March 20, 2008

Examination of IT Audit

2

Speaker: Ken Fortier, CISAIT Examination SpecialistFederal Reserve Bank of BostonSupervision, Regulation & Credit DepartmentLarge Bank Supervision

Disclaimer:The opinions expressed are those of the speaker and do not represent official policy or guidance of the Federal Reserve Bank of Boston or the Federal Reserve System.


About Today’s Session:

3


About Today’s Session:

Reach in for a handful of what you want.

4

Auditor or Examiner: What’s the difference?


5



Who we work for.

Auditor:Accountable to the Board of Directors

• Audit Plan approved by the Audit Committee

• Staffing & Budget governed by Executive Management

Examiner: Accountable to their Federal or State supervisory agency.

• Supervisory Plan set by supervising agency / interagency process

6



Our perspective.

Auditor:

“Internal” perspective:

• “Risk to the Institution” View

• Strives to maintain independence from control processes and management influence

• Focus on financial, operational & technical controls

• Provides “Opinion” on Compliance with Regulations & Guidances

• Reluctant to assess management capacity & effectiveness

• Does not assess Board / Audit Committee

7



Our perspective.

Examiner: “External” perspective

• “Risk to the Banking Industry” View

• Independent of institutional processes & management structure

• Focus on Risk Management Programs, including management’s ability to:– Identify, measure and monitor risk– Apply appropriate controls to mitigate risk – Adjust programs to changing Risk Profiles

• Emphasis on oversight control and governance

• Determines Regulatory Compliance

• Evaluates Management (capacity, effectiveness, commitment)

• Assesses the Audit Function

• Determines whether Board / Audit Committee responsibilities are being met

8



Our Approach.

Auditor: Performs Continuous Audit Processes for timely measurement of key risks/exposures

• Threshold Alerts (Transaction/Aggregation Limits; Volume/Capacity)

• Review of MIS (e.g. Business Line Metrics, Activity/Volume Reports, Exception Reports, Change Reports, Event Logs/Alerts, etc.)

• Maintains ongoing dialog with management (e.g., Performance; Strategic & Organizational Changes; etc.)

• Targeted Key Controls Validation

• Monitoring of New Initiatives & Systems Development Projects

Performs Audit Review of High-risk Projects

9



Our Approach.

Auditor:

Performs “Traditional” Audits of Business & IT Functions• Reviews Processes for Effectiveness & Efficiency• Assesses adequacy of defined policies, procedures & standards• Conducts Validation Testing to determine adherence to policies,

procedures and standards• May deploy Integrated Audit Software for independent control

validation or data extraction on production systems

May perform Horizontal, Vertical and Integrated (Financial/Operational/IT) Audits for comprehensive view

Performs Technical Audits (e.g., Application, Operating & Database Systems; Utility Software; Network & Telecom Components; etc.)

10



Our Approach.

Examiner: Performs Continuous Supervision Processes for timely measurement of key risks/exposures

• Monitors changes in institution risk profile:– Review MIS (e.g., Performance/Capacity Reports; Volume/Trend Reports; etc.) – Review Management Summary Reports (e.g., Board/AC Packages; Steering

Committee Reports; Project Status Reports; Incident Reports; etc.) – Review Assessment Summaries (e.g., Audit Reports; ERM Reports; Third-party

Test Summaries; DR Test Summaries; Regulatory Analysis; etc.)– Maintains ongoing dialog with management (e.g., Performance; Strategic &

Organizational Changes; etc.)• Conducts “Target” Exams (Large Bank Supervision)

– Risk Management Programs (e.g., Information Security Program, Business Continuity Planning, etc.)

– Business Lines/Functions (e.g., Consumer Finance; Wire Transfer; etc.)

Conducts Exams to Support Ratings Assignment• Leverage Continuous Monitoring and “Target” Exams• Uniform Rating System for IT (URSIT) Components: Management, Audit,

Development & Acquisition, Support & Delivery

11



Our Approach.

Examiner: Looks to leverage various monitoring, validation and assessment activities designed to ensure the reliability of controls

• Audit Reports

• Penetration Tests

• Network & Web Application Vulnerability Scans

• Systrust/Webtrust Reviews

• Business Continuity / Disaster Recovery Test Summaries

• System Patch Level & Anti-virus Maintenance Scans

May conduct validation testing on a risk-basis• Most agencies do not deploy independent validation/data extraction software

on bank systems

12



How we enact change.

Auditor:

Submits Audit Report to Management • Provides Conclusions on Audit Scope Areas

• Communicates Findings & Recommendations to Management – “Severity” ratings often applied to issues

• Seeks Management “Buy-in” to Recommended Action

• Requests Management Response & Evaluates Action Plan

• Assigns an Audit Control Rating (Auditor “Opinion” on Effectiveness/Reliability of Controls)

Tracks & Reports Status of Open Issues• Provides Summary to Audit Committee

• Escalation Channel: Executive Management, Audit Committee

13



How we enact change.

Examiner:

Issues Report of Examination to Executive Management / Board• Provides Conclusions on Exam Scope Areas • Communicates “Required Action” and “Recommendations” • Requires Response & Evaluates Management Action Plan• Assigns an Examination Rating (URSIT Rating for IT)• Risk-based approach to Issue / Action Plan Tracking• Escalation Channel: Additional Enforcement Actions

– Board Resolution, Civil Money Penalties, Cease & Desist, Removal– Impact on Applications Process (Mergers & Acquisitions, New Ventures)

14

Examination Review Points: IT Audit


15



Examination objectives & procedures are identified in the FFIEC IT Examination Handbook: Audit

Available at the FFIEC Website (www.ffiec.gov)• http://www.ffiec.gov/ffiecinfobase/html_pages/it_01.html

Other FFIEC IT Examination Handbooks Available:• Business Continuity Planning• Development & Acquisition• E-banking• Fedline• Information Security• Management• Operations• Outsourcing Technology Services• Retail Payment Systems• Wholesale Payment Systems• Supervision of Technology Service Providers

FFIEC Handbooks are a Guide to Examiners

16



Independence (FFIEC Handbook – Tier 1 Objective 5: Determine the level of Audit Independence)

Audit Charter: Establishes the authority and mission of the Audit Function

Defined by the Board

Precludes Conflict of Interest Duties

Authorizes full access to information, records and systems

Auditors (Internal & External) report directly to the Board or Board-level Audit Committee

Approval of the Audit Plan; Changes to the Audit Plan

Approval of “Out-of-Scope” Management Requests

Presentation of Audit Reports

Auditor has the ability to escalate issues to the Board Through normal Audit Committee process (Audit Committee Executive Session)

Through Direct Contact with AC Chairperson and Outside Directors

17



Independence

Administrative Reporting – “Degree of Control” management has on:What is reported to the Board

What is reviewed by Audit

Approval of Audit Staffing & Contract Requests

Department Compensation Levels • General Auditor compensation reviewed by Board/Audit Committee• Comparative data analysis by Audit Committee

Performance Appraisals & Measurement Criteria• Based on Job Descriptions, Audit Charter and Audit Committee Directives

Auditors are not responsible for ongoing control processes

Audit Ratings are assigned based on a defined structure, and are not “Negotiated” with Management

18


Examiner Review Points: IT Audit

Board & Audit Committee Oversight(FFIEC Handbook Tier 1 Objective 2: Determine the quality of the oversight and support of the IT

audit function provided by the Board and Senior Management.)

Board / Audit Committee:Defines the Authority & Mission of the Audit Function (Audit Charter)

Reviews and Approves the Audit Plan• Ensures the Audit Plan provides proper risk-based coverage of the “Universe” of

Audits– Does the Audit Committee know what is not being Audited?

• Ensures performance of the Audit Plan & Schedule

• Approves Major Deviations from the Plan

Maintains proper awareness of audit conclusions, significant findings, and management progress on significant issues.

• Reviews Audit Reports; Control Ratings Updates ; Issue Status Reports

• Ensures an appropriate level of Committee reporting

• Discussion is reflected in Board / Committee packages and meeting minutes

19



Board & Audit Committee Oversight

Board / Audit Committee:Approves the scope of engagement of External and Outsourced IT Auditors.

• Ensures the audit resource are independent and qualified

• Ensures the scope of review is adequate to support comprehensive assessment, including reviews of complex programs (e.g., Information Security, Vendor Management, etc.)

20



Staffing (FFIEC Handbook – Tier 1 Objective 4: Determine the qualifications of the IT Audit staff and its

continued development through training and continuous education.)

IT Audit staff is adequate in number and is technically competent to accomplish its mission

Staff level adequately supports the Audit Plan or adequate resources are secured through contract support

Staff is qualified to perform duties • Education, Experience & Certifications• Qualifications vs. Job Descriptions• Staff is qualified in the Technologies used

Specific expertise is secured where needed.

Training program ensures ongoing technical competence, consistent with technologies in use/planned.

Adequacy of Current Training Budget

Record of Past Training

21



IT Audit Policies, Standards & Procedures Formal and comprehensive Policies, Standards and Procedures are established to guide IT Audit activities and ensure consistency

Address who, what, where, when and how IT Audit activities will be conducted.

Address all key Audit activities

Controls are established to ensure adherence with policies, standards & procedures (e.g., Review & Approval processes; Quality Assurance reviews; etc.)

Reliable processes are established to update Audit policies, standards and procedures.

The expectation for well defined policies, standards and procedures applies to each of the following discussion topics.

22



Defining the IT Audit “Universe” (FFIEC Handbook – Tier 1 Objective 8, Step #1: Determine if the audit universe is well defined.)

IT Audit “Universe” is Properly Defined Addresses the inventory of applications & platforms in use (e.g., Applications; Operating Systems; RDMS; Utility Software; Network; Telecom; Hardware; Physical Locations; etc.)

Addresses the inventory of IT operations, functions & services (e.g., Information Security; Network Security; BCP/DR; Project Management; Development & Change Management; Vendor Management; Production Control; System Operations; etc.)

Addresses outliers to central IT functions

Reliable processes exist to identify & account for changes in the institution’s risk profile that may affect the Audit function (e.g., changes in technologies & processes; new products; organizational changes; etc.).

Examiners may compare the defined IT Audit “Universe” with Systems Inventory (BCP); Network Diagrams; Organizational Charts; etc.

23



Audit Risk Ratings (FFIEC Handbook – Tier 1 Objective 8: Determine the adequacy of Audit’s risk analysis

methodology in prioritizing the allocation of audit resources and formulating the IT Audit Schedule.)

Audit Risk Ratings: Analysis includes all appropriate risk factors. (e.g., Strategic; Financial Impact; Operational; Transaction; Technology; Reputation; Legal/Regulatory; etc.)

Assigned ratings are appropriate & supported.

• How do the assigned audit risk ratings compare with Business Impact Analysis (BCP) and Information Security risk rankings?

Applied to the “Universe” of Audits

Reliable processes exist to ensure audit risk ratings are consistently applied. (e.g., review & approval process; Quality Assurance review; etc.)

Reliable processes exist to ensure that significant changes are identified and addressed to ensure continued reliability of the risk ratings.

24



Audit Plans (FFIEC Handbook – Tier 1 Objective 7: Determine the adequacy of the overall audit plan in

providing appropriate coverage of IT risks.)

Audit is a process, not a single event. IT Audit activities are viewed as a whole when assessing the adequacy of coverage.

• Continuous Monitoring / Controls Validation

• Traditional Audits

• Integrated Audits

• Project Audits

Audit Plan:Ensures proper risk-based coverage of the Audit “Universe”

Provides for appropriate frequency of review for High and Medium risks.

Does not exclude Low risk areas

Meets appropriate (defined) standards for frequency of review

Audit Delineation and Budgeted Hours support comprehensive review

Is regularly met (without routine scope reductions)

25



Audit Reports(FFIEC Handbook – Tier 1 Objective 9: Determine the adequacy of the scope, frequency, and

timeliness of IT-related audit reports.)

Auditors accurately identify and consistently report weaknesses and risks.

Reports provide timely communication of issues:• Audit review period vs. report date

• Critical issues are discussed at time of discovery

Report Distribution: Addressed to an appropriate level of authority to affect corrective action.

Distribution to other affected parties/stakeholders

Audit Scope:Properly defined to provide clear understanding of coverage

Scope limitations and significant “Out-of-scope” items are identified

Appropriate to support conclusions• Core functions are included in scope of review• Compare with audit risk assessment

26



Audit Reports

Audit Findings:Accurate, complete and clearly defined

Properly identify Root Cause

Repeat issues are identified

Common issues are recognized across audits, and are linked for management attention.

Issues are properly categorized based on associated risks and compensating controls.

Audit Workpapers properly support decisions for issues not reported.

Audit Recommendations:Properly address Root Cause

Appropriate to prevent recurrence of issues

27



Audit Reports

Management Action Plan:Appropriate to resolve issues in a timely manner and minimize likelihood of recurrence; or

Auditor’s Note is provided to identify concerns with management’s action plan

Audit Rating:Consistent with volume & severity of identified issues

Consistent with stated conclusions.

28


FFIEC Guidelines: IT Audit Reports

FFIEC Basic Audit Report Guidelines – Report Should…Provide Scope and Objectives of the AuditSummarize all significant observationsProvide written notification to senior management and BoardHighlight exceptions, potential risk exposure and recommendations for remedial actionState an overall opinion of the function, improvement or decline since last audited, reasons for changesEnsure timely written responsesSatisfy audit objectivesEnsure conclusions are appropriate for work performedExercise sound judgment in separating significant/insignificant findings

FFIEC Additional Audit Report Considerations – Report should… Provide an opinion on the adequacy of stated action plans Provide a definition of Control Ratings Provide a definition of Risk Ratings When appropriate, establish a timeline for follow-up audit

29


Examiner Review Points: IT AuditIssues Tracking (FFIEC Handbook – Tier 1 Objective 6: Determine the existence of timely and formal follow-up and

reporting on management’s resolution of identified IT problems or weaknesses.)

Reliability of the Issue Tracking Process

Effectiveness in Securing Timely & Appropriate Corrective ActionExtent and Age of Open Audit Issues; Repeat Issues

Adequacy of Management Action (Addresses Root Cause; Timely Action)

Prior Issues History

Adequacy of Issue Escalation & Board Reporting Processes

Process for Issue Closure:Audit approval/agreement required prior to Issue Closure

Audit reviews/validates corrective action prior to closure (High Risk Issues)

30


Examiner Review Points: IT AuditAuditing Systems Projects (FFIEC Handbook – Tier 1 Objective 10: Determine the extent of Audit’s participation in

application development, acquisition, and testing, as part of the organization’s process to ensure the effectiveness of internal controls.)

Policies regarding Audit participation in SDLC projects are clearly defined.

Reliable processes are established to identify new projects.

An appropriate Project Risk Rating methodology is established and consistently applied.

Auditor is not simply repeating Project Management Office statements (“On-time” / “Within Budget”)

31



Auditing Systems Projects Appropriate risk-based audit coverage is provided for high-risk projects, including sufficient validation to conclude on the adequacy of project controls and testing activities.

Validation that Project Methodology is followed

Project status is accurately reported

Appropriate testing is being performed • Systems Integration Testing• User Acceptance Testing• Data Conversion Testing• Sample Test Plans / Test Scripts

Issues are followed-up / escalated

The Audit Committee is properly informed of Project Audit activity, including high-risk projects that Audit is not reviewing.

32



Process Audits and Technical Controls Validation

Auditing of IT processes is expected; however, process auditing alone is not enough. An appropriate level of audit review and testing must be performed to validate the reliability of technical controls.

The level of audit review and testing should be commensurate with the complexity and risk profile.

Leveraging alternative control validation processes is acceptable (e.g., third-party audits, vulnerability scans, penetration tests, etc.). However, the Auditor must:

assess the reliability of the alternative control validation; and

ensure that the scope is sufficiently comprehensive to support Audit objectives and conclusions.

33


Gaps in Technical Controls Validation – One Example

Loan Application Audit

Audit included limited review of application security.• User Access Rights• Password Configuration

The Auditor performed no validation of critical calculations. Auditor placed reliance upon a strong Change Management Process (recently audited). Process includes:

Comprehensive Testing of System CalculationsQuality Assurance Review & Approval for production system changesRestricted access to Production code, data, and interest rate tables.

So what’s the problem?

34


Gaps in Technical Controls Validation – One Example

Loan Application Audit The application was executing from a Test Environment. Interest rate calculations were incorrect.

The Auditor performed no validation to ensure that the Application was running from Production. (e.g., review of Job Logs; review for Test Library References in Production Link List or JCL)

35


Gaps in IT Audit Coverage – Other Examples

Application Audits:

Scope defined to include Application Security; however, Audit validation limited to user access provisioning. No review of security over application code and data.

Network Security Audits:

Reliance on Network Diagram (Visio Diagram) provided by Network Administrator without efforts to validate

• Network Mapping Tools• Network Addressing• Review with Network Administrators

Firewall AuditAuditor received no training in FW type; could not interpret FW rule set. Failed to identify “Holes” in FW (Stateful Inspection FW allowing UDP traffic.)

36


Questions???

Thank You!

Documents

Presentation to ISACA New England Chapter March 20, 2008 Examination of IT Audit