17 May 2011Utah Legislative Auditor GeneralSlide 1 Office of the Legislative Auditor General Performance Audits and Evaluation

17 May 2011Utah Legislative Auditor

General Slide 1

Office of theLegislative Auditor General

Performance Audits and Evaluation

Who We Are


General Slide 2

The Office of the Legislative Auditor General (OLAG) is constitutionally created by Article VI Section 33 of the Utah Constitution. The Legislative Auditor General has broad authority to conduct audits of any funds, functions, and accounts in any branch, department, agency or political subdivision of this state.

The Legislative Auditor General is appointed by the Legislature for six year terms.

The Auditor General reports directly to the bipartisan Audit Subcommittee of the Legislative Management Committee.

What We Do


General Slide 3

It is the mission of the Office of the Legislative Auditor General to serve the citizens of Utah and the Legislature by providing objective information, in-depth analyses, and useful recommendations that help Legislators and other decision makers.

The Office of the Legislative Auditor General accomplishes its mission by:

Improving Programs Reducing Costs Promoting Accountability

Completing The Puzzle


General Slide 4

Resolving Issues Identified By Legislators

What Some People Say We Do


General Slide 5

“The Washington State initiative sent ripples through the nation's public-sector auditing community, and for a very simple reason: Performance auditing is an evolving and somewhat controversial practice.” October 2009

“They continue to insist that the exercise focuses more on catching governments in the act than on proactively working to improve key government operations.” October 2009



General Slide 6



General Slide 7


General Slide 8

used

wit

h p

erm

issi

on

How We Cope


General Slide 9

F in a n c ia l C o m plia n ce

T ra d it io n a l

E ff ic ie n cy(O p e ra tio n a l)

E ffe ctiven e ss(P ro g ra m )

P e rfo rm a n ce

T yp es o f G o ve rn m e nta l A u d its


General Slide 10

Audit Complexity

DegreeOf

Difficulty

Time & DollarsExpended

Financial

Compliance

Efficiency(Operational)

Effectiveness(Program)

What Is Performance Auditing


General Slide 11

Performance audit refers to an examination of a program, function, operation or the management systems and procedures of a governmental or non-profit entity to assess whether the entity is achieving economy, efficiency and effectiveness in the employment of available resources. The examination is objective and systematic, generally using structured and professionally adopted methodologies.

Examples of Performance Audits


General Slide 12

Completing of the Puzzle Via Legislative Audit Requests

* Identify reasons why a heightened number of jail escapes and other inappropriate activities where happening in county jails.

* Identify instances of fraud, waste, and abuse in Utah’s Medicaid program.

* Review efficiency and effectiveness in Utah’s Medicaid managed care.

* Examine the use of O&M funding for institutions of higher education.

* Are criminal background check procedures in public education effective.

* Are discrimination complaints being processed effectively and equitably.


General Slide 13

OLAG Methodology

inputs activities outputs

Statute – Mission -- Objectives

• people• facilities• equipment• supplies• financial ability

ACTUAL—

• service(s)• product(s)

EFFICIENCY

THE MANNER IN WHICH AN AUDITEE USES ITS RESOURCES TO GENERATE SERVICES OR PRODUCTS

Efficiency Audits Compare How Well Inputs Are Used to Achieve

OutputsAre they performing economically and timely?


General Slide 14

District Planned Elementary Utilization

Actual Elementary Utilization

Planned Secondary Utilization

Actual Secondary Utilization

Alpine 62 % 49 % 127 % 87 %

Davis 55 50 102 60

Jordan 70 50 130 74

Murray 53 48 67 54

Park City 43 33 97 53

Salt Lake 76 61 120 81

Washington 65 58 94 83

Average 62 % 53 % 100 % 65 %

Smaller Buses Should Be Considered

In 12 select school districts

194 Class D school buses could have been replaced with Class C buses

For a potential savings of about $5 million

Class C Bus: Capacity = 72 Average Cost = $ 62,700

Class D Bus: Capacity = 84 Average Cost = $ 88,100


General Slide 15


General Slide 16

OLAG Methodology

inputs activities outputs

Statute – Mission -- Objectives

ACTUAL—


EFFECTIVENESS

THE EXTENT TO WHICH AN AUDITEE ACCOMPLISHES ITS OBJECTIVES, AND ULTIMATELY, ITS GOALS

INTENDED—


Effectiveness Audit Compares Intended Output to Actual Output

Are they performing well?


General Slide 17

Effectiveness Audit Compares Intended Output to Actual Output

Are they performing well?


General Slide 18

Audit Found: Management did not listen to the jail reviews

Action Taken: New management took action instituted standards and

consequences for not meeting standards

Jail Contracting: Conducted annual reviews of county jails. They were performed for the most part efficient – but they were not effective?

Audit Phases

• Survey Phase - Assess for risk

• Fieldwork Phase – Audit tests

• Report – Fieldwork areas developed into chapters

• Presentation – Audit is made public


General Slide 19


General Slide 20

Operating Methodology: “Attributes of a Finding”

1. Condition – What is happening?

2. Criteria – What should be happening?

3. Effect – What is the impact of the condition?

4. Cause – What created the condition?

5. Recommendation – What should be done? (corrective or preventative action)


General Slide 21

Operating Methodology: Condition

• What is really happening?

• What is working well?

• What is defective or deficient?

• Is the problem isolated or widespread?


General Slide 22

Operating Methodology: Tests For Condition

• Significant Resources

• Impact Upon the Public

• Management Controls

• Performance Measures

• Understanding of Mission and Goals

• Compliance with Statute and Rules


General Slide 23

Medicaid Fraud, Waste, Abuse: Condition

• 25 Year Old Analytical Tool (since last update)

• Unclear Recovery Methodology

• Employee Productivity

• Employee A = 47,384

• Employee B = 476,265

• Inadequate Performance Measures


General Slide 24

Operating Methodology: Criteria

Examples:

• Laws, policies or rules

• Accepted industry standards

• Goals and objectives

• Practices in other states or private sector

• Expert opinion or auditor opinion

• Generally accepted practices

Question

Which one can you

use?

S t r o

n g

e s t —

W e

a

k e

s t


General Slide 25

Medicaid Fraud, Waste, Abuse: Criteria

•Other state Medicaid programs (for general guidance on all areas)

•Federal laws and rules (for analytical tools)

•Accepted business practices (for employee productivity)


General Slide 26

Operating Methodology: Effect

• What is the impact or consequence?

• What is the extent of difference between criteria and condition?

• Is the difference significant in terms of cost, performance, etc.?


General Slide 27

Medicaid Fraud, Waste, Abuse: Effect

•Only 5% of Medicaid dollars were being reviewed for F,W,A

•National Statistics = $20.2 million

WeakestCriteria

WeakestEffect

Strongest

Criteria

Operating Methodology: Criteria & Effect Relationship

Laws or Mission Statements

Industry or Professional Standards

Historical Data or Goals & Objectives

Expert Opinion

Prudent Business Practices

Auditor Opinion

No Significant Impact

Implied Savings

Small Dollar Impact

Improved Services

Large Dollar Impact

Loss of Life or Public Danger

Strongest

Effect


General Slide 28


General Slide 29

Operating Methodology: Finding the Underlying

Cause• Ask why?

• Need documentation

• Lends strength to the recommendation


General Slide 30

Medicaid Fraud, Waste, Abuse: Cause

•Management

•Budget Constraints

•Lack of Adequate Training


General Slide 31

Operating Methodology: Recommendations

• What Should Be Done to Correct the Problem or Issue:

• A direct relationship between the cause and the recommendation.

• Work with organization to get agreement.

• Change is the overall goal.


General Slide 32

Operating Methodology: Recommendations


General Slide 33

Principles For A Successful Audit Function

• Resist Intimidation

– Understanding the source

– Anticipating the act


General Slide 34

Principles For A Successful Audit Function

• Exit for Understanding

– How: Reveal findings often

– Benefit: Strengthens audit findings

• Obtain Participation

– Recommendations can be joint effort

– Implementation is the key to success


General Slide 35

Contact Information

Kade R. Minchey

Office of the Legislative Auditor General

Utah State Capitol Complex

315 House Building

Salt Lake City, Utah 84114

(801) 538-1033 Office

Website: www.le.utah.gov/audit/olag.htm

Documents

17 May 2011Utah Legislative Auditor GeneralSlide 1 Office of the Legislative Auditor General Performance Audits and Evaluation