View
240
Download
1
Category
Preview:
Citation preview
8/8/2019 ASOSAI Performance Audit Guidelines
1/82
Fifth ASOSAI Research Project
Performance AuditingGuidelines
October 2000
8/8/2019 ASOSAI Performance Audit Guidelines
2/82
2 5th ASOSAI Research ProjectPerformance Auditing Guidelines
8/8/2019 ASOSAI Performance Audit Guidelines
3/82
3
Contents
Foreword
Preface
1 Introduction 7
Performance auditing guidelines 7Definition and scope of performance auditing 8
Performance auditing mandate and objective 10
Performance audit process 10General principles of performance auditing 13
2 Strategic Planning 16
Introduction 16Selecting audit topics 17
3 Initiating the Performance Audit 21
Introduction 21
Planning individual audits 21Understanding the business of an auditee 23
Audit criteria 25
4 Implementing the Performance Audit 30
Introduction 30
Audit approaches 32Refining the audit program 33
Audit test programs 35
Developing findings and recommendations 36
5 Evidence and Documentation 38
Audit evidence 38Competence, relevance and sufficiency of evidence 38
Characteristics of performance audit evidence 40
The evidential process 42
Documentation 44
6 Reporting 48Introduction 48
Reporting process 49
Report contents 52
7 Follow-up Processes 54
Introduction 54Assessing and reporting on follow-up activity 55
8/8/2019 ASOSAI Performance Audit Guidelines
4/82
4 5th ASOSAI Research ProjectPerformance Auditing Guidelines
8 Audit ManagementSome Important Issues 57
Consultation with auditees 57Quality assurance and quality control 59
Appendix A Important considerations for performance auditing in an
Information Technology environment 65Appendix B Potential impacts of performance auditing 70
Appendix C Key features of the auditee and its environment requiringunderstanding 72
Appendix D Important issues in the performance audit of government
programs/activities 75
Glossary of Terms 79
8/8/2019 ASOSAI Performance Audit Guidelines
5/82
5
Foreword
In 1997, at its 6 th Internat ional Seminar , held in Jakar ta , the Asian
Organization of Sup reme Au d it Institutions (ASOSAI) Assembly ap proved
the Jaka rta Declaration of Gu idelines on Prom oting Efficient an d Effective
Pu blic Ad min istration through Performan ce Au d iting. The Declaration
resolved that broad guidelines for performance auditing be developed.
In September 1998 the 26th ASOSAI Governing Board meeting, held in
Beijing, app roved that the 5th A SOSAI Research Project w ou ld be on the
topic of performance auditing guidelines.
The general framework for the development of ASOSAI performance
au d iting gu idelines has been ded u ced from the Jakarta Declaration. Theseguid elines are not a d etailed instruction m anu al and do not replace the
need for management and staff to use their professional judgement to
ensure the d elivery of a quality au dit p rodu ct.
The team for the research project comprised: Mr. Peter Robinson (Team
lead erAustralia), Messrs. I. P. Singh and A . K. Thaku r (Ind ia), Mr. Ab.
Rahma n bin Moham med and Ms.Azizah Arshad (Malaysia) and Mr. John
Oldroyd (New Zealand) .
T he r e sea r ch t e am deve loped a d r a ft o f t he pe r f o r m ance aud i t i ngguidel ines that was c i rcula ted to ASOSAI members for review and
comment in N ovember 1999. Comm ents from ASOSAI members were
taken into account in develop ing the final d raft of the guid elines. The
team wishes to express its appreciation for the contributions made by
m ember Sup reme Aud it Institution s. In October 2000, at the 8th ASOSAI
Assembly held in Chiang Mai, Thailand, the ASOSAI Governing Board
approved the performance auditing guidelines.
8/8/2019 ASOSAI Performance Audit Guidelines
6/82
6 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Preface
Perform ance au d iting is essentially a p rocess that u ses available evidence
to form an op inion on th e extent to w hich an agen cy utilises its resources
in an econom ic, efficient and effective man ner. Althou gh p erforman ce
a u d i t i n g i n v o l v e s t h e u s e o f p r o f e s s i o n a l j u d g e m e n t , a r o b u s t
methodology can provide an objective framework for this judgement.
Performance auditing offers considerable scope for adding real value to
pu blic ad ministration. It also aims to ensu re that ind ividu als and agencies
managing public resources remain accountable for their actions.
To be effective, performance auditors must have a clear understanding
of the objectives and p riorities of any area su bject to aud it. They mu stalso have an un d erstand ing of the p olicies for the p lanning and cond uct
of audi ts , audi t s tandards , audi t methodology and processes of the
particular Supreme Audit Institution (hereinafter referred to as SAI).
These guidelines provide a m ethodology and a broad framework for the
cond uct of the p erformance aud iting process and also p rovide the basis
by wh ich the qu ality of the aud it produ ct can be jud ged. The guid elines
will assist performance auditors:
to be attuned to the needs, expectations and p riorities of clients, wh ichin the case of SAIs are the legislature, government institutions and
those entrusted with the task of managing public affairs;
to add value to agencies through performance audit ing services;
t o p r ov ide an env ir onm en t tha t enab le s pe r fo r m ance aud i to r s t o
enh ance their skills and achieve their fu ll potential;
to achieve audi t excellence; and
to manage p erformance aud it operations efficiently and effectively.
The guidelines take into account relevant INTOSAI Auditing Standardsand are based on generally accepted principles of performance auditing
d is t il led f rom th e exp er ience of ASOSAI memb ers . They therefore
comprise contemporary performance auditing methodology and reflect
a bes t pract ice consensus for the cur rent environment , among the
mem ber SAIs. The guid elines are a living d ocum ent, w hich will need
to be upd ated as the environment changes and as p erformance aud iting
methodology and practice develop.
8/8/2019 ASOSAI Performance Audit Guidelines
7/82
7
1. Introduction
Performance auditing guidelines1.1 These guid elines p rovid e a framew ork for iden tifying, collecting
and analysing information an d provid e practical guidan ce to:
assist SAI performance auditors to manage and condu ct performance
audits ;
prom ote consistent, economical, efficient an d effective performance
audit practice;
e st a b li sh a b a s is fo r fu r t h e r d e v e lo p m e n t o f p e r fo r m a n ce a u d i t
methodology and professional development; and
set out a basic framework within w hich professional jud gement may
be exercised on analysing performance and reporting conclusions.
1.2 These gu ide l ines a re des igned for use by SAI audi tor s and
consultants who are involved in planning, carrying out, reporting and
man aging a performance audit . As such, the guid eline chap ters have
been structured in line with the major stages of the performance audit
process . The guidel ines wil l help audi tors to deal wi th the complex
demands of performance auditing.
1.3 The greater u se of information techn ology by ag encies, includ ing
the Internet, has meant that performance audits will be increasingly
un d ertaken in information techn ology environments. As SAIs may be
called up on to develop d etailed guidelines for p erformance aud iting in
an information technology environment, some of the more important
considerations hav e been outlined in Ap pend ix A.
1.4 Perform ance aud itors can be faced w ith consid erable variety and
ambigu ity in their work. They need skills in analysing organisationalactivities and m anag emen t practices. They can be faced w ith the need to
become fam iliar with a w ide ran ge of organ isational contexts and su bject
ma tters. They need the ability to write reports dealing with comp lex
issu es in a logical and thorou ghly sup por ted fashion. The gu idelines can
provide some assistance in these areas, but it is largely incumbent on
p erform ance aud itors to d evelop their skills in these areas by other m eans.
8/8/2019 ASOSAI Performance Audit Guidelines
8/82
8 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Definition and scope of performance auditing
1.5 Th e INTOSAI Auditing Standards1 define a performance audit
as an audit of the economy, efficiency and effectiveness with which the audited
ent ity uses its resources in carrying out its responsibilities.2
1.6 Economy is min imising the cost of resources used for an activit y, having
regard to the appropriate quality.3 Econom y issues focu s on t he cost of the
inpu ts and p rocesses. Econom y occurs w here equ al-qu ality resour ces
are acquired at lower prices.
1.7 Efficiency is the relationship between the output, in terms of goods,
services or other results, and the resources used to produce them.4 Efficiency
exis ts where the use of f inancia l , human, phys ical and informat ion
resources is such that output is maximised for any given set of resource
inpu ts, or inpu t is minimised for any given qu antity and quality of outp ut.
1.8 Effectiveness is the extent to which objectives are achieved and the
relationship between the intended impact and the actual impact of an activity.5
Effectiveness add resses the issue of whether th e program / activity has
achieved its objectives. When focusing on effectiven ess, it is imp ortant
to dis t inguish between the immediate outputs (or products) and the
u ltimate imp acts (or ou tcomes). Effectiveness is achieved , for instan ce,
w here there is imp roved achievement of a progr ams objectives. Ou tcomes
are important to the effectiveness of program s/ activities but may be more
difficult to measure and assess than the inpu ts and outp uts. Ou tcomes
will often be influenced by external factors and may require long-term
rather than short-term assessment.
1.9 Performance audit ing, also referred to as Value For Money
aud i t ing , shares s imi la r app roaches and method ologies to p rogram
evaluation bu t d oes not gen erally extend to assessing p olicy effectiveness
or policy alternatives. Ap art from examining the impa ct of outp uts, an
evaluation could includ e issues such as w hether ag ency plann ing reflected
the p rogram objectives, w heth er the ob jectives were consistent w ith p olicyand op tions for chan ging th e policy to achieve m ore effective outcomes.
1 Auditing Standards, Auditing Standards Committee, International Organization of Supreme Audit
Institutions, June 1992, p. 69.
2 Refer to Appendix B for examples of potential impacts of performance auditing.
3
Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, p. 664 Ibid, p. 67
5 Ibid, p. 67
8/8/2019 ASOSAI Performance Audit Guidelines
9/82
9
1.10 Performance auditing promotes public accountability and is an
aid to good corp orate governan ce. It also encomp asses the concept of
trad itional financial, compliance and p rop riety au d its. Com pliance au d its
d eal w ith regularity and legality aspects. Prop riety refers to the concept
of the best pr actice in pu blic man agem ent of pr ogram s/ activitiespu blic
funds should not be misused by the managers for personal benefit and
expend iture on program s should not exceed w hat the occasion deman ds.
Performance auditors should utilise these requirements in the course of
their audit, wherever applicable.
1.11 The performan ce au d itor will also be exp ected to ad d ress concerns
re la t ing to equi ty and e thics whi le assess ing the ef fect iveness of a
program/ activity. Equity in the context of program m anagement relates
to fairness and imp artiality in use of pu blic fund s. Ethics in ma nagin g
publ ic expendi ture enjoins the qual i t ies of hones ty and integr i ty inpersonal conduct and devotion to the duty as manager of public funds.
The SAI performance auditor should be conscious of equity and ethics
issues.
1.12 Performance aud its may examine and report on:
the qual ity of informat ion and advice available to Government for
the formulation of policy;
the existence and effectiveness of adm inistrative machinery in place
to inform the Government whether program objectives and targets
have been determined with a view to fulfilling policy objectives;
wh ether, and to wh at extent , s ta ted program object ives have been
met;
the economy, efficiency, effectiveness and ethics of the means u sed to
implement a program / activity; and
the intended and u nintended direct and indirect impacts of programs/
ac t ivi t ies ; for example , the environmental impact of government
activity.
1.13 Auditors should not confine the audit to what has been done
but should also examine what has not been done to meet the policy
objectives. Given the size, comp lexity and d iversity of agency operations,
it is generally imp racticable to attem pt to assess the overall performan ce
of d epartm ents or agencies. Consequ ently, performa nce aud its are usually
directed towards specific functions, activities, programs or operations
of the agency.
Introduction
8/8/2019 ASOSAI Performance Audit Guidelines
10/82
10 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Performance auditing mandate and objective
1.14 The aud it mand ate is generally set out in statute and dictates the
extent to w hich an SAI can aud it pu blic sector agencies. Where the
mandate is derived out of a broad interpretation of the term audit or
has been accepted by convention, auditors should be guided by their
best ju d gemen t in the context of any su ch conven tion. The performa nce
aud i t mand ate ordinar ily specifies the m inimu m aud i t and repor t ing
requirements; indicates what is required of the auditor ; and provides
the auditor with authority to carry out the work and report the results.
Accordingly, the auditor should be guided by the audit mandate when
planning the au d it. In m ost coun tries performance aud its are excluded
from commenting on Government policy.
1.15 Perform ance aud iting h as the objective of imp roving p u blic sectoradm inistration an d accoun tability by ad ding value throu gh an effective
program of per formance audi ts and re la ted products such as bet ter
pr actice gu ides. One of the main objectives of p erform ance aud iting is
to assist the peoples representatives in exercising effective legislative
control and oversight.
1.16 Performance aud its hav e a du al role. They p rovid e clients w ith
information and assurance about the quality of management of public
resources and they also assist pu blic sector m anag ers by ident ifying an d
prom oting bet ter managem ent pract ices . Per formance audi t ing maytherefore lead to better accoun tability, imp roved econom y a nd efficiency
in th e acquisition of resources, im pr oved effectiveness in a chieving p ub lic
sector p rogram objectives, a higher qu ality in pu blic sector service delivery
and imp roved man agement planning and control. The emp hasis placed
on the assurance role of performance auditing over the role of public
sector improvement will vary between SAIs.
Performance audit process
1.17 The first stage in p erforma nce aud iting is strategic plann ing, wh ich
requires the developm ent and maintenance of information on th e agency
that w ill assist in id entifying poten tial areas for aud it. Potential top ics
can then be analysed and ranked to form annu al audit strategy docum ents
for each agency. Chap ter 2 of these guid elines deals fu rther w ith strategic
planning.
8/8/2019 ASOSAI Performance Audit Guidelines
11/82
11
1.18 On ce a top ic has been selected for p erform ance aud it, the au d it is
initiated by the d evelopm ent of a plan d etailing th e cond uct of the au dit.
A preliminary study may be undertaken to gather information on the
topic and to identify s ignif icant issues for audit . Chapter 3 of these
guid elines deals with initiating the p erformance au d it.
1.19 The imp lementation stage of a perform ance aud it is dealt with in
particular in Chapters 4 and 5 of these guidelines and involves:
development and execut ion of an audi t program of procedures;
co ll ect ion and docum en ta t i on o f su f fi ci en t r e levan t and r e li abl e
evidence, including quantitative and qualitative analysis;
formulation of aud it findings, conclusions and recommend ations; and
development of discussion papers and confirmation of aud it find ings
at an exit conference.
1.20 A report on the audit may then be drafted for consideration by
the government, the legislature, the agency and th e pu blic. Throu ghou t
each s tage of the per formance audi t the emphasis should be on the
prod uction of a final report that is balanced an d has value ad d ed imp act.
The report-writ ing process should be viewed as a continuous one of
formu lating, testing an d revising conclusions, if necessary, about the au d it
topic. Therefore, issues such as the expected imp act and va lue of the
aud it, the likely imp rovements and savings resulting from the au d it andmethods of communicat ing audi t conclus ions should be cons idered
throu ghou t the aud it. Chapter6 of these guidelines deals furth er with
reporting.
1.21 Follow-up procedures should id entify and docum ent aud it imp act
and the progress of the agency in imp lementing au dit recomm end ations.
The cond u ct of follow -u p au dits is vital to the follow-up process to provid e
feedback to the SAI, the legislature and the government on performance
a u d i t e f f e c t i v e n e s s i n p r o d u c i n g i m p r o v e m e n t s i n p u b l i c s e c t o r
management. Follow-up processes are dealt with in Chapter7 of theseguidelines.
1.22 The key stages in the performance audit cycle are outl ined in
Figure 1.
Introduction
8/8/2019 ASOSAI Performance Audit Guidelines
12/82
12 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Figure 1Key Stages in the Performance Audit Cycle
Information
feedback
to the
strategic
planning
process
8/8/2019 ASOSAI Performance Audit Guidelines
13/82
13
General principles of performance auditing
Auditing standards
1.23 In conduct ing a per formance audi t the audi tor should fol low
INTOSAI Audi t ing Standards as we l l a s r e l evan t S AI s t anda r ds
applicable to performance audits.
1.24 Th e INTOSAI general auditing standards common to aud itors
and SAIs are:
(a) The auditor and the SA I must be independent ;
(b) The auditor and the SA I must possess the required competence; and
(c) The auditor and the SA I must exercise due care and concern in
complying with the IN TOSA I Auditing S tandards. This embraces
due care in planning, specifying, gathering and evaluating evidence,and in reporting findings, conclusions and recommendations. 6
1.25 The performance auditor and SAI must be, and be seen to be,
indep end ent and objective in the cond uct of aud its. That is , the SAI
mu st have freedom to act in accordance with its audit mandate without external
direction or interference of any kind.7
1.26 Th e INTOSAI general auditing standards also note that SAIs
should:
(a) Recruit personnel with su itable qualifications;
(b) Develop and t rain SA I employees to enable them to perform their
tasks effectively, and to define the basis for the advancement of
auditors and other staff;
(c) Prepare manuals and other written guidance and inst ructions
concerning the conduct of audits;
(d) Su pport the skills and experience available wit hin the SA I and
identify the skills which are absent; provide a good distribution of
skills to auditing tasks and assign a sufficient nu mber of persons forthe audit; and have proper planning and supervision to achieve its
goals at the required level of due care and concern; and
(e) Review the efficiency and effectiveness of the SAIs internal standards
and procedures. 8
Introduction
6
Ibid, p. 237 Ibid, pp. 24, 68
8 Ibid, p. 23
8/8/2019 ASOSAI Performance Audit Guidelines
14/82
14 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Professional behaviour
1.27 The aud itor shou ld comp ly with ethical principles and codes of
c o n d u c t g o v e r n i n g t h e a u d i t o r s p r o f e s s i o n a l b e h a v i o u r a n d
respon sibilities, w hich includ e:
in tegrity;
object ivity and fa irness;
confidentia lity; and
t echn ica l st anda rds .
1.28 Althou gh th ese gu idelines set out a coherent basis for condu cting
a performance audit, professional judgement (albeit largely applied on
the basis of relevant ru les and procedures) remains the m ost imp ortant
ingredient in performance audit work.
1.29 The performance aud itor should adop t an attitud e of professional
scepticism th rou ghou t the au d it, recognising that circum stances may exist
that could cause the information relating to performan ce to be materially
mis-stated.
Reasonable assurance
1.30 A performance audit conducted in accordance with applicable
audi t ing s tandards provides reasonable assurance as to whether the
information relating to performa nce is free from mat erial mis-statemen t.
Reasonable assurance re la tes to the accumulat ion of audi t evidence
necessary for the auditor to conclude whether there are any mater ial
mis-statements in the information relating to performance.
1.31 What is reasonable is dependent on the facts of that situation
and is to be d etermined by w hat evidence could reasonably be expected
to be gathered and wh at conclusions could reasonably be expected to be
draw n in the p articular situation.
Terms of the engagement1.32 The SAI should formally notify the auditee 9 of the details of the
engagement, preferably before the commencement of the audit, to help
in avoid ing any misun d er s tand ings . Wi th some engagem ents , the
objec t ive and scope of the audi t and the audi tor s ob l iga t ions a re
established by law. Even in those situ ations id entifying the terms of
engagement may still be informative for the agency.
9 Auditee refers to the agency being audited.
8/8/2019 ASOSAI Performance Audit Guidelines
15/82
15
Work performed by assistants
1.33 INTOSAI AuditingStandards10 state: The work of the audit staff at
each level and audit phase should be properly supervised during the audit, and
docum ented work should be reviewed by a senior member of the audit staff. When
work is delegated to members of the audi t team, the audi tor shouldcarefully direct, supervise and review the work delegated.
1.34 When mu lti-d isciplinary au d it teams are u sed, adequ ate direction,
supervis ion and review are necessary to ensure that team members
d ifferent p erspectives, experience and specialties are app rop riately u sed
in the aud it. All team m embers shou ld u nd erstand the objectives of the
aud it, the terms of reference of work assigned to them and the natu re of
obligations imposed on them by app licable aud iting stand ards. Adequ ate
d irection, sup ervision an d review w ill assist in such an un d erstand ing.
Using the work of an expert
1.35 When u sing the w ork performed by an expert, the aud itor should
obtain sufficient ap prop riate audit evidence to ensure that such w ork is
adequate for the purposes of the audit.
1.36 An exp ert is a person or firm p ossessing sp ecial skill, know ledge
and experience in a particu lar field oth er than au d iting. Becau se of the
diverse range of activities subject to performance auditing, the auditor
may need to obta in audi t evidence in the form of repor ts , opinions ,
valuations and statements of an expert. Although the aud itor may u se
the wor k o f an expe r t a s aud i t ev idence , t he aud i to r r e t a in s f u l l
responsibility for the conclusions in the audit report.
10 Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, p. 45
Introduction
8/8/2019 ASOSAI Performance Audit Guidelines
16/82
16 5th ASOSAI Research ProjectPerformance Auditing Guidelines
2. Strategic Planning
Introduction2.1 The SAI needs to select and sched ule au d it tasks that help achieve
its role and corpora te objectives. Au d it top ics shou ld be ran ked , for
examp le u sing a risk-based ap pr oach to topic selection, in ord er to make
the best use of lim ited SAI resou rces. For this pu rp ose the SAI ma y
prepare a program model which identifies the programs, projects and
activit ies undertaken by an agency, along with other factors such as
program objec t ives , expec ted ou tputs , ou tcomes and per formance
ind icators. The selection of top ics for perform ance aud it can be based
on a combination of relevant risk and materiality factors.
2.2 A well-structured strategic planning process, based on a sound
rationale, is necessary to ensu re tha t the resources of the SAI are u sed in
the most efficient an d effective m anner. The objectives of strat egic au d it
planning are to:
provid e a firm basis for the SAI man agement to give strategic direction
for futu re au dit coverage;
identify and select aud its with the potential to improve public sector
accountability and administration;
provide a p latform for commu nication w ith agencies and the legislature
on SAI audit strategies;
produ ce a work program that can be achieved with expected available
resources;
und erstand agency r isks and take them into account in audit selection;
an d
provide a basis for SAI accountabil ity.
2.3 The key d eliverable of the strategic plan ning p rocess is a d ocum ent
w hich is prep ared for the SAI man agem ent to en able it to critically assess
the proposed planning strategy for overall consistency with the SAIs
corporate objectives. It w ill also assist ma nagem ent to make ap pr op riate
resource allocations an d an assessment of the strategic plan ning pr ocess.
8/8/2019 ASOSAI Performance Audit Guidelines
17/82
17
2.4 The strategic plan should be supported by working papers that
include:
a descr ipt ion of the agency and i ts environment ;
an a s s es sm en t o f t he r is ks t o good p e r fo r m ance o f t he agencys
program s and activities;
a summ ary of the long-term strategic view of SAI performance audit
d irections in each agen cy;
recent SAI aud its , recent and prop osed inquir ies by the legislature,
and agency evaluations and internal aud its;
a l is t of potential areas for performance audit ; and
a l is t of aud its proposed for the next two or three years, the basis for
their selection and ind icative timin g wh ere possible. This list should
reflect resource availability and should also distinguish between newperformance audit topics, cross-agency audits and follow-up audits.
Selecting audit topics
2.5 Audit topics are generally selected on two grounds: firstly, to
focus on audits expected to add maximum value in terms of improved
accoun tability, econom y, efficiency and effectiveness; second ly, to en su re
appropriate coverage of program operations within the l imitations of
au d it resources available. The strategic plan ning p rocess is outlined in
Figure2.
Figure 2The Strategic Planning Process
Inputs
Potential audits
Priorities
Resourcerequirements
Audit Strategy Plan
Outputs
Budget
Previous audit strategy planning
Government views, budget papers, etc
Agency annual reports and evaluations
Media and external reports
Previous audit fieldwork
Analysis of performance indicators
Discussions with agencies, clients, etc.
Priorities of legislature
Priorities of government
Strategic Planning
8/8/2019 ASOSAI Performance Audit Guidelines
18/82
18 5th ASOSAI Research ProjectPerformance Auditing Guidelines
2.6 The analysis of risks of poor performance or, expressed another
way, risks of inadequate economy, efficiency and effectiveness will lead
to a list of p otential aud it topics. I t can be u sefu l to rank th e topics
subjectively against the following criteria:
overa ll est imated audit impact ; financia l mate ria lity;
r isk t o good m anagem en t;
significance of the program to the activities of the agency;
visibility of the program / activity as reflected in its political sensitivity
and national imp ortance; and
lack of recent audit coverage and other internal and external review
of the program/ activity.
Overall estimated audit impact
2.7 Of major importance in the final selection of topics is the added
value expected from the au d it. A p reliminary assessment of the au d its
likely benefits should be made at this strategic planning stage.
2.8 Appendix B lists some of the potential impacts of performance
auditing and classifies the benefits by reference to: economy; efficiency;
effectiveness; quality of service; p lanning , control an d man agem ent; and
accou ntability. Qu an tification is d esirable bu t un likely to be feasible at
the strategic planning stage.
Financial materiality
2.9 This criterion is based on an a ssessm ent of the total valu e of assets,
liabilities, annual expenditure and annual revenue of the selected audit
area. The more m aterial an area is, the h igher is its priority for selection
as an aud it topic.
Risks to good management
2.10 Assessment of risks to good performance in the agency requiresthe SAI to assess whether the management of the activity to be audited
is likely to be deficient in economy, efficiency and effectiveness.
2.11 Evidence of risk to good management includes:
management inaction in response to identified w eaknesses;
adverse comment by the legis la ture or media;
non-achievement of stated objectives such as revenu e raised or clients
assis ted;
h igh st aff t u rnover ;
8/8/2019 ASOSAI Performance Audit Guidelines
19/82
19
identified weaknesses in internal control;
s ignificant und erspending or overspending;
sudden program expansion or contract ion; and
overlapp ing or blurred accountabil ity relationships.
2.12 An agency p rogram or activity that is more comp lex to man age
and operates in an un certain environment is more likely to have p roblems
associated w ith p erforman ce. Some p ossible ind icators of high comp lexity
and uncertainty are:
highly decentralised operations with devolved management decision-
making responsibilities;
a mu lt ipl icity of interes ted p ar ties;
use of rapidly changing and sophisticated technology; a dynamic and compet it ive environment ; and
controversial social and p oli tical debate surrounding the issue.
2.13 The stage of the agencys program development should also be
kept in mind w hen assessing m anagement p erformance. For examp le, in
the development stages it will be particularly important for the agency
management to set measurable, operational objectives, which clear ly
identify how the program will contr ibute to the agencys objectives.
During p rogram imp lementation it will be imp ortant to ascertain w hetherapp ropriate performance measures are maintained and analysed to assess
performance, and whether there is a clear identif ication of roles and
respon sibilities for each level of the prog ram / activity. If the progr am
has been in place for some time it will be important to assess whether a
formal evaluation h as been u nd ertaken to ascertain wh ether the program
is continuing to m eet relevant n eeds and the extent to w hich th ose needs
still exist or are being met by other programs.
Significance
2.14 The s ignif icance of an audi t topic should have regard to the
magn itude of its organisational impacts. It will dep end on wh ether the
activity is comparatively minor and whether shortcomings in the area
concerned could flow on to other activities within the agency.
2.15 Significance will rate highly where the topic is considered to be
of particular importance to the agency and where improvement would
hav e a significant imp act on the operations of the agency. A low ran king
in relation t o significance w ou ld be expected w here th e activity is of a
routine natu re and the imp act of poor performan ce wou ld be restrictedto a sma ll area or be likely to hav e minim al impa ct.
Strategic Planning
8/8/2019 ASOSAI Performance Audit Guidelines
20/82
20 5th ASOSAI Research ProjectPerformance Auditing Guidelines
2.16 Cross-agency au d its are more likely to ran k high ly on significance.
Visibility
2.17 This criterion is similar to significance bu t is m ore concerned w ith
the external imp act of the p rogram . It is related to the social, econom ic
and environmental aspects of the act ivi ty and the impor tance of i t s
opera tions to the governm ent and the p ub lic. In considering th is criterion
some w eight w ould be attached to the impact of an error or irregularity
on p ub lic accoun tabili ty. It wou ld also have regard to the d egree of
interest by the legislatu re and pu blic in the outcom e of the au d it. Sub jects
that have been identified as current themes for the SAI would generally
w arran t a h igh ran king in terms of visibility.
Coverage
2.18 Coverage refers not only to previous SAI coverage but also to
other ind epen d ent review s of the activity. Such reviews may hav e been
conduc ted by in t e r na l aud i t , ex t e r na l cons u l t an t s o r gove r nm en t
comm ittees or the activity could h ave been su bject to prog ram evaluation.
As a general rule, a low ranking would occur when there has been a
substantial review of the activity within the past tw o years. A higher
ranking would be warranted where a review has been requested by the
legislature or the previou s review ind icated th at such a follow -up sh ould
occur.
2.19 The m ateriality, risk, significance and visibility of an activity w ill
also influ ence the ran king for coverage. If an activity has ran ked h ighly
on all or most of these elements , i t would be expected that the audit
coverage wou ld be more frequen t than for a lower ranked top ic. The
frequ ency of au d it coverage w ould also d epen d on the SAI strategic au d it
plan and on the availability of resources.
Cross-agency and theme audits
2.20 SAI audi t manager s should cons ider appropr ia te themes in
prop osing aud its. Au d its that cross several agencies, ad d ressing themes
w hich are of relevance to the entire p u blic sector or a d d ressing significant
nat ional concerns , can have very high imp act . Potent ia l theme and
cross-agency au d its can be evalua ted in th e same w ay as other au d its for
inclusion in the aud it pr ogram . That is, specific topics w hich rank mor e
highly in terms of risk, impact and materiality should receive priority.
8/8/2019 ASOSAI Performance Audit Guidelines
21/82
21
3. Initiating the Performance
Audit
Introduction
3.1 This chapte r ou t l ines the s teps involved wi th in i t i a t ing the
performance audit: the approach to be followed in planning individual
aud i t s , t he r equ i r em en t t o unde r s t and the aud i t e e s bus ine s s and
environment and the development of audit criteria.
Planning individual audits
3.2 INTOSAI Auditing Standards11 identify that the following are
the planning steps ordinarily includ ed in an au dit:
(a) collect in formation about the audited agency and it s organisation
in order to assess risk and t o determine materiality;
(b) define the objective and scope of the audit ;
(c) un dertake preliminary analysis to determine the approach to be
adopted and the nature and extent of enquiries to be made later;
(d) highlight special problems foreseen when plann ing the audit ;(e) prepare a budget and a schedule for the audit ;
(f) ident ify staff requirements and a team for the audit; and
(g) discuss wit h the audit ed agency t he scope, objectives and the
assessment criteria of the audit .
3.3 Planning consists of d eveloping a general strategy and a d etailed
app roach for the expected n ature, timing and extent of the aud it. The
audit plan is a key document for controlling and monitoring audits in
the SAI . The aud i tor shou ld deve lop and d ocum ent an aud i t p land escribing the expected scope and cond uct of the aud it and shou ld p lan
to have the audit work performed in an efficient, effective and timely
man ner. In add ition to the planning steps outlined above, the aud itor
should also consider:
determining the suitabili ty of aud it cr iteria;
determining an efficient and effective approach to cond ucting the audit;
11 Ibid, p. 44
8/8/2019 ASOSAI Performance Audit Guidelines
22/82
22 5th ASOSAI Research ProjectPerformance Auditing Guidelines
p lanning to use the work of an expert ;
relevant accountabil ity relationships;
remaining mindful of the users of audi t repor ts ;
reviewing the audi tees internal audi t sys tem;
reviewing re levant p revious audi t findings/ recommendat ions; and
docum ent ing t he aud it p lan.
3.4 A d e q u a t e p l a n n i n g o f t h e a u d i t w o r k h e l p s t o e n su r e t h a t
appropriate attention is devoted to important areas of the audit , that
po t en t i a l p r ob l em s a r e i den t i f i ed and tha t t he wor k i s com ple t ed
exped itiou sly. Planning also assists in p roper assignm ent of w ork to team
mem bers and in coordination of work p erformed by other aud itors and
experts. Aud it plann ing shou ld lead to the developm ent of a detailed
aud it program that id entifies the sp ecific aud it tasks to be un dertaken.
Determining the objectives of a performance audit
3.5 Audit objectives relate to the reasons for conducting the audit
and shou ld be established early in th e aud it process to assist in identifying
the matters to be aud ited and reported. For examp le, an au dit objective
ma y be to assess w heth er an agency uses app rop riate processes to identify
the human resources it needs to achieve its objectives.
3.6 The aud it objectives should ad d ress the concerns of accoun tability
and good governance and should facilitate financial control as well as
im pr ov ing the econom y, e f f i c i ency and e f f ec t i venes s o f p r og r am
management.
3.7 T he aud i t ob j ec t i ve s s hou ld be cond i t i oned by the need to
ma ximise the net benefits and imp acts from the au d it. In setting objectives
the audit team should take into account the roles and responsibilities of
the SAI and the expected net imp act of the au d it as defined in th e strategic
aud it plan. The au dit objectives and scope are interrelated and , since
changes in one usually affect the other , they need to be consideredtogether. It is good m anag emen t practice for aud it objectives and scop e
to be discussed w ith the agency man agement.
The audit program
3.8 The auditor should develop and document an audit program. 12
An audit program outlines the requirements and procedures necessary
12 Refer to Chapter 4 for more details on developing the audit program.
8/8/2019 ASOSAI Performance Audit Guidelines
23/82
23
to imp lement the au dit objectives and to m ake assessments against au dit
criteria. The main objectives of an au d it program are to:
e s ta b l is h a cl ea r r e la t i on s h i p b e t w e e n a u d i t o b je ct i v es , a u d i t
methodology, and the anticipated field work to be carried out;
ident ify and d ocument the procedures to be per formed; and
facilita te supervis ion and review.
3.9 An au dit p rogram shou ld set out and specify the following:
aud it objectives and sub-objectives concerning the efficient, economical
or effective operations of the area or agency being audited;
audit cr it er ia to be appl ied ;
the specific tasks to be performed d uring the fieldwork to determine
the extent to w hich agency operations m eet the requirements ou tlined
in audit criteria;
evidence to be collected, includ ing details of aud it sampling and the
timeframe of the activity to be covered;
procedures/ techniques for collecting the evidence;
the a lloca t ion and t iming of t a sks to be per formed by audi t t eam
members; and
any necessary specia l ins truct ions.
3.10 Progress against the planned aud it program should be monitoredand regular briefings provid ed to SAI man agement. The aud it program
should be revised as necessary du ring the course of the au d it.
Understanding the business of an auditee
3.11 It is imp ortant to develop a soun d un derstand ing of the aud itees
bus iness suf f ic ient to achieve the audi t object ives , to faci l i ta te the
identif ication of s ignif icant audit issues and to fulf i l assigned audit
responsibilities. This knowledge w ould includ e an un derstand ing of:
the manda te of the agency and the a reas being aud ited w ith in theagency;
objectives of the agency and relevant programs;
programs and per formance goals of the agency;
organisational and accountabil ity relationships w ithin the agency;
t h e in t e r n a l a n d e x te r n a l e n v ir o n m e n t of t h e ag e n cy a n d t h e
stakeholders;
external constraints affecting program delivery;
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
24/82
24 5th ASOSAI Research ProjectPerformance Auditing Guidelines
agency management processes and operat ions; and
resources of the agency.
Appendix C de ta i l s a r ange of key f ea tures , o f the audi tee and i t s
environment, that need to be understood by the auditor.
3.12 Obtaining the required k now ledge of the bu siness is a continu ous
and cumulative process of gather ing and assessing information, and
relating the resultant knowledge to audit evidence at all stages of the
aud i t . I t i s impor tant that au di tors weigh u p the costs of obta ining
information and the addit ional value of the information to the audit
outcomes, to ensure that m aximu m value is being obtained from th e aud it
resources expended.
3.13 Sources of information to assis t in understanding the agency
include:
enabling legislation and legislative speeches;
ministerial statements , government submissions and decisions;
r ecen t aud i t r epo r t s , r ev iews , eva lua t ions and inqu i r ie s i n to t he
agency;
the agencys st ra tegic and corporate plans , mission sta tement and
annu al report;
b ud g et statem en ts;
agency pol icy fi le s, man agement comm it tee and execut ive board
minutes;
agency organisation charts , internal guidelines and op erating manuals;
the agencys program evaluation and internal aud it plans and reports;
confe rence r eport s and minutes ;
discussions with agency management and key stakeholders; and
management in formation systems .
3.14 Past audits and other reviews can provide an extremely useful
source of in format ion . They can he lp avoid un necessa ry work in
examining areas that have been under recent scrut iny and highl ight
d eficiencies tha t ha ve not yet been remed ied. There is, how ever, no
substi tute for discussions with senior agency management to gain an
ove r a ll p r og r am pe r s pec t ive aga ins t t he backgr ound o f t he above
information.
8/8/2019 ASOSAI Performance Audit Guidelines
25/82
25
3.15 Perform ance mon itoring, accou ntability and eva luation p rocesses
w ithin the pu blic sector are generally agency-based. How ever, there
could a lso be a range of inform ation, wh ich crosses agencies, w hich ma y
also assist in the information-gathering stage, such as:
studies by indu stry, professional or special interest groups;
inquiries or previous reviews by the legislature;
informat ion held by coordinat ing agencies or by interdepar tmental
committees;
r esearch by academics;
work under taken by other governments overseas ; and
m ed ia cov era ge.
Role of preliminary study3.16 Once a topic has been selected for audit , the audit team may
conduct a preliminary study to fur ther understand the activity under
aud it and id entify fun dam ental issues. The preliminary stud y aims to
p rovid e sufficient ju stification to p roceed to a p erform ance au d it or else
to conclud e any fur th er work and report findings. At the end of the
p r e l i m i n a r y s t u d y t h e r e w i l l n o r m a l l y b e a r e p o r t t o s e n i o r S A I
man agement that sum marises the findings of the study an d recommend s
future action.
3.17 If a performance audit is recommended, the preliminary study
report should identify the major issues to be pursued, define the audit
objectives, scope and focus, estimate potential impacts and develop a
timetable and resource bud get to conclud e a timely and defensible audit
r epor t .
3.18 If a p erformance aud it is not recomm ended , the preliminary stud y
repor t should su mm arise the stud y conclusions. To finalise the preliminary
s tudy f ie ldwork, an exi t conference should be held with the agency
management.
Audit criteria
3.19 A u d i t c r i t e r i a a r e r e a s o n a b l e a n d a t t a i n a b l e s t a n d a r d s o f
performance against which economy, eff iciency and effectiveness of
activities can be assessed. They reflect a nor m ative (ie. d esirable) contro l
model for the subject matter under review. They represent good practicea
reasonable and informed persons expectation of what should be.
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
26/82
26 5th ASOSAI Research ProjectPerformance Auditing Guidelines
3.20 When cr i ter ia are compared with what ac tual ly exis ts , audi t
find ings are generated . Meeting or exceeding th e criteria might ind icate
b e s t p r a c t i c e , b u t f a i l i n g t o m e e t c r i t e r i a w o u l d i n d i c a t e t h a t
improvements can be made.
3.21 Typically, an audit will commence, following its identification bythe SAI p lann ing process, w ith one or more issues to be explored . To
explore these issues then requires the formulation of criteria, followed
by development of test programs to assess whether the cr i ter ia have
been met . But , in prep ar ing these programs and car rying them ou t ,
aud itors may realise that the issues previously d efined m ay need to be
extended or m odified.
3.22 The auditor should be satisfied that suitable criteria have been
identified to enable the auditor to assess the activities subject to audit
and to achieve the au d it objectives. Som e cha racteristics of suitab le criteria
include:
Reliability:Reliable criteria result in consistent conclusions w hen u sed
by an other au ditor in th e same circum stances.
Objectivity: Objective criteria are free from any bias of the au d itor or
management.
Usefulness: Useful criteria result in find ings and conclusions that m eet
users information needs.
Understandability: Understandable criteria are clearly stated and are
not subject to significantly different interpretations.
Comparability: Comparable criteria are consistent with those used in
performance audits of other similar agencies or activities and with
those used in p revious performance aud its within the agency.
Completeness:Com pleteness refers to the dev elopm ent of all sign ifican t
criteria appropriate to assessing performance in the circumstances.
Acceptabi l i ty: Acceptable cr i ter ia are those to which the audi tedagency, legislatu re, med ia and general p u blic are generally agreeable.
The high er th e d egree of acceptan ce of the criteria, the m ore effective
the performance audit.
3.23 Criter ia can perform a ser ies of important roles to assis t the
cond uct of a performance aud it as they can:
form a common basis for commu nication within the aud it team and
w ith SAI management concerning the n ature of the aud it;
8/8/2019 ASOSAI Performance Audit Guidelines
27/82
27
form a basis for commu nication with the agency management in that
the audit team will often solicit agency management understanding
of, and concu rrence with th e criteria and eventu al accep tance of au d it
find ings in light of those criteria;
l ink the objectives to the aud it test programs carr ied out d uring theimplementation phase;
form a basis for the data collection ph ase of the aud it, providing a
basis on w hich to bu ild pr oced u res for the collection of aud it evidence;
an d
provide the basis for audit findings, helping to add form and structure
to audit observations.
3.24 The degree to which criteria are successful in serving these uses
is of ten determined by their level of deta i l and the form they take.General aud it criteria are developed d uring the p reliminary stud y. As
the p reliminary stud y p rogresses these criteria are u sually expand ed and
ma d e more specific. By the end of the p reliminary stu d y, the criteria
s h o u l d b e s u f f i c i e n t l y d e t a i l e d t o g i v e c l e a r g u i d a n c e f o r t h e
implementation stage of the audit, particularly in the development of
specific audit programs to test the criteria.
3.25 I t is unrealis t ic to expect that activit ies , systems or levels of
p erform ance relating to econom y, efficiency an d effectiveness will alway s
fully meet the criteria. I t is imp ortan t to app reciate that satisfactoryperformance does not mean perfect p erformance but is based on w hat a
reasonab le person w ou ld expect, taking into account agency circumstan ces.
The general a im would, however , be high level per formance within
resource constraints.
3.26 SAIs should note the implications for performance auditing of
the emerging shift in the role of many governments from a provider of
services to a facilitator an d regulator and the increasing u se of private
operator s by the pu blic sector. The performan ce au d it focu s, on the
economy, efficiency and effectiveness of public sector activity, may have
to be modi f ied to take account of the inc reas ing pr iva t i sa t ion and
liberalisation of pu blic sector op erations in m any countries. SAIs will be
called upon to devise new approaches, techniques and criteria for the
per formance audi t of programs that may not be direct ly funded by
governments, but may affect the public at large by way of quality of
service, cost and equ ity of access. In this environm ent, p erforman ce
au d iting n eeds to take into account th e shift from a govern men t-centred
to a more people-oriented approach.
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
28/82
28 5th ASOSAI Research ProjectPerformance Auditing Guidelines
3.27 It is generally useful to obtain agency management input to the
d evelopmen t of criteria. Au d it criteria w ould typ ically be exposed to
the agency at the start of the main p erformance aud it. Any d isagreement
w ith agency man agem ent abou t criteria can then be identified, discussed
and, if possible, resolved at an early stage without impacting adversely
on au dit independence.
Sources of audit criteria
3.28 Each SAI w ill need to d evelop au d it criteria that a re valid for the
natu re of the activity und er review. These may includ e quanti tat ive
and/ or qualitat ive measures.
3.29 To avoid the necessity to create criteria from first principles for
each audit, the audit team should investigate the following sources of
existing criteria:
cr iter ia used previous ly in s imilar audi ts ;
cr iter ia publ ished by other SAIs ;
performance standards used by the agency, or previous inquiries by
the legislature;
agencies carrying out s imilar activities;
professional organisations and standard-sett ing bodies; and
general management and subject mat ter l itera ture .
3.30 These sources provide a basis for the development of suitable
criteria for the audit, but may require interpretation and modification to
ensure their relevance to the agency.
3.31 Criteria m ust b e realistic and take into account th e context of the
agen cy. Som e key criteria relate directly to the agency itself, for exam ple:
enabling and re la ted legis la t ion;
agency operat ing and procedures manuals ; and
central agency p olicies , s tand ards, directives and guidelines.
Agency performance information
3.32 Informat ion about the per formance of an agency underp ins
perform ance au d it ing. Perform ance information, ei ther qu anti tat ive
measures or qua l i t a t ive as sessments , i s fundamenta l to eva lua t ing
econom y, efficiency an d effectiveness. Criteria relating to sat isfactory
performance can be derived from the agencys own objectives or from
accepted indu stry and / or government standard s of performance.
8/8/2019 ASOSAI Performance Audit Guidelines
29/82
29
3.33 In examining p erformance information, aud itors shou ld:
consider w hether the agency has sufficient and reliable procedures in
place to measure and report on p erformance;
ascer ta in w hether the per formance measures in p lace are complete ,
relevant a nd ju stified on a cost-benefit basis;
e xa m i n e p r o ce d u r e s t o d e t er m i n e if th e y r e la t e t o th e a g e n cy s
corporate goals; and
consider whether the performance measures are incorporated into the
ma nagem ent d ecision-mak ing processes; that is, are they report ed an d
used within the agency.
3.34 These issues draw the performance auditor into a consideration
o f q u a n t i t a t i v e a n d q u a l i t a t iv e p e r fo r m a n c e i n f o r m a t i o n . Su c h
consideration should be an essential element in all performance audits.
3.35 Appendix D conta ins a l i s t o f impor tan t i s sues tha t can be
considered in a performan ce au d it of any gov ernm ent program or activity.
As such they serve as a useful checklist for performance auditors at the
start of a performan ce au d it. The qu estions are designed to help iden tify
potential issues and problems that can be explored in m ore detail.
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
30/82
30 5th ASOSAI Research ProjectPerformance Auditing Guidelines
4. Implementing the Performance
Audit
Introduction
4.1 The audit objectives and criteria will normally be tested by an
audit program of procedures that include:
observing, interviewing and documenting;
tes ting and checking; and
analysing.
Developing the audit program
4.2 In developing the au dit p rogram, it is important that the criteria:
relate to the aud it objectives; that is, enable relevant eviden ce to be
collected on issues w hich will maximise the imp act of the au d it;
are clear ly s tated and include sufficient detail to enable them to be
readily understood by those carrying out the audit;
are organised in a logical manner so that the aud it examination can be
conducted as efficiently as possible; form an efficient method of gather ing suf ficient evidence without
superfluous testing; and
t ake accoun t o f any ea r li er r ela t ed aud i t wor k and / o r pu b li shed
research on the topic.
4.3 Performan ce aud it programs n eed to be customised for each au d it.
Furth er factors to be considered w hen d eveloping th e program s includ e:
sizeaud it programs generally increase in s ize and complexity (more
detailed procedures, questionnaires and checklists) with increases inthe scope of the au d it;
geographic d i sper sionthe d i sper sion and loca t ion of s it e s to be
visited can marked ly affect the au d it pr ogram . Detailed procedu res
may be required to ensure consistency w hen different personnel are
carrying out the same audit at different locations;
aud it environmentmanagement receptiveness to being audited and
the sensitivity of the area in the agency will affect the way in which
procedures are developed and applied;
8/8/2019 ASOSAI Performance Audit Guidelines
31/82
31
the components of the program/ act ivity or the system to be aud ited,
eg. its inputs, processing activities, outputs and outcomes; and
w hether broad issues only have been identified, or specific cr iter ia
are available for audit examination.
4.4 Au dit man agers shou ld ensure that m eans of collecting app ropriate
eviden ce are listed aga inst each au d it criterion. The aud it ma nager shou ld
also consid er th e costs vis-a-vis the benefits of collecting evidence.
4.5 In some cases, a cr i ter ion that appears reasonable may not be
able to be tested at all. For examp le, a criterion that appropriate priority
be given to the development of an information system could n ot be assessed
w ithout kn owing all other priorities and their justifications. Thus the
criterion shou ld be revised. Criteria m ust also be reviewed for consistency
w ith th e legislation, for p racticality an d for comp rehension by field staff.4.6 Furth ermore, once preliminary research/ fieldw ork is un der way,
new issues may arise that warrant reconsideration and revision of the
initial criteria. H ow ever, before add ing new issu es, the likely imp act on
the audit budget and timetable must be considered.
4.7 In developing an aud it program it will not be p ossible to anticipate
all contingencies. In the early stages of an au d it, there is a need to retain
flexibility and to review the aud it pr ogram for ap pr opr iateness. I t is
preferable to start with a program outlining the approach to the audit
issues and revise and extend it as the aud it develops.
4.8 One approach to developing an audi t program is an i tera t ive
process such as the one outlined in Figure 3:
Figure 3The Audit Program Revision Cycle
Initiating the Performance Audit
4.9 In light of observations on the ou tcomes of this process, plan ning
f o r t he nex t s t age o f t he aud i t p r og r am can be m od i f i ed i . e . t he
interpretations/ conclusions arising from one stage of the aud it program
can be used to review and mod ify the next stage.
8/8/2019 ASOSAI Performance Audit Guidelines
32/82
32 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Audit approaches
4.10 There are a number of analytical approaches to undertaking an
au d it prog ram and mak ing assessments against criteria. These ap proaches
include:
analysis o f p rocedures ;
use of exist ing data or evidence provided by the agency;
analysis o f r esu lt s;
case stu d ies;
su rveys; an d
quantitat ive analysis.
Analysis of procedures
4.11 An analys is of procedures is of ten a s tar t ing point for audi t
analys is. The au d i tor w ould review sys tems in place for planning,
condu cting, checking and m onitoring the activity being aud ited . This
would involve interviews of managers and examination of documents
such as budgets, financial reports, program guidelines, annual or other
plans , procedures manuals , delegat ions and repor t ing requirements .
Procedures would be tested against established criteria or a desirable
control mod el. This would typ ically mean that procedu res w ould be
checked, among other things, for completeness, relevance against thelegislation, internal consistency and practicability.
Use of existing data
4.12 It is impo rtant for au d it staff to investigate the dat a held by agency
man agement and by o ther r e levant sources . This may inc lud e the
m a n a g e m e n t i n f o r m a t i o n s y s t e m s u s e d t o m a n a g e a g e n c y
program s/ activities and / or the data collected on ind ividu al program s.
Analysis of results
4.13 Analysis of resu lts from examin ing a nu mber of instan ces of agency
activity in a particular area w ill help decide w heth er agency performan ce
in that area conforms to audit criteria and is generally satisfactory.
Case studies
4.14 The case study is a method for learning about a complex issue,
based on a comprehensive und erstand ing of the particular instance. The
case stud y involves an extensive descrip tion and analysis of the p articu lar
issue within the context of the whole area under review.
8/8/2019 ASOSAI Performance Audit Guidelines
33/82
33
Surveys
4.15 Anoth er meth od of obtaining in sight into an agencys activities is
by the cond u ct of a surv ey. This is a m ethod of collecting information
from members of a population (such as the agencys staff, suppliers or
clients) to assess the incidence, distribution, and interrelation of eventsand conditions.
Quantitative analysis
4.16 Where pract icable , an ent i re populat ion should be analysed.
Where this is not feasible, due to cost and time constraints, sampling
techniques should be used . The natu re of the popu lation should be
examined to d ecide the most app ropriate sampling method ology. When
u sing either a statistical or a non -statistical samp ling ap proa ch, the au d itor
should select an ap prop riate aud it samp le, perform au dit procedu res on
the samp le and evaluate sample results so as to provide su fficient au dit
evidence.
4.17 When selecting an au dit samp le, the aud itor shou ld consid er the
specific au d it objectives and the attribu tes of the p opu lation from w hich
the aud itor wishes to draw th e samp le. In determining the sample size,
the au ditor should consider wh ether sampling risk w ill be redu ced to an
accept ably low level. The au d itor shou ld select samp le items so as to
have a reasonable expectation that all sampling units in the population
hav e an equ al chan ce of selection.
4.18 To each item selected th e aud itor shou ld ap ply au dit p rocedu res
ap pr opr iate to the pa rticu lar au d it objective. The au d itor shou ld consid er
what conditions would constitute an anomaly or error by reference to
the aud it objectives. The aud itor should consider the natu re and cause
of any errors identified and their possible effect on the particular audit
objective and on other areas of the audit.
4.19 Errors foun d in the samp le shou ld be p rojected to th e pop ulation.
The auditor should consider the effect of the projected error on theparticular audit objective and on other areas of the audit.
Refining the audit program
4.20 The initial au d it program shou ld be refined by d eveloping action
lists that reorgan ise the p rogram by au d it activity. For exam p le, for the
au d it activity interviews, an au d itor wou ld list all criteria/ issu es that
are to be ad d ressed in this way. At the same time, questions for the
interviews can be dev eloped from th e criteria/ issu es. Similarly, a list of
audit s teps should be established against other activit ies such as f i le
searches.
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
34/82
34 5th ASOSAI Research ProjectPerformance Auditing Guidelines
4.21 The result of this process is that for every criterion there is at
least one audit step (sometimes several will be necessary; for example,
interviews alone rarely p rovide ad equa te eviden ce). As well, for every
audit step there is a relevant issue related to the criterion.
4.22 It is also likely tha t the initial aud it progr am w ill require revisionto take accoun t of external factors. For examp le, the p recise natu re of
the d ata held (whether in d atabases or on files) may not be know n, nor
the likely difficulty of access. A decision is then necessary on w heth er to
aban d on th e test, m ake an in terim find ing of lack of satisfactory record-
keeping, or pu rsue an alternative means of gathering evidence.
4.23 I n c a r r y i n g o u t p l a n n e d a u d i t s t e p s a d d i t i o n a l r e l e v a n t
information, not explicitly covered by the audit program, may come to
light. In this case, the aud itor shou ld identify the issues highlighted by
the new information.
4.24 One of the p urp oses of the preliminary stu dy is to refine the au dit
pr ogram . It is also ad visable to review the prog ram early in the field w ork
s tage of the main aud i t to ensure it remains app ropr ia te. H owever,
revisions at this stage may lead to incompatibility of results from the
beginning to the end of the aud it. This w ill need to be consid ered by the
audit manager against the advantages of improving the program.
4.25 The level of detail of the final audit program will depend on a
number of factors:
the complexity of the aud it issues to be tested;
the extent of the aud it; for example, a large audit carr ied ou t in several
locations w ould need a d etailed program to ensure consistency; and
the level of the s taff carrying ou t the aud it; wh ere jun ior s taff have
responsibility for carrying out fieldwork, a more detailed program
would normally be appropriate.
4.26 For multiple-agency audits , the main issue in the program is
w heth er each au d it step need s to be carried ou t in each agency. Different
requ irements for samp le sizes ma y lead to some au d it steps being carried
ou t in on ly certain agen cies.
4.27 For single-agency audits, the auditor needs to be careful that the
instances chosen for analysis can be extrapolated to form a fair p ictu re of
the agency or pr ogram u nd er review as a w hole. In the case of cross-
agency audi t s , the re i s a s imi la r concern tha t agenc ies chosen for
examination are suff iciently representative or that the results can be
reasonably inferred across the public sector.
8/8/2019 ASOSAI Performance Audit Guidelines
35/82
35
Audit test programs
4.28 Audit test programs form an integral part of the audit program.
An audit test program refines audit criteria into a series of procedures
and / or activities (tests) to obtain relevant an d r eliable eviden ce up on
w hich conclusions may be d rawn .
4.29 Audit test programs are the key link between the development
of aud it objectives/ criteria and the condu ct of an au dit leading to credible
and d efensible find ings. This is illustrated in Figure4.
Figure 4The Development of Audit Test Programs
AuditObjective
TestMethodology
TestQuestions/Analysis
Examine findings and reassesscriteria if necessary.
Test Programs
FindingsAudit
Criteria
Initiating the Performance Audit
4.30 Test programs should not be prescr iptive but should have the
followin g characteristics:
Clear purpose: The purpose of the test program should be clear ly
defined from the outset to show relevance to the audit topic and to
effectively focus the audit.
Easily understood: The test program should be easy to understand,
w ith any jargon and acronym s explained.
Sound logic: A logical link should exist between the objectives of the
audit, the audit criteria and the audit test program.
Good layout/design: Page layout and design should be kept simple,
preferably in a working paper style format for easy referencing.
Good co-ordination: An au d it test program shou ld be used to clearly
define the roles of each team member to avoid duplication of work.
Flexibility: Test programs should remain flexible to allow for the
introdu ction of new evidence/ criteria, and the exclusion of outd ated
or irrelevant evidence/ criteria.
Cost effectiveness: Test programs must ultimately be cost effective;tha t i s , the t ime and r esources used on a t e s t p rogram mus t no t
outw eigh th e likely benefit such a p rogram w ill prod uce.
8/8/2019 ASOSAI Performance Audit Guidelines
36/82
36 5th ASOSAI Research ProjectPerformance Auditing Guidelines
Developing findings and recommendations
4.31 Audit findings are identified by relating audit observations to
au d it criteria. Au d it observations are based on th e analysis of information
collected du ring the aud it . Aud it findings should be d eveloped an d
eva lua t ed t h r oughou t t he va r ious phas e s o f a pe r f o r m ance aud i t .P o t en t i a l f i nd ings i den t i f i ed i n t he p l ann ing s t age o r du r ing t he
preliminary study should be followed up in the detailed examination
ph ase of the aud it.
4.32 I n s t ances w he r e ag ency p e r fo r m an ce exceeds t he expec t ed
per formance (as infer red f rom the audi t cr i ter ia) are good pract ice
findings, provided the targets/ benchmarks are realistically d etermined ,
and s hou ld a l s o be r epo r t ed . The p r oce ss o f ana lys ing ev idence ,
d eveloping findings and pr odu cing recommen d ations to resolve identified
areas of poor p ractice is su mm arised in Figure 5.
Figure 5The Process of Developing Recommendations
Audit criteria (what should be)
Audit evidence (what is)
Audit findings ('what is' compared with 'what should be')
Determine the causes and effects of the finding
Develop audit conclusions and recommendations
Estimate likely impacts of the recommendations wherever possible
4.33 The detailed evaluation of audit findings is generally completed
du ring the p reparation of discussion pap ers for d istinct segments of the
au dit or near the conclusion of the aud it f ieldw ork. H ow ever, some
evaluation m ay extend into the reporting p hase, as find ings are challenged
and further evid ence is obtained . It is at this latter stage that a final
decision is reached on the findings and recommendations that will be
reported. Once an aud it finding has been identified , two comp lementary
forms of assessmen t tak e p lace; an assessment of the significance of thef inding and the determination of the causes of good performance or
under-performance (where it is below that expected).
8/8/2019 ASOSAI Performance Audit Guidelines
37/82
37
4.34 The effect of a find ing m ay be qu antifiable in m any cases. For
e x a m p l e , t h e c o s t o f e x p e n s i v e p r o c e s s e s , e x p e n s i v e i n p u t s o r
u np rodu ctive facilities can u sually be estim ated. Ad d itionally, the effect
of inefficient p rocesses, for exam ple id le resources or p oor m anag emen t,
may become apparent in terms of time delays or wasted physical and
other resour ces. Qu alitative effects, as evid enced by a lack of control,
poor decisions or lack of concern for service, may also be significant.
The identified effect should demonstrate the need for corrective action.
The effect can also have occurred in the past, be occurring now or may
p ossibly occur in the futu re. If the effect occur red in the p ast the aud it
finding will stand only if the situation h as not already been remedied to
prevent it from recurring.
4.35 The aud itor shou ld id entify the un derlying cause of a find ing, as
this form s the basis for the recommen d ation. The cau se is that w hich, if
changed, would prevent similar findings. The cause may be outside the
control of the agency under audit, in which case the recommendation
shou ld d i r ect a t t en t ion ou ts id e the agency. H ow ever , the agency
concerned should be provided with the relevant part of the report for
commen t. This factor should also be taken into consideration w hile
developing the timeline for reporting.
4.36 The auditor may identify a cause-and-effect chain and have the
op tion of report ing the find ings at different po ints in the chain. In this
situation the auditor should highlight the most critical deficiency in the
chain.
4.37 Findings m ay be p resented in d iscussion pap ers for comment by
agency management. Agency responses can then be documented and
analysed. Where the agency d isagrees w ith the aud i t f ind ings and
recommendations, the reasons for such disagreement should be fully
analysed.
4.38 The recommendations made by the SAI regarding performance
aud its have to be argued in a logical mann er. Recomm endations should
indicate broadly what issues might be examined by management whenseeking solutions an d shou ld focus on the m ore significant issues requiring
at tent ion . The less s ignificant issues should b e refer red to agen cy
man agement for action.
4.39 Recomm end ations requ ire careful review to ensure that they are
practical and add value. The aud itor should ensure, for examp le, that
the recommendation addresses the objectives of the audit; ie. economy,
efficiency, effectiveness or accou ntability as app rop riate. A good test for
the auditor is to consider how the recommendation would be followed
up ; how the implementation of the recomm endation could be tested; andwha t s pec i f i c a c t i ons t he agency can unde r t ake t o im p lem en t t he
recommendation.
Initiating the Performance Audit
8/8/2019 ASOSAI Performance Audit Guidelines
38/82
38 5th ASOSAI Research ProjectPerformance Auditing Guidelines
5. Evidence and Documentation
Audit evidence5.1 Aud it evid ence is information collected an d used to sup port au dit
findings. The conclusions and recomm end ations in the aud it report stand
on the basis of such evidence. Consequ ently, perform ance au d itors m ust
give careful thou ght to the n ature, quality and amou nt of evid ence they
collect.
5.2 All field w ork shou ld be plan ned from the persp ective of acquiring
evid ence intended to sup port th e findings ap pearing in the final report.
The p lan for condu cting fieldw ork is the aud it program wh ich, in turn , isbased on the audi t object ives and on cr i ter ia where they have been
developed.
Competence, relevance and sufficiency of evidence
5.3 Th e INTOSAI Auditing Standards13 state Competent, relevant and
reasonable evidence should be obtained to support the auditors judgement and
conclusions regarding the organisation, program, activity or function under audit .
Competence of evidence5.4 Evid ence is comp etent (valid an d reliable) if it actua lly rep resents
w hat it pu rp orts to repr esent. The reliability of eviden ce can be ensu red
and assessed by considering the following:
cor robora t ion of ev idence is a pow er fu l t echnique for inc reas ing
reliability. This involves the au d itor looking for different typ es of
evidence from different sources;
evidence sourced from outside the agency is normally viewed as more
reliable for au di t p urp oses than informat ion generated within th e
agency;
documentary evidence is usually considered to be more reliable than
oral evidence;
evidence generated through d irect aud itor observation or analysis is
more reliable than indirectly obtained evidence;
the reliability of agency-generated information will be a fun ction of
the reliability of th e agen cys intern al control systems;
13 Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, pp. 50, 42
8/8/2019 ASOSAI Performance Audit Guidelines
39/82
39
ora l evidence that is cor roborated in w r it ing is more re liable than
oral evidence alone; and
original docum ents are more reliable than p hotocopies (if or iginals
are photocopied the auditor should note the source of original and
the date cop ied) . Photocop ies of eviden ce regard ed as being of s ignif icant impor tance to an audi t should, whenever poss ible , be
certified by relevant authorities.
Relevance of evidence
5.5 Relevance requires that the evidence bear a clear and logical
relationship to the aud it objectives and to the criteria. One app roach to
planning for data collection is to list, for each issue and criterion, the
nature and locat ion of evidence that is needed, as wel l as the audi t
procedure that is to be implemented.
Sufficiency of evidence
5.6 The aud itor shou ld obtain su fficient ap prop riate au dit evidence
to be able to draw reasonable conclusions on which to base the audit
r ep or t . Su f ficiency i s the measu re of qu ant i ty of au d i t ev iden ce .
Ap pr opr iateness is the measu re of qu ality of au d it evid ence, its relevance
to particular criteria and its reliability.
5.7 Evid ence is sufficient w hen there is enou gh r elevan t and reliable
information to p ersuade a reasonable person that the p erformance aud itfindings, conclusions and recomm endations are warranted and supp orted.
In determining whether documentary evidence is sufficient, the auditor
m ust take accou nt of the status of the d ocum ent; eg. d raft intern al agency
d ocum ents ma y hav e little statu s in terms of the agencys intentions and
decisions.
5.8 The factors that dic ta te the s t rength of evidence required to
sup port an observation in performance auditing includ e:
level of materiality or significance of the observation; d egree of risk associated w ith coming to an incorrect conclusion;
exper ience gained in previous audi t examinat ions on the degree of
reliability of the agencys records and representations;
known agency sens it ivity to an issue; and
cos t of obta ining the evidence rela t ive to the benefi ts in terms of
supporting the observation.
Evidence and Documentation
8/8/2019 ASOSAI Performance Audit Guidelines
40/82
40 5th ASOSAI Research ProjectPerformance Auditing Guidelines
5.9 E v i d e n c e g a t h e r e d d u r i n g a p e r f o r m a n c e a u d i t m a y b e
predominant ly qua l i t a t ive in na ture and r equi r es ex tens ive use of
professional jud gemen t. Accord ingly the aud itor w ould ord inarily seek
corroborating evid ence from d ifferent sou rces or of a d ifferent n atu re in
making assessments and forming conclusions.
5.10 When p lanning the aud it, the aud itor w ould id entify the probable
natu re, sources and a vailability of au d it eviden ce requ ired. The au d itor
should consider such factors as the availability of other audit reports or
stud ies and the cost of obtaining th e aud it evid ence.
Characteristics of performance audit evidence
5.11 I n pe r f o r m ance aud i t t e r m s , aud i t ev idence i s t he f ac t s o r
information used:
to conclud e whether an agencys management and employees have
accepted and carried out app ropriate actions to conform to operational
pr inciples , pol ic ies or s tandards for us ing resources ef fect ively,
efficiently and economically; and
to demonstrate to a third p ar ty that the aud itors findings are credible
and defensible.
5.12 Auditors need to be aware of potential problems or weaknesses
w ith perform ance au d it eviden ce. Potential pr oblems includ e:
evidence based on a s ingle source (this may imp act on re liabil ity,
validity and sufficiency);
o r al ev id e n ce n o t s u p p o r t a ble b y d o cu m e n t a t io n o r o b se r va t io n
(reliability);
evidence is not time-sensitive, ie. outd ated and does not reflect changes
(relevance);
evidence too expensive to obtain relative to benefits (relevance and
sufficiency);
source of evidence has a vested interest in outcome (reliability);
s a m p l es co l le ct e d a r e n o t r e p r e s en t a t i v e (r e le v a n c e, v a li d i t y,
sufficiency);
evidence may be related to an isolated occu rrence (validity, sufficiency);
evidence is incomplete, ie . does not dem onstra te a cause or effect
(reliability, sufficiency); and
evidence is conflicting (reliability).
5.13 Evidence can be ca tegor i sed as to i t s typephys ica l , o ra l ,documentary or analytical.
8/8/2019 ASOSAI Performance Audit Guidelines
41/82
41
Physical evidence
5.14 Physical evidence is obtained by observing people and events or
examining prop erty. The eviden ce can take the form of ph otograp hs,
charts, ma ps, graph s or other pictorial representations. A picture of an
unsafe condition is far more compelling than a written description.5.15 When the ob servation of p hysical cond ition is critical to achieving
the au d it objectives it should b e corroborated . This ma y be achieved by
h a v i n g t w o o r m o r e a u d i t o r s m a k e t h e o b s e r v a t i o n , i f p o s s i b l e
accompanied by representatives of the agency.
Oral evidence
5.16 Oral evidence takes the form of statements that are usually in
respon se to inqu ir ies or interview s. These s ta temen ts can pr ovid e
important leads not always obtainable throu gh other forms of aud it work.They can be m ad e by em p loyees of the ag ency, beneficiaries and clients
of the program being audi ted, exper ts and consul tants contacted to
provid e corroborating evidence in relation to an aud it, and by m embers
of the general p ub lic.
5.17 Corroboration of oral evidence is needed if it is to be used as
evidence rather than simp ly as backgroun d. This could be:
by wr it ten confirmat ion by the interviewee;
by weight of mu ltiple ind ependent sources revealing the same facts;or
by checking records la ter.
5.18 In assessing the reliability and relevance of oral evidence the
auditor needs to have regard to the credibility of the interviewee; that
is, the position, knowledge, expertise and forthrightness of the person
being interviewed .
Documentary evidence
5.19 Docum entary eviden ce in p hysical or electronic form is the most
common form of audit evidence. It may be external or internal to the
a g e n c y. E xt e r n a l d o c u m e n t a r y e v i d e n c e m a y in c l u d e l e t t e r s o r
m em
Recommended