PROCESS MINING OF EVENT LOGS IN AUDITING: …raw.rutgers.edu/docs/Innovations/Process20MiningSlides.pdf · Mieke Jans Michael Alles Miklos Vasarhelyi ... in contrast to what people

Mieke Jans Michael AllesMiklos Vasarhelyi

Hasselt University Rutgers Business School

PROCESS MINING OF EVENT LOGS IN AUDITING: OPPORTUNITIES AND CHALLENGES

Process Mining of Event Logs in

Auditing: Opportunities and Challenges

What is Process Mining of Event Logs?

• The basic idea of process mining is to extractknowledge from event logs recorded by aninformation system. Until recently, the information inthese event logs was rarely used to analyze theunderlying processes. Process mining aims atimproving this by providing techniques and tools fordiscovering process, control, data, organizational, andsocial structures from event logs. Fuelled by theomnipresence of event logs in transactionalinformation systems… process mining has become avivid research area.

• http://is.tm.tue.nl/staff/wvdaalst/BPMcenter/process%20mining.htm

2

http://is.tm.tue.nl/staff/wvdaalst/BPMcenter/process mining.htm
















Process Mining and Auditing

• Event Logs also referred to as Audit Trails.

• Audit trail: a chronological record of computersystem activities which are saved to a file on thesystem. The file can later be reviewed by the systemadministrator to identify users’ actions on the systemor processes which occurred on the system.

• Despite the presence of the word “audit” in the termaudit trail, it does not refer to auditing in theaccounting sense, but to the general potential anevent log provides to reconstruct past transactions.

• Little use made by auditors of process mining toexamine the data contained in event logs.

3



Recall World of Manual Accounting

• In a world of manual bookkeeping the data whichauditors had to rely on when checking whattransactions the client firm had undertaken and how ithad accounted for those transactions came entirelyfrom paper based ledgers.

• If ledger entries have been falsified, erased andoverwritten, added to or modified at another date andtime by the same or other party, auditors can detectthat only through physical scrutiny of the books.

• Similar to bankers having to rely on a bank teller’sfamiliarity with the signature of a customer todetermine whether a check was genuine or forged.

4



The WYSIWYG Problem in Auditing

• Hand written ledgers suffer from what we call the“what you see is what you get” or WYSIWYG problem:the only information that the auditor has is what theycan literally observe in front of them.

• Hence, the auditor has no way of verifying who madethose ledger entries and when they did so.

• What makes an event log such a unique andpotentially invaluable resource for auditing is not onlythat it provides the auditor with more data to analyze,but also because that additional data is recordedautomatically and independently of the person whosebehavior is the subject of the audit.

5



Meta-Data in Event Logs

• With an event log the auditor is no longer restricted tothe WYSIWYG ledger of transactions entered by theauditee, but also possesses an independent set of“meta-data” about the circumstances under which theauditee made those entries.

• That meta-data encompasses more than simple timestamps for transactions, for by taking advantage ofthat tracking data, the event log enables the auditorto reconstruct the history of any given transaction.

• Hence, the auditor is now able to trace therelationship of that particular entry and its author toall prior transactions by that or related parties.

6



The Digital Economy Facilitates Process Mining and Event Log Creation

• The emergence of the digital economy hasfundamentally altered both the way of runningbusinesses and of performing [continuous] audits.

• Most businesses of any significant size today storetheir data electronically thanks to the maturing oftechnologies for databases and computer networks.

• Systems such as Enterprise Resource Planning (ERP),Workflow Management (WFM), Customer RelationshipManagement (CRM), Supply Chain Management(SCM) or Busines-to-Business (B2B) all create andstore histories that potentially can be process minedby auditors.

7



Process Mining Still More Potential than Reality

• It is important to understand that the term event“log” promises more than what is the reality today:the event log is a database of time, process, andoriginator stamps, not a chronicle that an auditor cansimply read as if it were a story.

• While most modern IT systems—especially ERPsystems—record a history, that is the starting point ofcreating an event log, not one in itself.

• Extracting an event log from an ERP databaserequires considerable manual effort by the auditor.

• Logging can slow down ERP systems, so often thatfeature is turned off, especially if no one wants it.

8



An Example of An Event Log of an Invoice

9

Figure 1: Visualization of Input Data and

Event Log Data of an Invoice

Invoice number 003

Supplier: AT&T

Posting date: Feb 10th 2010

120 USD

Description: internet services Jan 2010

‘Signature of John’

‘Signature of Pete’

Invoice number 003

Supplier: AT&T

Posting date: Feb 10th 2010

120 USD


‘Signature of John’

‘Signature of Pete’

PLUS

- ‘Create Invoice’

Timestamp: Feb 12th

2010; 08:23 AM

Originator: Mike

Fields: supplier: AT&T, posting date:

02-10-2010, value: 100 USD,


- ‘Change’

Timestamp: Feb 12th

2010; 08:43 AM

Originator: John

Field changed: Value

Value old: 100 USD

Value new: 120 USD

-‘Sign’

Timestamp: Feb 12th

2010; 08:44 AM

Originator: John

Inp

ut d

ata

M

eta

-da

ta



Components of an Event Log: Input Data

• The left hand side of Figure 1 shows the data aboutthe invoice that is entered by the person making thedata entry into the firm’s ledger.

• This is what we call input data as it is characterizedby the controllable act of a person inputting the data.

• This input data is the type of data available at themoment this data is stored, such as the invoicenumber, the posting date, the supplier etc.

• This is also the type of transactional data that iscurrently used for auditing and monitoring.

10



Components of an Event Log: Meta Data

• The right hand side of Figure 1 shows the data that isstored in the event log of that same invoice.

• Formally speaking all input data is also part of theevent log, but clearly it is the entries that are uniqueto the event log that are of particular interest to anauditor because that data is recorded automatically bythe system and not inputted by the auditee.

• It is this meta-data which makes an event log oflarger dimension

• That contextual meta-data enables the auditor toidentifying relationships between this transaction andall other transactions in the database.

11



Comparing Input Data with Meta Data

• Event log reveals changes to the invoice and theidentity of other individuals who “touched” the invoicein any way during its progress through the purchaseto pay business process.

• For example, where we see a posting date of February10th in the input data, this could differ from thesystem’s timestamp recorded in the event log.

• The meta-data in the event log can potentially containmany pieces of information, like at what times whichfields are changed by which originators.

• Meta data serves as a check on the input data.

12



Using Process Analysis of Meta Data to Create a Narrative History of the Business Process

• Meta-data, when combined with the input data,enables the auditor to reconstruct the history of aparticular transaction.

• Thus, the following activities can be reconstructedfrom the data shown in the event log example ofFigure 1:

13

1. on Feb 12, 8:23 AM: Mike entered invoice No. 3 in system, filling out the supplier (AT&T),

posting date (02-10-2010), invoice value (100 USD) and description (internet services Jan 2010)

2. on Feb 12, 8:43 AM: John changed ‘Value’ from ‘100USD’ to ‘120USD’

3. on Feb 12, 8:44 AM: John signed invoice No. 3



Meta Data Expands Audit Evidence

• Where auditors used to have only the paper-writtengeneral ledgers with the data on it that the auditee(or anyone else) had written down, there is now eventlog data available that records everything thathappened to that ledger.

• For example, where the auditor used to rely on thesignature at the bottom of a document to verify whowas responsible for the document, the event logkeeps track of, amongst other things, the personopening/altering/signing the document.

• Critically for auditing, meta data in event log isindependent of the person entering the input data.

14



Process Mining Overcomes WYSIWYG Problem

• Thus the event log enable auditors to overcome theWYSIWYG constraint of the input data.

• This event log forms the start and also theopportunity to enact process mining.

• Based on the log of these events, a process model canbe used to gain insights about what actually happensin a process, in contrast to what people think happensin a process.

• Many other such comparisons can be undertaken bythe systematic analysis of a properly constructed,comprehensive event log.

• Key is creating such a log in the first place.

15



Steps in Creating an Event Log 1

• The event log data that is captured by the ERP systemis large and dispersed over numerous tables.

• In order to mine the event log and, hence, theprocess, a rigorous and defensible method ofstructuring the data needs to be developed.

• The first step is for the auditor to develop a holisticunderstanding of the activities that constitute theprocess being audited, trading off comprehensivenessagainst the size of the resulting event log.

• The second step in event log creation is the selectionof the process instance which undergoes the identifiedactivities, such as an invoice.

16




• The activity performed by the auditee is input databut the storage of the act itself is meta-data becauseit is beyond the control of the auditee.

• Each process instance has a unique identifier and isstored along with attributes that describe the processinstance, like for instance the size, the supplierinvolved etc.

• The activities performed on the process instance,called unique event entries, also have a uniqueidentifier which is stored along with the timestampand originator stamp.

17




• Aside from the time and originator stamps, otherattributes that describe the event or activity arestored. All these attributes are mainly stemming frommeta-data.

• Which activities and attributes will be captured in theevent log is jointly determined by capabilities of theERP system to record data and the judgment of theauditor as to the necessary scope of the event log.

18



Example of a Constructed Event Log

19

Unique Event Entries (UEE)

UEE-ID PI-ID Activity Originator Timestamp

UEE -ID 1 PI-ID 1 Activity A:

Change PO

Originator X xx/xx/xxxx

UEE -ID 2 PI-ID 1 Activity B:

Enter Goods Receipt

Originator Y xx/xx/xxxx

UEE -ID 3 PI-ID 1 Activity A:

Change PO

Originator Z

UEE -ID 4 PI-ID 2 … …

…

UEE -ID m PI-ID n

UEE attributes

UEE -ID Name Value

UEE -ID 1 field changed delivery address

UEE -ID 1 old value ‘previous delivery address’

UEE ID 2 Goods Receipt number GR0005014

UEE -ID 2 reference to invoice SP14V51

UEE -ID 3 field changed commercial discount %

…

UEE -ID m … …



Process Mining as an Audit Tool

• In auditing the information that is analyzed to issuean audit opinion is essentially the same as when anaudit was performed in a paper-based setting: inputdata, albeit now in a database and not a paper ledger.

• This data may now be analyzed using search queriesrather by manual examination, but that is simplyautomating an existing manual procedure.

• Process mining is reengineering audit practice to takefull advantage of the capabilities of digital businesses.

• Using event logs the auditor can examine not justwhat has happened, but also who did it, when, andwith whom, as well as similar transaction elsewhere.

20



Process Mining Based Audit tests 1

• Fraud typically takes place at times where there arefewer other employees around to ask questions, suchas at lunchtime.

• Hence, forensic accountants monitor the firm’s ledgersat lunchtime to see who else is on the system,whereas the person committing the fraud might wishto cover their tracks by entering a different time forwhen they undertook the fraudulent transaction.

• An event log will record the actual time of thistransaction and that information is available to theauditor without recourse to monitoring at preciselythe same time as the data is being entered.

21




• An anomaly that could be revealed by mining theevent log data and which may not be detected soreadily through other means is violations ofsegregation of duty controls.

• Some SOD violations are detectable with standardaudit tests.

• But employees not following required procedures likefirst getting approval and only then ordering thegoods would not be apparent by only looking attransactional data alone.

• Using the timestamps in the event log makes visiblethis circumvention of procedures.

22




• Consider a collaboration between a supplier and anemployee, systematically changing a purchase orderwithin the sustained margins after a last approval tothis purchase. The supplier gets paid more than wasagreed upon, and can provide kickbacks to theinvolved employee.

• This abuse can be discovered by analyzing event logdata since it captures changes to the invoice. Becauseof the stored originators of activities it is also possibleto analyze collaboration between employees and otherparties.

• Compensates for weaknesses in ERP controls.

23



Process Mining as an Analogy to Security Cameras (CCTV) 1

• An analogy that can be applied to event logs is that ofvideo surveillance used in businesses to safeguardassets and deter crime.

• The major difference between event logs and videosurveillance recordings is that storing transactionaldata is cheap and that it is time and location stampedso that it is feasible for an auditor to search foranomalies and track back history in the event of adetected problem, such as theft or fraud.

• By contrast, many surveillance cameras are actuallynon-functioning replicas since there is no possibility ofcost effectively monitoring their feeds.

24



Process Mining as an Analogy to Security Cameras (CCTV) 2

• Firms install fake cameras because they have adeterrent effect, which is also the greatest benefitfrom known systematic process mining of event logs.

• Just the chance that someone might be watching aperson’s behavior can serve to constrain thatbehavior.

• How much more effective this deterrence effect wouldbe then if an auditee knew that event logs wereindeed being automatically and continuouslymonitored for anomalies and subject to tests ofanalytic procedures?

25



Methods of Process Mining in Auditing 1

• Three fundamental process mining perspectives: theprocess perspective (“How?”), the organizationalperspective (“Who?”) and the case perspective(“What?”).

• The process perspective uses the time and locationstamps in the event log to help answer the questionof “How the process was undertaken?” with respect tothe following order of activities.

• This perspective can be used by auditors to comparethe process as it is meant to be performed againsthow it actually is and thus identify control failures andweaknesses.

26




• The organizational perspective uses the data in the'Originator' field in the event log to help answer thequestion of “Who was involved in the process?”

• In this perspective, underlying relations betweenperformers or between performers and tasks can bemade visible.

• The obvious use of this perspective in auditing is inchecking segregation of duty controls, eitherretrospectively, to check existing procedures, orprospectively, to verify integrity of controls whenpersonnel changes are expected (for example, due tolayoffs or expansion).

27




• The case perspective or the “What happened with thisparticular transaction?” question focuses on a singlecase, tracing back its history and relationships ofparties that are involved in that history.

• This perspective can be used to check assertions on acase basis, for instance “When a purchase order ischanged by over 2%, is this followed by a newapproval?”. (Assuming that the relative magnitude ofa change is stored as a attributes in the event log).

• This will also be useful to analyze the separatelystored attributes, for example the size of an order orthe related supplier.

28




• Another way of classifying process mining is by theapproach followed to search for answers to thesethree questions.

• Broadly speaking, there are at least five different suchtasks in process mining: a. process discovery, b.conformance check, c. performance analysis, d. socialnetworks analysis, e. decision mining and verification.

• The application of only some of these tasks toauditing have been examined in depth.

• Need for much more research on how the datacontained in event logs can be exploited throughprocess mining.

29



Caveats and Future Research

• It is essential that the integrity of the meta-data isassured, so that it is beyond the influence of theauditee. Otherwise the value of the event log as auditevidence is lost.

• This is a “guarding the guards” situation with meta-controls necessary.

• The format of the event log is still being intensivelyresearched, and it is important that auditors followdevelopments in this areas closely in order not to fallbehind other fields that use process mining.

• Similarly, new developments are arising in themethodology of process mining.

30



Next Steps

• Many of the examples of process mining as applied toauditing that we present in this paper take as theirstarting point familiar manual audit procedures.

• Thus, we are following the standard route intechnology adoption, which is to first automatemanual processes and only then, once a level ofcomfort is attained, to reengineer those processes totake full advantage of the capabilities of thetechnology

• Once the potential of process mining as applied toauditing is recognized, its true value added can befully investigated.

31

Documents

PROCESS MINING OF EVENT LOGS IN AUDITING: …raw.rutgers.edu/docs/Innovations/Process20MiningSlides.pdf · Mieke Jans Michael Alles Miklos Vasarhelyi ... in contrast to what people