CorporateFraud,LDA,and

Econometrics

DSSG ⋅2019March27

Dr.RichardM.Crowley

SMU

⋅

Slides:

rcrowley@smu.edu.sg @prof_rmc

rmc.link/DSSG

1

▪ Businessinsight

▪ Economictheory

▪ Psychologytheory

▪ Statistics

▪ Machinelearning

▪ Carefuleconometrics

Theproblem

▪ Detect:Classificationproblem

▪ Currently:Predictionproblem

▪ Misreporting:Theaccountingside

▪ Theapproachcombines…

Howcanwedetectifafirmiscurrently

involvedinamajorinstanceof

misreporting?

2

Whydowecare?

▪ Theabove,basedonAuditAnalytics,ignores:

▪ GDPimpacts:Enron’scollapsecost

▪ Societalcosts:Lostjobs,economicconfidence

▪ Anynegativeexternalities,e.g.compliancecosts

▪ Inflation:Incurrentdollarsitisevenhigher

The10mostexpensiveUScorporatefrauds

costshareholders12.85BUSD

~35BUSD

Catchingeven1moreoftheseastheyhappen

couldsavebillionsofdollars

3

WhatisMisreporting?

4 . 1

Misreporting:Asimpledefinition

Errorsthataffectfirms’accountingstatementsor

disclosureswhichweredoneseeminglyintentionallyby

managementorotheremployeesatthefirm.

4 . 2

Traditionalaccountingfraud

1. Acompanyisunderperforming

2. Managementcooksupsomeschemetoincreaseearnings

▪ WellsFargo(2011-2018?)

▪ Fake/duplicatecustomersandtransactions

3. Createaccountingstatementsusingthefakeinformation

4 . 3

Otheraccountingfraudtypes

▪

▪ Cookiejarreserve(secretpaymentsbyIntelofupto76%ofquarterly

income)

1. Thecompanyisoverperforming

2. “Saveup”excessperformanceforarainyday

3. Recognizerevenue/earningswhenneededtohitfuturetargets

▪

▪ Optionsbackdating

▪

▪ Relatedpartytransactions(transferring59MUSDfromthefirmto

familymembersover176transactions)

▪

▪ Improperaccountingtreatments(Notusingmark-to-market

accountingtofairvaluestuffedanimalinventories)

▪

▪ Goldreserveswereactually…dirt

Dell(2002-2007)

Apple(2001)

ChinaNorthEastPetroleumHoldingsLimited

CVS(2000)

CountrylandWellnessResorts,Inc.(1997-2000)

4 . 4

Wherearethesedisclosed?(US)

1. :AccountingandAuditingEnforcementReleases

▪ Highlightlarger/moreimportantcases,writtenbytheSEC

▪ Example:TheSummarysectionof

2. 10-K/Afilings(“10-K” ⇒annualreport,“/A” ⇒amendment)

▪ Note:notall10-K/Afilingsarecausedbyfraud!

▪ Benigncorrectionsoradjustmentscanalsobefiledasa10-K/A

▪ Note:

3. BytheUSgovernmentthrougha13(b)action

4. Inanoteinsidea10-Kfiling

▪ Thesearesometimesreferredtoas“littler”restatements

5. Inapressrelease,whichislaterfiledwiththeUSSECasan8-K

▪ 8-Ksarefiledformanyotherreasonstoothough

USSECAAERs

thisAAERagainstSanofi

AuditAnalytics’write-uponthisfor2017

Originaldisclosuremotivatedbymanagementadmission,

governmentinvestigation,orshareholderlawsuit

4 . 5

Whereareweat?

▪ Allofthemareimportanttocapture

▪ Allofthemaffectaccountingnumbersdifferently

▪ Noneoftheindividualmethodsarefrequent…

▪ Weneedtobecarefulhere(orcheckmultiplesources)

Fraudhappensinmanyways,formanyreasons

Itisdisclosedinmanyplaces.Allhavesubtlydifferent

meaningsandimplications

Thisisahardproblem!

4 . 6

PredictingFraud

5 . 1

Mainquestionandapproaches

▪ 1990s:Financialsandfinancialratios

▪ Misreportingfirms’financialsshouldbedifferentthanexpected

▪ Late2000s/early2010s:Characteristicsoffirmdisclosures

▪ Annualreportlength,sentiment,wordchoice,…

▪ Late2010s:Moreholistictext-basedMLmeasuresofdisclosures

▪ Modelingwhatthecompanydiscussesintheirannualreport

Howcanwedetectifafirmiscurrentlyinvolvedinamajor

instanceofmisreporting?

Allofthesearediscussedin

–IwillrefertothepaperasBCEforshort

Brown,CrowleyandElliott

(2018)

5 . 2

Whatweneedtoaddress:

1. Detectingvariedevents

▪ “Careful”featureselection(offloadtoeconometrics)

▪ Intelligentfeaturedesign(partiallyoffloadtoML)

2. Forbusinessusers…Interpretabilitymatters

▪ Psychology-styleexperiment

▪ Andaquasi-experiment

3. Predictivemodel

▪ Needclean,outofsampledesigns+backtesting

▪ Windoweddesign–datafrom1998won’thelptoday,butitwould

in1999

4. Infrequentevents

▪ Goodforsociety,badformodeling

▪ Carefuleconometrics

5 . 3

Mainresults

5 . 4

Issue1:Variedevents

6 . 1

Financialmodelbasedon

▪ 17measuresincluding:

▪ Logofassets

▪ %changeincashsales

▪ Indicatorformergers

▪ Theory:Purelyeconomic

▪ Misreportingfirms’

financialsshouldbe

differentthanexpected

▪ Perhapsmoreincome

▪ Oddcapitalstructure

Textualstylemodelbasedon

variouspapers

▪ 20measuresincluding:

▪ Lengthandrepetition

▪ Sentiment

▪ Grammarandstructure

▪ Theory:Communications

▪ Stylereflectscomplexity

andunintentionalbiases

▪ Somemeasuresadhoc

▪ Misreporting ⇒annual

reportwrittendifferently

Pastmodels

Dechow,etal.(2011)

Wetestedanadditional26financial&60stylevariables6 . 2

TheBCEmodel

1. Retainthevariablesfromthepreviousmodelsregressions

▪ Formsausefulbaseline

2. AddinanMLmeasurequantifyinghowmucheachannualreport(~20-

300pages)talksaboutdifferenttopics

▪ Trainonwindowsoftheprior5years

▪ Balancedatastaleness,dataavailability,andquantityoftext

▪ Optimaltohave31topicsper5years

▪ Basedonin-samplelogisticregressionoptimization

▪ Fromcommunicationsandpsychology:

▪ Whenpeoplearetryingtodeceiveothers,whattheysayiscarefully

picked–topicschosenareintentional

▪ Puttingthisinabusinesscontext:

▪ Ifyouaremanipulatinginventory,youdon’ttalkaboutinventory

Whydowedothis?—Thinklikeafraudster!

6 . 3

Whatthetopicslooklike

6 . 4

Howtodothis:LDA

▪ LDA:LatentDirichletAllocation

▪ Widely-usedinlinguisticsandinformationretrieval

▪ AvailableinC,C++,Python,Mathematica,Java,R,Hadoop,Spark,

…

▪ Weused

▪ isgreatforpython; isgreatforR

▪ UsedbyGoogleandBingtooptimizeinternetsearches

▪ UsedbyTwitterandNYTforrecommendations

▪ LDAreadsdocumentsallonitsown!Youjusthavetotellithowmany

topicstofind

onlineldavb

Gensim STM

6 . 5

Implementationdetails

1. Annualreportsareamess

▪ Fixedwidthtextfiles;properhtml;htmlexportedfromMSWord…

▪ Embeddedheximages

▪ Solution:Regexes,regexes,regexes

▪ Detailedinthepaper’swebappendix

2. Stemming,tokenizing,stopwords

3. FeedtoLDA

4. Tunehyperparameters(#oftopicsismostcrucial)

5. Finallyimplementthemodel

Theusualaddagethatdatacleaningtakesthelongeststill

holdstrue

6 . 6

Otherconsiderations

1. LDAprovidestheweightoneachtopic,butdocumentsvaryalotby

length

▪ Solution:Normalizetoapercentagebetween0and1

2. Thereisamechanicalcomponenttotopicsduetofirms’industries

▪ Solution:Orthogonalizetopicstoindustry

▪ Runalinearregressionandretain ε :

topic = α + β Industry + ε

i,firm

i,firm

j

∑ i,j j,firm i,firm

6 . 7

Issue2:Interpretability

7 . 1

LDAVerification

▪ LDAiswellvalidatedongeneraltext,noquestion

▪ Onekeyistopresentsomedetailsofthetopicstoensurecomfort

▪ Anotherkeyishavingpriorevidencetofallbackon

▪ WhetherLDAworksonbusiness-specificdocumentsisnotsowell

studied

▪ Moststudiesjustaskpeoplewhethertheyagreewiththehand-

codedtopiccategorizations

Wedecidedtofillthisgap

7 . 2

Experimentaldesign

▪ Whichworddoesn’tbelong?

1. Commodity,Bank,Gold,Mining

2. Aircraft,Pharmaceutical,Drug,Manufacturing

3. Collateral,Iowa,Residential,Adjustable

▪ 100individualsonAmazonTurk(20questionseach)

▪ Humanbutnotspecialized

Instrument:Awordintrusiontask

Participants

7 . 3

Quasi-experimentaldesign

▪ 3Computeralgorithms(>10Mquestionseach)

▪ Nothumanbutspecialized

1. GloVeongeneralwebsitecontent

▪ Lessspecificbutmorebroad

2. Word2vectrainedonWallStreetJournalarticles

▪ Morespecific,businessoriented

3. Word2vecdirectlyonannualreports

▪ Mostspecific

Theselearnthe“meaning”ofwordsinagivencontext

Runtheexactsameexperimentasonhumans

7 . 4

Experimentalresults

Experiment Internet WSJ Filings

10

20

30

40

50

60

70

ValidationofLDAmeasure(Intrusiontask)

Maximumaccuracy

Averageaccuracy

Minimumaccuracy

Randomchance

Datasource

%ofquestionscorrect

7 . 5

Issue3:Predictivemodeling

8 . 1

Backtesting

▪ So,wewillbacktest

▪ Usehistoricaldatatovalidateourmodel

▪ Problems:

1. Misreportingchangesovertime

2. Misreportingisunobservable(untilit’sobservable)

Wedon’tknowwhoismisreportingtoday

8 . 2

Movingtarget

▪ Implementamovingwindowapproach

▪ 5yearsfortraining+1yearfortesting

▪ Thestudyusesdatafrom1994through2012–14possiblewindows

▪ Ex.:topredictmisreportingin2010,trainondatafrom2005to2009

Problem:Nowwehave14models…

8 . 3

Comparingmultiplemodels

▪ Performancemeasures:

1. ROCAUC

2. Fisherstatistics

3. Performanceatareasonablecutoff(5%)

4. NDCG@k(usuallyusedinrankingproblems)

ROCAUCandFisherstatisticswillalsoallowusto

statisticallycompareacrossmodels

8 . 4

ROCAUCforwindowedapproaches

▪ ROCAUC

▪ Whatistheprobabilitythatarandomlyselected1isrankedhigher

thanarandomlyselected0

▪ Agoodscoreisabove0.70

▪ Aggregating:

▪ Simple:averageAUC

▪ Moreuseful:Poolpredictionstogether(withclusteringbyyear)

▪ ComparingROCAUCs

▪ Notsimple…

▪ Waldstatisticwithbootstrappedvarianceestimatesclusteredby

year

▪ ImplementedinStataasrocreg

8 . 5

▪ Comparingmodels:Variance-

Gammatest(seeBCE)

▪ Keyinsight:differenceof

X varshasthesameMGF

astheVarianceGammadist

▪ Calculationbelow

▪ KisthemodifiedBessel

functionofthesecondkind

Purelystatisticalmethod

▪ Fisherstatistic(Fisher1932)

▪ Combiningp-values(Note: p ∼ U 0, 1 )

▪ p-valuescomefromourout-of-samplepredictionmodel

▪ Calculatedas: X = −2 ln(p )

P(X > X ) = z K z dz

[ ]

∑i=1k

i

2

1 2 ∫−∞

X −X1 2

2 Γ(k)k√π

1∣ ∣k−

21

k−21 (∣ ∣)

8 . 6

Observability

▪ Theotherissueisthat,asofagivenyear,say2009,wedonotknow

everyfirmthatwasmisreporting

▪ Wecouldbuildanalgorithmwithperfectinformation,butitmayfall

flatoncurrent,noisydata!

▪ Itcouldalsogiveusafalseimpressionofanalgorithm’s

effectivenesswhenbacktesting

▪ Misreportingcantakealongtimetodiscover:Zale’sstartedin2004,

finishedin2009,andwasdisclosedin2011!

▪ Usedataonwhenamisreportingcasewasfirstdisclosed

▪ Ifthefraudwasn’tknownbytheendofthewindow,trainasifthat

was0(asitwasunobservablebackthen)

▪ Mimicsourcurrentsituation

Solution:Censorourdatatowhatwasknownatthepoint

intime

8 . 7

Issue4:Infrequentevents

9 . 1

Dealingwithinfrequentevents

▪ Fraudisinfrequent

▪ E.g.:Outof38,311firm-yearsofdata,thereare505firm-years

subjecttoAAERs

▪ Keyissue:Wemayhavemorevariablesthaneventsinawindow…

▪ Evenifwedon’t,convergenceisiffyusingalogisticmodel

▪ Afewwaystohandlethis:

1. Verycarefulmodelselection(keepitsufficientlysimple)

2. Sophisticateddegeneratevariableidentificationcriterion+

simulationtoimplementcomplexmodelsthatarejustbarely

simpleenough

▪ ThemainmethodinBCE

3. Automatedmethodologiesforpairingdownmodels(LASSO,

XGBoost)

9 . 2

Degeneratevariableidentification

1. Tosseveryinputintoamodel

2. CheckindependentnessusingaQRdecomposition

▪ Thiswillletusdetermineanorderfordroppinginputs

▪ A = Q × R,where Aisourfeaturematrix, Qisanorthogonal

matrix,and Risthetransformation

▪ Moreweightonthediagonalelementin Rmeansmore

independent(effectively)

▪ SameunderlyingmethodasaGram-Schmidtprocess

3. Removeexcessinputsiftoofew1s

▪ Why?Becauselogitcan’tconvergeiftherearemoreinputsthan

events(ornon-events)inthedata

Independentnessisausefulcriterionforremovingfeatures

withlowerlikelihoodofbeinguseful

9 . 3

Logisticiteration

1. RunalogitusingaNewton-Raphsonsolverfor50iterations

2. Checkconvergenceforsignsofquasi-completeness

▪ Standarderrorswillbeinthemillionsifquasi-complete

▪ Ifquasi-complete,dropthenextleastindependentvariableand

restart

3. Runa500iterationlogitusingaNewton-Raphsonsolver

4. Recheckconvergence

▪ Iffailed,dropthenextleastindependentvariableandrestart

Wewillessentiallygetthemostcomplexfeasiblemodel

withthemostindependentsetoffeatures

9 . 4

Finalcomments

10 . 1

Someotherinterestingresults

10 . 2

Waystobuildonthismodel

1. UseabettertokenizersuchasspaCy

▪ Ourtokenizerdidn’tdetectnounphrases

2. Useeconometricmethodsthatarebettersuitedforsparsity

▪ E.g.:XGBoost

3. ConsiderusingamorepowerfulLDAvariantsuchassupervisedLDA

(sLDA)

4. NoneedtostopatLDA–therehavebeenalotofadvancementsinNLP

since2003

Finalnote:Themotivationbehindourworkwasnottobuilda

bettermousetrap,buttoillustratetheusefulnessdocuments’

contenttobetterunderstandcompany/managerbehavior

10 . 3

Endmatter

11 . 1

Thanks!

Dr.RichardM.Crowley

SMU

⋅

Web:

rcrowley@smu.edu.sg @prof_rmc

rmc.link

Tolearnmore:

▪ Theseslidespubliclyavailableat

▪ Plentyoflinkstoclickthroughandexplore

▪ Technicaldetailspubliclyavailableat

rmc.link/DSSG

SSRN

11 . 2

▪ Predictionscoresfor1999

rankedinthe98thpercentile

▪ Firstpublicizedin2001

▪ IncreasesinIncometopicand

firmsizearethebiggestred

flags

▪ Predictionscoresfor2004

through2009rank97th

percentileorhighereachyear

▪ publishedin2011

▪ MediaandDigitalServices

topicsaretheredflags

Casestudies

AAER

11 . 3

▪ Logofassets

▪ Totalaccruals

▪ %changeinA/R

▪ %changeininventory

▪ %softassets

▪ %changeinsalesfromcash

▪ %changeinROA

▪ Indicatorforstock/bond

issuance

▪ Indicatorforoperatingleases

▪ BVequity/MVequity

▪ Lagofstockreturnminus

valueweightedmarketreturn

▪ BelowareBCE’sadditions

▪ Indicatorformergers

▪ IndicatorforBigNauditor

▪ Indicatorformediumsize

auditor

▪ Totalfinancingraised

▪ Netamountofnewcapital

raised

▪ Indicatorforrestructuring

Financialmodel

BasedonDechow,Ge,LarsonandSloan(2011)

11 . 4

▪ Logof#ofbulletpoints+1

▪ #ofcharactersinfileheader

▪ #ofexcessnewlines

▪ Amountofhtmltags

▪ Lengthofcleanedfile,

characters

▪ Meansentencelength,words

▪ S.D.ofwordlength

▪ S.D.ofparagraphlength

(sentences)

▪ Wordchoicevariation

▪ Readability

▪ ColemanLiauIndex

▪ FogIndex

▪ %activevoicesentences

▪ %passivevoicesentences

▪ #ofallcapwords

▪ #of“!”

▪ #of“?”

Stylemodel(late2000s/early2010s)

Fromavarietyofresearchpapers

11 . 5