Improvingthemanagementofstatisticalprocessestheuseof

QualityGatesPaulSchubertandSiu-MingTam

July2015

Outline• BackgroundonQuality

Gates• EffectivenessofQuality

Gatesinreducing‘qualityincidents’

• Addressingbarrierstomorewidespreadandeffectiveimplementation

• Abroaderapproachtostatisticalriskmanagement

2

‘Statisticalrisk’andqualitygates

• Statisticalrisk: • Qualitygates:

checkpointsordecisionpointsatvariousstrategicplacesinastatisticalprocessatwhichthequalityoftheprocess(anddata)atoruptothatpointisexplicitlyassessed

thechanceorlikelihoodthatsomethingcouldgowronginthestatisticalprocessaffectingthequalityorintegrityofthestatisticaloutputs

QualityGates

Handover

Within

Overviewofqualitygatescomponents

• Placement• QualityMeasures• Roles• Tolerances• Actions• Evaluation

Areasofrisk

• changestoprocesses,systems,methodologies• datatransformations• overlap/coordination/integrationwithotherareas

• knowledgemanagement

Reviewofqualityincidentsandreleaseincidents

0

1

2

3

4

5

0 1 2 3 4 5 6 7 8 9 10 11 12

Freq

uency

NumberofReleaseIncidentsintheQuarter

ReleaseIncidentsperQuarter

EffectivenessofQualityGatesinpreventingqualityorreleaseincidents

14.3%

61.4%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Collections withQualityGates Collections without QualityGates

ChanceofaReleaseIncidentOccuringDuringtheReferencePeriodforPLaSSandEESG

collections(Jan2010-Jun2014)

Independentaudit

• AssesstheeffectivenessofQualityGatesinimprovingqualityandmitigatingrisk

• Understandandsuggestsolutionstothebarrierstomorewidespreadadoptionbyteamsthroughouttheorganisation

ExtentofqualitygateadoptionTeam 1 2 3 4 5 6 7 8 9 10Qualitygatesinplace?

P P Partially Ï Ï Ï Partially Ï Ï Ï

Placement P P P P P Ï P Ï P Ï

QualityMeasures

P P P P P Partially P P P P

Roles P P P P P Partially P P P Partially

Tolerances P P P P Partially Ï P Ï Ï Ï

Actions P P Partially Ï Ï Ï Ï Ï Ï Ï

Evaluations P P Partially P P Ï P Ï P Partially

Findings– benefitsofQualityGates

• valuabletoolinmaintainingqualityandminimisingrisk

• clearlyassignsaccountabilityandresponsibility• promotesbetterprojectmanagementpractices• createsefficiency• formalisesprocesses• aidsinknowledgesharingandsuccessionplanning

Findings– barrierstoadoption

• lackoftimeandstaffresources• teamswithrelativelyinexperiencedstafforimmatureknowledgemanagementpracticesstrugglewiththeconceptandapplicationoftheQualityGateprocess

• perceptionofaddingalargeadministrativeburdentoexistingprocedures

• easiertoimplementformeasuresthatarequantitativeratherthanqualitativeinnature

Recommendations

• developaQualityGatepolicy– specifyteams,collectionsand/orcircumstanceswhereQualityGatesaretobemandatory

• improvesupportandtrainingmaterial• utilisefacilitatedworkshops• improveknowledgemanagementofbusinessprocesses

• developKPIstomonitortheeffectivenessofQualityGates

Abroaderdefinitionofstatisticalrisk

• Context– plannedchangestosystems,

processesormethods– errorsinsystems,processesor

methods– changesintheavailability(or

expertise)ofresources– changesintheavailabilityor

qualityofsourcedata– ‘realworld’eventsthatmay

notbecapturedadequately– asampleoradministrative

datasetthatbecomesunrepresentative

– achangeinusers’viewsorperceptionsofthefitnessofpurposeofABSstatistics

TheriskthattheABSpublishesstatisticsthatarenotconsideredbyusersasfitforpurpose.Thisincludes,butisnotlimitedto,timeseriesthatarenotcoherentandstatisticsthatarenotcoherentwithrelatedstatistics.

AbetterapproachtostatisticalriskmanagementintheABS

• impossibletoeliminatestatisticalrisk• mitigationofriskscanofteninvolvepotentiallydifferentapproaches– assessonthegroundsofcost-effectiveness

• mitigationapproachesneedtobeassessedcollectively

aimtoensurethatstatisticalrisksareunderstoodandthatthecollectiverisksaremanagedinacost-effectiveway

Elementsofmanagementofstatisticalrisk

• integrationwithbroaderriskmanagementapproach• mitigationappropriatetoimportanceofoutputs• determinationof‘fitforpurpose’ofeachstatisticsorclassofstatistic

• documentedstatisticalriskmanagementplans– includingQualityGates,contingencies

• clearidentificationofriskmanager• independentassessmentandreportingbyMethodologyDivision

• engenderingtherightculture(ledbyaDeputy)

ReferencesABS2010,QualityManagementofStatisticalProcessesUsingQualityGates,cat.no.1540.0,ABS,Canberra.Dufty,D2014,ReviewofQualityGatesandQualityIncidentsintheABS,internalreport,ABS,Canberra.Oakton2015,ApplicationofQualityGates,internalaudit,ABS,Canberra.Harper,P2015,Statistical riskmanagementintheABS– Abetterapproach, internalpaper,ABS,Canberra.

17

Questions?

paul.schubert@abs.gov.au