JoanneR.Stevenson,Ph.D.&JohnVargo,Ph.D.1

ChrisThomson,Ph.D.2&Lucy-JaneWalsh2

ResilientOrganisationsResearchReport2017/01

April2017

ResilientOrganisationsLtd.1

UniversityofCanterburyDigitalLab2

RESILIENCE&DATAINNEWZEALAND:THEDATAINTEGRATIONANDVISUALISATIONENMASSE

(DIVE)PLATFORM2016SUMMARY

ii

In2016,theNZCentreforEarthquakeResilience(QuakeCoRE)andtheResiliencetoNature’sChallenges(RNC)–NationalScienceChallenge,fundedasmallteamofresearchersfromResilientOrganisationsLtd.incollaborationwithUCCEISMIC(theCanterburyEarthquakeDigitalArchive)toinvestigatehowtobestenableteamsofresearcherstoaddresscomplexsocialproblemsthatwillmakeNewZealandmoreresilient.ThefocusofthisprogrammewastoidentifythetypesofdataQuakeCoREandRNCresearchteamswouldbeusing,howtheyplannedtoanalyseandsharethatdata,andhowdatamanagementpracticescouldenhancetheimpactoftheseresearchprogrammes.

BetweenMarchandNovember2016,theteaminitiatedaconsultationprocessinvolvingaseriesofworkshops,surveys,interviews,andsoftwareprototypedesignandtesting.

Theoutcomesofthisconsultationprocessresultedinseveralkeyoutcomes:

1. Theidentificationandclassificationofdatatypesthatresearcherswillbeusing.2. Theidentificationofcriticaldataneedsforresearchers,including:

a. Systemsforknowingaboutongoingresearch(beforepublication).b. Enhancedsearchabilityofdataacrossinstitutions.c. Systemsthatmakesharingresearchdatasafe,easy,anddesirable.d. Establishingstandardsandguidancefortransdisciplinarydatamanagementinaway

thatfacilitatesdataintegration,analysis,andvisualisation.e. Enhancingaccesstopublic,proprietary,andsensitivedatasources.f. Streamliningandclarifyingdatasharingagreementsfordatasetsthathave

significantreusevalueortowhichresearcherswilladdvalue.g. Theabilitytotrackdatareuse.

Theconsultationprocessalsoinvolved:

3. Evaluatingpre-existingsystemsthatcanmeetsomeoftheimmediateneedsofresilienceresearchersinthisspaceincludingDesignSafe,theNewZealandGeotechnicalDatabase,andEERIClearinghouseSystem.

4. Fosteringrelationshipsbetweenkeydataprovidingorganisationsandresearchers.5. Identifyinghumanandinstitutionalfactorsthatinhibitthesuccessofsuchboundary

pushing,transdisciplinary,andcross-institutionalresearchprogrammes.

Twofinaloutcomesmovedtheconsultationprocessintothedesignphaseforasystemthatcanbegintomeettheneedsofresilienceresearchersandpractitioners:

6. Thedevelopmentofseveralsoftwareusecasestoguidethedevelopmentoffuturedatasharingsystems.

7. Thecreationofaworkingprototypedatafederationportalsystem,whichwearecallingtheDataIntegrationandVisualisationEnMasse(DIVE)Platform.

TheconsultationprocessmadeitcleartherearesystemsthatcanmeetsomeoftheneedsofthoseworkingtoimproveresilienceinNewZealand.Thereare,however,stillsignificantunmetneedsthatwillhindertheprogressoftrulytrans-disciplinaryandtransformativeresearch.

Summary

iii

Outcomeswouldbeenhancedbyasystemthatisproblem-focused,ratherthandividedbyfundingordisciplinaryboundaries.Suchaproblem-focusedsystemwillenhancethevisibilityoftheworkgoingonto improvetheresilienceofNewZealand. Itwillbeaplacewherecommunitiesofresearchers,decisionmakers,dataholders,privateindustry,andcitizenscientistscanview,upload,anddownloaddata. Such a system should facilitate the creative collision of secondary and primary data, localnarratives,real-timehazardmonitoring,MātaurangaMāoriknowledge,andmulti-mediainformation.

WeproposethecontinueddevelopmentofDIVE intoan interactiveonlinespaceforresearchersandpractitionerstoorganiseandcommunicateinformationrelevanttotheirongoingresearchandinformationgatheredfromdisastereventsastheyunfold.Capturingthisdatainafederationportalthatiscurated,properlyarchived,andstrategicallysharedwillfacilitatefutureresearch,aidresponseand recovery actions and decisionmaking, andmay become a resilience building tool as broadercommunitiesareabletocontributedataonthehazardstheyareexperiencingorthetrendstheyareseeingintheircommunities.

ThisworkwasfundedbytheTertiaryEducationCommissionthroughtheCentreofEarthquakeResilience(QuakeCoRE)andbytheMinistryofBusinessInnovationandEmploymentthroughtheResiliencetoNature’sChallengesNationalScienceChallenge.Theauthorswouldliketoextendsincerethankstoeveryonewhogaveuptheirtimetoattendworkshops,filloutsurveys,andsharetheirexpertisethroughoutthedevelopmentoftheDIVEPlatform.WewouldalsoliketothankToniWiandLouiseHome-Dewarfortheirhelpinthepreparationofthisreport.

Acknowledgements

iv

TableofContentsSummary ii

1.Introduction 1

2.DIVEDevelopmentProcess 2

3.FindingsandOutputs 53.1 Processes 63.2 Institutionalstructures 63.3 Software 73.4 Stakeholders 73.5 ObservationsabouttheDataDevelopmentContext 7

4.Conclusions 9

References 10

AppendixA:TheDIVEPlatformInterface 11

AppendixB:UseCaseSummariesfromWorkshop3 13

AppendixC:Affiliatedresearchersroundtablepresentations 18

FromWorkshop3 18

1

NewZealandisexposedtoawiderangeofnaturaldisasters,innosmallpartbecauseNewZealandstraddlestheboundarybetweentwotectonicplateswithitsattendantriskofearthquakesandvolcaniceruptions.TheNewZealandGovernmenthascommittedacrossseveralplatformstobuildingtheresilienceofitspeople,places,andeconomytoensuresafety,stability,andprosperityinthefaceofsignificantexposuretodisruption.

Understandingthecurrentstateofthenation’sresilienceandachievingsystemicimprovementsrequirescross-institutionalandtransdisciplinarycollaborationandresearchinnovation.Suchmodels,however,presentchallengestothestatusquoofdatasharingandmanagement.InformationinefficienciesandgapshindertheprogressofthosetacklingNewZealand’smostcomplexandimportantissues.Thesuccessoftheseresearchprogrammesrequiresanewknowledgedevelopmentethos.Meaningfulcollaborationacrossinstitutionsanddisciplinesrequireseffectiveinformationmanagement.Thismeanscreatingspaceswheredatacanbecaptured,safelyshared,andmanagedtoensurequality,appropriateuse,andongoingdevelopment.

In2016,theNZCentreforEarthquakeResilience(QuakeCoRE)andtheResiliencetoNature’sChallenges–NationalScienceChallenge,fundedasmallteamofresearchersfromResilientOrganisationsLtd.incollaborationwithUCCEISMIC(theCanterburyEarthquakeDigitalArchive)toinvestigatehowtobestenableteamsofresearcherstoaddresscomplexsocialproblemsthatwillmakeNewZealandmoreresilient.

Usingadesign-thinkingapproachtheresearchteamdevelopedtheDataIntegrationandVisualisationEnmass(DIVE)platform.Althoughitisstillintheearlyphasesofdevelopment,theDIVEplatformencompassesdatamanagementprocesses,institutionalstructures,web-basedsoftware,andstakeholders(i.e.,thepeopleandorganisationsthatwilluseDIVE).TheaimoftheDIVEplatformistoenableteamsofresearchers,decisionmakers,andpractitionerstoaddresscomplexproblemsby:

1. Makingdatasharingsafe,easy,anddesirable;2. Establishingstandardsandguidancefortransdisciplinarydatamanagement;3. Enhancingaccesstopublic,proprietary,andsensitivedatasources;and4. Facilitatingdataintegration,analysis,andvisualisation.

Thisreportdetailstheworkconductedthroughout2016todeveloptheDIVEplatform.WebeginbydescribingthemethodsusedtodeveloptheDIVEplatform,followedbyasummaryoffindingsandoutputsofthestakeholderconsultationandinitialdevelopmentprocesses.ThissectionfeaturesthecurrentcapabilitiesoftheDIVEplatforminitsinitialprototypeform.Thereportconcludeswithanoverviewofthebenefitsandopportunitiesprovidedbytheplatform,andanoutlookforfuturedevelopment.

1.Introduction

2

TheDIVEdevelopmentteamusedadesign-thinkingapproachtoscopeanddesignthefirstDIVEprototype.Designthinkingisasolution-focusedprocesswhichincorporatesthein-depthinsightsofend-usersintoiterativeprototypedevelopment(Brown&Wyatt,2010).Descriptionsofdesignthinkingrefertoitasasystemofoverlappingspaces,asopposedtosequentialsteps.BrownandWyatt(2010)labelthesespaces:inspiration,ideation,andimplementation,whereinspirationis,“theproblemoropportunitythatmotivatesthesearchforsolutions”(p.33);ideationis“theprocessofgenerating,developing,andtestingideas”(p.33);andimplementationis,“thepaththatleadsfromtheprojectstageintopeople’slives”(p.33).

Theinspirationphasetypicallybeginswitha‘brief’,whichprovidesageneralframeworkofconstraintsandgoalsofthedesignprocessandbenchmarksagainstwhichprogresscanbemeasured.ForDIVEthebriefwastocreatedatamanagementsystemsthatenableteamsofresearcherstoaddresscomplexsocialproblemsthatmakeNewZealandmoreresilienttohazardsanddisasters.Thestandardagainstwhichthissystemwasbenchmarkediswhetheritisuseful,usable,andused.Theseprincipleswerelooselydefinedintheearlyphaseoftheprojectandcanbeguidedbyaseriesofquestionsbasedonanevolvingunderstandingofthesystem(SeeBox1).

Oncethebriefisset,the‘inspiration’phaseisfocusedonexploringtheneedsofstakeholdersthroughdirectconsultationandobservation.QuakeCoREandRNCresearchers,alongwithkeydataproviderssuchasgovernmentministries,localcouncils,andtheEarthquakeCommission,comprisedtheinitialstakeholdergroupwhoseneedsweassessed.Webeganthisprocesswithaworkshopandanassessmentofthewaystakeholdersareinteractingwithotherdatamanagementsystems(Figure1).Theseprocessesweresupplementedbyanonlinesurvey(theresultsofwhicharesummarisedinStevenson,Brown,&Vargo2016)andseveralinformalinterviewswithsubjectmatterexpertstogathermorein-depthinformationaboutuser-needsandprocessesthatmaybeuseful.

Thesecondspaceofdesignthinkingis‘ideation’,whereinsightsgeneratedintheinspirationphasearesynthesised.Themostsalientobservationsareidentifiedandtranslatedintovisionsandchoicesthatguidethedesignofthesystem(Brown&Wyatt,2010).InthefirstiterationoftheideationspacewesynthesisedthediscussionatWorkshop1,thestakeholdersurvey,andexpertinterviews,alongwithobservationsofhowresearchersareinteractingwithextantsystems(i.e.,DesignSafe,theNewZealandGeotechnicalDatabase,andtheAustralianUrbanResearchInfrastructureNetwork)intoareportandaseriesofusecases.

Box1.PrinciplesforevaluatingthesuccessofDIVE.

Useful:Isthedataup-to-date?Isthequalityofthedatabeingmanaged?Canothersunderstandandusethedatathatisbeinguploaded?

Useable:Isdatasearchableandaccessibleforawiderangeofusers?Forexample,canitfacilitate‘citizenscience’orcouncildatacollectioneffortsifthatiswhattheusersneed?

Used:Arecommunitiesofpracticebeingestablishedandself-sustaining?

2.DIVEDevelopmentProcess

3

Operationalusecasesdescribeaflowofoperationsforthoseinteractingwithasystem,andcanbeusedtoidentifythefunctions,operatingsystems,boundaries,andconstraintsthatarerelevanttopotentialusers(Summers,2012).Theusecasesfocusedonimportantdata-relatedchallengesfacingQuakeCoREandResiliencetoNature’sChallengesNationalScienceChallengeresearchers.

Thecasessynthesizeconceptsthatemergedfromtheinitialinspirationphase,including:

• Howsharingprimarydatasets(i.e.,generatedbyoriginalresearchforagivenpurpose)caninspirenovelsecondaryusesinawaythatenhancesthevalueoftheoriginalwork.

• Theadvantagesofafederatedsearchengineandtheabilitytoviewdatathatisnotstoredlocally.

• Thecomplexityofdatasetswithconfidentialelementsandtheneedforstreamlinedprocessesforsharingandaccessingsensitivedatasets.

• Theprocessesandsystemsneededtostandardiseandeasethecollectionofbuildinginventorydata,enhancebuildingdataintegration,andfacilitatedatasharing&visibility.

• Theneedtocaptureongoingandplannedresearchtoreduceoverlap,reducetheburdenondataprovidersandthesubjectsofsocialresearch,andtoimproveopportunitiesforcollaboration.

MoreontheseusecasescanbefoundinAppendixB.

Figure1.DesignThinkingMethodologyfortheDIVEPlatformshowingtheprogressionofphasesfrominspirationtoideationtoimplementation,andtheassociatedstagesofdevelopment.

InWorkshop2wepresentedsomeofthesesynthesizedfindingsandexaminedworkflowandassimilationprocessesandproblemsolvinginextantsystemsseekingfeedbackfromworkshopparticipants.Theoutcomesfromthisworkshopwereagainsynthesizedinareport.ThesubjectsdiscussedandoutcomesresultingfromWorkshops1and2canbefoundinStevensonetal.(2016)andStevensonandVargo(2016),respectively.

AfterWorkshop2weenteredthe‘implementation’phase–developingthepre-alphaprototype1oftheDIVEsoftwaresystem.Thisrepresentsthefirstiterationinamulti-prototyperoll-out.TheproposedDIVEversioningprogramme(Figure2)showsthecurrentstatusoftheDIVEPlatformandestimatesdevelopmentprogressoverthenextfewyears.

Thedevelopmentteamre-enteredtheinspirationphaseofthedesignprocessinWorkshop3.ThisworkshopwasheldinNovember2016attheUniversityofCanterbury,andincludedanoverview

1‘Pre-alpha’referstoallsoftwaredevelopmentactivitiesbeforeformaltesting.

• Workshops1,2&3• Interviews,surveys,observationInspiration

• Developmentofplatformusecases• Reporting&refinementIdeation

• Pre-alphaprototypedevelopment• AlphaprototypedevelopmentImplementation

4

andreflectionontheDIVEPlatformdevelopmentprocessoverthepreviousyear,ademonstrationoftheDIVEprototype’scurrentfeatures,andadiscussionabouthowtheDIVEPlatformmaybeusedtoaddresskeyissuesinresilienceresearch.ThiswasfollowedbyanAffiliatedResearchersRoundTable,whichincludedaseriesof9shortpresentations(summariesofthesepresentationscanbefoundinAppendixC),andarobustgroupdiscussion.

Figure2.DIVEVersioningProgrammeasofNovember2016.

Throughouttheinitialdevelopmentphase,approximately65stakeholdersandsubjectmatterexpertscontributedtotheDIVEdevelopmentprocess,engaginginworkshops(Figure3),surveys,andinterviews.Theworkshops,surveys,andinterviewshelpedthedevelopmentteamoutlinecriticalprocessesforsuccessfulinitiationoftheDIVEPlatformandprovidedimportantinputtotheinitialdesignoftheDIVEsoftwarepre-alphaprototype.Theworkshopswerealsoahelpfulstartingpointforformingtherelationshipsthatwillbecentraltothesuccessfulimplementationofatransdisciplinaryandcross-institutionalcollaborativeplatform.

Wewillcontinuethisiterativedesign-thinkingprocesstoensurethatDIVEbecomesasystemthatisuseful,useable,andusedbythethoseworkingtosolveNewZealand’sresiliencechallenges.

5

Figure3.TitlesofDIVEworkshopsheldin2016.

TheDIVEprogrammeisconstructedaroundabrieftodevelopadatamanagementsystemforresearchersworkingtowardenhancingresilienceinNewZealand.ItbecameclearearlyintheconsultationphasethatthereisjustifiedscepticismamongresearchersacrosstheNationalScienceChallengesandQuakeCoREaboutthepotentialefficacyofanewdatamanagementplatform.Resistanceisassociatedwiththreeprimaryissues:

1) Thereisalreadyasignificantreportingburdenonresearchers.Learninganewsystemormeetingdatasharingexpectationsrepresentadditionaltimecoststoresearchers.

2) Buildingasystemaroundarelativelyshort-livedresearchprogramme(whichincludesboththeNationalScienceChallengesandQuakeCoRE)presentstheriskoffuturedatalossorasystemthatisnotmaintainedandbecomesobsolete.

3) Astherearealreadylarge-scaledataintegrationschemesanddatarepositoriesinNewZealandandabroad,thereisariskofcreatingredundancy,inefficiencies,andmultiplecopiesofdata.

Whiletheseissuesdonotobviatetheneedfordatamanagementsystems,theyserveasimportantguidesforongoingdevelopment.ItwasclearfromthedesignandconsultationprocessthatexistingmanagementsystemsarenotcurrentlymeetingtheneedsofresearchersengagedintheResiliencetoNature’sChallengesandQuakeCoREresearchprograms.Problemswiththewaydataiscurrentlybeingmanagedinclude:

• Researchersandstakeholdersnotbeingawareofongoingworkordatasetsthathavenotyetbeenpublished;

• Researchershavingnocentralportalforfederateddatasharing;• Researchershavingdifferentunderstandingsofkeyconcepts,andthereforedifferent

systemsforrecordingandcommunicatingdata;• Havingnoagreedsystemformanagingandcommunicatingthequalityorcompletenessof

datasets;• Poororinconsistentmanagementofgeographicdata;and

Workshop3(November2016)Demonstratetheprototype&exploringkeyissuesinresilience

researchwithDIVEsoftwareapplications

Workshop2(July2016)Prototypingworkflow&assimilationprocesses

Workshop1(May2016)Identifyingdataintegration&federationissues&shortlisttools

3.FindingsandOutputs

6

• Lackofclarityaroundsharingdataforrestrictedaccessdatasets(i.e.,Tier2andTier3dataasinTable2).

Table2.Tiersofaccessforkeydisasterresiliencerelateddatasets.

Tier1 Tier2 Tier3

Somekeydatasets(i.e.taxstatisticsfromtheInlandRevenueDepartmentandCensusdata)arepubliclyavailableandwillbefederatedbyDIVEthroughthedata.govt.nzandDigitalNZAPI’s.

OtherkeydatasetshaverestrictedaccessandwillrequirefurthernegotiationtosetupstreamlinedaccessthroughDIVE–forexample,itemsheldintheNZGeotechnicalDatabase,EQCclaimsdata,andtheACCearthquakerelated-injurydatabase.Also,includedinthiscategoryarefilescreatedbyQuakeCoREandRNCresearcherssuchastheEconomicsofResilientInfrastructurebusinesssurveydata.

Themostdifficultkeydatasetsincludeproprietarydataheldbycommercialproviders,includingPaymarkandMarketviewretailtransactions,cellphoneusage,andprivateinsuranceclaims.Currentlythisdatahasbeenobtainedadhocbysomeresearchers–apossiblesolutiontoexploreisnegotiatingmoreinclusivedatasharingagreementsthroughtheDIVEplatformwithholdersofproprietarydata.

WiththesefindingsinmindwearetakingamultifacetedapproachtothedesignofDIVEgoingforward.TheDIVEPlatformincludestheprocesses,institutionalstructures,software,andstakeholdersusingtheplatform.

3.1 ProcessesProcessesarethefoundationofausefuldatamanagementplatform.Processesincludethestandardsthatdevelopersandusersespousetoensurethatdataiscapturedandsharedinwaysthatallowittobeunderstoodandusedbyothers.Processesalsorefertothewaydataanddatamanagementsystemsaremaintained.Suchprocessesinclude:

• Identifyingpotentialusersandassessingtheirneeds(whichwehavedoneaspartoftheDIVE2016workshopseries);

• Establishingongoingmonitoringofuserneedsandprioritisingareasforinvestment;• Buildingandcuratingrelationshipswithdataproviders(e.g.,EQC,LINZ,CrownResearch

Institutes);• Evaluatingandcreatingproceduresforaddressingresearchethicsandsecurityissues;• Creatingandmaintainingmetadatastandards,datadictionaries,datacatalogues,andother

relatedsystemsthatenhancedatasearchabilityandintegration.

TheseandotherprocesseswillbeinstrumentaltoensuringthattheDIVEPlatformachievesitsgoalofbeinguseful,usable,andused.

3.2 InstitutionalstructuresInstitutionalstructuresrefertothelong-termgovernanceandresourcingofDIVE.EnsuringtheongoingdevelopmentandmaintenanceofdatarequiredtoimproveNewZealand’sresilience–

7

includinguniquedatasetscreatedbyQuakeCoRE,ResiliencetoNature’sChallenges,andothers–needstobepartofthePlatformdevelopment.Identifyingappropriateinstitutionalstructureandplansisadevelopmentprioritygoingforward.

3.3 SoftwareThecentrepieceoftheDIVEPlatformistheweb-basedsoftwarethatprovidesadigitalportalforfederatingdata,searchengine,anddatamanagementandanalysisapplications.TheDIVE(pre-Alpha)prototypehasbeendevelopedusingCKAN,anopensourcedataplatformwhichprovidestoolstostreamlinepublishing,sharing,finding,andusingdata.Atitsbaseinstall,CKANactsasadatafederationportalandcatalogue,aswellashavingseveralbuilt-intoolsforexploringandunderstandingdata.

Currently,theplatformallowsuserstouploadandexploredatawithplanstoaddtotheintegrateandanalysefunctionalitieslater.Theplatformalsocomeswithmanyinbuiltwaysofvisualisingdata,suchasspreadsheetswhichcanbeviewedasrawdata,asagraph,oramapifcoordinatesareincluded.ImagescanbedisplayedandHTMLandPDFpagescanbeviewedandinteractedwithonthesite.

TheDIVEprototypealsoincludesanAPIwhichallowsdatatobepulledoutorpushedintotheplatform.TheAPIwillallowresearcherstointegratesomeofthetoolstheyarealreadyusingortoolscreatedspecificallyfortheDIVEwebsite.

ForfurtherexamplesoftheDIVEplatforminterfaceincludingsearchcapabilitiesanddatavisualisationsseeAppendixA.

3.4 StakeholdersThedriversbehindtheDIVEPlatformarethestakeholders.Stakeholdersrefertodataprovidersanddatausers,thoughmanystakeholderswillbeboth.StakeholdersguidethedesignanddevelopmentofDIVE.Theywilltestandrefineprototypeiterations,andthroughtheirinvestmentoftimeandinformationwilldeterminethesuccessoftheplatform.

ThestakeholdersfortheinitialprototypeoftheDIVEPlatformarepeopleinvolvedinQuakeCoRE,theResiliencetoNature’sChallenges–NationalScienceChallenge,CEISMIC,andseveralgovernmentandprivateresearchorganisationsthatwererepresentedatDIVEworkshopsthroughout2016.ThisbodyofstakeholderswillcontinuetogrowastheDIVEwebinterfacebecomesmorewidelyavailable.

3.5 ObservationsabouttheDataDevelopmentContextThedesignthinkingprocessassociatedwiththedevelopmentofDIVElendsitselftoaniterativecreativeprocess.ThecurrentDIVEsoftwareprototypeprovidesabasicdesignformeetingresearchers’immediateneedsandindicatespathwaysformoreadvancedprocessesgoingforward.

Figure4showsthehierarchyofdatamanagementneedsthatweidentifiedthroughstakeholderconsultation.ThemostbasicandfoundationalneedforstakeholdersworkingtoenhanceresilienceinNewZealandisensuringthatdataiscapturedandpreserved.Thisrequiresaportalforhostingandfederatingdataand,moreimportantly,appropriateintegrationofresearchdatamanagementandethicsconsiderationsintotechnicalsystems.TheNewZealandGovernmentOpenAccessLicensing(NZGOAL)Frameworkprovidesafoundationforachievingthis,butworkisstillrequired

8

withintheresearchcommunitytomakerobustmechanismsforthere-useofresearchdatapartofmainstreampractice.

Figure4.Hierarchyofdatamanagementneedsasidentifiedthroughstakeholderconsultation

Somedatasetswillonlyneedshort-termstorage.Inmostinstances,thiswillbemanagedbyresearcherswhointendtoterminatealldataaftertheprojectiscompleteandtheresultsarepublished.

Datasetswithre-usevalue,however,shouldbearchived,andwherepossiblemadeavailableforfutureuse.Researcherswilldesigntheirownadhocsystemsforshort-termstorage(e.g.,internalorexternalharddrives).Storagethatextendsbeyondthelifeoftheresearchprogramme(i.e.,archiving)needstobesaveddifferently,ideallyinatrustedrepositorymanagedbyastableinstitution.

ItwillbevaluablefortheDIVEPlatformprocessestoincludeguidelinesonthedesignofresearchprojectssothatresearcherscanmakeup-frontdecisionsabouthowtheirresearchdatamightbereused,andunderstandthetrade-offsinvolvedineitherwaivingconfidentialityorremovingpersonaloridentifyinginformationfromtheshareddataset.Thedatathatissharedforsuchprojectswouldlikelybesubjecttostrictercontrolstoensurenopersonalinformationissharedinerror.

ThecurrentDIVEinterfacecanassistinapplyingrelevantmetadatatorecordsandcanenhanceandtransformpre-existinginformationwhenitisingestedviatheCKANAPI.Aspartofthedataingestionprocess,theDIVEinterfacecanusevalidationandrequiredfieldstoensuredataquality.

Thenextlayersofdatamanagementonthehierarchyaresystemsthatmakedatashareable.Whendataisstandardiseditmeansthatitisconsistentlydescribedandrecordedusingasetofacceptedrules(i.e.,metadatastandards)tosystematisetheformataswellasthemeaning(U.SGeological

9

Survey,2017).Metadatastandardscanalsobedevelopedtoimprovedatadiscovery,includingguidingresearchersto:

• Includetermsfoundinpublishedtaxonomiesintheirmetadata,• Writedescriptivetitlesandabstracts,and• ProvidethedirectURLtoonlineresourcesinmetadata.

Thenextlayer,‘Trusted’referstothewayothersperceivethequality,accuracy,andprecisionofthedatabeingsharedinarepository.Metadatastandardsandtoolsthatfacilitateconsistentrecordkeeping(e.g.,electroniclabnotebooks)canenhancethequalityofdata,andmakeitmorelikelythatitisproperlystructuredandadequatelyannotated,andthereforecomprehensible.

Trustisbuiltinthequalityofdatawhenitisaccessedthroughaportalwhereinformationisreviewedandcurated(i.e.,organized,described,cleaned,enhanced,andpreserved)forre-use.Adatamanagerforsuchasystemmayreviewfrequenciesorproducesummarystatisticsformissingvaluesindatasets,reviewdataforconfidentialityissues,andperformotherchecksthatincreasetheconsistency,completeness,andusabilityofthedata.

Thelastlevel,transformativedatamanagement,referstodatamanagementsystemsthatbecomeself-sustaining,leadtoimprovedresearch,andenhanceoutcomesforcommunities.Forexample,systemscanallowpublisheddatatobetrackedtounderstandhowit’sbeingusedtoanswernewquestions.Certainaspectsofdatacapturecanbeautomatedandsystemscanallow‘citizenscientists’toinputtheirowndata.

ThisreportdetailsthedevelopmentoftheDataIntegrationandVisualizationEnmass(DIVE)platform.Theaimoftheplatformistoenablestakeholderstoaddresscomplexproblemsbystreamliningdatasharing,enhancingaccesstosensitivedatasources,andfacilitatingdataintegration,analysis,andvisualization.TheDIVEplatformoffersmanypotentialopportunitiesforresearchers,suchasagreateropportunitytocollaboratewithanopen-accessdatasourceandeasieraccesstodataforfasterproblemsolving.

Usingadesignthinkingapproachwithstakeholderconsultationandworkshopsaninitialprototypewascreatedallowingdatatobeuploadedandexploredonasingleplatform.However,thereareanumberofchallengesassociatedwithdatasharingsuchasstandardizingdatafromdiversesources,datagovernance,datainteroperability,andcreatingorganizationalframeworksthatmustbetackledalongsidefutureiterationsoftheprototypeastheplatformmatures(Medyckyj-Scottetal.,2016).

Ratherthandealingwithdataissuesreactivelyandinanadhocfashion,anintegratedplatformwithchallengesacknowledgedup-frontandaspartofthedesigncanhelptodealwithissuesastheyarise.Ultimately,thevalueoftheDIVEplatformisevidentasaproof-of-concept,wherebyasoftware-basedplatformwascreatedwiththepurposeoffacilitatingdataprocessestosupportaproblem-basedfieldofenquiry.EnhancingresilienceinNewZealandiscriticaltoinvestinginthewellbeingofcommunities,organizations,andindividuals.Usingdatamoreeffectivelyisakeystepinachievingthisgoal.

4.Conclusions

10

Medyckyj-Scott,D.,Stock,K.,Gibb,R.,Gahegan,M.,Dzierzon,H.,Schmidt,J.,&Collins,A.(2016).OurLandandWaterNationalScienceChallenge:ADataEcosystemforLandandWaterDatatoAchievetheChallengeMission.LandcareResearchContractReport.

Stevenson,J.R.,&Vargo,J.(2016a).DataIntegrationandVisualisationPlatform:UsingtheDIVEPrototypetoAnswerSomeQuestionsandRaiseOthers.[WorkshopPresentation].DataIntegrationandVisualisationEnmasseWorkshop:29November2016.Christchurch,NewZealand.Retrievedfrom:https://wiki.canterbury.ac.nz/pages/viewpage.action?pageId=52692301

Stevenson,J.R.,Vargo,J.,&Brown,C.(2016).DataIntegrationandVisualisation:PrototypingtheQuakeCoREDataPlatformforDiverseNeeds.ResilientOrganisations/QuakeCoREResearchReport.Retrievedfrom:https://wiki.canterbury.ac.nz/pages/viewpage.action?pageId=52692301

Stevenson,J.R.,&Vargo,J.(2016b).DataIntegrationandVisualisationEnMasse(DIVE)PlatformReport:ProgrammeUpdateSeptember2016.ResilientOrganisations/QuakeCoREResearchReport.Retrievedfrom:https://wiki.canterbury.ac.nz/pages/viewpage.action?pageId=52692301

Summers,B.L.(2012).Effectivemethodsforsoftwareandsystemsintegration.CRCPress.

U.SGeologicalSurvey.(2017).USGSDataManagement.UnitedStatesGeologicalSurvey,U.S.DepartmentoftheInterior.RetrievedFebruary15th,2017,from:https://www2.usgs.gov/datamanagement/plan/datastandards.php

Watt,J.,&Brown,T.(2010).Designthinkingforsocialinnovation.StanfordSocialInnovationReview.RetrievedJune20th,2013.

References

11

Figure1.HomepageoftheDIVEpre-alphaprototypebuiltonaCKANinterface.

Figure2.AsearchwithfacetsintheDIVEplatform.

AppendixA:TheDIVEPlatformInterface

12

Figure3.MapviewinDIVEsearchinterfaceofGeonetearthquakedatafromFebruary2011.

Figure4.GraphviewofGeonetearthquakedatafromFebruary2011.

13

ThefollowingslidessummarisetheDIVESoftwareusecasespresentedinWorkshop3inNovember2016.ThefullpresentationcanbefoundontheDIVEWikiPage.2

2DIVEWiki:https://wiki.canterbury.ac.nz/pages/viewpage.action?pageId=52692301

www.resorgs.org.nz

UseCases

UseCase1:Federation&analysis

UseCase2:Addressingobstaclestodatasharing&dataintegration

UseCase3:Creatingasafehomeforuniquedata&metadatastandardisation

UseCase4:Enhancingsearchability

UseCase1:NewQuestionsfromOldData

Ishazardplanningadequateinsociallydeprivedcommunities?

DIVE Platform Hosts & Harvests Existing Data• Plan Content Analysis 2014 Uploaded by QC

researcher• NZDep Data, University of Otago - Wellington• Active faults database, GNS

Integrates & Displays• Joins the relevant data in a relational

database and displays preview. This may also occur after download or by integrating all of the data on an open source GIS Web App like Koordinates.

AppendixB:UseCaseSummariesfromWorkshop3

14

UseCase1:Conclusions• Researchersarecreatingdatasetsthathavevaluefar-beyondtheirinitialresearchquestions.

• ThesecanbeuploadedtoDIVEandinspiresecondaryusesthattheoriginalresearchershadnotevenconsidered.

• Federatedsearchandviewingdatathatisnotstoredlocallywillenhanceexplorationandinnovation

UseCase2:ImprovingDataAccess&AddingValuetoDataLinkingBuildingPropertiestoEarthquakeInducedDamageandBusinessDowntimeusingFEMAP-58andREDIAssessments

GemmaCremen,JackW.Baker,SoniaGiovinazzi,EricaSeville[1]

BuildingProperties DamageBenchmarks

CEBADatabase

CCCBuildingFootprint

CEBADatabase

ERISurveyDatabase

• 3Non-disclosureagreements(NDAs)

• 3waysofidentifyingthesamebuildings• Prupi*,UniqueID,

Lat/Long

*Prupi istheIDusedbytheChristchurchCityCouncilbeforetheChristchurchearthquake.TheyswitchedtheuniqueIDsystemaftertheearthquake.TheEQCusesadifferentsetofIDsforbuildings.

15

UseCase2:ImprovingDataAccess&AddingValuetoData

“Preliminaryresults[show]fromanefforttousetheFEMAP-58andREDi seismic

assessmentprocedurestopredictdamageanddowntimeforasampleofbuildings…

[that]bothproceduresappeartooverestimateearthquakeimpacts...Future

evaluationsofalargersetofbuildingswillprovidemoredefinitiveresults.”(Cremen

etal.,2016).

UseCase2:Conclusions• GemmaCremen andcolleagueswillcreateavaluableuniquedatasetwiththecombineddatabases,butthereisnoclearwaytosharethisvalue-addeddatasetwithoutallfutureresearcherssigningnon-disclosureagreementswithalloftheoriginaldataholders.

• TheDIVEplatformhasthepotentialtodevelopprocessesthateaseadministrativeburden.• Buildingonexistingexpertiseandexperiencewithdatasharingagreements,permissions,ethics,andcopyrights.

16

UseCase3:HostingUniqueData

&MetadataStandardisation

[2]ScatteredmasonrythathasfallenfromStJohnthe

BaptistChurchinLatimerSquare

CollectionFieldsforBuildingInventories

BuildingName

BuildingsAddress

YearBuilt

Constructiontype

Descriptionofconcreteextractionlocation

Testingstandardsused

Specimendimensions

Concretecompressionstrength

Concreteelasticmodulus

Concretedensity

Descriptionofsteelextractionlocation

Testingstandardsused

Specimendimensions

Steeltensilestrength

Steelelasticmodulus

CapitalValue(anddateofvalue)…

UseCase3:Conclusions

• QuakeCoREResearchersandotherswillbecollectingrichdatabasesaboutNewZealand’sbuildinginventories.Thereiscurrentlyno‘home’forthisdata.

• WewillworktowarddevelopingaBuildingInventoryModuleforDIVEthatwillstandardise andeasethecollectionofbuildinginventorydata,enhancebuildingdataintegration,andfacilitatedatasharing&visibility.

• Worktowardbetterliveupdating,archiving,andversioncontrolprocessesforcollaborativelyproduceddatasets.

17

ThefullpresentationcanbefoundontheDIVEWikiPage:

https://wiki.canterbury.ac.nz/pages/viewpage.action?pageId=52692301

UseCase4:KaikouraEarthquakes

• Capturingplannedandongoingresearch/datacollection

• Paralleldatabases• Sourcing,

uploading,andmanagingdataqualityisalotofwork

• Thereisagreatneedtocaptureongoingandplannedresearch

• Sourcing,uploading,andcapturinghighqualitymetadatadataisalotofwork• Howcanbetterautomateand/ordistributethisburden

• Visualising metadatacanmakeitmoresearchable

UseCase4:Conclusions

18

SlidesareavailableforthepresentationsdescribedinTable1inaseparatedocument.

Table1.AffiliatedresearchersroundtablepresentationsfromWorkshop3,2016.

Presenter SelectedAffiliations PresentationTitle ThesisorHighlights

MatthewHughes

UniversityofCanterbury&QuakeCoRE

Findingandsharingdatainapost-disasterenvironment

Relationshipbuildingandestablishingstrongdegreesoftrustwithdataprovidingagenciesiscriticalforsuccessfulandmeaningfulco-creation.

DavidJohnston

NaturalHazardsResearchPlatform,JointCentreforDisasterResearch(MasseyUniversity)&UnitedNationsOfficeforDisasterRiskReduction

Largescalecross-institutioncollaboration

TheInternationalCouncilforScience’sCODATA,theIRDR’sDisasterLossData(DATA)project,andtheRHISEgroup(researchingthehealthimplicationsofseismicevents)provideusefulinternationalandnationalguidanceforthedevelopmentofDIVE.

AustenGanley

UniversityofAuckland&theBiologicalHeritageNationalScienceChallenge

AvirtualhubfornationwidesharingofenvironmentalDNAsequencedata

AspartoftheBH–NSC,developingasystemwhereDNAdatacanbesharedandusedbythosewithalllevelsofexpertise.

Criticalareasincludevisualization,databasecreation,standardizingmetadatacollection,andpotentiallycreatinganappsopeoplecollectandcontributemetadataeasily(likeGeoNet).

ByronCochrane

LandInformationNewZealand

Internationalstandardsandbestpractices

LINZisdrawingoninternationalstandards,bestpractices,andexistingtoolsandexpertise(e.g.,spatialdataonthewebworkinggroup)todeveloptheirspatialdatainfrastructure.

Sheng-LinLin

GNS PrinciplesfordevelopingONEstandardized,extensible,updatablebuildingdataset

Therearesignificantchallengeswithmanagingandsharingearthquakeaffectedandearthquakepronebuildingdata,includingnonation-wideuniqueID,nostandardizeddatacaptureprocess,andissueswithconfidentiality.

AppendixC:AffiliatedresearchersroundtablepresentationsFromWorkshop3

19

ChrisBowie

OpusInternationalConsulting

NaturalHazardsResearchPlatform–NeighbourhoodRecoveryTrajectories

Researchersneedtobudgettoassessthequalityofandto‘clean’largedatasets–itisaresourceandtimeconsumingprocess.Researcherswouldbenefitfromacollaborativeweb-spacetosharedata.

KyleDow

ChristchurchCityCouncil

‘Addresses–who’sdoyouuse’&‘Knowingwhoisusingyourdata’

Therearedataqualityissuestobeawareofwhenusingaddressestospatiallylocatedata.AbetteroptionistousebuildingIDsratherthanaddresseswherepossible.

Dataproviders(e.g.,theCCC)haveconcernsaboutmanaginguseandmonitoringhowpeopleuseandchangethedata.

AlistairRitchie

Landcare Pervasiveissuesaffectingouroperationalandcommercialsystems

Datamanagementpresentstechnical,social,andfinancialproblems.AkeychallengeforLandcare:howtointegrateanddisseminatedatadescribingmanyaspectsoftheenvironmentinawaythatsupportsmonitoring,analysisandreporting.

SimonKingham

UniversityofCanterbury&theBetterHomes,Towns,andCitiesNationalScienceChallenge

Nextgenerationinformationforbetteroutcomes

Weneedtoalignthewaywetalkaboutspatialdata.Ifwewantdatathatcan‘worktogether’focusingonspatialstandardswillhelpsignificantly.

20

InNewZealand,thereareseveralinformationaggregationportalsforhazardsdata–notablyTonkin&Taylor’sGeospatialPortal,theNZGeotechnicalDatabase,theCEISMICdigitalarchive,andthepost-earthquakedataclearinghousesystemhostedbyTheEarthquakeEngineeringResearchInstitute(EERI).

Theseareusefulrepositories,butareheavilyfocusedonearthquakeandgeotechnicaldata.NZGDfocusesexclusivelyondatarelevanttoengineers,andCEISMICprovidesmainlydocumentaryresources,withanemphasisonimages,newsmedia,videoandaudiofiles,andassociatedculturalheritagecollections.

TheEarthquakeEngineeringResearchInstitute(EERI)supportsaninternationalvirtualclearinghousesystem,whichhasbeenemployedtocapturevaluabledatagatheredbyresearchersandpractitionersworkingacrossmanyinstitutionsintheaftermathofthe2010and2011Canterburyearthquakesandthe2016Kaikouraearthquake.TheKaikouraEarthquakeVirtualClearinghousewebsitewasestablishedforthosewishingtopublishinformationrelevanttothepublicandinternationalresearchers.Thiseffortwaslargelydrivenbytheearthquakeengineeringcommunity,butmanagedtocapturemediaarticles,photographs,mapsandsocial-economicinformationthatarerelevantacrossabroadrangeofapplications.However,theEERIdataclearinghousesaregearedtowardsinitialdatacaptureafteranearthquake,andarenotupdatedwithrecoverydata.

https://www.nzgd.org.nz/HelpSupport/AboutNZGD.pdf?AspxAutoDetectCookieSupport=1

http://www.ceismic.org.nz

http://www.eqclearinghouse.org

http://www.eqclearinghouse.org/2016-11-13-kaikoura/

OtheraggregationportalsforhazardsdatainNewZealandincludeNIWA’sHistoricWeatherEventsCatalogue,GeoNet’sGeohazardsapplicationsanddata,andplaceswherecitizenscanengageinthecollectionofcriticalenvironmentaldata,suchasGeoNet’sFeltReportsandNIWA’scommunityairqualityobservationnetwork.

Thereisnotyetaunifiedspacewhereindependentresearchersandcitizenscientistscansharetheinformationtheyarecollectingacrossmultiplehazardsinawaythatiscuratedandarchived.Mountainsofinformationarecollectedfollowingtheseevents.Someofthedatacollectedbycrownresearchersispublishedontheirinstitutions’websites,andlessonslearnedarepublishedinthemonthsandyearsfollowingtheevent,butmostdataarestoredinanadhocmannerandeasilylostorburied.

https://felt.geonet.org.nz

https://www.niwa.co.nz/atmosphere/researchprojects/CommunityObservationNetworksforAir/rangiorasurvey

AppendixD:ExistingHazardsDatabases&DesignSafe-CI

21

DesignSafe-CIisaflexible,extensible,community-drivencyber-infrastructureforthenaturalhazardsengineeringresearchcommunity,basedintheUS.

Featuresinclude:

• Excellenttoolsforanalysingdata• Interactiveforums• Regularworkshopsforusers

DeseignSafe-CIfacilitatescollaborationanddatasharingwithotherresearchers,andsupportsthefulllifecycleofdataneededtoaddresscivilinfrastructurethreatsposedbynaturalhazards.

LiketheNZGDitisgearedtowardengineers,andalthoughtheyareveryinterestedinengagingwithaglobalresearchcommunity,mostofthedataontheplatformiscurrentlyfromresearchersintheUS.

https://www.designsafe-ci.org