This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 700475

beAWAREEnhancingdecisionsupportandmanagementservicesinextremeweather

climateevents

700475

D3.2Empiricalstudyandannotationof

multilingualandmultimediamaterialinbeAWARE

Disseminationlevel:

Public

Contractualdateof

delivery:

31August2017

Actualdateofdelivery:

10September2017

Workpackage:

WP3Earlywarninggeneration

Task: T3.2,T3.3

Type: Report

ApprovalStatus:

FinalDraft

Version: 0.4

Numberofpages:

31

Ref. Ares(2017)4413776 - 10/09/2017

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 700475

Filename: D3.2_beAWARE_empirical_study_of_multilingual_and_multimedia_material_2017-09-08_v0.4.docx

Abstract

Thisdeliverablereportstheresultsoftheempiricalstudyofthemultimediainputs,namelyaudio,text,audioandvisualdata,pertinenttothebeAWAREpilots,inaccordancewiththeusecasesandinitial requirements as described in D2.1. As described, the data collection includes: i) historicalmaterial made available by the user partners, ii) datasets compiled within beAWARE, and iii)publiclyavailabledatasetsthatarerelevanttothebeAWAREdomains.TheoutcomesofthestudydelineatetheinitialsetofspecificationsandrequirementswithrespecttothetargetedinformationofinterestforthecontentanalysisanddistillationtasksaddressedinT3.2andT3.3respectively.Theinformationinthisdocumentreflectsonlytheauthor’sviewsandtheEuropeanCommunityisnotliableforanyusethatmaybemadeoftheinformationcontainedtherein.Theinformationinthisdocumentisprovidedasisandnoguaranteeorwarrantyisgiventhattheinformationisfitforanyparticularpurpose.Theuserthereofusestheinformationatitssoleriskandliability.

Co-fundedbytheEuropeanUnion

D3.2–V0.4 D8.1–V1.0

Page3

HistoryVersion Date Reason Revisedby

0.1 14.07.2017 Deliverablestructure,abstract S.Dasiopoulou

0.2 29.07.2017 Contributionstosections2,3&4 D.Liparas,K.Avgerinakis,E.Michail,J.Grivolla

0.3 29.08.2017 Updatestosections3&4 K.Avgerinakis,E.Michail,J.Grivolla

0.3.1 08.09.2017 Revision,completionandfinalizationofsections2,3&4;executivesummary,introduction&conclusionsections

S.Dasiopoulou

0.4 09.09.2017 Internalreview AnastasiosKarakostas(CERTH)

AuthorlistOrganisation Name ContactInformation

UPF StamatiaDasiopoulou stamatia.dasiopoulou@upf.edu

UPF JensGrivolla jens.grivolla@upf.edu

UPF LeoWanner leo.wanner@upf.edu

CERTH DimitriosLiparas dliparas@iti.gr

CERTH KonstantinosAvgerinakis koafgeri@iti.gr

CERTH EmmanouilMichail michem@iti.gr

D3.2–V0.4 D8.1–V1.0

Page4

ExecutiveSummaryThisdeliverablereportsontheaudio,textandvisualdatasetsthathavebeencollectedandstudiedduring theperiodM1-M8aspartof the investigationscarriedoutwithinT3.2andT3.3towardsthedevelopmentofrespectivecontentanalysistechniquesthatwillenablethedistillation of the pertinent semantic information. More specifically, D3.2 explicates themultitudeofaudio,textandvisualinformationinputsconsideredwithinbeAWARE,basedonthe initial requirements anduse case scenario descriptions laid out inD2.1, describes thecurrentlyacquireddatasets,andreportsonthefindingsoftheirempiricalstudytowardstheelicitationofpreliminaryspecificationsandrequirementsfortherespectiveanalysistasks.

D3.2–V0.4 D8.1–V1.0

Page5

AbbreviationsandAcronyms

ASR AutomaticSpeechRecognition

CCTV ClosedCircuitTeleVision

OWL WebOntologyLanguage

RDF ResourceDescriptionFramework

UAV UnmannedAerialVehicles

D3.2–V0.4 D8.1–V1.0

Page6

TableofContents

1 INTRODUCTION............................................................................................................7

2 MULTIMEDIADATAINPUTSINBEAWARE.....................................................................92.1 Audiodatainputs............................................................................................................92.2 Textualdatainputs.........................................................................................................92.3 Visualdatainputs.........................................................................................................10

3 MULTIMEDIADATACOLLECTION................................................................................113.1 Historicaldatasetsprovidedbyuserpartners...............................................................113.2 DatasetscompiledwithinbeAWARE.............................................................................133.3 Benchmarkdatasets......................................................................................................15

3.3.1 Audiodatasets.................................................................................................................153.3.2 Textdatasets...................................................................................................................163.3.3 Visualdatasets.................................................................................................................16

4 EMPIRICALSTUDY......................................................................................................194.1 Audiodatastudy...........................................................................................................194.2 Textdatastudy.............................................................................................................20

4.2.1 Tweetlocationmetadata.................................................................................................204.2.2 Tweetsprovenanceandinformationcontent.................................................................214.2.2.1 Authorship...................................................................................................................214.2.2.1 Categories...................................................................................................................234.2.2.2 Location.......................................................................................................................27

4.3 Visualdatastudy..........................................................................................................27

5 SUMMARY..................................................................................................................30

D3.2–V0.4 D8.1–V1.0

Page7

1 IntroductionbeAWARE advocates an integrated solution to support and enhance crisis managementencompassingforecasting,earlywarninggeneration,transmissionandroutingofemergencyrelated information, aggregated analysis of multimodal data, and management of thecoordinationbetweenthefirstrespondersandtheauthorities.To achieve this, it capitalizes on amultitude of inputs, including pertinentmeteorologicaland forecasting data (e.g. heat index, forecasted values and duration forwind speed andtemperature,emergencypredictionmodelssuchastheAMICOfloodwarningsystem,etc.),sensor readings that provide real-time opportunistic sensing information (e.g. water levelsensors, temperature sensors, etc.), cameras affording continuous streams of the visualcontext for the referencephysical vicinity, socialmedia, aswell as direct communicationsbetweenciviliansandauthorities (e.g. calls toemergencynumbers, incidence reports sentvia the beAWARE mobile application to the PSAP), and between first responders andauthorities (e.g. radiocommunication,assignedtaskstatusupdatessentvia thebeAWAREmobile application). The informationextracted through the analysis of theheterogeneousinput sources feeds the aggregation and semantic integration techniques for decisionsupportandearlywarningsgenerationthataredevelopedwithinWP4.Inthisdeliverable,weelaborateonthemultimediacontentinputs,namelyaudio,textandvisual,andexplicatethedatasetsthathavebeencurrentlycollectedaswellastheoutcomesof their empirical study that lay the groundwork for the semantic analysis tasks T3.2 andT3.3.Asdescribedinthefollowing,thecollectionofthedatasetsservesatwofoldpurpose:i)toaffordreferencematerialforunderstandingthecharacteristicsandidiosyncrasiesofthematerial under analysis, thus allowing for adequate technical solutions, and assess thegamutofinformationofinteresttobetargetedforextraction;ii)tomakeavailabledatafortraining,whenandasrelevanttotheinvestigatedcontentanalysistechniques,aswellasforevaluationandbenchmarking.Toensureacomprehensiveandadequately insightfulapproach,avarietyofdatasetshavebeencollectedforconsideration,includinghistoricaldatamadeavailablebytheconsortium,datasets compiled within beAWARE, as well as publicly available benchmark datasets.Naturally,thecollectionofdatasetsisacontinuousprocess,drivenbythespecificationsandrequirements of the use case scenarios, as further worked out during the course of theproject, as well as of the investigations into corresponding analysis tools. Thus, thisdeliverable only reflects the current state of affairs; further updates, as needed, will bereportedinD3.3(M17)andD3.4(M34)thatwillpresentthebasicandadvancedtechniquesforcontentdistillationfrommultimediamaterial.Likewise,theobservationsoftheempiricalstudy serve as preliminary guidelines for the development of the T3.2 and T3.3 contentanalysis techniques,alsoexpected tobe revisedand further refined,as thecorrespondingresearchanddevelopmentactivitiesprogress.The remaining of the document is structured as follows. Section 2 outlines the use ofmultimedia data within the targeted use case scenarios, as described in D2.1. Section 3describesthecurrentlycollecteddatasets,whichencompasshistoricaldatamadeavailable

D3.2–V0.4 D8.1–V1.0

Page8

by consortium partners, datasets that have been compiled within, and relevant publiclyavailablebenchmarkdatasets.Followingthedatasetsdescriptions,wereporttheoutcomesof the carried out empirical study are in Section 4. Last, Section 5 summarizes the datacurrently drawn observations with respect to the collection of relevant data and thepreliminaryobservationsresultingfromtheiranalysis.

D3.2–V0.4 D8.1–V1.0

Page9

2 MultimediadatainputsinbeAWAREbeAWAREaimstoenhancesituationalawarenessandsupportdecisionmakingduringcrisismanagementby capitalizingon the integrationandaggregatedanalysisofmeteorological,forecastingandsensordata,aswellasofamultitudeofpertinentmultimediadatainputs.Inthe context set by the targeted pilots and respective scenario descriptions, as laid out inD2.1, these multimedia inputs pertain to the following information sources: i) incidencereportsbyciviliansviacalls toemergencynumbersandauthorities’callcenters,aswellasviathebeAWAREmobileapplication,whichwillenablethesharingoftextualmessagesandalsoofcapturedimages/videos;ii)communicationsbetweenfirstrespondersandauthoritiesviainternalcommunicationchannels(e.g.radiocommunication)aswellasviathebeAWAREmobileapplication;iii)informationpostedonsocialmedia,namelytextualmessagesaswellasaccompanying imagesandvideos; iv) imagesandvideoscapturedvia staticandmobilecameras. These act complementary allowing not only for amore complete apprehensionand assessment of the reference situational context, but also for cross-checking andvalidating of the accuracy and trustworthiness of the extracted information, crucialprerequisitesincriticalapplicationssuchasemergencymanagement.Inthefollowing,webrieflyreviewtheoverallcontextofusagepertinenttothethreetypesof consideredmultimedia data, namely audio, text and visual, before continuingwith thedescriptionofthecurrentlycompileddatasetsandtheoutcomesoftheempiricalanalysis,inSections3and4respectively.

2.1 Audiodatainputs

Asnotedabove, theaudio inputsconsideredwithinbeAWAREconsist in incidencereportsby civilians to emergency numbers and call centers, which may involve human or non-human receivers, as well as, voice-based communications between authorities and firstresponders through currently deployed internal communication channels (e.g. radio). Inaccordancewiththeplannedpilots, theconsidered languagesareGreek(heatwavepilot),Italian(floodpilot),andSpanish(fireforestpilot),aswellasEnglish,whichwillbeusedforcontrolanddemonstrationpurposes.Todistilltheconveyedinformation,AutomaticSpeechRecognition (ASR) techniques will be first deployed in order to denoise, enhance andeventuallytranscribethecapturedaudiostreams intotext; thesemanticsof the latterwillsubsequentlybeextractedbymeansoftextanalysisandfedtothesemanticintegrationandinterpretationtoolsofWP4forsupportingdecisionmakingandwarninggeneration.Astheconditions pertaining to emergency communications aggravate considerably acoustic andenvironmental robustness challenges associated with the recognition of spontaneousspeech,includinghighnoiselevels,fragmented,emotionallychargedutterances,hesitations,and so forth, ASR investigations will focus, in addition to ensuring adequate vocabularycoverage,oneffectivecompensationstrategies.

2.2 Textualdatainputs

Theconsideredtextualinputsinclude:i)postsonsocialmedia,inparticularTwitter,crawledviathebeAWAREsocialmediamonitoringservice(T4.1), ii)messagesfedtothesystemby

D3.2–V0.4 D8.1–V1.0

Page10

civilians and first responders via the beAWARE mobile application, as well as iii)transcriptions of the audio inputs described above. Leaving aside processing variationspertinent to the textual inputs provenance (e.g. twitter messages need to undergonormalizationtoaccountfortheuseofabbreviationsand idiosyncraticexpressions,emoji,etc.),thedistillationoftheconveyedinformationconsistsintheextractionofthementionedentities and the relations between so as to capture the communicated events/situations(e.g. a car being carried away in the flooded river, traffic interrupted, etc.). Thesewill berepresentedinastructured,formalmanner,namelyRDF/OWL,thatabstractingawayfromlanguage variations facilitates their semantic integration and aggregated interpretation,tasks addressed in WP4. The key challenge in the analysis and understanding of theconsideredtextualinputsliesintheidiosyncraticnatureofallthreetypesoftextualinputs,compared to proper written language. Naturally, as for audio inputs, the consideredlanguagesareGreek,Italian,SpanishandEnglish.

2.3 Visualdatainputs

Thevisual inputsutilizedwithinbeAWARE include imagesandvideosobtained fromstaticand aerial (e.g. UAV, satellite) cameras deployed on the pilot sites, crawled Twittermessages, as well as reports sent to the PSAP by first responders and civilians using thebeAWAREmobileapplication.Thecapturedvisualmaterialwillcontributetothedetectionof emergencies and the enhancement of the real-time contextual understanding, throughthe recognition of emergency indicative situations, such as smoke, traffic bottlenecks,floodedareas,elementsatpossiblerisk(e.g.people,cars,andbuildings),etc.Keychallengesinvolve theneed for real-timeprocessingofhigh-discriminatingquality inorder toextractinformationthateffectivelycomplementsandenrichesthatoftheaudioandtextmodalities.Thischallengewillbetackledbydeployingapowerfulserveronthecloud,whichwillbeabletorunonlinestate-of-the-artdeep learningalgorithmsandextractfromthevisualcontenthighlyaccurateconclusionsinareasonablenearrealtimecomputationalcost.Additionally,embeddedsystems,suchastheJetsonTX2moduleandRaspberryPi3modelB,couldalsobeusedinsomespecialcases,wherethecomputervisionalgorithmsdonotrequirealargeamount of memory and energy resources, deductions could bemade on-site, where thecrisiseventoccurs,andsendtheresultstotheserverinsteadofthemedia.

D3.2–V0.4 D8.1–V1.0

Page11

3 MultimediadatacollectionInthefollowingwedescribethecurrentdatasetsandtheunderpinningacquisitionprocess,which, as aforementioned followed a three-pronged approach, involving: i) historical dataalready in thedisposalof thebeAWAREuserpartners, ii)datacompiledwithinbeAWARE,andiii)publiclyavailabledatasetsforbenchmarkanddevelopmentpurposes.Thecompileddatasets form the basis for the empirical study towards the elicitation of preliminaryrequirements and specifications for the targeted multimedia content analysis techniques(T3.2,T3.3),andalsoprovidereference,trainandevaluation,material forsupportingtheirdevelopment.Althoughthedatasetsthatwerecompiled,andwillcontinuebeingcompiled,withinbeAWAREareclearlytheonesofhigherinterest,thehistoricalandpubliclyavailableonesareofequally important,especiallyforkickstarting insightsandinvestigations,since,as described in the following, the opportunity to collect actual relevant data variesconsiderablydependingonthemodalityconsidered.

3.1 Historicaldatasetsprovidedbyuserpartners

Theconsortium’suserpartnerspossessandstartedmakingavailable,aconsiderablevolumeofhistoricaltext,imageandvideodata,fromtheearlyonsetoftheproject,namelystartinginM2inresponsetothefirstdiscussionsduringthekick-offmeeting,inordertoassistthetechnicalpartnerswith theunderstandingandelicitationof specificationspertinent to theanalysis of such data. Table 1 summarizes the datasets, their media types and the userpartnerthathasprovidedthem.

Table1.Audio,textandvisualhistoricaldataprovidedbybeAWAREuserpartners

Partner DataDescription MediaType

AAWA ~1Gofimages(UAVandsatelliteones)andvideos,collectedfromYouTubeandotherpublicwebsources,thatdepictthefloodthatoccurredintheregionofVicenzainNovemberof2010

Image,Video

AAWA Reportssentthroughthemobileapp,collectedduringtrialsorganizedwithintheFP7WeSenseItproject(2012-2016)

Text,Image

FBBR ~3MBoffireimagesfromtheirprivaterecordingsPicturesoffireevents Image

HRT ~1.67GofimageandvideosthatwererecordedbyfirstrespondersinrealfireincidentsinGreeceduringthelastdecade

Image,Video

PLV ~25MBofflood,fireandheatwaverelatedimagesandvideos,recordedbyfirstrespondersandcitizensintheValenciaregion(2014-2016)

Image,Video

PLV Tweetsrelatedtoflood,fireandheatwaverelatedeventsintheValenciaregion,postedbyValenciapolice(2014-2016)

Text

D3.2–V0.4 D8.1–V1.0

Page12

Ascanbeseen,thehistoricaldataincludeintheirmajoritytextualandvisualmaterial,andthoughveryuseful forpreliminary investigations,additionalmaterial is required toensuretheadequatecoverageofthescopeofthepilotsandspecificusecasesscenariostargetedbybeAWARE (e.g.audio inputs,descriptive reportsenabled through thebeAWAREenvisagedmobile application, etc.). Examples of the providedmaterial are showing in Figure 1 andFigure2.

Figure1-ExampleoffloodandfireeventsimagesfromUAV,satelliteandmobilecameras.

Figure2-ScreenshotfromexampleWeSenseItreportinformingabouthigh-waterlevel.

D3.2–V0.4 D8.1–V1.0

Page13

3.2 DatasetscompiledwithinbeAWARE

Thedatasetsthathavebeencompiled,andthataretobecompiled,withinbeAWAREduringitslifetime,fallintotwocategories.ThefirstreferstodatasetsthatwillbecollectedduringthefieldtrialsplannedforM18-M21andM30-33forthethreepilots,namely:

• PUC1–flood:tobedeployedandimplementedbyAAWAinVicenza,Italy;• PUC2–fire:tobeimplementedanddeployedbyPLVinValencia,Spain;• PUC3-heatwave:tobeimplementedanddeployedbyHRTinThessaloniki,Greece.

Collectedwithinfieldtrialsthatsimulatetheunfoldingofemergencyevents,thesedatasetsnotonlywill providemodality-correlated information,butwill alsoafford temporally- andspatially-associated inputs, thus allowing the validationof the analysis and interpretationoutcomesforboththedistinctmodalitiesandtheiraggregation.

Thesecondcategoryreferstodatasetsthathavebeenandwillbecompiledbyconsortiummembers, technical and user partner ones, to support the analysis of distinctmultimediacontent types and/or specific analysis objectives, e.g. smoke detection in aerial images,smokedetectionfromstaticcameravideos,extractionofinformationaboutelementatriskfrom twitter messages, and so forth. The respective, currently underway and planned,activitiesofuserpartnersarelistedinthefollowing.

• AAWAisresponsibleforsupportingthecollectionofaudio,textandvisualdatarelatedtofloodemergenciesintheareaofVicenza.Tothisend,theycontributetothedefinitionof criteria for the crawling of flood-related tweets in Italian and their annotation fordevelopmentandevaluationpurposes.Furthermore,theyhavestartedthecompilationofsimulatedemergencycallsrecordings,currentlyamountingto20and includingboth“clean” and noisy data, based on the simulation trials carried out within the FP7WeSenseIt project. AAWAwill also provide surveillance videos from already availablestatic cameras that monitor regions of interest in the city of Vicenza(http://www.bacchiglione.it/); these will be used to build reference patterns for therecognitionofnon-floodscenesandforfloodprediction.

• FBBRisresponsibleforsupportingthecollectionofvisualdatarelatedtotheforestfireusecasescenarios, servingcomplementary to thedatacollectionbyPLV.This involvesimagesandvideosfrommobiledevicesandstaticcamerasintheregionofDenmarkthatmonitor regions of danger. As Danish is not among the languages targeted for theplanned pilots, no efforts for the collection of audio and textual material has beencurrentlyscheduled.

• HRTisresponsibleforsupportingthecollectionofaudio,textandvisualdatarelatedtotheheatwaveusecasescenarios.Tothisend,theyhavealreadyprovidedalistofsocialmedia account references from involved authorities and contribute actively to thedefinitionofcriteria forcrawlingheatwaverelated tweets inGreekaswellas to theirannotation; moreover, within D2.1’s scenarios definition, they have provided apreliminarylistofexemplartextualmessagesthatcouldbesentbyfirstrespondersandcivilians via themobile application. Furthermore, likewise to theAAWA initiative, theyhavepreparedaninitialsetofemergencycallrecordingsinGreek,affordingboth“clean”andnoisydata.HRTwillalsoprovideaccesstoclosecircuitsurveillancecameras(CCTV)

D3.2–V0.4 D8.1–V1.0

Page14

thatmonitor highway roads in Greece in order to support the implementation of thetraffic congestion related use case scenarios aspects. Last, being operational andinvolved in fire emergenciesmanagement,HRTwill also provide videos related to fireeventsfrommobileandstaticcameras.

• PLV isresponsibleforsupportingthecollectionofaudio,textandvisualdatarelatedtotheforestfireusecasescenarios.Tothisend,theycontributetothedefinitionofcriteriaforthecrawlingoffire-relatedtweetsinSpanishandtheirannotationandhavestartedthecompilationofsimulatedemergencycallsrecordings,alsoincludingboth“clean”andnoisy data. Furthermore, PLV will provide videos from static cameras monitoring theforestoftheDevesaregion,aswellasflood-relatedvideostowhichtheyhaveaccess.

Parallel to the aforementioned user partners-centered data collection activities, furtherdatasetsarebeinggatheredandcompiledby technicalpartners,within thecorrespondingcontentanalysistasks.

Figure3-ScreenshotofthecrawlingandannotationtooldevelopedforTwittermessages.

More specifically, and as part of thework on socialmediamonitoring carried out withinT4.1,CERTHhasdevelopedatweetscrawlerandannotationtoolthatcontinuouslygatherstweets of possible relevance to the domains of the three pilots, while enabling manualrelevanceannotations;ascreenshotfortheflooddomainisshowninFigure3.Furthermore,CERTH has been collecting relevant visualmaterial from video-sharingwebsites, includingDailymotion1andYouTube2.

1 http://www.dailymotion.com/

D3.2–V0.4 D8.1–V1.0

Page15

3.3 Benchmarkdatasets

ComplementingthehistoricaldatacollectionsandthedatasetscompiledwithinbeAWARE,several public available datasets have been identified. In addition to contributing to thedevelopmentof respectivemultimedia analysis solutions by providing trainingdata, thesewillenabletoevaluateandcomparethebeAWAREapproachesnotonlywithrespecttotheprojectgoals,butalsowithrespecttothecurrentstateoftheart.Thecompletelistfollows.

3.3.1 Audiodatasets

Generic audio datasets, containing annotated speech recordings, are being collected forthreepurposes:i)thedevelopmentoftheASRalgorithmsandtheinitialtrainingoftheASRtool, ii) the realization of the empirical study regarding the most important aspects andinformationforthemanagementofacrisiseventandiii)thefinaladaptationofthelanguagemodel and the refinement of the ASR dictionary. The collected datasets include generalannotated speech recordings as well as general emergency data, as described in thefollowing.

As far as general annotated speech recordings are concerned, the following datasets ofconversationsandphonecalls,inboth“clean”andnoisyenvironmentshavebeenidentified:

§ TalkBank(English,Greek,Spanish)http://talkbank.org/browser/index.php§ VoxForge(English,Greek,Italian,Spanish)

http://www.voxforge.org/el/Downloads§ ELRA(English,Italian,Spanish)http://www.elra.info/en/catalogues/free-

resources/free-lrs-set-1/http://www.elra.info/en/catalogues/free-resources/free-lrs-set-2/

§ CMUCensusDatabase(English)http://www.speech.cs.cmu.edu/databases/an4/

§ CMU_SINdatabase(English)http://www.festvox.org/cmu_sin/§ SantaBarbaraCorpusofSpokenAmericanEnglish(English)

http://www.linguistics.ucsb.edu/research/santa-barbara-corpus§ EMOVOCorpus(Italian)

http://voice.fub.it/activities/corpora/emovo/index.html§ LibriSpeechASRcorpus(English)http://www.openslr.org/12/§ Speech/SongDatabase–RAVDESS(English)

http://smartlaboratory.org/ravdess/

With respect to general emergency event data, which form the basis for the conductedempiricalstudyandwillserveasreferencefortheinitialadaptationoftheASRmodule,thelist of relevant datasets includes speech recordings of real life emergency calls to 911reporting general incidents, online inspected on YouTube. Examples, representativeexamplesinclude:

• https://www.youtube.com/watch?v=q-Tr0u35Tek• https://www.youtube.com/watch?v=qSywoZAO2SE

2 https://www.youtube.com/

D3.2–V0.4 D8.1–V1.0

Page16

• https://www.youtube.com/watch?v=P8t_weHgrJwIn addition, we have considered emergency call speech corpora from published studies,including the NineOneOne project’s Spanish to English code-switching corpora. A licenseagreementdocument isbeingprepared inorder toenable theacquisitionanduseof realrecordingsfromactualemergencycallcenters,aswellasfrompublishedstudiesofresearchgroupsandinstitutes;examplesofthelatterincludethedatabaseofemergencycallsusedinBaicerek’s3 study aswell as Lefter’s4 database of spontaneous emotional speech from anemergencycall-centre.

3.3.2 Textdatasets

Thegrowingadoptionof informationpostedonsocialmediaduringdisastersasmeansforassistingincrisismanagementandrecoveryhasgivenrisetoaplethoraofresearchstudieson toolsandmethodologies,underpinnedby respectivedatasets.Thediversity throughofthe pertinent objectives, scope and domains renders many of no direct interest for thetextualanalysispursuitswithinbeAWARE,themainissuebeingthatdatasetsforlanguagesand/ordomains thatdonotmatch thoseof beAWARE cannotbeused for addressing thelinguistic phenomena (synctactic, idiomatic, etc. constructions) and vocabulary coveragepertinent to thebeAWARE languages andpilot domains. This is not the casehowever forCrisisLex5, a large-scale repository of crisis-related social media data media that includescollections of crisis data aswell as lexicons of crisis terms. The compiled collections spanvarious languages,albeitnotofequalbreadth,andlikewisevariousdomains,rangingfromhuman-inducedcrisisevents(e.g.protests,accidents,attacks)toclimate-relatedandnaturaldisasters(e.g.hurricanes,floods,earthquakes),andareorganizedinsixsub-collections6withvarying levels of annotation. Of interest to beAWARE are the relevant climate-relatedcollectionsaffordedinEnglishandItalian.

3.3.3 Visualdatasets

BasedontheinitialusecaserequirementsandscenariodescriptionsexplicatedinD2.1,fourkey visual contexts, namely fire, smoke, flood and traffic scenes, havebeen identified. Assuch,thesearchandselectionofpubliclyavailabledatasetshavebeendrivenbytheneedtoeffectively detect as well as distinguish such contexts from others with visually similarcharacteristics(e.g.firefromwaterreflections,smokefromcloudformations,etc.). Inthefollowing,welisttheidentifiedrelevantdatasetsforeachcategory.

Firedatasets§ TheMivia fire dataset (http://mivia.unisa.it/datasets/video-analysis-datasets/

fire-detection-dataset/)containsseveralvideosamples(31)thatdepictfirecasesthat occur in both indoor (i.e. office scenario) and outdoor (i.e. forest fire

3 http://ieeexplore.ieee.org/document/5943720/citations 4 http://ii.tudelft.nl/~iulia/papers/ijidss_il.pdf 5 http://crisislex.org/ 6 http://crisislex.org/data-collections.html

D3.2–V0.4 D8.1–V1.0

Page17

scenario)environments.Thedatasetalsocontainsnon-firesituationsinordertoevaluatethediscriminationpowerofthefiredetector.

§ The FIRESENSE fire detection dataset (http://signal.ee.bilkent.edu.tr/VisiFire/),whichwas compiledwithin the FIRESENSEproject, includesboth forest fires aswell as human-induced ones, recorded by static cameras. The dataset wasbroadlyusedthelastdecadetoevaluatefiredetectionalgorithms.Thedemofireclips (http://signal.ee.bilkent.edu.tr/ VisiFire/Demo /FireClips/) contain somefurthervideosamples,depictingfiresituationsrecordingwithastaticcamera.

§ The Rabot 2012 dataset (http://multimedialab.elis.ugent.be/rabot2012/) is arather simple dataset that contains, among others, video samples of firescenariosinindoorenvironments.

Smokedatasets

§ The Mivia smoke dataset (http://mivia.unisa.it/datasets/video-analysis-datasets/smoke-detection-dataset/) is composed by 149 videos, each lastingapproximately15minutesandiswidelyusedforsmokedetectionevaluation.Itisaverychallengingdataset,sinceitcontainsscenesandelementsredhousesinawidevalley,mountainsatsunset,sunreflectionsinthecamera,andclouds.

§ The FIRESENSE smoke detection dataset, also compiled within the FIRESENSEproject(http://signal.ee.bilkent.edu.tr/VisiFire/).

§ The Visor smoke detection dataset (http://www.openvisor.org/video_videosInCategory.asp?idcategory=8) contains a wide number of smoke video samplesand is used to evaluate the discrimination power of smoke detectors. Severalmotions also exist in the videos, so the separation between smoke and non-smoke regions is determined as a quite challenging task for state-of-the-artsmokedetectors.

§ TheVisiFiresmoke(http://signal.ee.bilkent.edu.tr/VisiFire/Demo/SmokeClips/)andforestsmoke(http://signal.ee.bilkent.edu.tr/VisiFire/Demo/ForestSmoke/)datasets were recorded by the Bilkent University and are broadly used toevaluate smoke detection algorithms. The video samples are recorder by usingstaticcamerasinbothforestandofficeenvironments.

Flooddatasets

§ TheVideoWaterdatabase (https://staff.fnwi.uva.nl/p.s.m.mettes/) consistsof260 high-quality videos that contain water depictions of predominantly 7subcategories, namely canals, fountains, lakes, oceans, ponds, rivers, andstreams,aswellasnon-waterdepictingsamplesthatcontainobjectswithsimilarspatialandtemporalcharacteristics,suchasclouds/steam,fire,flags,trees,andvegetation.Thedatasetisusedformodelingwaterdynamicsandcanbeusedtoevaluateflooddetectionalgorithms.

§ The Dyntex database (http://dyntex.univ-lr.fr/) is a diverse collection of high-qualitydynamictexturevideosthatconsistsofmorethan650sequences,whichcanbeusedinordertomodelwater,smoke,fireandothertexturedynamics.Itisbroadlyusedforevaluatingwater,fireandsmokedetectionalgorithms.

§ The Moving Vistas dataset (http://www.umiacs.umd.edu/users/nshroff/DynamicScene.html),broadlyusedfordynamicsceneunderstanding,comprises

D3.2–V0.4 D8.1–V1.0

Page18

10videos foreach of the following13categories:Avalanche, IcebergCollapse,Landslide,Volcanoeruption,Chaotictraffic,Smoothtraffic,Tornado,Forestfire,Waterfall,Boilingwater,Fountain,WavesandWhirlpool,thusmakingitrelevantfor several beAWARE pertinent detection purposes, such as traffic, water, andfirerecognition.Largevariationsinthebackground,illumination,scaleandviewrenderitaverychallengingdatasetthatcanbeusedformodelingandevaluatingdynamictexturealgorithms.

Trafficdatasets§ TheUA-Detrac dataset (http://detrac-db.rit.albany.edu/) is a challenging real-

worldmulti-object detection andmulti-object tracking benchmark. The datasetconsistsof10hoursofhigh-quality videos captured froma static cameraat24differentlocationsleadingtomorethan140thousandvideoframesconsistingof8250vehicles.Thedatasetisusedbroadlytoevaluateobjectdetectionandmulti-objecttrackingmethods.

D3.2–V0.4 D8.1–V1.0

Page19

4 EmpiricalStudyInthissection,wereportthefindingsoftheempiricalstudycarriedoutwithrespecttothethree types of multimedia data inputs considered in the project, using as reference thedatasetsdescribedinthepreviousSection.Asnotedinthebeginningofthedocument,theobservationsdrawnarepreliminaryandareexpectedtobefurtherrefinedandelaborated;this isnotonlyattributed to the fact that themajorityof currentdatasetsareexternal tobeAWARE,butalsototheearlyphaseofthetaskspertinenttothedevelopmentoftoolsfortheharvestingandanalysisofsuchdata.Inthefollowing,welistthekeyobservations.

4.1 Audiodatastudy

As aforementioned, audio inputs in beAWARE include emergency calls by civilians toauthorities, and voice-based communications between first responders and authorities.Given the currently available audio datasets, the empirical study drew mainly upon thecollected speech corpora of cross-domain emergency calls recordings so as to afford acomprehensive understanding of the types of information that authorities and firstresponders seek for decisionmaking during emergencies. In addition, the empirical studytook into account the simulated calls compiled by user partners within beAWARE; beingcurrently limited though in extent, as aforementioned they amount to 20 recordings foreach of the three emergency domains, they need to be considerably extended beforeallowingforcomprehensive insights intotheactualcoveragerequiredfortheASR lexiconsandthelanguagemodelsexpressivityforthepurposesofbeAWARE.Yet,wecouldalreadydeterminethekeyinformationsoughtafterduringsuchcalls:

• Typeofevent,soastomobilizetheappropriatetypeofpersonnel.• Severityandextent,inordertoclassifytheevent,theurgeforhelp,aswellasthe

requiredpersonnelandequipment.• Addressof the reported incidence; if thepersoncalling isnotable toprovide the

address,ageneraldescriptionofthesurroundingareaisrequested.• Number of people in danger and their condition, as well as overall number of

peopleinvolved.• Identificationdetailsandcharacteristicsofthepeopleinvolvedsuchasname,age,

sex,phonenumbers• Informationaboutbystandersthatcouldbeofassistance;e.g.ifthereisanydoctor

orspecializedpersonnelaround.• Incaseofanemergencytakingplacewithinabuilding,keyinformationincludesthe

type of the building, the floor where the incident takes place, its spatialconfiguration,theexistenceoftoxicorinflammablematerialsetc.

Similarinformationissoughtafterduringcommunicationsamongfirstrespondersandfirstresponders and authorities, alongwithmore specialized updates, such as the progress ofassignedtasksandthecurrentseverityoftheincident.

D3.2–V0.4 D8.1–V1.0

Page20

4.2 Textdatastudy

The inputs in beAWARE include Twitter posts collected in real-time through the TwitterStreamingAPI, reportssentbyciviliansandfirst responderstothePSAPviathebeAWAREmobile applications, as well we transcriptions of the captured audio communications. AsTwitter posts comprise the most populous of the three and given the limited historicalreport-relevantmaterialmadeavailable,comparedtothealreadyconsiderablevolumesoftweetsthathavebeencrawledusingbeAWAREdevelopedtechnologies,theempiricalstudyhasfocusedontheexaminationofthecollectedTwitterdataandtheirparticularities.

At the timeofwriting, thebeAWAREtweetcollectionamountsa total to5,797,042posts.Table 2 shows the distribution across the languages and emergency domains (fire, flood,heatwave)addressedbythepilots.

Table2–beAWAREtweetcollectiondistributionLanguage/Domain Fire Flood Heatwave

English 81,693 1,022,967 367,839

Greek 27,526 2,988 50,540

Italian 141,745 12,003 685

Spanish 3,158,257 712,556 217,442

As the collected tweets have been crawled based on predefined lists of domain-specifickeywords,asopposedtomoresemanticcriteria,theyaresubjectto lexicalambiguity(e.g."floodedwithmessages","he'sonfire",etc.),andhencenotallofthemrefernecessarilytoactual fire, flood and heat wave. Since within the overall beAWARE pipeline, the Twittermessagesthatarefedtotextualanalysishavebeenfirstfilteredsothatmostlyrelevantonesare kept, the empirical study was based on a subset of tweets that had been manuallyannotated with relevant judgments, and which amounted, at the time, to 7,914 tweets,most of which in Italian, Spanish and English. Interestingly enough, slightly over half themanuallyannotatedtweetswerejudgedtoberelevant,i.e.refertoactualfire,floodorheatwaveevents.

4.2.1 Tweetlocationmetadata

Akeyaspectof interest for theanalysisof crisis situations is the locationof themessage,whetheritistheoriginofthemessage,oralocationthatthemessageisabout.

Twitter allows users to share their location asmetadatawith their tweets; however, thisoption is notwidely used. In our sample, only 88,287 out of 5,797,042 have any locationinformationassociatedwith them(oraround1.5%),and inmanycases the location is toocoarse, indicatingonly thecountryor cityoforigin.Only4,244 tweets (or0.07%of them)haveexactgeo-coordinateswhichwouldallowthesystemtoknowtheexactlocation,wherethetweetoriginatedfrom.

D3.2–V0.4 D8.1–V1.0

Page21

Thismakesitveryimportanttobeabletoextractthelocationfromalternativesources,andinparticular,fromthetextitself,assumingthatrelevantinformationisprovided(e.g.nameofastreet,nameofbridgeormonumentwhichcanbesubsequentlygeo-localized,etc.),orthrough more integrated approaches, as needed further down the overall processingpipeline, e.g. making use of preceding and following tweets by the same user and thepertinent spatiotemporal correlations or through the aggregated interpretation ofmultimodalinformation.

4.2.2 Tweetsprovenanceandinformationcontent

Inordertodeterminewhatinformationcan(potentially)beextractedfromthetweets,weproceeded to manually annotate a sample of messages, starting from the subset of therelevantmarkedonesforthefollowinglanguage-domaincombinations:

• English-floods(536of1,015weremarkedasrelevant)• Spanish-fires(615/973)• Greek-heatwave(755/897)

For each of these collections, 200 tweets were manually annotated in more detail asdescribed in the following.The reason fornot includingan Italiandatasetat themoment,was that the English dataset, chosen for demonstration purposes, already covered thedomain associated with the Italian pilot, and as in the current stage of analysis intra-language differences within the same domain are out of scope (being pertinent todevelopmentchoicesduringtheimplementationoftherespectivemultilingualtextanalysismodule,theyareplannedforonceafirstbaselineversionisfirstmadeavailable).

4.2.2.1 AuthorshipTweetscanoriginatefromavarietyofsources.Sinceprovenanceisanimportantfactorforassessingthecredibilityandtrustworthinessoftheconsideredinformation,weselectedtodifferentiatebetweenmessagesfromauthorities,suchasthepoliceorthefiredepartment("official"), messages from news outlets ("media") and others ("personal/unknown"). Thedistributions of the author types for the three considered collections are shown inError!Referencesourcenotfound..

Asobserved,messagesfrommediaoutletsmakeupasignificantproportionofthecollectedtweets,whereas official accounts of authorities aremuch less prominent. The second biggroupconsistsofpersonalaccounts(oraccountsforwhichnoprofessionalaffiliationcouldbe established). However, going through the messages, it becomes clear that, in theirmajority,the"personal"messagesoriginatefromrespectivenewsmediaandarefrequentlygenerated by readers clicking on the "share on Twitter" buttons on the respective newssites.Whilewedidnotmanuallyannotatethisaspect,itcanbesomewhatbeapproximatedbylookinghowmanyofthetweetscontainaURL(whichmostcommonlypointstoanewsarticle). For the "English flood" collection, 80%of tweets containaURL, 84% for "Spanishfire",and95%forthe"Greekheatwave".

D3.2–V0.4 D8.1–V1.0

Page22

Figure4-Distributionofauthortypesfortheconsideredcollections

A second interesting observation is that here are important differences between thecollections,withmediaoutletsaccountingfor57%oftheSpanishTweets,comparedto35%oftheEnglishones.Thisisalsoreflectedintheotherdimensionsanalyzedinthefollowingsections.Whetherthereareunderlyingculturaland/oremergencytyperelatedcorrelations,remains tobe foundonce thebasic versionof themultilingual text analysis is set up andsuchanalysiswillbefullyautomated.

D3.2–V0.4 D8.1–V1.0

Page23

4.2.2.1 CategoriesTable3 lists the categories that emergedduring themanual inspectionandannotationofthecollectedtweetsandthathavebeenusedinordertocreateapreliminarycategorizationofthetypesofconveyedinformation.

Table3–ListoftweetscategoriesCategory Description

Advice adviceonwhattodo,inaspecificcrisissituationoringeneral

Comment generalcommentsaboutasituation

Enquiry questionsaboutasituation

Warning awarningaboutapotentialcrisissituation

Forecast predictionsaboutthedevelopmentoftheforecastedemergency

Miscellaneous commentarythatmayormaynotberelatedtoacrisissituation7

News newsaboutanongoingorpastcrisissituation(usuallynotactionable/notaimedatpersonsaffectedbythesituation)

Places/routes adviceonsafe/reliefplacesorescape/blockedroutesforaffectedpersons

Status_update an update on an ongoing situation in a way that is potentially relevant /actionableforaffectedpersons

Thedifferencesbetweenthesecategoriesarenotalwaysclear-cut,as itoftendependsonthe (implicit) intentionof the senderaswell as the receiver.As such,amessagecanbeawarningtopeopleunawareofasituationthatmayaffectthem,astatusupdatetoothers,andmerely"news"topeopleunaffectedbytheevent,andourcategorizationissomewhatsubjectiveastothemostlikelyreadingofanygivenmessage.Someexamples:

• News:"MillionsunderfloodthreatinCarolinashttps://t.co/halp4PiAPw".(Themajorityoftherecipientsofthismessageareunlikelytobeaffectedbythesituation,anditistoovaguetobeactionable.)

• Statusupdate:"PocketsofheavyraincontinuetofallbetweentheRoanokeareaandtheSouthside.Floodingissuespersist.https://t.co/0L6xABKZvz"(Appearstobeaimedmainlyatpeopleaffectedbythesituation.)

• Forecast:"Bynoonweshouldhaveoveraninchofrainaddedtothealreadysaturatedsoils.FloodingstillanissueTuesday.https://t.co/gTQEDr4BpA"

• Warning:"#PrepperNewshttps://t.co/ZdeEkHcqQVWarning:Floodingpossiblethisweek-TimesDailyhttps://t.co/Hv2y69Zaz9"

7 Note that while we only examine tweets previously marked as "relevant", this does not mean that they all refer to the same emergency event, only that the use of crisis-related terms is non-metaphorical.

D3.2–V0.4 D8.1–V1.0

Page24

• Advice:"#Floodtip:Avoiddrivingthroughpoolsofstandingwater.Watercouldbecoveringfallenpowerlines&otherdebri…https://t.co/7LnBySuBPU"

• Comment:"Crap!Ihopetherainstopsbeforeanymoreflooding!Whatadayalready!Ihopeitgetslotsbetter!https://t.co/4vIm58i7O1"

Thedistributionsofthetweets’categoriesfortheconsideredcollectionsareshowninFigure5.

D3.2–V0.4 D8.1–V1.0

Page25

Figure5–Distributionofcategories,Greek,heatwavecollection

Overall, we see that many messages are general news, with little practical impact oractionable information.This isespecially true for the"Spanish fire"collectionwherenewsrepresent80%ofallmessages.The"Englishflood"collectionissomewhatmorevaried,withmuch more practically relevant information such as warnings, forecasts, status updates,information on places and routes to use or avoid, etc. It also contains more personalcomments,includingmessagesofempathy.The“Greekheatwave”collection,ontheotherhand,containsaconsiderablenumberofwarningsandforecastmessages,aswellasadvicesand information on places of relief, as during the reference period, Greece experiencescontinuousheatwaves.

The differences between the collections are largely due to the way the data is collected(selectionofkeywords,etc.),butalsothetypesofemergencyeventstakingplaceduringthetime frame of collection. As such, the Spanish language tweets are dominated by newsabout the London Grenfell Tower fire, with little practical impact to people getting theirinformation from Spanish language sources, which may have significantly biased thecollection. The flood-related English collection, on the other hand, contains much moreinformationaimedatlocalresidentsworldwide(intheUS,UK,Australia,SriLanka,etc.),beitfromlocalnewsmediaorfrompersonalaccounts.Table4,Table5and

Table 6 show a further breakdown of the respective tweet categories distribution, byindicatingtherespectiveauthortype.

Table4-Categoriesandauthors,English,floodcollection Institution/

companyMedia Official Personal/

unknownTotal

advice 3 2 1 3 9

comment 1 0 0 20 21

enquiry 0 0 0 1 1

D3.2–V0.4 D8.1–V1.0

Page26

forecast 3 6 0 5 14

miscellaneous 1 7 0 15 23

news 0 39 0 62 101

places/routes 0 1 0 5 6

status_update 1 10 0 6 17

warning 2 5 0 1 8

Total 11 70 1 118 200

Table5-Categoriesandauthors,Spanish,firecollection Media Official Personal/

unknownTotal

advice 2 0 1 3

comment 1 0 3 4

miscellaneous 4 1 3 8

news 96 0 64 160

status_update 7 7 4 18

warning 4 1 2 7

Total 114 9 77 200

Table6-Categoriesandauthors,Greek,heatwavecollection

Media Official

Personal /unknown Total

advice 3 1 0 4

comment 17 0 0 17

forecast 41 1 1 43

miscellaneous 0 5 5

news 8 0 0 8

places/routes 5 1 0 6

status_update 9 0 0 9

warning 104 1 3 107

D3.2–V0.4 D8.1–V1.0

Page27

Total 187 4 9 200

4.2.2.2 Location

Asalreadymentioned,thoughanoptionofferedbyTwitter, locationmetadatatendtonotbe commonly used. In the "English flood" collection, only 15 out of 537 tweets (2.8%)contained location information; the “Spanish fire” collection follows a similar distribution,with 15 out of 600 (2.5%),while for the "Greek heatwave" collection, only 4 out of 756tweets(0.005%)containedlocationmetadata.

Accordingtothemanualannotation,thenumberoftweet,whosetextcontainedsomekindoflocationmentionwasasfollows:

• English,flood:150/200(75%)• Spanish,fire:178/200(89%)• Greek,heatwave:29/200(14.5%)

Despite thehighpercentage, in themajorityof cases the location informationprovided isverycoarse(atthecity,state,orevencountrylevel)andthusoflimiteduse,whenitcomesto extracting actionable information. Interestingly, the same can applywhen very specificlocationinformationisprovided,whenmoregeneralinformationthatwouldallowtomakesenseofitislacking,e.g.providingastreetnamewithoutmentioningthecitythestreetisin,"Chester Bridge reopens after being closed due to flooding". In those cases, additionalcontext isneeded,e.g. fromsurrounding tweetsby the sameuser,orother tweets in thesamethread(ifthereareresponses).

Delving intothe individualcollections,the"Englishflood"onescontainsmanymoreusefullocationmentions (i.e. specific and relevant to affected people), while the "Spanish fire"collectioncontainsmostlyverygeneric location information (e.g.London).Thismightbeareflectionof themuchgreater varietyofmessage types in the former collection,whereasthelatterconsistsinamuchgreaterproportionof(oftennon-local)news.The“Greekheatwave”collection,notonlyincludesaconsiderablylowerpercentageoftweetswithsomeofsortlocationinformationmention,but,whensuchinformationispresent,itiseitheratcityor prefecture level; yet, this is largely attributed to the country-wide impact of the heatwaveeventsmentionedinthecollection.

4.3 Visualdatastudy

As aforementioned, as far as image and video inputs are concerned, and based on thecurrently available use case scenarios and respective specifications laid out in D2.1, keytypes of information to be targeted by visual content analysis involve scenes that depictflooded areas, smoke, fire and traffic congestions. The goal and challenge is to developalgorithms, of low computational cost, that can accurately discriminate in real-time suchsituations, and eventually enable to detect and predict elements at risk (people, cars,monuments,etc.).Inadditiontoaforementionedcontent-wiseobservationswithrespecttothevisualinformationofinterestthatdrovethecollectionofrespectivedatasets,welooked

D3.2–V0.4 D8.1–V1.0

Page28

into the characteristics entailed by the mainly considered capture devices, namely CloseCircuitTeleVision(CCTV)surveillancesystemsandmobilecameras,aswellasthepotentialaddedvaluethatcouldbebroughtthroughtheincorporationofUAVandwearablecameras:

§ CCTV surveillance systems are deployed in remote areas (i.e. across a forest orembankment)orregionsofinterest(i.e.acrossacity’sbridge,overahighway),soastomonitortheareaoverregulartimeintervals.Theusedframepersecondrecordingrateandframeresolutionarelowduetostorageandenergyconsiderations,asthecapturing procedure involves particularly large volumes of data (e.g. 5 days ofrecording could easily amount to 100G of visual data). The captured data aretransferredregularlyinacentralserver,fromwheretheacquisitionandsubsequentfeedingtothebeAWAREsystemandimageandvideocontentswilltakeplace.

§ Mobilevisualdata include imageandvideocontentscapturedbyciviliansandfirstresponders, posted on social media, i.e. Twitter and video-sharing websites (e.g.YouTube,Dailymotion, etc.) or feddirectly to the systemvia thebeAWAREmobileapplication. This data are usually accompaniedwith textual information that couldpossiblehelpananalysttodeductmoremeaningfulconclusions.Thegreatamountandvariabilityofsuchdata,duringacrisisscenario,requiresapowerfulserverandawell-designed big data strategy (i.e. targeted crawling, accurate data filtering,multimodalfusion)tobedeployedanddealwiththenatureofthisdata.

§ UAV visual data, captured by drones, usually comprise videos of short duration.Within our investigations, the main motivation would be that of first responders,trainedonUAV flight technology, operating such systems to capture theunfoldingsituationintheaffectedareaanduploadingthemtothebeAWAREsystemforonlineprocessing. It isworthmentioning that the stabilizers that havebeendeployedonthe latest UAVs afford high flexibility and thus applicability in even challengingemergencysituations.

§ Wearablecamerasarearelativelynewtechnologythathasbeenadoptedtorecordextremesportsandsincethenhasbeenexpandedtomorefieldsofinterest,suchascrisisscenario.Firstresponderscouldusethistechnologytocapturesmallvideosorimage samples and transfer them to a central center in order to deduct furtherconclusions.Thedatausuallycontainagreatdealofnoisethatisproducedfromthemovingcameraandrequirestabilizationalgorithmstoacquireasafeconclusion.

Last,andincollaborationwiththeuserpartners,theuseofsatelliteimagesthatcapturetheregion of interest before and after the crisis incident will be investigated in order todeterminewhetherthereisapotentialforaddedvaluewithinthetargetedpilots.Figure6showssomeexampletakenfromhttps://www.planet.com/disasterdata/datasets/

D3.2–V0.4 D8.1–V1.0

Page29

Figure6.Disasterdatasetforcompensatingcrisisevents

D3.2–V0.4 D8.1–V1.0

Page30

5 SummaryData collection is an important step towards the development of accurate and robustcontent analysis techniques that can adequatelymeet the pertinent, functional and non-functional,specificationsandrequirements.AddressingthemultimediainputsconsideredinbeAWARE,namelyaudio,textandvisualdatathatoriginatefromthemultitudeofinvolvedsources(Twitterposts,reportssentbyciviliansandfirstrespondersviathebeAWAREmobileapplication,callstoemergencynumbers,internalcommunicationsbetweenfirstrespondersand authorities, static cameras, etc.), this deliverable described the currently collecteddatasetsandthefindingsoftheirempiricalstudy.

Asoutlined,datacollectionactivitieshavebeeninitiatedfromtheearlyonsetoftheproject,startingwithhistoricaldataprovidedbytheuserpartnersandcontinuingwiththesearchforrelevantpubliclyavailableones;inparallel,tasksforthecollectionofdatawithinbeAWAREhavebeenlaunchedandalsoplannedduringthecourseoftheproject, inaccordancewiththescheduledfieldtrials.Anoverallobservationisthatasfarastextandvisualdataareconcerned,awealthofthemcan be acquired quite straightforwardly, both within beAWARE (Twitter messages arecontinuouslycrawledandannotated,userpartnersalreadydeploysuchdata,e.g.AAWAisusing static cameras across bridges tomonitor thewater level and velocity during floods,etc.)aswellasfromthirdparties.Audiodata,ontheotherhand,involvingemergencycallsand communications between first responders and authorities, present certain challengesdue to imposed privacy and security restrictions. However, as aforementioned, we arealready intouchwithemergencycallcentersfrombothuserpartnersandthirdparties,aswellaswithresearchgroupsthatpossesssuchdata inordertosortoutrespective licenseagreement documents; in parallel, and in collaboration with the user partners, we havealreadyinitiatedthecompilationofrecordingsofsimulatedcallstoensuretheavailabilityofbeAWARE-specificaudioinputs.

The empirical study of the collectedmaterial resulted to a number of interesting findingswith respect to the types of information of interest with respect to the development ofcorresponding analysis techniques, as well as the subsequent integration and aggregatedinterpretation. The study of the collected audio datasets outlined the key informationaspects that authorities and first responders seek, during the report of an incidence. Theexaminationofthecrawledtweetsrevealedpreliminarycontentandprovenance/authorshipcategories,while highlighting the need for contextualization, as individual tweets, in theirmajority,donotprovidemuchactionableinformation(e.g.locationinformationmaybetooconciseorabbreviatedtothepointofneedingadditionalexternalinformationtobemappedtoreal-worldlocations).Inaddition,thetypesofdifferencesobservedinthedistributionofthecontentsoftweetsforthethreeemergencydomains(fire,flood,heatwave),suggesttheexistence of correlations between the type of emergency and what people post, thusallowing to delineate, even coarsely, theunderlyingdomain’s scope and coverage.A finalobservation,tobefurtherinvestigatedasmoredataarebeencrawled,istowhatextentthekeywordsusedforfiltering,result inskewedconclusionswithrespecttothewealthoftheinformationthatisactuallybeingposted.Asfarasthevisualdataareconcerned,empiricalstudyshowedthat thevideoanalysismodulesdependsignificantly fromthenatureof the

D3.2–V0.4 D8.1–V1.0

Page31

capturingdeviceandthewaythatdataareacquired.Fastandlightweightprocessescanrunonline, but most of the data should be transferred into a powerful server and analyzedoffline. Recent developments onwearable and UAV devices pave new grounds for visualcontext analysis and understanding, yet, with a great deal of challenges that need to beaddressedbeforesafeconclusionscouldbededucted.

Last,andarealreadynoted, thedeliverablereportsonthedatasetsandempiricalanalysisfindingsthatcorrespondtothetimeofwriting.Asthisisacontinuousprocess,updatesandfurther refinements are expected andwill be reported in the upcoming deliverablesD3.3andD3.4 thatpresent thebasicandadvanced, respectively, techniques for theanalysisoftheaudio,textandvisualbeAWAREinputs.