View
2
Download
0
Category
Preview:
Citation preview
EMSODataManagementPlatform
PasqualeAndrianiasdasdasdOceanologyInternational2018London,Excel/March15,2018
Context
HIGH-LEVELARCHITECTURE
Dataaccess
DMPtools
DATAMANAGEMENTPLATFORM
Real-tim
eAsynch.
Dataingestion
Ingestionspeed
PHYSICALINFRASTRUCTURE
EMSO DMP • Production VO: vo.emsodev.eu • Cloud Compute: 11 VMs (8 CPUs + 16GB RAM + 40GB) • File Storage: 5 TB
Jun-2016«TestVO» Aug-2016«SLAdefinition» Nov-2016«ProductionVO»
FROMSOLUTION… datause
dataprocessing
WebHDFSAPI
datapublishing
dataacquisition
EMSODEVDATAMANAGEMENTPLATFORM
datacuration
MOODA
FROMSOLUTION…(DATAACQUISITION)
DataacquisitionPushTransferFlow:dataissenttoaDMPservice.PullTransferFlow:dataisretrievedviaAPIexposedbyanEGIMSOSserver.
dataacquisition
PUSHtransferflow
PULLtransferflowAPI
FROMSOLUTION…(DATACURATION)Datacuration
• Datamanipulation(datascrapingandmunging)• Datacollection(datapersistintodifferentstorage)
dataacquisition
datacuration
FROMSOLUTION…(DATAUSE)
TheDMPprovidesthescientificuserwithasetofdifferenttools(DMPToolS):• EMSODEVAPIallowsscientificusersandotherEuropeaninitiativesintheoceansciencesto
interfacewiththedataavailablewithintheDMP.• MOODA(ModuleforOceanObservatoryDataAnalysis)isacustomapplicationdevelopedfor
thescientificcommunity.• GRAFANAisadashboardtoquery,visualiseandunderstandthemetricsstoredintotheDMP.
MOODA
FROMSOLUTION…(DATAPROCESSING)
dataprocessing
• ApacheSpark:fastandgeneralengineforlarge-scaledataprocessing.Itisabletoexecutebothbatchandstreamingdataprocessing.
• ApacheZeppelin:interactivedataanalyticstool.Itisabletoperformdataanalysiswithdifferentprocessingsysteminterpreters.
ODVFileOceanDataView
…TOOPERATION
12hoursofSeaWaterTemperature
The EMSO-Link project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements N° 731036.
Thankyouforyourattention.
PasqualeAndrianipasquale.andriani@eng.it
Recommended