ESGF + DOCKER

LucaCinquiniNASA/JetPropulsionLaboratory+CaliforniaIns:tuteofTechnology

ESGFF2FWorkshop,Washington,DC,December2016

DockerinaNutshell

• Dockeristheleading“containeriza:on”technology:runanapplica:onasa“blackbox”onanyDocker-enabledserver‣ Build:imagesarebuiltasbundlesthatincludetheapplica:onitself,allofitsdependencies,and“just-enough-opera:ng-system”‣ Ship:imagesarehostedononlinerepositoriessuchasDockerHub‣ Run:imagesarerunascontainersonanyhostthatincludesaDockerdaemon

•WhyusingDockerforESGF?‣ Promisestogreatlyimproveinstalla:onandmaintenanceofanESGFnode‣ PartofDREAMstrategyformodularizingtheESGFarchitectureandpor:ngittootherdomains

Demo:InstallandRunanESGFNode

• Pre-requisite:installDockerEngineonhost(Linux,MacOSX,Windows)• Instruc:ons:‣ gitcloneh`ps://github.com/ESGF/esgf-docker.git‣ cdesgf-docker‣ exportESGF_HOSTNAME=‣ exportESGF_CONFIG=‣ ./esgf_node_init.sh‣ docker-composeup

h`ps://hub.docker.com/esggub/

h`ps://my-node.esgf.org/

CoG

TDS

Solr

https://github.com/ESGF/esgf-docker.git

ESGF/DockercompleteNodeArchitecture

• AllESGFservicesarerunasindependent,interac:ngDockercontainers• SpecificNodeconfigura:on(cer:ficates,passwords,XMLfiles)savedin$ESGF_CONFIG• SpecificNodedata(Postgresdb,Solrindexes,TDScatalogs,NetCDFfiles,CoGsitemedia)storedonDockervolumes

• Customnetworksisolateapplica:onsforaddi:onalsecurity(forexample,Postgresdb)

Threads Data Server+ OpenID Relying Party

(Tomcat)

ESGF DATA NODE

ESGF POSTGRES DB

TDS CATALOGS

ESGF CONFIG DATA

ESGF SOLR(Jetty)

SOLR DATA

ESGF CONFIGURATION(site specific)

esgf-search webapp(Tomcat)

ESGF INDEX NODE

ESGF COG

COG DATA

httpd + mod_wsgi

ESGF WEBUI NODE

ESGF IDP NODE

esgf-idp webapp(Tomcat)

ESGF/Docker complete Node Architecture

POSTGRES DATA

ESGF DATA

ESGF FTP

docker-compose-fdocker-compose.ymlup

DockerDetails

Dockerfile:“recipe”forbuildingaDockerimage

docker-compose.yml:configura:onfileforbundlingseveralimages

ESGF/DockerDataNodeArchitecture

docker-compose-fdocker-compose-data-node.ymlup

Threads Data Server+ OpenID Relying Party

(Tomcat)

ESGF DATA NODE

ESGF POSTGRES DB

TDS CATALOGS

ESGF CONFIG DATA

ESGF CONFIGURATION(site specific)

ESGF/Docker Data Node Architecture

POSTGRES DATA

ESGF DATA

ESGF FTP

ESGF/DockerIndexNodeArchitecture

docker-compose-fdocker-compose-index-node.ymlup

UsingstandardSolrreplica:on+distributedsearch

ESGF CONFIG DATA

ESGF SOLR(Jetty)SOLR DATA

ESGF CONFIGURATION(site specific)

esgf-search webapp(Tomcat)

ESGF INDEX NODE

httpd + mod_wsgi

ESGF WEBUI NODE

ESGF/Docker Index Node Architecture

Slave Shard Master Shard

ESGF/DockerIndexNodewithSolrCloud

docker-compose-fdocker-compose-solr-cloud.ymlup

ESGF CONFIG DATA

ESGF SOLR CLOUD(Jetty)SOLR DATA

ESGF CONFIGURATION(site specific)

esgf-search webapp(Tomcat)

ESGF INDEX NODE

httpd + mod_wsgi

ESGF WEBUI NODE

ESGF/Docker Solr Cloud Architecture

Shard 1 Shard 2 Shard 3ESGF SOLR CLOUD

(Jetty)

Shard 1 Shard 2 Shard 3ESGF SOLR CLOUD

(Jetty)

Shard 7 Shard 8 Shard 9

• Solr-Cloudadvantages:‣ Automa:cdistributedindexingandsearching(nocustomconfigura:on)‣ Loadbalancingandhighavailability‣ Faulttolerance‣ Scalability

AdvantagesofusingDockerforESGF

• Installa:on:‣ Installa:onscriptsaremuchmoremodularandmuchsmaller‣ Installa:onprocessismusheasier:simplydownloadtheimages,nocompila:oninvolved‣ Easierupgrades,possiblyonemoduleata:me,andreversible‣ Everybodyrunsexactlythesamesokware

• Architecture:‣ Candefineanddeploynewarchitecturesbysimplywri:ngnewconfigura:onfiles‣ Canintroducenewmodulesbysimplywri:ng&wiringnewimages(solr-cloud,nginx,…)

• Portability:ESGFnodewillrunonanyplanorm(Linux,MacOSx,Windows)includingCloud

• Scalability:modulescanbescaledarbitrarilybyrunningmorecontainers(e.g.TDS,WPS…)‣ Caveat:applica:onmustbewri`entoenabledistributedaccesstodata

• OtherDocker/Swarmadvancedfeatures:‣ Scalabilityontomul:plehostsclusters‣ Highavailability,faulttolerance,rollingupdates,…

DisadvantagesofusingDockerforESGF

•Dockerisanewparadigmforbuildingandrunningapplica:ons:‣ Newknowledgeforapplica:ondevelopers‣ Newtrainingfornodeadministrators‣ Excellentandup-to-datedocumenta:onisamust

•MustporttheremainingESGFmodulestoDocker:‣ ESGFPublisherClient‣ Globus‣MyProxyornewOAuthserver‣ Desktop+Dashboard‣ LAS

•Mustdevelopaprocesstomigrateallapplica:ondata:‣ ESGFPostgresdatabase(usersanddata)‣ TDScatalogs‣ Solrindexes‣ CoGPostgresdatabaseandsitedata

PossibleFutureRoadmap

•Establisha“whaleteam”ofexpertstoworkon:‣ Testthecurrentinfrastructure‣ Porttheremainingmodules‣Workonotheroutstanding:ckets‣ Developates:ngsuite‣ Developdatamigra:ontools‣ Reviseandexpandthedocumenta:on‣ Re-usecurrentesgf-iwtbiweeklymee:ngs?

•Migra:ngtoDockerwouldtake6-12months‣Mustsupportcurrentorupgradedinstalledinthemean:me‣MightwanttoswitchakerCMIP6-lookingintothelong-termlongevityofESGF…