Embed Size (px)
Citation preview
eResearchAustralasiaConference|Melbourne–Australia|10-14October-2016
Chiminey:ConnectingResearcherstoCloud,HPCandBigDataResources
IanThomas1,ImanYusuf2,HeinzSchmidt3,GrischaMeyer4,SteveAndroulakis5,DanielDrumm61.eResearchOffice,RMITUniversity,Melbourne,Australia,[email protected] 2.eResearchOffice,RMITUniversity,Melbourne,Australia,[email protected],RMITUniversity,Melbourne,Australia,[email protected],MonashUniversity,Melbourne,Australia,[email protected],MonashUniversity,Melbourne,Australia,Steve.Androulakis@monash.edu6.ARCCentreofExcellenceforNanoscaleBioPhotonics,RMITUniversity,Melbourne,[email protected]
ABSTRACTManyscientificexperimentshaveatwofoldchallenge:theyarechallengingasexamplesofcomplicateddomain-specificresearch tasks (i.e.,acomplicatedanalysesof thequantumphysicsapproaches)andat thesametimechallenging inthat the corresponding computations and datasets are too large-scale to be executed on a single desktopmachine.Thatis,moresophisticatedapproachesarerequired,suchascloud-basedandhigh-performancecomputing(HPC)solutions.Enablingofscientificexperiments increasingly includesdata,software,computationalandsimulationelements,oftenembarrassinglyparallel, long runninganddata-intensive. Increasingly, suchexperimentsare run inacloud environment or on high-end clusters and supercomputers.Many disciplines in sciences and engineering (andoutside computer science) find the requisite computational skills distracting from their focus on their sciencedomain. In this presentationwedescribeChiminey, our containerized cloud-basedplatform that provides a reliablecomputinganddatamanagementservice.
BACKGROUNDAny new technology usually means both opportunities and challenges, as technology often mandates some initialknowledgeacquisitiontaskof itsusers.Cloudcomputing [1]enablesacquisitionofvery largecomputingandstorageresourcesthatcanbeintegratedwithbigdatatechnologiesformassive-scalecomputation.Nevertheless,failurewhilesettingupacloud-basedexecutionenvironmentorduringtheexecutionitselfisarguablyinevitable:someoralloftherequestedvirtualmachines(VMs)maynotbesuccessfullycreated/instantiated,orthecommunicationwithanexistingVMmayfailduetolong-distancenetworkfailure–givencloudsdatacentersaretypicallyremoteandcommunicationcrossesmanynetworkboundaries.Also,alltasksofsuchparallelcomputationsarerequiredtocomplete,thereforethefailureofanyoneofthemmaycorrupttheresultinsomeway.Statisticallythismeansthatthereliabilityoftheoveralltaskcompletionistheproductofthatoftheindividualtasks–andwithverymanythousandsormillionsofcomputetasks,thismayquicklybecomeavanishinglysmallnumber. WhenusingcloudcomputingplatformsandbigdatatechnologieslikeHadoop,scientistsrequireoperationalskillsandtosomeextentknowledgeofaspectsoffaulttolerance.Scientistsneedtolearn,forexample,howtocreateandsetupvirtualmachines (VMs),collect the resultsofexperiments,and finallydestroyVMs.Suchchallengesdistract theuserfromfocusingontheircoregoals.Thus,theChimineyplatformencapsulatestheseproblemsandisolatesthemfromtheuser.Thisallowstheusertofocusondomain-specificproblems,andtodelegatethetooltodealwiththedetail thatcomes with accessing high-performance and cloud computing infrastructure, and the data management challengesposed.
CHIMINEY:CONNECTINGRESEARCHERSTOCLOUD,HPCANDBIGDATARESOURCESIn this presentation we describe Chiminey, our cloud-based platform that provides a reliable computing and datamanagementservice.Chimineyenablesdomainscientiststoexecutenewandlegacyapplicationsoncloud-based,bigdata (i.e. Hadoop), and high-performance computing (HPC) facilities without the need to have a technicalunderstanding of cloud computing, HPC, fault tolerance, or datamanagement. Chiminey handles failure during theexecutionofapplications. ItalsocuratesandvisualisesexecutionoutputsusingMyTardis,throughwhichdatacanbesharedwithcollaboratorsorwiththepublic.WedevelopedChimineyunderdirectionbyquantumphysicistsandmolecularbiologists,toreducethelearningcurveindatamanagementandsoftwareplatformsrequiredfor thecomplexcomputational targetsystems. Thisopen-source
eResearchAustralasiaConference|Melbourne–Australia|10-14October-2016
platform has been applied across two research disciplines, physics (material characterisation) and structural biology(understandingmaterialsat theatomic scale), toassess itsusabilityandpracticality.Thedomainexpertsnoted timesavings for computinganddatamanagement, auser-friendly interface for the computation setup, and theability tovisualizecalculationresultsas2Dor3Dgraphs.Thiscontainerisedplatformprovidesanabstractionofresearchtasksassmartconnectors.Theinfrastructureprovidedfor these connectors allows the domain scientists to choose the target platform, and then the target platform ismanagedautomatically; itacceptsall thenecessaryparameterstorunmany instancesoftheirprogramregardlessofwhether this runs on a peak supercomputer, a commercial cloud like Amazon EC2 or (in Australia) the nationalfederatedcloudsystemNeCTAR.Chimineynegotiateswithtargetsystemschedulers,dashboardsanddatabases,andprovidesaneasy-to-usedashboardinterfacetomonitortherunningjobs,regardlessofthespecifictargetplatform.Thesmartconnectorencapsulatesandvirtualisesanumberoffurtheraspectsthatthedomainscientistsdirectingoureffortfoundnecessaryordesirable. Previouswork:ThefirstprototypeofChimineyisdiscussedin[2].Aformalmodelofaplatformforscalableandfault-tolerantcloudcomputationsaswellastheimplementationofthisplatformastheChimineyplatformwasintroducedin[3]. The formalmodel allowsus to have a precise and concise specificationof theplatformon the logical level.Wepresenttherefinedformalmodelofacloud-basedplatformandthelatestversionofitsopen-sourceimplementation,withtheemphasisonusabilityandreliabilityaspectsin[4].ThefeasibilityoftheChimineyplatformisshownusingcasestudiesfromtheTheoreticalChemicalandQuantumPhysicsgroupatRMITUniversity.
REFERENCES1.M.Armbrust,A.Fox,R.Griffith,etal.,Aviewofcloudcomputing,Commun.ACM53(2010)50–58.2.I.Yusuf,I.Thomas,M.Spichkova,S.Androulakis,G.Meyer,D.Drumm,G.Opletal,S.Russo,A.Buckle,H.Schmidt,Chiminey:Reliablecomputinganddatamanagementplatform in thecloud, in:Proceedingsof the37th InternationalConferenceonSoftwareengineering,pp.677–680.3. M.Spichkova, I.Thomas,H.Schmidt, I.Yusuf,D.Drumm,S.Androulakis,G.Opletal,S.Russo,Scalableandfault-tolerantcloudcomputations:Modellingandimplementation,in:ProceedingsICPADS2015,pp.396–404.4.M.Spichkova,H.Schmidt,I.Thomas,I.Yusuf,S.Androulakis,G.Meyer,Managingusabilityandreliabilityaspectsincloudcomputing,in:Proceedingsofthe11thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering
ABOUTTHEAUTHORSIan Thomas is a software developer and system administrator at the eResearch Office of RMIT University. He hasworkedindatacurationforoutputofhigh-performancecomputingsystems,microscopydataformaterials,andscreenmedia objects (film and television). His current work is in institutional metadata stores, containerized researchworkflowsandincloud-basedplatformsinsupportofeResearchapplications.ImanYusufisasoftwareengineerandresearchacademicattheeResearchOfficeofRMITUniversity.Shehasworkedon projects that focus on distributed computing, in particular,cloud computing, and big data. She is one of thedevelopersofChiminey.Her research interestsaregridandcloudcomputing, fault tolerance, reliability, component-basedsoftwarearchitecture,andbigdata.Heinz is Professor of Software Engineering at RMITUniversitywhere he is theDirector of eResearch and heads theDistributed Software Engineering and Architecture lab in Computer Science. Heinz is also an adjunct professor atMälardalen University in Sweden. Heinz received his PhD from Bremen University, Germany. He has over 30 yearsexperiencewithcomponent-basedandobject-orientedarchitecture,especiallyinparallelanddistributedsystemsandlanguages inpractice, researchandeducation.Heinz is aneminent researcherwhohaspublishedover120 refereedarticles,supervisedover25higher-degreeresearchstudents,andlecturesinsoftwareengineering,distributedsystemsandenterprisearchitecture.PriortoRMITHeinzheldpositionsatMonashUniversity,theCSIROandANUinCanberra,attheGermanNationalResearchCentreforInformationTechnologyandtheInternationalComputerScienceInstituteat the University California, Berkeley. Prof Schmidt has led large university-industry research collaborations, in theEuropean ESPRIT program and the Australian Collaborative Research Center program, among others with SIEMENS,ABB,DECandOlivetti,IBMandothers.
eResearchAustralasiaConference|Melbourne–Australia|10-14October-2016
GrischaMeyeristhetechnicalleadontheInstrumentIntegrationprojectatMonashUniversity.HehasbeenworkingontheMyTardisprojectforover3yearsandisthecurrentMyTardiscoremaintainer.SteveAndroulakisisthemanageroftheMonashBioinformaticsPlatform.TheMBPprovidesbioinformaticssupportandtrainingtotheresearchcommunity.HehasalsocreatedandledtheMyTardisprojectforfiveyears.DanielDrummisapostdoctoral researcher in theRMITUniversitynodeof theAustralianResearchCouncilCentreofExcellenceforNanoscaleBioPhotonics(CNBP),studyingoptimalsearchalgorithms,quantumimagingtechniques,two-dimensional materials, and switchable fluorescent marker molecules. Within RMIT’s Theoretical Chemical andQuantumPhysicsgroup,hehasalsoworkedonabinitiohigh-performancecomputingsimulationsofdefectsandarraysof defects in semiconductors, organo-metallic crystals, ternary chalcogenide glasses, dye-sensitised solar cells, andtransition-metaloxides,aswellasdatacurationandworkflowautomationforthistypeofresearch.
