e516 - Engineering Cloud Computing4.5.5 Exercise Cloud API 4.6 Week 6: REST 4.6.1 Lecture Material 4.6.2 Lab Activities 4.6.3 Graded Lab Activity 4.7 Week 7: MapReduce - 4.7.1 Lecture

E516-ENGINEERINGCLOUDCOMPUTING

GregorvonLaszewski

(c)GregorvonLaszewski,2020

E516-ENGINEERINGCLOUDCOMPUTING

1PREFACE1.1Disclaimer☁1.1.1Acknowledgment1.1.2Extensions

2SYLLABUS2.0.1SyllabusEngineeringCloudComputing☁2.0.1.1Instructor2.0.1.2Audience2.0.1.3Coursesummary2.0.1.4References2.0.1.5Tools2.0.1.6CourseSchedule2.0.1.7Attendance2.0.1.8Assignments2.0.1.9GradeBreakdown2.0.1.10ProjectExamples2.0.1.11StatementonAcademicMisconduct

2.1Calendar☁2.1.1Deadlines

3OVERVIEW3.1Summary☁3.2Communication☁3.2.1ClassMaterial3.2.2Piazza3.2.3ClassResources3.2.4OnlineMeeting3.2.5Assignments3.2.6PostYourBio3.2.7EvolvingDocument3.2.8Books3.2.9YouGetCreditforImprovingtheBooks3.2.10OngoingAgenda

3.3QuickTipsfortheClass☁3.3.1Requirements3.3.2TimeCommitment3.3.3CourseMaterialList3.3.4Help3.3.5HowtoTakethisClass3.3.6Assignments3.3.6.1TechnologyReview3.3.6.2Project3.3.6.2.1License3.3.6.2.2ProjectReport3.3.6.2.3ProjectCode3.3.6.2.4ProjectData3.3.6.2.5WorkBreakdown3.3.6.2.6Bibliography3.3.6.2.7Reproducibility3.3.6.2.8ListofDeliverables3.3.6.2.9ExampleOutlineofaReport

3.3.7Submission3.3.8BonusProjects3.3.9Participation3.3.10GitHubandPycharmMarkdown

4WEEKLYAGENDA4.1Week1:CourseIntroduction☁4.1.1Assignments4.1.2ClassVideos4.1.3LabActivities4.1.3.1AccountCreation4.1.3.2Survey4.1.3.3Bio4.1.3.4Python3.8

4.1.4OnlineMeetingRecording4.2Week2:CloudDataCenters☁4.2.1LectureMaterial4.2.2LabActivities4.2.2.1ReceiveyourHID

4.2.2.2README.yml4.2.2.3ChameleonCloud4.2.2.4ReviewPython4.2.2.5SetupyourComputer4.2.2.5.1Python

4.2.2.6UbuntuMultipass4.2.2.7Optional:PlagiarismCertificate

4.2.3OnlineMeetingRecording4.3Week3:CloudComputingArchitectures☁4.3.1LectureMaterial4.3.2LabActivities4.3.2.1Review:venvinpython3(Graded)4.3.2.2Review:Anaconda(Graded)4.3.2.3PythonLanguagereview4.3.2.3.1Dicts4.3.2.3.2f-StringsinPython34.3.2.3.3PythonClasses4.3.2.3.4PythonModules

4.3.2.4GradedActivities4.3.3ChapterSelection4.3.4ProjectSelection4.3.5WorkingAhead4.3.6OnlineMeetingRecording

4.4Week4:Openstack☁4.4.1Video4.4.2LectureMaterial4.4.3Lab:OpenStack4.4.4Namingofvms4.4.5Horizon4.4.6CloudmeshOpenStackinterface4.4.7OpenStackCommandlineClient4.4.7.1InstallationofCloudmeshCloudBundle4.4.7.2SSH4.4.7.3Configuration4.4.7.4CloudmeshMongo4.4.7.5StartaVM

4.4.8WorkingwiththeVM4.4.9Exercise

4.5Week5:CloudVMComputeService☁4.5.1Videos4.5.2LectureMaterial4.5.3Lab:VMComputeServices4.5.4ExerciseMultipass4.5.5ExerciseCloudAPI

4.6Week6:REST☁4.6.1LectureMaterial4.6.2LabActivities4.6.3GradedLabActivity

4.7Week7:MapReduce�☁4.7.1LectureMaterial4.7.2LabActivity:HadoopInstallation4.7.2.1Exercises:

4.7.3LabActivity:PythonMapReduce4.7.4LabActivityMap/ReduceontheCloud4.7.4.1PythonWordCountinMapReduce4.7.4.2RunHadoopMapReduceinPython

4.7.5LabActivity:MapReduceonthecloud4.7.5.1HadoopClusterSetup4.7.5.2Configuration4.7.5.3OperatingtheHadoopCluster4.7.5.4RunHadoopMapReduceinPython

4.8Week8:ProjectReview☁4.9Week9:Containers-Docker☁4.9.1LectureMaterial4.9.2LabActivityDocker4.9.3LabBookmanager4.9.4 Optional Lab Activity Container REST service (as part ofmanyprojects)

4.10Week11:GraphQLandMessaging☁4.10.1LectureMaterial4.10.2LabActivityGraphQL4.10.3OptionalLabActivityMQTT

4.11Week12:GO☁4.11.1OptionalLabActivityGoRESTService

4.12Week13:FaaS�☁4.12.1LabActivity

4.13WeekN-16☁5APPENDIX5.1FAQs☁5.1.1CourseManagent5.1.1.1Whenisthelastdatoftheclass?5.1.1.2Whatdothe+-intheFAQsinPiazzamean?5.1.1.3PleasereadtheFAQ5.1.1.4WheredoIfindthematerialforthisclass?5.1.1.5Wheretofindtheweeklyagenda?5.1.1.6Whendotheonlinemeetingstakeplace?5.1.1.7WhyswitchthecomputertoEnglishlanguagemode?5.1.1.8WhyshouldInotusedarkmode5.1.1.9WaituntilGregorsaysENTER5.1.1.10Characternamesonthekeyboard5.1.1.11WherecanIfindtherecordingsoftheonlinemeetings?5.1.1.12DoIneedtoreadtheclassmaterialtotaketheclass?5.1.1.13Doesthisclasshaveamidtermandfinal?5.1.1.14Balanceyourclasses.5.1.1.15Whenisthelastdayofthisclass?5.1.1.16WillyouprovidesupportduringtheSpringbreak?5.1.1.17Lab:WhycanInotsolveanassignmentin2minutes?5.1.1.18 Lab: How much time do I need to spend on eachquestion?5.1.1.19IdonotunderstandPiazza?5.1.1.20WhyyoushouldnotreadthePiazzamailbutuseClick Hereinstead?5.1.1.21CanIsearchintheePubreader?5.1.1.22WheredoIfindtheGitHubdirectory?5.1.1.23WherecanIfindthehiddirectory?5.1.1.24DoIneedthefuturesystems.orgaccount?

5.1.2HardwareandSoftwareRelatedQuestions5.1.2.1CanIusemyworkcomputerforthisclass?

5.1.2.2 How much menmory have these days developmentcomputers?5.1.2.3WhatHardDrivedoyourecommend?5.1.2.4CanIuseWindows1809?5.1.2.5RasperryPIClusterComponents5.1.2.6Whichoperatingsystemcanweuseforclass?5.1.2.7WhatdoIneedtorunHyper-VforWindows?5.1.2.8WherecanIgetWindows10EDUforfree?5.1.2.9CanIrunmultipassinWindows10Home?5.1.2.10WhatSoftwarewillinneedforthisclass?5.1.2.11Whatisthebesteditorforclass?5.1.2.12CanIuseVS-Code?5.1.2.13Whataarethebiggesissueswenoticed?

5.1.3Markdown5.1.3.1Whatisthebesteditorforbibliographies?5.1.3.2CanIjustcreatanempty.bibfile?5.1.3.3Whatisan�inthemarkdownfile?5.1.3.4Howtosetthecolumnwidthinpycharmto80characters?5.1.3.5Whatisagoodspellsheckerformarkdown?5.1.3.6 Why must i not use the words bellow and above inmarkdown5.1.3.7WhymustIavoidscreenshotsforterminaloutput?

5.1.4EpubandDocumentRelatedQuestions5.1.4.1Isthereanepubreaderforchrome?

5.2ePubReaders☁5.3FAQ☁5.3.1Whereisthemanual5.3.2Videos5.3.3Havingcontroloveryourcomputers5.3.4DeploymentTipCollection5.3.4.1InstallMongoDB5.3.4.2Doanupdateofthesource5.3.4.3InteractwithChameleonCLoud5.3.4.4Draft:CmsMostlyautomatedinstallonmacOSandLinux

5.3.5Switchingbetweencmsandcmsd5.3.6WhatisKeyError:‘auth’

5.4FAQforWindows☁5.4.1Videos5.4.2CMSrestrictions5.4.2.1CommandthatdonotworkonWindows

5.4.3Prerequisits5.4.4VisualStudioBuildTools5.4.5MongoDB5.4.6WhichversionofWindowsdoIneedtousemultipass?5.4.7multipassHyper-VinstalationonWindows5.4.7.1GeneralWindowsTips5.4.7.1.1WheredoIfindthereleasenumbersforWindows?5.4.7.1.2HowcanIgettheEDUVersion?5.4.7.1.3IssueswithdockeronWindows5.4.7.1.4UsernamesinWindowsmustnothavespaces

5.4.8Whattodoifwindowsstilldoesnotworkforyou?5.4.9HowtoinstallsshonWindows

5.5FAQchameleoncloud☁5.5.1Didcmseverworked?5.5.2Howcheckifchmaeleoncloudworks5.5.3HowdoIuploadmykeytochameleon?5.5.4Whatistheerror…?5.5.5Whatisthechameleonaccountbug?5.5.6YoumustnotuseIUGuestnetwork5.5.7Tetheringyoucomputerthroughacellphone5.5.8CanIkeepmyinstancerunningwhennotinuse?5.5.9WhatisWARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!5.5.10ssh-keygenvsputty5.5.11CanIgivemyownnetworkwhenbooting5.5.12HowdoIuploadthesecgroups5.5.13ArethereadditionalVideos5.5.13.1Startingvmsinhorizon5.5.13.2StartingVMsincloudmesh

6REFERENCES

1PREFACE

TueMar1722:35:18EDT2020☁

1.1DISCLAIMER☁

ThisbookhasbeengeneratedwithCyberaideBookmanager.

Bookmanagerisatooltocreateapublicationfromanumberofsourcesontheinternet.Itisespeciallyusefultocreatecustomizedbooks,lecturenotes,orhandouts.Contentisbestintegratedinmarkdownformatasitisveryfasttoproducetheoutput.

Bookmanagerhasbeendevelopedbasedonourexperienceover thelast3yearswithamoresophisticatedapproach.Bookmanager takes the lessonsfromthisapproachanddistributesatoolthatcaneasilybeusedbyothers.

Thefollowingshieldsprovidesomeinformationaboutit.Feelfreetoclickonthem.

pypipypi v0.2.33v0.2.33 LicenseLicense Apache2.0Apache2.0 pythonpython 3.73.7 formatformat wheelwheel statusstatus stablestable buildbuild unknownunknown

1.1.1ACKNOWLEDGMENT

If you use bookmanager to produce a document you must include thefollowingacknowledgement.

“This document was produced with Cyberaide Bookmanagerdeveloped by Gregor von Laszewski available athttps://pypi.python.org/pypi/cyberaide-bookmanager. It is in theresponsibility of the user to make sure an authoracknowledgement section is included in your document.Copyright verification of content included in a book isresponsibilityofthebookeditor.”

https://github.com/cloudmesh-community/book/blob/master/chapters/version.md

https://github.com/cyberaide/bookmanager/blob/master/bookmanager/template/disclaimer.md

https://pypi.python.org/pypi/cyberaide-bookmanager


https://github.com/cloudmesh/cyberaide-bookmanager/blob/master/LICENSE




https://travis-ci.com/cloudmesh/cyberaide-bookmanager

Thebibtexentryis

1.1.2EXTENSIONS

Wearehappytodiscusswithyoubugs,issuesandideasforenhancements.Pleaseusetheconvenientgithubissuesat

https://github.com/cyberaide/bookmanager/issues

Please do notfilewith us issues that relate to an editors book.Theywillprovideyouwiththeirownmechanismonhowtocorrecttheircontent.

@Misc{www-cyberaide-bookmanager, author = {Gregor von Laszewski}, title = {{Cyberaide Book Manager}}, howpublished = {pypi}, month = apr, year = 2019, url={https://pypi.org/project/cyberaide-bookmanager/} }

https://github.com/cyberaide/bookmanager/issues

2SYLLABUS

2.0.1SYLLABUSENGINEERINGCLOUDCOMPUTING☁

LearningObjectives

Getaquickoverviewofwhattheclassisabout

2.0.1.1Instructor

[email protected]:Byappointment

2.0.1.2Audience

Werecommendyouknowoneprogramminglanguage.Knowledgeofpythonisofadvantagebutnotrequired.Pythoniseasy.AllassignmentsandprojectsareconductedinPython.

2.0.1.3Coursesummary

This class introduces you to state-of-the-art cloud computing concepts anengineeringapproaches.Thiswillincludevirtualmachines,containers,andMap/Reduce. The course has Labs in which you can practically exploretheseconcepts.Youwill,forexample,havetheoptiontocreateacloudaspartofthiscourseandexplorecloudcomputingtoolsandframeworks.

2.0.1.4References

Thecoursedoesnothavearequiredreadinglist.Weprovidethefollowingreferencesasstartingpointerstowhatwediscuss:

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/syllabus-2020.md

mailto:[email protected]

CloudComputingLinuxPythonMarkdown

However,pleaserememberwewillselecttopicsandincludematerialthatisormay not be covered in these books. The bookswill evolve during thesemester.

2.0.1.5Tools

You are required to have a computer to log into the cloud.We give youaccess toanOpenStackcloud.Access toAzure,AWS,Googleandotherscanbeachievedthroughtheirfreetier.

2.0.1.6CourseSchedule

Week References1 CourseIntroduction2 CloudDataCenters3 PythonforCloudComputing,StartofProjectSelection4 CloudArchitectures5 VirtualizationI-OpenStack6 VirtualizationII-AWS,Azure,Google7 Multi-CloudEnvironment8 CloudTechnologyPresentation-ProjectReview9 Containers-Docker,Kubernetes10 MapReduce11 Messaging12 REST13 GO

https://laszewski.github.io/book/cloud/

https://laszewski.github.io/book/linux/

https://laszewski.github.io/book/python/

https://laszewski.github.io/book/writing/

Week References14 ProjectWork15 ProjectsDue16 ProjectImprovements

For each of the topics, you will find one or more relevant chapters orsectionsinouronlinebook.

2.0.1.7Attendance

Attendanceaccountsfor10%ofyourfinalgrade.Ifyouneedtoskipclassforanyreason,youneedtonotifytheinstructorandTAs.

2.0.1.8Assignments

This course will not have exams. Instead, we have the following gradedassignmentcategories:

LabAssignments(pass/fail):areassignmentsthatwillbeconductedonaweeklybasis.Theywillhelpyoumakesureyounotonlyunderstandthematerialtheoreticallybuttrythemout.

Cloud Technology Chapter with Examples (Graded): (This can besubstituted if your project is related to Cloudmesh or NIST whileprovidingamanual andusageexample).This is adocument about aCloudtechnologythatisnotyetincludedinourbookstointroduceaninterested party to it. It should not contain advertisements but be arational description of the technology with examples that you havetriedyourself.Youwillhavetogiveanon-plagiarizedpresentationanddocument about it. The document is formulated as a chapter to thebookwithproperreferencesandworkingexamples.Due:Midterm

ProjectAssignments(Graded):Themostimportantpartoftheclassforwhich you will be working throughout the semester. Up to three

students canwork in aproject. In caseofgroupprojects, theprojectdeliverables are increased. Due: after midterm every weeksubmissions.

To support the project evolution, it has three submissions that are spreadthroughout the semester.Each submissionbuildson thepreviousone andmodifies previous documents into a consistent project report anddocumentationforyourproject.

ProjectOutline

Adescriptionofwhatyourprojectisaboutandhowitrelatestocloudcomputingandaddress:

Whatistheproblemyoutrytoaddress?Whatareyoudoingtoaddressthisproblem?Howareyouaddressingthisproblem?Whatisthearchitecturethataddressestheproblemthatyouwillimplement?

CodeandDocumentationReview

You will be asked to have a meeting with the TA’s and/orinstructor to showcase your code and have at least one reviewpriortoyourfinalsubmission.

ThiswilltakeusuallytakeplacethroughtheLabhoursonregularbasis.

A first project discussionmust have been done at least once atmidtermtime.

FinalProjectSubmission

All code and documentationmust be checked intoGitHubwellbeforethesemesterisover.Thisallowsustogiveyoufeedbackforimprovements.

In addition to thesemajormilestones, youwill have a project discussionandupdatetoreportoneveryweekstartingfromthemidtermdate.

Pleasenotethatthesyllabusissubjecttochange.Wemayalsochangetheorder.Studentsinthisclassoftencomefromawidevarietyofbackgroundsand experiences. As such, the instructor reserves the right to change thecontent of the course to accommodate the needs and expectations of thestudents.

2.0.1.9GradeBreakdown

70%Project10%Chapter10%Assignments10%Particpation

There is nowritten exam for themidtermorfinal in this class.However,therewillbeaprojectreviewatmidtermandfinaltime.Duringthelasttwoweeks of the semetser, intensive reviews are conducted. Please do notassumethatafteryouhavehandedinyourproject,youaredone.Youhavetheabilitytoimproveitbasedonfeedback.

Thedeliverablesformidtermandprojectreviewesarediscussedelsewhere

2.0.1.10ProjectExamples

Asmalllistofprojectexamplesareavailableat:

https://cloudmesh.github.io/cloudmesh-manual/projects/index.html

Wewilladdtothislistasthesemesterprogresses.Welikeyoutoidentifyaprojectwithinthefirtsmonthofthesemester.

2.0.1.11StatementonAcademicMisconduct

https://cloudmesh.github.io/cloudmesh-manual/projects/index.html

Students will be expected to uphold and maintain academic andprofessional honesty and integrity as outlined in the Code of StudentRights,Responsibilities,andConduct.Casesofacademicmisconductwillbe handled according to the student disciplinary procedures described inIU’spolicies.

2.1CALENDAR☁

Pleasealertusifwemadeamistake!

HerewenotetheIUclanedarandmaptheweeknumbers.Pleasenotethatthespringbreakweekisnotincludedinthenumbers.Westronglysuggestyouplnavacationwisely.OftenSPringbreakisagoodtimetocatchuporworkahead.

Spring2020

Theofficialcalendarispostedat

https://registrar.indiana.edu/official-calendar/official-calendar-spring.shtml

BeginsMon,Jan13MLKJr. DayMon,Jan20AutoWSun,Mar15SpringBreakMar15-Mar22FinalExamsMay4-May8

EndsFri,May8

Thislistwillbeupdatedthroughoutthesemester.

Week Date GradedAssignmants

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/calendar-2020.md

https://registrar.indiana.edu/official-calendar/official-calendar-spring.shtml

Week Date GradedAssignmants1 13Jan-19Jan Piazza,Bio,Survey2 20Jan-26Jan Python3.8,Multipass,HID,github3 27Jan-2Feb TechnologyChapterandProjectUpdate4 3Feb-9Feb TechnologyChapterandProjectUpdate5 10Feb-16Feb TechnologyChapterandProjectUpdate6 17Feb-23Feb ProjectProposal7 24Feb-1Mar ProjectUpdate8 2Mar-8Mar ProjectUpdate9 9Mar-15Mar ProjectUpdate- 16Mar-22Mar Break,goodtimetoworkahead10 23Mar-29Mar ProjectUpdate11 30Mar-5Apr ProjectUpdate12 6Apr-12Apr ProjectUpdate13 13Apr-19Apr ProjectUpdate14 20Apr-26Apr ProjectDue15 27Apr-3May ProjectImprovements16 3May-8May ProjectImprovements

In addition to the listedGraded Assignments Labs will be posted. Allactivities will be listed in the weekly updated Lecture Notes so you canidentifywhathastobedone.

Pleaseconsulttheweeklylecturenotesforthecompletelistofactivities.

2.1.1DEADLINES

Feb3:DataCenterAssignmentdueFeb3:LastdaytoproposethechapterFeb10:cloudmesh sys generate commandassignmentdue

Feb10:LastdaytosubmitthefirstversionofthechapterFeb10:LastdaytosubmityourprojectproposalFeb 17: Last day to submit your peer review of another studentschapterFeb24:LastdaytoimproveyourownchapterApr26:FirstProjectreviewofthealmostfinalprojectMay3 -8:opportunity to improve.Remember thatwehavea lotofprojects,sonoteveryonecansubmitonthe8th.Mar8:Lastdaytosubmityourmidtermprojectreport

3OVERVIEW

3.1SUMMARY☁

Pleasewatchthefollowingvideoandreadthischapter.

OverviewEngineeringCloudComputing

Youmaylikesomeofthequotesfromstudents:

“BestclassIevertookatIU.”

“ThankstoyouIgotajobattopratedcompany.”

“Mycompanyhasadoptedlessonslearnedfromyourclass.”

“The level of support and expertise the professor provides isoutstandingandunmatchedwithinIU.”

3.2COMMUNICATION☁

3.2.1CLASSMATERIAL

Most classmaterial is distributed as ePubs and PDF or resources on theWeb. Our ePubs render on many computers very nicely. Thus werecommendyouuse ePub insteadof thePDFversions of our documents.PleaseinstallanePubReaderifyoursystemdoesnothaveone.

macOS:hasabuildePubreadercalledBooks.Windows10:CalibreLinux:CalibreOthers:Linux:Calibre

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/summary.md

https://www.youtube.com/watch?v=PU93waozGJk

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/communication.md

https://calibre-ebook.com/



Naturally, youcan read the ePubs alsoonyour tablets insteadofonyourcomputers,ifyouhaveatabletwithlotsofmemory.

Pleasebeawarethatthedocumentswillbeupdatedweekly,andyouneedto download them accordingly. Make sure you clear the cache if yourbrowserpreventsthedownloadofthenewestversion.

3.2.2PIAZZA

ThisclassusesPiazzaforcommunication.Itisyourresponsibilitytoenrollin thePiazza for theclass.A link toourPiazza isprovided inCANVAS.Piazzaworksjustlikeaforuminwhichyoucanaskquestionsorpostnotes.Incaseofaquestion,studentsandteacherscanformulateanswers.Athreadsysteminthequestioncanbeusedtogathertheanswer.

https://piazza.com/iu/spring2020/e516spring2020/resources

Youwill get a grade for participation in the class discussions on Piazza.PleasenotethatwedonotrecommendyougofromCANVAStoPiazzaasCANVAShasanbadHTMLlayoutandinfact,doesnotallowyoutoseethePiazzahyperlinks.Instead,gotoPiazzadirectly.Useourlink.

3.2.3CLASSRESOURCES

Allclassmaterialwillbepostedat


3.2.4ONLINEMEETING

Ifyouareanonlinestudent:

ApollhasbeenpostedthatyoumustfilloutbeforeFridayofthefirstweekofthesemesterforpossiblemeetingtimes.

https://piazza.com/class/k4xfd8nxutk4ka?cid=9




Thelinktothemeetinginformationisat


3.2.5ASSIGNMENTS

All assignments will be posted in our lecture notes. Graded assignmentswillalsoappearinCANVAS.Youcanfindthelinktotheminourresourcesection:


3.2.6POSTYOURBIO

Toassure thatyouhaveaccess toPiazzaandcanpost to it,please,postaformalBioto:

https://piazza.com/class/k4xfd8nxutk4ka

PleaseinvestigatewhataformalBiois.

3.2.7EVOLVINGDOCUMENT

Thisdocumentisimprovedthroughoutthesemesterwithweeklyactivities.Werecommendthatyouregularlydownloadanewversion.BeawarethatsomebrowsersorePubreadersmaycachethepreviousversion.Thusmakesuretocheckoutthenewestversion.

3.2.8BOOKS

This class does not use books from publishers as cloud computing isconstantlychanging,andbythetimeyouplacethepurchaseorder,thebookmayalreadybeoutdated. Instead,wehavepreparedonlinebooks thatarecontinuallyevolving.Itisevenexpectedthattoolsorservicesyouusewillevolveduringthesemester.



https://piazza.com/class/k4xfd8nxutk4ka

WehavethefollowingbooksinePubandPDFavailable.TheePubversionisourpreferredversion:

ListofBooks

TheweeklyLecturenotesarecontainedat

e516:CloudComputingEngineering

Thispagewillbringyoutothefollowingbooksthatweuseaspartofe516.ItwillalsoclearlyspecifywhichassignmentsyouneedtodoaspartoftheweeklyLabactivities.

CloudComputingLinuxPythonMarkdown

3.2.9YOUGETCREDITFORIMPROVINGTHEBOOKS

Pleasenote that improving thedocument inGitHubviapull requestswillget you credits. This is an easy opportunity for you to get an excellentgrade.

3.2.10ONGOINGAGENDA

WewilladdhereweeklythetopicsandLabsthatwewillcoverinthecomingweek(s).Pleasebeawarethatjustasinregularlectures,weaddtheagenda items and follow the syllabus. We may change the order of thetopicscoveredinthesyllabusasappropriate.

3.3QUICKTIPSFORTHECLASS☁

3.3.1REQUIREMENTS

https://laszewski.github.io/#books

https://laszewski.github.io/book/e516/





https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/quicktips.md

Werecommendthatyouknowoneprogramminglanguage.AlthoughmostactivitiesaredoneinPython,thisprogramminglanguagedoesnothavetobePython,asitcaneasilyandquicklybelearnedthroughoutthesemester.Nobackgroundincloudcomputingisneeded.

3.3.2TIMECOMMITMENT

Anyclassatauniversityrequiresasignificanttimecommitment.Duetothedifferentbackgrounds thestudentshave, it isdifficult topredict theactualtimeneeded.Onaverage,weseethatstudentsspend6hoursontheclassifthey do participate every week. Students with little programmingexperiencespendupto12hours.

Naturally, you need to balance your course work. If another class hasassignmentseveryweekandyoudonotspendatleast6hoursonthisclass,also you will have a hard time with this class. If you spend less than 6hours, please, work ahead and do not just spend the 6 hours on anotherclass.

3.3.3COURSEMATERIALLIST

Course material is distributed as ePubs, PDF, Video, or Presentations.However,youarerequiredtoresearchsometopicsontheinternetasCloudComputing is a rapidly evolvingfield, andyou likelyfind themostup todate informationon theWeb.However,notall informationon theWeb isaccurateorworks.Itwillbeinyourresponsibilitytofilterthatinformation.

ThroughoutthesemesterwealsoaskyoutohelpusupdatingtheePubandtouseadditionalresourcesasappropriate.

Thelistofresourcesfortheclassinclude:

CloudComputing[1]IntroductiontoPython[2]LinuxforCloudComputing[3]ScientificWritingwithMarkdown[4]


Otherdocumentsmayinclude:

CloudTechnologies,GregorvonLaszewski[5]ProjectReportFormat,GregorvonLaszewski[6]

3.3.4HELP

Ifyoutakeourclass,pleaseusePiazzatoaskforhelp.Thisisimportantasdifferent Teaching Assistants may answer questions (TAs) based onexpertise.Please,donotsende-mailtotheinstructors.TAsarenotallowedtoanswere-mailsendtothempersonally.OnPiazzaalwaysusethesendtoinstructorsgroup.

3.3.5HOWTOTAKETHISCLASS

This class is attended by studentswith greatly different backgrounds andtimeschedules.Tobemostflexibleandaddressall students thereare twodifferentwaysonhowyoucantakethisclass.

Way 1: Chronological order. The lecture notes are orderedchronological.Thusyoucan followour lecture also in chronologicalorder.

Way2:Freeform.HereyousimplylookattheSyllabustableforthesemesterandidentity,whateversectionyoufeellikereading(whenitbecomesavailable).However,makesureyouconductourweeklyLabactivities.

PleasenotethatwehavesetasideregularLabactivities.Theactivitiesarepass-failandwillbeintegratedintoyourgrade.Youarecertainlyallowedtoworkahead,butpleasebeaware thatbasedon feedbackandobservation,wemaymakemodificationstotheLabs.

Typically,Labactivitiesaresupposedtobecompletedwithinoneweekasitalertsusofproblems,youmighthavethatwecanthenaddress.

Labactivitieswillnotreceiveanycreditifyouarearesidentialstudent,andtheactivityhasnotbeencompletedwithinoneweek.

However, residential students will get twoDelay a Lab for OneWeekpassesthatyoucanapplytoanyoftheLabsandstillgetcredit.

Ifyouareanonlinestudent,werecommendthatyoufinishtheLabsalsoinoneweek.However,youwillgeteightDelayaLabforOneWeekpassesthatyoucanapplytoanyoftheLabs.

Pleasenote that if youwouldneed to postpone a lab for twoweeks, youneedtousetwopasses.Labpassesexpireonemonthbeforethelastdayofclass.Youwillhavetocompleteallthelabsatthattime.Nocreditwillbegiven at that time if this deadline ismissed for anydelayedLabs asTAsmustfocustheirattentiononprojectsupport.

Labpassesdonotapplytootherassignmentsandduedates.

3.3.6ASSIGNMENTS

BesidestheLab’s,wehaveonlyafewassignmentsinthisclass.TheLab’swillprepareyoutowardsachievingtheseassignments.

3.3.6.1TechnologyReview

Students doing projects related to Cloudmesh are exempt fromwriting a technology review, but are expected to do moreprogramming and making sure the project has a manual andproperdocumentation.

As part of cloud engineering, you will be exposed to a large set oftechnologies. To sharpen your skills in analyzing and evaluating thesetechnologies, youwill be asked to prepare a to be determined number oftechnologyreviewsthatarebeingaddedtoaclassproceeding.

This includes writing a substantial non-plagiarized document that can beaddedasachaptertothelecturenotes.Thereviewmustbedoneonatopicthat is not yet included in our book. The review will not includeadvertisement statements form those that have developed the technologybutwillqualitativelydescribe the technologyandpotentiallycontrast it tootherrelatedtechnologies.Inaddition,youwillhavetodevelopanexampleshowcasinghowtousethetechnology.Theminimal lengthofareviewisabout 800words.Otherswill try out your reviewand test if the exampleworks.Yourgradewillbeinfluencedbythereviewofotherstudents.Youwillalsogetagradeforthereviewyouprovidetootherstudents.

Anexampleofsuchasectionisthe

GraphQL

intheCloudComputingbook.

Alternatively,youcanprepareseveraldifferentsmallersections(atleast5)that may not have an example in it but are more of descriptive nature.Samplesectionscontributedbystudentsinclude:

SectionMicrosoftNafikDataCenterintheDatacenterChapterSectionLambdaExpressionsinIntroductiontoPython

It is expected from you that you self identify a sectionyourself, as this shows competence in the area of cloudcomputing.If,however,youdonotknowwhattoselect,youmustattendanonlinehourwithus inwhichwe identifya topicwithyou.Technologies thatarenot repeatabledue toenormouscostorlicensingissuesneedtogetpriorapproval.

This year’s technology reviewswill center around the use ofAI servicesfrommajorcloudproviders.

3.3.6.2Project

The objective of the project is to define a clear problem statement andcreate a framework to address that problem as it relates to multi-cloudcomputingandengineering.

Aprojectistheprimaryactivitythatyouchoseaspartofyourclass.Thisincludesaprojectreportormanualandworkingdocumentedprojectcode.Youwillcreateasignificantnon-trivialprojectrelatedtocloudcomputingand cloud engineering. Up to three students can collaborate. The projectcould be built on top of a previous project but must have significantadditions or modifications. If a previous project is used, a detaileddiscussionistobeheldonwhathasbeenimprovedandisdifferent.

Inthisclass,itisespeciallyimportanttoaddressthereproducibilityofthedeployment.Atestandbenchmarkonmultipleclouds,possiblyincludingadownloadable* dataset, must be used to verify the correctness of yourapproach.

Theprojectmustdemonstrate

1.Theuseofmultiplecloudsorcloudservices.2.Theimplementationofanabstractinterfacetothesecloudsorservices.3.TheimplementationofanOpenAPISpecificationandinterface.4.Theimplementationofaneasydeployment.5.Theimplementationofabenchmark.6.The easy replication of the project from source from your shortdocumentationion(viatheAI’s).

7.TheintegrationoftheprojectintoCloudmeshcommands.Thisissupereasy.

8.AssumeyourprojecthasNmembersthenyouneedtouseandprovideatestinginfrastructurethatworksonN+1clouds.

3.3.6.2.1License

All projects are developed under an open-source license, such as theApache 2.0 License. You are required to add a LICENCE.txt file anddescribehowothersoftwareifused,canbereusedinyourproject.Ifyour

projectusesdifferentlicenses,pleaseaddaREADME.mdfilethatdescribeswhichpackagesareusedandwhatlicensesthesepackageshave.

3.3.6.2.2ProjectReport

Forcertainprojects,therequirementofareportcanbewavedoris significantly reduced while replacing it with moreprogrammingactivities.Thisincludes

AnyprojectthatenhancesCloudmeshBuilding a large cloud cluster with Raspberry Pi’s (whilecontributingamanualorachapteraboutit)AnyApplicationprojectshowcasingNISTbigdatareferencearchitectureuse.

However you still have to do a manual and usage examples,benchmarksandpytestsforthem.

Forallothersthefollowingguidelineexists:

Aproject report is tobedelivered andcontinuously improved throughoutthe semester inGitHub. It includes not just the analysis of a topic, but ashort description of the Architecture and code, with benchmarks anddemonstrated use.Obviously, it is longer than a term paper and includesdescriptionsaboutthereproducibilityoftheapplication.AREADME.mdisprovided thatdescribeshowotherscan reproduceyourproject and run it.Remember that tables andfigures do not count towards the paper length.Thefollowingminimallengthisrequired:

800words,onestudentintheproject1200words,twostudentsintheproject1400words,threestudentsintheproject

Projects with more students are expected to do more programming. ThereportiswritteninmarkdownandcheckedintoGitHub.Thereportwillbemade available in a class proceeding. A report could be substituted by a

manualandbenchmarksuponapproval. In thiscase,aone-pageextendedabstracthastobewritten,whichincludesthelinktothemanual.

3.3.6.2.3ProjectCode

YouareexpectedtodeliveradocumentedandreproduciblecodewithunitteststhatallowsaTAtoreplicatetheprojectwithease.IncaseyouuseVMorcontainerimages,theymustbecreatedfromscratchlocallyandmaynotbe uploaded to services such as DockerHub. You can, however, reuseapprovedvendoruploaded imagessuchas fromubuntuorcentosorotherLinuxdistributions.Allcode,scripts,anddocumentationmustbeuploadedtogithub.comundertheclass-specificGitHubdirectory.

3.3.6.2.4ProjectData

DataistobehostedonIU’sGoogledriveifneeded.Ifyouhavelargerdata,it should be downloaded from the internet. It is your responsibility todevelopadownloadprogram.ThedatamustnotbestoredinGitHub.YouareexpectedtowriteapythonprogramthatdownloadsthedataeitherfromtheWeborIU’sdatastorage.

3.3.6.2.5WorkBreakdown

Onlyneededwhenyouhave2ormoremembersinyourteam.

Thisisaseparatedocumentcalledbreakdown.mdthatisincludedinyourprojectdirectory.Itdescribesinabulletlistwhodidwhatintheproject.Ifyouareateamofone,suchadocumentisnotneeded.Itdoesnotcounttowardsthepage length of the document. It must include explicit URLs to the githistory thatdocuments thestatistics todemonstratemore thanonestudenthas worked on the project. If you can not provide such a statistic or allcheck-inshavebeenmadebya single student, theprojecthas shown thattheyhavenotadequatelyusedgit.Thus,pointswillbedeductedfromtheproject. Furthermore, if we detect that a student has not contributed to aproject, we may invite the student (alone) to an oral presentation of allaspects of the project, including a live demonstration of the examples in

real-time to determine the project grade. So if you work in a teamcontributeanddonothaveyourteammembersdoallyourwork.

3.3.6.2.6Bibliography

All bibliography has to be provided in a BibTex file thatmust either bevalidatedwith jabref orwith emacs. Please be advised doing referencescorrectlytakessometime,soyouwanttodothisearlyandthroughoutthesemester.Whatwouldtakelessthan5minutesaweekcouldquicklyaddupto multiple hours at the end of the semester. Please note that exports ofEndnote or other bibliographymanagement tools do not lead to properlyformattedBibTeXfiles,despite theirclaimsofdoingso.Youwillhave tocleanthemup,andwerecommendtodoittheotherwayaround.Hence,theeasiestway tomanage your bibliography iswith jabref* or emacs.Makesurelabelsonlyincludecharactersfrom[a-zA-Z0-9-].Usedashesandnotunderscore and colons (_ ,:) in the label.Your labelsmust bemeaningfulandunique.Wewill deductpoints if you submit an invalidBibTexfile toGitHub. So pleasemake sure your file is validated. You can even createyourowncheckswithtoolssuchasbiber.

Wewillteachyouwhattodo.Itiseasy.

3.3.6.2.7Reproducibility

Ingeneral,anyprojectmustbedeployablebytheTAs.Ifittakeshourstodeploy your project, please talk to us before the final submission. Thisshouldnotbethecase.Also,ifittakes100stepstodobyhand,wearesureyoucanautomate them,asyouare likelydoingsomethingwrongorhavenot thoughtabout theengineeringaspectofcloudcomputingwhereaswetendtoautomatemostofthesteps.Apartofyourgradewillbehowmuchyourprojectisautomated.

Thebestwaytoassurereproducibilityistousepytest.Wewilldiscusshowtodothatinclass.


3.3.6.2.8ListofDeliverables

You have plenty of time to execute awonderful project, but you need towork consistently on it. Starting one week before the deadline will bedisastrous.

In general, your deliverableswill include the following (Wewill addressandexplaintheminaLab):

Providebenchmarks.Take results in a cloud services and your local PC (ex: ChameleonCloud,echoKubernetes).Makesureyour systemcanbecreatedanddeployedbasedonyourdocumentation.EachteammembermustprovideabenchmarkontheircomputerandacloudIaaS,wherethecloudisdifferentfromeachteammember.We require you towrite one ormore pytest’s that deploys, run, kill,view,clean thatdeploysyourenvironment, runs theapplication,killsit,viewstheresult,andcleansupafterward.For python use a requirements.txt file and develop a setup.py so yourcodecanbeinstalledwithpip install .FordockeruseaDockerfile

Wewillteachyouwhattodo…

3.3.6.2.9ExampleOutlineofaReport

(If not exempt) write a report that typically includes the followingsections:

AbstractIntroductionDesign

ArchitectureImplementation

TechnologiesUsedResults

DeploymentBenchmarks

ApplicationBenchmarks(Limitations)Conclusion(WorkBreakdown)

Your reportwillnot have aFutureWork section as this implies thatyouwilldowork in the future, andyourpaper is incomplete.Hencewewould not grade it. Instead, you can use an optionalLimitationssection.

Do communicate your status and add aWorkbreakdown section inwhichyououtlinewhichtasksneedtobedoneandbywhomincaseofa group project.Once you have done a task simply includemaker ataskasfollows

* [done, Gregor] This was Gregor's task to showcase how to mark it

Incaseyouhaveanexemptionfortheprojectreport,youneedtousesphinxanddocumentyourcodeaspartofamanual.Wewillexplainthedetailsinoneofourlabs.

3.3.7SUBMISSION

All submissions are conducted via GitHub if not otherwise instructed.TechnologyreviewsaretobeaddedtothebookGitHubrepowiththehelpofpullrequests.TheTA’swillworkwithyoutointegratethem.

Asweareworkingcontinuouslythroughoutthesemester,youmustindicateyour activities in aREADME.yml file in yourGitHub repo. TheGitHubRepo,wewilldefineforyouinthefirst2weeksofthesemester.

AnexampleoftheREADME.yamlfileisshownnext

--- owner: firstname: Gregor lastname: von Laszewski hid: fa18-523-00 community: i523 semester: fa18 chapter: - keyword: IoT

YouMUST run yamllint on the README.yml file. YAML errors will cause pointdeductions. Any invalid yaml file will result in point deductions. Pleasekeepyouryamlfilevalidatany time.Ourscriptsdependon it.TheyamlfilewillalsobeusedtocreatealistforTAstoreviewyourdeliverables.Ifitis not in theYAMLfile itwill not be reviewed.Pleasenote that it is notsufficienttojustrunyamllint,buttocompareyouryamlfilecarefullywiththeREADME.ymlexamples.Makesureyoudotheindentationwith2spaces,donotuse the TAB character and make sure you use the list and attributeorganizationwithproperdashplacement.Workwith theTAs if youhavedifficulties.Ifyoucopy,onlycopyfromtherawcontentinGitHub.Ifyouworkonmore


3.3.8BONUSPROJECTS

This class will not have any bonus projects, as all additional activitiesshouldbeputinyourprojectorchapter/reviewcontribution.However,wewillrecognizeextraordinaryeffortsintheseactivities.

3.3.9PARTICIPATION

Inadditiontotheseartifacts,therewillalsobeaparticipationcomponentinclass that will be determined based on your productive contributions toPiazzatohelpothersthathavequestionsandcontributionstothebookstoo,for example, improve sectionswith spelling,grammaror content.Wecansee from theGitHub history if you conducted such improvements.Makesurethattechnicalcontributions,workonallOSesandarenotjusttargetingasingleOSiftheimprovementisofgeneralnature(exceptionsapply).

title: Role of Big Data in IoT url: https://github.com/cloudmesh-community/fa18-523-00/blob/master/chapter1/paper.md group: fa18-523-00 fa18-523-01 - keyword: Datacenter title: Green IT data centeres in the US url: https://github.com/cloudmesh-community/fa18-523-00/blob/master/chapter2/paper.md group: fa18-523-00 project: - keyword: Cloud Cluster title: Raspberry PI Cloud Cluster url: https://github.com/cloudmesh-community/fa18-523-00/tree/master/project-report group: fa18-523-00 fa18-523-01 code: TBD

3.3.10GITHUBANDPYCHARMMARKDOWN

PleasebeawarethattheePubsusemarkdownasdefinedinpandoc.NeitherGitHubnorPycharmsupport thisversionofMarkdown. Instead, theyusesomeenhancements that arenot supported.Thusdonot assume thatyourGitHubwill render inourproceedings.Weuse themostprimitiveversionof markdown, but use the enhancement for citations and references inciteproc and citeref.We have a docker image that allows you to run theversionofmarkdownonyourreportsanddocuments.

TheTA’swillprovideyouwithsupportonhowtodothis.Itistrivial.

4WEEKLYAGENDA

4.1WEEK1:COURSEINTRODUCTION☁

Weprovideinthisdocumentalltheactivitieswedoineachweek.

Sections marked with a� are not yet released and are likely to bemodified.

4.1.1ASSIGNMENTS

Allgradedclassassignmentsarepostedat

https://iu.instructure.com/courses/1858708/assignments

4.1.2CLASSVIDEOS

Pleasewatchthefollowingvideo

OverviewEngineeringCloudComputing(21:05)

Next,reviewthefollowingchaptersinthebook

CloudComputing,GregorvonLaszewski[1]:

Chapter:Preface,ePubReadersChapter:OverviewChapter:DefinitionofCloudComputing(anditsvideos)

4.1.3LABACTIVITIES

Summary:

[]Createanaccountonhttps:github.com

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week1-intro.md

https://iu.instructure.com/courses/1858708/assignments

https://www.youtube.com/watch?v=PU93waozGJk


https://github.com/

[]FillouttheClassSurvey[]PostyourprofessionalbiotoPiazza

[]Setupyourcomputer

[]Installpythononyourcomputer(Python3.8)[]Updatepip

InstallanePubReader

macOS:Build-inuseBooksWindows10:Calibreothers:Calibre

Ifyouhavequestions,askonPiazzaNext,weprovidesomemoredetailsofeachoftheassignmentsinthissummary.

4.1.3.1AccountCreation

Aspartoftheclass,youwillneedanumberofaccounts.Thisincludes:

piazza.com(usedforcommunication)github.com(usedfortheprojectandotherclassartifacts)chameleoncloud.org (free cloud account)https://www.chameleoncloud.org/user/register/

Optionalaccounts(applyonceyouneedthem,somearetimelimited):

google.com(optional)aws.com(optional)azure.com(optional)WatsonfromIBM(optional)googleIaas(optional)

Wewillneveraskyouforyourpasswords.

4.1.3.2Survey

https://forms.gle/4Zpi2vzpZ5kFSoD1A



https://piazza.com/class/jzkfveoqwri3e4

https://www.chameleoncloud.org/user/register/

Inthisactivity,youwillbefillingoutaformwithinformationaboutyousowecanassesshowtointegrateyouinthisclassbest.Itisnotimportantthatyouknowanyofthetechnologiesweaskyouabout.

ClassSurvey

4.1.3.3Bio

Thisactivityservestwopurposes.First,ittellsuswecancommunicatewithyou within Piazza, and second, you can introduce yourself to others inPiazzatopotentiallybuildprojectorstudyteams.

4.1.3.4Python3.8

Please set up a computer onwhich you can do Python development.Werecommendthatyouusepythonfromhttp://python.org.Wewillnotprovideanysupportforanaconda,asanacondahashundredsoflibrariesthatwearenot interested in using. Also note that conda modifies your environmentwithouttellingyou.Certainly,youcanusevirtualbox,orcontainersincaseyouliketoisolateyourdevelopmentenvironmentfromyourothersystems.Please remember that conda has significant disadvantages while oftenworkingwithoutdated libraries.As thedeveloper of future software, youcertainlywanttoavoidthis.Ifyouusepythonfrompython.orgwerequireyouthatyouusevenv.

Themanualmayhavereferencestoolderversionsofpython.Alsoitcouldbethatsomeservicesdonotworkinnewerpythonversionsyet.Itwillbeyourresponsibilitytofindthatoutandmakessuretheservicesyouusehavean appropriate python version. We will not accept any project targetingPython2orsmallerthanPython3.7.4.Youmustusepipversion19.3.1ornewer. As it may be that newer versions of Python come out during thesemester,welikethatyouinstallthenewestversion.Attimethisnotewaswrittenthisis3.8.1.Ifanewerversioncomesout,pleaseinstallthis.

Itmaybethatsomecloudservicesmaynotworkwiththatversion.Inthatcaseitisyourresponsibilitytouseorinstallinparallelaversionofpython



http://python.org/

thatworks.

4.1.4ONLINEMEETINGRECORDING

OnlineMeetingRecordingforWeek12020(1:29:39)

AcceptingtheGitHubinvitationifyoucannotfinditine-mail,andcreationofGitHubissuesToDolists(1:29:39)

Recodrdingsfrompastyears


4.2WEEK2:CLOUDDATACENTERS☁

4.2.1LECTUREMATERIAL

Anewversionofthefollowingbookshavebeenreleased:

e516 Lecture Notes Engineering Cloud Computing, Gregor vonLaszewski[7]:CloudComputing,GregorvonLaszewski[1]:

Please read the section inCloudComputing[1]: aboutDataCenters.Wedecidednottodoavideoasthematerialinthissectionfrequentlychanges,andvideosbecametooquicklyoutdated.

Dothefollowingassignments:

Due: twoweeks,shareyourfindingsonPIazzaand improvebasedon thediscussionsandother students’ contributions.We recommendyou start inthefirstweek.Gregorwillthanintegratethecontributionsintoadocumentthatwecanthenuseinthesecondweekforimprovements.Pleasealsonotethat this assignment is used for you to learn markdown with references,

https://www.youtube.com/watch?v=m-Zgg2iTlkQ

https://www.youtube.com/watch?v=HIM7d8TFZRs

https://www.youtube.com/watch?v=k5218WUo6Cc

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week2-center.md




links,andBibTeXentries.NotethatGitHubdoesnotsupportthisversionofmarkdown.

E.Datacenter.2.aE.Datacenter.2.bE.Datacenter.3E.Datacenter.4E.Datacenter.5E.Datacenter.8

Optional:

E.Datacenter.9

Optional�

WemaypostonPiazzaexamplesfromlastsemesterthatyoucanleverageandimprove.Remember, theymayalreadybeoutdatedbynowandcouldneedupdates. Itwill takeussometimetopublish them.Butyoucanfindthemalreadyyourselfifyoulookinpreviousstudents’repositories.

4.2.2LABACTIVITIES

4.2.2.1ReceiveyourHID

Due:oneweek(needssurveytobefilledout)

MakesureyouhavereceivedanHIDonGitHub.Lookitupat

https://github.com/cloudmesh-community

MakesuretoaccepttheGitHubinvitationandtrytoaddafile.YoucanuseeithertheGUIwaywithCreatenewFileorifyouarefamiliarwithGitHubusethecommandlinetools.NextweekwewillintroduceyoutoconvenienttoolssoyoucandevelopyourprogramsmoreeasilyinGitHub.


4.2.2.2README.yml

Due:oneweek

Make sure the information in yourREADME.yamlfile is accurate.Makesure to change the value in the community attribute and use your classnumber.Thiswillbeeithere516orb649orsimilar.Pleasenotewhenweaskyou foryourhidnumber, it is the entirehidnumber, not just the lastthreedigits.

4.2.2.3ChameleonCloud

Due:Now

Makesureyouhavecompletedtheapplicationforthechameleoncloudaswewillusethatinourlabsoon.

4.2.2.4ReviewPython

Due:throughoutthesemester

Thisactivitywillgoonthroughouttheentiresemester.YoudonothavetobeaPythonexpertwithinoneday,butifyouhaveknowledgegaps,pleasereviewandconsultourPythonBook.Startwiththelanguagefeatures.

PythonforCloudComputing,GregorvonLaszewski[2]:

WetypicallyusePython3.8;thebookmaybeabitoutdatedinthataspect.

4.2.2.5SetupyourComputer

4.2.2.5.1Python

--- owner: firstname: Gregor lastname: von Laszewski hid: hid-000 community: e516 semester: sp20


Due:OneWeek

Make sure you have Python set up. Read up on virtualized pythonenvironmentsasusedinPython3.8.Itssupersimple.Understandwhatthecommand

ThoseusingWindows,pleasefindouthowtodoitonWindows,provideadocumentvenv-windows.mdinyourHIDdirectorydescribingit.

Those using conda/anaconda, please find out how to do it withconda/anaconda,provideadocumentvenv-conda.mdinyourHIDdirectorydescribingit.

Make sureyou create a separatevenv for this class calledENV3 thatwewilluseforthisclassandshouldnotbeusedforotherclassesaswedonotwant to create side effects. Please also be aware that at times it may benecessary to delete your python environment in case you do somethingwronganditwouldbeunwisetocombineallyourpythonactivitiesintoonepythoninstall.

Asyoucansee,thisissupereasy.

We recommend PyCharm for an IDE if you have not yet picked one.PyCharm automated code improvements will be used for most of ourprojects. It really helps you!We will teach you throughout the semesterhowtouseit.

4.2.2.6UbuntuMultipass

Due:oneweek

LocatetheSectionaboutMultipassinthebook

CloudComputing,GregorvonLaszewski[1]:

$ cd ~ $ python -m venv ENV3 $ source ENV3/bin/activate


andreadit.

Installubuntumultipassonyourcomputer.Notethatyoumusthaveatleast8GBofmainmemoryonyourcomputer.However,youmayneedtoclosechrome or PyCharm. Please monitor your consumption of memory todetermineifyouliketobuymorememoryforyourcomputerandupgradeit.Acomputerwith4GBislikelynottobeusableforthisassignment.

Incaseyoudonothaveacomputeronwhichyoucanexecutethis,pleaseconsult with us in Piazza. You may have to do more work than otherstudents.

Forexample(a)youmayhavetoimmediatelystartusingchameleoncloudand learnOpenStack as an alternative; (b) youmayneed to use a virtualbox;(c)oryoumayhavetobuyyourselfacheapuptodatecomputeronwhich you can execute the class. You must be able to besuperuser/administrotor of that computer. The computer must supportvirtualmachinesandDocker.

Pleasealsonote that ifyouhaveWindowsHome10onyoursystem,youneedtoupgradeittoProorEDU.Makesuretoconfigureitas64bit.

Itisinyourresponsibilitytohavetheproperequipment.

Analogy:Whenyouplaytennis,yougetyourselfatennisracket,andyoudonotshowupwithapingpongpaddletoplayit.

DoAssignments:

E.Multipass.1E.Multipass.5

4.2.2.7Optional:PlagiarismCertificate

Due:optional,butifyoudonotunderstandwhatplagiarismis,pleasefindout.

AtIUyouarerequired toknowtheseconceptsbeforeyoutakeanyclass.Thustheexcuse:“Ididnotknowthatthisisplagiarism”doesnotapplytoanyclassatIUincludingthisone.

Whenwritingcontributionsthatcanbeintegratedintotheclassmaterial,itisimportantthatitisnotplagiarized.Itismostimportantthatyounotjustcopy content from Web pages, but make appropriate modifications andprovidecredittowhereyoufoundthisinformation.

As you certainlywill have learned from other classeswhat plagiarism isand how not to plagiarize, youwill probably be fine.However,we oftenfindoneortwostudentsinaclassthatdoesnotknowwhatitis.ThereforeweSTRONGLYRECOMMEND thatyou take theplagiarismcertificateofferedbyIU.Thishastheadvantagethatyoucanshowitinotherclassesandshowyouare informed.Pleasealso read thecodeofconduct rulesatIU.

Forthisreason,weincludedthismaterialalsoinpartintheBook

ScientificWritingwithMarkdown

PleasereadthechapteraboutPlagiarism.

Pleasebeawarethatwemayconductsuperiorplagiarismtests.Ourtestsareevenbetter thanTurn-it-inwhileweareable to identify translations fromotherlanguages,copiesoftextfromnon-publicexternaluniversityservers,andcompanyarchivesthatturn-it-indoesnotcheck.IUhasastrictpolicythatwemustfollow.Plagiarism/cheatingcould lead toexpulsionfromtheuniversity.TheargumentIdidnotknowwhatplagiarismisdoesnotcountaccordingtoIUrules.YoumustknowwhatitispriortoyoutakingacourseatIU.


Itisnotthesubjectofthisclasstoteachyouwhatplagiarismis.Wedothisjusttoremindyoutoavoidanyuncertaintyaboutit.


Meeting a: markdown, github, pycharm and markdown (25:27)https://youtu.be/uGTlDwff41cMeeting b: multipass, pycharm, questions (1:31:48)https://youtu.be/30CdNYN7gHs

Recodrdingsfrompastyears

TheonlinemeetingrecordingfromTueSep3,2019isavailablefromthislink:

OnlineMeetingRecordingforWeek2(48:38)

4.3WEEK3:CLOUDCOMPUTINGARCHITECTURES☁

Goals:

StartofChapterSelectionStartofProjectSelectionReviewPythonUsecloudmesh-installerGenerateacommandwithcmssys



e516 Lecture Notes Engineering Cloud Computing, Gregor vonLaszewski[7]CloudComputing,GregorvonLaszewski[1]

https://youtu.be/uGTlDwff41c

https://youtu.be/30CdNYN7gHs

https://www.youtube.com/watch?v=kezPx0QHrt8

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week3-arch.md



Introduction toPython forCloudComputing,Gregor vonLaszewski[2]

ReadingAssignments:

1.Read in the book Cloud Computing [1] the chapter about CloudArchitectures.

2.ReadinthebookIntroductiontoPythonforCloudComputing[2] thechapteraboutthePythonlangugae

3.Focusonrefreshingyourpythonknowledgeabout lists,set,dict,andclasses.

4.Deadlines in the e516 Lecture Notes Engineering Cloud Computing[7]wereupdated.

4.3.2LABACTIVITIES

Youshouldhavesetupinthefirstweekoftheclasssetuppython3.8onyourcomputer.Ifyouhavenotyetdoneso,pleasedo.Werecommendthatyouusepythonfrompython.org.

Wefoundoutoverthelasttwoweeksthatsomestudentsrelyonanacondaforotherclassesbutdonotknowhowtouseit inanacondaavirtualenv.Pleasefindout.

Ifyoucannot,wefounditmaybeeasiertocreateforthisclassadifferentuserasyourotherteachersmayhavegivenyouwronginstructionsonhowto installpythonor anaconda.Toavoid this,youmay just start fresh inanew user and do this class in that user. Make sure the user hasadministrativeprivileges.

E.g.,Ifyouusecondaoranacondait isuptoyoutofigureouthowtodothis.Youcanpostguidestothatinpiazza,andwewilltrytoincludetheminfuturelecturenotes.

Whateveryoudo,youmustuseapythonvirtualenvironment.





4.3.2.1Review:venvinpython3(Graded)

Tocreateavenvpleaseusethecommand

Pleaseunderstandthefollowingconcepts(nosubmissionneeded):

HowdoyouactivatethevirtualenvinyourOS?Howdoyoumodifyyour .bashrcfileso that thepythonvenvis loadedautomaticallyuponthestartofanewterminal?In case you use zsh, either switch to bash or describe how do youmodifyzshsothatthepythonvenvisloaded.Whydo you need to use venv for this class?Provide a one-paragraphanswerinyournotebook.mdfile.

4.3.2.2Review:Anaconda(Graded)

Due:beforethesemesterbreak

This labonlyhas tobedoneby thoseusinganaconda/conda.Thoseusingpythonfrompython.orgdonothavetodothisassignment.

Pleaseprovideyouranswersinthenotebook.mdfile.

What problemsmay you encounter when using anaconda as pythondeveloper?

Letusassumeyouuseanacondaonyourvirtualmachinesinthecloud.Let us assume you start 1000 vms all using anaconda.What is theoverheadinwastedspaceonthesemachinesifyoujustwantedtouseassimpleregularpythonprogramontheseVMs?

Howdoyoufindouthowmuchspaceisusedbyyourprogramanditslibraries?

Howdoyouswitchbetweenanacondaandregularpython3.8.1

$ python -m venv ~/ENV3

Whatisthedifferencebetweenconda,miniconda,anaconda?

Doesanacondaprovideavirtualenvironment?

Why do you want to use a virtual environment even forconda/anaconda?

Isanacondamodifyingyourbashrc,zshprofile,orregistry?

Showcasehowtoinstallcloudmeshunderandaconda.

4.3.2.3PythonLanguagereview

4.3.2.3.1Dicts

Weliketoremindyouaboutdictsinpython.

Readupondictsandexperimentwiththem.

Howdoyoumergethecontentoftwodicts?

4.3.2.3.2f-StringsinPython3

Python3providessomeverynicewayofusingvariablenamesaspartofstringmanipulations.

Locate the PEP 498 and study it:https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep498

Tryoutthefollowing

test = "Gregor" msg = f"This is a test {test}" print (msg)

def f(test): msg = "This is a test {test}".format(**locals()) print (msg)

from cloudmesh.common.debug import VERBOSE d = {"test": "Gregor"} VERBOSE(d)

https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep498

Inoneoftheexampleslocals()isused.

Whatdoeslocals()do?Whatdoes**intheformatstatementdo?

4.3.2.3.3PythonClasses

Thiscanbecompletedatalatertimethroughouttheclass:

Whatisselfinclasses?Whydoesselfneedtobeusedinregularmethoddefinitionsinclasses?WhatcanIdowithinitandwhyisitused?Whatisclsand@classmethods?Whywouldoneuse@staticmethod?Write a Provider.py class to interfacewithmultipass. If you can forsomereasonnotusemultipass,emulateitsbehaviorwhileusingprintstatements.

4.3.2.3.4PythonModules

Thiscanbecompletedatalatertimethroughouttheclass:

Whatisasetup.pyfile(usegoogle)Whatisthedifferencebetweenpip install .andpip install -e .HowdoIuninstallapythonmodule?Mypythonvirtualenvironmentisbroken,Whatdoyoudonow?Use the command cms sys generate comamnd and create a new command.Inspect thecode that isgenerated.Takea lookat setup.pyWecreatethisallforyouautomatically;-)

WewillbediscussingthesequestionsintheLabs(online/andresidential).

4.3.2.4GradedActivities

InthebookIntroductiontoPythonforCloudComputing[2]pleasedothefollowingassignments


E.Cloudmesh.Common.1E.Cloudmesh.Common.2E.Cloudmesh.Common.3E.Cloudmesh.Common.4E.Cloudmesh.Common.5E.Cloudmesh.Shell.1E.Cloudmesh.Shell.2E.Cloudmesh.Shell.3

4.3.3CHAPTERSELECTION

As part of this class you will be conducting a technology review whilewriting a chapter that looks at a topic related to cloud computing. ThechapteriswritteninmarkdownandplacedinyourHIDdirectoryat

Pleasenotethatallcharactersofthefilenamearelowercase.

This year’s topic focusses mostly aroundCloud AI Services but we alsoaccept other topics that you may find interesting and are not covered insubstantiallengthinanyofourbooks.

We have posted here some initial ideas for chapter contributions.Remember that you must not write an introductory paragraph to yourcontributionsastheyaremostlikelyalreadycoveredinthebooks.Instead,youmustonlyfocusonthetopicathand.

Sopleasevisitthelinkandselectatopicorsuggestyourown

https://docs.google.com/spreadsheets/d/1QxlFCSQI66-zR9H6uI9-N5F99fi23o3enrU51QJdQ7c/edit?usp=sharing

Pleaseonlyfilloutthewhitelinesandleavetheyellowonceuntouched.Iftherearemore than twopeople inacategory,pleasecoordinatewitheachotherhowtowriteitwhileonlyprovidingasingledocument.Dependingonwhatthetopicisabout,youmayhavetoexpandit.Pleasenotethatthisisa

chapter/report.md

https://docs.google.com/spreadsheets/d/1QxlFCSQI66-zR9H6uI9-N5F99fi23o3enrU51QJdQ7c/edit?usp=sharing

multipage activity and not a one-sentence activity. References must beincludedasproperreferencesattheend;wewillteachyouintimeonhowtodothat.Examplesaregiveninthemarkdownbook.

Allchaptersneedtoinclude

a writeup that introduces the technology without plagiarism andadvertisementclaimsbythedevelopersofthattechnology.anexampleonhowtouseitthatcanbereplicatedandyouhavetriedyourself,ifpossible.referencesinBibTeXformat

Duedates:

Feb3rd:LastdatetoproposeachapterFeb10th:LastdatetosubmitthefirstdraftFeb 17:Last date to return a peer review (you get a grade from theauthorofthechapter,thereviewerwillalsocheckyourexamples)Feb24:Lastdatethehandinthefinalversionofthechapter

4.3.4PROJECTSELECTION

Deadline:Feb16lastdaytohandinyourprojectproposal

Youwillbeselectingacloud-relatedprojectoverthenext2weeksthatyouwillbedevelopinguntiltheendofthesemester.Theprojectmusthavethefollowingrequirements:

Programming must be done using python if another programminglanguage isused,pleasecontactusand justify theuse.Youwillalsohavetodopartofyourprogramming,to,forexample,coordinatedthedeployment or the benchmark likely in python. We also acceptJavaScriptifyouliketodevelopaGUIforCloudmesh.Wealsohavesomestartusingelectron.

TheprojectmustuseOpenAPI3.0todefineaRESTservice.Wewillteachyouhowtodothatinfutureactivities.

The project must use conexion to automatically generate the restservice(pleasedonotuseswaggercodegen).Wewillteachyouhowtodothatinafutureactivity.

The project must use at least 1 + n clouds for each team member.where n is the number of team members with maximum 3 teammembers.Wewillteachyouhowtodothatinafutureactivity.

The code must use pytests.We will teach you how to do that in afutureactivity.

Thecodemustusecms sys generate.Thisisoneofthelabassignmentsforthisweek.

Thecodemustusethecloudmesh benchmark/stopwatchclass.Thisisoneofthelabassignmentsforthisweek.

If anAI service isusedwe recommend touse scikit-learnorKearosandcontrast itwithAIservicesofferedbycloudproviders.Thiswillbediscussedinupcominglectures.

Youmustprovideareportwithmeaningfulbenchmarks.Thereportiswritten inmarkdown.Wewill discusswith you how to do that in afuturelecture.

Please note that the above requirements also hold true if you usetechnologies such as Hadoop, Spark, Kubernetes, AWS, Azure, Google,OpenStack,…

Certainprojectswillhavecustomdeliverablesthatwewillrefinewithyouonceyouhavechosensuchaproject.Forexample,ifyouweretochoseacloudmesh project you will be asked to develop a manual instead of areport.However,youstillhavetodothebenchmarks.

Apreliminarylistofprojectsisavailableat.

https://cloudmesh.github.io/cloudmesh-manual/projects/

https://cloudmesh.github.io/cloudmesh-manual/projects/

OthersarealsopostedinPiazza.Wewilladdadditionalprojectideasoncetheybecomeavailable.PleasenotethatweconsidertheCloudmeshrelatedprojects easy as we introduce you gradually in all aspects as part of theclasstodeliverasuccessfulproject.

ThosewantingtochosetheRaspberryPIClusterortheRobotBoat,pleasecontact the instructors via piazza.Wewant to set up ameeting to betterdiscussthisproject.

Whenitcomestothescopeoftheproject,rememberaprojecttakesupto12weekstobecompleted.Itisnotallowedtojustsearchongithuboranotherbookandreplicateaprojectdonebysomeoneelse.Yourprojectmusthavea novel component. Please note this class is called Engineering CloudComputingwemustseeclearlyanaspectthatyouengineerthecloud,e.g.,the setup of a reproducible cloud environment is mandatory. Please askquestions and understand this.While you are able to use all services, allimagesmust be created from scratch, and wemust be able to reproducethem.Wewilldeclineallprojectsthatpointustoimagesthatyouaskustodownload and uploaded by you on github, dockerhub or similar. Instead,you must provide us with scripts, dockerfiles, makefiles or similar thatcreatetheimages.Youareallowedtouseimageshostedbymajorvendorssuchasanubuntu19.04 imageoronchameleoncloud the images startingwithcc-*andsoon…

4.3.5WORKINGAHEAD

Obviously, we will be introducing you to somemore advanced conceptsthat are not yet finalized in the books. You can certainly use documentsfromtheinternettolearnaboutsuchconcepts.

TheconceptswewillneedarelistedintheSyllabus.

Onthepythonside,wewillintroduceyouto

pytestGitHubAPIasanexampleforaRESTservice

AzurepythonAPIAWSbotoOpenstackSDKAPIGooglecloudAPI…

OntheCloudmeshsidewewillintroduceyousoonto

cloudmesh.yamlaspreliminarydocumentedinthecloudmeshmanualcloudmeshcloudbundlecloudmeshstoragebundle


Labmeeting:

1. Online Meeting Recording discussing Projects, multipass,cloudmesh(2:16:40)

2. Online Meeting Recording discussing Projects, Twillo projectdiscussion,bibtexjabref,cloudmesh(1:05:40)

Ihaveonlywrittendownsomeofthetopicsifyoufindadditionaltopicswetalkedabout,pleasecorrectthemviaapullrequestingithub.

FYI.Twillohasnotbeenchosenbythestudent.

Thisisarecordingfromlastyear.ThismeetingtookplaceSep10,2019,8-9pmEST


4.4WEEK4:OPENSTACK☁

https://www.youtube.com/watch?v=I-k8QuuFfJY

https://www.youtube.com/watch?v=Vm65mP8st9I

https://www.youtube.com/watch?v=DD5rBVk3kZw

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week4-openstack.md

PleasenotethatthelinktoOpenstackhaschanged,thisisnotreflectedinthevideos.Pleaseusethenewlink

https://kvm.tacc.chameleoncloud.org/project/

Pleasenotethatwesimplifiedthecloudmesh-instalerasmanystudentsforgot to specify the -eoption.Wehave simplyommitted the -e from thenewcloudmesh-installer

Makesuretoalwaysupdatethingswith

Wherebundleisthenameofthebundleyouliketoupdate

4.4.1VIDEO

Incaseyouhavenotyetcreatedansshkey,thefollowingvideoisuseful.ThiscanbereplicatedonanyLinux,macOSandWindows10machine,incaseofWindows10usegitbash

SSHkeygen(4:07)

AVideobrieflysummerizinganintroductiontocloudmeshformulticloudenvironments including pointedts to projects related to virtual directories,computeservices,andvirtualclusters

CloudmeshVersion4(44:01)

A video on how to start an login into a virtual machine on Horizon ispresentednext:

OpenStackHorizon(10:49)

cd cm pip install cloudmesh-installer -U cloudmesh-installer pull BUNDLE

https://kvm.tacc.chameleoncloud.org/project/

https://www.youtube.com/watch?v=pQb2VV1zNIc

https://www.youtube.com/watch?v=VhgGIE2lTpQ

https://www.youtube.com/watch?v=ofAVUX1gvUg

Avideohowtostartvmswithcloudmeshisavailablehere:

Cloudmeshcmsvmboot(15:07)

A number of videos on explaining the internals of cloudmesh is avalablehere:

Thefirstvideo introducesyou tocloudmeshcommona library toexecutecommandsonyourosandinteractwithit

cloudmesh-common(13:40)

Thesecondvideoexplainsdetailsaboutcloudmeshcommputeandalsotellsyou aboutwhat is expected for theproject to bedeveloped.However thevideodoesnotyetexplainhowtodoavirtualcluster fromtheProviders,whichisonegoalforthissemester.

cloudmesh-cloud(15:09)

The third video tells you about the cloudmesh-storage project that issupposedtobedonintheproject,thisincludes

bettertestscompletionofadditionalprovidersusingofProcesspoolandmongodbformanagingfilestobecopied

the creation of aREST service that includes the pool to allow copyfromonecloudtoanother.

cloudmesh-storage(8:03)


ABookforChameleonCloudisavailable

https://www.youtube.com/watch?v=GgNUXYaB7kQ

https://www.youtube.com/watch?v=o9AcgNheVls

https://www.youtube.com/watch?v=sghmbRldyKQ

https://www.youtube.com/watch?v=eKPMD7qDXSc

https://laszewski.github.io/book/chameleon/


e516 Lecture Notes Engineering Cloud Computing, Gregor vonLaszewski[7]CloudComputing,GregorvonLaszewski[1]Introduction toPython forCloudComputing,Gregor vonLaszewski[2]

4.4.3LAB:OPENSTACK

This week we will be learning how to manage virtual machines onOpenStack.YouarerequestedtoexploretheGUIinterfacewhichiscalledhorizon so you can. verify your activities easily in case you have issueswiththecommandlinetools.However,ourmaingoalwillbethatyouusecommandlinetoolstointeractwithChameleoncloud.

Whateveryoudoafteryouaredonewith theVMsyouneed to terminatethemsoyoudonotunnecessarilywastecomputetime.

4.4.4NAMINGOFVMS

YourHIDisoftheformfa16-516-NNNorsimilarforotherclasses.Pleasenotethatthenumberisuniqueacrossclasses.Thisidentifiesyouandifyoustart a vm in a shared space such as chameleonwe can use it to identifypeopleandnotifythemeasily.

Thuspleasusethefollowingnamingscheme

NNN-firstname-i

wheretopicisatopicforthevmsuchaswebserverandiisanumbersuchas1asyoumaystartmultiplevms.

Pleaseneverstartmorethan3vmswithoutconsultationwithGregoraswewillrunoutofnodehoursfortheclassifwedoso.




4.4.5HORIZON

TheinformationonhowtouseHorizonisavailableintheChameleonbook.

AvideoofthemeetingonTuesday17Sepwillbemadeavailable

4.4.6CLOUDMESHOPENSTACKINTERFACE

TheinformationonhowtouseitisavailableintheChameleonbook.

AvideoofthemeetingonTuesday17Sepwillbemadeavailable

4.4.7OPENSTACKCOMMANDLINECLIENT

TheinformationonhowtouseitisavailableintheChameleonbook.

4.4.7.1InstallationofCloudmeshCloudBundle

⚠ Do these only after you have completed the cloudmesh shell relatedassignmentfromlastweek.

Compute:

The next is optional and only for those that chose a cloudmesh storagerelatedproject.Theinstallationofthestoragebundleincludestheinstallationofthecomputebundle.

Toseetheavailablecommandstype

4.4.7.2SSH

$ cd cm $ cloudmesh-installer git clone cloud $ cloudmesh-installer install cloud

$ cd cm $ cloudmesh-installer git clone storage $ cloudmesh-installer install storage

$ cms help

Makesureyouhaveapasswordprotectedsshkey

4.4.7.3Configuration

1.Change the username and password for the chameleon cloud in~/.cloudmesh/cloudmesh.yaml

2.Changethepasswordinthemongodbsection

3.Changetheinformationintheprofilesection

4.4.7.4CloudmeshMongo

Setting up cloudmeshMongo is discussed in the cloudmeshmanual.Wesuggestyoudotheonediscussedinthedistributionsectionforyoursystem.It is actually built into cloudmesh, but before you do it, we suggest youbackupyormachine.

Onceyousetitupusecms initwhichwipesthedbandshouldonlybeusedonce

afterthatalwaysuse

4.4.7.5StartaVM

OpenStack could be over-utilized and that a VMmay not start before atimeout.Gregorobserved70%successrate.

Task:wheneveryoustartavm,pleasekeeparecordifitstartedornot

$ ssh-keygen

$ cms admin mongo install

$ cms init

$ cms start

$ cms stop

dothisinafileonyourGitHubcalledchameleon-success.md

make sure the failures are not recorded based on programming errors orwrongparameters.

Usethecommands

⚠switchondebuggingandtrace

⚠Explainthedifferencebetween--refreshandnotusingit.

4.4.8WORKINGWITHTHEVM

Logintothevmwith

4.4.9EXERCISE

1.StartavmwithHorizon,login,andterminateit2.StartavmwithCloudmesh,login,andterminateit3.Use robo3T or a similar program to brows in the CloudmeshMongoDB

4.5WEEK5:CLOUDVMCOMPUTESERVICE☁

4.5.1VIDEOS

Watchthevideosfromweek4relatedtoCloudmesh


success: 7 failure: 3

cms set cloud=chameleon cms image list --refresh cms flavor list --refresh cms vm boot cms image list --refresh

cms ssh

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week5-compute.md



PleasereadthechapterinthebookCloudComputing[1]about

HypervisorIaaS,pickonecloudyouliketogetfamiliarewithandfocusonthat

1.Donotreadthesectionsmarkedwith

Construction

2.DonotusePythonLibCloudwewantyoutofindthevendorspythonlibraries whcih are typically superior and have better access to theinformation related to IaaS for the provider. This will make yourprojectsmucheasierinthelongrun.

4.5.3LAB:VMCOMPUTESERVICES

4.5.4EXERCISEMULTIPASS

Multipass.1:Showyouhaveinstalledcloudmesh-multipasbyrunningthecommandcms multipass images

Multipass.2: Pick one method that is not yet implemented incloudmesh-multipass.Verify that it does not yet exist, and if its not,createapullrequest.

4.5.5EXERCISECLOUDAPI

Thisassignmentmaytake2weekstocompletedependentonexperienceofstudnetswithPython.ChoseanAPIrelatedtooneofthecloudsyouchose




for your project. Please remember that we do have cloudmesh and asignifican amount of Providers exist. You are allowed to reuse thecloudmeshproviders.

CloudAPI.1:FindacloudyoulikeandidentifythenativePythonAPI(butnotlibcloud,incaseofopenstackusetheopenstacksdkwhichisnewandnotnovaandtheotherapi’s).

CloudAPI.2:DemonstrateaPythonprogramtolistimages,flavorsandvirtual machines. You can create a vm via the GUI on your cloud,nameitwithyourfirstname.

CloudAPI.3: When it comes to credential management, please usecloudmesh.yaml and Config(). Learn how to do that. Under NOCIRCUMSTANCESpostyourpasswordsor addfiles toGitHub thatincludeyourpasswordsorothercredentials.

CloudAPI.4: Develop a python program that adds keys and securitygroups(difficult).

CloudAPI.5:Showcasethatyoucansshintothevm(difficult)

Thisexercisecanbedoneopenlyinclassandeveryonecansharecodewitheveryone,aslongasyouacknowledgethestudentwithnameandhid.

4.6WEEK6:REST☁



e516 Lecture Notes Engineering Cloud Computing, Gregor vonLaszewski[7]CloudComputing,GregorvonLaszewski[1]ChameleonCloud

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week6-REST.md



https://laszewski.github.io/book/chameleon/

Introduction toPython forCloudComputing,Gregor vonLaszewski[2]

GothroughChapterRESToftheCloudComputing[1]book.Thischapterincludes,

AnintroductiontoRESTAnoverviewofframeworksusingRESTAnoverviewofOpenAPIandhowitrelatestoRESTAnintroductionofusingGitHubwhichisalsoaRESTservicefromaPythonAPI.

ThechapteralsincludesasectiononOpenAPI2andEvewhichishoweverforthisclassnotrelevant.

4.6.2LABACTIVITIES

Implement the REST service explained in the Section OpenAPI RESTServiceviaIntrospectioninyourlocalmachine

Completetheexercises:

OpenAPI.Conexion.1OpenAPI.Conexion.2OpenAPI.Conexion.3OpenAPI.Conexion.4OpenAPI.Conexion.5

Completetheexercises:

E.OpenAPI.1E.OpenAPI.2E.OpenAPI.3E.OpenAPI.4E.OpenAPI.5

4.6.3GRADEDLABACTIVITY



InthislabactivityyouwillbeaddingmorefunctionalitytotheRESTcpuexampleanddeployingtheserviceintheChameleonCloud.

Youmayneedtoinstallpy-cpuinfolibrarytohelpyoucollectinformationfortheserviceimplementation.

1.ChangethecpuGETmethod,toworkinaoperatingsysteminvariantway(i.e. usepythonlibrariestodeterminetheCPUname,ratherthansystemcalls)

2.Playaroundwithpy-cpuinfolibraryinUbuntuenvironmentjusttogetanideaabouttheinformationyoucouldretrieve.IsuggestyouspawnanUbuntu18.04containeronyourlocalmachineforthis.

3.Add a GET method to get cache size of the CPU. URL parameter{level}shouldspecifythecachelevel.

GEThttp://localhost:8080/cloudmesh/cpu_cache/{level}

level=‘l3’forl3cachesizelevel=‘l2’forl2cachesize

Thereturnshouldbeadictionaryasfollows.

4.If cache level is not specified, the following dictionary should bereturned.

5.Addthesemethodstothecpu.yamldefinition.

6.Deploytheservicelocallyandtesttheservicesusingswagger-UIandcurl.

{ "l3" : "8448 KB" }

{ "caches":{ "l3":"8448 KB", "l2":"1024 KB" } }

https://pypi.org/project/py-cpuinfo/0.2.3/

7.Create am1.small instance inChameleonCloudwithubuntu 18.04image.Deployyournewwebserviceinthecloudinstance.YoushouldusecloudmeshcommandstostarttheseVMs.

8.Usecurltocallthewebserviceinthecloud.(YouareNOTexpectedtoexposetheservicethroughthepublicIPaddress)

9.Capture the outputs for each service paths in a meaningful way(images, screenshots, etc) and compile a Markdown file. TheMarkdown should include, the new cpu.yaml file, server.py, andcpu.pyfilestogetherwithoutputs.Furthermore,youshouldincludethecloudmeshcommandsthatyouhaveusedintheprocessandyoumayinclude improvements to the service such as handling malformedrequests,etc.

10.In the Markdown file, discuss what are the ways you could addsecurityfortheseservices(YoudoNOTneedtoimplementthis).

4.7WEEK7:MAPREDUCE�☁




PleasereadtheHadoopsectionsinCloudComputing[1]

WatchtheseonlinevideosforHadoopMapReduce:

MapReduce,Hadoop,andSpark(19:02)HadoopA

Hadoop13:19HadoopB

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week7-mapreduce.md




https://youtu.be/HfuP2RJnQ6k?t=73

https://youtu.be/-N5PpD2sy3Q?t=17

Hadoop12:57HadoopC

Hadoop15:14HadoopD

Note:AllMapReduce/HadoopLabactivitiesareoptoional

4.7.2LABACTIVITY:HADOOPINSTALLATION

ThefollowingexerciseswillguideyoutoinstallHadooponasinglenodeand then run aMapReduce job in Python. Please figure out the requiredcommandlines.Thesecommandsareavailableinthebooksections:CloudComputing,GregorvonLaszewski,Ed.2019[1]

4.7.2.1Exercises:

InstallprerequisitesoftwareInstallJavaInstallSSHInstallMaven

ConfigureprogrammingenvironmentsJAVA_HOME

core-site.xml

hdfs-site.xml

SShlocalhostwithoutapassphraseStartHadoopHDFS

FormatHDFSfilesystemStartNameNodeandDataNodeviacommandlinesCheck NameNode status via Web Interface: NameNode -http://localhost:9870/

StartHadoopYARNConfigure

mapred-site.xml

yarn-site.xml

StartYARNviacommandline

https://youtu.be/BaRHay32I80?t=18

https://youtu.be/MYOosbF6-dA?t=20


Check YARN status via Web Interface: ResourceManager -http://localhost:8088/

Afterfinishingallthesesteps,youaregoodtomoveforwardtoMapReduceprogramming.

4.7.3LABACTIVITY:PYTHONMAPREDUCE

SeeinthemapSectionofthePythonbook:

The basic syntax of the map function expects a function object and anynumberofiterableslikealistordictionary.Itexecutesthefunctionobjectforeachelementinthesequenceandreturnsalistoftheelementsmodifiedbythefunctionobject.

4.7.4LABACTIVITYMAP/REDUCEONTHECLOUD

4.7.4.1PythonWordCountinMapReduce

ThiscodeisusedasMapper.

def multiply(x): return x * 2

map(multiply2, [2, 4, 6, 8]

# Output [4, 8, 12, 16]

#!/usr/bin/env python

import sys for line in sys.stdin: line = line.strip() words = line.split() for word in words: print('%s\t%s' % (word, 1))

#!/usr/bin/env python

from operator import itemgetter import sys current_word = None current_count = 0 word = None for line in sys.stdin: line = line.strip() word, count = line.split('\t', 1) try: count = int(count) except ValueError: continue

https://cloudmesh-community.github.io/book/vonLaszewski-python.epub

ThiscodeisusedasReducer.

Please note these code snippets are used to demonstrate the idea ofMapper and Reducer in Python. It leaves some bugs by intentions.PleasedebugtheabovecodeorwriteyourownversionofMapReduce.

4.7.4.2RunHadoopMapReduceinPython

4.7.5LABACTIVITY:MAPREDUCEONTHECLOUD

4.7.5.1HadoopClusterSetup

Hadoop’s Java configuration is driven by two types of importantconfigurationfiles:

Read-only default configuration - core-default.xml, hdfs-default.xml,yarn-default.xmlandmapred-default.xml.Site-specificconfiguration-etc/hadoop/core-site.xml,etc/hadoop/hdfs-site.xml,etc/hadoop/yarn-site.xmlandetc/hadoop/mapred-site.xml.

Additionally,youcancontroltheHadoopscriptsfoundinthebin/directoryof the distribution, by setting site-specific values via theetc/hadoop/hadoop-env.shandetc/hadoop/yarn-env.sh.

ToconfiguretheHadoopclusteryouwillneedtoconfiguretheenvironmentin which the Hadoop daemons execute as well as the configuration

if current_word == word: current_count += count else: if current_word: print('%s\t%s' % (current_word, current_count)) current_count = count current_word = word if current_word == word: print('%s\t%s' % (current_word, current_count))

bin/hadoop jar <path_to_hadoop_libs>/hadoop-*streaming*.jar \ -file /<path_to_mapper>/mapper.py \ -mapper /<path_to_mapper>/mapper.py \ -file /<path_to_reducer>/reducer.py \ -reducer /<path_to_reducer>/reducer.py \ -input <input_file_path> \ -output <output_file_path>

parametersfortheHadoopdaemons.

HDFS daemons are NameNode, SecondaryNameNode, and DataNode.YARNdaemonsareResourceManager,NodeManager,andWebAppProxy.IfMapReduce is tobeused, then theMapReduceJobHistoryServerwillalso be running. For large installations, these are generally running onseparatehosts.

4.7.5.2Configuration

ConfigureallHadoopdaemons:

NameNode:HDFS_NAMENODE_OPTSDataNode:HDFS_DATANODE_OPTSSecondaryNameNode:HDFS_SECONDARYNAMENODE_OPTSResourceManager:YARN_RESOURCEMANAGER_OPTSNodeManager:YARN_NODEMANAGER_OPTSWebAppProxy:YARN_PROXYSERVER_OPTSMap Reduce Job History Server:MAPRED_HISTORYSERVER_OPTS

Configure Namenode to use parallelGC and a 4GB Java Heap, thefollowingstatementshouldbeaddedinhadoop-env.sh:

4.7.5.3OperatingtheHadoopCluster

TostartaHadoopclusteryouwillneedtostartboththeHDFSandYARNcluster.Thefirst timeyoubringupHDFS, itmustbeformatted.Formatanewdistributedfilesystemashdfs.

FormatNameNode(onadedicatenodeinthecluster)StartNameNode(onadedicatenodeinthecluster)StartDataNode(oneachnodeofthecluster)StartResourceManager(onadedicatenodeinthecluster)

export HDFS_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"

StartNodeManager(oneachnodeofthecluster)

4.7.5.4RunHadoopMapReduceinPython

4.8WEEK8:PROJECTREVIEW☁

Makesuretohavemademajorprogressinyourproject

Documentitin

githubbyupdatingreport.bibgithubbycheckinginyourcurrentcode

4.9WEEK9:CONTAINERS-DOCKER☁




Please read the section in Cloud Computing[1]: about Containers. Wedecidednottodoavideoasthematerialinthissectionfrequentlychangesandvideosbecametooquicklyoutdated.

4.9.2LABACTIVITYDOCKER

Dothefollowingassignmentswithallitsquestions:

$ bin/hadoop jar <path_to_hadoop_libs>/hadoop-*streaming*.jar \ -file /<path_to_mapper>/mapper.py \ -mapper /<path_to_mapper>/mapper.py \ -file /<path_to_reducer>/reducer.py \ -reducer /<path_to_reducer>/reducer.py \ -input <input_file_path> \ -output <output_file_path>

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week8-midterm.md

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week9-container.md




E.Docker.1E.Docker.2E.Docker.3

Placeyoursolutionsinyourgithubhiddirectory

4.9.3LABBOOKMANAGER

Read the section in the Container chapter about book manager. Doassignments

E.container.bookmanager.1E.container.bookmanager.2

4.9.4OPTIONALLABACTIVITYCONTAINERRESTSERVICE(ASPARTOFMANYPROJECTS)

Develop a simpleREST service usingOpenAPI.Use the “Introspection”principalwiththeconexionwebservice.DevelopacontainerthatrunstheREST service. Write a script or pytest that contacts the container andreturnstheresultfromtheRESTservice.

WriteaDockerfilethatcreatetheservice

Writeacloudmeshcommand(rememberwedidthisbefore) thatmanagesandinteractswiththeserver,butitactuallyinteractswiththecontainerandnotjustthenativeservice.

Here is an example list sof commands (replacemyservicewith a comandthatisnotusedalreadyincloudmesh-cloudorcloudmesh-storage:

cms myservice start starts the service cms myservice stop stops the service cms myservice PATH Connects to the running service and returns the Object curl ... Document how you interact with your service

Obviously you can use services from your project. A more elaborateversionof tiswill be in yourfinal project.Develop commands thatmakesenseforyou,Thiscan includeget,upload,delete,andupdateactionsforexample.

4.10WEEK11:GRAPHQLANDMESSAGING☁




Please read thechapter in thebookCloudComputing[1]aboutGraphQLandMessaging.

WehavespendsignificanttimetoteachyouhowtogenerateRESTservicesfromOpenAPIspecifications.ThishastheadvantagethatyoudonothavetodevelopaserverasitcanbeoutomaticallygeneratedfromtheOpenAPIspecification.

Furthermore clients could be developed with Swagger codeGen in thesupported large number of supported programming languages. HoweverusingSwaggercodeGen isnot required for thisclassandgoesbejond thescope we like you to explore as part of this class. So if you are usingSwaggercodegen,youneedtocontactGregortoidentifywhyyouwouldneedthisforthisclass.Ingeneralitisnotneeded.

Instead we like to focus on two different technologies. One is calledGraphQlandtheotheriscalledMQTT.

Theinformationaboutthemareincludedinthebook:

CloudComputing,GregorvonLaszewski[1]

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week11-messaging.md





Pleaselocatethechaptersandread

4.10.2LABACTIVITYGRAPHQL

Beforedoingtheseoptionalassignments,makesurethatyourprojectisinverygoodshape.Theseoptionalassignmentsareungraded.

E.GraphQL.1E.GraphQL.3

4.10.3OPTIONALLABACTIVITYMQTT


E.MQTT.2

4.11WEEK12:GO☁

As you have noticed cloud computing has a lot to do with parallelcomputing,servers,andmessaging.

Whilepythonisagreatlanguageforcloudcomputingwealsoliketomakeyouawareofsomeotherlanguages.OneoftheselanguagesisGO.GohasalotoffeaturesincludedfromCSP.

PleasereadthechapterinthebookCloudComputing[1]aboutGo.

4.11.1OPTIONALLABACTIVITYGORESTSERVICE


E.Go.1

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week12-go.md


4.12WEEK13:FAAS�☁

Recently you have heard abouut Function as a Service.We will providesome introductory material and discuss some concrete FaaS frameworksandFaaSofferingsbycloudproviders.

PleasereadthechapterinthebookCloudComputing[1]aboutFaaS.

4.12.1LABACTIVITY

Therestofthesemestryoucanexploraspectswetaughtyouandfocusonyourprojects.

4.13WEEKN-16☁

TheagendaforthenextweekswilllikelyfollowtheSyllabus.Adjustmentsmayneedtobemadebasedonpaceoftheclass.

We will try to release new material on Tuesdays’s before midnight.AssignmentswillthanbedueonTuesdaythefollwoingweekat9aminthemorning.IfthingsareunclearpleaseaskquestionsonPiazza.

https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/week13-faas.md


https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/weekn.md

5APPENDIX

5.1FAQS☁

5.1.1COURSEMANAGENT

5.1.1.1Whenisthelastdatoftheclass?

ThelastdayofclassisMay8th

5.1.1.2Whatdothe+-intheFAQsinPiazzamean?

=FAQisavailableinthelecturenotes=FAQwillnotbeavailableinthelecturenotes

without + or -: FAQ is not yet reviewed for inclusion in the lecturenotes

5.1.1.3PleasereadtheFAQ

Beforeaskingaquestion,please read theFAQhereandonPiazza tofindanswersorengageindicussionsthatresultinanswers.

5.1.1.4WheredoIfindthematerialforthisclass?


5.1.1.5Wheretofindtheweeklyagenda?

We have lecture notes for this class distributed as epub and PDF. WerecommendcreatingforthelinkshortcutsinyourWebbrowsersoyoucanclickonthemquickly.However,allresourcesarepostedat


https://github.com/cloudmesh-community/book/blob/master/chapters/class/516/faq.md



Youcanfindalllinksandmorealsoat

https://laszewski.github.io

5.1.1.6Whendotheonlinemeetingstakeplace?

TBD:see:https://piazza.com/iu/spring2020/e516spring2020/resources

Onlinemeetingsareforresidentialstudentsnotasubstitutefor theFridayMeetings.

5.1.1.7WhyswitchthecomputertoEnglishlanguagemode?

Wewillnot support any student that isnotusingEnglish in the languagesetting in their computer becausewe can not read it.You can set it backafterthemeetings.

5.1.1.8WhyshouldInotusedarkmode

Be kind to people with vision problems. The “dark” mode has becomereallypopular.However, ithassomereallybigdisadvantage.Thoseofuswhich are having poor vision can not see the light font on the blackbackground.

So if you like to effectivelyworkwith those that have vision issues youmustuseawhitebackgroundanddarkfonts.

Thishasnowbecomearequirementandnotanoption.

IatetndednnowseveralsessionsinwhichIcouldnotreadthescreenduetothe use of black background. This is valid for pycharm as well as theterminal you use to execute your programs.You can switch back after azoommeeting.

Wewill immediatly stop support if you use dark themes andmove on tootherstudents.

https://laszewski.github.io/


CMD.execaneasilybeswitchedtolightmodewithcolor f0

5.1.1.9WaituntilGregorsaysENTER

WhenworkingwithGregor, on installs try tobepatient andWAITTILLHESAYSENTER.

5.1.1.10Characternamesonthekeyboard

Weusethefollowingnames

~=tildeortwiddleorsquiggly’#`=hash‘:’=colon‘[]’=sqaurebraketbasedonwhencontextisclearwejustsaybracket‘{}’=curlybraketbasedonwhencontextisclearwejustsaybracket‘()’=roundbracketbasedonwhencontextisclearwejustsaybracket‘<>’ = angle bracket based on when context is clear we just saybracket‘<>’=urlbracketbasedonwhencontextisclearwejustsaybracket

5.1.1.11WherecanIfindtherecordingsoftheonlinemeetings?

Themeetingrecordingsaregenerallyavailabletwodaysafterthemeeting.However, jf thebooksincludetheinformation,wemayoptnot topublishthemandinsteadrefertothebook.

TherecordingsareaddedtotheLecturenotes.

5.1.1.12DoIneedtoreadtheclassmaterialtotaketheclass?

Yes,thisisarealquestionfromapaststudent.IUdoesnotofferclassesthatgiveyouacertificatewithoutunderstandingthematerial.Tounderstandit,youmuststartbyreadingit.

5.1.1.13Doesthisclasshaveamidtermandfinal?

Althoughwedonothaveafinalormidterm,wedohavedeliverables fortheclass.ThesedeliverableswillbeclearlystatedintheLecturenotes.

Thistheansweisyes,butitwillnotbeanexamandincludesthereviewofyourgradedassignments.

5.1.1.14Balanceyourclasses.

Thisclassrequiresatminimum6hoursaweek.Westatethisexplicitlyasweoftenfindstudentswiththewrongimpressionthattheycanspend100%ofawekes timeonother classes.Please set6hours aside andmake sureyou realy do it. If you finish in a week in less time, please conductadditionalwork. This class is designed so that you do spend 6 hours perweeknomatetrwhat.

5.1.1.15Whenisthelastdayofthisclass?

ThelastdayofclassisMay8th.

5.1.1.16WillyouprovidesupportduringtheSpringbreak?

There should not be a reason for this as you need to work during thesemester and will be likely in good shape if you continiuosly work.Howeveritmaybepossibletoprovidesupporthours.Theytypicallycanbescheduledonrequestbetween9-10amduringtheSpringbreak.

TheMon,Tue8-9pmmeetingwilltakeplaceuponrequestTheresidentialclass fduring the spring break is canceled, you can request to attend theonlinemeetings.

The meetings must be requested 1-2 working days ahead. Ther ei snogurantee that I am inmeetings if they are not confirmed.Cancelationsofmeetingsmaybepossible.

5.1.1.17Lab:WhycanInotsolveanassignmentin2minutes?

Insomecases,ittakesyouaconsiderableefforttoresearchtheanswer.AnexampleistheDatacenterquestionaboutenergyconsumptionorcost.Itisnot expected that google may return a result quickly. You may need toconsiderotherpublicationssuchasgooglescholar,orprofessionaljournals.Ifyou,aftersomehours,havenotfoundananswer,maybethereexistsnosuchanswer.Checkwhatotherstudentshavedoneandaskthemhowtheyfound the information. Remember, this class is a community. Discuss onPiazza.

5.1.1.18Lab:HowmuchtimedoIneedtospendoneachquestion?

This question can not be answered as it depends on the question. Somequestionsmaytakehours.Thetimestudentstypicallyspendonthisclassis6-8hours.However,sometimesitmaybelessormoredependentonyourbackground. Please be reminded that many students with many differentbackgrounds attend this class. For example, if we ask you to do somepythonprogramming,andyouhavenobackgroundinPython,thattaskmaytakeyoulongerthansomeonewhohasPythonbackground.However,afteryouhavedonesomePythontasks,thenextpythontaskwillbeeasier.LetusassumeyoudonotknowPython,butit tookyouonly1hourtodotheassignmentsfortheweek,werecommendthatyoudotherestofthetimetolearn Python. This way , when the python assignment starts you are aPythonexpertalso.

5.1.1.19IdonotunderstandPiazza?

Attend the online hours and ask questions there or watch the recordedonlinemeetingsasweexplainedittherealso.

5.1.1.20WhyyoushouldnotreadthePiazzamailbutuseClick Hereinstead?

Piazza sends you e-mails. You can control the frequency in the piazzasettings.However,sometimeswedomakecorrectionstothepostafteryoureceivedthee-mail.Topreventanyissues,wedorecommendthat insteadofreadingthee-mail,youjustusetheclickherefeature,whichisplacedontopofeachmail.

Thisway,youareuptodateinsteadoflookingatapoteniallyoutdatede-mail. I activated e-mail notifications, so you have the best f these twomodes.

5.1.1.21CanIsearchintheePubreader?

Yes,consulttheHelpinyourepubreaders’documentation.

5.1.1.22WheredoIfindtheGitHubdirectory?

Seehttps://github.com/cloudmesh-communityandsearchforyourname.Itwill take some time to setupyour repository.A formwillbe sent toyouthatasksyoutogiveusyourgithub.comaccount.Beremindedthatweusegithub.com and not IU GitHub, which has, in the past, been unreliable.Thus,ifyoudonothaveanaccountongithub.com,pleasecreateonenow.

Therepositorywillbecreatedforyouafteryoufilledoutthesurvey.Itwilltakesometimeasthisisaaprocessdonebytheinstructor.

YOuwillknowthatyourrepositoryhasbeencreatedasyouwillreceiveane-mailfromgithubtotheemailaddressthatyouhaveregisteredingithub.Ifyoudonotreceivedane-mailitmaybeinyourspamorjunkfolder.

YOucanalsogototherepositoryandbrowsethroughthemandlocatetheoneunderyourname.

Ifitisthereyouneedtoaccepttheinvitationifyouhavenotdonesoviathee-mail.Thisiseasy:

1.Locateyourrepositoryinhttps://github.com/cloudmesh-community


2.Letsassumeyouhavenumbersp20-516-0003.Go to https://github.com/cloudmesh-community/sp20-516-000/invitations

4.Accepttheinvitation

5.1.1.23WherecanIfindthehiddirectory?

Weoften refer to theGitHubdirectory as thehiddirectory, seeWhere tofindtheGitHubdirectory?

5.1.1.24DoIneedthefuturesystems.orgaccount?

No.

5.1.2HARDWAREANDSOFTWARERELATEDQUESTIONS

5.1.2.1CanIusemyworkcomputerforthisclass?

Pleasediscussthiswithyourcompany.

Every year we have bad experiences with students that use a workcomputer.THisisnotalwaysthecase,butfrequentlyfirewallareinstalled.programms that blok execution or other restrictions are imposed on themachine. Therfore we have decided, to not provide any support for anystudentusingaworkcomputerevenifyouhaveadministrativepriviledges.Only if you are completely responsible for the machine we will providesuggestions.Thefirstonewehaveis:buyabackupdrive,byanotheronefromwhichyoudoalbiitandleaveyourworkpartitionalone.

Anotheroneisifthisdoesnotworkuseavirtualmachineinthecloud.

Orfindadifferentcomputerthatyouhavefullcontrolabout?

5.1.2.2Howmuchmenmoryhavethesedaysdevelopmentcomputers?

Developmentcomputersthesedayshave16GB,howeveryoucangetawaylikelywith8G.

5.1.2.3WhatHardDrivedoyourecommend?

SSD.

5.1.2.4CanIuseWindows1809?

No. itisbrokenandwillnotwork.Donoteventry!

5.1.2.5RasperryPIClusterComponents

Pleasehelpwithyoursuggestions.

ThisisonlyrelevantforthosethatwanttousethePIasaproject.

Residentialstudentswillgetaclusterforthedurationoftheclassthattheycantakehome.

Next we describe the components that are needed to create a clusteryourself

Wearecollectinginthisspreadsheetinformationaboutwhichcomponentsare needed for a raspberry pi cluster. We estimate a cluster costs about$333.32

https://docs.google.com/spreadsheets/d/1bK6XssA_1NStsAkyxYBuzwmXOVxXEt-oIkvKoqz91L4/edit?usp=sharing

ThosethatselectitasprojectandcancometothebuildingIaminandcangetaclusterwithupto5nodesfortheduration.Wehaveabout200PIs.Wehave PI3B and PI3B+.We do not have the newer Pi4s which are muchmuchnicer.SoifyoubuymakesuretogetPI4’s.Wewill,however,haveabout5PI4’sclosetomyofficewhichyoucanuseinthebuilding(theyareonorder)

https://docs.google.com/spreadsheets/d/1bK6XssA_1NStsAkyxYBuzwmXOVxXEt-oIkvKoqz91L4/edit?usp=sharing

Wewillmakechangestothespreadsheetbasedonyourfeedback.Youcanplay with the count.When ordering a PI4 I recommend to get the 4GBversion.Thatisthebestpricepoint.

Theonlythingthatisabitopenisthepowersupply.Fortheclusterinthebuilding,we elected touse anAnkermultiUSBpower supply.However,youwillneedacableanditmayjustbecheaperforyoutobuythestandardpowersuppliesforthepi4s.Ifoundoneonamazonthatevenhasaswitch.

Networking may be able to be used from your modem. However, itsworthwhiletothinkaboutaseparateswitch.

Youcanalsogowithlessthan4GBforthePiversion,butthe4GBseemsinmyviewthebestvalueinmyview.

5.1.2.6Whichoperatingsystemcanweuseforclass?

macOS10.14.8ornewerUbuntu18.04Ubuntu19.10Windows10EDU>=1903Windows10PRO>=1903anyuptodateDebiandistributionCentOS(recent)Fedora(recent)Raspbian(newest)forPi

WerecommendatthsitimeUbuntuoverCentosandFedoraasmanyhaveexperiencewithUbuntu.

ThefollowingOSarenotusable

Windows7or8iOSAndroid

5.1.2.7WhatdoIneedtorunHyper-VforWindows?

Youneed

1.Windows10Enterprise,Pro,orEducation.2.64-bitProcessorwithSecondLevelAddressTranslation(SLAT).3.CPUsupportforVMMonitorModeExtension(VT-conIntelCPUs).4.Minimumof4GBmemory.

Checkyoursystemifyourcomputerfulfillstheserequirements.Ifnotthereis no need to even try! You will have to look up your computerspecificationsandgoogleif2and3apply

Hereisthelinkaboutthis:

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v

5.1.2.8WherecanIgetWindows10EDUforfree?

https://iuware.iu.edu/Windows/Title/2977

Afterinstalationyoumustupdateit.makesureyouhaveatleast1909

5.1.2.9CanIrunmultipassinWindows10Home?

yes. See the documentation in out Cloud computing book and use“virtualbox”asadriver.youmustinstallvirtualboxonyoursystem.

https://discourse.ubuntu.com/t/installing-multipass-for-windows/9547

5.1.2.10WhatSoftwarewillinneedforthisclass?

The software can not be listed up front as it may change during thesemester.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v


https://discourse.ubuntu.com/t/installing-multipass-for-windows/9547

Herewhatweknowatthistime:

Required:

Epubreader(OSXBooks,AllothersCalibreoralternativeyoufind)

Python3.8.1https://www.python.org/downloads/OnWindows:ssh,VisualC++BuildToolsCLI,gitbash,multipass

OnOSX:xcode

Optional,butthebestPythoneditor:

Pycharmcommunityversionhttps://www.jetbrains.com/pycharm/

Optional, but we will use this to introduce you to some programmingconcepts:

Multipass:<https://multipass.run/ifmultipassdoesnotwork,youcanusetheallpowerfulprintstatement;-)

5.1.2.11Whatisthebesteditorforclass?

Pycharm,butitusesalotofmemory

5.1.2.12CanIuseVS-Code?

Yesbutithaslotsoffeaturesmissing:

https://tzutalin.blogspot.com/2019/04/set-up-visual-studio-code-for-python-in.html

For group projects pyCharm code inspect is very nice and keeps codeuniform. Hence we do not recommend using VS-Code for pythondevelopmentinthisclass.

https://www.python.org/downloads/

https://www.jetbrains.com/pycharm/

https://tzutalin.blogspot.com/2019/04/set-up-visual-studio-code-for-python-in.html

##@Whatisthebestwaytousepycharmfortheclass

Firstcloneyourhidrepository.Makesureyoufollowourinstructionsandpuitinafoldercmsoitseasierforyoutofollowourinstructions.

Whenusingpycharmyoudonotcreateanewproject,instead,youuseOpen.Nowbrowsetothecmfolder(notthehidfolder)andchosethecmfolder.

5.1.2.13Whataarethebiggesissueswenoticed?

1.Nottakingtimetofilloutaform2.Forgettingapassword3.ChameleonCloudaccountcreationbug4.Notupdatingconfigurationfiles5.Notupdateingthesoftware6.Spendingallyourtimeonanotherclass

5.1.3MARKDOWN

5.1.3.1Whatisthebesteditorforbibliographies?

1.jabref2.emacs

Pleasestayawayfromendnote,itusedtohavesignificantstabilityissues.

5.1.3.2CanIjustcreatanempty.bibfile?

No. undernocircumstancescreateanemptybibfile.youmustputatleastonebibtexentryinitbeforecommittingtogithub,otherwiseourautomatedscriptsthatusepandobreak.

$ cd ~ $ mkdir cm $ cd cm $ git clone sp20-516-00 $ cd sp20-516-00

5.1.3.3Whatisan�inthemarkdownfile?

Thisindicatesanissuethatyoumustfix.

5.1.3.4Howtosetthecolumnwidthinpycharmto80characters?

Werequireyoutoformatallmarkdownin80characterwidthtext.THiscanbe enabled in pycharm and a shortcut can be created to invoke it on aparagraphlevel.

ForPycharm2019.3versiononMacOS,youcangoto

PyCharm -> preferences -> Editor -> Code Style -> Markdown

Iset‘HardWrapat’tobe80tofollowPEP-8styleguide.

Alsoset‘wrapontyping’tobeYesallowinglinestobreakeautomaticallywhanyouwrite.

Seealsoourcloudmeshmanualforcodeconventions:

https://cloudmesh.github.io/cloudmesh-manual/dev/manual.html#documentation-management

5.1.3.5Whatisagoodspellsheckerformarkdown?

Grammarly.

5.1.3.6Whymustinotusethewordsbellowandaboveinmarkdown

PleasenotethataccordingtopublisherssuchasOReileytheword“bellow:and”above"areforbiddentobeusedindocumentation.Soneverusethesewords again. As we publish our documents in epub below and aboveabsolutelydonotmakeanysenseasthecontentfloatsandispotentiallyondifferentpagesdependentonyouroutputdevice.

https://cloudmesh.github.io/cloudmesh-manual/dev/manual.html#documentation-management

5.1.3.7WhymustIavoidscreenshotsforterminaloutput?

Please note that is is not allowed to use screenshots of terminals andwemust use Copy and paste and use markdown code mode to frame it.Obfiously we can search in text, but not in eimages as part of epub ormarkdown.

5.1.4EPUBANDDOCUMENTRELATEDQUESTIONS

5.1.4.1Isthereanepubreaderforchrome?

Astudentreportedthisworkswell:

Howtoinstall"

1.OpenChrome2.Clickonthe‘Apps’icon3.GotoGoogleChrome’s‘Webstore’4.Search‘epubreader’5.Make sure you install the one with the following logo shown inFigure 1

6.Openanepubfile(eithersavedondriveorclickingonalink)7.Mouseoverthebottomtofindsettingsinthereader8.Set your column width to your desire and match of your screenresolution

9.Enjoy

Figure1:EPubreader

Sometimesitsworthwhiletoclearthecache.Thereisalsoausefulutilityforchromethatcandothis:

https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn

ItwillinstallarecycelingiconthatwhenyoupressonitclearthecacheforthecurrentlydisplayedURL.

5.2EPUBREADERS☁

ThisdocumentisdistributedinePubformat.EveryOShasasuitableePubreader to view the document. Such readers can, in many cases, also beintegrated into aWeb browser so that when you click on an ePub, it isautomaticallyopenedinyourbrowser.AsweuseePubs,thedocumentcanbescaledbasedontheuser’spreference.Ifyoueverseecontentthatdoesnotfitonapage,werecommendyouzoomout tomakesureyoucanseetheentirecontent.

Wehavemadegoodexperienceswiththefollowingreaders:

macOSX:Books,whichisabuildinebookreader

https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn

https://github.com/cloudmesh-community/book/blob/master/chapters/preface/reader.md

https://www.apple.com/apple-books

Windows10:calibreLinux:calibre

IfyouhaveaniPadorTabletwithenoughmemory,youmayalsobeabletousethem.

Other browsers are available. Please report if you like onemore than theoncespecifiedhere,soweaddyoursuggestion.

Youmaywanttoadjustthezoomofyourreadertoincreaseordecreaseit.Pleaseadjustyourzoomtoalevelthatiscomfortableforyou.OnmacOSwithalargermonitor,wefoundthatzoomingoutmultipletimesresultsinverygoodrenderingallowingyoutoseethesourcecodewithouthorizontalscrolling.

5.3FAQ☁

5.3.1WHEREISTHEMANUAL

AManualisavailableat

https://cloudmesh.github.io/cloudmesh-manual

5.3.2VIDEOS

TBD

5.3.3HAVINGCONTROLOVERYOURCOMPUTERS

Cloudmesh is easy to install, butwe have seen issueswith the followingthatyoumustsolveyourself:

1.Virus checkers switch on that prevent instalation of programs andcontainers

2.Computersmanagedbyaworkplacewithstrictersecurityrules.



https://github.com/cloudmesh/cloudmesh-manual/blob/master/docs-source/source/faq/install.md

https://cloudmesh.github.io/cloudmesh-manual

Anyofthesecasesareuptoyoutofindoutandsolve.WehavetestedCMSonanoperationg system thatwasdownloaded stright fromMicrosoft andverofieditworks.

5.3.4DEPLOYMENTTIPCOLLECTION

Thebiggestissueistoinstallcmsandconfigureit.Ifyouhaveissueswiththis,usecmsd.

SSH:asyoualreadycreatedakeyfortheHorizonassignmentyoucanskipthisone

5.3.4.1InstallMongoDB

osxandLinux:

Windows:

installthemsi,BUTDONOTcheckonserverinstall,compass.YoumustuninstallallpreviousversionsofMongoDB

5.3.4.2Doanupdateofthesource

5.3.4.3InteractwithChameleonCLoud

cms admin mongo install

pip install cloudmesh-installer -U cloudmes-installer git clone cloud cloudmesh-installer git clone openstack cloudmes-installer git pull cloud cloudmesh-installer git pull openstack cloudmes-installer install cloud cloudmesh-installer install openstack cms help cms gui quick cms test cms init

cms key list

cms test cms gui quick

cms key list --cloud=chameleon

removeallkeysyouseewith

5.3.4.4Draft:CmsMostlyautomatedinstallonmacOSandLinux

Note:Useatyourownrisk

IfyoudinotunderstandlinebylinewhatthisdoesyoushouldnotexecuteitVerifycorrectnettworksonlinuxandOSX,Ihavethisnotyettestedongitbash,onwindowsyoumustremovethemongoinstall lineandhavecl,mongo,gitbashpreviouslyinstalled.

Youmusthaveusedssh-keygentogeneratethekeybefore

InstallPython:

Saveinfileinstall-python.sh

Saveinfileinstall-cms.shandchangevalues

cms key delete NAMEOFTHEKEY --cloud=chameleon

cms key list cms key upload --cloud=chameleon cms image --refresh cms vm --refresh cms vm boot cms vm --refresh cms vm log cms vm ssh

#! /bin/sh -x rm -rf ~/ENV3 rm -rf ~/cm/cloudmesh-common rm -rf ~/cm/cloudmesh-cmd5 rm -rf ~/cm/cloudmesh-sys rm -rf ~/cm/cloudmesh-configuration rm -rf ~/cm/cloudmesh-test rm -rf ~/cm/cloudmesh-gui rm -rf ~/cm/cloudmesh-cloud rm -rf ~/cm/cloudmesh-openstack rm -rf ~/cm/cloudmesh-inventory rm -rf ~/cm/cloudmesh-openstack rm -rf ~/.cloudmesh python3.8 -m venv ~/ENV3 echo "Please activate your Virtual Environment" echo " source ~/ENV3/bin/activate"

#! /bin/sh -x pip install pip -U

pip install cloudmesh-installer -U mkdir cm cd cm cloudmesh-installer git clone openstack cloudmesh-installer git pull openstack cloudmesh-installer install openstack

makethescriptsexecutable

Nowlestuseit

StartyourENVwith

Installcms

Startavm

cloudmesh-installer install openstack cms set cloud=chameleon cms help cms debug off

#cms test # its in your responsiblity to fix this this test may not work yet properly

cms config set cloudmesh.data.mongo.MONGO_AUTOINSTALL=True

#cms config set [email protected] #cms config set cloudmesh.profile.firstname=TBD #cms config set cloudmesh.profile.lastname=TBD #cms config set cloudmesh.profile.github=TBD #cms config set cloudmesh.profile.user=TBD

#cms config set cloudmesh.cloud.chameleon.credentials.auth.username=TBD #cms config set cloudmesh.cloud.chameleon.credentials.auth.password=TBD cms config set cloudmesh.cloud.chameleon.credentials.auth.project_id=7767f9aac3c143de8b1f0e6acc70f159 cms config set cloudmesh.cloud.chameleon.default.network=e01ee12e-fd22-4b21-a050-d3e015f42bb1

cms config set cloudmesh.data.mongo.MONGO_PASSWORD=TBD rm -rf ~/.cloudmesh/cloudmesh.yaml.bak.*


#cms test

cms gui quick # Set your Mongo username = `admin` and password what you want. sleep 2 cms init sleep 2

# LET US NOW ASSUME YOU HAVE IN cloiudmes.profile.user=xyz

Replace xyz with what you have in profile cms key delete xyz --cloud=chameleon cms key upload xyz --cloud=chameleon cms key list cms key list --cloud=chameleon

cms vm list --refresh

chmod a+x install-cms.sh install-python.sh

./install-python.sh

source ~/ENV3/bin/activate

./install-cms.sh

cms vm boot

5.3.5SWITCHINGBETWEENCMSANDCMSD

Itisbesttodeletethe.cloudmeshdirectoryandstartnew.

HoweveryoucanalsotrytosettheMONGOmodeaccordingly.

Forcmsyouuse

Forcmsdyouuse

Remeber that cmsd will automatically upon –setup change the mode torunning.SOifyouliketoswithcbackdoso.

Pleasebeaware therecoudlbe issueswithMongoruningnativelyor inacontainer.Tryitout.IFitdoesnotwork,removethe.cloudmeshdirectory

5.3.6WHATISKEYERROR:‘AUTH’

Ifyouseethis:

Youhaveanoutdatedversionofthe.cloudmesh/cloudmesh.yamlfile.stopmongoandreinstallcloudmeshandtheENV3

5.4FAQFORWINDOWS☁

5.4.1VIDEOS

ShowcasethecompleteinstallofCMSinavideoassumongyouhavedonetheprerequisitsasdiscussedlater

https://iu.zoom.us/rec/share/yuZcE_Lg2DlLSKOd7hmEXaALRrbHaaa80yVK-aAOn0iCwnrZnokx6Rxwl91bIWqy

cms config set cloudmesh.data.mongo.MODE=native

cmsd config set cloudmesh.data.mongo.MODE=running

KeyError: 'auth',

https://github.com/cloudmesh/cloudmesh-manual/blob/master/docs-source/source/faq/windows.md

https://iu.zoom.us/rec/share/yuZcE_Lg2DlLSKOd7hmEXaALRrbHaaa80yVK-aAOn0iCwnrZnokx6Rxwl91bIWqy

https://iu.zoom.us/rec/share/tJZUK5DJ_2BOa5WWwRzQd758Qav4aaa823NPqPcOy06Vcf-xaVmuyG3Bnq70OujX

Other:

https://www.youtube.com/watch?v=5GpwfSjM9Lg

PleaseNote that this video discusses cloudmesh-windows starting ataround23minutes.Howeversincethenitwasgeneralizedandallowstobe runonotherOSes.To reflect this change itwas renamed fromcloudmesh-windowstocloudmesh-test.Thecommandisnowcalled

asalreadydocumentedelsewhere.

5.4.2CMSRESTRICTIONS

5.4.2.1CommandthatdonotworkonWindows

InsteadyouneedtoinstallMongoformtheMSI.butnotselecttheservicemode

5.4.3PREREQUISITS

Allofthefollowingcommandsmustreturnavalidpath.Ifnotinstallthem

YoumustusePython3.8.1,64bit

Nowtypein

$cms test


where cls where mongod where ssh where ssh-keygen

python --version

python

https://iu.zoom.us/rec/share/tJZUK5DJ_2BOa5WWwRzQd758Qav4aaa823NPqPcOy06Vcf-xaVmuyG3Bnq70OujX

https://www.youtube.com/watch?v=5GpwfSjM9Lg

andseeifyoufind64bit

5.4.4VISUALSTUDIOBUILDTOOLS

Youmusthaveclinstalled.Fordeatiledinstructiononhowtodothissee

https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019

Aftertheinstallyoumustadditspathtotheenvironmentvariables.Attimeofwritingthiswas

Thisisalsoextensivelydiscussedinourregularinstallnotes

5.4.5MONGODB

TheinstructionsforMongoDBareprovidedat

https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/

Please make ssure to switch of all Mongo Services and uninstall allpreviousversionsofMongoDB.THanpleaseinstallversion4.2.3.

However there are different ways to install Mongo.We do not want toinstallMongoasaservice.Whengoingtothedoscumentation,youwillseetwopanels.OneMongoDBService(whichwedonotwanttouse)andoneMongoDB. The later containse the proper install instructions includingimages.

5.4.6WHICHVERSIONOFWINDOWSDOINEEDTOUSEMULTIPASS?

ThisisareminderthatyoumustmakesureyourOSisuptodate.Youmustuseatleastversion1903orlater.Pleasechecktherequirementsattheirwebsite

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\Hostx64\x64

https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019

https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/

Youcanusemultipassinallversionofwindowsfulfillingthisrequirement.

f you use Windows 10 Home you must configure multipass to usevirtualbox.

However,tousehyperVwindows10mustbe

the64bitversion.youmustcheckthis.beEDUorPRO

Seehttps://multipass.run/

Instructions:

Multipass InstallationonWindowsMachinePrerequisiteWindows10ProorEDUv1903orabove

5.4.7MULTIPASSHYPER-VINSTALATIONONWINDOWS

Gotohttps://multipass.runanddownloadwindowsinstaller.

1.Runinstallerwithdefaultsettings.2.EnsurethatVirtualizationsupportisenabledinBIOS.Ifnot,enableit.3.Open PowerShell in admin mode by pressing windows+X andselecting

4.WindowPowerShell(Admin)Option.5.Enable Hyper-v Hypervisor by using following command inPowerShell.

6.Restartmachine.7.Afterreboot,runcommandtocheckmultipassinstallation.

8.IfSuccessfulinstallationmustlaunchacontainer.

$ Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor

$ multipass launch --name ubuntu-lts

https://multipass.run/

https://multipass.run/

5.4.7.1GeneralWindowsTips

5.4.7.1.1WheredoIfindthereleasenumbersforWindows?

https://docs.microsoft.com/en-us/windows/release-information/

5.4.7.1.2HowcanIgettheEDUVersion?

ForIUstudentsyoufollow


5.4.7.1.3IssueswithdockeronWindows

For anyone using Docker Desktop forWindows and seeing some insaneerrorwithcmsd --setup:

RestartDockerDesktop.Evidentlythere’sabugthatcausesDocker’sclocktostopwhenyourhostcomputersleeps, so thebuildenginegetsstuck inthepast

5.4.7.1.4UsernamesinWindowsmustnothavespaces

If youhave a spce inyourusername, simply create a newWindowsuseraccountwithoutspaces.

5.4.8WHATTODOIFWINDOWSSTILLDOESNOTWORKFORYOU?

First of all we have to say the issue is not with cloudmesh but yourcomputer.WeasteachershouldnotbethesystemadministratorfixingyourOS.Therforyouhaveseveraloptions.Herearejustaview:

1.useavirtualmachineonyourcomputer(multipassorvirtualbox)2.useavirtualmachineonchameleoncloud3.useanubuntucontainerandinstallcloudmeshinthecontainer

E: Release file for http://... is not valid yet

https://docs.microsoft.com/en-us/windows/release-information/


4.dualbootintoubuntu

Makesureyoudobackups.

5.4.9HOWTOINSTALLSSHONWINDOWS

https://winaero.com/blog/enable-openssh-server-windows-10/https://ittutorials.net/microsoft/windows-10/enable-ssh-windows-10-command-prompt/https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse

5.5FAQCHAMELEONCLOUD☁

AllFAQ’srelatedtochameleoncloudasume

1.youhavelookedattheinstanceswithhorizonbeforeyouusecms2.cmsisinstalled3.cms initworkedand

5.5.1DIDCMSEVERWORKED?

cmsWorkedfromdayoneovthisclass.Duetoanchangewithinchameleoncloud some parameters needed to be changed in the cloudmesh.yaml file. Weinformedtheentireclasstodeletethecloudmesh.yamlfileandcreateitagainviacms help. However, we found that this was often not executed and a lot oferrorscouldbetracedbacktothatthefilewasnotdeleted.Chameleonalsoupdatedthewaythenetworkworksandwehadtointegrateatthattimealsoa mechanism to deal with this. Again if you deleted your yaml file andrecreatedityouwillhavenoissue.

IfyoucompletelyeraseyourENV3,thecloudmesh-*folderincmanddoafreshinstall you can avoid this. As we had students not following our clearinstructions,wewillnolongerhelpifyouhavenot

1.deletedthe.cloudmeshdirectory

https://winaero.com/blog/enable-openssh-server-windows-10/

https://ittutorials.net/microsoft/windows-10/enable-ssh-windows-10-command-prompt/

https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse

https://github.com/cloudmesh/cloudmesh-manual/blob/master/docs-source/source/faq/chameleon.md

2.deletedyourvirtualnv(forexampleENV3)3.deletedyoursourcecodincmrelatedtocloudmeshMakesurenottodelete the directories unrelated to that. Be careful that you do notdeletethecloudmesh-*folderthatyouworkedon.makeabackupcopy.

4.afterthenewinstallandhelpcallusethecommandcms testandobserveandinterpretetheoutput.

Cloudmeshistrivialtoinstall.

5.5.2HOWCHECKIFCHMAELEONCLOUDWORKS

Thisreturnsthelistoffkavors

5.5.3HOWDOIUPLOADMYKEYTOCHAMELEON?

5.5.4WHATISTHEERROR…?

1.Ifyougetthiserror

thenyouhavenotuploadeyourkeytochameleon

5.5.5WHATISTHECHAMELEONACCOUNTBUG?

Preriquisite:

1.We assue you start by deleting your .cloudmesh directory and haveinstallaproperversionofcms.

2.Before you do anything verify if you can remember your password.You must logout of the chameleon cloud.org dashboart and login,

cms set cloud=chameleon cms flavor list --refresh

cms key upload --cloud=chameleon cms key list --cloud=chameleon

UnboundLocalError: local variable 'key' referenced before assignment

while not using an auto completionWeb password completer.Makesuretoremoveyourpasswordifyouuseoneofthem.

Chameleon has several strange bugs we know of that will hopefully befixed

1.Any account in chameleon thatwants to useOpenStackKVMmustfirst visit horizon and through the visit activate the acoount. Theactivationvia thee-mail that theysendtoyoudoesnotfullyactivateyouraccount.

2.Whenactivationyouraccount inhorizon,youmeust login toa loginscreen.However,sometimesweobservedthatscreendonotcompletethe activation andyoudonot get forwarded tohorizon. INsteadyouendinanendlessloopforactivationrequests.

3.Some students despite the issue of being able to login reported thattheycannotusecmseventhoughtheytypedinthepasswordcorrectly.TofindthisoutyouhavetotalktoGregorwhichunfortunatelyneedsyourusernameandpassword.toverifyifyouraccountworksornot.

4.IfGregor canverify that your accountworks (e.g. he does this on acomputer on whichwe checked chameleon cloudworks for anotheraccount),youwillhavetoapplyforanewchameleonaccountwithane-malthathasnotbeenusedat

UniversityofChicagoIndianaUniversityjetstreamXSEDEChameleonCloud

Thanyoudothepreviousstepsandcheckifyoucanusethisaccountin cms. If it works fine, if it does not file a ticket with chameleoncloud.

Gregor completely hats to do 3. and 4. but chameleon cloud have notprovidedanypropersolutions.

5.5.6YOUMUSTNOTUSEIUGUESTNETWORK

IUGUESToranyotherinsecurenetwork:

IfyouareatIU,chameleonclouddoesnotallowyoutoconnecttoitfromIUguest.Soifyousaycmsvmlist–refreshandithangs therefora longtimeyouwillatonepointsee“Failedtodiscoveravailableidentityversionswhencontactinghttps://kvm.tacc.chameleoncloud.org:5000/v3.AttemptingtoparseversionfromURL.”

This indicates you can not connect to chamelon cloud as you are on aninsecurenetwork.YoucandoaCRTLChere

Ifyouwaitlonger,youwillsee

5.5.7TETHERINGYOUCOMPUTERTHROUGHACELLPHONE

This will liekly not work as your cell phone company has likely somerestrictions.

5.5.8CANIKEEPMYINSTANCERUNNINGWHENNOTINUSE?

Noyoushoutterminateandrestartit.Howeverifyouinitiallyhaveissueskeepitrunningandfindinthemanaulhowtosuspendandresumeit.Youcanalsodothisfromhorizonifyouprefer.

5.5.9WHATISWARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://kvm.tacc.chameleoncloud.org:5000/v3/auth/tokens: HTTPSConnectionPool(host='kvm.tacc.chameleoncloud.org', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(' \< urllib3.connection.VerifiedHTTPSConnection object at 0x7f63fc14a040 \> : Failed to establish a new connection: [Errno 110] Connection timed out'))

https://kvm.tacc.chameleoncloud.org:5000/v3

THisishapeningwhenapreviouslyusedipaddressisusedonanothervmYouneedtoremovetheipfrom~/.ssh/kownhostsbyusing

ssh-keygen-RIPADDRESS

whereIPADDRESSistheaddresstoberemoved,andthantryagain

5.5.10SSH-KEYGENVSPUTTY

usessh-keygenasitisfareasier.

5.5.11CANIGIVEMYOWNNETWORKWHENBOOTING

Yes

5.5.12HOWDOIUPLOADTHESECGROUPS

5.5.13ARETHEREADDITIONALVIDEOS

5.5.13.1Startingvmsinhorizon

Avideowas shot byRahul. I do recommend however if you can just tokeepthekeynameonyourcomputertobeid_rsa.

Inhorizonyounaturallyuseakeynamesuchasrahul-222(e.g. whatrahulused)

Thereasonisthatinthefutureweusecloudmesnandwehaveadefaultkeyforcloudmeshsetasid_rsa.Thissimplifiesdebuggingincasesomethingdoesnotgoright

https://www.youtube.com/watch?v=-2RgDrhN02Q

cms vm boot --name=gregor-02 \ --image=CC-Ubuntu18.04 \ --network=e01ee12e-fd22-4b21-a050-d3e015f42bb1

cms sec group load --cloud=chameleon

https://www.youtube.com/watch?v=-2RgDrhN02Q

5.5.13.2StartingVMsincloudmesh

Rahul has done a video to showhow to start vms in cloudmeshAlsoweadded the network id to the cloudmesh.yaml file, so if you have a newversionitsnolongerneededtoaddthenetwork

https://www.youtube.com/watch?v=bDue67M1xl0

https://www.youtube.com/watch?v=bDue67M1xl0

6REFERENCES

☁

[1]G.vonLaszewski,Cloudcomputing,Fall2019.Bloomington,Indiana:Indiana University, 2019 [Online]. Available:https://laszewski.github.io/book/cloud/

[2] G. von Laszewski, Python for cloud computing, Fall 2019.Bloomington, Indiana: Indiana University, 2019 [Online]. Available:https://laszewski.github.io/book/python/

[3]G.vonLaszewski,Linuxforcloudcomputing,Fall2019.Bloomington,Indiana: Indiana University, 2019 [Online]. Available:https://laszewski.github.io/book/linux/

[4] G. von Laszewski, Scientific writing with markdown, Fall 2019.Bloomington, Indiana: Indiana University, 2019 [Online]. Available:https://laszewski.github.io/book/writing/

[5] G. von Laszewski, Cloud technologies, Fall 2019. Bloomington,Indiana: Indiana University, 2019 [Online]. Available: https://cloudmesh-community.github.io/book/vonLaszewski-cloud-technologies.epub?raw=true

[6] G. von Laszewski, “Project format example.” Aug-2019 [Online].Available: https://github.com/cloudmesh-community/proceedings-fa18/tree/master/project-report

[7] G. von Laszewski, E516 cloud computing engineering, Fall 2019.Bloomington, Indiana: Indiana University, 2019 [Online]. Available:https://laszewski.github.io/book/e516/

https://github.com/cloudmesh-community/book/blob/master/chapters/empty.md





https://cloudmesh-community.github.io/book/vonLaszewski-cloud-technologies.epub?raw=true

https://github.com/cloudmesh-community/proceedings-fa18/tree/master/project-report


Documents

e516 - Engineering Cloud Computing4.5.5 Exercise Cloud API 4.6 Week 6: REST 4.6.1 Lecture Material 4.6.2 Lab Activities 4.6.3 Graded Lab Activity 4.7 Week 7: MapReduce - 4.7.1 Lecture