View
44
Download
10
Category
Preview:
Citation preview
MicroserviceswithClojure
Developevent-driven,scalable,andreactivemicroserviceswithreal-timemonitoring
AnujKumar
BIRMINGHAM-MUMBAI
MicroserviceswithClojureCopyright©2018PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishingoritsdealersanddistributors,willbeheldliableforanydamagescausedorallegedtohavebeencauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
CommissioningEditor:RichaTripathiAcquisitionEditor:AiswaryaNarayananContentDevelopmentEditor:AkshadaIyerTechnicalEditor:AbhishekSharmaCopyEditor:SafisEditingProjectCoordinator:PrajaktaNaikProofreader:SafisEditingIndexer:FrancyPuthiryGraphics:JasonMonteiroProductionCoordinator:DeepikaNaik
Firstpublished:January2018
Productionreference:1230118
PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.
ISBN978-1-78862-224-0
www.packtpub.com
Tomymother,Mrs.InduSrivastava,myfather,Mr.DilipKumar,andtomylovelywife,Aishwarya,fortheircontinuoussupportandencouragement.AllthetimethatIhavespentonthisbookshouldhavebeenspentwiththem.FortripsthatwecanceledandforweekendsthatIspentatmydesk.
Tomyfamily,teachers,andcolleagues.Theyhaveextendedtheircontinuoussupport,providedcriticalfeedback,andmadeitpossibleformetofocusonthisbook.
mapt.io
Maptisanonlinedigitallibrarythatgivesyoufullaccesstoover5,000booksandvideos,aswellasindustryleadingtoolstohelpyouplanyourpersonaldevelopmentandadvanceyourcareer.Formoreinformation,pleasevisitourwebsite.
Whysubscribe?SpendlesstimelearningandmoretimecodingwithpracticaleBooksandVideosfromover4,000industryprofessionals
ImproveyourlearningwithSkillPlansbuiltespeciallyforyou
GetafreeeBookorvideoeverymonth
Maptisfullysearchable
Copyandpaste,print,andbookmarkcontent
PacktPub.comDidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewsletters,andreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
Contributors
AbouttheauthorAnujKumaristheco-founderandchiefarchitectofFORMCEPT,adataanalyticsstartupbasedinBangalore,India.Hehasmorethan10yearsofexperienceindesigninglarge-scaledistributedsystemsforstorage,retrieval,andanalytics.
Hehasbeeninindustryhacking,mainlyintheareaofdataintegration,dataquality,anddataanalyticsusingNLPandmachinelearningtechniques.HehaspublishedresearchpapersatACMconferences,gotafewpatentsgranted,andhasspokenatTEDx.
PriortoFORMCEPT,hehasworkedwiththeOracleServerTechnologiesdivisioninBangalore,India.
Iwouldliketothankmytechnicalreviewer,MichaelVitz,forhisvaluablefeedbackandthePackteditorialteamforanexcellentfeedbacklooptocomeupwithgoodqualitycontent.IwouldalsoliketothankmyteachersandFORMCEPTteammembers,whohavehelpedmeonvarioustopicscoveredinthisbook.Andespecially,Iwouldliketothankmyparents,mywife,andmyentirefamilyfortheircontinuousencouragement.
AboutthereviewerMichaelVitzhasmanyyearsofexperiencebuildingandmaintainingsoftwarefortheJVM.Currently,hismaininterestsincludemicroserviceandcloudarchitectures,DevOps,theSpringFramework,andClojure.
AsaseniorconsultantforsoftwarearchitectureandengineeringatINNOQ,hehelpsclientsbybuildingwell-craftedandvalue-providingsoftware.
HealsoisthewriterofacolumnintheGermanmagazine,JavaSPEKTRUM,wherehepublishesarticlesaboutJVM,infrastructure,andarchitecturaltopicseverytwomonths.
PacktissearchingforauthorslikeyouIfyou'reinterestedinbecominganauthorforPackt,pleasevisitauthors.packtpub.comandapplytoday.Wehaveworkedwiththousandsofdevelopersandtechprofessionals,justlikeyou,tohelpthemsharetheirinsightwiththeglobaltechcommunity.Youcanmakeageneralapplication,applyforaspecifichottopicthatwearerecruitinganauthorfor,orsubmityourownidea.
TableofContents
PrefaceWhothisbookisfor
Whatthisbookcovers
TogetthemostoutofthisbookDownloadtheexamplecodefiles
Conventionsused
GetintouchReviews
1. MonolithicVersusMicroservicesDawnofapplicationarchitecture
Monolithicarchitecture
MicroservicesDatamanagement
Whentousewhat
MonolithicapplicationstomicroservicesIdentifyingcandidatesformicroservices
Releasecycleandthedeploymentprocess
Summary
2. MicroservicesArchitectureDomain-drivendesign
Boundedcontext
Identifyingboundedcontexts
Organizingaroundboundedcontexts
ComponentsHexagonalarchitecture
MessagingandcontractsDirectmessaging
Observermodel
Servicecontracts
ServicediscoveryServiceregistry
Servicediscoverypatterns
DatamanagementDirectlookup
Asynchronousevents
Combiningdata
Transactions
AutomatedcontinuousdeploymentCI/CD
Scaling
Summary
3. MicroservicesforHelpingHandsApplicationDesign
Usersandentities
Userstories
Domainmodel
MonolithicarchitectureApplicationcomponents
Deployment
Limitations
MovingtomicroservicesIsolatingservicesbypersistence
Isolatingservicesbybusinesslogic
Messagingandevents
Extensibility
WorkflowsforHelpingHandsServiceproviderworkflow
Serviceworkflow
Serviceconsumerworkflow
Orderworkflow
Summary
4. DevelopmentEnvironmentClojureandREPL
HistoryofClojure
REPL
ClojurebuildtoolsLeiningen
Boot
ClojureprojectConfiguringaproject
Runningaproject
Runningtests
Generatingreports
Generatingartifacts
ClojureIDE
Summary
5. RESTAPIsforMicroservicesIntroducingREST
RESTfulAPIsStatuscodes
Namingconventions
UsingRESTfulAPIsviacURL
RESTAPIsforHelpingHandsConsumerandProviderAPIs
ServiceandOrderAPIs
Summary
6. IntroductiontoPedestalPedestalconcepts
Interceptors
Theinterceptorchain
ImportanceofaContextMap
CreatingaPedestalserviceUsinginterceptorsandhandlers
Creatingroutes
Declaringrouters
Accessingrequestparameters
Creatinginterceptors
Handlingerrorsandexceptions
Logging
Publishingoperationalmetrics
Usingchainproviders
Usingserver-sentevents(SSE)CreatinginterceptorsforSSE
UsingWebSocketsUsingWebSocketwithPedestalandJetty
Summary
7. AchievingImmutabilitywithDatomic
DatomicarchitectureDatomicversustraditionaldatabase
Developmentmodel
Datamodel
Schema
UsingDatomicGettingstartedwithDatomic
Connectingtoadatabase
Transactingdata
UsingDatalogtoquery
Achievingimmutability
Deletingadatabase
Summary
8. BuildingMicroservicesforHelpingHandsImplementingHexagonalArchitecture
Designingtheinterceptorchainandcontext
CreatingaPedestalproject
DefininggenericinterceptorsInterceptorforAuth
Interceptorforthedatamodel
Interceptorforevents
CreatingamicroserviceforServiceConsumerAddingroutes
DefiningtheDatomicschema
Creatingapersistenceadapter
Creatinginterceptors
Testingroutes
CreatingamicroserviceforServiceProviderAddingroutes
DefiningDatomicschema
Creatingapersistenceadapter
Creatinginterceptors
Testingroutes
CreatingamicroserviceforServicesAddingroutes
DefiningaDatomicschema
Creatingapersistenceadapter
Creatinginterceptors
Testingroutes
CreatingamicroserviceforOrderAddingroutes
DefiningDatomicschema
Creatingapersistenceadapter
Creatinginterceptors
Testingroutes
CreatingamicroserviceforLookupDefiningtheElasticsearchindex
Creatingqueryinterceptors
Usinggeoqueries
Gettingstatuswithaggregationqueries
CreatingamicroserviceforalertsAddingroutes
CreatinganemailinterceptorusingPostal
Summary
9. ConfiguringMicroservicesConfigurationprinciples
Definingconfigurationparameters
Usingconfigurationparameters
UsingOmniconfforconfigurationEnablingOmniconf
IntegratingwithHelpingHands
ManagingapplicationstateswithmountEnablingmount
IntegratingwithHelpingHands
Summary
10. Event-DrivenPatternsforMicroservicesImplementingevent-drivenpatterns
Eventsourcing
UsingtheCQRSpattern
IntroductiontoApacheKafkaDesignprinciples
GettingKafka
UsingKafkaasamessagingsystem
UsingKafkaasaneventstore
UsingKafkaforHelpingHandsUsingKafkaAPIs
InitializingKafkawithMount
IntegratingtheAlertServicewithKafka
UsingAvrofordatatransfer
Summary
11. DeployingandMonitoringSecuredMicroservicesEnablingauthenticationandauthorization
IntroducingTokensandJWT
CreatinganAuthserviceforHelpingHandsUsingaNimbusJOSEJWTlibraryforTokens
CreatingasecretkeyforJSONWebEncryption
CreatingTokens
Enablingusersandrolesforauthorization
CreatingAuthAPIsusingPedestal
MonitoringmicroservicesUsingELKStackformonitoring
SettingupElasticsearch
SettingupKibana
SettingupLogstash
UsingELKStackwithCollectd
Loggingandmonitoringguidelines
DeployingmicroservicesatscaleIntroducingContainersandDocker
SettingupDocker
CreatingaDockerimageforHelpingHands
IntroducingKubernetes
GettingstartedwithKubernetes
Summary
OtherBooksYouMayEnjoyLeaveareview-letotherreadersknowwhatyouthink
Preface
Themicroservicearchitectureissweepingtheworldasthedefactopatternforbuildingscalableandeasy-to-maintainweb-basedapplications.Thisbookwillteachyoucommonpatternsandpractices,showingyouhowtoapplythemusingtheClojureprogramminglanguage.ItwillteachyouthefundamentalconceptsofarchitecturaldesignandRESTfulcommunication,andshowyoupatternsthatprovidemanageablecodethatissupportableindevelopmentandatscaleinproduction.ThisbookwillprovideyouwithexamplesofhowtoputtheseconceptsandpatternsintopracticewithClojure.
Whetheryouareplanninganewapplicationorworkingonanexistingmonolith,thisbookwillexplainandillustratewithpracticalexampleshowteamsofallsizescanstartsolvingproblemswithmicroservices.Youwillunderstandtheimportanceofwritingcodethatisasynchronousandnon-blocking,andhowPedestalhelpsusdothis.Later,thebookexplainshowtobuildReactivemicroservicesinClojure,whichadheretotheprinciplesunderlyingtheReactiveManifesto.Wefinishoffbyshowingyouvarioustechniquestomonitor,test,andsecureyourmicroservices.Bytheend,youwillbefullycapableofsettingup,modifying,anddeployingamicroservicewithClojureandPedestal.
WhothisbookisforIfyouarelookingforwardtomigrateyourexistingmonolithicapplicationstomicroservicesortakingyourfirststepsintomicroservicearchitecture,thenthisbookisforyou.YoushouldhaveaworkingknowledgeofprogramminginClojure.However,noknowledgeofRESTfularchitecture,microservices,orwebservicesisexpected.
WhatthisbookcoversChapter1,MonolithicVersusMicroservices,introducesmonolithicandmicroservicearchitectureanddiscusseswhentousewhat.Italsocoversthepossiblemigrationplansofmovingfrommonolithicapplicationstomicroservices.
Chapter2,MicroservicesArchitecture,coversthebasicbuildingblocksofmicroservicearchitectureanditsrelatedfeatures.Itdiscusseshowtosetupmessagingandcontracts,andmanagedataflowsamongmicroservices.
Chapter3,MicroservicesforHelpingHandsApplication,introducesasampleHelpingHandsapplicationanddescribesthestepsthatwillbetakenintherestofthebooktobuildtheapplicationusingmicroservices.Further,thechaptercomparesandcontraststhebenefitsofusingamicroservices-basedarchitecturecomparedwithamonolithicone.
Chapter4,DevelopmentEnvironment,coversClojureandREPLatahighlevelandintroducestheconceptsofLeiningenandBoot—thetwomajorbuildtoolsforanyClojureproject.TheemphasiswillbeonLeiningenwithabasicintroductiontoBootonhowtosetupaClojureprojectforimplementingmicroservices.
Chapter5,RESTAPIsforMicroservices,coversthebasicsoftheRESTarchitecturalstyle,variousHTTPmethods,whentousewhat,andhowtogivemeaningfulnamestoRESTfulAPIsofmicroservices.ItalsocoversthenamingconventionsforRESTAPIsusingtheHelpingHandsapplicationasanexample.
Chapter6,IntroductiontoPedestal,coverstheClojurePedestalframeworkindetailwithalltherelevantfeaturesprovidedbyPedestal,includinginterceptorsandhandlers,routes,WebSockets,server-sentevents,andchainproviders.
Chapter7,AchievingImmutabilitywithDatomic,givesanoverviewoftheDatomicdatabasealongwithitsarchitecture,datamodel,transactions,andDatalogquerylanguage.
Chapter8,BuildingMicroservicesforHelpingHands,isastep-by-step,hands-on
guidetobuildandtestmicroservicesfortheHelpingHandsapplicationusingthePedestalframework.
Chapter9,ConfiguringMicroservices,coversthebasicsofmicroservicesconfigurationanddiscusseshowtocreateconfigurablemicroservicesusingframeworkssuchasOmniconf.Italsoexplainsthestepstomanagetheapplicationstateeffectivelyusingavailablestate-managementframeworkssuchasMount.
Chapter10,Event-DrivenPatternsforMicroservices,coversthebasicsofevent-drivenarchitecturesandshowshowtouseApacheKafkaasamessagingsystemandeventstore.Further,itdiscusseshowtouseApacheKafkabrokersandsetupconsumergroupsfortheeffectivecoordinationofmicroservices.
Chapter11,DeployingandMonitoringSecuredMicroservices,coversthebasicsofmicroservicesauthenticationusingJWTandhowtosetupareal-timemonitoringsystemusingtheELKStack.ItalsoexplainsthebasicconceptsofcontainersandorchestrationframeworkssuchasKubernetes.
TogetthemostoutofthisbookTheJavaDevelopmentKit(JDK)isrequiredtorunanddevelopapplicationsusingClojure.YoucangettheJDKfromhttp://www.oracle.com/technetwork/java/javase/downloads/index.html.Itisalsorecommendedthatyouuseatexteditororanintegrateddevelopmentenvironment(IDE)ofyourchoiceforimplementation.SomeoftheexamplesinthechaptersrequireLinuxasanoperatingsystem.
Downloadtheexamplecodefiles
Youcandownloadtheexamplecodefilesforthisbookfromyouraccountatwww.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisitwww.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.
Youcandownloadthecodefilesbyfollowingthesesteps:
1. Loginorregisteratwww.packtpub.com.2. SelecttheSUPPORTtab.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchboxandfollowtheonscreen
instructions.
Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:
WinRAR/7-ZipforWindowsZipeg/iZip/UnRarXforMac7-Zip/PeaZipforLinux
ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/Microservices-with-Clojure.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!
ConventionsusedThereareanumberoftextconventionsusedthroughoutthisbook.
CodeInText:Indicatescodewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandles.Hereisanexample:"ThepersistenceprotocolServiceDBconsistsofupsert,entity,anddeletefunctions."
Ablockofcodeissetasfollows:
{
"query":{
"term":{
"status":"O"
}
}
}
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:
(defnhome-page
[request]
(log/counter::home-hits1)
(ring-resp/response"HelloWorld!"))
Anycommand-lineinputoroutputiswrittenasfollows:
%leinrun
Noname|Hello,World!
%leinrunClojure
Clojure|Hello,World!
Bold:Indicatesanewterm,animportantword,orwordsthatyouseeonscreen.Forexample,wordsinmenusordialogboxesappearinthetextlikethis.Hereisanexample:"ClickontheCreateavisualizationbutton."
Warningsorimportantnotesappearlikethis.
Tipsandtricksappearlikethis.
GetintouchFeedbackfromourreadersisalwayswelcome.
Generalfeedback:Emailfeedback@packtpub.comandmentionthebooktitleinthesubjectofyourmessage.Ifyouhavequestionsaboutanyaspectofthisbook,pleaseemailusatquestions@packtpub.com.
Errata:Althoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyouhavefoundamistakeinthisbook,wewouldbegratefulifyouwouldreportthistous.Pleasevisitwww.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetails.
Piracy:IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,wewouldbegratefulifyouwouldprovideuswiththelocationaddressorwebsitename.Pleasecontactusatcopyright@packtpub.comwithalinktothematerial.
Ifyouareinterestedinbecominganauthor:Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,pleasevisitauthors.packtpub.com.
ReviewsPleaseleaveareview.Onceyouhavereadandusedthisbook,whynotleaveareviewonthesitethatyoupurchaseditfrom?Potentialreaderscanthenseeanduseyourunbiasedopiniontomakepurchasedecisions,weatPacktcanunderstandwhatyouthinkaboutourproducts,andourauthorscanseeyourfeedbackontheirbook.Thankyou!
FormoreinformationaboutPackt,pleasevisitpacktpub.com.
MonolithicVersusMicroservices
"Theoldorderchangethyieldingplacetonew"
-AlfredTennyson
Awell-designedmonolithicarchitecturehasbeenthekeytomanysuccessfulsoftwareapplications.However,microservices-basedapplicationsaregainingpopularityintheageoftheinternetduetotheirinherentpropertyofbeingautonomousandflexible,theirabilitytoscaleindependently,andtheirshorterreleasecycles.Inthischapter,youwill:
LearnaboutthebasicsofmonolithicandmicroservicesarchitecturesUnderstandthemonolithic-firstapproachandwhentostartusingmicroservicesLearnhowtomigrateanexistingmonolithicapplicationtomicroservicesCompareandcontrastthereleasecycleanddeploymentmethodologyofmonolithicandmicroservices-basedapplications
DawnofapplicationarchitectureEversinceAdaLovelace(https://en.wikipedia.org/wiki/Ada_Lovelace)wrotethefirstalgorithmforAnalyticalEngine(https://en.wikipedia.org/wiki/Analytical_Engine)inthe19thcenturyandAlanTuring(https://en.wikipedia.org/wiki/Alan_Turing)formalizedtheconceptsofalgorithmandcomputationviatheTuringmachine(https://en.wikipedia.org/wiki/Turing_machine),softwarehasgonethroughmultiplephasesinitsevolution,bothintermsofhowitisdesignedandhowitismadeavailabletoitsendusers.Theearliersoftwarewasdesignedtorunonasinglemachineinasingleenvironment,andwasdeliveredtoitsendusersasanisolatedstandaloneentity.Intheearly1990s,asthefocusshiftedtoapplicationsoftware,theindustrystartedexploringvarioussoftwarearchitecturemethodologiestomeetthedemandsofchangingrequirementsandunderlyingenvironments.Oneofthesoftwarearchitecturesthatwaswidelyadoptedwasmultitierarchitecture,whichclearlyseparatedthefunctionsofdatamanagement,businesslogic,andpresentation.Whentheselayerswerepackagedtogetherinasingleapplication,usingasingletechnologystack,runningasasingleprogram,itwascalledamonolithicarchitecture,stillinusetoday.
Withtheadventoftheinternet,softwarestartedgettingofferedasaserviceovertheweb.Withthischangeindeploymentandusage,itstartedbecominghardtoupgradeandaddfeaturestosoftwarethatadoptedamonolithicarchitecture.Technologystartedchangingrapidlyandsodidprogramminglanguages,databases,andunderlyinghardware.Companiesthatwereabletodisintegratetheirmonolithicapplicationsintoloosely-coupledservicesthatcouldtalktoeachotherwereabletoofferbetterservices,betterintegrationpoints,andbetterperformancetotheirusers.Theywerenotonlyabletoupgradetothelatesttechnologyandhardware,butalsoabletooffernewfeaturesandservicesfastertotheirusers.Theideaofdisintegratingamonolithicapplicationintoloosely-coupledservicesthatcanbedeveloped,deployed,andscaledindependentlyandcantalktootherservicesoveralightweightprotocol,wascalledmicroservices-basedarchitecture(https://en.wikipedia.org/wiki/Microservices).
CompaniessuchasNetflix,Amazon,andsoonhavealladoptedamicroservices-basedarchitecture.IfyoulookatGoogleTrendsintheprecedingscreenshot,youcanseethatthepopularityofmicroservicesisrisingdaybyday,butthisdoesn'tmeanthatmonolithicapplicationsareobsolete.Thereareapplicationsthatarestillsuitedformonolithicarchitecture.Microserviceshavetheiradvantages,butatthesametimetheyarehardtodeploy,scale,andmonitor.Inthischapter,wewilllookatbothmonolithicandmicroservices-basedarchitectures.Wewilldiscusswhentousewhatandalsotalkaboutwhenandhowtomigratefromamonolithictoamicroservices-basedarchitecture.
MonolithicarchitectureMonolithicarchitectureisanall-in-onemethodologythatencapsulatesalltherequiredservicesasasingledeployableartifact.Itworksonasingletechnologystackandisdeployedandscaledasasingleunit.Sincethereisonlyonetechnologystacktomaster,itiseasytodeploy,scale,andsetupamonitoringinfrastructureformonolithicapplications.EachteammemberworksononeormorecomponentsofthesystemandfollowsthedesignprincipleofSeparationofConcerns(SoC)(https://en.wikipedia.org/wiki/Separation_of_concerns).Suchapplicationsarealsoeasiertorefactor,debug,andtestinasinglestandalonedevelopmentenvironment.
Applicationsbasedonmonolithicarchitecturemayconsistofoneormoredeployableartifactsthatarealldeployedatthesametime.SuchamonolithicarchitectureisoftenreferredtoasaDistributedMonolith.
Forexample,averycommonmonolithicapplicationisawordprocessingapplication;MicrosoftWordisinstalledviaasingledeployableartifactandisentirelybuiltonMicrosoft.NETFramework(https://www.microsoft.com/net/).Therearevariouscomponentswithinwordprocessingapplication,suchastemplates,import/export,spell-checker,andsoon,thatworktogethertohelpcreateadocumentandexportittheformatofchoice.
Monolithicarchitectureappliesnotonlytostandaloneapplications,butalsotoclient-serverbasedapplicationsthatareprovidedasaserviceovertheweb.Suchclient-serverbasedapplicationshaveaclearlydefinedmultitierarchitecturethatprovidestherelevantservicestoitsendusersviaauserinterface.
Theuserinterfacetalkstoapplicationendpointsthatcanbeprogrammedusingwell-definedinterfaces.
Atypicalclient-serverapplicationmayadoptathree-tierarchitecturetoseparatethepresentation,businesslogic,andpersistencelayerfromeachother,asshownintheprecedingdiagram.Componentsofeachlayertalkstrictlytothecomponentsofthelayerbelowthem.Forexample,thecomponentsofthepresentationlayermaynevertalktothepersistencelayerdirectly.Iftheyneedaccesstodata,therequestwillberoutedviathebusinesslogiclayerthatwillnotonlymovethedatabetweenthepersistencelayerandthepresentationlayer,butalsodotherequiredprocessingtoservetherequest.Adoptingsuchacomponent-basedlayeredarchitecturealsohelpsinisolatingtheeffectofchangetoonlythecomponentsofdependentlayersinsteadoftheentireapplication.Forexample,changestothecomponentsofthebusinesslogiclayermayrequireachangeinthedependentcomponentsofthepresentationlayerbutcomponentsofthepersistencelayermayremainintact.
EventhoughamonolithicapplicationisbuiltonSoC,itisstillasingleapplicationonasingletechnologystackthatprovidesallrequiredservicestoitsusers.Anychangetosuchanapplicationrequirestobecompatiblewithalltheencapsulatedservicesandunderlyingtechnologystack.Inadditiontothat,itisnotpossibletoscaleeachserviceindependently.Anyscalingrequirementismetbydeployingmultipleinstancesoftheentiresystemasasingleunit.Ateam
workingonsuchamonolithicapplicationscalesovertimeandhastoadapttonewertechnologiesasawhole,whichisoftenchallengingduetotherapidlychangingtechnologylandscape.Iftheydonotchangewiththetechnology,theentiresoftwarebecomesobsoleteovertimeandisdiscardedduetoincompatibilitywithnewersoftwareandhardware,orashortageoftalent.
MicroservicesMicroservicesareafunctionalapproachwellappliedtosoftware.Ittriestodecomposetheentireapplicationfunctionallyintoasetofservicesthatcanbedeployedandscaledindependently.Eachservicedoesonlyonejobanddoesitwell.Ithasitsowndatabase,decidesitsownschema,andprovidesaccesstodatasetsandservicesthroughwell-definedapplicationprogramminginterfacesthatarebetterknownasAPIs,oftenpairedwithauserinterface.APIsfollowasetcommunicationprotocols,butservicesarefreetochoosetheirowntechnologystackandcanbedeployedonhardwareofchoice.
Inamicroserviceenvironment,asshownintheprecedingdiagram,therearenolayerslikeinmonoliths;instead,eachserviceisorganizedaroundaboundedcontext(https://en.wikipedia.org/wiki/Domain-driven_design#Bounded_context)thataddsabusinesscapabilitytotheapplicationasawhole.Newcapabilitiesinsuchanapplicationareaddedasnewservicesthataredeployedandscaledindependently.Eachuserrequestinamicroservices-basedapplicationmaycalloneormoreinternalmicroservicetoretrievedata,processit,andgeneratetherequiredresponse,asshowninthefollowingdiagram.Suchsoftwareevolvesfasterandhaslowtechnologydebt.Theydonotgetmarriedtoaparticulartechnologystackandcanadoptanewtechnologyfaster:
DatamanagementInamicroservices-basedapplication,databasesareisolatedforeachbusinesscapabilityandaremanagedbyonlyoneserviceatatime.AnyrequestthatneedsaccesstothedatamanagedbyanotherservicestrictlyusestheAPIsprovidedbytheservicemanagingthedatabase.Thismakesitpossibletonotonlyusethebestdatabasetechnologyavailabletomanagethebusinesscapability,butalsotoisolatethetechnologydebttotheservicemanagingit.However,itisrecommendedforthecallingservicetocacheresponsesovertimetoavoidtightcouplingwiththetargetserviceandreducethenetworkoverheadofeachAPIcall.
Forexample,aservicemanaginguserinterestsmightuseagraphdatabase(https://en.wikipedia.org/wiki/Graph_database)tobuildanetworkofusers,whereasaservicemanagingusertransactionsmightusearelationaldatabase(https://en.wikipedia.org/wiki/Relational_database)duetoitsinherentACID(https://en.wikipedia.org/wiki/ACID)propertiesthataresuitablefortransactions.ThedependentserviceonlyneedstoknowtheAPIstoconnecttotheservicefordataandnotthetechnologyoftheunderlyingdatabase.
Thisiscontrarytoamonolithiclayeredarchitecture,wheredatabasesareorganizedbybusinesscapability,whichmaybeaccessedbyoneormorepersistencemodulesbasedontherequest.Iftheunderlyingdatabaseisusingadifferenttechnology,theneachofthemodulesaccessingthedatabaseshavetocomplywiththesametechnology,thusinheritingthecomplexityofeachdatabasetechnologythatithasaccessto.
Databaseisolationshouldbedoneatthedatabaselevelandnotatthedatabasetechnologylevel.Avoiddeployingmultipleinstancesofthesamerelationaldatabaseorgraphdatabaseasmuchaspossible.Instead,trytoscalethemondemandandusetheisolationcapabilityofthesesystemstomaintainseparatedatabaseswithinthemforeachservice.
Theconceptofmicroservicesisverysimilartoawell-knownarchitecturecalled
service-orientedarchitecture(SOA)(https://en.wikipedia.org/wiki/Service-oriented_architecture).Inmicroservices,thefocusisonidentifyingtherightboundedcontextandkeepingthemicroservicesaslightweightaspossible.Insteadofusingacomplexmessage-orientedmiddleware(https://en.wikipedia.org/wiki/Message-oriented_middleware)suchasESB(https://en.wikipedia.org/wiki/Enterprise_service_bus),asimplemodeofcommunicationisusedthatisoftenjustHTTP.
"ArchitecturalStyle[ofMicroservices]isreferredtoasfine-grainedSOA,perhapsserviceorientationdoneright"
-MartinFowleronmicroservices
WhentousewhatThemonolithiclayeredarchitectureisoneofthemostcommonarchitecturesinuseacrossthesoftwareindustry.Monolithicarchitecturesarewellsuitedfortransaction-orientedenterpriseapplicationsthathavewell-definedfeatures,changelessoften,andhavecomplexbusinessmodels.Forsuchapplications,transactionsandconsistencyareofprimeimportance.Theyrequireadatabasetechnologywithbuilt-insupportforACIDpropertiestostoretransactions.Ontheotherhand,microservicesaresuitedbetterforSoftware-as-a-Service,internet-scaleapplicationsthatarefeature-firstapplicationswitheachfeaturefocusedonasinglebusinesscapability.Suchapplicationschangerapidlyandarescaledpartiallyperbusinesscapabilityondemand.Transactionsandconsistencyinsuchapplicationsarehardtoachieveduetomultipleservices,ascomparedtomonolithsthatareimplementedassingleapplications.
Itisrecommendedtostartwithawelldesigned,modularmonolithicapplicationirrespectiveofthedomaincomplexityortransactionalnature.Generally,allapplicationsstartasamonolithicapplicationthatcanbedeployedfasterasasingleartifactandlatersplitintomicroserviceswhentheapplication'scomplexitybeginstooutweightheproductivityoftheteam.
Theproductivityoftheteammaystartdecreasingwhenchangestothe
monolithicapplicationstartaffectingmorethanonecomponent,asshownintheprecedingdiagram.Thesechangesmaybearesultofanewfeaturebeingaddedtotheapplication,adatabasetechnologyupgrade,ortherefactoringofexistingcomponents.Anychangesmadetotheapplicationmustkeeptheentireteamin-sync,especiallythedeploymentteam,ifthereareanychangesrequiredinthedeploymentprocesses.Communicatingsuchchangesinalargeteamoftenresultsinacoordinationnightmare,multiplechangerequests,andin-turn,reducestheoverallproductivityoftheteamworkingontheapplication.
Productivityalsodependsontheinitialchoicesmadewithrespecttothetechnologystackanditsflexibilityofimplementation.Forexample,ifanewfeaturerequiresalibrarythatisreadilyavailablewithadifferenttechnologystackoraprogramminglanguage,itbecomeschallengingtoadoptasitdoesnotconformtotheexistingtechnologystackoftheapplicationcomponents.Insuchcases,theteamendsupimplementingthesamefeaturesetforthecurrenttechnologystackfromscratch,andthatinturnreducesproductivityandfurtheraddstothetechnologydebt.
Beforestartingwithmicroservices,firstsetupbestdesignprinciplesamongteammembers.Next,trytoevaluatetheexistingmonolithwithregardtocomponentsandtheirinteraction.Ifrefactoringcanhelpreducethedependencybetweenthecomponents,dothatfirstinsteadofdisintegratingyourapplicationintomicroservices.
MonolithicapplicationstomicroservicesMostapplicationsstartasamonolith.Amazon(http://highscalability.com/amazon-architecture)startedwithamonolithicPerl(https://en.wikipedia.org/wiki/Perl)/C++(https://en.wikipedia.org/wiki/C%2B%2B)application,andTwitter(http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)startedwithamonolithicRails(https://en.wikipedia.org/wiki/Ruby_on_Rails)application.Bothorganizationshavenotonlygonethroughmorethanthreegenerationsofsoftwarearchitecturalchanges,buthavealsotransformedtheirorganizationalstructuresovertime.Today,allofthemarerunningonmicroserviceswithteamsorganizedaroundservicesthataredeveloped,deployed,scaled,andmonitoredbythesameteamindependently.Theyhavemasteredcontinuousintegrationandcontinuousdeliverypipelineswithautomateddeployment,scaling,andmonitoringofservicesforreal-timefeedbacktotheteam.
IdentifyingcandidatesformicroservicesThetop-mostchallengeinmigratingfromamonolithicapplicationtomicroservicesistoidentifytherightcandidatesformicroservices.Awellstructuredandmodularizedmonolithicapplicationalreadyhaswell-definedboundaries(boundedcontexts)thatcanhelpdisintegratetheapplicationintomicroservices.Forexample,theUser,Orders,andInterestmodulesalreadyhavewell-definedboundariesandaregoodcandidatestocreatemicroservicesfor.Iftheapplicationdoesnothavewell-definedboundaries,thefirststepistorefactortheexistingapplicationtocreatesuchboundedcontextsformicroservices.Eachboundedcontextmustbetiedtoabusinesscapabilityforwhichaservicecanbecreated.
Anotherapproachinidentifyingtherightcandidatesformicroservicesistolookatthedataaccesspatternsandassociatedbusinesslogic.Ifthesamedatabaseisbeingupdatedbymultiplecomponentsofamonolithicapplication,thenitmakessensetocreateaservicefortheprimarycomponentwithassociatedbusinesslogicthatmanagesthedatabaseandmakesitaccessibletootherservicesviaAPIs.Thisprocesscanberepeateduntildatabasesandtheassociatedbusinesslogicaremanagedbyoneandonlyoneservicethathasasmallsetofresponsibilities,modeledaroundabusinesscapability.
Forexample,amonolithicapplicationconsistingofUser,Interest,andOrderscomponentscanbemigratedintomicroservicesbypickingonecomponentatatimeandcreatingamicroservicewithanisolateddatabase,asshownintheprecedingdiagram.Tostartwith,firstpicktheonewiththeleastdependency,theUsermodule,andcreatetheUserServiceservicearoundit.AllothercomponentscannowtalktothisnewUserServiceforUserManagement,includingauthentication,authorization,andgettinguserprofiles.Next,picktheOrdersmodulebasedontheleastdependencylogic,andcreateaservicearoundit.Finally,picktheInterestmoduleasitisdependentonboththeUserandOrdersmodules.Sincewehavethedatabasesisolated,wecanalsoswapoutthedatabaseforInterestwithmaybeagraphdatabasethatisefficienttostoreandretrieveuserinterestsduetoitsinherentcapabilityofstoringrelationshipsasagraph.
Inadditiontoorganizingyourmicroservicesaroundbusinesscapabilitiesanddatabaseaccesspatterns,lookforcommonareas,suchasauthentication,authorization,andnotification,thatcanbeperfectedonceasaserviceandcanbeleveragedbyoneormore
microserviceslater.
ReleasecycleandthedeploymentprocessOnceamonolithicapplicationisdisintegratedintomicroservices,thenextstepistodeploythemintoproduction.Monolithicapplicationaremostlydeployedasasingleartifact(JARs,WARs,EXEs,andmore)thatarereleasedafterextensivetestingbythequalityassurance(QA)team.Typically,developersworkonvariouscomponentsoftheapplicationandreleaseversionsfortheQAteamtopickandvalidateagainstthespecification,asshownundertheOrgStructureofmonolithicarchitectureinthefollowingdiagram.Eachiterationmayinvolvetheadditionorremovaloffeaturesandbugfixes.Thereleasegoesthroughmultipledevelopers(dev)andQAteamiterationsuntiltheQAteamflagsoffthereleaseasstable.OncetheQAteamflagsofftherelease,thereleasedartifactishandedovertotheITopsteamtodeployitinproduction.Ifthereareanyissuesinproduction,theITopsteamasksthedevteamtofixthem.Oncetheissuesarefixed,thedevteamtagsanewreleaseforQAthatagaingoesthroughthesamedev-QAiterationsbeforebeingmarkedasstableandeventuallyhandedovertoIT/ops.Duetothisprocess,anyreleaseforamonolithicapplicationsmayeasilytakeuptoamonth,oftenthreemonths.
Ontheotherhand,formicroservices,teamsareorganizedintogroupsthatfully
ownaservice.Theteamisresponsiblefornotonlydevelopingtheservice,butalsoforputtingtogetherautomatedtestcasesthatcantesttheentireserviceagainsteachchangesubmittedfortheservice.Sincetheserviceistobetestedinisolationforitsfeatures,itisfastertorunentiretestsuitesfortheserviceforeachchangesubmittedbythedevelopers.Additionally,theteamitselfcreatesdeployablebinariesoftenpackagedintocontainers(https://en.wikipedia.org/wiki/Linux_containers),suchasDocker(https://en.wikipedia.org/wiki/Docker_(software)),thatarepublishedtoacentralrepositoryfromwheretheycanbeautomaticallydeployedintoproductionbysomewell-knowntools,suchasKubernetes(https://en.wikipedia.org/wiki/Kubernetes).Theentiredevelopmenttoproductiontimelineiscutshorttodays,oftenhours,astheentiredeploymentprocessisautomated.WewilllearnmoreaboutdeployingmicroservicesinproductionandhowtousethesedeploymenttoolsinPart-4,thelastpartofthisbook.
Thereisareasonwhyalotofmicroserviceprojectsfailandonlyafewsucceed.Migratingfromamonolithicarchitecturetomicroservicesmustnotonlyfocusonidentifyingtheboundedcontexts,butalsotheorganizationalstructureanddeploymentmethodologies.Teamsmustbeorganizedaroundservicesandnotprojects.Eachteammustowntheservicerightfromdevelopmenttoproduction.Sinceeachteamownstheresponsibilityfortesting,validation,anddeployment,theentireprocessshouldbeautomatedandtheorganizationmustmasterit.Developmentanddeploymentcyclesmustbeshortwithimmediatefeedbackviafine-grainedmonitoringofthedeployedmicroservices.
Automationiskeyforanysuccessfulmicroservicesproject.Testing,deployment,andmonitoringmustbeautomatedbeforemovingmicroservicestoproduction.
SummaryInthischapter,welearnedaboutmonolithicandmicroservicesarchitecturesandwhymicroservicesarebecomingpopularintheindustry,especiallywithweb-scaleapplications.Welearnedabouttheimportanceofdatabaseisolationwithmicroservicesandhowtomigrateamonolithicapplicationtomicroservicesbyobservingthedatabaseaccesspattern.Wealsodiscussedtheimportanceofthemonolith-firstapproachandwhentomovetowardsmicroservices.Weconcludedwithacomparisonofmonolithicandmicroservicesarchitectureswithregardtothereleasecycleanddeploymentprocess.
Thenextchapterofthisbookwilltalkaboutmicroservicearchitectureindetail;wewilllearnmoreaboutdomain-drivendesignandhowtoidentifytherightsetofmicroservices.InChapter3,MicroservicesforHelpingHandsApplication,thelastchapterofPart-1,wewillpickareal-lifeusecaseformicroservicesanddiscusshowtodesignitusingtheprinciplesofmicroservicearchitecture.
MicroservicesArchitecture
"Gathertogetherthethingsthatchangeforthesamereasons.Separatethosethingsthatchangefordifferentreasons."
-RobertMartin,SingleResponsibilityPrinciple
Softwarearchitectureplaysakeyroleinidentifyingthebehaviorofthesystembeforeitisbuilt.Awell-designedsoftwarearchitectureleadstoflexible,reusable,andscalablecomponentsthatcanbeeasilyextended,verified,andmaintainedovertime.Sucharchitecturesevolveovertimeandhelppavethewayfortheadoptionofnext-generationarchitectures.Forexample,awell-designedmonolithicapplicationthatisbuiltontheprinciplesofSeparationofConcern(SoC)iseasiertomigratetomicroservicesthananapplicationthatdoesnothavewell-definedcomponents.Inthischapter,youwill:
LearnasystematicapproachtodesigningmicroservicesusingtheboundedcontextLearnhowtosetupcontractsbetweenmicroservicesandisolatefailuresLearnhowtomanagedataflowsandtransactionsamongmicroservicesLearnaboutservicediscoveryandtheimportanceofautomateddeployment
Domain-drivendesignIdealenterprisesystemsaretightlyintegratedandprovideallbusinesscapabilitiesasasingleunitthatisoptimizedforaparticulartechnologystackandhardware.Suchmonolithicsystemsoftengrowsocomplexovertimethatitbecomeschallengingtocomprehendthemasasingleunitbyasingleteam.Domain-drivendesignadvocatesdisintegratingsuchsystemsintosmallermodularcomponentsandassigningthemtoteamsthatfocusonasinglebusinesscapabilityinaboundedcontext(https://en.wikipedia.org/wiki/Domain-driven_design#Bounded_context).Oncedisintegrated,allsuchcomponentsaremadeapartofanautomatedcontinuousintegration(CI)processtoavoidanyfragmentation.Sincethesecomponentsarebuiltinisolationandoftenhavetheirowndatamodelsandschema,thereshouldbeawell-definedcontracttointeractwiththecomponentstocoordinatevariousbusinessactivities.
ThetermDomain-drivendesignwasfirstcoinedbyEricJ.Evansasthetitleofhisbookin2003.InPart-IV,Evanstalksabouttheboundedcontextandtheimportanceofcontinuousintegration,whichformsthebasisofanymicroservicesarchitecture.
BoundedcontextAdomainmodelisaconceptualmodelofabusinessdomainthatformalizesitsbehavioranddata.Asingleunifieddomainmodeltendstogrowincomplexitywithbusinesscapabilitiesandincreasesthecollaborationoverheadamongtheteamduetohighcoupling.Toreducecoupling,domain-drivendesignrecommendsdefiningamodelforeachbusinesscapabilitywithawell-definedboundarytoseparatethedomainconceptswithinthemodelfromtheonesoutside.Eachsuchmodelthenfocusesonthebehavioranddataconfinedtoasinglebusinesscapability,andthusgetsboundedbyasingleapplicationcontext,calledaboundedcontext.Monolithicapplicationstendtohaveaunifieddomainmodelfortheentirebusinessdomain,whereasformicroservices,domainmodelsaredefinedforeachidentifiedboundedcontext.
Forexample,insteadofdefiningasingleunifieddomainmodelforane-commerceapplication,itisbettertodividetheapplicationintoboundedcontextsofCustomer,Sales,andMarketinganddefineadomainmodelforeachofthesecontexts,asshownintheprecedingdiagram.Suchfocuseddomainmodelscanthenconquereachcontextbasedonbusinesscapabilities.Forexample,CustomerContextcanfocusonlyonuserandprofilemanagement,SalesContextcanhandleordersandtransactions,andMarketingContextcankeeptrackofuserinterestsforfocusedmarketing.
IdentifyingboundedcontextsOneofthemostchallengingtasksindesigningamicroservices-basedarchitectureistogettheboundedcontextrightforeachmicroservice.Itisaniterativeprocessthatrequiresathoroughunderstandingofbusinesscapabilitiesandbusinessdomain.Businesscapabilitiesmustnotbeconfusedwithbusinessfunctionsorprocesses.Businesscapabilitiestargetthewhatpartofabusinessandhaveanoutcome,whereasabusinessprocesstargetsthehowpart.Forexample,alertingisabusinesscapability,whereassendinganemailisabusinessprocess.Abusinesscapabilitymayincorporateoneormorebusinessprocesses.
Boundedcontextsmusttargetbusinesscapabilitiesandnotbusinessprocesses.Toidentifytherightboundedcontexts,itisrecommendedtostartwithamonolithicapplicationwithasingleunifiedmodelandanalyzeititerativelyovertimeforhighcouplingareas.Often,highcouplingareasaregoodtargetpointstosplitthedomainmodelintosub-domainmodelsthatcaninteractusingfixedcontractsandhelpreducetightcoupling.However,suchsub-domainsmustbefurthervalidatedagainstbusinesscapabilitiestomakesurethattheytargetonlyonebusinesscapability.
Microservicesmustbeorganizedaroundabusinesscapabilitywithinaboundedcontextandowntheirpresentation,businessdomain,andpersistencelayer.Theymusttakeresponsibilityfortheend-to-enddevelopmentstackincludingfunctions,datamodel,persistence,userinterface,andthecontracttoaccesstheserviceusingAPIs,oftenoverHTTP(S).
Organizingaroundboundedcontexts"Anyorganizationthatdesignsasystem(definedbroadly)willproduceadesignwhosestructureisacopyoftheorganization'scommunicationstructure"
-MelvynConway,1967
Generally,theorganizationstructurecontributesheavilytothedesignoftheapplication.Therefore,boundedcontextsshouldneverbeidentifiedonthebasisoftheexistingstructureoftheorganization.Instead,theorganizationmustbestructuredaroundboundedcontextssuchthattheentireteamcanworkonaserviceinisolation.
Atypicalorganization,workingonamonolithicapplication,isbuiltaroundlayersofpresentation,businesslogic,andpersistence,asshownintheprecedingdiagram.TheytendtohaveaseparateteamofUIdesignersandUI/UXexpertsforthepresentationlayer,ateamofbackenddeveloperstoimplementthedomainmodel,andateamofdatabaseadministratorstocreateadatabasefordeveloperstoaccess.Suchanorganizationstructureisidealforanapplicationwithasmallersetofbusinesscapabilities.
Oncetheapplicationgrows,anychangestotheapplicationrequirecommunicationtobemadeacrosstheteamofUIengineers,backenddevelopers,anddatabaseadministrators.Often,suchcommunicationleadstosomanyback-and-forthexchangesinvolvingdesigndocumentsandspecificationchangesthat
itbecomesoverwhelmingfortheteamstotranslatetherequirementscorrectlytotheimplementation.Suchcommunicationoverheadaddsdelaystotheprojectandbringsdowntheproductivityoftheentireteamworkingontheproductasawhole.
Boundedcontext,ifidentifiedcorrectly,solvesthisproblembylocalizingtheteamofUIdevelopers,backenddevelopers,anddatabaseadministratorstofocusonasinglebusinesscapability.Suchboundariesmakesurethatthecommunicationbetweentheteamsisboundedbyafixedcontractthatissetattheservicelevel.Thismakesitpossibletoreduceaconsiderablecommunicationoverhead,asanychangesmadetoaserviceareconfinedwithintheteamworkingontheservice.Forexample,thelocalizedteamofUserServiceandOrdersServicewillcommunicateonlytodiscusstheserviceAPIsthattheuserserviceisexposingforOrdersServicetogetthecustomerdetails.Sinceanychangestothecustomerschemaortheordersschemashouldnotimpacteachotherasperthedefinitionofboundedcontext,itisnotrequiredtocommunicatesuchchangestotheotherservice.
Components
Oncetheboundedcontextsareidentifiedformicroservicesandtheorganizationstructureisaligned,eachmicroservicemustbeconsideredasaproductthatistested,deployed,andscaledinisolationbythesameteamthatdevelopedit.Awell-designedmicroservicemustneverexposeitsinternaldatamodeltotheoutsideworlddirectly.Instead,itmustmaintainaservicecontractthatmapstoitsinternalmodelsuchthatitcanevolveovertimewithoutaffectingthedependentmicroservices.
Component-basedsoftwareengineering(https://en.wikipedia.org/wiki/Component-based_software_engineering)definesacomponentasareusablemodulethatisbasedontheprinciplesofSoCandencapsulatesasetofrelatedfunctionsanddata.Inthecontextofmicroservicesarchitecture,itisrecommendedtoimplementeachserviceasacomponentthatisindependentlyswappableanddeployablewithoutaffectinganyothermicroservices.
HexagonalarchitectureHexagonalarchitecture(http://alistair.cockburn.us/Hexagonal+architecture),alsoknownastheportsandadapterspattern,aimstodecouplebusinesslogicfromotherpartsofthecomponent,especiallythepersistenceandserviceslayers.Acomponent,builtontheportsandadapterspattern,exposesasetofportstowhichoneormoreadapterscanbeaddedasnecessary.Forexample,totestandverifythecorebusinesslogicinisolation,amockdatabaseadaptercanbepluggedinandlaterreplacedwitharuntimedatabaseadapterinproduction.
Aportisanentrypointthatisprovidedbythecorebusinesslogictointeractwithotherpartsofthecomponent.Anadapterisanimplementationofaport,andtheremaybemorethanoneadapterdefinedforasingleportbasedontherequirement.Forexample,aRESTadapterisusedtoacceptrequestsfromexternalusersorothermicroservicescomponents.ItinternallycallstheserviceAPIportdefinedbythecorebusinesslogicthatperformsthatrequestedoperationandgeneratesaresponse.Similarly,adatabaseadapterisusedbythecorebusinesslogictointeractwiththeexternaldatabaseviaitsdatabaseport,asshowninthefollowingdiagram:
Basedontheirapplicabilityandusage,theRESTadapteranddatabaseadapterareoftenreferredtoasprimaryandsecondaryadaptersrespectively.Similarly,theserviceAPIportanddatabaseportarereferredtoasprimaryandsecondaryportsrespectively.Primaryportsarecalledbytheprimaryadaptersandtheyactasthemaininterfacebetweenthecorebusinesslogicanditsusers,whereassecondaryportsandsecondaryadaptersareusedbythecorebusinesslogictogenerateeventsorinteractwithexternalservices,likeadatabase.Primaryadaptershelpvalidateservicerequestswithrespecttoaserviceschemaandcallcorebusinesslogicfunctions,whereasthecorebusinesslogiccallsthefunctionsofsecondaryadapterstohelptranslatetheapplicationschematotheexternalserviceschema,likethatofadatabase.Primaryadaptersareinitializedwiththeapplication,butreferencestothesecondaryadaptersarepassedtothecorebusinesslogicviadependencyinjection.
Thenamehexagonalarchitectureisderivedfromthestructureofacomponentthathassixports,butthatisnotarule.Theideaofrepresentingthearchitectureasahexagonisjusttoremovethenotionofone-dimensionallayeredarchitectureandhaveroomtoinsertportsandadaptersasrequired.
Messagingandcontracts
Inmonolithicapplications,messagingbetweencomponentsismostlyachievedusingfunctioncalls,whereasformicroservices,itisachievedusinglightweightmessagingsystems,oftenHTTP(S).Usingalightweightmessagingsystemisoneofthemostpromisingfeaturesofmicroservicesandmakesiteasiertoadoptandscale,ascomparedtoservice-orientedarchitecture(SOA)thatusesacomplexmessagingsystemwithmultipleprotocols.Microservicesaremoreaboutkeepingtheendpointssmartandthecommunicationchannelsassimpleaspossible.
Inamicroservicesarchitecture,oftenmultiplemicroservicesneedtointeractwitheachothertoachieveaparticulartask.Theseinteractionscanbeeitherdirect,viarequest-response-based(https://en.wikipedia.org/wiki/Request-response)communication,orthroughalightweightmessage-orientedmiddleware(MOM)(https://en.wikipedia.org/wiki/Message-oriented_middleware).Directmessagingissynchronous,thatis,therequesterwaitsfortheresponsetobereturned,whereasamessage-orientedmiddlewareisprimarilyusedforasynchronouscommunication.
DirectmessagingIndirectmessaging,eachrequestissentdirectlytothemicroserviceonitsAPIendpoint.Suchrequestsmaybeinitiatedbyusers,applications,orbyothermicroservicesthatintegratewiththetargetmicroservicetocompleteaparticulartask.MostlytheendpointstohandlesuchrequestsareimplementedusingREST(https://en.wikipedia.org/wiki/Representational_state_transfer),whichallowsresourceidentifierstobeaddresseddirectlyviaHTTPbasedAPIswithasimplemessagingstyle.
RESTisanarchitecturalstylewithpredefinedoperationsbasedonHTTPrequestmethods,suchasGET,PUT,POST,andDELETE.ThestatelessnatureofRESTmakesitfast,reliable,andscalablewithmultiplecomponents.
Forexample,inatypicalmicroservices-basede-commerceapplication,anadministrativeusermaywishtocreate,update,ormanageusersviatheUserService'sRESTendpoint.Inthatcase,theusercandirectlycalltheRESTAPIoftheUserService,asshowninthefollowingdiagram:
Similarly,amobileapplicationmaywanttoqueryuserinterestsviaInterestService'sRESTendpointofthee-commerceapplication.SincetheInterestServicedependsonboththeUserServiceandtheOrdersService,itmayfurtherinitiatetwoseparatecallstotheRESTendpointofthesetwoservicestogetthedesiredresponsewhichitcanmergewiththeuserinterestsdataandgeneratetheresponsefortherequestingapplication.Mostly,allthesekindsofrequest-responsearesynchronousinnatureasthesenderexpectsaresponsewiththeresultsinthesamecall.Asynchronousrequestsaremostlyusedtosendalertsormessagestorecordanoperation,andinthosecases,thesenderdoesn'twaitfortheresponseandexpectstheactiontobetakeneventually.
Avoidbuildinglongchainsofsynchronouscallsasthatmayleadtohighlatencyandanincreasedpossibilityoffailureduetomultiplehopsofintermediaterequestsandnetworkround-tripsamongtheparticipatingservices.
MessageformatsusedwithRESTendpointsaremostlytext-basedmessageformatssuchasJSON,XML,orHTML,assupportedbytheendpointimplementation.BinarymessageformatssuchasThrift(https://en.wikipedia.org/wiki/Apache_Thrift),ProtoBuf(https://en.wikipedia.org/wiki/Protocol_Buffers),andAvro(https://avro.ap
ache.org/)arealsopopularduetotheirwidesupportacrossmultipleprogramminglanguages.
Usedirectmessagingonlyforsmallermicroservices-baseddeployments.Forlargerdeployments,itisadvisabletogowithAPIgatewaysthatactasthemainentrypointforallclients.SuchAPIgatewayshelpmonitorrequestsformicroservicesandalsoassistinthemaintenanceandupgradeoperations.
ObservermodelTheobservermodelusesamessagebroker(https://en.wikipedia.org/wiki/Message_broker)atitscoretosendmessagesamongmicroservices.Amessagebrokerprovidescontentandtopic-basedroutingusingthepublish-subscribepattern(https://en.wikipedia.org/wiki/Publish-subscribe_pattern),whichmakesthesenderandreceiverindependentofeachother.Allobservingmicroservicessubscribetooneormoretopicsthroughwhichtheycanreceivethemessagesandalsoconnecttotopicsonwhichtheycanpublishthemessagesforotherobservers.Allinteractionsdoneviamessagebrokersareasynchronousinnatureanddonotblockthesender.Thishelpsinscalingboththepublishersandsubscribersindependently.Amicroservicesarchitecturethatisbuiltononlyasynchronousandnon-blockinginteractionsusingmessagebrokerscalesverywell.
Messagebrokersarealsousedtomanageworkloadsinscenarioswheretherateofpublishedmessagesishigherthantherateatwhichthesubscriberisabletoprocessthemessages.Messagebrokersalsoprovidereliablestorage,multipledeliverysemantics(atleastonce,exactlyonce,andsoon),andalsotransactionmanagementthatisusefulfordatamanagementacrossmicroservices.BinarymessageformatssuchasThrift,ProtoBuf,andAvroarepreferredovertextformatsformessagebrokers.
Intheobservermodel,alltherequestsgeneratedbyusers,applications,ormicroservicesarepublishedonatopictowhichoneormoremicroservicescansubscribeandreceivethemessageforprocessing,asshownintheprecedingdiagram.Thegeneratedresultscanalsobewrittenbacktoatopicthatcanbelaterpickedbyanothermicroservice,whichmayeitherreporttheresponsebacktotheapplicationorpersistitwithinadatastore.Ifasubscriberfails,themessagebrokercanreplaythemessage.Similarly,ifallsubscribersarebusy,themessagebrokercanaccumulatethemessagesuntiltheyareprocessedbythesubscribers.
Theobservermodelhelpstoachievebetterscalabilityascomparedtodirectmessagingattheexpenseofasinglepointoffailureofthemessagebroker.
Servicecontracts
Contractsarerequiredforentitiestointeractwitheachother.Theydefinethemessageformatandmediumofcommunicationfortheparticipatingentities.Inamonolithicenvironment,itissimplerforcomponentstointeractwiththetargetcomponentsusingtheinterfacesandfunctionsexposedbythem.Sinceafunctionclearlydefinesitsmessageformatasinputparameters,themessagepassingbetweenthecomponentscanbedonebyjustafunctioncallwiththerequiredinputparameters.Functioncallsarefurthersimplifiedinamonolithduetoacommonunderlyingtechnologystack.Contractsarealsoeasiertomaintainformonolithicapplicationsbecauseanychangedonetothecontractistestedandverifiedacrosscomponentsforcompatibility.Suchapplicationsarealsoversionedasawholeandnotper-component.Thefollowingtablecomparesandcontrastsmonolithicandmicroservicearchitecturalstyles:
Monolithicarchitecture Microservices
Entity Components Servicesbycapability
EndpointInterfacesandfunctions
RESTURIs(HTTP)/Thrift/Avro/Protobuf
Medium Functioncalls
HTTP/publish-subscribeviamessagebroker(observer)
Contract Functiondefinition
APIspecification(Swagger,RAML)/messageserialization(ThriftIDL,AvroSchema)
Version Singleversion Separateversionsforeachservice
Technology Single Polyglot
Ascomparedtoamonolith,inamicroservices-basedenvironment,thereisa
servicedeployedforeachbusinesscapabilitythatmayormaynotbeusingthesametechnologystack.Insuchcases,servicecontractsbecomeabsolutelymandatoryformicroservicestounderstandthemessageformatsandcommunicationmediumacceptedbyothermicroservicesandinteractwiththem.Moreover,thesemessageformatsneedtobelanguage-agnostictoallowanymicroservicetocommunicateirrespectiveofthetechnologystackinwhichtheyareimplemented.
Microservicesshouldneverexposetheirinternaldatamodeldirectlyasapartofamessagecontracttotheexternalworld.Theinternaldatamodelmustbedecoupledfromtheexternalservicecontractandthereshouldbeawaytoconvertandvalidatethecontractatentryandexit.Thishelpstoevolvethedatamodelofamicroserviceinisolationwithoutaffectingthecontractwithothermicroservices.Ifthereisachangerequiredintheservicecontract,itmustbeversionedandeachversionofthecontractmustbesupportedbythemicroserviceaslongasitisinusebyanyexternalservice.Aversionshouldbediscardedonlywhentherearenootherservicesusingtheobsoleteversionandthereisnolongeraneedtorollbacktheservicetoitspreviousversion.
Avoidmultipleversionsofservicecontracts(andmessageformats)asmuchaspossible.Chooseaflexiblemessageformatthatcanevolveovertimewithoutbreakingpreviousversions.
MicroservicesthatexposeRESTAPIsprimarilyusetheRESTAPIdefinitionandHTTPverbs(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods)todefinewell-formedURIs.ServicecontractsforREST-basedAPIsaredefinedusingframeworkssuchasSwagger(https://swagger.io/)andRAML(https://raml.org/).MicroservicesthatusetheobserverpatterntendtoacceptmessagesinThrift,Avro,orProtoBufformats.Eachoftheseframeworkshasawaytodefinelanguage-agnosticspecificationsandsupportsmostofthepopularprogramminglanguages.
ServicediscoveryAlltheAPIsexposedbyamicroserviceareaccessibleviatheIPaddressandportofthehostmachineonwhichthemicroserviceisdeployed.SincemicroservicesaredeployedinavirtualmachineoracontainerthathasadynamicIP,itisquitepossiblethattheIPsandportsthatareallocatedtothemicroserviceAPIsmaychangeovertime.Therefore,IPaddressesandportsofservicesshouldneverbehardcodedbythedependingmicroservice.Instead,thereshouldbeacommondatabaseofalltheservicesthatareactiveforthecurrentapplication.Suchadatabaseofservicesiscalledtheserviceregistryinmicroservicesarchitectureandisalwayskeptuptodatewiththelocationofthemicroservices.ItkeepsthedetailsofalltheactivemicroservicesincludingtheircurrentIPaddressandport.Microservicesthenquerythisserviceregistrytodiscoverthecurrentlocationoftherequiredmicroservicesandconnecttothemdirectly.
ServiceregistryTheserviceregistryactsasadatabaseofmicroservices.ItmusthaveadedicatedstaticIPaddressorafixedDNSnamethatmustbeaccessiblefromalltheclients,asshowninthefollowingdiagram.Sincealltheclientsdependonaserviceregistrytolookupthetargetservices,italsobecomesasinglepointoffailurefortheentiremicroservicesarchitecture.Therefore,theimplementationoftheserviceregistrymustbeextremelylightweightandshouldsupporthighavailabilitybydefault.SomecommontoolsthatcanbeusedasaserviceregistryareApacheZookeeper(http://zookeeper.apache.org/),etcd(https://github.com/coreos/etcd),andconsul(https://www.consul.io/):
Tokeeptheregistryuptodate,microservicesshouldeitherimplementthestartup/shutdowneventtoregister/deregisterwiththeserviceregistrythemselvesorthereshouldbeanexternalserviceconfiguredtokeeptrackofservicesandkeeptheregistryuptodate.Someorchestrationtools,suchasKubernetes(https://kubernetes.io/),supportserviceregistryoutoftheboxandmaintaintheregistryfortheentireinfrastructure.
Clientsmustcachethelocationoffrequentlyusedmicroservicestoreducedependencyontheserviceregistry,butthelocationmustbesyncedperiodicallywiththeserviceregistryforup-to-dateinformation.
ServicediscoverypatternsMicroservices-basedapplicationsmayoftenscaletosuchalargenumberofservicesthatitmaynotbefeasibleforeachmicroservicetokeepatrackofallotheractiveservicelocations.Insuchscenarios,theserviceregistryhelpsindiscoveringmicroservicestoperformaparticulartask.Thereareprimarilytwopatternsforservicediscovery—client-sidediscoveryandserver-sidediscovery,asshowninthefollowingdiagram.
Intheclient-sidediscoverypattern,theresponsibilityfordeterminingthelocationofservicesbyqueryingtheserviceregistryisontheclient.Therefore,theserviceregistrymustbeaccessibletotheclienttolookupthelocationoftherequiredservices.Also,eachclientmusthaveservicediscoveryimplementationbuilt-inforthispatterntowork.
Ontheotherhand,intheserver-sidediscoverypattern,theresponsibilityforconnectingwiththeserviceregistryandlookingupthelocationofservicesisofarouteroragatewaythatactsasaloadbalanceraswell.Clientsjustneedtosendarequesttoarouterandtheroutertakescareofforwardingtherequesttotherequiredservice.OrchestrationtoolssuchasKubernetessupportserver-sidediscoveryusingproxies.
Theserver-sidediscoverypatternmustbepreferredforalarge-scaledeployment.Itcanalsobeusedasacircuitbreakertopreventresourceexhaustionbycontrollingthenumberofopenrequeststoaservicethathasencounteredconsecutivefailuresorisnotavailable.
DatamanagementInamicroservices-basedarchitecture,thedatamodelandschemamustnotbesharedamongboundedcontexts.Eachmicroservicemustimplementitsowndatamodelbackedbyadatabasethatisaccessibleonlythroughtheserviceendpoints.Microservicesmayalsopublisheventsthatcanbeconsideredasalogofthechangestheserviceappliestoitsisolateddatabase.Keepingapplicationdatauptodateacrossmicroservicesmayalsoaddtothenetworkoverheadanddataduplication.
DirectlookupAlthoughmicroserviceshavetheirownisolatedpersistence,anapplicationimplementedusingmicroservicesmayneedtosharedataamongasetofservicestoperformtasks.Inamonolithicenvironment,sincethereisacommondatabase,itiseasiertosharedataandmaintainconsistencyusingtransactions.Inamicroservicesenvironment,itisnotrecommendedtoprovidedirectaccesstothedatabasemanagedbyaservice,asshowninthefollowingdiagram:
Forexample,whenauserplacesaneworder,theOrderServicemayneedaccesstothedeliveryaddress;thatis,theuseraddressfromtheUserService.Similarly,oncetheorderisplaced,theUserServicewouldliketoupdatethetotalordersplacedtilldatebytheuserinitsdatabase.SincetheuserdatabaseismaintainedbytheUserServiceandtheOrdersDatabaseismaintainedbytheOrdersService,thesetwoserviceswillgettherequireddetailsviatheAPIsexposedbytheotherserviceonly.Theyshouldnotbeallowedtodirectlyupdateoraccessdatabasesmaintainedbyanotherservice.Thishelpsinalwaysmaintainingasinglecopyoftheuserandorderdatabasesthatisnotaccessibletoanyothermicroservicedirectly.
AsynchronouseventsGettingdataviaserviceendpointssynchronouslymaybecomeoverwhelmingforservicesthatmaintainawidelyuseddatabase,liketheusersdatabase.Therefore,itisrecommendedforservicestomaintainaread-onlycacheforsuchdatabasesandkeepituptodateasynchronouslyusingevents,asshowninthefollowingdiagram:
Forexample,insteadoflookinguptheaddressorordercountusingserviceendpointssynchronously,servicessuchasUserServiceandOrdersServicecanpublishtheeventsofinterestonamessagequeueinorderofoccurrence.TheUserServicecanthenreceivetheorderseventfromtheOrdersServiceviatheMessageBrokerandupdateitsdatabasewiththeorderscountorcacheit.Similarly,theOrdersServicecanreceiveanyaddressupdateeventfromtheUserService,keeptheaddressuptodatefortheuserwithinitscache,andrefertoitasandwhenrequiredtogenerateordersforusers.
Microservicesshouldalwayshaveanisolateddatabase,butitisnotrecommendedtocreateseparateservicestoisolateimmutabledatabasessuchasgeolocations,PINcodes,domainknowledge,andsoon.Sincethesedatabasedonotchangethatoften,itisfair
enoughtoshareandcachetheseacrossmicroservices.
CombiningdataInamonolithicenvironment,combiningdataiseasy;youneedtojustjointwotablestocreatetherequiredview.Inmicroservices,datasetsaredistributedacrossmicroservicesandcombiningthemrequiresmovingthedataacrossmicroservices,whichmayinvolvesignificantnetworkandstorageoverhead.Italsobecomeschallengingtokeepthecombineddatauptodate.Therearemultiplewaystosolvetheproblemofcombiningdataorjoinsinamicroservicesarchitecturebasedonthescopeoftherequest.
Forexample,ifyouwishtobuildanordersummarypageforaparticularuser,youneedtogetonlythatuser'sdatafromtheUserServiceandalltheordersforthatuserfromtheOrdersService.Thesecanbeobtainedindependentlyandjoinedattherequestingserviceleveltogeneratetheordersummary,asshownintheprecedingdiagram.Thesekindsofjoinworkwellfor1:Njoins.
Real-timejoinsworkwellforlimiteddatasets,butitisexpensivetocombinedatainrealtimeforeachrequest.ImaginetensofthousandsofsimilarrequestshittingtheOrderSummaryServiceeverysecond.Insuchscenarios,services
shouldinsteadkeepdenormalized(https://en.wikipedia.org/wiki/Denormalization)combineddatainacachethatiskeptuptodateusingtheeventsgeneratedbythesourceservices.Theservicecanthenrespondtotherequestsbyjustlookingupthisdenormalizeddatacacheinrealtime.Thisapproachscaleswellattheexpenseofdatabeingnearrealtime.Thedatainthecachemightbeoffbythetimesourceservicegeneratestheeventandtargetservicepicksitupandmakeschangestoitscache.
Forexample,asshownintheprecedingdiagram,anInterestServicemayreceiveuserinterestsviaitsAPIendpoint,butitmayneedtheuserandorderdetailsfromtheUserandOrdersservicesrespectively.Insteadofdirectlylookingupdetailsforeachuserinterest,theInterestServicemaysubscribetotheeventsgeneratedbytheuserandordersserviceandinternallykeepadenormalizedcacheviewofinterestdatathatisreadilyavailablewithalltherequireddetailsofusersandorders.
TransactionsEachmicroservicecanuseadatabaseofitschoice.ThechosendatabasesmayormaynothavetheACIDproperty(https://en.wikipedia.org/wiki/ACID)andsupporttransactions.Thisisoneofthereasonswhydistributedtransactionsarehardtoimplementwithmicroservices.However,businesstransactionsinvolvingchangesacrossmultiplebusinessentitiescannotbeomittedentirely,andthereforemicroservicesimplementdistributedtransactionsbyusingdataworkflows,asshowninthefollowingdiagram:
Microservicespublisheventswhenevertheymakeachangetothedatabase.Theeventscontainthetypeofchangealongwithimmutabledataaboutthebusinessentitiesthatwereaffectedbythischange.Otherservicesthenlistentotheseeventsasynchronouslyandperformthechangesstrictlyintheorderinwhicheventswerepublished.Asingletransactionmaycontainoneormoreeventsthatmayresultincascadingeventsgeneratedbythemicroservicesthatareaffectedbyit.Duetotheasynchronousnatureoftheeventflow,theconsistencyachievedacrossmicroservicesinthiscaseiseventual(https://en.wikipedia.org/wiki/Eventual_consistency).
Ifatransactionfails,theservicethatencountersthefailuregeneratescompensatoryeventstonullifythechangesmadeacrossmicroservicesthathavealreadyprocessedthetransactioneventsinthechain.Thecompensatoryeventsflowbackwardstowardstheoriginofthetransaction,asshownintheprecedingdiagram.Compensatoryeventsareidempotentinnatureandretrieduntiltheysucceed.
ThetransactionpatternformicroservicesisinspiredbySagas(http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf)andwasproposedbyHectorGarcia-MolinaandKennethSalem,aspublishedinanACMpaperin1987.Sagasmaybeimplementedasasetofworkflows,whereateachstepoffailureacompensatingactionistriggeredtobringthesystembacktoitsoriginalstateasitwasbeforetheworkflowwastriggered.
AutomatedcontinuousdeploymentThecorephilosophyofamicroserviceenvironmentmustbebasedontheYoubuildit,yourunit(https://queue.acm.org/detail.cfm?id=1142065)model;thatis,theteamworkingonthemicroservicemustownitendtoend,rightfromdevelopmenttodeployment.Theinfrastructurerequiredfortheteamtointegrate,test,anddeployanychangesmustbecompletelyautomated.Thismakesitpossibletobuildacontinuousintegrationandcontinuousdelivery(CD)pipeline,whichisthebackboneofmicroservices-basedarchitectures.
CI/CDThetermCI/CDcombinesthepracticesofCIandCDtogetherasaseamlessprocess.Inatypicalmicroservices-basedsetup,theteamcontinuouslyworksonenhancingthefeaturesofthemicroserviceandfixingtheissuesthatareencounteredinproduction.
Theentirecyclefromdevelopmenttodeploymenthasthreemajorphases,asshowninthefollowingdiagram:
Inthefirstphase,theteamcommitsthechangestoaversioncontrolrepository.Theversioncontrolrepositoryusedformicroservicesshouldbecommonforalltheservicesandapplications.Consolidatingalltheimplementationsinasingleversioncontrolsystemhelpsinautomationandrunningapplication-wideintegrationandacceptancetests.Someversioncontrolservicesalsosupportafine-grainedcollaborationsystemthatallowsthedeveloperstonotonlysharetheirchangeswithagroupofdevelopers,butalsohaveactivereviewsandafeedbacksystemwithintheteambeforethechangesarecommitted.
VersioncontrolsystemslikeGitareidealforadistributedteamworkingonmultiplemicroservicesduetoitsinherentfeatures.ServicelikeGitHubandBitbucketaresomecloudhosting
providersforGitthatalsohavethecapabilitytobuildtriggersforCI/CDsystems.
Inthesecondphase,aCI/CDsystemsuchasJenkinsisusedtobuildthechangesandruntheunittestsforthemicroserviceforwhichthechangeiscommitted.Runningthetestsforeachchangerequesthelpsindetectingissuesbeforethechangesareintegratedwiththerestoftheapplication.Italsohelpstodetectanyregressions(https://en.wikipedia.org/wiki/Software_regression)thatmighthavebeenintroducedduetotherecentchanges.Ifthereareanytestfailures,analertissentbacktotheteam,especiallythecommitterwhosubmittedthechangerequest.
Theteamthenfixesthetestsandsendsthechangerequestagaintotheversioncontrolsystemthatin-turntriggersthebuildtoretestthechanges.Thisprocessrepeatsuntilallthetestssucceed.Oncethetestssucceed,theCI/CDsystemmergesthechangeswiththemainlineandpreparesareleaseartifactfortheservice.Artifactsformicroservicesareoftenpackagedascontainersthatarereadilydeployablebyorchestrationtools.Packagingmicroservicesinacontaineralsohelpsinautomatingthedeploymentandscalingoftheserviceson-demand.
Dockerisonepreferredtechnologyforpackagingmicroservices.IthasseamlessintegrationwithmultipleorchestrationtoolssuchasKubernetesandMesos(http://mesos.apache.org/).
Inthethirdphase,theCI/CDsystempublishesthereleasestoacentralrepositoryandinstructstheorchestrationtoolstopickthelatestversionofthemicroservicethatcontainstherecentchanges.Theorchestrationenginethenpullsthelatestreleasefromtherepositoryanddeploysitinproduction.Alltheinstancesinproductionaremonitoredbyautomatedtoolsthatgeneratealertsfortheteamifthereareanyissues.Ifthereareanyissuesencounteredbytheteam,theteamfixestheissuesandsubmitsthechangerequesttotheversioncontrolsystemthattriggersthebuildandtheentireprocessrepeatstopushthechangestoproduction.
Generally,theorchestrationenginedoesarollingupgradeoftheservicewhiledeployingupdates,butsometeamsprefertodotheA/Btestingbyupgradingonlyasubsetofdeployedserviceinstancesandroll-outonlywhenthetestssucceedforthatsubset.SuchdeploymentsareoftenreferredtoasBlueGreenDeployment(h
ttps://martinfowler.com/bliki/BlueGreenDeployment.html).
Suchanautomatedenvironmenthelpstheteamcutshorttheentiredevelopment-to-deploymentcyclefrommonthstodaysanddaystohours.LargecompaniessuchasGoogle,Facebook,Netflix,Amazon,andsoonarenowabletopushmultiplereleasesinadayduetosuchautomatedenvironmentsandrobusttestingprocesses.
ScalingTheArtofScalability(http://theartofscalability.com/)bookusesascalecubemodeltodescribethreeprimaryscalingpatternsforanapplication,asshowninthefollowingdiagram.Thex-axisofthecuberepresentshorizontalscaling;thatis,deployingthesameinstanceoftheapplicationbyjustcloningthemandfront-endingbyaloadbalancertodistributetheloadevenlyamongtheinstances.Thisscalingpatternisquitecommonforhandlingahighnumberofservicerequests.Thez-axisofthecubeaddressesscalingbydatapartitioning.Inthiscase,eachapplicationinstancedealswithonlyasubsetofdata.Thisscalingpatternisparticularlyusefulforapplicationswherethepersistencelayerbecomesabottleneck:
They-axisofthecubeaddressesscalingbysplittingtheapplicationbyfunctionorservice.Thispatternrelatesdirectlytothemicroservicespattern.Thefaceofthecubecreatedusingthexy-axiscombinesthebestpracticesofscalingformicroservices-basedarchitecture.Microservicesareidentifiedbysplittinganapplicationbyboundedcontexts(y-axis)andscaledbycloningeachinstance(x-axis).Formicroservices,cloningisdonebydeployingmultipleinstancesofaservicecontainer.
SummaryInthischapter,welearnedaboutdomain-drivendesignandtheimportanceofidentifyingtherightboundedcontextformicroservices.Welearnedaboutthehexagonalarchitectureformicroservicesandvariousmessagingpatterns.Wealsodiscusseddatamanagementpatternsformicroservicesandhowtosetupserviceregistriestodiscovermicroservices.Weconcludedwiththeimportanceofautomatingtheentiremicroservicedeploymentcycleincludingtesting,deployment,andscaling.Inthenextchapter,wewillintroduceareal-lifeusecaseformicroservicesandlearnhowtodesignanapplicationusingtheconceptslearnedinthischapter.
MicroservicesforHelpingHandsApplication
"Welearnbyexampleandbydirectexperiencebecausetherearereallimitstotheadequacyofverbalinstruction."
-MalcolmGladwell,Blink:ThePowerofThinkingWithoutThinking
Microservicesaregainingpopularityforinternet-scaleapplicationsthattargetconsumers.Inthischapter,youwilllearnhowtoapplyprinciplesofmicroservicesarchitecturetodesignasimilar,internet-scale,fictitiousapplicationcalledHelpingHandsthatconnectshouseholdserviceproviderswithconsumersofservicessuchashomecleaning,appliancerepair,pestcontrol,andsoon.Inthischapter,youwill:
LearnhowtogatherrequirementsandcaptureuserstoriestodesigntheHelpingHandsapplicationLearntheimportanceofmonolithic-firstdesignLearnhowtomovetowardsamicroservices-baseddesignLearnhowtouseevent-drivenarchitecturewithmicroservices
DesignOneofthebestwaystodesignasoftwaresystemistocapturethebusinessdomain,itsusers,andtheirinteractionwiththesystemasauserstory(https://en.wikipedia.org/wiki/User_story).Userstoriesareaninformalwayofcapturingtherequirementsofasoftwaresystem.Inuserstories,thefocusisontheendusersandtheinteractionsthatarepossiblebetweentheusersandthesystem.
UsersandentitiesThefirststepinwritinguserstoriesfortheHelpingHandsapplicationistounderstandtheusersandentitiesofthesystem.Primarily,therearetwousersofthesystem—ServiceConsumersandServiceProviders,asshowninthefollowingdiagram.ServiceConsumerssubscribetooneormoreservicesprovidedbytheServiceProviders.Thecoreentityoftheapplicationistheservice.Aserviceisanintangible,temporal,andlimitedassetthatprovidersownandprovidetotheconsumerson-demandataprice.
ServiceProvidersregisteroneormoreserviceswiththesystemthatcanbesubscribedtobytheconsumers.Allservicesareregisteredwithavailabletimeslotsandthedurationforwhichtheycanbeoffered.EachservicedurationandtimeslothasanassociatedpricethattheServiceConsumerhastopaytomakeuseoftheservice.ServiceProvidersarealsoresponsibleformaintainingtheavailabilitystatusoftheserviceswiththesystem.ServiceConsumerscansearchforservicesavailablefromasetofprovidersandcanpickaprovideroftheirchoice.Oncechosen,theServiceConsumercanscheduleaservicefromtheServiceProviderbasedonavailability.
Userstories
ThenextstepistolisttheuserstoriesthatwillbesupportedbytheHelpingHandsapplication.Herearetheuserstoriesfortheapplication:
AsaServiceConsumer,IcancreateanaccountsothatIcansearchforservicesandbookthemAsaServiceConsumer,IcansearchforrequiredservicessothatIcanbookoneformytaskAsaServiceConsumer,IcansubscribetooneormoreservicessothatIcangetmytaskdoneonaregularbasisAsaServiceConsumer,IwouldliketoratetheservicesofferedsothatotherscanbenefitfromthefeedbackandchoosethebestservicesofferedforaparticulartaskAsaServiceConsumer,IwanttogetnotificationssothatIcangetremindedoftheservicescheduleAsaServiceProvider,IcancreateanaccountsothatIcanregisteroneormoreservicesforconsumersAsaServiceProvider,IwanttoregisteroneormoreservicessothatIcangetservicerequestsAsaServiceProvider,IwanttospecifytheservicelocationareasothatIcangetonlyservicerequeststhatareneartomyplaceAsaServiceProvider,IwanttospecifythepriceandavailabilitysothatIcangetonlyservicerequeststhatarefeasibletoserveandtheonesIaminterestedinAsaServiceProvider,IwanttogetnotificationswhenaservicerequestisplacedsothatIcanattendtoit
Apartfromuserstories,therearesomenon-functionalrequirements(https://en.wikipedia.org/wiki/Non-functional_requirement)aswellthatmustbeaddressedbytheHelpingHandsapplication:
Alltheimplementationsmustbetrackedandversionedinarevisioncontrol
system.TheHelpingHandsapplicationwillusetheexternalhostingserviceGitHubtotrackthecodebase.Allthedependenciesmustbeexplicitlydeclared.TheHelpingHandsapplicationwillbeimplementedusingClojure(https://clojure.org/)withLeiningen(https://leiningen.org/)fordependencymanagement.Allservicesmusthaveauthenticationandauthorizationbuiltin.Configurationsmustbespecifiedexternallyandnothardcodedintheapplication.Alleventsmustbeloggedtounderstandthestateoftheapplicationandmonitoritinproduction.
Non-functionalrequirementsareapartoftwelve-factormethodology(https://12factor.net/),whichcoverstwelvedifferentaspectsthatmustbeaddressedbytheapplication.Part-3andPart-4ofthisbookaddresssomeoftheseimportantaspectsfortheHelpingHandsapplicationindetail.
DomainmodelBasedontheuserstories,ServiceOrderandService(catalog)arethetwocoredomainsoftheHelpingHandsapplication.Apartfromthesetwodomains,useraccounts,provideraccounts,andnotificationsarethegenericdomainsthatformtheHelpingHandsapplication.ServiceConsumer,ServiceProvider,Service,andServiceOrderareentitiesoftheHelpingHandsapplication.Theseareshownalongwiththeirfieldsinthefollowingdiagram:
EachServiceConsumerhasanIDassignedinthesystemandhasName,Address,Mobile,andEmaildefinedassomeoftheattributes.TheGeoLocationofeachconsumerisderivedfromtheaddressinformation.EachServiceProvideralsohasanIDassignedinthesystemwithName,Mobile,
ActiveSince,andOverallRatingassomeoftheattributes.AServiceisregisteredbytheServiceProviderandhasauniqueIDdefinedinthesystem.ItisregisteredagainstaServiceTypeandhasoneormoreServiceAreasdefined.BasedontheServiceAreas,thesystemderivestheGeoLocationBoundarywheretheserviceisofferedtotheconsumers.AServicealsohasanHourlyCostsetbytheServiceProviderandkeepsanOverallRatingbasedonthepreviousorders.
AServiceConsumersubscribestotheServiceandplacesaServiceOrderfortheServiceProvidertofulfill.EachServiceOrderhasanIDdefinedinthesystemandhasanassociatedServiceID,ProviderID,andConsumerID.EachorderentryhasanassociatedTimeSlotbasedonwhichtheCostisdeterminedbythesystem.TheServiceOrderalsohasastatusthatisupdatedbytheServiceProvideroncetheorderiscompleted.TheorderalsokeepsaRatingbetween1and5,asratedbytheconsumerfortheserviceprovided.
TheHelpingHandsapplicationalsoalertstheServiceConsumerandtheServiceProviderwithrespecttotherelatedServiceOrder.AnychangeinStatusisnotifiedtoboththeparticipantsviaSMSsentovertheregisteredmobilenumber.TheServiceConsumercanalsosubscribetoServiceandreceiveallupdateswithrespecttoservicestatus,availability,costchanges,andsoon.
MonolithicarchitectureTheHelpingHandsapplicationcanbedesignedusingathree-layeredarchitectureofpresentation,businesslogic,andpersistence.Basedonthedomainmodel,therecanbefourmaintablesintheHelpingHandsapplicationdatabasecorrespondingtoeachentity.Therewillbeasingledatabasethatwillstoreallthedatainthedesignatedtable.Thedatabasemustbeaccessibletoallthecomponentsofthesystem.Thebusinesslogiclayerwillhavewell-definedcomponentsbasedontheprincipleofSeparationofConcerns(SoC).ComponentswilladdressalluserstoriesfortheHelpingHandsapplication.
Applicationcomponents
Toaddresstheuserstories,therewillbethreemaincomponentsandtwohelpercomponents,asshowninthefollowingdiagram.TheRegistrationComponentwillmanagealltheuseraccountsandrelatedCRUDoperations.TheServiceComponentwillhandleallservice-relatedoperationssuchascreate,update,andlookup.TheOrderComponentwillhelpplacetheorders,searchforthehistoricalorder,andalsoratetheservices.ItwillalsohelptomaintainastatusfortheserviceorderandgeneraterelevantalertsintheformofSMSandemailusinganexternalservice:
Therewillbetwohelpercomponents—GeoLocationandAlerting.ThesecomponentswillhelpconnecttheapplicationtoexternalservicestogetgeolocationtagsandsendalertsintheformofSMSandemailviaprovidedAPIs.Components,theirresponsibilities,andrelateddatabasesaresummarizedinthefollowingtable:
Component Responsibilities Component
RegistrationComponent
Create/update/deleteaccountforserviceconsumersCreate/update/deleteaccountforserviceproviders
GeoLocationComponenttogetthelongitudeandlatitudebasedontheaddress
ServiceComponent
Create/update/deleteservicesSearchforservicesbykeywords,types,location,andsoon
GeoLocationComponenttogetthelongitudeandlatitudebasedonservicearea
OrderComponent
Create/updateordersSearchforordersbykeywords,types,timespan
AlertingComponenttogeneratealertswithrespecttoorders
GeoLocationComponent
Lookuplongitudeandlatitudebyaddress ExternalService(API)
AlertingComponent
SendemailandSMSalerts ExternalService(API)
Componentssuchasauthenticationanduserssuchasadministratorsandsoonhavebeenintentionallyomittedfromtheexplanationtofocusonlyonthecorecomponentsandfeaturesoftheapplication.
DeploymentTheHelpingHandsapplicationcanbedeployedasasingleartifactinanapplicationserver.Oncetheservicebecomespopularitmustbescaledtohandleincomingservicelookupsandorderrequests.Theeasiestwaytoscaletheapplicationforincomingrequestsistodeploymultipleinstancesoftheapplicationandfrontenditwithaloadbalancer,asshowninthefollowingdiagram.Theloadbalancerthenhelpstodistributetherequestsamongthedeployedinstancesoftheapplication.Sincealltheinstancesusethesamedatabase,allofthemarealwaysin-syncandhaveaccesstoconsistentdata.Consistencyisachievedduetotheinherentcapabilitiesofthedatabaseusedfortheapplication;itsupportsACIDtransactions:
LimitationsAlthoughamonolithicarchitecturefortheHelpingHandsapplicationfulfillsthepurpose,ithassomeinherentlimitations.Thereisahighcouplingofthecomponentswiththeconsumersandproviderstable.Anychangeinthesetwotableswillaffectalmostallthecomponentsofthesystemandwillrequireredeploymentoftheentireapplication.
TheHelpingHandsapplicationalsodependsontwoexternalservices.Supposeoneoftheseservicesshutsdownorthereisarequirementtomovetoabetterservice;thecorrespondinggeolocationoralertingcomponentwillbechangedandwillresultintheredeploymentofalltheinstancesoftheapplication,eventhoughtherewasnochangeinthecorefunctionalityandservicesoftheapplication.Thisaddstothedeploymentoverheadforsimplechangesaswell.
Sincetheentireapplicationisdeployedasasingleartifact,scalinganapplicationscalesallthecomponentsoftheapplicationequally.Forexample,toscalewithincomingorderandservicelookuprequests,theRegistrationComponentisunnecessarilyscaledwiththeorderandservicecomponents.Thisalsoincreasestheloadonthedatabasethatishandlingalltheincomingrequestsfromthecomponents.Often,requestsfromonecomponentcanaffectthedatabaseperformanceforothercomponentsaswellandreducetheperformanceoftheentireapplication.
AnotherlimitationofthecurrentmonolithicarchitectureoftheHelpingHandsapplicationisitsdependencyonasingledatabasetechnology.Inpractice,afuzzysearchforservicesusingtagsandlookupusinggeolocationscanbesupportedbetterbydatabasessuchasElasticsearch(https://www.elastic.co/products/elasticsearch)ascomparedtorelationaldatabasessuchasMySQL(https://www.mysql.com/).Relationaldatabasesarebettersuitedfortransactionaloperationssuchascreatingserviceordersandmaintaininguseraccounts.Withthecurrentarchitecture,thereisonlyonedatabasetechnology,andthataffectstheefficiencyoftheapplicationandmakesitlessflexible.
MovingtomicroservicesThelimitationsofamonolithicarchitectureforHelpingHandscanbeaddressedbyseparatingoutthecomponentsalongwiththedatabaseasamicroservice.Theseservicescanthenmakeinformedchoicesaboutthetechnologystackanddatabasethatsuitthemwell.Theseservicescanbedeveloped,changed,anddeployedinisolationaspertheconceptsofamicroservices-basedarchitecture.Toidentifytheboundedcontextforthecomponentsofanexistingmonolithicapplication,itisrecommendedtolookatthedatabaseaccesspatternandrelatedbusinesslogicfirst,isolatethem,andthenlookatthepossibilitiestoisolatethecomponentsfurtherbasedonbusinesscapabilities.
IsolatingservicesbypersistenceIntheexistingmonolithicapplicationofHelpingHands,theconsumersandprovidersdatabasetablesareaccessedbyallthecorecomponentsofthesystem,asshowninthefollowingdiagram.Thesetablesareprimecandidatesforbeingwrappedaroundaserviceandisolatedinaseparatedatabasethatisaccessibleonlytothecorrespondingservicedirectly.AllotherservicesmusttalktotheServiceConsumerserviceandtheServiceProviderserviceforanydetailsinsteadofdirectlyaccessingtheconsumersandprovidersdatabases.
Sincethereisaseparateservicecreatedtohandletherequestsforconsumersandproviders,thereisnoneedtohaveaservicecorrespondingtotheRegistrationComponent.TheServiceConsumerserviceandServiceProviderservicecannowhandlealltherequeststoregister,modify,ordeleteconsumersandproviders,respectively.Similarly,theserviceandorderservicescannowhandlealltherequestsrelatedtoservicesandorders,respectively,byisolatingthecorrespondingdatabases.TheorderservicecannowtalktoServiceConsumer,ServiceProvider,andServicetogettherequireddetailsfortheorder.
TheHelpingHandsapplicationwillbeusingacombinationoftheDatomic(http://www.datomic.com/)andElasticsearch(https://www.elastic.co/products/elasticsearch)databasesforvariousmicroservices.Part-3ofthisbookdiscussesthepersistencelayerindetail,andthelastchapterofPart-2introducesDatomic.
IsolatingservicesbybusinesslogicOncepersistence-basedservicesareisolated,thenextstepistoevaluateexistingcomponentsformicroserviceswithrespecttobusinesslogic.ApartfromdroppingtheRegistrationComponentinfavorofseparateservicesforconsumerandprovider,anewservicecalledlookupcanbecreatedtoconsolidateallthesearchoperationsintooneserviceandallowuserstosearchacrossapplicationentities,asshowninthefollowingdiagram.Sincedatabasesofconsumers,providers,services,andorderscannotbesharedwithlookupservices,itcankeepadenormalized(https://en.wikipedia.org/wiki/Denormalization)viewofthesedatabasescontainingonlythefieldsthatneedtobesearched.
Geolocation-basedquerieswillalsobelimitedtolookupservices,sothereisnoneedtomaintainaseparategeolocationservice;instead,theLookupServiceitselfcanqueryforthegeolocation.
Sincegeolocationsrarelychange,theLookupServicecancachethemandmaintainadatabaseofwell-knownandalreadyqueriedgeolocationsaswellforbetterperformance.
TheAlertingComponentcanbeisolatedasaseparateserviceasitwillberequiredbymultipleservices,includingOrder,ServiceConsumer,andService
Provider,tosendalertstotheusers.AlertsmaybesentviaSMSoremail,andtheAlertingServicecanuseexternalservicestosendthealerts.Sincealertsmustnotbeoverwhelmingforusers,theAlertingServicecangroupbyuserallthealertsthatarerequestedinashortperiodoftimeandsendthemasasinglenotificationmessage.
Donotattempttoaggressivelystartdisintegratingyourcomponentsintomicroservices.Focusonthebusinesscapabilitiesandnotonfeaturesandusecases.Forexample,insteadofcreatingaseparateserviceforsendingemailsandsendingSMS,itisrecommendedtocreateasingleAlertingServicewithbothcapabilities.
MessagingandeventsThenextstepfortheHelpingHandsapplicationistodefinetheinteractionsbetweentheidentifiedmicroservices.Microservicescaneitherinteractbydirectlysendingthemessagestootherserviceendpointssynchronouslyortheycansubscribetotheeventsgeneratedbyothermicroservicesandreceivethemessagesasynchronously.Asynchronousmessagesrelyontheunderlyingmessagebrokeranditsdurability.Messagebrokersnotonlyhelptoscaletheapplicationbyholdingthemessagesyettobeprocessedinthequeue,butalsosupportdurabledeliveries.Evenifaservicefails,itcanberestoredandallowedtostartprocessingpendingmessagesfromthepointwhereitleftoff.CombiningbothsynchronousandasynchronousmessagepatternsfortheHelpingHandsapplicationgivesusaflexibleandperformantarchitecturetoaccomplishagiventask,asshowninthefollowingdiagram:
AlltheservicesoftheHelpingHandsapplicationmustpublishchangelogeventsrelatedtothebusinessentitiesonamessagequeuethatcanbereadby
anyservicethatsubscribestoit.Thepublishedeventsmustberetainedbythemessagequeueforaconfiguredamountoftime,beyondwhichtheeventsmaybediscarded.Forexample,allthecoreservices—ServiceConsumer,ServiceProvider,Service,andOrder—publisheventsontheirdesignatedmessagequeuesattheallocatedtopic.
TheLookupServicemustsubscribetoalltheeventspublishedbytheConsumer,Provider,Service,andOrderservicestomaintainalocaldenormalizeddatabasetosupportsearchqueries.Itmustaddgeolocationdetailsbyqueryingtheexternalserviceandcachingtheresultslocally.Anychangesdonetotheconsumers,providers,services,andordersdatabasesmustbecommunicatedtotheLookupServiceviaevents,asynchronously.TheLookupServicemayalsopublisheventstoitsdesignatedmessagequeueforotherservicestoconsume.Theseeventsareoftenusefultoanalyzethenumberofsearchqueriesreceived,trendingservices,andsoon.
ServicessuchasAlertingarebestsuitedforsuchasynchronousmessages.TheAlertingServiceshouldnotonlyrelyonthemessagebrokerforvariousdeliverysemantics,suchasat-leastorexactly-oncedeliverybutmustalsoreadbatchesofalerts,combinealertsforthesameuserandsendthemasasingleconsolidatedalert.
ServicessuchasOrderServicemayalsorelyondirectmessagestoretrievedetailsoftheconsumer,provider,andtheservicebeforeregisteringanorderfortheuser.Oncetheorderisregistered,achangelogeventmustbepublishedbytheOrderServicefortheLookupServicetomaketheorderavailabletobesearched.
Eventlogsarealsousefultosetupadeepmonitoringandreportinginfrastructureformicroservices.Part-4ofthisbookdescribesthemonitoringandreportingpatternforthemicroservicesarchitecturethatisbasedoneventlogs.
ExtensibilityAmicroservices-basedarchitecturefortheHelpingHandsapplicationnotonlymakesiteasiertodeployandscalebutalsomakesithighlyextensible.Forexample,bydeployingaseparateLookupServiceforsearchoperations,itisnowpossibletouseadatabasesuchasElasticsearchonlyforsearchoperationsandDatomicforallothermicroservicesthatrequireconsistenttransactions.
Inthefuture,ifthereisabettertechnologyavailable,itwillbeeasiertodeployserviceswithanewertechnology.Newertechnologymayalsocomewithhardwarechallenges.Forexample,adatabasesuchasMapD(https://www.mapd.com/)runsonGPUs(https://en.wikipedia.org/wiki/Graphics_processing_unit).Tousesuchdatabases,microservicesneedtorunonspecializedhardware.Sincemicroservicescanbedeployedinisolationonthesameoranentirelyseparatemachine,itispossibletodeployservicesthatneedGPUsonmachinesthatsupportGPUswithoutaffectingthewaytheyinteractwithotherservices.Thisisoneoftheadvantagesofmicroservices—youarenotboundbythetechnologyorunderlyinghardwareandanychangesdonearelocalizedwithintheboundedcontextoftheservice.
Dataanalyticsisnowanintegralpartofanyweb-basedapplication.Itnotonlyhelpsusunderstandtheusagepatternsoftheapplication,butalsohelpsprovidebetterservicestotheusers.Bygeneratingeventlogsforallthechangesdonetoentities,itisalsopossibletofurtherextendtheHelpingHandsapplicationtoanalyzeusagepatterns.Forexample,onecanlistentoeventsgeneratedbytheOrderServiceandstudyserviceusagepatternsbydemography.Itshouldalsobepossibletoanalyzethepopularityofservicesbylocationandprovidecustomizedofferstousersasnotifications.
WorkflowsforHelpingHandsDataworkflowsarethebackboneofanymicroservices-basedarchitecture.Theydefinethesequenceofmessagesandeventsthataregeneratedamongtheservicestoaccomplishadesiredtask.Aworkflowmayconsistofbothsynchronousandasynchronousmessages.
Workflowsshowninthissectionareonlyforexplanatorypurposesanddonotconformtotheexactsemanticsofsequencediagrams(https://en.wikipedia.org/wiki/Sequence_diagram).Detailsofauthentication,authorization,validation,anderrorconditionshavebeenomittedintentionally.
ServiceproviderworkflowTheserviceproviderworkflowconsistsofServiceProvider,LookupService,andAlertingService.TheServiceProviderComponentexposesendpointsforuserstocreateandupdateserviceprovidersfortheHelpingHandsapplication,asshowninthefollowingdiagram:
Forthecreateoperation,theServiceProviderservicefirstvalidatestheinputrequestfortherequiredparametersandprivileges,thenstartsatransactiontoaddanewserviceproviderwithinthesystem.AnewIDfortheserviceproviderisautomaticallygeneratedbytheServiceProvider.Oncethetransactionissuccessful,itemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup,andpublishesacreatealertonthemessagequeueoftheAlertingServiceforittopickandsendanalerttotheinterestedusers.
Toupdatetheserviceproviders,theusersendstheupdaterequesttotheServiceProvideranditinitiatesanewtransactiontoupdateitslocaldatabase.Oncethetransactionissuccessful,itemitsanupdateeventonitseventlogqueueforthe
LookupServicetopick-up.Updateoperationsarenotsentasalertsornotificationstousers.Ifthisisarequirement,itcanbeenabledbypublishingamessagefortheAlertingServiceonitsrequestqueue.
ServiceworkflowTheserviceworkflowdescribesthemessageinteractionamongtheServiceComponent,ServiceProvider,LookupService,andAlertingService,asshowninthefollowingdiagram.TheserviceexposesendpointsforuserstocreateandupdateservicesfortheHelpingHandsapplication.Eachservicemusthaveaserviceprovideralreadydefinedfortheapplication:
Tocreateanewservice,theusercallstheAPIendpointforServicewiththerequiredparameters.TheproviderIDmustbespecifiedtocreateanewservice.OncetheServiceComponentreceivestherequest,itfirstsendsadirectmessagetotheServiceProvidertovalidatethespecifiedproviderIDandmakesurethattheproviderisalreadyregistered.Iftheproviderisalreadyregistered,theServiceProviderreturnstheproviderdetailssynchronouslytotheService.
Oncetheproviderisvalidated,theServiceComponentstartsatransactiontoaddanewserviceintoitslocaldatabase.AnewIDfortheserviceisautomaticallygeneratedbytheServiceComponent.Oncethetransactionissuccessful,theServiceComponentemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup.TheLookupService
pullsthecreateevent,looksupthegeolocationcorrespondingtotheserviceareas,andupdatestheservicedetailsinitslocaldenormalizeddatabase.TheServiceComponentalsopublishesanewservicealertonthemessagequeueoftheAlertingServicetosendanalerttotheinterestedusers.
Toupdatetheservicedetails,theusersendsanupdaterequesttotheServiceComponentwiththeserviceID.Sincetheupdateoperationisperformedonlyforexistingservices,theserviceproviderisnotvalidatedinthiscase.TheServiceComponentinitiatesatransactiontoupdatetheservicedetailsandupdatesthespecifiedfields.Oncethetransactionissuccessful,itemitsanupdateeventontheeventlogqueuefortheLookupServicetopickup.TheLookupServicethenupdatesthelocaldatabasewiththechangesandalsoupdatesthegeolocationsfortheserviceareasiftheyarepartofthechangelog.Alertsarenotsentforserviceupdates.
ServiceconsumerworkflowTheserviceconsumerworkflowdescribesthemessageinteractionamongtheServiceConsumer,Lookup,andAlertingService.TheServiceConsumercomponentexposesendpointsforuserstocreateandupdateserviceconsumersfortheHelpingHandsapplication,asshowninthefollowingdiagram.
Tocreateanewserviceconsumer,theusersendsarequestwithvalidparameterstotheServiceConsumerComponent.Thecomponentthenvalidatestherequestandiftherequestisvalid,itstartsatransactiontoaddanewconsumertotheapplication.Foreachnewconsumer,anIDisautomaticallygeneratedbyServiceConsumerComponent.
Oncethetransactionissuccessful,theServiceConsumerComponentemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup.Afterreceivingthecreateevent,theLookupServicelooksupthegeolocationcorrespondingtotheaddressoftheconsumerandthenstorestheconsumerdetailsalongwiththegeolocationinitslocaldenormalizeddatabase.TheServiceConsumercomponentalsopublishesanewconsumeralertonthemessagequeueoftheAlertingServicetosendanalerttointerestedusers:
Toupdatetheconsumerdetails,theusersendstheupdaterequesttotheServiceConsumerComponentalongwiththeconsumerID.TheServiceConsumerComponentinitiatesatransactiontoupdatetheconsumerdetailsandupdatesthespecifiedfields.Oncethetransactionissuccessful,itemitsanupdateeventontheeventlogqueuefortheLookupServicetopickup.TheLookupServicethenupdatesthelocaldatabasewiththechangesandalsoupdatesthegeolocationfortheconsumeraddressifitisapartofthechangelog.Alertsarenotsentforconsumerupdates.
OrderworkflowTheorderworkflowinvolvesmostoftheservicesintheHelpingHandsapplication.TheOrderServicereceivestherequestfromtheconsumertocreateaneworderforthechosenservice.TheconsumersendsthecreaterequesttotheOrderServicewiththeproviderandservicedetailsalongwiththetimeslot.TheOrderServicethenvalidatestheorderdetailsbysendingadirectsynchronousrequesttotheServiceConsumer,ServiceProvider,andService.Ifthedetailsarevalid,thatis,theserviceisalreadyregisteredbythespecifiedproviderandtheconsumerisavaliduserinthesystem,thentheOrderServicesendsadirectsynchronousrequesttotheLookupServicetomakesurethattheserviceisavailableinthevicinityoftheconsumerlocationbasedonthegeolocationoftheconsumeraddressandtheservicearea.
Iftheserviceisfeasiblefortheconsumeraddress,theOrderServicestartsatransactionandcreatesaneworderinthesystem,asshowninthefollowingdiagram:
Oncethetransactionissuccessful,theOrderServiceemitsacreateeventonitseventlogqueuefortheLookupService;itreceivestheevents,updatestheorderlistwithinitslocaldatabase,andmakesitavailableforuserstosearch.TheOrderServicealsopublishesaneworderalertonthemessagequeueoftheAlertingServicetosendanalerttoboththeconsumerandtheprovideroftheservice.
Toupdatetheordertimeslot,status,orrating,theusersendsanupdaterequesttotheOrderService.SinceanupdatemessagerequiresanexistingorderID,theOrderServicedoesnotvalidatetheproviderandconsumerfortherequest,itjustmakessurethatitisanexistingorder.Inadditiontovalidatinganexistingorder,theOrderServicealsosendsadirectsynchronousrequesttotheLookupServicetomakesurethattherequestedtimeslotandservicearestillavailablefortheconsumer.Iftheyareavailable,thentheOrderServicestartsa
transactionandupdatesitslocaldatabase.Oncethetransactionissuccessful,itemitstheupdateeventonitseventlogqueuefortheLookupServiceandalsosendsanalerttotheAlertingServicebypublishinganupdateorderalert.Anychangemadetotheorderissentasanalerttoboththeconsumerandprovideroftheserviceorder.
Inpractice,therewillbeseparateservicesandworkflowsforauthenticationandauthorization.Thoseworkflowshavebeenintentionallyomittedfromthischaptertofocusonlyonthecoreservicesandworkflows.Part-4ofthisbookdescribesauthenticationandauthorizationservicespatternsindetail.
SummaryInthischapter,wedesignedanapplicationcalledHelpingHands,usingthebestsoftwareengineeringpractices.Westartedwithamonolithicarchitectureandarguedwhyamicroservices-basedarchitectureiswellsuitedfortheHelpingHandsapplication.Inthenextpartofthisbook,wewillfirsttakealookatthebasicdevelopmenttoolsandlibrariesthatwewillbeusingtobuildourHelpingHandsapplication.
DevelopmentEnvironment
"Themechanicthatwouldperfecthisworkmustfirstsharpenhistools."
-Confucius
Thedevelopmentenvironmentconsistsoftoolsandlibrariesthatareusefultoimplement,debug,andmakechangestosoftwaresystems.Theefficiencyofadevelopmentteamishighlydependentonthedevelopmentenvironmentandtechnologystackathand.Inthischapter,youwilllearnhowtosetupadevelopmentenvironmentformicroservicesusingtheClojureecosystem.Thischapterwillhelpyouto:
LearnthehistoryofClojureandfunctionalprogrammingLearntheimportanceofREPLLearnhowtobuildyourapplicationusingClojurebuildtoolsLearnaboutwellknownintegrateddevelopmentenvironments(IDEs)
ClojureandREPLClojure(https://clojure.org/)isadialectoftheLisp(https://en.wikipedia.org/wiki/Lisp_(programming_language)programminglanguageandprimarilyrunsonaJavavirtualmachine(JVM).TheothertargetimplementationsincludeClojureCLR(https://github.com/clojure/clojure-clr),whichrunsonCommonLanguageRuntime(CLR),andClojureScript,whichcompilestoJavaScript.AlthoughClojureusesaJVMasitsunderlyingruntimeengine,itemphasizesafunctionalprogramminglanguagewithimmutabilityatitscore.AlldatastructuresofClojureareimmutable.SinceClojureisadialectofLisp,italsotreatscodeasdataandisknowntobehomoiconic.ItssyntaxisbuiltonS-expressions(https://en.wikipedia.org/wiki/S-expression)thatarefirstparsedasadatastructureandthentranslatedintoconstructsoftheJavaprogramminglanguagebeforebeingcompiledintoJavabytecode.Clojurealsosupportsmetaprogrammingwithmacro(https://en.wikipedia.org/wiki/Macro_(computer_science)).
ClojureintegrateswellwithexistingJavaapplicationsduetoitsprimarysupportforunderlyingJVMsanditsentireecosystem.AllexistinglibrariesthatrunonaJVMintegrateseamlesslywithClojure,includingMaven,Java'sbuildsystem.
HistoryofClojureClojureisafunctionalprogramminglanguagethattreatsfunctionsasfirst-classobjects;thatis,itsupportspassingfunctionsasargumentstootherfunctionsandalsoreturningthemasvaluesfromotherfunctions.Italsosupportsanonymousfunctions,assigningfunctionstovariablesandstoringthemindatastructures.
ThetimelineintheprecedingdiagramshowstheevolutionoffunctionalprogrammingthatledtothedevelopmentofClojure.TheconceptoffunctionalprogrammingoriginatedfromaformalsystemcalledLambdacalculusthatwasintroducedbyAlonzoChurch(https://en.wikipedia.org/wiki/Alonzo_Church)in1930.Churchproposedauniversalmodelofcomputationthatwasbasedonfunctionabstractionanditsapplicationusingvariablebindingandsubstitution.Church'smodellaidthefoundationoffunctionalprogramminglanguagesthatareknowntoday,includingClojure.
Almostthreedecadeslaterin1958,JohnMcCarthy(https://en.wikipedia.org/wiki/John_McCarthy_(computer_scientist))introducedtheLispprogramminglanguage,whichwashighlyinfluencedbythenotationsofLambdacalculus.Itwasdistinctiveduetoitsfullyparenthesizedprefixnotation.Itssourcecodewasmadeoflists,hencethenameLIStProcessor.LISPintroducedmanyotherconceptsincludingtreedatastructure,dynamictyping,high-orderfunctions,andread-eval-print-loop(REPL).TwowidelypopularLispdialectsareScheme(https://en.wikipedia.org/wiki/Scheme_(programming_language))andCommonLisp(https://en.wikipedia.org/wiki/Common_Lisp),bothinventedbyGuySteele(https://en.wikipedia.org/wiki/Guy_L._Steele_Jr.).Schemewasintroducedin1970followedbyCommonLispin1984.ClojureisalsooneofthedialectsofLispthatwasintroducedin2007byRichHickey.
InadditiontobeingadialectofLISP,ClojurealsochoseJVMasitsruntimeduetoitsinherentadvantagesandmaturedecosystem.Javawasfirstintroducedin
1994-1995byJamesGosling(https://en.wikipedia.org/wiki/James_Gosling).Javaisanobject-orientedlanguagewithafocusonconcurrency.Javabecamewidelypopularinenterprisessoonafteritsreleaseduetoitsefficientdependencymanagementandabilitytowriteonce,andrunanywhere(https://en.wikipedia.org/wiki/Write_once,_run_anywhere).Javaalloweddeveloperstowritetheircodeonce,compileitintobytecode,andthenrunitonallplatformsthatsupportJVMwithoutrecompilingthecode.
ClojureinheritedallthecapabilitiesofJVMwiththefocusonimmutabilityandLISP-likeparenthesizedprefixnotation.Itisbuiltontheconceptsofimmutabilityandpersistentdatastructures,whichmakesithighlyconcurrent.ItusestheEpochalTimeModel,whichorganizesprogramsbyidentities,whereeachidentityisdefinedasaseriesofimmutablestatesovertime.Duetoitshighlyconcurrentnature,Clojureisbestsuitedtobuildingmassivelyparallelsoftwaresystemsthatarerobustandutilizemodernmultiprocessorhardwaretothefullest.
TheEpochaltimemodelwasintroducedbyRichHickeyinhiskeynotetalkonArewethereyet?(https://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey)attheJVMLanguagesSummitin2009.HerevisiteditinhiskeynotetalkonEffectivePrograms-10YearsofClojure(https://www.youtube.com/watch?v=2V1FtfBDsLU)atClojure/Conjin2017.
REPLRead-eval-print-loopisaninteractiveprogrammingenvironmentthattakesexpressionsasinputsfromtheuser,parsesthemintoadatastructurefortheprogramminglanguage,evaluatesthem,andprintstheresultbacktotheuser.Oncedone,itreturnstothereadstateandwaitsforuserinput,thusformingaloop,asshowninthefollowingdiagram.REPLwaspioneeredbyLispandisnowavailableforotherprogramminglanguagesaswell:
REPLisalsothedefaultinteractiveprogrammingenvironmentforClojure.ItcompilesanexpressionintoJavabytecode,evaluatesit,andreturnstheresultoftheexpression.TouseClojureREPL,downloadaClojurerelease(1.8.0)fromClojureDownloads(https://repo1.maven.org/maven2/org/clojure/clojure/1.8.0/clojure-1.8.0.zip)andextractit.SinceClojurerunsonJVM,youneedtofirstmakesurethatyouhaveJava1.6+orhigherinstalledonyourmachine.TovalidateyourJavaversion,usethejavacommandwiththe-versionoption.ThisbookusesJava1.8forallexamples:
%java-version
javaversion"1.8.0_121"
Java(TM)SERuntimeEnvironment(build1.8.0_121-b13)
JavaHotSpot(TM)64-BitServerVM(build25.121-b13,mixedmode)
TouseClojureREPL,firstunziptheClojure1.8release:
%unzipclojure-1.8.0.zip
Archive:clojure-1.8.0.zip
creating:clojure-1.8.0/
...
inflating:clojure-1.8.0/clojure-1.8.0-slim.jar
inflating:clojure-1.8.0/clojure-1.8.0.jar
inflating:clojure-1.8.0/pom.xml
inflating:clojure-1.8.0/build.xml
inflating:clojure-1.8.0/readme.txt
inflating:clojure-1.8.0/changes.md
inflating:clojure-1.8.0/clojure.iml
inflating:clojure-1.8.0/epl-v10.html
Intheunzippedclojure-1.8.0folder,theclojure-1.8.0.jarJAR(https://en.wikipedia.org/wiki/JAR_(file_format))isanexecutableJARthatcontainstheClojurecompiler.ItisalsousedtostartClojureREPLusingthejavacommand:
%cdclojure-1.8.0
%java-jarclojure-1.8.0.jar
Clojure1.8.0
user=>(+123)
6
user=>
TheprecedingmethodofstartingtheREPLwiththeClojureJARfileworkswelluntilthelaststablereleaseofClojure1.8atthetimeofwritingthisbook.WithClojure1.9andonwards,itisrecommendedtousetheClojurecljtool(https://clojure.org/guides/deps_and_cli)orabuildtoolsuchasLeiningen(https://leiningen.org/)tosetupaprojectandstartREPLfortheprojectwiththerequiredlibraries.
repl.it(https://repl.it)isanonlineutilitythatallowyoutoaccessClojureREPLonlineoverthewebbrowser.
ClojurebuildtoolsBuildtoolsareofprimeimportanceforanyprogramminglanguage.Theynotonlyhelptogenerateadeployableartifactfortheapplication,butalsomanagethedependenciesoftheapplicationthroughoutitsdevelopmentlifecycle.LeiningenandBoot(http://boot-clj.com/)aretwowidelyusedbuildtoolsforClojure.SinceClojureisahostedlanguageforJVM,ClojurebuildtoolsprimarilygenerateJARsasdeployableartifactsforallClojureprojects.
LeiningenLeiningenisabuildandprojectmanagementtoolthatiswritteninClojureandiswidelyusedacrossClojureprojects.ItdescribesaClojureprojectusinggenericClojuredatastructures.ItintegrateswellwiththeMavenrepositoryfromtheJavaworldandtheClojars(https://clojars.org/)repositoryofClojurelibraries,fordependencymanagementandreleases.Leiningenhasbuilt-insupportforpluginstoextenditsfunctionality.ItalsoprovidesanoptiontodesignapplicationtemplatesthatcanhelpcreateaClojureapplicationcodelayoutwithasingleleincommand.Toseparateprojectconfigurationfromdevelopment,test,andproduction,italsohastheconceptofprofiles,whichcanbeusedtochangetheprojectconfigurationatbuildtimebyusingitwiththeleincommand.
Tosetuplein,downloadthelatestLeinscript(https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein)fromtheLeiningenGitHubrepository(https://github.com/technomancy/leiningen),makeitexecutableusingthechmodcommand,andmakesurethatthescriptisavailableonyourpath,forexample,bycopyingitto~/binor/usr/local/bininLinux.Oncetheleinscriptisavailableonyourpath,runittodownloadthelatestself-installpackageandsetuptheenvironment.LeindownloadstheexecutableJARforLeiningenfirsttime:
#Downloadthelatestscript(latestversion)
%wgethttps://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
#Makeitexecutable
%chmoda+xlein
#Runleintosetup
%lein
DownloadingLeiningen...
LeiningenisatoolforworkingwithClojureprojects.
...
#Validateleinversion
%leinversion
Leiningen2.8.0onJava1.8.0_121JavaHotSpot(TM)64-BitServerVM
#Downgradetoversion2.7.1,i.e.theversionusedinthebook
%leindowngrade2.7.1
#Validateleinversion
%leinversion
Leiningen2.7.1onJava1.8.0_121JavaHotSpot(TM)64-BitServerVM
Notethattheleinscriptalwaysdownloadsthemostrecentlyreleasedversion.Onceleinissetup,youcanstartREPLdirectlyfromtheleincommand:
%leinrepl
nREPLserverstartedonport43952onhost127.0.0.1-nrepl://127.0.0.1:43952
REPL-y0.3.7,nREPL0.2.12
Clojure1.8.0
JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13
Docs:(docfunction-name-here)
(find-doc"part-of-name-here")
Source:(sourcefunction-name-here)
Javadoc:(javadocjava-object-or-class-here)
Exit:Control+Dor(exit)or(quit)
Results:Storedinvars*1,*2,*3,anexceptionin*e
user=>(+123)
6
user=>
BootBootisanalternativebuildsystemforClojurethatisgainingpopularity.BoottreatsbuildscriptsasClojureprogramsthatcanbeextendedusingpods(https://github.com/boot-clj/boot/wiki/Pods).PodsalsohelptoisolateclasspathsforbetterdependencymanagementandusesmultipleClojureruntimes.InsteadoftheprofilesandpluginsofLeiningen,Bootusesthecommonconceptoftasks(https://github.com/boot-clj/boot/wiki/Tasks).TasksareusedacrosstheBootprogramtomodifythebuildenvironmentasperthecontext.
TosetupBoot,downloadthelatestboot.shscript(https://github.com/boot-clj/boot-bin/releases/download/latest/boot.sh)fromtheBootGitHubrepository(https://github.com/boot-clj/boot),makeitexecutableusingthechmodcommand,andmakesurethatthescriptisavailableonyourpath,forexample,bycopyingitto~/binor/usr/local/bininLinux.OncetheBootscriptisavailableonyourpath,runittosetuptheenvironment,asshownhere:
#Downloadbootscript
%curl-fsSLoboothttps://github.com/boot-clj/boot-
bin/releases/download/latest/boot.sh
#Makeitexecutable
%chmod755boot
#Runboottosetup
%boot
Downloadinghttps://github.com/boot-clj/boot/releases/download/2.7.2/boot.jar...
Runningforthefirsttime,BOOT_VERSIONnotset:updatingtolatest.
Retrievingmaven-metadata.xmlfromhttps://repo.clojars.org/(3k)
Retrievingboot-2.7.2.pomfromhttps://repo.clojars.org/(2k)
Retrievingboot-2.7.2.jarfromhttps://repo.clojars.org/(3k)
#http://boot-clj.com
#SatOct2119:43:24IST2017
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_VERSION=2.7.2
BOOT_CLOJURE_VERSION=1.8.0
OnceBootissetup,youcanstartREPLdirectlyfromthebootcommand:
%bootrepl
nREPLserverstartedonport33140onhost127.0.0.1-nrepl://127.0.0.1:33140
REPL-y0.3.7,nREPL0.2.12
Clojure1.8.0
JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13
Exit:Control+Dor(exit)or(quit)
Commands:(user/help)
Docs:(docfunction-name-here)
(find-doc"part-of-name-here")
FindbyName:(find-name"part-of-name-here")
Source:(sourcefunction-name-here)
Javadoc:(javadocjava-object-or-class-here)
Examplesfromclojuredocs.org:[clojuredocsorcdoc]
(user/clojuredocsname-here)
(user/clojuredocs"ns-here""name-here")
boot.user=>(+123)
6
boot.user=>
ThisbookwillfocusonLeiningenasthebuildtoolfortheClojureprojects,butitisimportanttoknowthatLeiningenisnottheonlybuildtoolavailableforClojure.BootisalsoanoptionthatcanbeusedinsteadofLeiningen.
ClojureprojectAClojureprojectisadirectorythatcontainssourcefiles,testfiles,resources,documentation,andprojectmetadata.SourcefilesareprimarilyfromClojure,butaprojectmaycontainJavasourcefilesaswell.LeiningenhasadefaultprojecttemplatethatcanbeusedtoquicklycreateaClojureprojectstructureusingtheleinnew<project-name>command:
#Createanewproject'playground'
%leinnewplayground
Generatingaprojectcalledplaygroundbasedonthe'default'template.
Thedefaulttemplateisintendedforlibraryprojects,notapplications.
Toseeothertemplates(app,plugin,etc),try`leinhelpnew`.
#Showthe'playground'projectdirectorystructure
%treeplayground
playground
├──CHANGELOG.md
├──doc
│└──intro.md
├──LICENSE
├──project.clj
├──README.md
├──resources
├──src
│└──playground
│└──core.clj
└──test
└──playground
└──core_test.clj
6directories,7files
EachClojureprojectcontainsaprojectmetadatafile,project.clj,whichdefinesalltheprojectdependencies,profiles,andpluginsthatarerequiredfortheproject.Forexample,theproject.cljfileoftheplaygroundprojectliststheprojectmetadataanddependenciesbasedonthedefaultprojecttemplateofLeiningen,asshownhere:
(defprojectplayground"0.1.0-SNAPSHOT"
:description"FIXME:writedescription"
:url"http://example.com/FIXME"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]])
ThedefprojectisaClojuremacrothatisdefinedbyLeiningenandactsasacontainerforallprojectmetadatadirectives.Thedefaulttemplateomitsthe
defaultconfigurationdirectives.
ConfiguringaprojectAgoodprojectconfigurationmustdefineseparateprofilesfordevelopment,test,andproduction.Itshouldalsohavedirectivestotesttheimplementation,checkcodequality,andgeneratetestreportsanddocumentationfortheapplication.Eachprojectconfigurationshouldalsohavetheentrypointintotheapplicationdefinedusingthe:maindirective,whichpointstothenamespacethatcontainsthemainfunction,asshownhere:
(defprojectplayground"0.1.0-SNAPSHOT"
:description"PlaygroundProject"
:url"http://example.com/playground"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]]
:mainplayground.core
:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]
[org.clojure/tools.nrepl"0.2.12"]]}
:uberjar{:aot:all:omit-sourcetrue}
:doc{:dependencies[[codox-theme-rdash"0.1.1"]]
:codox{:metadata{:doc/format:markdown}
:themes[:rdash]}}
:dev{:resource-paths["resources""conf"]
:jvm-opts["-Dconf=conf/conf.edn"]}
:debug{:jvm-opts
["-server"
(str"-agentlib:jdwp=transport=dt_socket,"
"server=y,address=8000,suspend=n")]}})
Projectconfigurationshouldexplicitlylistsourceandtestfilelocationsusingthe:source-pathsand:test-pathsdirective.IfbothClojureandJavasourcecodefilesarepresentintheproject,thenitisrecommendedtoorganizethesourcefileundersrc/cljforClojureandsrc/jvmforJava.Similarly,testfilescanbekeptundertest/cljandtest/jvmforClojureandJava,respectively.Areferenceproject.cljfilewiththerequiredprojectconfigurationisshowninthefollowingcodesnippet:
(defprojectplayground"0.1.0-SNAPSHOT"
...
:mainplayground.core
:source-paths["src/clj"]
:java-source-paths["src/jvm"]
:test-paths["test/clj""test/jvm"]
:resource-paths["resources""conf"]
...
)
Projectresourcefiles,suchasstartup,shutdownscripts,andmore,canbekeptintheresourcesdirectoryandtheprojectconfigurationfilescanbekeptintheconfdirectory.Boththedirectoriesmustbespecifiedunderthe:resource-pathsdirective,asshownintheprecedingsnippet.Togeneratedocumentationandtestreports,andcheckcodequality,thereareanumberofthird-partypluginsavailablethatcanbeaddedtotheprojectconfiguration.Forexample,Codox(https://github.com/weavejester/codox)canbeusedtogenerateAPIdocumentation,Cloverage(https://github.com/cloverage/cloverage)canbeusedtotestcodecoverage,andtest2junit(https://github.com/ruedigergad/test2junit)canbeusedtogeneratetestreports.Addthesepluginsintheconfigurationfile,asshownhere:
(defprojectplayground"0.1.0-SNAPSHOT"
...
:resource-paths["resources""conf"]
:plugins[[:lein-codox"0.10.3"]
;;CodeCoverage
[:lein-cloverage"1.0.9"]
;;Unittestdocs
[test2junit"1.2.2"]]
:codox{:namespaces:all}
:test2junit-output-dir"target/test-reports"
:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]
[org.clojure/tools.nrepl"0.2.12"]]}
:uberjar{:aot:all:omit-sourcetrue}
:doc{:dependencies[[codox-theme-rdash"0.1.1"]]
:codox{:metadata{:doc/format:markdown}
:themes[:rdash]}}
:dev{:resource-paths["resources""conf"]
:jvm-opts["-Dconf=conf/conf.edn"]}
:debug{:jvm-opts
["-server"
(str"-agentlib:jdwp=transport=dt_socket,"
"server=y,address=8000,suspend=n")]}})
SomeofthesepluginsmayalsorequireextraLeiningendirectivestobedefinedforthem.Dependenciesthatarecommonacrossprofilesmustbelistedunderthe:dependenciesdirectiveofthedefprojectmacro,andtherestofthedependenciesmustbelistedundertherespectiveprofiles,asshownintheprecedingconfiguration.
TheLeiningenGitHubrepositoryhasasample.project.cljfile(https://github.com/technomancy/leiningen/blob/master/sample.project.clj)thatlistsallsupportedprojectdirectivesforaClojureprojectthatismanagedbyLeiningen.ThisfileactsasdetaileddocumentationforallthefeaturesofLeiningen.
Runningaproject
Ifthe:maindirectiveisdefinedintheproject.cljfileoftheproject,thentheprojectcanberunbydirectlycallingtheleinruncommand.leinrunexpectsthenamespacespecifiedunder:maindirectivetocontaintheClojuremainfunction.Forexample,theplayground.corenamespacemusthaveamainfunctiondefinedforleinruntowork,asshowninthefollowingimplementation.Sincethe:source-pathsparameterintheupdatedconfigurationofproject.cljpointstosrc/cljinsteadofthedefaultsrc/,thecore.cljsourcefile,asshowninthefollowingcode,mustresidewithinthesrc/clj/playground/directoryasperthenamespaceandconfiguredsourcepaths:
(nsplayground.core
(:gen-class))
(defnfoo
"Idon'tdoawholelot."
[x]
(printlnx"|Hello,World!"))
(defn-main
[&args]
(foo(or(firstargs)"Noname")))
Ifthemainfunctionisdefinedintheplayground.corenamespace,thenitcanbeexecutedusingleinrun:
%leinrun
Noname|Hello,World!
%leinrunClojure
Clojure|Hello,World!
Runningtests
Torunallthetestsdefinedunderthe:test-pathsdirective,runtheleintestcommand.SincethedefaulttemplateofLeiningenhasonlyonetestdefinedanditalwaysfails,therewillbeatestfailurereported,asshowninthefollowingcodesnippet.Sincethe:test-pathsparameterintheupdatedconfigurationofproject.cljpointstotest/cljinsteadofthedefaulttest/,thegeneratedcore_test.cljtestsourcefilemustresidewithinthetest/clj/playground/directoryasperthenamespaceandconfiguredtestpathsforteststorun,asshownhere:
%leintest
leintestplayground.core-test
leintest:onlyplayground.core-test/a-test
FAILin(a-test)(core_test.clj:7)
FIXME,Ifail.
expected:(=01)
actual:(not(=01))
Ran1testscontaining1assertions.
1failures,0errors.
Testsfailed.
Generatingreports
AgoodprojectdeliverablemustincludetestandcoveragereportswithassociatedAPIdocumentation.Basedontheconfiguredpluginsunderthepluginsdirectiveofproject.clj,leincanbeusedtogeneratevariousreportsaspertheplugindocumentation.Theexecutionofeachofthepluginsusingtheleincommandandtheircorrespondingresultsareshownhere:
%leintest2junit
Usingtest2junitversion:1.2.2
RunningTests...
Writingoutputto:target/test-reports
Creatingdefaultbuild.xmlfile.
Testing:playground.core-test
Ran1testscontaining1assertions.
>1failures,0errors.
Testsfailed.
Testsfailed.
%leinwith-profiles+testcloverage
Loadingnamespaces:(playground.core)
Testnamespaces:(playground.core-test)
Loadedplayground.core.
Instrumentednamespaces.
Testingplayground.core-test
FAILin(a-test)(core_test.clj:7)
FIXME,Ifail.
expected:(=01)
actual:(not(=01))
Ran1testscontaining1assertions.
1failures,0errors.
Rantests.
Producedoutputinplayground/target/coverage.
HTML:target/coverage/index.html
|-----------------+---------+---------|
|Namespace|%Forms|%Lines|
|-----------------+---------+---------|
|playground.core|17.65|60.00|
|-----------------+---------+---------|
|ALLFILES|17.65|60.00|
|-----------------+---------+---------|
Errorencounteredperformingtask'cloverage'withprofile(s):
'base,system,user,provided,dev,test'
Suppressedexit
%leinwith-profiles+doccodox
GeneratedHTMLdocsinplayground/target/doc
Asaresultoftheexecutionofthecommandsshownintheprecedingsnippet,theoutputoftest2junitisgeneratedunderthetarget/test-reports/xml/playground.core-test.xmlfile,whichisastandardJUnit(https://en.wikipedia.org/wiki/JUnit)testreport.Coveragereportsaregeneratedundertarget/coverage/,whichcontainsanindex.htmlfilethatcanbeviewedusingawebbrowsertoseeadetailedreportandreviewtheinstrumentedcodepathsthatwerecoveredwiththetestcases.Thedocumentationfortheprojectisgeneratedunderthetarget/docfile,whichcontainstheindex.htmlfilethatcanbeviewedinawebbrowsertoreviewthegenerateddocs.
Generatingartifacts
ClojureartifactsaregeneratedasrunnableJARs.TogeneratearunnableJAR,runtheleinuberjarcommand,asshowninthefollowingsnippet.Theartifactcanbeconfiguredusingtheuberjarprofileinproject.clj:
%leinuberjar
Compilingplayground.core
Createdtarget/playground-0.1.0-SNAPSHOT.jar
Createdtarget/playground-0.1.0-SNAPSHOT-standalone.jar
%java-jartarget/playground-0.1.0-SNAPSHOT-standalone.jar
Noname|Hello,World!
%java-jartarget/playground-0.1.0-SNAPSHOT-standalone.jarClojure
Clojure|Hello,World!
ClojureIDE
Anintegrateddevelopmentenvironment(IDE)isasoftwareapplicationthatprovidesutilitiesforprogrammerstodevelop,build,anddebugsoftwareapplications.Itconsistsofacodeeditor,buildautomationtool,andadebugger.ClojurehasagrowingnumberofIDEsavailable,outofwhichEmacs(https://en.wikipedia.org/wiki/Emacs)andVim(https://en.wikipedia.org/wiki/Vim_(text_editor))standout.AlthoughthisbookdoesnotcoverIDEs,whereveroneisreferredto,usesEmacsastheIDE.BothEmacsandVimaretexteditorsthatneedadditionalpluginstosupportClojure.EmacsneedsCIDER(https://github.com/clojure-emacs/cider),whereasVimgetsREPLsupportusingFireplace(https://github.com/tpope/vim-fireplace).
SomeotherClojureIDEsthatarewidelyusedare:
Eclipse(https://www.eclipse.org/home/index.php)withCounterclockwise(http://doc.ccw-ide.org/)LightTable(http://lighttable.com/)SublimeText(https://www.sublimetext.com/)withSublimeREPL(https://github.com/wuub/SublimeREPL)Atom(https://atom.io/)withnrepl(https://atom.io/packages/nrepl)Cursive(https://cursive-ide.com/)
Summary
Inthischapter,wefocusedonthedevelopmentenvironmentfortheHelpingHandsapplication.SinceClojureisourlanguageofchoiceforimplementation,wefirstlookedatthehistoryofClojureandLispandunderstoodwhyitiswellsuitedforourusecase.WealsolookedattheREPLenvironmentandtwobuildtoolsforClojure—LeiningenandBoot.Further,wedefinedareferenceLeiningenprojectconfigurationforourapplicationandlearnedhowtorunanapplicationandtestit.Wealsolearnedhowtogeneratedocumentationandreports,andhowtocreateadeployableartifact.Attheend,webrieflylookedattheClojureIDEsthatcanmakeourapplicationdevelopmentworkeasy.
Inthenextchapter,wewilllearnaboutRESTspecification.WewilllearnhowtodefineRESTAPIsformicroservicesintheHelpingHandsapplicationthatcanhelpwithdirectmessagingamongtheservices.
RESTAPIsforMicroservices
"Comingtogetherisabeginning;keepingtogetherisprogress;workingtogetherissuccess."
-HenryFord
OneofthemodesofinteractionamongmicroservicesisdirectmessagingviaAPIs.RESTAPIsareoneofthewaystomakethedirectmessagingpossibleamongmicroservices.RESTAPIsenableinteroperabilityamongmicroservicesirrespectiveofthetechnologystackinwhichtheyareimplemented.Inthischapter,wewillcoverthefollowingtopics:
TheconceptofRESTHowtodefineRESTURIswithappropriatemethodswithstatuscodesHowtouseREST-basedHTTPURIsusingutilitiessuchascURLRESTAPIsfortheHelpingHandsapplication
IntroducingRESTRESTstandsforrepresentationalstatetransfer,anddefinesanarchitecturalstylefordistributedhypermediasystems.Itfocusesoncreatingindependentcomponentimplementationsthatarestatelessandscalewell.ApplicationsthatsupportRESTdonotstoreanyinformationabouttheclientstateontheserverside.SuchapplicationsrequireclientstomaintainthestatethemselvesandusetheREST-styleimplementations,suchasHTTPAPIsexposedbytheapplication,totransferthestatebetweenthemandtheserver.ClientswhousetheRESTAPImayquerytheserverforthelateststateandrepresentthesamestateattheclientside,thuskeepingtheclientandserverinsync.
Thestateofanapplicationisdefinedbythestateoftheentitiesofthesystem.EntitiescanberelatedtotheconceptofresourcesinRESTarchitecture.AresourceisakeyabstractionofinformationinREST.Itisaconceptualmappingtoasetofentitiesthatisdefinedbytheresourceidentifier.AnyinformationthatcanbenamedorbethetargetoftheRESTresourceidentifiercanbearesource.Forexample,aconsumerisaresourceforaServiceConsumerserviceandanorderisaresourceforanOrderservice.
ResourceidentifiersinRESTaredefinedasAPIendpointsthataremappedtoHTTPURIs.TheURIspointtooneormoreresourcesandallowthecallersto
performoperationsontheresources,asshownintheprecedingimage.AlltheoperationssupportedbyRESTarestateless;thatis,noneoftheclientstateispersistedontheserverside.ThestatelessnatureoftheRESTAPIshelpsincreatingservicesthatscalehorizontallywithoutdependingonsharedstate.
RESTstyleissuitedtomicroservicesfordirectcommunicationandbuildingsynchronousAPIstogetaccesstotheentities.EachentitythatismanagedbyamicroservicecanbedirectlymappedtotheconceptofaresourceinREST.
TheconceptofRESTwasdefinedbyRoyFieldingin2000asapartofhisPhDdissertationonArchitecturalStylesandtheDesignofNetwork-basedSoftwareArchitectures(https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation_2up.pdf).ARESTfulsystemconformstosixguidingprinciples(https://en.wikipedia.org/wiki/Representational_state_transfer#Architectural_constraints)ofclient-server,statelessness,cacheability,layeredsystem,code-on-demand,anduniforminterface.
RESTfulAPIs
WebserviceAPIsthatconformtoRESTarchitectureiscalledRESTfulAPIs.MicroservicesmostlyimplementtheHTTP-basedRESTfulAPIsthatarestatelessandhaveabaseURIandamediatype(https://en.wikipedia.org/wiki/Media_type)fortherepresentationofresources.ItalsosupportspredefinedstandardoperationsthataremappedtoHTTPmethodssuchasGET,POST,PUT,DELETE,andmore.
Forexample,asshowninthefollowingtable,theOrderservicemaydefineanAPI/orderstogetaccesstotheordersthatitmaintains.ItcansupportaGETmethodtolookupalltheordersorgetaspecificorderbyspecifyingtheorderID.ItcanalsoallowclientstocreatenewordersbyusingthePOSTmethodorcreateanorderwithaspecificIDbyusingthePUTmethod.Similarly,itcansupportthePUTmethodtoupdateorderdetailsandtheDELETEmethodtodeleteanorderbyspecifyingtheorderIDexplicitly.
URI HTTPmethod Operation Description
GEThttps://server/orders GET Read Getsalltheorders
GET
https://server/orders/1GET Read Getsthedetailsoforder
withIDas1
POST
https://server/ordersPOST Create Createsaneworderand
returnstheID
PUT
https://server/orders/100PUT Create Createsaneworderwith
IDas100
PUT
https://server/orders/2PUT Update Updatesanexistingorder
withIDas2
DELETE
https://server/orders/1DELETE Delete Deletesanexistingorder
withIDas1
GET,PUT,POST,andDELETEarethemostwidelyusedHTTPmethodsforRESTfulAPIs.OthermethodsincludeHEAD,OPTIONS,PATCH,TRACE,andCONNECT(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods).EachrequestthatissenttotheURIsgeneratesaresponsethatmaybeHTML,JSON,XML,oranydefinedformat.JSONisthepreferredformatformicroservicesthathandlethecoreoperationsoftheapplication.Forexample,theresponsesgeneratedbytheRESTfulAPIsoftheOrderservicewillcontainaJSONobjectoforderdetailsoraJSONarrayoforders,whereeachorderisrepresentedasaJSONobjectwithkey-valuepairscontainingtheorderdetails.
Statuscodes
Statuscodesareimportantforclientstounderstandtheoutcomeoftherequestandtakerequiredactionontheresponse.Successfulrequestsreturnaresponsewiththerequiredrepresentationofresources,butfailedrequestsgeneratearesponserepresentingtheerrorinstead.Statuscodeshelpclientstobewellawareofthecontentoftheresponseandtakenecessaryactionattheclientside.
AllmicroserviceswithRESTfulAPIsmustsupportappropriateHTTPstatuscodes(https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)andsendthemtotheclientbasedontheoutcomeoftherequestedoperation.SomeofthestatuscodesthatmustbeimplementedbyallAPIsofmicroservicesare:
Statuscode Usage
200OK
Standardresponsecodeiftherequestwassuccessful.AGETrequestmustreturntheresourcedetailsintheresponse,andaPOSTrequestmustreturntheoutcomeoftheresponse.
201CreatedSentwhentherequestresultsinthecreationofaresource.TheresponsemustcontaintheURIoftheresource.
400Bad
Request
Client-sideerrorresponsecodeiftherequestcontainsinvalidorinsufficientparameters.Theresponsemustincludedetailsoftheissueswithrespecttorequestparametersforclienttocorrect.
401
Unauthorized
Sentwhenauthenticationisrequiredtoaccessaresourceandeitherclientrequestisnotauthenticatedordoesnotcontainauthentication/authorizationtokensasexpectedbytheserver.
403
Forbidden
Sentwhentheclientrequestisvalidbutitisnotallowedtoaccesstheresource,possiblyduetoinsufficientprivileges.
404Not
Found Sentwhentherequestedresourceisnotfoundontheserver.
405Method
NotAllowedSentwhentherequestedmethodisnotsupportedbytheresource.Forexample,aresourcemaysupportonlytheGETmethod.Inthatcase,sendingaPUTrequestmayresultinaresponsewiththiserrorcode.
500Internal
ServerError
Genericerrorresponsecodesentbytheserverwhentherequestfailswithanunknownerrorthatisnothandledbytheserverandthereisnootherappropriateerrormessagetosend.
TheprecedingstatuscodesmustbeimplementedfortheRESTAPIs.ThereareotherstatuscodesaswellthatmaybeusedbytheAPIs.Toreviewallthedefinedstatuscodes,takealookattheHTTPstatuscodesguide.
Namingconventions
RESTAPIsmustbeorganizedaroundresourcesofthesystemandmustbeeasytounderstandbyjustlookingattheURIs.URIsmustfocusononeresourcetypeatatimewithoperationsmappedtoHTTPrequestmethods.TheymayhaveoneormorerelatedresourcesthatcanbefurthernestedintheURI.
GoodexamplesofURIsare/users,/consumers,/orders,and/services,not/getusers,/showorders,andmore.AbasicruletodefineAPIURIsistotargetnounsasresourcesandverbsasHTTPmethods.Forexample,insteadofcreatingaURIas/getusers,itisrecommendedtohavetheURIas/userswiththeHTTPmethodasGET.Similarly,insteadof/showorders,theURIGET/ordersmustbecreatedtoprovidealltheordersintheresponsetotheclient.GuidelinesfornamingURIsarefurtherexplainedinthefollowingdiagram:
ItisrecommendedtoalwaysuseSSL(https://en.wikipedia.org/wiki/Transport_Layer_Security)byonlyusingHTTPS-basedURIsforalltheAPIs.TheAPIsmustbeversionedtoupgradethemovertimewithoutbreakingtheintegrationwithexistingservices.TospecifytheversionoftheAPI,amajorversioncanbeusedasaprefixtoalltheAPIendpoints.Sub-versionsmustnotbeaddedtotheURI.Ifrequired,theycanbespecifiedusingacustomheaderparameter.
Versionscanalsobeintroducedviaamediatype.Itisperfectlyfine
toincludesub-versionsaswellwiththemediatype.Versionsmayincludebreakingchangesaswellfortheclients,butitisrecommendedtokeepthechangesasbackwardcompatibleaspossible.
URIsmayalsocontainapplicationqualifiersthatsegmenttheAPIendpointsbasedonthetargetcomponentsorbusinessoperations.EachURImustbefocusedonasingleresourcetype,andthatnamemustbeusedinitspluralform.Toretrievespecificresourcesofthespecifiedtype,anIDmustbeappendedtotheURI.HerearesomeoftheexamplesofURIsthatfollowtheeasy-to-understandnamingconvention:
URI DescriptionGEThttps://server/v1/orders GetsalltheordersGEThttps://server:9988/v1/orders GetsalltheordersGEThttps://server/v1/tech/orders GetsalltheordersGEThttps://server/v1/orders/7 GetsanorderwithID7POSThttps://server/v1/orders CreatesaneworderandreturnstheID
PUThttps://server/v1/orders/17UpdatestheorderID17,createsnewifnotpresent
GET
https://server/v1/orders/7/feedbacks GetsallfeedbacksfororderID7GET
https://server/v1/orders/7/feedbacks/1 GetsthefeedbackwithID1fororderID7DELETEhttps://server/v1/orders/3 DeletestheorderwithID3PUThttps://server/v1/orders/7/rate UpdatestheratingfororderID7
GEThttps://server/v1/orders?
fields=ts,id,name
Getsalltheorderswithonlyts,idandnamefieldoftheordersintheresponse
GEThttps://server/v1/orders?
status=open&sort=-ts,name
Getsalltheordersthathavestatusasopen,sortedbytsindescendingorderandnameinascendingorder
UsingRESTfulAPIsviacURLcURLisacommand-lineutilitytotransferdatausingvariousprotocols,includingHTTPandHTTPS.cURLiswidelyusedtotryRESTfulAPIsusingitscommand-lineinterface.Forexample,let'stakealookattheAPIsprovidedbyMetaWeather(https://www.metaweather.com/api/)toqueryalocationandgetitsweatherinformation.SinceitisapublicAPIthatmaintainstheweatherinformationofwell-knownplaces,itallowsonlyGETrequeststogetthedetailsoftheresourcesanddoesnotallowcreatingorupdatingresources.
Togettheweatherofalocation,theMetaWeatherapplicationrequirestheresourceIDasdefinedbythesystem.TogettheresourceID,MetaWeatherprovidesasearchAPItolookuptheresourcesbyasearchquery.ThesearchAPIusestheGETmethodwiththequeryastheURLparameterintherequestandreturnstheJSONresponsewithresourcesthatmatchthequery,asshownhere:%curl-XGET"https://www.metaweather.com/api/location/search/?query=bangalore"[{"title":"Bangalore","location_type":"City","woeid":2295420,"latt_long":"12.955800,77.620979"}]
ThewoeidfieldintheresponseistheresourceIDrequiredbytheweatherAPIthatcanbeusedtogettheweatherdetailsofthequeriedlocation:
%curl-XGET"https://www.metaweather.com/api/location/2295420/"
{
"title":"Bangalore",
"location_type":"City",
"woeid":2295420,
"latt_long":"12.955800,77.620979",
"timezone":"Asia/Kolkata",
"time":"2017-10-24T21:06:34.146240+05:30",
"sun_rise":"2017-10-24T06:11:13.036505+05:30",
"sun_set":"2017-10-24T17:56:02.483163+05:30",
"timezone_name":"LMT",
"parent":{
"title":"India",
"location_type":"Country",
"woeid":23424848,
"latt_long":"21.786600,82.794762"
},
"consolidated_weather":[
{
"id":6004071454474240,
"weather_state_name":"HeavyCloud",
"weather_state_abbr":"hc",
"wind_direction_compass":"NE",
"created":"2017-10-24T15:10:08.268840Z",
"applicable_date":"2017-10-24",
"min_temp":18.458000000000002,
"max_temp":29.392000000000003,
"the_temp":30.0,
"wind_speed":1.9876466767411649,
"wind_direction":51.745585344396069,
"air_pressure":969.60500000000002,
"humidity":63,
"visibility":11.150816730295077,
"predictability":71
},
...
],
"sources":[
{
"title":"BBC",
"slug":"bbc",
"url":"http://www.bbc.co.uk/weather/",
"crawl_rate":180
},
...
]
}
TheMetaWeatherapplicationAPIsalsosupporttheOPTIONSmethodthatcanbeusedtofindoutthedetailsoftheAPI.Forexample,sendingtherequesttothesamesearchAPIwiththeOPTIONSHTTPmethodprovidestherequireddetailsintheresponse:
%curl-XOPTIONS"https://www.metaweather.com/api/location/search/?query=bangalore"
{"name":"LocationSearch","description":"","renders":["application/json"],"parses":
["application/json","application/x-www-form-urlencoded","multipart/form-data"]}
TheMetaWeatherapplicationalsorespondswiththeappropriatestatuscodeandmessageintheresponseforaninvalidresourceID:
%curl-v-XGET"https://www.metaweather.com/api/location/0/"
...
*Trying172.217.26.179...
*Connectedtowww.metaweather.com(172.217.26.179)port443(#0)
...
*compression:NULL
*ALPN,serveracceptedtousehttp/1.1
>GET/api/location/0/HTTP/1.1
>Host:www.metaweather.com
>User-Agent:curl/7.47.0
>Accept:*/*
>
<HTTP/1.1404NotFound
<x-xss-protection:1;mode=block
<Content-Language:en
<x-content-type-options:nosniff
<strict-transport-security:max-age=2592000;includeSubDomains
<Vary:Accept-Language,Cookie
<Allow:GET,HEAD,OPTIONS
<x-frame-options:DENY
<Content-Type:application/json
<X-Cloud-Trace-Context:507eec2981a7028e50e596fcb651acb7;o=1
<Date:Tue,24Oct201716:07:36GMT
<Server:GoogleFrontend
<Content-Length:23
<
*Connection#0tohostwww.metaweather.comleftintact
{"detail":"Notfound."}
RESTAPIsforHelpingHandsTheHelpingHandsapplicationhasConsumerandProviderservicesthatexposetheRESTAPIstomanageconsumersandprovidersfortheapplication.EachserviceproviderintheapplicationcanregisteroneormoreservicesthataremanagedbyServiceAPIs.Apartfromtheseservices,thereisanOrderservicethatmanagesalltheordersplacedbytheconsumersfortheserviceandservedbytheprovidersoftheservice.TheHelpingHandsapplicationalsoprovidesanapplication-wideLookupservicethatprovidesasingleAPItolookupservicesandordersbyvicinity.AllAPIsprovidedbytheHelpingHandsmicroservicesalsohandleerrorswithappropriateerrormessagesintheresponseandthecorrespondingstatuscode.
ConsumerandProviderAPIs
TheAPIsprovidedbytheConsumerandProviderservicestargettheconsumersandprovidersresourcesofthesystem,respectively:
URI Method Params Description
/consumers POST Details(JSON) CreatesanewconsumerandreturnstheconsumerID
/consumers/1 PUTDetailstobeupdated(JSON)
Updatesthedetailsofanexistingconsumer
/consumers GETFields(CSV),sort(CSV),page
Getsalltheconsumersbasedonrequestparams
/consumers/1 DELETE - Deletesthespecifiedconsumer
/providers POST Details(JSON) CreatesanewproviderandreturnstheproviderID
/providers/1 PUTDetailstobeupdated(JSON)
Updatesthedetailsofanexistingprovider
/providers/1/star PUT - Incrementsthestarsfortheproviderbyone
/providers GETFields(CSV),sort(CSV),page
Getsalltheprovidersbasedonrequestparams
/providers/1 DELETE - Deletesthespecifiedprovider
ServiceandOrderAPIs
TheAPIsprovidedbytheServiceandOrderservicestargettheproviderservicesandordersresourcesofthesystem,respectively:
URI Method Params Description
/services POST Details(JSON) CreatesanewserviceandreturnstheserviceID
/services/1 PUTDetailstobeupdated(JSON)
Updatesthedetailsofanexistingservice
/services/1/star PUT - Incrementsthestarsfortheservicebyone
/services GETFields(CSV),sort(CSV),page
Getsalltheservicesbasedonrequestparams
/services/1 DELETE - Deletesthespecifiedservice
/orders POST Details(JSON) CreatesaneworderandreturnstheorderID
/orders/1 PUTDetailstobeupdated(JSON)
Updatesthedetailsofanexistingorder
/orders GETFields(CSV),sort(CSV),page
Getsalltheordersbasedonrequestparams
/orders/1 DELETE - Deletesthespecifiedorder
APIsforauthenticationandauthorizationhavebeenintentionallyleftoutfromthediscussion,butthesewillbeaddressedinPart-4ofthisbook.
Summary
Inthischapter,welearnedabouttheconceptofRESTandhowtodesignRESTfulAPIswithaneasy-to-understandnamingconvention.Wealsolearnedaboutvariousrequestmethodsandstatuscodes,andwhentousewhat.Attheend,werevisitedtheHelpingHandsapplicationtolistdowntheRESTAPIsthatwillberequiredfortheapplicationacrossmicroservices.
Inthenextchapter,wewilltakealookataClojureframeworkcalledPedestal(http://pedestal.io/)thatwewillbeusingtodesignmicroservicesfortheHelpingHandsapplication.PedestalwillalsobeusedtoexposeRESTAPIsforvariousmicroservicesoperations.
IntroductiontoPedestal
"Aconceptualframeworkisa'framethatworks'toputthoseconceptsintopractice."
-PaulHughes
Microservicesareimplementedusingaparticulartechnologystack,whichservesasingle-boundedcontextandprovidesservicesaroundit.Theservicesexposedfortheexternalworldmustbescalableandsupportdirectmessagingaswellasasynchronousrequests.Pedestal(http://pedestal.io/)isonesuchClojureframeworkthatfitsinwelltocreatereliableandscalableservicesformicroservice-basedapplications.ItalsofitsinwellwiththeClojurestackoftheHelpingHandsapplication.Inthischapter,youwill:
LearnaboutthebasicconceptsofPedestalLearnhowtodefinePedestalroutesandinterceptorsLearnhowtohandleerrorswithPedestalinterceptorsLearnhowtopublishoperationalmetricswithPedestalLearnhowtouseServer-sentEventsandWebSocketswithPedestal
PedestalconceptsPedestalisanAPI-firstClojureframeworkthatprovidesasetoflibrariestobuildreliableandhighlyconcurrentservicesthataredynamicinnature.Itisanextensibleframeworkthatisdata-drivenandimplementedusingprotocols(https://clojure.org/reference/protocols)toreducethecouplingbetweenitscomponents.Itfavorsdataoverfunctionsandfunctionsovermacros.Itallowsforthecreationofdata-drivenroutesandhandlersthatcanapplyadifferentbehavioratruntimebasedonincomingrequests.Thismakesitpossibletocreatehighlyflexibleanddynamicservicesthatarewellsuitedformicroservice-basedapplications.Italsosupportsthebuildingofscalableasynchronousservicesusingserver-sentevents(SSE)andWebSockets:
ThePedestalarchitectureisbasedontwomainconcepts,InterceptorsandContextMap,andtwosecondaryconcepts,ChainProvidersandNetworkConnectors.AllthecorelogicofthePedestalframeworkhasbeenimplementedasInterceptors,buttheHTTPconnectionhandlerhasbeenseparatedtocreateaninterfaceforChainProviderthatsetsuptheinitialContextMapandqueueofInterceptorstostarttheexecution.PedestalincludesaservletchainprovideroutoftheboxthatworkswithalltheHTTPserversthatworkwithservlets.ItalsosetsuptherequiredkeysintheContextMapthatareexpectedbytheInterceptorsasperthecontract.PedestalapplicationsarenotlimitedtojustHTTP.Customchainproviderscanbewrittentosupportotherapplicationprotocolsaswell.Theycanalsoworkwithdifferenttransportprotocols,suchasreliableUDPbasedonthetargetchainproviderandtheunderlyingnetworkconnector.
Pedestalinitiallyhadtwoseparateparts—thePedestalapplication
andPedestalserver.PedestalapplicationwasaClojureScript-basedfrontendframeworkthathasbeendiscontinued.NowthefocusisonlyonPedestalservertobuildreliableservicesandAPIs.
InterceptorsPedestalinterceptorsarebasedonasoftwaredesignpatterncalledinterceptor.Interceptorisaserviceextensionthatregisterseventsofinterestwiththeframeworkandisinvokedbytheframeworkwhenthoseeventsoccurwithinthecontrolflow.Oncetheinterceptorisinvoked,itexecutesitsfunctionality,andthenthecontrolflowreturnstotheframework.MostofPedestalcorelogicismadeupofoneormoreinterceptorsthatcanbecomposedtogethertobuildachainofinterceptors.
InterceptorinPedestalisdefinedbyaClojureMap(https://clojure.org/reference/data_structures#Maps)thatcontains:name,:enter,:leave,and:errorkeysasshowninthefollowingdiagram.The:namekeycontainsanamespacedkeyword(https://clojure.org/reference/namespaces)fortheinterceptorandisoptional.The:enterand:leavekeysspecifyaClojurefunctionthattakesaContextMapasinputandreturnsaContextMapasoutput.Eitherthe:enteror:leavekeymustbedefinedfortheinterceptortoberegisteredwiththePedestalframework.Thefunctionspecifiedbythe:enterkeyiscalledbythePedestalframeworkwhenthedataflowsintotheinterceptor.Thefunctionspecifiedby:leaveiscalledwhentheresponseisbeingreturnedbytheinterceptor:
Thefunctionspecifiedby:erroriscalledwhenanexceptioneventistriggeredbytheinterceptorexecutionoriftheexecutionfailswithanerror.Tohandletheexceptionevent,the:errorkeyspecifiesaClojurefunctionthattakestwoarguments—aContextMapandanex-info(https://clojuredocs.org/clojure.core/ex-info)exceptionthatwasthrownbytheinterceptor.IteitherreturnsaContextMap,
optionally,withtheexceptionreattachedforotherinterceptorstohandle,orthrowsanexceptionforthePedestalframeworktohandle.
TheinterceptorchainInterceptorsinPedestalcanbecomposedasaninterceptorchainthatfollowstheChainofResponsibility(https://en.wikipedia.org/wiki/Chain-of-responsibility_pattern)designpattern.Eachinterceptordoesexactlyonejobandwhencomposedtogetherasaninterceptorchain,theyachieveabiggertaskconsistingofoneormorejobs.
TheChainofResponsibilitypatternhelpswithnavigatingthecompositestructureoftheinterceptorchain.ThecontrolflowwithintheinterceptorchainiscontrolledbyaContextMap.SinceaContextMapitselfispassedasinputtoeachinterceptor,interceptorscanoptionallyadd,remove,orreorderinterceptorsinthechain.Thisisoneofthereasonswhymostofthemodulesofawebframework,suchasRouting,ContentNegotiation,RequestHandlers,andmore,arealsoimplementedasinterceptorsbyPedestal.
Interceptorfunctionsfor:enterand:leavemustreturnContextMapasavaluefortheexecutionflowtocontinuewiththenextinterceptor.Ifthefunctionsreturnanil(https://clojure.org/reference/data_structures#nil)value,aninternalservererrorisreportedbythePedestalframeworkandtheexecutionflowterminates.Aninterceptormayreturnacore.async(https://clojure.github.io/core.async/)channelinsteadoftheContextMap.Inthatcase,thechannelistreatedlikeapromisetodelivertheContextMapinthefuture.OncethechanneldeliverstheContextMap,thechainexecutorclosesthechannel:
Thechainexecutorcallseachinterceptorinthecore.asynclibrary'sgoblock(https://clojure.github.io/core.async/#clojure.core.async/go),sooneinterceptormaybecalledonadifferentthreadthanthenextbutallbindingsareconveyedtoeachinterceptor.InscenarioswheretheinterceptormaytakealongtimetoprocesstherequestormakeanexternalAPIcall,itisrecommendedtouseagoblocktosendachannelasthereturnvalueandletPedestalcontinuewiththeexecutionasynchronously.WhenPedestalreceivesachannelasanoutput,ityieldstheinterceptorthreadandwaitsforavaluetobeproducedbythechannel.Onlyonevalueisconsumedfromthechannel,anditmustbeaContextMap.
Asshownintheprecedingdiagram,whileexecutingtheinterceptorchain,all:enterfunctionsarecalledintheorderofinterceptorslistedinthechain.Onceallthe:enterfunctionsoftheinterceptorsarecalled,theresultingContextMapissentthroughthe:leavefunctionoftheinterceptorsbutinreverseorder.Sinceanyoftheinterceptorsinthechaincanreturnasynchronously,Pedestalcreateswhatitcallsavirtualcallstackofinterceptors.Itkeepsaqueueofinterceptorsforwhichithastocallthe:enterfunctionandalsomaintainsastackofinterceptorsforwhichthe:enterfunctionhasbeencalled,butthe:leavefunctionispending.KeepingastackallowsPedestaltocallthe:leavefunctioninthereverseorderofthe:enterfunctioncallsoftheinterceptorsinthechain.BothofthesequeuesandstacksarekeptintheContextMapandareaccessibletotheinterceptors.
SinceinterceptorshaveaccesstotheContextMap,theycanchangetheexecutionplanfortherestoftherequestbymodifyingthesequenceofinterceptorsinthechain.Theycannotonlyenqueueadditionalinterceptorsbutcanalsoterminatetherequesttoskipalltheremaininginterceptorsinthechain.
ImportanceofaContextMap
AcontextisjustaClojureMap(https://clojure.org/reference/data_structures#Maps)thatistakenasinputbytheinterceptorandalsogeneratedasanoutput.ItcontainsallthevaluesthatcontrolaPedestalapplication,includinginterceptors,chains,executionstacks,queues,andcontextvaluesthatmaybegeneratedbyinterceptorsorrequiredbyinterceptorsremainingintheinterceptorchain.AContextMapalsocontainsasequenceofpredicatefunctions.Ifanyoneofthepredicatefunctionreturnstrue,thechainisterminated.AlltherequiredfunctionsandsignalkeysarealsodefinedwithintheContextMaptoenableasynccapabilitiesandfacilitatetheinteractionbetweentheplatformandinterceptors.ThetableshownherelistssomeofthekeysthattheinterceptorchainkeepsintheContextMap:
Key Type Description
:bindingsMap(var->value)
Installedusingwith-bindings(https://clojuredocs.org/clojure.core/with-bindings)priortoexecution
:io.pedestal.interceptor.chain/error Exception Mostrecentexceptionthattriggerederrorhandling
:io.pedestal.interceptor.chain/execution-
id Opaque UniqueIDfortheexecution
:io.pedestal.interceptor.chain/queueQueueofinterceptors
Interceptorstobeexecutednext
:io.pedestal.interceptor.chain/stackStackofinterceptors
Interceptorslefttobeexecuted
:io.pedestal.interceptor.chain/terminators
Collectionofpredicates
Checksforvalidterminationpredicatesaftereach:enterfunction
Ifthe:bindingsmapisalteredbyaninterceptorandreturnedintheoutputContextMap,thenPedestalwillinstallthenewbindingsasthreadlocalbindingspriortotheexecutionofthenextinterceptorinthechain.The:io.pedestal.interceptor.chain/queuecontextkeycontainsalltheinterceptorsthatarelefttobeexecuted.Thefirstinterceptorinthequeueisthenextoneconsideredtobeexecutedbycallingthe:enterfunction.Thiskeymustbeusedonlyfordebuggingpurposes.Tomakeanychangestothequeueorexecutionflow,enqueue,terminate,orterminate-when(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.chain.html#var-enqueue)intheinterceptorchainmustbecalledinsteadofchangingthevalueofthiskey.
Theterminationpredicatesspecifiedbythe:io.pedestal.interceptor.chain/terminatorskeysarecheckedforatruepredicateaftereach:enterfunctioncall.Ifthereisavalidpredicatefound,Pedestalskipsallotherremaininginterceptors':enterfunctionsandbeginsexecutingthe:leavefunctionofinterceptorsinthestacktoterminatetheexecutionflow.
The:io.pedestal.interceptor.chain/stackcontextkeycontainstheinterceptorsforwhichthe:enterfunctionhasalreadybeencalledbutthe:leavefunctionispending.Theinterceptoratthetopofthestackisexecutedfirsttomakesurethatthe:leavefunctioniscalledinthereverseorderof:enterfunctioncalls.
Inadditiontothekeysaddedbytheinterceptorchain,ContextMapmaycontainkeysthatareaddedbyotherinterceptorsaswell.Forexample,aservletinterceptor(http://pedestal.io/reference/servlet-interceptor),providedbyPedestaloutofthebox,addsservlet-specifickeystotheContextMap,suchas:servlet-request,:servlet-response,:servlet-config,and:servlet.WhenworkingwithHTTPserver,ContextMapalsohasthe:requestand:responsekeys,whichhaverequest(http://pedestal.io/reference/request-map)andresponse(http://pedestal.io/reference/response-map)mapsassignedtothem,respectively.
PedestalextendsbeyondjustHTTPservices.YoucanextendservicestoKafka-like(https://kafka.apache.org/)systemsandalsousedifferentprotocolssuchasSCTP,ReliableUDP,UDT,andmore.Thepedestal.servicePedestalmoduleisacollectionofHTTP-specificinterceptors.
CreatingaPedestalservicePedestalprovidesaLeiningen(https://leiningen.org/)templatenamedpedestal-servicetocreateanewprojectwiththerequireddependenciesanddirectorylayoutforaPedestalservice.Tocreateanewprojectusingthetemplate,usetheleincommandwiththetemplatenameandaprojectnameasshownhere:
#Createanewproject'pedestal-play'withtemplate'pedestal-service'
%leinnewpedestal-servicepedestal-play
Retrievingpedestal-service/lein-template/0.5.3/lein-template-0.5.3.pomfromclojars
Retrievingpedestal-service/lein-template/0.5.3/lein-template-0.5.3.jarfromclojars
Generatingapedestal-serviceapplicationcalledpedestal-play.
Theleincommandwillcreateanewdirectorywiththespecifiedprojectnameandaddalltherequireddependenciestotheproject.cljfile.Itwillalsoinitializetheserver.cljandservice.cljfileswiththecodetemplateforasamplePedestalservice.Thecreatedprojectdirectorytreeshouldlookliketheoneshownhere:
#Showthe'pedestal-play'projectdirectorystructure
%treepedestal-play
pedestal-play
├──Capstanfile
├──config
│└──logback.xml
├──Dockerfile
├──project.clj
├──README.md
├──src
│└──pedestal_play
│├──server.clj
│└──service.clj
└──test
└──pedestal_play
└──service_test.clj
5directories,8files
Toruntheproject,justusetheleinruncommandanditwillcompileandstartthesampleservicedefinedbythetemplateatport8080.Totesttheservice,openthehttp://localhost:8080URLandhttp://localhost:8080/aboutinabrowserandobservetheresponse.ThefirstURLreturnstheresponseasHelloWorld!,whereasthesecondURLreturnsClojure1.8.0-servedfrom/about:
BoththeendpointscanalsobeaccessedfromcURLasshownhere:
%curl-vhttp://localhost:8080
*RebuiltURLto:http://localhost:8080/
*Trying127.0.0.1...
*Connectedtolocalhost(127.0.0.1)port8080(#0)
>GET/HTTP/1.1
>Host:localhost:8080
>User-Agent:curl/7.47.0
>Accept:*/*
>
<HTTP/1.1200OK
<Date:Fri,03Nov201706:30:00GMT
<Strict-Transport-Security:max-age=31536000;includeSubdomains
<X-Frame-Options:DENY
<X-Content-Type-Options:nosniff
<X-XSS-Protection:1;mode=block
<X-Download-Options:noopen
<X-Permitted-Cross-Domain-Policies:none
<Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'
'strict-dynamic'https:http:;
<Content-Type:text/html;charset=utf-8
<Transfer-Encoding:chunked
<
*Connection#0tohostlocalhostleftintact
HelloWorld!
%curl-vhttp://localhost:8080/about
*Trying127.0.0.1...
*Connectedtolocalhost(127.0.0.1)port8080(#0)
>GET/aboutHTTP/1.1
>Host:localhost:8080
>User-Agent:curl/7.47.0
>Accept:*/*
>
<HTTP/1.1200OK
<Date:Fri,03Nov201706:30:09GMT
<Strict-Transport-Security:max-age=31536000;includeSubdomains
<X-Frame-Options:DENY
<X-Content-Type-Options:nosniff
<X-XSS-Protection:1;mode=block
<X-Download-Options:noopen
<X-Permitted-Cross-Domain-Policies:none
<Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'
'strict-dynamic'https:http:;
<Content-Type:text/html;charset=utf-8
<Transfer-Encoding:chunked
<
*Connection#0tohostlocalhostleftintact
Clojure1.8.0-servedfrom/about
Toruntheapplicationindevmode,startaREPLsessionviaCIDER(https://cider.readthedocs.io/en/latest/up_and_running/#launch-an-nrepl-server-and-client-from-emacs),andcallthepedestal-play.server/run-devfunctiontostarttheserverindevmode.
UsinginterceptorsandhandlersInthepedestal-playproject,theservice.cljsourcefiledefinestwointerceptors,about-pageandhome-page,whichareusedfortheservice.Additionally,itusestwoHTTP-specificinterceptors,body-params(http://pedestal.io/api/pedestal.service/io.pedestal.http.body-params.html#var-body-params)andhtml-body(http://pedestal.io/api/pedestal.service/io.pedestal.http.html#var-html-body),whichareprovidedoutoftheboxbythepedestal.servicemodule.Thepedestal-play.servicenamespacesnippetisshownherewiththeinterceptordeclarations:
(nspedestal-play.service
(:require[io.pedestal.http:ashttp]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[ring.util.response:asring-resp]))
;;UsedbyGET/about
(defnabout-page
[request]
(ring-resp/response(format"Clojure%s-servedfrom%s"
(clojure-version)
(route/url-for::about-page))))
;;UsedbyGET/
(defnhome-page
[request]
(ring-resp/response"HelloWorld!"))
(defcommon-interceptors[(body-params/body-params)http/html-body])
...
InsteadoftakingaContextMapasinputandreturningtheContextMap,theabout-pageandhome-pageinterceptorstaketherequestmap(http://pedestal.io/reference/request-map)asanargumentandreturntheresponsemap(http://pedestal.io/reference/response-map).SuchinterceptorsarecalledhandlersandaretreatedasfunctionsbythePedestalframework.HandlersinthePedestalframeworkdonothaveaccesstoContextMap;therefore,theycannotchangethesequenceofinterceptorsinthechainandthatisthereasontheycanonlybeusedattheendoftheinterceptorchaintoconstructtheresponsemap.Theresponsefunction,usedbythehome-pageandabout-pagehandlers,isautilityfunctionprovidedbytheRingframework(https://github.com/ring-clojure/ring)tocreateaRingresponsemapwith200status,noheaders,andagivenbodycontent.
Thebody-paramsfunctionreturnsaninterceptorthatparsestherequestbodybasedonitsMIMEtype(https://en.wikipedia.org/wiki/Media_type)andaddstherelevantkeystotherequestmapwiththecorrespondingbodyparameters.Forexample,itwilladda:form-paramskeywithalltheformparametersfortherequestswiththecontenttypeofapplication/x-www-form-urlencoded.Theinterceptorspecifiedbythehtml-bodyvaraddstheContent-Typeheaderparametertotheresponsewiththetypeastext/html;charset=UTF-8.Sincehtml-bodyactsontheresponse,thefunctionforthisinterceptorisdefinedforthe:leavekey,whereasforbody-paramsthefunctionisdefinedforthe:enterkeyofitsinterceptormap.
AlltheinterceptorsinPedestalimplementtheIntoInterceptorprotocol(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.html#var-IntoInterceptor).PedestalextendstheIntoInterceptorprotocoltoaMap,Function,List,Symbol,oraVarandallowsinterceptorstobedefinedinanyoftheseextendedforms.ThemostcommonwayofdefininganinterceptorisaClojureMap(https://clojure.org/reference/data_structures#Maps)with:enter,:leave,and:errorkeys.Itcanbedefinedasafunction,aswell,tobeconsideredahandler,suchastheabout-pageandhome-pageofthepedestal-playproject.
Creatingroutes
InterceptorsinPedestalareattachedtoroutesthatdefinetheendpointsforclientstointeractwiththeapplication.PedestalprovidesthreeprominentwaysofdefiningRoutes(http://pedestal.io/reference/routing-quick-reference#_routes)—verbose,table,andterse.AllthreearedefinedasClojuredatastructureswithverbosebeingdefinedasaMap(https://clojure.org/reference/data_structures#Maps),tableasaset(https://clojure.org/reference/data_structures#Sets),andterseasavector(https://clojure.org/reference/data_structures#Vectors).Thesethreeformatsaredefinedforconvenienceonly.Pedestalinternallyconvertsallroutedefinitionsintotheverboseformbeforeprocessingthemusingtheconvenientfunctionexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes).TheverbosesyntaxisalistofMapswithkeywordsthatdefinearoute.Alistofkeywordsandtheirdescriptionsareshowninthefollowingtable:
Keyword Samplevalue Description:route-name
:pedestal-
play.service/about-page Uniquenamefortheroute:app-name :pedestal-play Optionalnamefortheapplication:path /about/:id PathURI
:method :getHTTPverb;canbe:get,:put,:post,:delete,:any,andmore
:scheme :http Optional,suchas:httpand:https:host somehost.com Optionalhostname:port 8080 Portnumber
:interceptorsVectorofinterceptors
InterceptorMapswith:name,:enter,:leave,and:errorkeys
:query-
constraints
{:name#".+":search#"
[0-9]+"} Constraintsonqueryparameters,ifany
Additionally,thereareafewmorekeysderivedfromthe:pathparameterof
verbosesyntax,asfollows:
Keyword Samplevalue Description:path-re #"/\Qabout\E/([^/]+)" Regexusedtomatchthepath:path-parts ["about":id] PartsofthePathURI:path-params [:id] Pathparameters:path-constraints {:id"([^/]+)"} Constraintsforpathparameters,ifany
Thepedestal.routePedestalmodulecontainstheimplementationforbothRoutesandRouters.RoutersarespecialinterceptorsthattakeRoutesasinputinverboseformandprocesstheincomingrequeststotheRoutesbasedontheimplementation.Routersmakesurethattherequestsareprocessedthroughtheinterceptorsaspertheinterceptorchain.AnychangeintheinterceptorchainbyinterceptorsishandledbyRoutersefficiently.
Inthepedestal-playproject,theservice.cljsourcefiledefinestworoutes,GET/andGET/about,whichareboundedbytheinterceptorchainwithhome-pageandabout-pagehandlersattheendrespectively,asshowninthefollowingcode:
;;TabularRoutes
(defroutes#{["/":get(conjcommon-interceptors`home-page)]
["/about":get(conjcommon-interceptors`about-page)]})
The:app-name,:host,:port,and:schemekeyscanbespecifiedasmapsthatapplytoalltheroutesspecifiedinthelistofroutes,asshowninthefollowingcode:
(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}
["/":get(conjcommon-interceptors`home-page)]
["/about":get(conjcommon-interceptors`about-page)]})
Toseetheverbosesyntaxfortheroutes,usetheexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes)functionasshowninthefollowingREPLsession.Theoutputoftheexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes)functioniswhatispassedtothePedestalrouter.The:route-nameintheverboseformatisderivedfromthenameofthelastinterceptorinthechainorthesymbolattheendofthechainthatresolvestoafunction,suchas:pedestal-play.service/home-page:
;;REPL
pedestal-play.server>(io.pedestal.http.route/expand-routespedestal-
play.service/routes)
({:app-name"PedestalPlay",
:route-name:pedestal-play.service/home-page,
:scheme:http,
:host"localhost",
:port8080,
:path"/",
:method:get,
:path-re#"/\Q\E",
:path-parts[""],
:path-params[],
:interceptors
[{:name:io.pedestal.http.body-params/body-params,
:enter#function[io.pedestal.interceptor.helpers/on-request/fn--9231],
:leavenil,
:errornil}
{:name:io.pedestal.http/html-body,
:enternil,
:leave#function[io.pedestal.interceptor.helpers/on-response/fn--9248],
:errornil}
{:namenil,
:enter#function[io.pedestal.interceptor/eval157/fn--158/fn--159],
:leavenil,
:errornil}]}
{:app-name"PedestalPlay",
:route-name:pedestal-play.service/about-page,
:scheme:http,
:host"localhost",
:port8080,
:path"/about",
:method:get,
:path-re#"/\Qabout\E",
:path-parts["about"],
:path-params[],
:interceptors
[{:name:io.pedestal.http.body-params/body-params,
:enter#function[io.pedestal.interceptor.helpers/on-request/fn--9231],
:leavenil,
:errornil}
{:name:io.pedestal.http/html-body,
:enternil,
:leave#function[io.pedestal.interceptor.helpers/on-response/fn--9248],
:errornil}
{:namenil,
:enter#function[io.pedestal.interceptor/eval157/fn--158/fn--159],
:leavenil,
:errornil}]})
ThesameRoutescanbedefinedinatersesyntaxaswell,usingthevector(https://clojure.org/reference/data_structures#Vectors)ofnestedvectors.Eachvectordefinesanapplication,optionally,withanapplicationname,scheme,host,andport.Eachapplicationdeclarationhasoneormorenestedvectorsthatdefinetheroutes.Eachvectoraddsapathsegmentrepresentingthehierarchicaltreestructureofroutes,asshowninthefollowingcode:
(defroutes
`[["PedestalPlay":http"localhost"8080
["/"{:gethome-page}
^:interceptors[(body-params/body-params)http/html-body]
["/about"{:getabout-page}]]]])
Eachroutevectorcontainsapathsegment,suchas/about,interceptormetadatamap,constraintsmap,verbmap,andchildroutevectors,ifany.Interceptorsdefinedintheinterceptormetadatamapareappliedtoeveryroutedefinedintheverbmap,andtheverbkeyintheverbmapcontainsthevalueofverb-specifichandlerfunctionsoralistofinterceptors.
DeclaringroutersRoutersarefunctionsthatareaddedasinterceptorstothechaintoanalyzetherequestsbasedonthedefinedRoutes.Pedestalcreatesarouterbasedonthevaluesof:io.pedestal.http/routesand:io.pedestal.http/routerkeys,asspecifiedintheservicemap(http://pedestal.io/reference/service-map).TheservicemapcontainsallthedetailsforPedestaltocreateaserviceincludingarouter,routes,chain-providerproperties,andmore.Itactsasabuilder(https://en.wikipedia.org/wiki/Builder_pattern)forPedestalservices.
Pedestalprovidesthreebuilt-inroutersthatcanbespecifiedusingthe:map-tree,:prefix-tree,and:linear-searchkeywords.The:map-treeisthedefaultrouterthathasconstanttimecomplexitywhenappliedtoallroutesthatarestatic.Itfallsbacktoprefix-treeifanyrouteshavepathparametersorwildcards.
Ifthevalueof:io.pedestal.http/routerisspecifiedasafunctionthenthatfunctionisusedtoconstructarouter.Thefunctionmusttakeoneargument,thatis,thecollectionofroutesinverboseformat,andmustreturnarouterthatsatisfiesrouterprotocols(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.router.html#var-Router).
Accessingrequestparameters
ServletChainProvider(http://pedestal.io/api/pedestal.service/io.pedestal.http.impl.servlet-interceptor.html)attachesarequestmaptotheContextMapwiththe:requestkeybeforethefirstinterceptorisinvoked.Therequestmapcontainsalltheform,query,andURLparametersspecifiedbytheclientoftheAPI.Theseparamsaredefinedasmapsofkey-valuepairs,whereeachkeyrepresentstheparameterspecifiedbytheclient.Alltheparamsareoptionalandarepresentonlyiftheyarespecifiedbytheclientintherequest.Hereisalistofthekeysthatarequestmapmaycontain:
Key Usedfor:path-
params PresentifanypathparametersarespecifiedandfoundbytheRouter
:query-
params
Presentifthequery-params(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-query-params)interceptorisused(default)
:form-
params
Presentifthebody-params(http://pedestal.io/api/pedestal.service/io.pedestal.http.body-params.html#var-body-params)interceptorisusedandtheclientsendstherequestwithapplication/x-www-form-urlencodedasthecontenttype
:json-
params
Presentifthebody-paramsinterceptorisusedandtheclientsendstherequestwithapplication/jsonasthecontenttype
:edn-
params
Presentifthebody-paramsinterceptorisusedandtheclientsendstherequestwithapplication/ednasthecontenttype
:params Mergedmapofpath,query,andrequestparameters
Apartfromparameters,therequestmapalsohasthesekeysthatarealwayspresentandcanbeusedbyinterceptorsinthechain:
Key Type Usedfor
:async-
supported? Boolean Trueifthisrequestsupportsasynchronousoperations
:body ServletInputStream Bodyoftherequest
:headers Map Requestheaderssentbytheclientwithallnamesconvertedtolowercase
:path-info String Requestpath,belowthecontextpath;alwayspresent,atleast/
:protocol String Nameandversionoftheprotocolwithwhichtherequestwassent
:query-
string String Thepartoftherequest'sURLafterthe?character
:remote-
addr String IPAddressoftheclient(orthelastproxytoforwardtherequest)
:request-
method KeywordHTTPverbused,inlowercaseandinkeywordformasdeterminedbythemethod-paraminterceptor(default)
:server-
name String Hostnameoftheservertowhichtherequestwassent
:server-
port Int Portnumbertowhichtherequestwassent
:scheme String Thenameoftheschemeusedfortherequest,suchashttp,https,orftp
:uri String RequestURIfromtheprotocolnameuptothequerystring
Toreviewtherequestmap,addanewdebug-pagehandlertothepedestal-playprojectandmapittotheroute/debugasshowninthefollowingcodesnippet.Thehandlerdebug-pagejustreturnstherequestmapintheresponsewithonlytheparameterkeysofinterest.ItalsoconvertsitintoaJSONstringusingtheCheshire(https://github.com/dakrone/cheshire)library:(defndebug-page[request](ring-resp/response
(cheshire.core/generate-string(select-keysrequest[:params:path-params:query-params:form-params]))))
;;CommonInterceptorsusedforallroutes(defcommon-interceptors[(body-params/body-params)http/html-body])
(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}["/":get(conjcommon-interceptors`home-page)]["/about":get(conjcommon-interceptors`about-page)]["/debug/:id":post(conjcommon-interceptors`debug-page)]})
ThecURLrequesttothe/debugroutenowprovidestheentirerequestmapthatcanbeinspectedforthepath,query,andformparamsasshowninthefollowingexample:curl-XPOST-d"formparam=1""http://localhost:8080/debug/1?qparam=1"{"params":{"qparam":"1","formparam":"1"},"path-params":{"id":"1"},"query-params":{"qparam":"1"},"form-params":{"formparam":"1"}}
CreatinginterceptorsAPedestalinterceptorcanbedefinedasamapwiththekeys:name,:enter,:leave,and:error.Forexample,aninterceptormsg-playcanbedefinedforthe/hellorouteofthepedestal-playprojecttochangethequeryparameternametouppercaseatthetimeofentryusingthe:enterfunctionandappendingagreetingatthetimeofexitusingthe:leavefunction.Itisfollowedbythehandlerhello-pagethatreadsthequeryparametersandaddsaHellogreeting.Takealookatthefollowingexample:
;;Handlerfor/helloroute
(defnhello-page
[request]
(ring-resp/response
(let[resp(clojure.string/trim(get-inrequest[:query-params:name]))]
(if(empty?resp)"HelloWorld!"(str"Hello"resp"!")))))
(defmsg-play
{:name::msg-play
:enter
(fn[context]
(update-incontext[:request:query-params:name]clojure.string/upper-case))
:leave
(fn[context](update-incontext[:response:body]
#(str%"Goodtoseeyou!")))})
;;CommonInterceptorsusedforallroutes
(defcommon-interceptors[(body-params/body-params)http/html-body])
(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}
["/":get(conjcommon-interceptors`home-page)]
["/about":get(conjcommon-interceptors`about-page)]
["/debug/:id":post(conjcommon-interceptors`debug-page)]
["/hello":get
(conjcommon-interceptors`msg-play`hello-page)]})
ThecURLrequesttothe/helloroutewiththenamequeryparameternowprovidestheresultasexpectedwithboth:enterand:leaveeventsfiringforthemsg-playinterceptor:
%curl"http://localhost:8080/hello?name=clojure"
HelloCLOJURE!Goodtoseeyou!
Ifthequeryparameterisnotspecified,itreturnstheHelloWorldgreetingaspertheimplementationofthehello-pageinterceptor:
%curl"http://localhost:8080/hello?name="
HelloWorld!Goodtoseeyou!
HandlingerrorsandexceptionsInterceptorsmightthrowanexceptionduetoanerrorintheimplementation.Forexample,trycallingthe/helloroutewithnoqueryparameter.Itfailsandthrowsanexceptionduetotheusageoftheupper-casefunctiononanilvalueofthe:nameparameter.Theexceptionisthrownbythe:enterfunctionofthemsg-playinterceptorbutthereisno:errorfunctiondefinedfortheinterceptortohandletheexception.SucherrorsmustbehandledgracefullyanderrorsmustbereportedtothecallerusingappropriateHTTPstatuscodes.Inthiscase,iftherequiredparameter:nameisnotdefined,thentherouteshouldreturnaresponsewithaHTTP400BadRequest,alongwithameaningfulmessageforthecaller.
PedestalunifieserrorhandlingforbothsynchronousinterceptorsthatreturnContextMap,andasynchronousinterceptorsthatreturnachannel.Itcatchesallexceptionsthrownwithinaninterceptorandbindsittothe:io.pedestal.interceptor.chain/errorkeyintheContextMap.Oncetheerrorisattachedtothekey,Pedestalstartslookingforthenextinterceptorinthechainthathasan:errorfunctionattachedtoit.
Tohandletheexceptionwiththe/hellorouteofthepedestal-playproject,an:errorfunctioncanbedefinedforthemsg-playinterceptor.The:errorfunctioncanthencatchanyexceptionthrownbytheinterceptorandassociateanappropriateresponsewiththeContextMap.Takealookatthefollowingexample:
(defmsg-play
{:name::msg-play
:enter
(fn[context]
(update-incontext[:request:query-params:name]
clojure.string/upper-case))
:leave
(fn[context](update-incontext[:response:body]
#(str%"Goodtoseeyou!")))
:error
(fn[contextex-info]
(assoccontext:response{:status400:body"Invalidname!"}))})
ThecURLrequestforthe/helloroutenowprovidesanappropriateresponsewiththecorrectstatuscodeandamessage,asshowninthefollowingexample:
%curl-i"http://localhost:8080/hello"
HTTP/1.1400BadRequest
...
Invalidname!
The:errorfunctionreceivestwoarguments,ContextMapandanex-infoexception.ItcaneitherreturnaContextMaptocatchtheerrororupdatethe:io.pedestal.interceptor.chain/errorkeywiththeexceptiontolookforahandler.Itcanalsore-throwtheexceptionorthrowanewexceptiontosignalsomethingwentwrongwhilehandlingtheexception.Inbothcases,Pedestalwillstartlookingforahandleroftheexception.Pedestalkeepstrackofalltheexceptionsthatwereoverriddenbyaddingtheminsequencetothekey:io.pedestal.interceptor.chain/suppressedofContextMap.
Pedestalalsoprovidesanerror-dispatch(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.error.html#var-error-dispatch)macrotobuilderror-handling(http://pedestal.io/reference/error-handling#_error_dispatch_interceptor)interceptorsthatusepatternmatchingtoselectaclause.
LoggingThePedestalmodulepedestal.logcontainscomponentsforloggingandalsoreportingruntimeoperationalmetrics.PedestalusesLogback(https://logback.qos.ch/)forlogginganditcanbeconfiguredbycreatingalogback.xmlfileintheprojectconfigdirectory.Natively,logback-classicimplementsSLF4J(https://www.slf4j.org/),whichisusedbyPedestalaswellforlogging.Pedestalimplementseachlogginglevel—trace,debug,info,warn,anderror—asmacrosthattakekey-valuepairsasparameters,printedusingtheprfunction(https://clojuredocs.org/clojure.core/pr).TologanexceptionviaPedestallogger,the:exceptionkeymustbeusedwithajava.lang.Throwableobjectasavalueassignedtoit.
ThedefaultprojecttemplateofPedestalcontainsalogback.xmlfileintheconfigdirectoryalongwiththerelevantdependenciesaddedintheproject.cljfileforrequiredloggerimplementations.Thedefaultloggingconfigurationofthepedestal-playprojectlogsthelogbackconfigurationintheconsoleandalsointhelogfile,asmentionedinthelogback.xmlfile.Takealookatthefollowingexample:%leinrun18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-CouldNOTfindresource[logback.groovy]18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-CouldNOTfindresource[logback-test.xml]18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-Foundresource[logback.xml]at[file:/pedestal-play/config/logback.xml]...18:53:30,686|-INFOinch.qos.logback.core.joran.action.AppenderAction-Abouttoinstantiateappenderoftype[ch.qos.logback.core.rolling.RollingFileAppender]18:53:30,688|-INFOinch.qos.logback.core.joran.action.AppenderAction-Namingappenderas[FILE]18:53:30,691|-INFOinch.qos.logback.core.joran.action.NestedComplexPropertyIA-Assumingdefaulttype[ch.qos.logback.classic.encoder.PatternLayoutEncoder]for[encoder]property18:53:30,711|-INFOin
c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Archivefileswillbelimitedto[64MB]each.18:53:30,712|-INFOinc.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Nocompressionwillbeused18:53:30,713|-INFOinc.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Willusethepatternlogs/pedestal-play-%d{yyyy-MM-dd}.%i.logfortheactivefile18:53:30,716|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-Thedatepatternis'yyyy-MM-dd'fromfilenamepattern'logs/pedestal-play-%d{yyyy-MM-dd}.%i.log'.18:53:30,716|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-Roll-overatmidnight.18:53:30,719|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-SettinginitialperiodtoTueNov0718:53:30IST201718:53:30,719|-WARNinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-SizeAndTimeBasedFNATPisdeprecated.UseSizeAndTimeBasedRollingPolicyinstead18:53:30,721|-INFOinch.qos.logback.core.rolling.RollingFileAppender[FILE]-Activelogfilename:logs/pedestal-play-2017-11-07.0.log...
Oncetheserverisstarted,thedefaultPedestalloggerlogseachrouteaccessrequestasanINFOmessageinthelog:Creatingyourserver...INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208INFOo.e.j.server.handler.ContextHandler-Startedo.e.j.s.ServletContextHandler@62765e11{/,null,AVAILABLE}INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@58fef400{HTTP/1.1,[http/1.1,h2c]}{0.0.0.0:8080}INFOorg.eclipse.jetty.server.Server-Started@4829msINFOio.pedestal.http-{:msg"GET/hello",:line80}
Pedestal'sservletinterceptor(http://pedestal.io/reference/servlet-interceptor)providesa
defaulterrorhandlerthatlogstheHTTPrequestsandalsoexceptions,ifany.Italsoemitstheexceptionstacktraceintheresponsebodyindevelopmentmode.Pedestal'sloggingisbackedbytheLoggerSourceprotocol,whichcanbeimplementedforcustomloggers.
PublishingoperationalmetricsOperationalmetricsareusefultounderstandtheusageandperformanceoftheapplicationbyobservingitsruntimestateasreportedbythemetrics.PedestalprovidesaloggingcomponentthatusestheMetricslibrary(http://metrics.dropwizard.io/3.2.3/)topublishthemetricstoJMX(https://en.wikipedia.org/wiki/Java_Management_Extensions)bydefaultviaMetricRegistry(http://metrics.dropwizard.io/3.1.0/getting-started/#the-registry).TheprotocolimplementedbyPedestalmetricsisMetricRecorder,whichcanbeimplementedforcustommetricsimplementation.Bydefault,MetricRecorderprovidesfourtypesofrecordersasshowninthefollowingtable:
Metricrecorder Usage
Gauge Usedfortheinstantaneousmeasurementofavalue
Counter Usedtoincrement/decrementasinglenumericmetric
Histogram Usedtomeasurethestatisticaldistributionofvalues(min,max,mean,median,percentiles)
Meter Usedtomeasuretherateofatickingmetric
Tocountthenumberofrequestsreceivedforeachrouteofthepedestal-playproject,acountercanbeaddedandincrementedeverytimethecorrespondinghandleriscalled:
(nspedestal-play.service
(:require[io.pedestal.http:ashttp]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.log:aslog]
[ring.util.response:asring-resp]))
(defnabout-page
[request]
(log/counter::about-hits1)
(ring-resp/response(format"Clojure%s-servedfrom%s"
(clojure-version)
(route/url-for::about-page))))
(defnhome-page
[request]
(log/counter::home-hits1)
(ring-resp/response"HelloWorld!"))
(defndebug-page
[request]
(log/counter::debug-hits1)
(ring-resp/response
(cheshire.core/generate-string
(select-keysrequest[:params:path-params:query-params:form-params]))))
(defnhello-page
[request]
(log/counter::hello-hits1)
(ring-resp/response
(let[resp(clojure.string/trim(get-inrequest[:query-params:name]))]
(if(empty?resp)"HelloWorld!"(str"Hello"resp"!")))))
Bydefault,thecounterwillbepublishedviaJMXandcanbelookedupusingtheJVMmonitoringtoolJConsole(https://en.wikipedia.org/wiki/JConsole).TopublishthemetricstoJMX,startthePedestalapplicationinREPLusingpedestal-play.server/run-devandaccessvariousroutesdefinedforthepedestal-playapplication.Next,openJConsoleandconnecttotheclojure.mainprocessformetrics.Itwillstartlistingtheroutemetricsundertheio.pedestal.metricsMBean(https://en.wikipedia.org/wiki/Java_Management_Extensions#Managed_beans).Takealookatthefollowingscreenshot:
UsingchainprovidersPedestalprovidesservletinterceptoraschainprovidersforHTTP-basedwebapplicationsoutofthebox.Itconnectsanyservletcontainertotheinterceptorchain.Bydefault,thePedestalapplicationtemplateusestheJettywebserver(https://www.eclipse.org/jetty/),butitalsohassupportforchainprovidersthatworkwithserversotherthanJettyaswell,suchasImmutant(http://immutant.org/)andTomcat(https://tomcat.apache.org/).
Tostarttheapplicationwiththedefaultserverandchainprovider,thatis,forJetty,runleinrunwithinthepedestal-playapplicationandobservethelogmessages.ItshowsthelogsfortheJettyserver,thatis,theserverinuse:
Creatingyourserver...
INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208
INFOo.e.j.server.handler.ContextHandler-Started
o.e.j.s.ServletContextHandler@62765e11{/,null,AVAILABLE}
INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@4789995a{HTTP/1.1,
[http/1.1,h2c]}{0.0.0.0:8080}
INFOorg.eclipse.jetty.server.Server-Started@4755ms
INFOio.pedestal.http-{:msg"GET/about",:line80}
INFOio.pedestal.http-{:msg"GET/hello",:line80}
Touseadifferentchain-provider,say,Immutant,changetheservicemaptouseImmutantasthechain-provider.Takealookatthefollowingcode:
(defservice{:env:prod
::http/routesroutes
::http/resource-path"/public"
;;Either:jetty,:immutantor:tomcat
::http/type:immutant
::http/port8080
...
})
Also,changetheproject.cljfiletousetheImmutantimplementationasfollows:
(defprojectpedestal-play"0.0.1-SNAPSHOT"
:description"FIXME:writedescription"
:url"http://example.com/FIXME"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
;;Removethislineanduncommentoneofthenextlinesto
;;useImmutantorTomcatinsteadofJetty:
;;[io.pedestal/pedestal.jetty"0.5.3"]
[io.pedestal/pedestal.immutant"0.5.3"]
;;[io.pedestal/pedestal.tomcat"0.5.3"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-
api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
:min-lein-version"2.0.0"
:resource-paths["config","resources"]
...
:main^{:skip-aottrue}pedestal-play.server)
Now,theleinrunlogsshowUndertow(http://undertow.io/)beingused,thatis,thewebserverusedbyImmutantlibrariesfortheweb.TheroutesandloggingworkexactlysameaswiththeJettywebserver:
INFOorg.xnio-XNIOversion3.4.0.Beta1
INFOorg.xnio.nio-XNIONIOImplementationVersion3.4.0.Beta1
WARNio.undertow.websockets.jsr-UT026010:Bufferpoolwasnotseton
WebSocketDeploymentInfo,thedefaultpoolwillbeused
INFOorg.projectodd.wunderboss.web.Web-Registeredwebcontext/
Creatingyourserver...
INFOio.pedestal.http-{:msg"GET/about",:line80}
INFOio.pedestal.http-{:msg"GET/hello",:line80}
Pedestalisnotjustlimitedtowebapplications.InterceptorsinPedestalcanbeusedinmessageprocessinganddataflowapplications,aswell,andarenotlimitedtoonlyrequest/reply-basedwebservices.
Usingserver-sentevents(SSE)Server-sentevents(SSE)(https://www.w3.org/TR/eventsource/)isastandardthatenablesefficientserver-to-clientstreamingusingatwo-partimplementation.ThefirstpartistheEventSourceAPIthatisimplementedattheclientsidetoinitiatetheSSEconnectionwiththeserver,andthesecondpartisthepushprotocolthatdefinestheeventstreamdataformatthatisusedfortheserver-to-clientcommunication.
TheEventSourceAPIofSSEisdefinedasapartoftheHTML5standardbyW3C(https://en.wikipedia.org/wiki/World_Wide_Web_Consortium)andisnowsupportedbyallthemodernwebbrowsers:
SSEsareprimarilyusedtopushreal-timenotifications,updates,andcontinuousdatastreamsfromservertoclientonceaninitialconnectionhasbeenestablishedbytheclient.Generally,thenotificationsandupdatesarepulledbytheclientbysendingarequesttotheAPIsorpolling(https://en.wikipedia.org/wiki/Polling_(computer_science))theserverforupdates.Pollingrequiresanewconnectiontobeestablishedbetweentheclientandserverforeachrequesttopullthenotificationsandupdates,asshownintheprecedingdiagram.SSEinsteadfocusesonthepushmodelinwhichtheconnectionisestablishedoncebytheclientandislong-lived.Allthenotifications,updates,anddatastreamsarepushedbytheserveroverthe
sameclientconnection.Ifaconnectionislost,theclientautomaticallyreconnectswiththeservertokeeptheconnectionactive.TheflowbetweentheserverandtheclientisfurtherillustratedintheprecedingdiagramforbothpollingandSSEmodesofinteraction.
ThePedestalservicecomponentincludessupportforSSEaswell.Itsendsallitseventsasapartofasingleresponsestreamthatiskeptaliveovertimebysendingeventsand/orperiodicheartbeatdata.Iftheresponsestreamisclosedortheconnectionsisinterrupted,theclientcansendarequesttoreopenitandcontinuetoreceivetheeventsnotificationsfromtheserver.
CreatinginterceptorsforSSEAninterceptorforSSEcanbecreatedusingthestart-event-stream(http://pedestal.io/api/pedestal.service/io.pedestal.http.sse.html#var-start-event-stream)functionprovidedbythePedestalservicecomponent.Ittakesafunctionasinputandreturnsaninterceptor.Thefunctionexpectedbythestart-event-streamfunctioniscalledbyPedestaloncetheinitialHTTPconnectionisestablishedwiththeclient;theHTTPresponseispreparedandPedestalnotifiestheclientthatanSSEstreamisstarting.Ittakesachannelasinput(https://clojure.github.io/core.async/),andContextMap.Tosendtheeventstotheclient,thefunctionjustpublishesthemonthechannelprovidedasanargumenttothefunction.Inadditiontothechannelprovidedasanargumenttothefunction,ContextMapalsocontainsachannelassociatedwiththe:response-channelkey.ThischannelisdirectlyconnectedwiththeresponseOutputStreamandmustnotbeusedtosendeventstotheclient.
TocreateaninterceptorforSSEinthepedestal-playproject,defineafunctionsse-stream-readyandpassitasanargumenttothestart-event-stream.Thestart-event-streamreturnsaninterceptorthatisassignedtoaroute/eventsthatinitiatesSSE.Thefunctionsse-stream-readyreadsarequestparametercounterforthenumberofmessagestobesenttotheclientanddefaultstofive.ItpublishesaMaponthechannelevent-chwithtwokeys,:nameand:data,thatcontainastringvalue.Itisrecommendedtousenamedeventsastheyarehelpfulforclientstotakeappropriateactionbasedonthenameoftheevent.Oncetherequirednumberofeventsaresent,itsendsacloseeventandclosesthechannel.Onceitclosesthechannel,Pedestalcleansuptheconnection.Takealookatthefollowingcode:
(nspedestal-play.service
(:require[io.pedestal.http:ashttp]
[io.pedestal.http.sse:assse]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.log:aslog]
[ring.util.response:asring-resp]
[clojure.core.async:asasync]))
(defnsse-stream-ready
"Startssendingcountereventstoclient."
[event-chcontext]
(let[count-num(Integer/parseInt
(or(->(context:request)
:query-params:counter)"5"))]
(loop[countercount-num]
(async/put!
event-ch{:name"count"
:data(strcounter",T:"
(.getId(Thread/currentThread)))})
(Thread/sleep2000)
(if(>counter1)
(recur(deccounter))
(do
(async/put!event-ch{:name"close":data"Iamdone!"})
(async/close!event-ch))))))
(defcommon-interceptors[(body-params/body-params)http/html-body])
(defroutes#{{:host"localhost":port8080:scheme:http}
["/":get(conjcommon-interceptors`home-page)]
["/about":get(conjcommon-interceptors`about-page)]
["/debug/:id":post(conjcommon-interceptors`debug-page)]
["/hello":get
(conjcommon-interceptors`msg-play`hello-page)]
["/events":get
[(sse/start-event-streamsse-stream-ready)]]})
Theroute/eventsisdefinedundertheroutesofthepedestal-playapplicationthatareusedtoreceiveeventsoverSSE.Ithasaninterceptorassignedthatiscreatedbycallingthestart-event-streamfunctionwiththesse-stream-readyfunctionthatpublishestheevents.Whenarequestreachesthisinterceptor,itpausestheinterceptorexecutionandsendsHTTPresponseheaderstotheclientstatingthataneventstreamisstarting,andinitiatesatimedheartbeattokeeptheconnectionalive.Oncetheconnectionisestablished,itcallsthesse-stream-readyfunctionwiththechannelandthecurrentContextMap.
TotrytheSSEendpoint,runthepedestal-playapplicationusingleinrunandusecURLtosendthegetrequesttothe/eventsendpoint.Takealookatthefollowingexample:
%curl-i-XGET"http://localhost:8080/events"
HTTP/1.1200OK
Date:Wed,08Nov201707:07:51GMT
X-Frame-Options:DENY
X-XSS-Protection:1;mode=block
X-Download-Options:noopen
Strict-Transport-Security:max-age=31536000;includeSubdomains
X-Permitted-Cross-Domain-Policies:none
Cache-Control:no-cache
X-Content-Type-Options:nosniff
Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'
'strict-dynamic'https:http:;
Content-Type:text/event-stream;charset=UTF-8
Connection:close
event:count
data:5,T:22
event:count
data:4,T:22
event:count
data:3,T:22
event:count
data:2,T:22
event:count
data:1,T:22
event:close
data:Iamdone!
Bydefault,itsendsfiveevents,butthatcanbecontrolledusingtherequestqueryparametercounter.Takealookatthefollowingexample:
%curl-XGET"http://localhost:8080/events?counter=2"
event:count
data:2,T:22
event:count
data:1,T:22
event:close
data:Iamdone!
TheSSEinterceptorsendsapartialHTTPresponsetotheclientasapartoftheconnectioninitializationprocessitself,therefore,anydownstreaminterceptorsarenotallowedtochangetheContextMapandtheresponsemap.Theycanonlyexaminethem.
PedestalsupportstheuseofLast-Event-ID(https://www.w3.org/TR/eventsource/#last-event-id)aswell,whichallowstheclienttoreconnectandresumefromthepointwhereitgotdisconnected.BasedontheSSEspec(https://www.w3.org/TR/eventsource/),PedestalsupportsassigningastringIDtotheSSEstreamthatcanbereferredtobytheclientintheLast-Event-IDheadertoresume.
UsingWebSocketsWebSocketsisacommunicationsprotocol(https://en.wikipedia.org/wiki/Communication_protocol)thatprovidesfull-duplex(https://en.wikipedia.org/wiki/Duplex_(telecommunications)#FULL-DUPLEX)communicationchannelsoverasingleTCPconnectionbetweenclientandserver.ItallowsclientstosendmessagestotheserverandreceiveservereventsoverthesameTCPconnectionwithoutpolling.ComparedtoServer-SentEvents(SSE)(https://www.w3.org/TR/eventsource/),WebSocketssupportfull-duplexcommunicationbetweenclientandserverinsteadofaone-waypush.Also,SSEsareimplementedoverHTTP,whichisanentirelydifferentTCPprotocolcomparedtoWebSocket.Althoughbothprotocolsaredifferent,theybothdependontheTCPlayer.
RFC6455(https://tools.ietf.org/html/rfc6455)statesthatWebSocketisdesignedtoworkoverHTTPports80and443aswellastosupportHTTPproxiesandintermediaries,thusmakingitcompatiblewiththeHTTPprotocol.Toachievecompatibility,theWebSockethandshakeusestheHTTPUpgradeheader(https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header)tochangefromtheHTTPprotocoltotheWebSocketprotocol.
Server-senteventsareusefultosendnotificationsoralertsfromtheserverasandwhentheyoccur.Iftheapplicationrequiresaninteractivesessionbetweentheclientandtheserver,thenWebSocketsmustbepreferred.
UsingWebSocketwithPedestalandJettyPedestalprovidesout-of-the-boxsupportforJettyWebSocketsasapartofitspedestal.jettymodule.TocreateandregisteraWebSocketendpoint,Pedestalprovidestheadd-ws-endpointsfunctionthatacceptsaServletContextHandlerandaMapofWebSocketpathstotheactionMap.BasedontheprovidedWebSocketpaths,itproducesthecorrespondingservletsandaddsthemtothecontextoftheservletcontainer,thatis,Jettyinthiscase.TheservletcontainerthenmakestheWebSocketpathsavailablefortheclientstoconnecttousingtheWebSocketprotocol.TheWebSocketendpointsarecommunicatedtotheJettycontainerusingthe:context-configuratorkeyofthemapassignedtothe::http/container-optionskeyofPedestal'sservicemap.
Thepedestal-playprojectdefinesaws-pathsmapthatcontainsasingleWebSocketpath,/chat.Theactionsdefinedforthe/chatpathare:on-connect,:on-text,:on-binary,:on-error,and:on-close.Thestart-ws-connectionfunction,providedbyPedestal,acceptsafunctionoftwoarguments—theJettyWebSocketsessionanditspairedcore.asyncchannel—andreturnsafunctionthatcanbeusedasan:on-connectactionhandler.Forotheractions,thesamplepedestal-playprojectjustlogsamessage.Takealookatthefollowingexample:
(nspedestal-play.service
(:require[io.pedestal.http:ashttp]
[io.pedestal.http.sse:assse]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.log:aslog]
[io.pedestal.http.jetty.websockets:asws]
[ring.util.response:asring-resp]
[clojure.core.async:asasync]))
;;Atomtoholdclientsessions
(defws-clients(atom{}))
(defnnew-ws-client
"Keepstrackofallclientsessions"
[ws-sessionsend-ch]
(async/put!send-ch"Welcome!")
(swap!ws-clientsassocws-sessionsend-ch))
(defws-paths
{"/chat"{:on-connect(ws/start-ws-connectionnew-ws-client)
:on-text(fn[msg]
(log/info:msg(str"Client:"msg)))
:on-binary(fn[payloadoffsetlength]
(log/info:msg"BinaryMessage!":bytespayload))
:on-error(fn[t]
(log/error:msg"WSErrorhappened":exceptiont))
:on-close(fn[num-codereason-text]
(log/info:msg"WSClosed:"
:reasonreason-text))}})
(defservice{:env:prod
::http/routesroutes
::http/resource-path"/public"
::http/type:jetty
::http/port8080
;;Optionstopasstothecontainer(Jetty)
::http/container-options
{:h2c?true
:h2?false
:ssl?false
:context-configurator#(ws/add-ws-endpoints%ws-paths)}})
Inthepedestal-playexample,thefunctionprovidedtothestart-ws-connectionfunctionisnew-ws-client,afunctionthatsendsaWelcome!messagetoeachnewclientandkeepstrackofclientsessions.Thepedestal-playexamplealsodefinesacoupleofutilitymethods,send-and-close!andsend-message-to-all!,tosendmessagesfromtheserversidetotheclientsconnectedtotheWebSocket.Takealookatthefollowingexample:
(defnsend-and-close!
"Utilityfunctiontosendmessagetoaclientandclosetheconnection"
[message]
(let[[ws-sessionsend-ch](first@ws-clients)]
(async/put!send-chmessage)
(async/close!send-ch)
(swap!ws-clientsdissocws-session)
(log/info:msg(str"ActiveConnections:"(count@ws-clients)))))
(defnsend-message-to-all!
"Utilityfunctiontosendmessagetoallclients"
[message]
(doseq[[^org.eclipse.jetty.websocket.api.Sessionsessionchannel]
@ws-clients]
(when(.isOpensession)
(async/put!channelmessage))))
Totestthe/chatWebSocket,startthepedestal-playapplicationinREPLusingthepedestal-play.server/run-devfunction:
pedestal-play.server>(defsrv(run-dev))
Creatingyour[DEV]server...
INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208
INFOo.e.j.server.handler.ContextHandler-Started
o.e.j.s.ServletContextHandler@15d129a1{/,null,AVAILABLE}
INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@766b5ac6{HTTP/1.1,
[http/1.1,h2c]}{0.0.0.0:8080}
INFOorg.eclipse.jetty.server.Server-Started@220932ms
#'pedestal-play.server/srv
pedestal-play.server>
Now,opentheJavaScriptconsoleinawebbrowserandstarttheWebSocketsessionusingthefollowingcommands:
//connectstothe'/chat'endpointwith'ws'protocol
w=newWebSocket("ws://localhost:8080/chat")
//logallthemessagesreceivedfromserveronconsole
w.onmessage=function(e){console.log(e.data);}
//messagetobeshownwhenserverclosestheconnection
w.onclose=function(e){
console.log("Theconnectiontotheserverhasclosed.");}
//sendamessagetoserver
w.send("HellofromtheClient-1!");
AnymessagethatissentfromtheclientisreceivedbytheserverandloggedontheREPL.Similarly,anymessagesentbytheserverusingthefunctionsend-message-to-all!isbroadcastedtoalltheactiveclientconnections.ToclosetheWebSocketconnection,callthesend-and-close!functionthatwillpickthefirstclientconnection,sendamessage,andcloseit.Italsologsthenumberofactiveclientconnectionsasshowninthefollowingcode:
INFOpedestal-play.service-{:msg"Client:HellofromtheClient-1!",:line113}
INFOpedestal-play.service-{:msg"Client:HellofromtheClient-2!",:line113}
pedestal-play.server>(pedestal-play.service/send-message-to-all!"HellofromPedestal
Server!")
nil
pedestal-play.server>(pedestal-play.service/send-and-close!"GoodbyefromPedestal
Server!")
INFOpedestal-play.service-{:msg"ActiveConnections:1",:line102}
nil
INFOpedestal-play.service-{:msg"WSClosed:",:reasonnil,:line119}
pedestal-play.server>(pedestal-play.service/send-and-close!"GoodbyefromPedestal
Server!")
INFOpedestal-play.service-{:msg"ActiveConnections:0",:line102}
nil
INFOpedestal-play.service-{:msg"WSClosed:",:reasonnil,:line119}
pedestal-play.server>
ThefollowingscreenshotcapturestheWebSocketinteractionsamongaREPLsessionandtwobrowserclients,connectedviaaJavaScriptconsole:
Summary
Inthischapter,welearnedabouttheconceptsofthePedestalframeworkandhowtousethemtocreateAPIs.WealsolearnedhowtologusefulinformationfordebuggingandmonitoringtheruntimestateoftheapplicationusingJMXmetrics.WealsolookedatvariouswebserverpluginsthatcanbeusedwithPedestal.Finally,welookedathowPedestalcanbeusedforSSEsandWebSocketsforclient-serverinteraction.
Inthenextchapter,wewilltakealookataDatomicdatabasethatwillbeusedforpersistencebythemicroservicesoftheHelpingHandsapplication.DatomiciswritteninClojureandfitsinwellwiththeHelpingHandsapplication,whichrequirestransactionsandtemporalqueries.
AchievingImmutabilitywithDatomic
"Mostofthebiggestproblemsinsoftwareareproblemsofmisconception."
-RichHickey
Microservicesdependontheunderlyingdatabasetoreliablystoreandretrievedata.OftenapplicationslikeHelpingHandsneedtostoreusertransactionsconsistentlyalongwithuserlocationsthatmaychangeovertime.Insteadofupdatingtheuserlocationpermanentlyandlosingthehistoryofthechanges,agoodapplicationmustmaintainthechangeindatasothatitcanbequeriedovertime.Suchrequirementsexpectthedatastoredinthedatabasetobeimmutable.Datomic(http://www.datomic.com/)isonesuchdatabasethatnotonlyprovidesdurabletransactionsbutalsohastheconceptofimmutabilitybuiltintoitscoresothatuserscanquerythestateofthedatabaseoveraperiodoftime.DatomicisalsowritteninClojure,whichisthetechnologystackofchoicefortheHelpingHandsapplication.Inthischapter,youwilllearnaboutthefollowing:
DatomicarchitectureanditsdatamodelHowtostoreandretrievedataasfactswithDatomicDatalogquerylanguagetoretrievefactsHowtoqueryimmutablefactswithanexample
DatomicarchitectureDatomicisadistributeddatabasethatsupportsACID(http://docs.datomic.com/acid.html)transactionsandstoresdataasimmutablefacts.Datomicisfocusedonprovidingarobusttransactionmanagertokeeptheunderlyingdataconsistent,adatamodeltostoreimmutablefacts,andaqueryenginetohelpretrievedataasfactsovertime.Insteadofhavingitsownstorage,itreliesonanexternalstorageservice(http://docs.datomic.com/storage.html)tostorethedataondisk.
DatomicversustraditionaldatabaseAtypicaldatabaseisimplementedasamonolithicapplicationthatcontainsthestorageengine,queryengine,andthetransactionmanagerallpackagedasasingleapplicationtowhichclientsconnecttostoreandretrievedata.Datomic,ontheotherhand,takesaradicalapproachofseparatingouttheTransactionManager(Transactor)asaseparateprocesstohandleallthetransactionsandcommitthedatatoanunderlyingStorageServicethatactsasapersistencestoreforallthedatamanagedbyDatomic.Ahigh-levelarchitectureofDatomicanditscomparisonwithtraditionaldatabasesisshownhere:
ClientsofDatomicarecalledpeersandhavetheapplicationcodeandpeerlibrary(http://docs.datomic.com/integrating-peer-lib.html)thatconnectwiththeTransactortostoredataconsistently.Peersalsoquerytheunderlyingstorageservicefordataandmaintainacachetoreducetheloadontheunderlyingstorageservice.PeersalsoreceiveupdatesfromtheTransactoraswellandaddtothecache.PeersinDatomicarethickclientsthatcanbeconfiguredtocache(http://docs.datomic.com/caching.html)thedatain-memory,oruseexternalcachingsystemslikeMemcached(https://en.wikipedia.org/wiki/Memcached)tostoretheobjects.Thecachemaintainedbypeersalwayscontainsimmutablefactsthatarealwaysvalid.
DatomicalsohasaPeerServerthatallowslightweightclientslikethatof
traditionaldatabasestoconnecttoitdirectlytoquerythedatabase.Itactsasacentralqueryprocessorforalltheclientsconnectedtoit.Datomicprovidesaconsole(http://docs.datomic.com/console.html)aswell,whichhasagraphicaluserinterfacetomanageschema,examinetransactions,andexecutequeries.
Datomicisdesignedfortransactionaldataandmustbeusedtostoreuserprofiles,orders,inventorydetails,andmore.Itshouldnotbeusedforhigh-throughputusecasessuchasthosefoundinIoT(https://en.wikipedia.org/wiki/Internet_of_things),whichrequiresatime-seriesdatabasetostoreincomingdatawithhighvelocity.
DevelopmentmodelDatomicprovidestwodevelopmentmodels(http://docs.datomic.com/clients-and-peers.html)—PeerandClient.BoththemodelsrequireaTransactor(http://docs.datomic.com/transactor.html)toberunningtohandlethetransactionsandstorethedataconsistently.IntheClientmodel,aPeerServer(http://docs.datomic.com/peer-server.html)isrequiredinadditiontotheTransactortocoordinatethestorageandqueryrequestsfromtheclients.
Boththedevelopmentmodelsandtheparticipatingcomponentsareshowninthefollowingdiagram:
ClientsarelightweightinthecaseofaclientmodelasalltheinteractionwiththeTransactorandstorageengineishandledbythePeerServeronbehalfoftheclient,butitaddsanadditionalhopofrequestsasalltherequestsareroutedthroughPeerServerinsteadofdirectlyconnectingwiththeTransactorandthestorageengine.DatomicprovidesaseparatePeerLibraryandClientLibrary(http://docs.datomic.com/project-setup.html)forpeerandclientmodes,respectively.
DatamodelDatomicstoresdataasfactswitheachfactbeingafive-tuple(https://en.wikipedia.org/wiki/Tuple).ThesefactsarecalledDatoms.EachdatomconsistsofanEntityID,Attribute,andValuethatformthefirstthreepartsofthefive-tuple.Thefourthpartdefinesthetimestampatwhichthefactwascreatedandholdstrue.Thefifthpartconsistsofabooleanvaluethatdetermineswhetherthedefineddatomisanadditionorretractionofafact.Multipledatomsofthesameentitycanberepresentedasanentitymapwithanattributeandvalueaskey-valuepairs.TheEntityIDfortheentitymapisdefinedusingthekey:db/id:
Intheprecedingexample,therearefourattributes—order/name,:order/status,:order/rating,and:order/contactdefinedfortheorderentitywiththeID1234.
SchemaEachdatabaseinDatomichasanassociatedschemathatdefinesalltheattributesthatcanbeassociatedwiththeentities.Italsodefinesthetypeofvalueeachattributecancontain.Theattributesarethemselvestransactedasdatoms,thatis,theyarealsoconsideredasentitieswithassociatedattributesthataredefinedbyDatomic.TheattributessupportedbyDatomicforschemadefinitionsareasshowninthefollowingtable:
Attribute Type Description
:db/identNamespacedkeyword
Uniquenameofanattributeintheform<namespace>[.<nested-namespace>]/<name>,suchas:order/name.Namespacesareusefultopreventcollisionsbuttheycanbeomitted.:dbisarestrictednamespaceusedbyDatomicinternally.
:db/valueType Keyword
Definesthetypeofvalue.Supportedtypesare:
:db.type/keyword
:db.type/string
:db.type/boolean
:db.type/long
:db.type/bigint
:db.type/float
:db.type/double
:db.type/bigdec
:db.type/ref
:db.type/instant
:db.type/uuid
:db.type/uri
:db.type/bytes
:db/cardinality
Specifieswhethertheattributeissingle-valuedormulti-valued.Possiblecardinalitytypesare:
Keyword :db.cardinality/one
:db.cardinality/many
Datomicalsodefinessomeoptionalschema(http://docs.datomic.com/schema.html)attributessuchas:db/doc,:db/unique,:db/index,:db/fulltext,:db/isComponent,and:db/noHistory.The:db/ident,:db/valueType,and:db/cardinalityattributesaremandatory.
Datomicalsosupportsindexes(http://docs.datomic.com/indexes.html)thatcanbeenabledforanattributeusingthe:db/indexschemaattribute.Internally,DatomicmaintainsfourindexesthatcontaindatomsorderedbyEAVT,AEVT,AVET,andVAET,whereEisentity,Aisattribute,Visvalue,andTistransaction.
UsingDatomicDatomiccanbedownloadedfreelyfromitsGetDatomic(http://www.datomic.com/get-datomic.html)website.TostartwithDatomic,downloadtheDatomicfreeeditionthatincludesamemorydatabaseandembeddedDatalogqueryengine.ThefreeeditionisalsolimitedtotwosimultaneouspeersandembeddedstoragethatshouldbegoodenoughtotryoutDatomicfeaturesandworkwithitsdatamodel.
GettingstartedwithDatomicTosetupDatomic,downloadandextractthefreeedition'sdatomic-free-x.x.xxxx.xx.zipfile.ThefreeversionofDatomicdoesnotrequireanylicensekey.Forotherversions,registrationismandatorytoobtainalicensekeywhichmustbeaddedtotransactorpropertiesforDatomictowork.DatomicdistributioncontainstwoJARs,datomic-free-x.x.xxxx.xx.jaranddatomic-transactor-free-x.x.xxxx.xx.jar.Thedatomic-freeJARfilecontainsapeerlibraryanddatomic-transactorcontainstheimplementationofthetransactor.ThedistributionalsocontainsabinfolderthathasalltherequiredscriptstostartDatomiccomponentsasshownhere:datomic-free-0.9.5561.62├──bin├──CHANGES.md├──config├──COPYRIGHT├──datomic-free-0.9.5561.62.jar├──datomic-transactor-free-0.9.5561.62.jar├──lib├──LICENSE├──pom.xml├──README├──resources├──samples└──VERSION
5directories,8files
Tostartwiththefreeedition,createanewClojureLeiningenprojectandaddthedatomic-freedependency.Tobuilduponthepedestal-playproject,addthedependenciestotheexistingprojectconfigurationfileproject.cljofprojectpedestal-playasshownhere:(defprojectpedestal-play"0.0.1-SNAPSHOT":description"FIXME:writedescription":url"http://example.com/FIXME":license{:name"EclipsePublicLicense":url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"][io.pedestal/pedestal.service"0.5.3"][frankiesardo/route-swagger"0.1.4"]
;;Removethislineanduncommentoneofthenextlinesto;;useImmutantorTomcatinsteadofJetty:[io.pedestal/pedestal.jetty"0.5.3"];;[io.pedestal/pedestal.immutant"0.5.3"];;[io.pedestal/pedestal.tomcat"0.5.3"]
;;DatomicFreeEdition[com.datomic/datomic-free"0.9.5561.62"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-api]][org.slf4j/jul-to-slf4j"1.7.22"][org.slf4j/jcl-over-slf4j"1.7.22"][org.slf4j/log4j-over-slf4j"1.7.22"]]:min-lein-version"2.0.0":resource-paths["config","resources"]...:main^{:skip-aottrue}pedestal-play.server)
Thedependencyofcom.datomic/datomic-freewillpulltherequireddependencyfromClojars.Datomicdistributionalsoprovidesabin/maven-installscripttoinstalltheJARshippedinthedistributioninthelocalMaven(https://maven.apache.org/)repositoryfromwhereLeiningencanpullitfortheproject.
Now,startaREPLusingleinreplorjack-inusingtheEmacsCIDERplugintostartusingDatomicAPIs.Thenamespacerequiredtoaccessthepeerlibraryisdatomic.api.IncludethenamespaceinaREPLsessionasshownhere:%leinreplnREPLserverstartedonport33835onhost127.0.0.1-nrepl://127.0.0.1:33835REPL-y0.3.7,nREPL0.2.12Clojure1.8.0JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13Docs:(docfunction-name-here)(find-doc"part-of-name-here")
Source:(sourcefunction-name-here)Javadoc:(javadocjava-object-or-class-here)Exit:Control+Dor(exit)or(quit)Results:Storedinvars*1,*2,*3,anexceptionin*e
pedestal-play.server=>(require'[datomic.api:asd])nilpedestal-play.server=>
TheDatomicfreeeditioncomeswithin-memorystorage.Thedatastoredwiththein-memorystorageisonlyavailableforthelifetimeoftheapplicationprocesswhenworkingwiththeDatomicfreeedition.
Connectingtoadatabase
Toconnecttoadatabase,first,definethedatabaseURIandcreateadatabaseusingthecreate-databasefunctionofthedatomic.apinamespace.IttakesasinputadatabaseURIthatdefinesthestorageenginetobeused,thatis,memforin-memoryandthedatabasetobecreated,thatis,hhorderforHelpingHandsorders.Itreturnstrueifthedatabaseiscreatedsuccessfullyasshownhere:
pedestal-play.server>(require'[datomic.api:asd])
nil
pedestal-play.server>(defdburi"datomic:mem://hhorder")
#'pedestal-play.server/dburi
pedestal-play.server>(d/create-databasedburi)
true
pedestal-play.server>
Oncethedatabaseiscreated,connecttoitusingtheconnectfunctionprovidedbythedatomic.apinamespace.Itreturnsadatomic.peer.LocalConnectionobjectthatcanbeusedtotransactwiththedatabase.Takealookatthefollowingexample:
pedestal-play.server>(defconn(d/connectdburi))
#'pedestal-play.server/conn
pedestal-play.server>conn
#object[datomic.peer.LocalConnection0x299180eb
"datomic.peer.LocalConnection@299180eb"]
pedestal-play.server>
TransactingdataDatomicneedstoknowabouttheattributestobeusedfortheentitiesinthedatabasebeforehand.Bothattributes(schema)andfacts(data)aretransactedasdatomsusingthetransactfunctionofthedatomic.apinamespace.Datomscanbetransactedindividuallyorclubbedtogetherasapartofsingletransactionbywrappingtheminavector(https://clojure.org/reference/data_structures#Vectors).Forexample,alltheattributesforhhordercanbetransactedusingasingletransactionbyspecifyingalltheentitymapstogetherwithinavectorandpassingthatvectorasanargumenttothetransactfunctionasshowninthefollowingexample:
pedestal-play.server>
(defresult
(d/transactconn[{:db/ident:order/name
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"DisplayNameofOrder"
:db/indextrue}
{:db/ident:order/status
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"OrderStatus"}
{:db/ident:order/rating
:db/valueType:db.type/long
:db/cardinality:db.cardinality/one
:db/doc"Ratingfortheorder"}
{:db/ident:order/contact
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"ContactEmailAddress"}]))
#'pedestal-play.server/result
Ifthe:db/idkeyisnotdefinedasapartoftheentitymap,itisaddedbyDatomicautomatically.ThetransactfunctiontakesasparameteraDatomicconnectionandavectorofoneormoredatomstotransact.Itreturnsapromise(https://en.wikipedia.org/wiki/Futures_and_promises)thatcanbedereferencedtoseethetransacteddatoms,aswellasthebeforeandafterstateofthedatabase.Takealookatthefollowingexample:
pedestal-play.server>(pprint@result)
{:db-beforedatomic.db.Db@6c5316fa,
:db-afterdatomic.db.Db@351b51d3,
:tx-data
[#datom[1319413953431250#inst"2017-11-22T13:01:30.632-00:00"13194139534312true]
#datom[6310:order/name13194139534312true]
#datom[63402313194139534312true]
#datom[63413513194139534312true]
#datom[6362"DisplayNameofOrder"13194139534312true]
#datom[6344true13194139534312true]
#datom[6410:order/status13194139534312true]
#datom[64402313194139534312true]
#datom[64413513194139534312true]
#datom[6462"OrderStatus"13194139534312true]
#datom[6510:order/rating13194139534312true]
#datom[65402213194139534312true]
#datom[65413513194139534312true]
#datom[6562"Ratingfortheorder"13194139534312true]
#datom[6610:order/contact13194139534312true]
#datom[66402313194139534312true]
#datom[66413513194139534312true]
#datom[6662"ContactEmailAddress"13194139534312true]
#datom[0136513194139534312true]
#datom[0136413194139534312true]
#datom[0136613194139534312true]
#datom[0136313194139534312true]],
:tempids
{-922330166810959814363,
-922330166810959814264,
-922330166810959814165,
-922330166810959814066}}
Oncetheattributesaredefinedforthehhorderdatabase,theycanbeassociatedwiththeentities.Toaddaneworder,transactwiththeregisteredattributesasshowninthefollowingexample:
pedestal-play.server>
(deforder-result
(d/transactconn[{:db/id1
:order/name"CleaningOrder"
:order/status"Done"
:order/rating5
:order/contact"abc@hh.com"}
{:db/id2
:order/name"GardeningOrder"
:order/status"Pending"
:order/rating4
:order/contact"def@hh.com"}]))
#'pedestal-play.server/order-result
Forexample,twoorderswithIDs1and2havebeenaddedtothehhorderdatabase,whichcannowbequeriedviaits:db/idorotherdefinedattributevalues.
UsingDatalogtoquery
Datomicofferstwowaystoretrievethedatafromthedatabase—pull(http://docs.datomic.com/pull.html)andquery(http://docs.datomic.com/query.html).ThequerymethodofretrievingfactsfromDatomicdatabasesusesanextendedformofDatalog(https://en.wikipedia.org/wiki/Datalog).Toquerythedatabase,theqfunctionofdatomic.apineedstoknowthestateofthedatabasetorunthequeryon.Thecurrentstateofthedatabasecanberetrievedusingthedbfunctionofdatomic.api.Takealookatthefollowingexample:
pedestal-play.server>(d/q'[:find?e?n?c?s
:where[?e:order/rating5]
[?e:order/name?n]
[?e:order/contact?c]
[?e:order/status?s]]
(d/dbconn))
#{[1"CleaningOrder""abc@hh.com""Done"]}
pedestal-play.server>
EachDatomicquerymusthaveeithera:findand:whereclauseora:findand:inclausepresentasapartofthequeryconstruct.Aquery,whengivenasetofclauses,scansthroughthedatabaseforallthefactsthatsatisfythegivenclausesandreturnsalistoffacts.Datomicquerygrammar(http://docs.datomic.com/query.html#grammar)definesallthepossiblewaystoquerythedatabase.Herearesomeoftheexamplestoquerythehhorderdatabase:
;;ReturnsonlytheentityIDoftheentitiesmatchingtheclause
(d/q'[:find?e
:where[?e:order/rating5]]
(d/dbconn))
#{[1]}
;;findalltheentitieswiththethreeattributesandentityID
(d/q'[:find?e?n?c?s
:where[?e:order/name?n]
[?e:order/contact?c]
[?e:order/status?s]]
(d/dbconn))
#{[1"CleaningOrder""abc@hh.com""Done"][2"GardeningOrder""def@hh.com"
"Pending"]}
;;using'or'clause
(d/q'[:find?e?n?c?s
:where(or[?e:order/rating4][?e:order/rating5])
[?e:order/name?n]
[?e:order/contact?c]
[?e:order/status?s]]
(d/dbconn))
#{[1"CleaningOrder""abc@hh.com""Done"][2"GardeningOrder""def@hh.com"
"Pending"]}
;;usingpredicates
(d/q'[:find?e?n?c?s
:where[?e:order/rating?r]
[?e:order/name?n]
[?e:order/contact?c]
[?e:order/status?s]
[(<?r5)]]
(d/dbconn))
#{[2"GardeningOrder""def@hh.com""Pending"]}
AchievingimmutabilityAllthefactspresentinaDatomicdatabaseareimmutableandarevalidforanygiventimestamp.Forexample,thestatusofanorderwith:db/id2inthecurrentstateofthedatabaseissetasPending.Takealookatthefollowingexample:
(d/q'[:find?e?s
:where[?e:order/status?s]]
(d/dbconn))
#{[1"Done"][2"Pending"]}
Now,trytoupdatethevalueofthe:order/statusattributefororderID2toDonebytransactingwithits:db/id.Takealookatthefollowingexample:
;;updatethestatusattributeto'Done'fororderID'2'
(defstatus-result(d/transactconn[{:db/id2:order/status"Done"}]))
#'pedestal-play.server/status-result
;;querythelateststateofdatabase
(d/q'[:find?e?s:where[?e:order/status?s]](d/dbconn))
#{[2"Done"][1"Done"]}
Aftertransacting,thestatusoftheorderID2nowshowstheupdatedstatusinthecurrentstateofthedatabase.AlthoughthequeryshowsthatthestatusoforderID2isnowDone,DatomicdoesnotoverwritethevalueofthestatusfororderID2in-place.Instead,itaddsanewdatomwiththerecenttransactiontimestamp.Wheneverthequeryisexecutedwiththecurrentstateofthedatabase,thatis,usingthedbfunctionofdatomic.api,italwaysreturnsthefactswiththemostrecenttimestamp.
Toretrievethepreviousstatusoftheorder2,usethestateofthedatabasebeforethetransactionthatupdatedthestate.Thereturnvalueoftransactcontainsa:db-beforekeythatcanbeusedtorunthesamestatusqueryonthedatabasestatebeforethetransaction.Takealookatthefollowingexample:
;;querythestatusonpreviousstate
(d/q'[:find?e?s
:where[?e:order/status?s]]
(@status-result:db-before))
#{[1"Done"][2"Pending"]}
TheresultreturnsthepreviousstatusoforderID2,thatis,Pending.ImmutabilityisoneofthemostpowerfulfeaturesofaDatomicdatabaseandisveryusefulto
trackthechangesinthedatabase.
Deletingadatabase
Todeleteanexistingdatabase,usethedelete-databasefunctionofthedatomic.apinamespace.IttakesasinputthetargetdatabaseURIandreturnstrueifthedeletionsucceedsasshowninthefollowingexample:
pedestal-play.server>(d/delete-databasedburi)
true
DatomichasaDayofDatomic(http://www.datomic.com/training.html)seriesthatprovidesin-depthdetailsaboutDatomicdatabaseswithdetailedexamplesandtutorialstolearnfrom.
Summary
Inthischapter,welearnedaboutDatomicarchitectureandhowitisradicallydifferentfromtraditionaldatabases.Welearnedaboutitsdatamodelandhowitstoresdatoms.WealsolearnedhowtoretrievefactswithDatomicAPIsanditsDatalog-basedqueryengine.Wealsolookedatitsimmutabilityconstructsandhowtoquerydatabasesincurrentaswellashistoricalstates.
Inthenextpartofthisbook,wewillfocusontheimplementationofmicroservicesfortheHelpingHandsapplication,whichwillusePedestalasthebaseframeworktodesignAPIsandDatomicforpersistence.
BuildingMicroservicesforHelpingHands
"It'snottheideas;it'sdesign,implementationandhardworkthatmakethedifference."
-MichaelAbrash
Identifyingboundedcontextisthefirststeptowardsbuildingasuccessfulmicroservices-basedarchitecture.Designingforscaleandimplementingthemwiththerighttechnologystackisthenextandthemostcrucialstepinbuildingamicroservices-basedapplication.Thischapterbringstogetherallthedesigndecisionstakeninthefirstpartofthebook(Chapter2,MicroservicesArchitectureandChapter3,MicroservicesforHelpingHandsApplication)anddescribesthestepstoimplementthemusingthePedestalframework(Chapter6,IntroductiontoPedestal).Inthischapter,youwilllearnhowto:
ImplementHexagonaldesignformicroservicesCreatescalablemicroservicesforHelpingHandsusingPedestalImplementworkflowsformicroservicesusingthePedestalinterceptorchainImplementthelookupserviceofHelpingHandstosearchforservicesandgeneratereports
ImplementingHexagonalArchitectureHexagonalArchitecture(http://alistair.cockburn.us/Hexagonal+architecture),asshowninthefollowingdiagram,aimstodecouplethebusinesslogicfromthepersistenceandtheservicelayer.Clojureprovidestheconceptofaprotocol(https://clojure.org/reference/protocols)thatcanbeusedtodefinetheinterfaces,thatactasportsofHexagonalArchitecture.Theseportscanthenbeimplementedbytheadapters,resultinginadecoupledimplementationthatcanbeswappedbasedontherequirement.ExecutionoftheseadapterscanthenbetriggeredviaPedestalinterceptorsbasedonthebusinesslogic.
Designingtheinterceptorchainandcontext
TheinterceptorchainmustbedefinedforeachmicroserviceoftheHelpingHandsapplicationseparately.Eachinterceptorchainmayconsistofinterceptorsthatauthenticatetherequest,validatethedatamodels,andapplythebusinesslogic.InterceptorscanalsobeaddedtointeractwiththePersistencelayerandtogenerateeventsaswell.ThelistofprobableinterceptorsthatmaybeapartofHelpingHandsservicesinclude:
Auth:UsedtoauthenticateandauthorizerequestsreceivedbytheAPIendpointsexposedbythemicroservice.Validation(datamodel):Usedtovalidatetherequestparametersandmaptheexternaldatamodelwiththeinternaldatamodelasexpectedbythebusinesslogicandunderlyingpersistentstore.Businesslogic:Oneormoreinterceptorstoimplementthebusinesslogic.TheseinterceptorsprocesstherequestparametersreceivedbytheAPIendpoints.Persistence:PersistthechangesusingoneormoreadaptersthatimplementthePortProtocoldefinedforthemicroservicedatamodeltobepersisted.Events:Generateeventsasynchronouslybothforothermicroservicesaswellasformonitoringandreporting.Theseinterceptorsareaddedattheendofthechaintogeneratechangelogeventsofthepersistentstoreforothermicroservicestoconsume.
PedestalcontextisaClojuremapthatcontainsallthedetailsrelatedtotheinterceptorchain,requestparameters,headers,andmore.Thesamecontextmapcanalsobeusedtosharedatawithotherinterceptorsinthechain.Insteadofaddingakeytothecontextmapdirectly,itisrecommendedtokeepaparentkey,suchastx-data,thatcontainsamapofkeysthatarerelatedtothedatabeingprocessedbythemicroservice.Itmayalsocontainthevalidateduserdetailsforothermicroservicestoconsume.
CreatingaPedestalprojectTostartwiththeimplementation,createaprojectbythenameofhelping-handsandinitializeitwithaPedestaltemplateasdiscussedearlierinChapter6,IntroductiontoPedestal.Oncetheprojecttemplateisinitialized,updatetheproject.cljfileandotherconfigurationparametersasperthedevelopmentenvironmentsetupoftheplaygroundapplicationofChapter4,DevelopmentEnvironment.Onceconfigured,theproject.cljfileshouldcontainalltherequireddependenciesandplugins,asshownhere:
(defprojecthelping-hands"0.0.1-SNAPSHOT"
:description"HelpingHandsApplication"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
;;Removethislineanduncommentoneofthenextlinesto
;;useImmutantorTomcatinsteadofJetty:
[io.pedestal/pedestal.jetty"0.5.3"]
;;[io.pedestal/pedestal.immutant"0.5.3"]
;;[io.pedestal/pedestal.tomcat"0.5.3"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-
api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
:min-lein-version"2.0.0"
:source-paths["src/clj"]
:java-source-paths["src/jvm"]
:test-paths["test/clj""test/jvm"]
:resource-paths["config","resources"]
:plugins[[:lein-codox"0.10.3"]
;;CodeCoverage
[:lein-cloverage"1.0.9"]
;;Unittestdocs
[test2junit"1.2.2"]]
:codox{:namespaces:all}
:test2junit-output-dir"target/test-reports"
;;IfyouuseHTTP/2orALPN,usethejava-agenttopullinthecorrectalpn-boot
dependency
;:java-agents[[org.mortbay.jetty.alpn/jetty-alpn-agent"2.0.5"]]
:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]
[org.clojure/tools.nrepl"0.2.12"]]}
:dev{:aliases{"run-dev"["trampoline""run""-m""helping-
hands.server/run-dev"]}
:dependencies[[io.pedestal/pedestal.service-tools"0.5.3"]]
:resource-paths["config","resources"]
:jvm-opts["-Dconf=config/conf.edn"]}
:uberjar{:aot[helping-hands.server]}
:doc{:dependencies[[codox-theme-rdash"0.1.1"]]
:codox{:metadata{:doc/format:markdown}
:themes[:rdash]}}
:debug{:jvm-opts
["-server"(str"-agentlib:jdwp=transport=dt_socket,"
"server=y,address=8000,suspend=n")]}}
:main^{:skip-aottrue}helping-hands.server)
Theprojectdirectorystructureshouldcontaintherequiredfiles,asshownhere:
.
├──Capstanfile
├──config
│└──logback.xml
├──Dockerfile
├──project.clj
├──README.md
├──src
│├──clj
││└──helping_hands
││├──core.clj
││├──persistence.clj
││├──server.clj
││└──service.clj
│└──jvm
└──test
├──clj
│└──helping_hands
│├──core_test.clj
│└──service_test.clj
└──jvm
9directories,11files
Notethetwonewsourcefiles,core.cljandpersistence.clj,thathavebeencreatedmanuallyandaddedtotheprojectstructurealongwithacore_test.cljfilefortestcases.Thisprojectactsasatemplateforeachmicroservicethatfurtherextendstheservice.cljfilewiththeimplementationofroutesanddirectmessagingendpoints.Initializationoftheapplicationhappensincore.cljalongwiththeimplementationofinterceptorsandbusinesslogic.Thepersistencelayer,alongwithitsprotocol,isdefinedinpersistence.clj.ThenextstepistostartdefiningthegenericinterceptorsforHelpingHandsmicroservices.
DefininggenericinterceptorsInadditiontothebusinesslogic,eachmicroserviceneedstoauthenticaterequests,validateincominginputparameters,mapexternaldatamodelstointernaldatamodelsforbusinesslogic,andgenerateeventsforothermicroservicesbasedontheactiontaken.AllofthesecapabilitiescanalsobeimplementedasaPedestalinterceptorforbetterflexibilityandreducedmaintenanceoverheadduetotheseparationofconcernofeachPedestalinterceptor.
InterceptorforAuthMicroservicesthatallowuserstoregisterservices,lookupservices,andcreateordersmustintegratewiththeAuthservicetomakesurethattherequestsreceivedbytheservicearegenuineandthesenderisauthorizedtoperformtherequestedtask.InsteadofembeddingAuthlogicinallthemicroservices,itisrecommendedtoseparateitoutasamicroservicewithwhichallotherservicescaninteractviadirectmessagingtoauthenticatethesenderoftherequest.
TheinteractionwiththeAuthservicecanbeembeddedwithinaPedestalinterceptorthatfrontendstheinterceptorchainforallthesecuredAPIs.ThisinterceptorshouldcapturetheauthenticationandauthorizationdetailsintherequestandsendthemtotheAuthservicetovalidate.IftheAuthservicefailstovalidatetherequest,theinterceptorshouldterminatethechainandreturnaHTTP401Unauthorizedresponse,elseitshouldaddthesenderprofiledetailstotherequestandforwardittothenextinterceptorinthechain.
Forexample,authisaninterceptorthatlooksforatokenintherequestheader.Ifitispresent,itlooksuptheuserdetailsandpopulatesthemunderthe:userkeywordofthePedestalcontextmap.Ifatokenisnotpresentintheheader,itaddsaHTTP401Unauthorizedresponsewithamessageinthebodyandterminatesthechain:
(nshelping-hands.service
(:require[cheshire.core:asjp]
[io.pedestal.http:ashttp]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.interceptor.chain:aschain]
[ring.util.response:asring-resp]))
...
(defauth
{:name::auth
:enter
(fn[context]
(let[token(->context:request:headers(get"token"))]
(if-let[uid(and(not(nil?token))(get-uidtoken))]
(assoc-incontext[:request:tx-data:user]uid)
(chain/terminate
(assoccontext
:response{:status401
:body"Authtokennotfound"})))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
;;Tabularroutes
(defroutes#{["/":get(conjcommon-interceptors`auth`home-page)]})
Asofnow,forsimplicity,let'sassumethattheget-uidfunctionjustreturnsaresponsewithuidasafieldthathasafixedvalue,hhuser,thatcanbepickedbyinterceptors,suchashome-page,toconstructtheresponsefurtherdowntheinterceptorchain:
(nshelping-hands.service
(:require[cheshire.core:asjp]
[io.pedestal.http:ashttp]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.interceptor.chain:aschain]
[ring.util.response:asring-resp]))
...
(defnhome-page
[request]
(ring-resp/response
(if-let[uid(->request:tx-data:user(get"uid"))]
(jp/generate-string{:msg(str"Hello"uid"!")})
(jp/generate-string{:msg(str"HelloWorld!")}))))
(defn-get-uid
"TODO:IntegratewithAuthService"
[token]
(when(and(string?token)(not(empty?token)))
;;validatetoken
{"uid""hhuser"}))
Now,trytorequesttheroute/withandwithoutatoken.ItwillreturntheHTTP200OKresponsewiththemessagecontainingthefixeduserhhuser;whereas,withoutatoken,itwillreturnaHTTP401Unauthorizedresponse.Inthelattercase,executionofthehome-pageinterceptorisskippedentirelyincludingboththe:enterand:leavefunctions:
%curl-i-H"token:1234"http://localhost:8080
HTTP/1.1200OK
...
{"msg":"Hellohhuser!"}
%curl-ihttp://localhost:8080
HTTP/1.1401Unauthorized
...
Authtokennotfound
TheAuthserviceandrelatedinterceptorareexplainedindetailinPart-4ofthisbookunderChapter11,DeployingandMonitoringSecuredMicroservices.
InterceptorforthedatamodelOncetherequestisauthenticatedandauthorizedbytheAuthinterceptor,itshouldbevalidatedagainstthedatamodelofthemicroservice.ThisvalidationlogiccanalsobeimplementedasaPedestalinterceptorthatcanreviewtheinputparametersspecifiedwiththerequestandmakesurethatitconformstothedatamodel.Iftherequestparametersarevalid,thisinterceptorshouldforwardtherequesttothenextinterceptorinthechainwiththerequireddetails,elseitshouldterminatethechainandreturnaHTTP400BadRequestresponse.
Thedatamodelvalidationinterceptorcanalsoaddadditionaldetailstotherequest.Forexample,iftherequestcontainsjusttheserviceID,thisinterceptorcanpullintheservicedetailsandserviceproviderdetailsandaddittothelistoftheparametersfortherestofthechaintoprocess.ItcanalsovalidatethepresenceandabsenceofthespecifiedserviceID.
Forexample,tocreateanorder,arequestmustcontaintheserviceIDforwhichtheorderistobecreated.Thedatamodelinterceptor,inthiscase,canfirstvalidatethattherequestcontainstheserviceIDandifitdoes,itcanvalidatetheserviceIDwiththeexternalmicroservicethatmanagestheServicedatabaseandpullinadditionaldetails:(defn-get-service-details"TODO:GettheservicedetailsfromexternalAPI"[sid]{"sid"sid,"name""HouseCleaning"})
(defdata-validate{:name::validate
:enter(fn[context](let[sid(->context:request:form-params:sid)](if-let[service(and(not(nil?sid))(get-service-detailssid))](assoc-incontext[:request:tx-data:service]service)(chain/terminate(assoccontext
:response{:status400:body"InvalidServiceID"})))))
:error(fn[contextex-info](assoccontext:response{:status500:body(.getMessageex-info)}))})
;;Tabularroutes(defroutes#{["/":post(conjcommon-interceptors`auth`data-validate`home-page)]})
Now,thePOST/endpointusesbothauthanddata-validateinterceptorstomakesurethatbeforetherequesthitsthehome-pageinterceptor,ithasbeenauthenticatedandcontainstherequiredservicedetails.ThebehavioroftheendpointisshownhereusingcURLrequests:%curl-i-XPOSThttp://localhost:8080HTTP/1.1401Unauthorized...
Authtokennotfound
%curl-i-XPOST-H"token:1234"http://localhost:8080HTTP/1.1400BadRequest...
InvalidServiceID
%curl-i-XPOST-H"token:1234"-d"sid=1"http://localhost:8080HTTP/1.1200OK...
{"msg":"Hellohhuser!"}
ValidationinterceptorscoveredinthischapteruseClojurecoreconditionalfunctionstoachievethedesiredvalidation.WiththeClojure-1.9.0release,itisrecommendedtomovetoClojurespec(ht
tps://clojure.org/guides/spec)forallsuchvalidations.
InterceptorforeventsHelpingHandsmicroservices,suchasServiceProviderandServiceConsumer,generatechangelogeventsfortheLookupservicetopickupandupdatethelocaldatabaseforServiceConsumerstolookup.ThisrequirescertaineventstobegeneratedeverytimeachangeispushedtothelocaldatabaseoftheServiceProviderandServiceConsumerservice.TheseeventscanbedeterminedbyaninterceptorthatispresentinthechainrightbeforethePersistenceinterceptorthatpersiststhechangestothedatabase.Apartfromchange-logevents,eachmicroservicemayalsopublisheventsformonitoringandreporting.Alltheseeventscanbegeneratedbythesameinterceptor.
Theeventsinterceptorgeneratesalltheeventsrequiredbyothermicroservicesandtomonitortheentireapplication.Chapter10,EventDrivenPatternsforMicroservices,andChapter11,DeployingandMonitoringSecuredMicroservices,talkaboutusingexternalframeworkssuchasKafkatopublishandconsumeeventsforcoordinationandbuildinganevent-drivendatapipelineformicroservices.
CreatingamicroserviceforServiceConsumer
TheServiceConsumermicroserviceexposesAPIsforenduserstoregisterasaconsumeroftheHelpingHandsapplication.Tolookupregisteredservicesandplaceanorder,theuseroftheapplicationmustberegisteredasaConsumer.AspertheworkflowofServiceConsumerdefinedinChapter3,MicroservicesforHelpingHandsApplication,itrequiresthefollowingAPIstocreatenewconsumers,getconsumerprofiles,andupdateconsumerdetails:
URI Description
GET
/consumers/:id/?
flds=name,address
Getsthedetailsoftheconsumerwiththespecified:idifthe:idisspecified,elseitgetsthedetailsoftheauthenticatedconsumer.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.
PUT
/consumers/:id CreatesanewconsumerwiththespecifiedID.POST/consumers CreatesanewconsumerandreturnstheID.DELETE
/consumers/:id DeletestheconsumerwiththespecifiedID.
AddingroutesPedestalroutesareaddedforeachoftheidentifiedAPIs.TheinterceptorchainforeachAPIconsistsofAuth,Validation,BusinessLogic,andEventinterceptorsthatauthorizetheincomingrequests,validatetherequiredparameters,applythebusinesslogic,andgeneratetherelevantevents,respectively,forothermicroservicesandmonitoringframeworks.
Sincetheinterceptorchainofeachrouteendswithacommongen-eventsinterceptor,a:route-namemustbedefinedforeachroutetomakesurethattheyreceiveauniquename.If:route-nameisnotspecified,PedestalwilltrytoassignthenameofthelastinterceptortoeachrouteandwillfailwiththeRoutenamesarenotuniqueexception.Addtheroutesfortheconsumerserviceintheservice.cljfile,asshowninthefollowingcodesnippet:(nshelping-hands.consumer.service(:require[helping-hands.consumer.core:ascore][io.pedestal.http:ashttp][io.pedestal.http.route:asroute][io.pedestal.http.body-params:asbody-params]))
(defcommon-interceptors[(body-params/body-params)http/html-body])
;;Tabularroutes(defroutes#{["/consumers/:id":get(conjcommon-interceptors`auth`core/validate-id`core/get-consumer`gen-events):route-name:consumer-get]["/consumers/:id":put(conjcommon-interceptors`auth`core/validate-id`core/upsert-consumer`gen-events):route-name:consumer-put]["/consumers":post(conjcommon-interceptors`auth`core/validate`core/create-consumer`gen-events):route-name:consumer-post]
["/consumers/:id":delete(conjcommon-interceptors`auth`core/validate-id`core/delete-consumer`gen-events):route-name:consumer-delete]})
Chapter11,DeployingandMonitoringSecuredMicroservices,discussestheimportanceofeventsgeneratedbythemicroservicesforreal-timemonitoringandgeneratingalerts.
DefiningtheDatomicschema
TheServiceConsumermicroserviceusesDatomicasthelocaldatabasetostoretheconsumerdetails.Theschemafortheconsumerdatabaseconsistsofthefollowingattributes:
:db/ident :db/valueType :db/cardinality :db/index :db/fulltext
:consumer/id :db.type/string :db.cardinality/one true false
:consumer/name :db.type/string :db.cardinality/one true true
:consumer/address :db.type/string :db.cardinality/one true true
:consumer/mobile :db.type/string :db.cardinality/one false -
:consumer/email :db.type/string :db.cardinality/one true -
:consumer/geo :db.type/string :db.cardinality/one false -
CreatingapersistenceadapterPersistenceprotocolfortheConsumerserviceconsistsofupsert,entity,anddeletefunctionsthatareimplementedbyeachadapterofthisport.AdaptersimplementingthepersistenceprotocolarethenusedwithinthePedestalinterceptortocreate,update,query,anddeleteConsumerdetails.Theimplementationoftheprotocolandcorrespondingrecordisshowninthefollowingcodesnippetthatmustbeaddedtothepersistence.cljsourcefile:
(nshelping-hands.consumer.persistence
"PersistencePortandAdapterforConsumerService"
(:require[datomic.api:asd]))
;;--------------------------------------------------
;;ConsumerPersistencePortforAdapterstoPlug-in
;;--------------------------------------------------
(defprotocolConsumerDB
"Abstractionforconsumerdatabase"
(upsert[thisidnameaddressmobileemailgeo]
"Adds/Updatesaconsumerentity")
(entity[thisidflds]
"Getsthespecifiedconsumerwithallorrequestedfields")
(delete[thisid]
"Deletesthespecifiedconsumerentity"))
;;--------------------------------------------------
;;DatomicAdapterImplementationforConsumerPort
;;--------------------------------------------------
(defn-get-entity-id
[connid]
(->(d/q'[:find?e
:in$?id
:where[?e:consumer/id?id]](d/dbconn)(strid))
ffirst))
(defn-get-entity
[connid]
(let[eid(get-entity-idconnid)]
(->>(d/entity(d/dbconn)eid)seq(into{}))))
(defrecordConsumerDBDatomic[conn]
ConsumerDB
(upsert[thisidnameaddressmobileemailgeo]
(d/transactconn
(vector(into{}(filter(compsome?val)
{:db/idid
:consumer/idid
:consumer/namename
:consumer/addressaddress
:consumer/mobilemobile
:consumer/emailemail
:consumer/geogeo})))))
(entity[thisidflds]
(when-let[consumer(get-entityconnid)]
(if(empty?flds)
consumer
(select-keysconsumer(mapkeywordflds)))))
(delete[thisid]
(when-let[eid(get-entity-idconnid)]
(d/transactconn[[:db.fn/retractEntityeid]]))))
Forexample,theConsumerDBprotocoldefinestheportforthePersistencelayeroftheConsumerservicethatisimplementedbytheConsumerDBDatomicrecord(https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/defrecord)thatactsasanadaptertomanagetheconsumerdatabasewithintheDatomicdatabase.Thehelping-hands.consumer.persistencenamespacealsoprovidesacreate-consumer-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:
(defncreate-consumer-database
"Createsaconsumerdatabaseandreturnstheconnection"
[d]
;;createandconnecttothedatabase
(let[dburi(str"datomic:mem://"d)
db(d/create-databasedburi)
conn(d/connectdburi)]
;;transactschemaifdatabasewascreated
(whendb
(d/transactconn
[{:db/ident:consumer/id
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"UniqueConsumerID"
:db/unique:db.unique/identity
:db/indextrue}
{:db/ident:consumer/name
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"DisplayNamefortheConsumer"
:db/indextrue
:db/fulltexttrue}
{:db/ident:consumer/address
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"ConsumerAddress"
:db/indextrue
:db/fulltexttrue}
{:db/ident:consumer/mobile
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"ConsumerMobileNumber"
:db/indexfalse}
{:db/ident:consumer/email
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"ConsumerEmailAddress"
:db/indextrue}
{:db/ident:consumer/geo
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"Latitude,LongitudeCSV"
:db/indexfalse}]))
(ConsumerDBDatomic.conn)))
Thecreate-consumer-databasefunctionacceptsadatabasenameasinput,suchasconsumer,andcreatesaDatomicdatabaseURI.ThedatabaseURIprefixdatomic:mem://signifiesanin-memorydatabaseofDatomicthatisusedinthisexample.Thecreate-consumer-databasefunctiontriestocreateadatabaseandifitsucceeds,ittransactstheschemafortheconsumerdatabase.ItcreatesaconnectiontothedatabaseandreturnsitwrappedasaConsumerDBDatomicrecordthatcanthenbeusedtoupsertconsumers,retrievethem,anddeletethembasedontherequirement.
CreatinginterceptorsThehelping-hands.consumer.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheConsumermicroservice.Validationinterceptorsvalidatetheinputparametersandmakesurethatalltherequiredfieldsarepresentintherequest.Theyalsopreparetheinputparametersforthebusinesslogicinterceptorsbycreatinga:tx-datakeywithalltherequiredparameters.
Validationinterceptorsmayalsoperformtransformations,suchaschangingtheCSVofthefldsparametertoavectoroffieldnamesthatisrequiredasaninputfortheentityfunctiondefinedbyConsumerDBprotocol.TheyterminatetherequestwithaHTTP400BadRequeststatusiftherequiredparametersarenotpresent.TheyalsodefinetheerrorhandlertocatchtheexceptioninthechainandreportthemtotheclientwithaHTTP500InternalServerErrorstatus.Theimplementationofthevalidationinterceptorisshowninthefollowingcodesnippet:
(nshelping-hands.consumer.core
"InitializesHelpingHandsConsumerService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.consumer.persistence:asp]
[io.pedestal.interceptor.chain:aschain])
(:import[java.ioIOException]
[java.utilUUID]))
;;delaythecheckfordatabaseandconnection
;;tillthefirstrequesttoaccess@consumerdb
(def^:privateconsumerdb
(delay(p/create-consumer-database"consumer")))
;;--------------------------------
;;ValidationInterceptors
;;--------------------------------
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))]
(if(and(not(empty?params))
;;anyoneofmobile,emailoraddressispresent
(or(params:id)(params:mobile)(params:email)(params:address)))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body(str"OneofAddress,emailand"
"mobileismandatory")})))))
(defvalidate-id
{:name::validate-id
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidConsumerID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate
{:name::validate
:enter
(fn[context]
(if-let[params(->context:request:form-params)]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"Invalidparameters"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Thehelping-hands.consumer.corenamespacealsodefinestheinterceptorstogetconsumerdetails,performupsertoperations,createaconsumerwithageneratedID,anddeletetheconsumer,asshownhere:
;;--------------------------------
;;BusinessLogicInterceptors
;;--------------------------------
(defget-consumer
{:name::consumer-get
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
entity(.entity@consumerdb(:idtx-data)(:fldstx-data))]
(if(empty?entity)
(assoccontext:response{:status404:body"Nosuchconsumer"})
(assoccontext:response{:status200
:body(jp/generate-stringentity)}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defupsert-consumer
{:name::consumer-upsert
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
id(:idtx-data)
db(.upsert@consumerdbid(:nametx-data)
(:addresstx-data)(:mobiletx-data)
(:emailtx-data)(:geotx-data))]
(if(nil?@db)
(throw(IOException.
(str"Upsertfailedforconsumer:"id)))
(assoccontext
:response{:status200
:body(jp/generate-string
(.entity@consumerdbid[]))}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defcreate-consumer
{:name::consumer-create
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
;;generatearandomIDifitisnotspecified
id(str(UUID/randomUUID))
tx-data(if(:idtx-data)tx-data(assoctx-data:idid))
;;createconsumer
db(.upsert@consumerdbid(:nametx-data)
(:addresstx-data)(:mobiletx-data)
(:emailtx-data)(:geotx-data))]
(if(nil?@db)
(throw(IOException.
(str"Upsertfailedforconsumer:"id)))
(assoccontext
:response{:status200
:body(jp/generate-string
(.entity@consumerdbid[]))}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defdelete-consumer
{:name::consumer-delete
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
db(.delete@consumerdb(:idtx-data))]
(if(nil?db)
(assoccontext:response{:status404:body"Nosuchconsumer"})
(assoccontext:response{:status200:body"Success"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
TestingroutesTheroutesdefinedfortheConsumerserviceallowuserstocreateanewconsumer,queryforitsproperties,anddeleteit.Asofnow,fortheAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.Tostartwith,createaconsumerusingthePUT/consumers/:idroutewiththeIDsetas1,asshownhere:%curl-i-H"token:123"-XPUT-d"name=ConsumerA"http://localhost:8080/consumers/1HTTP/1.1200OK...
{"consumer/id":"1","consumer/name":"ConsumerA"}
Itcreatesanewconsumerandreturnstheconsumerentity.Toaddanewfield,usethesameAPIwiththesameconsumerIDandspecifythenewfields.Itwilldoanupsertoperationontheentityandaddthenewfields,asshownhere:
curl-i-H"token:123"-XPUT-d"email=user1@helpinghands.com"
http://localhost:8080/consumers/1
HTTP/1.1200OK
...
{"consumer/id":"1","consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}
Togettheentity,usetheGET/consumers/:idroute.IftheconsumerIDisnotfound,itreturnsaHTTP404NotFoundresponse:
%curl-i-H"token:123""http://localhost:8080/consumers/1"
HTTP/1.1200OK
...
{"consumer/id":"1","consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}
%curl-i-H"token:123""http://localhost:8080/consumers/1?
flds=consumer/name,consumer/email"
HTTP/1.1200OK
...
{"consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}
%curl-i-H"token:123""http://localhost:8080/consumers/2"
HTTP/1.1404NotFound
...
Nosuchconsumer
ThePOST/consumersroutecanalsobeusedtocreateaconsumerwitharandomID:
%curl-i-H"token:123"-XPOST-d"name=ConsumerX&email=userx@helpinghands.com"
http://localhost:8080/consumers
HTTP/1.1200OK
...
{"consumer/id":"b46cdbbb-06a1-4375-9287-
2230e3ad8ded","consumer/name":"ConsumerX","consumer/email":"userx@helpinghands.com"}
%curl-i-H"token:123""http://localhost:8080/consumers/b46cdbbb-06a1-4375-9287-
2230e3ad8ded"
HTTP/1.1200OK
...
{"consumer/id":"b46cdbbb-06a1-4375-9287-
2230e3ad8ded","consumer/name":"ConsumerX","consumer/email":"userx@helpinghands.com"}
Todeleteaconsumer,usetheDELETE/consumers/:idroutewithanexistingconsumerID:
%curl-i-H"token:123"-XDELETE"http://localhost:8080/consumers/b46cdbbb-06a1-
4375-9287-2230e3ad8ded"
HTTP/1.1200OK
...
Success
%curl-i-H"token:123""http://localhost:8080/consumers/b46cdbbb-06a1-4375-9287-
2230e3ad8ded"
HTTP/1.1404NotFound
...
Nosuchconsumer
CreatingamicroserviceforServiceProvider
TheServiceProvidermicroserviceexposesAPIsforenduserstoregisterasaserviceprovideroftheHelpingHandsapplication.ServiceproviderscanregisteroneormoreserviceswiththeHelpingHandsapplicationthattheyarewillingtofulfillifanorderisplacedagainstit.AspertheworkflowofServiceProviderdefinedinChapter3,MicroservicesforHelpingHandsApplication,thefollowingAPIsarerequiredtocreatenewproviders,getproviderprofiles,andupdateproviderdetails:
URI Description
GET
/providers/:id/?
flds=name,mobile
Getsthedetailsoftheserviceproviderwiththespecified:idifthe:idisspecified,elseitgetsthedetailsoftheauthenticateduserregisteredasaserviceprovider.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.
PUT/providers/:id CreatesanewproviderwiththespecifiedID.POST/providers CreatesanewproviderandreturnstheID.PUT
/providers/:id/rate Addstothelatestratingsfortheprovider.DELETE
/providers/:id DeletestheproviderwiththespecifiedID.
Addingroutes
RoutesfortheServiceProvidermicroserviceareverysimilartotheConsumerservice.TheProviderserviceadditionallyprovidesadedicatedroutetoregisteraratingfortheProvider,asshownhere:
(defroutes#{["/providers/:id"
:get(conjcommon-interceptors`auth`core/validate-id
`core/get-provider`gen-events)
:route-name:provider-get]
["/providers/:id"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-provider`gen-events)
:route-name:provider-put]
["/providers/:id/rate"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-provider`gen-events)
:route-name:provider-rate]
["/providers"
:post(conjcommon-interceptors`auth`core/validate
`core/create-provider`gen-events)
:route-name:provider-post]
["/providers/:id"
:delete(conjcommon-interceptors`auth`core/validate-id
`core/delete-provider`gen-events)
:route-name:provider-delete]})
DefiningDatomicschema
TheServiceProvidermicroserviceusesDatomicasthelocaldatabasetostoretheproviderdetails.Theschemafortheproviderdatabaseconsistsofthefollowingattributes:
:db/ident :db/valueType :db/cardinality :db/index :db/fulltext
:provider/id :db.type/string :db.cardinality/one true false
:provider/name :db.type/string :db.cardinality/one true true
:provider/mobile :db.type/string :db.cardinality/one false -
:provider/since :db.type/long :db.cardinality/one false -
:provider/rating :db.type/float :db.cardinality/many false -
Creatingapersistenceadapter
Persistenceprotocolconsistsofupsert,entity,anddeletefunctions,similartotheConsumerservice,thataredefinedinthepersistence.cljsourcefile,asfollows:
(nshelping-hands.provider.persistence
"PersistencePortandAdapterforProviderService"
(:require[datomic.api:asd]))
;;--------------------------------------------------
;;ProviderPersistencePortforAdapterstoPlug-in
;;--------------------------------------------------
(defprotocolProviderDB
"Abstractionforproviderdatabase"
(upsert[thisidnamemobilesincerating]
"Adds/Updatesaproviderentity")
(entity[thisidflds]
"Getsthespecifiedproviderwithallorrequestedfields")
(delete[thisid]
"Deletesthespecifiedproviderentity"))
;;--------------------------------------------------
;;DatomicAdapterImplementationforProviderPort
;;--------------------------------------------------
(defn-get-entity-id
[connid]
(->(d/q'[:find?e
:in$?id
:where[?e:provider/id?id]](d/dbconn)(strid))
ffirst))
(defn-get-entity
[connid]
(let[eid(get-entity-idconnid)]
(->>(d/entity(d/dbconn)eid)seq(into{}))))
(defrecordProviderDBDatomic[conn]
ProviderDB
(upsert[thisidnamemobilesincerating]
(d/transactconn
(vector(into{}(filter(compsome?val)
{:db/idid
:provider/idid
:provider/namename
:provider/mobilemobile
:provider/sincesince
:provider/ratingrating})))))
(entity[thisidflds]
(when-let[provider(get-entityconnid)]
(if(empty?flds)
provider
(select-keysprovider(mapkeywordflds)))))
(delete[thisid]
(when-let[eid(get-entity-idconnid)]
(d/transactconn[[:db.fn/retractEntityeid]]))))
Thehelping-hands.provider.persistencenamespacealsoprovidesacreate-provider-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime:
(defncreate-provider-database
"Createsaproviderdatabaseandreturnstheconnection"
[d]
;;createandconnecttothedatabase
(let[dburi(str"datomic:mem://"d)
db(d/create-databasedburi)
conn(d/connectdburi)]
;;transactschemaifdatabasewascreated
(whendb
(d/transactconn
[{:db/ident:provider/id
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"UniqueProviderID"
:db/unique:db.unique/identity
:db/indextrue}
{:db/ident:provider/name
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"DisplayNamefortheProvider"
:db/indextrue
:db/fulltexttrue}
{:db/ident:provider/mobile
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"ProviderMobileNumber"
:db/indexfalse}
{:db/ident:provider/since
:db/valueType:db.type/long
:db/cardinality:db.cardinality/one
:db/doc"ProviderActiveSinceEPOCHtime"
:db/indexfalse}
{:db/ident:provider/rating
:db/valueType:db.type/float
:db/cardinality:db.cardinality/many
:db/doc"Listofratings"
:db/indexfalse}]))
(ProviderDBDatomic.conn)))
Creatinginterceptors
Thehelping-hands.provider.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheProvidermicroservice.Validationinterceptorsvalidatetheinputparametersandmakesurethatalltherequiredfieldsarepresentintherequest.ValidationinterceptorsforProviderroutesworkexactlyinthesamewayasthatoftheConsumermicroserviceroutes.Additionally,theyvalidatethedatatypeoftheratingandsincefieldstomakesurethatthebusinessmodelgetsthevaluesintheformatexpectedbytheDatomicdatabase.Theimplementationofvalidationinterceptorsisshowninthefollowingcodesnippet:
(nshelping-hands.provider.core
"InitializesHelpingHandsProviderService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.provider.persistence:asp]
[io.pedestal.interceptor.chain:aschain])
(:import[java.ioIOException]
[java.utilUUID]))
;;delaythecheckfordatabaseandconnection
;;tillthefirstrequesttoaccess@providerdb
(def^:privateproviderdb
(delay(p/create-provider-database"provider")))
;;--------------------------------
;;ValidationInterceptors
;;--------------------------------
(defn-validate-rating-ts
"Validatestheratingandtimestamp"
[context]
(let[rating(->context:request:form-params:rating)
since_ts(->context:request:form-params:since)]
(try
(let[context(if(not(nil?rating))
(assoc-incontext[:request:form-params:rating]
(Float/parseFloatrating))context)
context(if(not(nil?since_ts))
(assoc-incontext[:request:form-params:since]
(Long/parseLongsince_ts))context)]
context)
(catchExceptionenil))))
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))
ctx(validate-rating-tscontext)
params(if(not(nil?ctx))
(assocparams
:rating(->ctx:request:form-params:rating)
:since(->ctx:request:form-params:since)))]
(if(and(not(empty?params))
(not(nil?ctx))
;;anyoneofidormobile
(or(params:id)(params:mobile)))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body(str"ID,mobileismandatory"
"andrating,sincemustbeanumber")})))))
(defvalidate-id
{:name::validate-id
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidProviderID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate
{:name::validate
:enter
(fn[context]
(if-let[params(->context:request:form-params)]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"Invalidparameters"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Otherinterceptors,suchasget-provider,upsert-provider,create-provider,anddelete-provider,aresimilartointerceptorsdefinedfortheroutesoftheConsumerservice,asshownhere:
;;--------------------------------
;;BusinessLogicInterceptors
;;--------------------------------
(defget-provider
{:name::provider-get
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
entity(.entity@providerdb(:idtx-data)(:fldstx-data))]
(if(empty?entity)
(assoccontext:response{:status404:body"Nosuchprovider"})
(assoccontext:response{:status200
:body(jp/generate-stringentity)}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defupsert-provider
{:name::provider-upsert
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
id(:idtx-data)
db(.upsert@providerdbid(:nametx-data)
(:mobiletx-data)(:sincetx-data)
(:ratingtx-data))]
(if(nil?@db)
(throw(IOException.
(str"Upsertfailedforprovider:"id)))
(assoccontext
:response{:status200
:body(jp/generate-string
(.entity@providerdbid[]))}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defcreate-provider
{:name::provider-create
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
;;generatearandomIDifitisnotspecified
id(str(UUID/randomUUID))
tx-data(if(:idtx-data)tx-data(assoctx-data:idid))
;;createprovider
db(.upsert@providerdbid(:nametx-data)
(:mobiletx-data)(:sincetx-data)
(:ratingtx-data))]
(if(nil?@db)
(throw(IOException.
(str"Upsertfailedforprovider:"id)))
(assoccontext
:response{:status200
:body(jp/generate-string
(.entity@providerdbid[]))}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defdelete-provider
{:name::provider-delete
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
db(.delete@providerdb(:idtx-data))]
(if(nil?db)
(assoccontext:response{:status404:body"Nosuchprovider"})
(assoccontext:response{:status200:body"Success"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
TestingroutesTheroutesdefinedfortheProviderserviceallowuserstocreateanewprovider,queryforitsproperties,anddeleteit.Asofnow,fortheAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.Tostartwith,createaproviderusingthePUT/providers/:idroutewithIDsetas1:%curl-i-H"token:123"-XPUT-d"name=ProviderA"http://localhost:8080/providers/1HTTP/1.1200OK...
{"provider/id":"1","provider/name":"ProviderA"}
Toaddarating,usethePUT/providers/:id/rateroutewiththesameIDasthatofProviderA:
%curl-i-H"token:123"-XPUT-d"rating=5.0"http://localhost:8080/providers/1/rate
HTTP/1.1200OK
...
{"provider/id":"1","provider/name":"ProviderA","provider/rating":[5.0]}
Togettheprovider,usetheGET/providers/:idroute.IftheproviderIDisnotfound,itreturnsaHTTP404NotFoundresponse:
%curl-i-H"token:123""http://localhost:8080/providers/1"
HTTP/1.1200OK
...
{"provider/id":"1","provider/name":"ProviderA","provider/rating":[5.0]}
%curl-i-H"token:123""http://localhost:8080/providers/2"
HTTP/1.1404NotFound
...
Nosuchprovider
Todeleteaprovider,usetheDELETE/providers/:idroutewithanexistingproviderID:
%curl-i-H"token:123"-XDELETE"http://localhost:8080/providers/1"
HTTP/1.1200OK
...
Success
%curl-i-H"token:123""http://localhost:8080/providers/1"
HTTP/1.1404NotFound
...
Nosuchprovider
CreatingamicroserviceforServices
TheServicemicroservicemanagesthelistofservicesofferedbytheserviceprovidersviatheHelpingHandsapplication.ItexposesAPIsforserviceproviderstoregisterservicesthatareofferedbythem.AspertheworkflowofService,definedinChapter3,MicroservicesforHelpingHandsApplication,thefollowingAPIsarerequiredtocreateanewservice,getservicedetails,andupdateservicedetails.EachservicemustalreadyhavetheserviceproviderregisteredwiththeHelpingHandsapplicationviatheServiceProvidermicroservice.Sinceonlyanexistingserviceprovidercanregisteraservice,theserviceproviderIDisretrievedbytheAuthtokenreceivedwiththerequestcallingtheServiceAPItocreateanewservice:
URI Description
GET
/services/:id/?
flds=name,mobile
Getsthedetailsoftheservicewiththespecified:idifthe:idisspecified.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.
PUT/services/:id CreatesanewservicewiththespecifiedID.POST/services CreatesanewserviceandreturnstheID.PUT
/services/:id/rate Addstothelatestratingsfortheservice.DELETE
/services/:id DeletestheservicewiththespecifiedID.
Addingroutes
RoutesforServiceareverysimilartotheProviderservice.Italsodefinesroutestocreate,modify,rate,anddeleteservices.TheroutetocreateaserviceexpectsaproviderIDasamandatoryparametertomakesurethateachserviceisassociatedwithaprovideratthetimeofcreation.Also,anychangeinproviderIDusingthePUT/services/:idrouteisvalidatedagainsttheProviderservicetomakesurethatthespecifiedproviderexistsandisregisteredwiththeHelpingHandsapplication.Theroutesareshowninthefollowingcodesnippet:
;;Tabularroutes
(defroutes#{["/services/:id"
:get(conjcommon-interceptors`auth`core/validate-id-get
`core/get-service`gen-events)
:route-name:service-get]
["/services/:id"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-service`gen-events)
:route-name:service-put]
["/services/:id/rate"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-service`gen-events)
:route-name:service-rate]
["/services"
:post(conjcommon-interceptors`auth`core/validate
`core/create-service`gen-events)
:route-name:service-post]
["/services/:id"
:delete(conjcommon-interceptors`auth`core/validate-id-get
`core/delete-service`gen-events)
:route-name:service-delete]})
DefiningaDatomicschemaTheServicemicroserviceusesDatomicasthelocaldatabasetostoretheservicedetails.ServicemaintainsaproviderIDaswell.Althoughintheschemaitisdefinedastype:db.type/string;ifacommondatabaseisusedfortheproviderandservicethenitisrecommendedtodefinetheproviderIDoftype:db.type/reftomakebetteruseofDatomicentityreferences.Theschemafortheservicedatabaseconsistsofthefollowingattributes:
:db/ident :db/valueType :db/cardinality :db/index :db/fulltext
:service/id :db.type/string :db.cardinality/one true false
:service/type :db.type/string :db.cardinality/one true true
:service/provider :db.type/string :db.cardinality/one false -
:service/area :db.type/string :db.cardinality/many true true
:service/cost :db.type/float :db.cardinality/one false -
:service/rating :db.type/float :db.cardinality/many false -
:service/status :db.type/string :db.cardinality/one false -
ThereisalsoaGeoLocationfieldmentionedinthedatamodeloftheServicedatabaseinChapter3,MicroservicesforHelpingHandsApplication.ThisfieldisaderivedfieldthatiscomputedandstoredbytheLookupserviceinsteadofbeingstoredintheServicedatabase.Thisfieldisusedonlyforgeolocation-basedqueriesthattheLookupservicewillbehandling.
Creatingapersistenceadapter
PersistenceprotocolServiceDBconsistsofupsert,entity,anddeletefunctions,similartotheProviderservice,thataredefinedinthepersistence.cljsourcefile,asshownhere:
(nshelping-hands.service.persistence
"PersistencePortandAdapterforService"
(:require[datomic.api:asd]))
;;--------------------------------------------------
;;ServicePersistencePortforAdapterstoPlug-in
;;--------------------------------------------------
(defprotocolServiceDB
"Abstractionforservicedatabase"
(upsert[thisidtypeproviderareacostratingstatus]
"Adds/Updatesaserviceentity")
(entity[thisidflds]
"Getsthespecifiedservicewithallorrequestedfields")
(delete[thisid]
"Deletesthespecifiedserviceentity"))
;;--------------------------------------------------
;;DatomicAdapterImplementationforServicePort
;;--------------------------------------------------
(defn-get-entity-id
[connid]
(->(d/q'[:find?e
:in$?id
:where[?e:service/id?id]](d/dbconn)(strid))
ffirst))
(defn-get-entity
[connid]
(let[eid(get-entity-idconnid)]
(->>(d/entity(d/dbconn)eid)seq(into{}))))
(defrecordServiceDBDatomic[conn]
ServiceDB
(upsert[thisidtypeproviderareacostratingstatus]
(d/transactconn
(vector(into{}(filter(compsome?val)
{:db/idid
:service/idid
:service/typetype
:service/providerprovider
:service/areaarea
:service/costcost
:service/ratingrating
:service/statusstatus})))))
(entity[thisidflds]
(when-let[service(get-entityconnid)]
(if(empty?flds)
service
(select-keysservice(mapkeywordflds)))))
(delete[thisid]
(when-let[eid(get-entity-idconnid)]
(d/transactconn[[:db.fn/retractEntityeid]]))))
Thehelping-hands.service.persistencenamespacealsoprovidesacreate-service-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:
(defncreate-service-database
"Createsaservicedatabaseandreturnstheconnection"
[d]
;;createandconnecttothedatabase
(let[dburi(str"datomic:mem://"d)
db(d/create-databasedburi)
conn(d/connectdburi)]
;;transactschemaifdatabasewascreated
(whendb
(d/transactconn
[{:db/ident:service/id
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"UniqueServiceID"
:db/unique:db.unique/identity
:db/indextrue}
{:db/ident:service/type
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"TypeofService"
:db/indextrue
:db/fulltexttrue}
{:db/ident:service/provider
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"AssociatedServiceProviderID"
:db/indexfalse}
{:db/ident:service/area
:db/valueType:db.type/string
:db/cardinality:db.cardinality/many
:db/doc"ServiceAreas/Locality"
:db/indextrue
:db/fulltexttrue}
{:db/ident:service/cost
:db/valueType:db.type/float
:db/cardinality:db.cardinality/one
:db/doc"HourlyCost"
:db/indexfalse}
{:db/ident:service/rating
:db/valueType:db.type/float
:db/cardinality:db.cardinality/many
:db/doc"Listofratings"
:db/indexfalse}
{:db/ident:service/status
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"StatusofService(A/NA/D)"
:db/indexfalse}]))
(ServiceDBDatomic.conn)))
CreatinginterceptorsThehelping-hands.service.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheServicemicroservice.SincebusinessmodelinterceptorsofServiceareexactlythesameasthoseoftheProviderserviceanddependonlyontheValidationinterceptors,let'sfocusonlyontheinterceptorsthatvalidatetheinputparametersforServiceroutes.Theimplementationofthevalidationinterceptorisshowninthefollowingcodesnippet:
(nshelping-hands.service.core
"InitializesHelpingHandsServiceService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.service.persistence:asp]
[io.pedestal.interceptor.chain:aschain])
(:import[java.ioIOException]
[java.utilUUID]))
;;delaythecheckfordatabaseandconnection
;;tillthefirstrequesttoaccess@servicedb
(def^:privateservicedb
(delay(p/create-service-database"service")))
;;--------------------------------
;;ValidationInterceptors
;;--------------------------------
(defn-validate-rating-cost
"Validatestheratingandcost"
[context]
(let[rating(->context:request:form-params:rating)
cost(->context:request:form-params:cost)]
(try
(let[context(if(not(nil?rating))
(assoc-incontext[:request:form-params:rating]
(Float/parseFloatrating))context)
context(if(not(nil?cost))
(assoc-incontext[:request:form-params:cost]
(Float/parseFloatcost))context)]
context)
(catchExceptionenil))))
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))
ctx(validate-rating-costcontext)
params(if(not(nil?ctx))
(assocparams
:rating(->ctx:request:form-params:rating)
:cost(->ctx:request:form-params:cost)))]
(if(and(not(empty?params))
(not(nil?ctx))
(params:id)(params:type)(params:provider)
(params:area)(params:cost)
(contains?#{"A""NA""D"}(params:type))
(provider-exists?(params:provider)))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body(str"ID,type,provider,areaandcostismandatory"
"andrating,costmustbeanumberwithtype"
"havingoneofvaluesA,NAorD")})))))
(defvalidate-id
{:name::validate-id
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidServiceID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate-id-get
{:name::validate-id-get
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))]
(if(and(not(empty?params))
(params:id))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidServiceID"}))))
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidServiceID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate
{:name::validate
:enter
(fn[context]
(if-let[params(->context:request:form-params)]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"Invalidparameters"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
ForService,thevalidationrulealsoincludesvalidatingagivenproviderIDagainstanexternalProviderServicetomakesurethattheprovideroftheserviceisalreadyregistered.Todoso,thevalidationinterceptorofServicemakesanexternalAPIcallasynchronouslyviatheprovider-exists?functionandpassesPedestalcontexttobusinesslogicinterceptorsonlywhenitfindsthattheproviderIDisvalid.
clj-http(https://github.com/dakrone/clj-http)isaClojurelibrarythatiswidelyusedtocreateHTTPclientstomakeAPIcallstoexternalservices.TomakeaGETcalltoanexternalservicelikethatofGET/providers/:id,seeclj-httpGET(https://github.com/dakrone/clj-http#get).
Testingroutes
TheroutesdefinedforServiceallowuserstocreateanewservice,queryforitsproperties,anddeleteit.Asofnow,forAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.HereareasampleofcURLrequeststocreate,query,anddeleteaserviceofferedbytheHelpingHandsapplication:
;;Createanewservicewithrequiredparameters
%curl-i-H"token:123"-XPUT-d"type=A&provider=1&area=bangalore&cost=250"
http://localhost:8080/services/1
HTTP/1.1200OK
...
{"service/id":"1","service/type":"A","service/provider":"1","service/area":
["bangalore"],"service/cost":250.0}
;;GetservicepropertiesbyID
%curl-i-H"token:123"http://localhost:8080/services/1
HTTP/1.1200OK
...
{"service/id":"1","service/type":"A","service/provider":"1","service/area":
["bangalore"],"service/cost":250.0}
;;Deletetheservice
%curl-i-H"token:123"-XDELETEhttp://localhost:8080/services/1
HTTP/1.1200OK
...
Success
;;Validateservicenolongerexists
%curl-i-H"token:123"http://localhost:8080/services/1
HTTP/1.1404NotFound
...
Nosuchservice
CreatingamicroserviceforOrder
TheOrderServicereceivestherequestfromServiceConsumerstocreateaneworderfortheserviceprovidedbyaparticularserviceprovider.ItexposesthefollowingAPIsforconsumerstocreateaneworder,getorderdetails,andgetthelistofordersplacedbythem.ItalsoallowstheconsumerstoratetheOrderbasedonthequalityofservicereceived.TocreateanewOrder,APIsexpectaserviceIDandproviderIDtobespecifiedalongwiththerequireddetailssuchastimeslot,andmore.TheconsumerIDispickedfromtheAuthtokenthatisreceivedasapartofrequestheaders.TheIDsspecifiedforServiceandServiceProvidermustalreadyberegisteredwiththeHelpingHandsapplicationviatheServiceandServiceProvidermicroservices.CreationofanOrderalsomakessurethattherequestedserviceisofferedwithinthevicinityoftheconsumerbasedonthegeolocationoftheconsumerandservicebeingrequested:
URI DescriptionGET/orders/?
flds=cost,status Getsalltheordersplacedbytheauthenticatedconsumer.
GET
/orders/:id/?
flds=name,mobile
Getsthedetailsoftheorderwiththespecified:idandplacedbytheauthenticatedconsumer.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.
PUT/orders/:idCreatesaneworderwiththespecifiedIDfortheauthenticatedconsumer.
POST/ordersCreatesaneworderfortheauthenticatedconsumerandreturnstheID.
PUT
/orders/:id/rate
Addstothelatestratingsfortheorder.ConsumerIDoftheordermustmatchtheauthenticatedconsumerID.
DELETE
/orders/:id
DeletestheorderwiththespecifiedIDfortheauthenticatedconsumer.ConsumerIDoftheordermustmatchtheauthenticatedconsumerID.
Addingroutes
TheOrderServiceallowsconsumerstocreateneworders,modifythem,rate,anddeletethem.ItalsoallowsanauthenticatedusertogetalltheordersplacedbytheauthenticateduserID.Theroutesrequiredforthisserviceareshowninthefollowingcodesnippet:
;;Tabularroutes
(defroutes#{["/orders/:id"
:get(conjcommon-interceptors`auth`core/validate-id-get
`core/get-order`gen-events)
:route-name:order-get]
["/orders"
:get(conjcommon-interceptors`auth`core/validate-all-orders
`core/get-all-orders`gen-events)
:route-name:order-get-all]
["/orders/:id"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-order`gen-events)
:route-name:order-put]
["/orders/:id/rate"
:put(conjcommon-interceptors`auth`core/validate-id
`core/upsert-order`gen-events)
:route-name:order-rate]
["/orders"
:post(conjcommon-interceptors`auth`core/validate
`core/create-order`gen-events)
:route-name:order-post]
["/orders/:id"
:delete(conjcommon-interceptors`auth`core/validate-id-get
`core/delete-order`gen-events)
:route-name:order-delete]})
DefiningDatomicschema
TheOrdermicroserviceusesDatomicasthelocaldatabasetostoretheorderdetails.Theschemafortheorderdatabaseconsistsofthefollowingattributes:
:db/ident :db/valueType :db/cardinality :db/index :db/fulltext :db/unique
:order/id :db.type/string :db.cardinality/one true false :db.unique/identity
:order/service :db.type/string :db.cardinality/one false - -
:order/provider :db.type/string :db.cardinality/one false - -
:order/consumer :db.type/string :db.cardinality/one false - -
:order/cost :db.type/float :db.cardinality/one false - -
:order/start :db.type/long :db.cardinality/one false - -
:order/end :db.type/long :db.cardinality/one false - -
:order/rating :db.type/float :db.cardinality/many false - -
:order/status :db.type/string :db.cardinality/one false - -
Creatingapersistenceadapter
PersistenceprotocolOrderDBconsistsofupsert,entity,anddeletefunctions,similartotheProviderservice.Additionally,itdefinesanordersfunctionthatcanlistalltheordersofthegivenauthenticatedconsumerID,asshownhere:
(nshelping-hands.order.persistence
"PersistencePortandAdapterforOrder"
(:require[datomic.api:asd]))
;;--------------------------------------------------
;;OrderPersistencePortforAdapterstoPlug-in
;;--------------------------------------------------
(defprotocolOrderDB
"Abstractionfororderdatabase"
(upsert[thisidserviceproviderconsumer
coststartendratingstatus]
"Adds/Updatesanorderentity")
(entity[thisidflds]
"Getsthespecifiedorderwithallorrequestedfields")
(orders[thisuidflds]
"Getsalltheordersoftheauthenticateduserwithallorrequestedfields")
(delete[thisid]
"Deletesthespecifiedorderentity"))
;;--------------------------------------------------
;;DatomicAdapterImplementationforOrderPort
;;--------------------------------------------------
(defn-get-entity-id
[connid]
(->(d/q'[:find?e
:in$?id
:where[?e:order/id?id]](d/dbconn)(strid))
ffirst))
(defn-get-entity
[connid]
(let[eid(get-entity-idconnid)]
(->>(d/entity(d/dbconn)eid)seq(into{}))))
(defn-get-entity-uid
[connuid]
(->>(d/q'[:find?e
:in$?id
:where[?e:order/consumer?id]](d/dbconn)(struid))
(into[])flatten))
(defn-get-all-entities
[connuid]
(let[eids(get-entity-uidconnuid)]
(map#(->>(d/entity(d/dbconn)%)seq(into{}))eids)))
(defrecordOrderDBDatomic[conn]
OrderDB
(upsert[thisidserviceproviderconsumer
coststartendratingstatus]
(d/transactconn
(vector(into{}(filter(compsome?val)
{:db/idid
:order/idid
:order/serviceservice
:order/providerprovider
:order/consumerconsumer
:order/costcost
:order/startstart
:order/endend
:order/ratingrating
:order/statusstatus})))))
(entity[thisidflds]
(when-let[order(get-entityconnid)]
(if(empty?flds)
order
(select-keysorder(mapkeywordflds)))))
(orders[thisuidflds]
(when-let[orders(get-all-entitiesconnuid)]
(if(empty?flds)
orders
(map#(select-keys%(mapkeywordflds))orders))))
(delete[thisid]
(when-let[eid(get-entity-idconnid)]
(d/transactconn[[:db.fn/retractEntityeid]]))))
Theget-all-entitiesfunctionqueriestheDatomicdatabaseforalltheordersthathave:order/consumersettothegivenconsumerIDthatisprovidedasaparametertotheendpointthatrequestsalltheordersfortheconsumer.Italsoallowstopickonlythespecifiedfieldsacrosstheorders.Thehelping-hands.order.persistencenamespacealsoprovidesacreate-order-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:
(defncreate-order-database
"Createsaorderdatabaseandreturnstheconnection"
[d]
;;createandconnecttothedatabase
(let[dburi(str"datomic:mem://"d)
db(d/create-databasedburi)
conn(d/connectdburi)]
;;transactschemaifdatabasewascreated
(whendb
(d/transactconn
[{:db/ident:order/id
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"UniqueOrderID"
:db/unique:db.unique/identity
:db/indextrue}
{:db/ident:order/service
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"AssociatedServiceID"
:db/indexfalse}
{:db/ident:order/provider
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"AssociatedServiceProviderID"
:db/indexfalse}
{:db/ident:order/consumer
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"AssociatedConsumerID"}
{:db/ident:order/cost
:db/valueType:db.type/float
:db/cardinality:db.cardinality/one
:db/doc"HourlyCost"
:db/indexfalse}
{:db/ident:order/start
:db/valueType:db.type/long
:db/cardinality:db.cardinality/one
:db/doc"StartTime(EPOCH)"
:db/indexfalse}
{:db/ident:order/end
:db/valueType:db.type/long
:db/cardinality:db.cardinality/one
:db/doc"EndTime(EPOCH)"
:db/indexfalse}
{:db/ident:order/rating
:db/valueType:db.type/float
:db/cardinality:db.cardinality/many
:db/doc"Listofratings"
:db/indexfalse}
{:db/ident:order/status
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"StatusofOrder(O/I/D/C)"
:db/indexfalse}]))
(OrderDBDatomic.conn)))
CreatinginterceptorsThehelping-hands.order.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheOrdermicroservice.TheAuthinterceptoristhegenericinterceptorthatreadsthetokenandupdatestheuserIDfield:uidfortheOrderroutestogetalltheordersforanauthenticateduser.Forsimplicityoftheimplementation,theAuthinterceptorassumesthatthetokenpassedintheheaderissettotheconsumerID.
ThevalidationinterceptorforOrderroutesvalidatesbothserviceIDandtheproviderIDoftheordertomakesurethatbothproviderandserviceareregisteredwiththeHelpingHandsapplicationandthesameproviderprovidesthespecifiedservice.Theservice-exists?,provider-exists?,andconsumer-exists?functionsvalidatetheservice,provider,andconsumer,respectively.Additionally,thevalidationinterceptorchecksfortherightvalueofstatusandthevalueofrating,cost,start,andendtobeoftypenumber,asshowninthefollowingcodesnippet.TheimplementationofthesefunctionsaresameasthatoftheConsumer,Order,andServicemicroservicesimplementationsexplainedearlier:
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))
ctx(validate-rating-cost-tscontext)
params(if(not(nil?ctx))
(assocparams
:rating(->ctx:request:form-params:rating)
:cost(->ctx:request:form-params:cost)
:start(->ctx:request:form-params:start)
:end(->ctx:request:form-params:end)))]
(if(and(not(empty?params))
(not(nil?ctx))
(params:id)(params:service)(params:provider)
(params:consumer)(params:cost)(params:status)
(contains?#{"O""I""D""C"}(params:status))
(service-exists?(params:service))
(provider-exists?(params:provider))
(consumer-exists?(params:consumer)))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body(str"ID,service,provider,consumer,"
"costandstatusismandatory.start/end,"
"ratingandcostmustbeanumberwithstatus"
"havingoneofvaluesO,I,DorC")})))))
(defvalidate-id
{:name::validate-id
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidOrderID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate-id-get
{:name::validate-id-get
:enter
(fn[context]
(if-let[id(or(->context:request:form-params:id)
(->context:request:query-params:id)
(->context:request:path-params:id))]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:path-params))]
(if(and(not(empty?params))
(params:id))
(let[flds(if-let[fl(:fldsparams)]
(maps/trim(s/splitfl#","))
(vector))
params(assocparams:fldsflds)]
(assoccontext:tx-dataparams))
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidOrderID"}))))
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidOrderID"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
(defvalidate-all-orders
{:name::validate-all-orders
:enter
(fn[context]
(if-let[params(->context:tx-data)]
;;GetuserIDfromauthuid
(assoc-incontext[:tx-data:flds]
(if-let[fl(->context:request:query-params:flds)]
(maps/trim(s/splitfl#","))
(vector)))
(chain/terminate
(assoccontext
:response{:status400
:body"Invalidparameters"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Thevalidate-id-getinterceptorisusedfortheGET/orders/:idrequestsandvalidatesonlytheorderIDandtheorderfieldsparameter.Similarly,thevalidate-all-ordersinterceptorisusedwiththeGET/ordersroutetogetalltheordersoftheauthenticatedconsumer.TheimplementationofinterceptorsforthebusinesslogicoftheOrderserviceissimilartothatofthepreviousimplementationofConsumer,Provider,andService.Additionally,theOrderservicedefinesinterceptorstogetallordersoftheauthenticatedconsumerthatusetheordersfunctionoftheOrderDBprotocol,asshownhere:
(nshelping-hands.order.core
"InitializesHelpingHandsOrderService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.order.persistence:asp]
[io.pedestal.interceptor.chain:aschain])
(:import[java.ioIOException]
[java.utilUUID]))
;;delaythecheckfordatabaseandconnection
;;tillthefirstrequesttoaccess@orderdb
(def^:privateorderdb
(delay(p/create-order-database"order")))
(defget-all-orders
{:name::order-get-all
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
entity(.orders@orderdb(:uidtx-data)(:fldstx-data))]
(if(empty?entity)
(assoccontext:response{:status404:body"Nosuchorders"})
(assoccontext:response{:status200
:body(jp/generate-stringentity)}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Testingroutes
TotesttheroutesoftheOrderservice,createoneormoreordersbythesameconsumerIDandtrytoqueryalltheordersforthesameconsumerID.Forsimplicity,theroutesassumethevalueofthetokenspecifiedintheheaderastheconsumerIDfortheGET/ordersroute.HereisalistofcURLrequeststodemonstratetheprocessofcreating,querying,anddeletingtheorders:
;;AddanorderforConsumerwithID1
%curl-i-H"token:1"-XPUT-d"service=1&provider=1&consumer=1&cost=500&status=O"
http://localhost:8080/orders/1
HTTP/1.1200OK
...
{"order/id":"1","order/service":"1","order/provider":"1","order/consumer":"1","order/cost":500.0,"order/status":"O"}
;;AddanotherorderforConsumerwithID1
%curl-i-H"token:1"-XPUT-d"service=2&provider=2&consumer=1&cost=250&status=O"
http://localhost:8080/orders/2
HTTP/1.1200OK
...
{"order/id":"2","order/service":"2","order/provider":"2","order/consumer":"1","order/cost":250.0,"order/status":"O"}
;;AddanorderforConsumerwithID2
%curl-i-H"token:2"-XPUT-d"service=1&provider=1&consumer=2&cost=250&status=I"
http://localhost:8080/orders/3
HTTP/1.1200OK
...
{"order/id":"3","order/service":"1","order/provider":"1","order/consumer":"2","order/cost":250.0,"order/status":"I"}
;;GetallordersofconsumerwithID1
%curl-i-H"token:1"http://localhost:8080/orders
HTTP/1.1200OK
...
[{"order/id":"1","order/service":"1","order/provider":"1","order/consumer":"1","order/cost":500.0,"order/status":"O"},
{"order/id":"2","order/service":"2","order/provider":"2","order/consumer":"1","order/cost":250.0,"order/status":"O"}]
;;GetallordersofconsumerwithID2
%curl-i-H"token:2"http://localhost:8080/orders
HTTP/1.1200OK
...
[{"order/id":"3","order/service":"1","order/provider":"1","order/consumer":"2","order/cost":250.0,"order/status":"I"}]
;;GetallordersofconsumerwithID1withspecificfields
%curl-i-H"token:1""http://localhost:8080/orders?flds=order/service,order/status"
...
[{"order/service":"1","order/status":"O"},{"order/service":"2","order/status":"O"}]
;;DeleteorderwithID2
%curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/2
HTTP/1.1200OK
...
Success
;;MakesurethatOrderwithID2nolongerexists
curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/2
HTTP/1.1404NotFound
...
Nosuchorder
;;CheckordersforconsumerwithID1doesnotlistOrderwithID2now
curl-i-H"token:1""http://localhost:8080/orders?flds=order/service,order/status"
HTTP/1.1200OK
...
[{"order/service":"1","order/status":"O"}]%
;;DeletetheorderwithID3thatwastheonlyorderforconsumerwithID2
%curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/3
HTTP/1.1200OK
...
Success
;;MakesuretherearenootherordersleftforconsumerwithID2
%curl-i-H"token:2"http://localhost:8080/orders
HTTP/1.1404NotFound
...
Nosuchorders
CreatingamicroserviceforLookupTheLookupserviceisusedtosearchforservicesbytype,geolocation,andavailability.ItsubscribestotheeventsgeneratedbytheConsumer,Provider,Service,andOrdermicroservicesasanObserverandkeepsadenormalizeddatasetthatisfastertoqueryforrequiredServices.TheLookupservicealsoaddslongitudeandlatitudefortheServicesandConsumersthatisderivedfromtheiraddressandserviceareaorlocality.Togetthelongitudeandlatitudefromtheservicearea,itdependsonanexternalAPI.TheLookupserviceprovidesthefollowingAPIsforconsumerstosearchforaserviceandalsofiltertheservicesbytype,ratings,andproviders:
URI Description
GET/lookup/?q=queryFiltersalltheservicesbasedonthespecifiedquery.
GET/lookup/?q=query&type=typeFiltersalltheservicesofagiventypebasedonthespecifiedquery.
GET/lookup/geo/?
tl=40.73,-74.1&br=40.01,-71.12
Looksuptheservicebythegivenlatitude-longitudesetfortop-left(tl)andbottom-right(br)boundingboxpointsorwithinaradiusofapre-defineddistancefromtheconsumerlocation.
GET/validate/:service/?
tl=40.73,-74.1&br=40.01,-71.12
Validatesifthespecified:serviceIDiswithintheboundingboxregionspecifiedbythetop-left(tl)andbottomright(br)boundingboxpointsorwithinaradiusofapre-defineddistancefromtheconsumerlocation.
GET/status
Getsthecurrentstatusoftheeventsincludingthenumberofordersplacedovertime,trendingordertypes,trendinglocation,toppreferredserviceproviders,andmore.
GET/status/consumer/:idGetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedconsumerID.
GET/status/provider/:id GetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedproviderID.
GET/status/service/:typeGetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedtypeofservices.
TheLookupservicedoesnotprovideanyAPIstoupdatethedataasitmaintainsonlyadenormalizedviewofeventsreceivedfromConsumer,Provider,Service,andOrderservices.Ifthereareanychangesrequiredinthedata,theymustbedonewiththeAPIsexposedbytheircorrespondingmicroservicesthatmanageit.TheLookupservicewillthenreceivethechangeeventsandreflectthechangesinitslocaldatabase.
DefiningtheElasticsearchindex
TheLookupmicroserviceusesElasticsearchasthelocaldatabasetostorealltheeventsthataredenormalizedacrossdatabasesmaintainedbyConsumer,Provider,ServiceandOrdermicroservices.Elasticsearchprovidessub-secondresponseforthesearchqueriesandalsosupportsgeolocation-basedqueries.ItalsosupportsaggregationandanalyticsoutoftheboxtogeneratevariousreportsfortheHelpingHandsapplication.HereisamappingfortheElasticsearchindexthatisrequiredfortheLookupservice:
Field Type Mapping Analyzer Descriptionoid string - keyword OrderIDcid string - keyword UsedforconsumerID
pid string - keywordUsedforserviceproviderID
sid string - keyword UsedforserviceIDstype string - keyword Usedforservicetype
locality string - standardUsedtostorethelocalityoftheorder
geo string(lat,long) geo_point - Usedforgeolocation-basedqueries
cost float - - Usedtostorethetotalcostoftheorder
ts_startdate
(yyyyMMDD'T'HH:mm:ss) - - Startdate-timeoftheorder
ts_enddate
(yyyyMMDD'T'HH:mm:ss) - - Enddate-timeoftheorder
rating float
Ratinggivenforthe
- - order
status string - keywordStatusoftheorder,oneofO,I,D,C
TheeventsarestoredintoElasticsearchwiththeprecedingindexschemainLookupindex.TheeventsarereceivedbytheLookupServiceviatheKafkatopicthatitsubscribesto.ThedetailsofKafka,theconceptoftopicsandhowtosubscribetoitforevents,havebeenexplainedinChapter10,Event-DrivenPatternsforMicroservices.Inthischapter,thefocuswillbeonhowtoquerytheElasticsearchindexthathasthefieldsaspertheschemaoftheLookupindex.
CreatingqueryinterceptorsAlltheroutesoftheLookupserviceareoftype:getastheyareusedonlytoquerythedata.SincethedataresideswithElasticsearch,therequestsneedstobemappedtorelevantElasticsearchqueriestogettherequiredresult.InterceptorstoqueryElasticsearchdatareadtherequiredfieldsfromthe:tx-datafieldofthePedestalcontextassetbythevalidationinterceptors.TheimplementationofthevalidatorthatwrapsthequeriestoElasticsearchisthesameasthatofotherservicesexplainedearlier.
ElasticsearchdefinesaQueryDSL(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)thatmustbeusedtoquerythedataagainsttheElasticsearchindex.TheQueryDSLisbasedonJSONandinvolvescreatingaJSONstructureofqueryclausesthatarewrappedwithinaqueryorafiltercontext(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html).Queryclausesmaybeoftypeleafqueryclausesorcompoundqueryclauses.
Leafqueryclauseslookforaparticularvalueinaparticularfield,asshowninthefollowingcodesnippetofanElasticsearchquery.Forexample,queryingforalltheordersthathavestatusopen;thatis,O.Thetermquery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html),usedtogettheopenorders,queriesonlyforexactmatches.Sincethestatusfieldofthelookupschemahasthemappingdefinedasthatoftypekeyword,itsupportsexactmatchesviatheKeywordAnalyzer(https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-keyword-analyzer.html)ofElasticsearch:{"query":{"term":{"status":"O"}}}
Compoundqueryclausesareusedtowraponeormorequeryclausethatmaybeoftypeleaforcompound,asshowninthefollowingcodesnippet.Forexample,queryingforalltheordersthathavestatusdone;thatis,D,andhavearatingof
fourormorethanthat.ElasticsearchprovidesaBoolQuery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html)tocreateBooleancombinationsofoneormorequeriestoformacompoundclause.Forstatus,itusesatermqueryandforratingitusesarangequery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html)ofElasticsearchthatarecompoundedbyamustclausethatmakesboththeconditionstobesatisfiedforanordertobereturnedasaresponseofthisquery:{"query":{"bool":{"must":[{"term":{"status":"D"}},{"range":{"rating":{"gte":4}}}]}}}
ElastischisaClojureclientthatcanbeusedtocreateElasticsearchindexesandquerythem.ElasticsearchalsoprovidestheJavaRESTClient(https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html)andJavaAPIs(https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html)thatcanalsobeusedwithClojure.
Usinggeoqueries
Geoqueries(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html)ofElasticsearchallowfindingtherecordsusingaboundingbox,distancefromagivengeopoint,ortherecordslyingwithinthepolygonmadeupofgeopointsspecifiedasaboundingregion.TheHelpingHandsapplicationrequiresgeoqueriesfortwoofitsroutes,GET/lookup/geoandGET/validate/:service.
Thefirstrouteallowsconsumerstolookupavailableserviceswithinthespecifiedboundingboxofalatitudeandlongitudepairthatcanbeselectedviaamapbydrawingarectangle.Alternatively,consumerscanalsolookupaservicewithin,say,a5kmradiusoftheirlocationspecifiedbytheirgeolocation.Similarly,thesecondroutevalidatesthattheselectedserviceID:servicelieswithintheallowedlimitsofaconsumergeolocationboundary.Boththeroutesinternallyuseeitheraboundingboxquery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-query.html)oradistancequery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html)ofElasticsearch,asshownhere:
{
"query":{
"bool":{
"must":{
"match_all":{}
},
"filter":{
"geo_bounding_box":{
"geo.location":{
"top_left":{
"lat":13.17,
"lon":77.38
},
"bottom_right":{
"lat":12.73,
"lon":77.88
}
}
}
}
}
}
}
{"query":{"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{
"distance":"5km","geo.location":{"lat":12.97,"lon":77.59}}}}}}
Geoqueriesbasedonbounding-boxanddistancerequiresfieldstohavethegeo_point(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html)mappingdefined.Tolookupbydefiningashape,fieldsmusthavegeo_shape(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html)mappingdefined.
GettingstatuswithaggregationqueriesElasticsearchalsosupportsaggregations(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html)thatprovideaggregatedresultsbasedonthegivenqueries.AggregationsareusefulfortheHelpingHandsapplicationtogetthestatusoftheordersandgenerateanalyticsreportsthatcanbeusedtounderstandtheusageoftheapplication.
Forexample,totakealookattheordersreceivedeverymonth,adatehistogramaggregationcanbecreatedonthets_startfield:
{
"aggs":{
"monthly_orders":{
"date_histogram":{
"field":"ts_start",
"interval":"month"
}
}
}
}
Similarly,togetthestatsonratingsreceivedacrossorderssofar,statsaggregationcanbeusedontheratingfield,asshowninthefollowingexample.Statsreturnsthecount,min,max,avg,andsumofthevaluesofthespecifiedfield;thatis,ratinginthiscase:
{
"aggs":{
"rating_stats":{
"stats":{
"field":"rating"
}
}
}
}
Toknowthecurrentstatusoftheordersacrosstheapplication,termsaggregation(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html)canbeusedonthestatusfield,asshowninthefollowingexample.Termsaggregationreturnsthecurrentcountofalltheorderstatusesacrossthesystemtogetareportofhowmanyordersareopen,inprogress,done,orclosed:
{
"aggs":{
"order_status":{
"terms":{
"field":"status"
}
}
}
}
ElasticsearchisalsousedtobuildamonitoringsystemfortheHelpingHandsapplication.SuchamonitoringsystemreliesheavilyonaggregationqueriesofElasticsearchtobuildadashboardtounderstandtheruntimestateofthesystem.ThemonitoringsystemfortheHelpingHandsapplicationisdescribedinPartIV,Chapter11,DeployingandMonitoringSecuredMicroservices.
Creatingamicroserviceforalerts
TheAlertserviceisusedtosendemailalertsandSMS.Alertscanbegeneratedatvariouslevelsbyothermicroservices.Forexample,successfulcreationofaconsumer,provider,service,oranordermayrequireanemailtobesenttotherelevantstakeholders.Similarly,alertsmayberequiredwheneverthereisachangeinthestatusoftheorderoraratingisreceived.TheAlertservicedoesnotmaintainalocaldatabase,itjustgenerateseventsforeachsuccessfulalertsentthatcanbetrackedformonitoringpurposes.ThefollowingtableliststheendpointsfortheAlertservice:
URI Params Description
POST
/alerts/emailto,cc,subject,body
Sendsanalertviaemailtooneormorerecipients.
POST/alerts/sms to,body SendsanalertviaSMStooneormorerecipients.
Addingroutes
Mostly,theAlertservicewilllistenforeventsasanObserverandwillnotreceiverequeststosendalertsviaroutes.Ifitisrequiredtosendalertssynchronously,the/alerts/emailand/alerts/smsroutescanbeused,asdefinedinthefollowingcodesnippet:
(defroutes#{["/alerts/email"
:post(conjcommon-interceptors`auth`core/validate
`core/send-email`gen-events)
:route-name:alert-email]
["/alerts/sms"
:post(conjcommon-interceptors`auth`core/validate
`core/send-sms`gen-events)
:route-name:alert-sms]})
CreatinganemailinterceptorusingPostalPostal(https://github.com/drewr/postal)isaClojurelibrarythatallowssendingemail.ItrequiresSMTPconnectiondetailsandthemessageasamapcontainingtherequireddetailsofto,from,cc,subject,andbodytosendasanemail.PostalcanbeusedwithinthePedestalinterceptortosendanemailiftherequiredfieldsarevalidatedandpresentinthecontext,asshownhere:
(nshelping-hands.alert.core
"InitializesHelpingHandsAlertService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[postal.core:aspostal]
[helping-hands.alert.persistence:asp]
[io.pedestal.interceptor.chain:aschain])
(:import[java.ioIOException]
[java.utilUUID]))
;;--------------------------------
;;ValidationInterceptors
;;--------------------------------
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(->context:request:form-params)]
(if(and(not(empty?params))
(not(empty?(:toparams)))
(not(empty?(:bodyparams))))
(let[to-val(maps/trim(s/split(:toparams)#","))]
(assoccontext:tx-data(assocparams:toto-val)))
(chain/terminate
(assoccontext
:response{:status400
:body"Bothtoandbodyarerequired"})))))
(defvalidate
{:name::validate
:enter
(fn[context]
(if-let[params(->context:request:form-params)]
;;validateandreturnacontextwithtx-data
;;orterminatedinterceptorchain
(prepare-valid-contextcontext)
(chain/terminate
(assoccontext
:response{:status400
:body"Invalidparameters"}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
;;--------------------------------
;;BusinessLogicInterceptors
;;--------------------------------
(defsend-email
{:name::send-email
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
msg(into{}(filter(compsome?val)
{:from"admin@helpinghands.com"
:to(:totx-data)
:cc(:cctx-data)
:subject(:subjecttx-data)
:body(:bodytx-data)}))
result(postal/send-message
{:host"smtp.gmail.com"
:port465
:ssltrue
:user"admin@helpinghands.com"
:pass"resetme"}
msg)]
;;sendemail
(assoccontext:response
{:status200
:body(jp/generate-stringresult)})))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
PostaldependsontheunderlyingSMTPservertoacceptthecredentialsandallowthird-partyclientstosendemails.ServicessuchasGmailmayrestrictthird-partyclientstouseusernameandpassword.
Tosendalerts,itisrecommendedtouseexternalservicessuchasAmazonSES,AmazonSNS,andmoreastheyarereliabletouseandfollowapay-per-usemodel.
SummaryInthischapter,wefocusedonthestep-by-stepimplementationofHelpingHandsmicroservicesusingthePedestalframework.WelearnedhowtoimplementHexagonalArchitectureusingClojureprotocols(https://clojure.org/reference/protocols)andPedestalinterceptors.WealsoimplementedtherequiredmicroservicesfortheHelpingHandsapplicationinPedestal.Inthenextchapter,wewilllearnhowtoconfigureourmicroservicesandmaintaintheruntimestateoftheapplicationthatincludesconnectionwithpersistentstorageandmessagequeuestostoredataandsendevents,respectively.
ConfiguringMicroservices
""Ican'tchangethedirectionofthewind,butIcanadjustmysailstoalwaysreachmydestination.""
-JimmyDean
Microservicesmustbeconfigurabletoadapttotheenvironmentinwhichtheyaredeployed.Theymustsupportexternalconfigurationparametersthatcanbespecifiedatruntimetoconfigurethemaspertheenvironmentinwhichtheyaredeployed.Oncetheconfigurationparametersaredefined,amicroservicemustbeabletoeffectivelypropagatetheconfigurationsacrossitsmodules.Theseconfigurationparametersmightthenbeusedtoinitializedatabaseconnectionsormaintainotherapplicationstatesthatmustbesharedacrossthemodulesofamicroservice.Allthemodulesmusthaveaccesstotheexactsamestateatruntime.Thischapterprovideseffectivesolutionstobuildsuchconfigurableservicesthatcanmanagetheirruntimestateseffectively.Inthischapter,youwilllearnhowtodothefollowing:
ApplyconfigurationprinciplestobuildhighlyconfigurableservicesUseOmniconfforconfigurationValidateconfigurationatstartupandregisteritforuseatruntimeUseamountlibrarytocomposeandmanageapplicationstates
ConfigurationprinciplesAlltheapplicationparametersthatarerelatedtotheenvironmentandaffecttheapplicationstatemustbemadeconfigurable.Forexample,theconnectionstringforaDatomicdatabasethatisusedbyHelpingHandsservicescanbemadeconfigurablesothatitcanbeupdatedexternallytopointtoaspecificinstanceofDatomicinproduction.Configurationparametersalsomakeitpossibletotesttheapplicationinvariousenvironments.Forexample,ifaDatomicconnectionstringismadeconfigurableforHelpingHandsservices,itcanbeusedtotesttheserviceswithanin-memoryinstanceofDatomicinlocaldevelopmentenvironmentsandlaterchangedtopointtotheproductioninstanceofDatomiconcetheyaredeployed.
Definingconfigurationparameters
Applicationsmustsupportmultiplewaysofdefiningtheconfigurationparameters.Usingcommand-lineargumentsisoneofthemostcommonwaysofspecifyingtheconfigurationparametersfortheapplication.Environmentvariablesandexternalconfigurationfilescanalsobespecifiedforapplicationstopicktheconfigurationparametersatruntime.SinceClojureusesJVMasitsruntimeengine,applicationsbuiltinClojurecanacceptconfigurationparametersasJavapropertiesaswell.
Applicationsthatacceptconfigurationparametersfrommultiplesourcesmustdecideonthepreferenceofvarioussources.Forexample,configurationparametersspecifiedatthecommandlineasJavapropertiescanoverwritethevaluesdefinedbytheenvironmentvariablesthatinturncanoverridethedefaultconfigurationdefinedintheconfigurationfile.
UsingconfigurationparametersOneoptiontoprovideaccesstoconfigurationparametersistoloadthematstartupandpassthemasargumentstothefunctions.Inthiscase,everytimeanewconfigurationparameterisadded,itmayresultinthechangeofthefunctionsignaturethatcanaffectallthefunctionsdependentonit.
Configurationparametersmustnotbetieddirectlytotheargumentsofthefunctionbecauseconfigurationparametersthatareloadedatstartuptimemaynotchangethroughoutthelifecycleoftheservice.Instead,configurationparameterscanbereadonceatstartupandkeptasimmutableconstantsthatcanbedirectlyaccessedbyallthefunctionsthatrequireoneormoreconfigurationparameter.
Readingtheconfigurationparametersfromvarioussourcesandmakingthemaccessibledoesnotguaranteetheconfigurationwillbecorrectunlesstheyareusedbytheapplication.Forexample,iftheapplicationneedsaportnumberasaconfigurationparameter,itmustbeverifiedassoonasitisreadandmustbeashortpositivenumberwithamaximumvalueof65535.Ifthesechecksarenotperformedatthetimetheconfigurationsareread,theyaregoingtoresultinruntimeexceptionslaterduringtheapplicationlifecycle.Detectingsuchconfigurationissuesatalaterpointintimeiscostlyastheconfigurationneedstobeupdatedandtheapplicationneedstoberedeployedtopickupontheupdatedconfiguration.
UsingOmniconfforconfigurationOmniconf(https://github.com/grammarly/omniconf)isanopensourceconfigurationlibraryforClojureprojectsthatcanbeusedtoconfiguremicroservicesoftheHelpingHandsapplication(refertoChapter3,MicroservicesforHelpingHandsApplication,andChapter8,BuildingMicroservicesforHelpingHands).Omniconfnotonlyallowstheapplicationtodefinethepreferencewithrespecttovariousconfigurationsourcesbutalsotoverifythematstartup.Internally,itkeepsalltheconfigurationparametersstoredasanimmutableconstantthatcanbeaccessedasaregularClojuredatastructure.
Omniconfisoneoftheoptionsforconfigurationmanagement.Libraries,suchasEnviron(https://github.com/weavejester/environ),Config(https://github.com/yogthos/config),Aero(https://github.com/juxt/aero),andFluorine(https://github.com/reborg/fluorine)canalsobeusedforconfigurationmanagement.
EnablingOmniconf
ToenableanOmniconflibraryforanexistingproject,suchasHelpingHandsConsumerService,addtheOmniconfdependencytotheproject.cljfileandaddJVMopts,conftothedevprofilethatpointstotheconf.ednOmniconfconfigurationfile:
(defprojecthelping-hands-consumer"0.0.1-SNAPSHOT"
:description"HelpingHandsConsumerApplication"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
[io.pedestal/pedestal.jetty"0.5.3"]
;;DatomicFreeEdition
[com.datomic/datomic-free"0.9.5561.62"]
;;Omniconf
[com.grammarly/omniconf"0.2.7"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-
api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
...
:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]
[org.clojure/tools.nrepl"0.2.12"]]}
:dev{:aliases{"run-dev"["trampoline""run""-m"
"helping-hands.consumer.server/run-dev"]}
:dependencies[[io.pedestal/pedestal.service-tools"0.5.3"]]
:resource-paths["config","resources"]
:jvm-opts["-Dconf=config/conf.edn"]}
:uberjar{:aot[helping-hands.consumer.server]}
:doc{:dependencies[[codox-theme-rdash"0.1.1"]]
:codox{:metadata{:doc/format:markdown}
:themes[:rdash]}}
:debug{:jvm-opts
["-server"(str"-agentlib:jdwp=transport=dt_socket,"
"server=y,address=8000,suspend=n")]}}
:main^{:skip-aottrue}helping-hands.consumer.server)
IntegratingwithHelpingHandsTheHelpingHandsservicesthatwereimplementedinthepreviouschapterusedafixedDatomicdatabaseURI,suchasdatomic:mem://consumer,fortheconsumerdatabasemanagedbytheconsumerservice.InsteadoffixingthenameofthedatabaseandDatomicURI,itmustbemadeconfigurablesothatitcanbechangedatthetimeofdeployment.Forexample,considerascenariowhereyouwishtoruntwoinstancesofConsumerservicebutwithseparateconsumerdatabases.ItwillnotbepossibletodosoiftheDatomicdatabaseURIishardcodedintheimplementationandnotmadeconfigurable.
Omniconfrequiresalltheconfigurationparameterstobedefinedviatheomniconf.core/definefunction.FortheConsumerserviceoftheHelpingHandsapplication,addanewhelping-hands.consumer.confignamespaceandinitializetheconfigurationasshownhere:
(nshelping-hands.consumer.config
"DefinesConfigurationfortheService"
(:require[omniconf.core:ascfg]))
(defninit-config
"Initializestheconfiguration"
[{:keys[cli-argsquit-on-error]:asparams
:or{cli-args[]quit-on-errortrue}}]
;;definetheconfiguration
(cfg/define
{:conf{:type:file
:requiredtrue
:verifieromniconf.core/verify-file-exists
:description"MECBOTconfigurationfile"}
:datomic
{:nested
{:uri{:type:string
:default"datomic:mem//consumer"
:description"DatomicURIforConsumerDatabase"}}}})
;;like-:some-option=>SOME_OPTION
(cfg/populate-from-envquit-on-error)
;;loadpropertiestopick-Dconffortheconfigfile
(cfg/populate-from-propertiesquit-on-error)
;;Configurationfilespecifiedas
;;EnvironmentvariableCONForJVMOpt-Dconf
(when-let[conf(cfg/get:conf)]
(cfg/populate-from-fileconfquit-on-error))
;;like-:some-option=>(java-Dsome-option=...)
;;reloadJVMargstooverwriteconfigurationfileparams
(cfg/populate-from-propertiesquit-on-error)
;;like-:some-option=>-some-option
(cfg/populate-from-cmdcli-argsquit-on-error)
;;Verifytheconfiguration
(cfg/verify:quit-on-errorquit-on-error))
(defnget-config
"Getsthespecifiedconfigparamvalue"
[&args]
(applycfg/getargs))
Inthisexample,thereisamandatoryconfigurationparameter,:conf,definedtobeof:filethatmustpointtotheconf.ednconfigurationfile.Thereisalsoaverifierattachedtoitthatvalidatesthepresenceoftheconf.ednfilebasedonthedefinedlocation.Also,thereisa:datomicconfigurationparameterdefinedthatisnestedandhasa:uriparameter,definedasstringtype,withadefaultvalueofdatomic:mem://consumerthatpointstoanin-memoryDatomicdatabase.
Afterdefiningtheconfigurationparameters,theimplementationchecksfortheconfparametervaluebyfirstloadingtheJVMproperties.The-DconfJVMpropertypointstotheconf.ednfileasdefinedinthedevprofileofproject.clj.Theconfigurationparametersarereadinthesequenceofenvironmentvariables,propertiesfile,andcommandline,eachoverwritingthevaluesdefinedbytheprevioussourceaspertheloadingsequence.
Aget-configutilitymethodisalsodefinedwithinthesamenamespaceforothermodulestolookuptheconfigurationparametersthatareloadedbythisnamespaceusingOmniconf.Toloadtheconfigurationatstartup,calltheinit-configmethodattheapplicationentrypoint,thatis,helping-hands.consumer.server,asshownhere:
(nshelping-hands.consumer.server
(:gen-class);for-mainmethodinuberjar
(:require[io.pedestal.http:asserver]
[io.pedestal.http.route:asroute]
[helping-hands.consumer.config:ascfg]
[helping-hands.consumer.service:asservice]))
...
(defnrun-dev
"Theentry-pointfor'leinrun-dev'"
[&args]
(println"\nCreatingyour[DEV]server...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
(->service/service;;startwithproductionconfiguration
...
;;Wireupinterceptorchains
server/default-interceptors
server/dev-interceptors
server/create-server
server/start))
(defn-main
"Theentry-pointfor'leinrun'"
[&args]
(println"\nCreatingyourserver...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
(server/startrunnable-service))
Now,ifyoutrytoruntheConsumerserviceindevmodeatREPL,Omniconfwillloadtheconfiguration,verifyit,andmakeitavailableviathehelping-hands.consumer.config/get-configmethod.Sincewehaven'tcreatedtheconf.ednfileinthespecifiedconf/location,theverifiershouldfailattheinitializationstepitself,asshownhere:
helping-hands.consumer.server>(defserver(run-dev))
Creatingyour[DEV]server...
CompilerExceptionjava.io.FileNotFoundException:config/conf.edn(Nosuchfileor
directory),compiling:(form-init3431514182044937086.clj:118:44)
Addaconf.ednfileundertheconfigdirectoryanddefineonlytheDatomicURIconfiguration,asshownhere:
{:datomic{:uri"datomic:mem://consumer"}}
Now,theconfigurationisvalid,asitfindsthedefinedconf.ednfile.Inthiscase,asshownintheREPLsessioninthefollowingcodesnippet,Omniconfwilldumptheloadedconfigurationthatcanthenbeverifiedtomakesurealltheconfigurationparametersareloadedasexpected.Notethattherequired:confparameterisnotdefinedintheconf.ednfile,butitisdefinedastheJVMpropertythatisalsoreadinthesequence:
;;startserviceindevmode
helping-hands.consumer.server>(defserver(run-dev))
Creatingyour[DEV]server...
Omniconfconfiguration:
{:conf#object[java.io.File0x27a8b5d3"config/conf.edn"],
:datomic{:uri"datomic:mem://consumer"}}
#'helping-hands.consumer.server/server
;;trylookinguptheconfigurationparameter
helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic)
{:uri"datomic:mem//consumer"}
helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic
:uri)
"datomic:mem//consumer"
Now,trychangingtheDatomicURlparameterintheconfig.ednfiletodatomic:mem://consumer-sample.Itwilloverwritethedefaultvalueandcanthenbe
usedbytheapplicationusingtheget-configmethodasshownhere:
helping-hands.consumer.server>(defserver(run-dev))
Creatingyour[DEV]server...
Omniconfconfiguration:
{:conf#object[java.io.File0x60e62394"config/conf.edn"],
:datomic{:uri"datomic:mem://consumer-sample"}}
#'helping-hands.consumer.server/server
helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic)
{:uri"datomic:mem://consumer-sample"}
helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic
:uri)
"datomic:mem://consumer-sample"
ThepersistencenamespaceoftheConsumerservicecannowreadthedatabaseURIdirectlyfromtheconfigurationinsteadofexpectingitasanargumentofthecreate-consumer-databasefunction:
(nshelping-hands.consumer.persistence
"PersistencePortandAdapterforConsumerService"
(:require[datomic.api:asd]
[helping-hands.consumer.config:ascfg]))
...
(defncreate-consumer-database
"Createsaconsumerdatabaseandreturnstheconnection"
[]
;;createandconnecttothedatabase
(let[dburi(cfg/get-config[:datomic:uri])
db(d/create-databasedburi)
conn(d/connectdburi)]
;;transactschemaifdatabasewascreated
(whendb
(d/transactconn
[{:db/ident:consumer/id
:db/valueType:db.type/string
:db/cardinality:db.cardinality/one
:db/doc"UniqueConsumerID"
:db/unique:db.unique/identity
:db/indextrue}
...
]))
(ConsumerDBDatomic.conn)))
OmniconfworkswithboththeLeiningenandBootbuildtoolsofClojure.Formoredetailsandusageinformationofallthepossibleoptions,takealookattheexample-lein(https://github.com/grammarly/omniconf/tree/master/example-lein)andexample-boot(https://github.com/grammarly/omniconf/tree/master/example-boot)projectsofOmniconf.
ManagingapplicationstateswithmountOncetheconfigurationparametersaredefinedusingOmniconf,theyareaccessibleacrossthenamespacesasimmutabledata.Theconfigurationparametersareoftenusedtocreatestatefulobjects,suchasdatabaseconnections.Forexample,intheConsumerserviceproject,Omniconfmadeitpossibletocreateaconsumerdatabasebydirectlylookingupthe:datomic:uriconfigurationparameterwithinthecreate-consumer-databasefunction.
Thehelping-hands.consumer.persistence/create-consumer-databasefunctionhasasideeffectofdatabasebeingcreatedandalsoanewconnectionbeinginitializedtoconnecttothecreateddatabase.ThisconnectionhasastatethatmustbesharedacrossothernamespacesoftheHelpingHandsConsumerservicethatneedaccesstothedatabase.Inthecurrentimplementation,theconnectionwasinitializedatthefirstcalltothehelping-hands.consumer.core/consumerdbasshownhere:(nshelping-hands.consumer.core"InitializesHelpingHandsConsumerService"(:require[cheshire.core:asjp][clojure.string:ass][helping-hands.consumer.persistence:asp][io.pedestal.interceptor.chain:aschain])(:import[java.ioIOException][java.utilUUID]))
;;delaythecheckfordatabaseandconnection;;tillthefirstrequesttoaccess@consumerdb(def^:privateconsumerdb(delay(p/create-consumer-database)))
Insteadofcreatingastateusingdelay,thestatemanagementcanbehandledeffectivelyusingalibrary,suchasmount(https://github.com/tolitius/mount).Creatingapplicationstatesusingmountallowsforthereloadingoftheentireapplicationstateusingstartandstopfunctionsprovidedbythemount.corenamespace.The
mountlibraryalsohelpswithstatecompositionbyallowingtheapplicationtostartonlywithspecificstatesandatthesametimeswappingotherswithnewvalues.Italsosupportsruntimearguments.
Component(https://github.com/stuartsierra/component)isanotherClojurelibrarythatiswidelyusedtomanagethelifecycleofstatefulobjectsinaClojureproject.mountisanalternativetotheComponentlibrarywithkeydifferences(https://github.com/tolitius/mount/blob/master/doc/differences-from-component.md#differences-from-component),oneofthembeingComponent'srequirementoftheentireappbeingbuiltarounditscomponentobjectmodel.
Enablingmount
ToenablethemountlibraryfortheHelpingHandsConsumerserviceapplication,addamountdependencytotheproject.cljfileasshowninthefollowingcodesnippet.Also,createanewhelping-hands.consumer.statenamespacethatwillbeusedtodefinethestatesusingthemount.core/defstatefunction,whichcanbereferredtobyothernamespacesoftheprojecttogetaccesstothecurrentstateofthedefinedobject,suchastheDatomicdatabaseconnection:
(defprojecthelping-hands-consumer"0.0.1-SNAPSHOT"
:description"HelpingHandsConsumerApplication"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
[io.pedestal/pedestal.jetty"0.5.3"]
;;DatomicFreeEdition
[com.datomic/datomic-free"0.9.5561.62"]
;;Omniconf
[com.grammarly/omniconf"0.2.7"]
;;Mount
[mount"0.1.11"]
...
]
...
:main^{:skip-aottrue}helping-hands.consumer.server)
IntegratingwithHelpingHandsTointegratemountwiththeHelpingHandsConsumerserviceproject,createastateforaDatomicdatabaseconnectionwithinthehelping-hands.consumer.statenamespace,asshownhere:
(nshelping-hands.consumer.state
"InitializesStateforConsumerService"
(:require[mount.core:refer[defstate]:asmount]
[helping-hands.consumer.persistence:asp]))
(defstateconsumerdb
:start(p/create-consumer-database)
:stop(.stopconsumerdb))
The:startclauseiscalledatthetimeofstartupand:stopiscalledatshutdown.ThefunctionstopisdefinedfortheConsumerDBprotocol,asshowninthefollowingexampleunderthehelping-hands.consumer.persistencenamespace:
(nshelping-hands.consumer.persistence
"PersistencePortandAdapterforConsumerService"
(:require[datomic.api:asd]
[helping-hands.consumer.config:ascfg]))
(defprotocolConsumerDB
"Abstractionforconsumerdatabase"
(upsert[thisidnameaddressmobileemailgeo]
"Adds/Updatesaconsumerentity")
(entity[thisidflds]
"Getsthespecifiedconsumerwithallorrequestedfields")
(delete[thisid]
"Deletesthespecifiedconsumerentity")
(close[this]
"Closesthedatabase"))
...
(defrecordConsumerDBDatomic[conn]
ConsumerDB
...
(close[this]
(d/shutdowntrue)))
Next,addthestartupandshutdownhooksformountattheapplicationentrypointunderthehelping-hands.consumer.servernamespace.Theshutdownhookdefinedat
theapplicationentrypointcallsthe:stopclauseofmountthatmustcleanupalltheresourcesrelatedtothestatefulobject,thatis,theDatomicconnectionasshowninthefollowingexample:
(nshelping-hands.consumer.server
(:gen-class);for-mainmethodinuberjar
(:require[io.pedestal.http:asserver]
[io.pedestal.http.route:asroute]
[mount.core:asmount]
[helping-hands.consumer.config:ascfg]
[helping-hands.consumer.service:asservice]))
...
(defnrun-dev
"Theentry-pointfor'leinrun-dev'"
[&args]
(println"\nCreatingyour[DEV]server...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(->service/service;;startwithproductionconfiguration
...
;;Wireupinterceptorchains
server/default-interceptors
server/dev-interceptors
server/create-server
server/start))
(defn-main
"Theentry-pointfor'leinrun'"
[&args]
(println"\nCreatingyourserver...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(server/startrunnable-service))
Oncemountissetuptoinitializethestateatstartup;thedefinedstateconsumerdbcannowbereferredtoacrossnamespacesanduseddirectlyforthehelping-hands.consumer.corenamespaceasshownhere:
(nshelping-hands.consumer.core
"InitializesHelpingHandsConsumerService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.consumer.persistence:asp]
[io.pedestal.interceptor.chain:aschain]
[helping-hands.consumer.state:refer[consumerdb]])
(:import[java.ioIOException]
[java.utilUUID]))
;;delaythecheckfordatabaseandconnection
;;tillthefirstrequesttoaccess@consumerdb
;;NOLONGERREQUIREDDUETOMOUNT
;;(def^:privateconsumerdb
;;(delay(p/create-consumer-database)))
...
;;Usethereferredstatefulconsumerdbdirectlyintheinterceptor
(defupsert-consumer
{:name::consumer-upsert
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
id(:idtx-data)
db(.upsertconsumerdbid(:nametx-data)
(:addresstx-data)(:mobiletx-data)
(:emailtx-data)(:geotx-data))]
...))
:error...})
SummaryInthischapter,wefocusedonhowtobuildconfigurableapplicationsthatcanadaptaspertherequirementsanddependenciesathand.WelookedatanopensourceconfigurationutilitycalledOmniconfthatprovidesaneffectivewaytodefineandvalidateconfigurationparametersforClojureapplications.
WealsolookedathowtheruntimestateoftheapplicationcanbecomposedandsharedamongvariousnamespacesoftheClojureapplication.Welookedatanopensourcelibrarycalledmountthathelpsapplicationstomanageandcomposetheirstatesatruntimewithoutaffectingtheoverallstructureoftheimplementation.
Inthenextchapter,wewilllearnhowtoadoptevent-drivenarchitectureforHelpingHandsmicroservices.WewillalsolearnhowtobuilddataflowsforthemicroservicesofHelpingHands.
Event-DrivenPatternsforMicroservices
""Thesinglebiggestproblemwithcommunicationistheillusionthatithastakenplace.""
-GeorgeBernardShaw
Microservicesaddressasingleboundedcontextandaredeployedindependentlyononeormorephysicalmachinesthataredistributedacrossanetwork.Althoughtheyaredeployedinisolation,theyneedtointeractwitheachothertoaccomplishapplication-leveltasksthatmaycutacrossmultipleboundedcontexts.Thechoiceofcommunicationmediumandmethodhasagreatimpactontheperformanceanddurabilityoftheentiremicroservice-basedarchitecture.Eventsareoneofthemethodsofasynchronouscommunicationamongmicroservicestoexchangedataofinterest.Part-1ofthebookexplainstheimportanceoftheobservermodelandhowamessagebroker(https://en.wikipedia.org/wiki/Message_broker)helpsinsendingandreceivingeventsinamicroservicesarchitecture.Inthischapter,youwill:
Learnaboutevent-drivenpatternsforeffectivemessagingamongmicroservicesLearnhowtouseApacheKafkaasamessagebrokerformicroservicesLearnhowtouseApacheKafkaforEventSourcingLearnhowtointegrateApacheKafkawithHelpingHandsmicroservices
Implementingevent-drivenpatternsEvent-drivenpatternsaddresstheobservermodelofcommunicationtosendmessagesamongmicroservices.Themessagesaresentandreceivedthroughamessagebrokerthatactsasaconnectingbridgebetweenthesenderandthereceiver.Inamicroservicesarchitecture,thesemessagesmaybegeneratedaseventsasanoutcomeoftheactiontakenbythemicroservice.Messagesforwhichthesourcemicroservicedoesnotexpectaresponsefromthetargetservicecanbepublishedaseventsasynchronously,insteadofsendingthemoveraRESTAPIfordirectcommunication.Sinceitisnotadirectcommunication,aneventcanbepublishedonceandconsumedbymorethanonemicroservicethathassubscribedtoreceiveit.Moreover,thesenderdoesnotgetblockedbythereceiverforeacheventthatispublished.
Messagebrokersalsohelptobuildaresilientarchitectureforevent-drivencommunicationasreceiversneednotbeavailablewhiletheeventisbeingproducedandtheycanconsumethemessagesatwill.Incaseoffailures,thereceivercanberestartedanditcanstartconsumingtheeventswhereitleftoff.Duringthedowntime,themessagebrokeritselfactsasaqueueandcachesalltheeventsthatwerenotconsumedbythereceiverandmakesitavailabletothereceiverondemand.
Asynchronouscommunicationviaeventsalsodecouplesthesenderfromthereceiverthathelpsinscalingboththesidesindependently.Thisfeatureisofprimeimportanceinamicroservice-basedarchitectureasitallowsthemicroservicestobedeployedindependentlyofothermicroservicesfromwhichitconsumestheeventsviaamessage-broker.Event-drivenpatternsarealsousedforasynchronoustaskssuchasstoringauditlogstokeeptrackofruntimestateoftheapplicationandsendingalerts.
Theobservermodel,alongwithvariousdatamanagementpatterns,areexplainedinChapter2,MicroservicesArchitectureofthisbook.
EventsourcingEventscanbeusedtoadvertisethechangesinthestateoftheentitiesmanagedbyamicroservice.Insuchcases,eventscarrytheupdatedstateoftheentityincludingtheentityidentifierandthevaluesofthefieldsthathavechanged.Itisalsorecommendedtoincludeauniqueidentifierandversionnumberaswellwitheachupdateevent.Anyinterestedmicroservicecanthensubscribetotheseeventsviaamessagebrokerandreceivethechangestoupdatethestateoftheentitylocally.TheLookupServiceoftheHelpingHandsapplicationisonesuchservicethatlistenstoallthestatechangeeventsgeneratedbytheconsumer,provider,andordermicroservicestokeepanupdateddatasetforuserstolookup.
Oneofthemainadvantagesofpublishingallthestatechangesacrossmicroservicesasimmutableeventsistomakesurethatthestateoftheentiremicroservice-basedapplicationcanberebuiltbyjustprocessingtheseeventsinthesequenceinwhichtheyarepublished.Boththecurrentstateoftheapplicationaswellasthestateinthepastcanbereconstructedbyprocessingtheseeventsintheexactsamesequence.Abilitytoreplaytheeventsnotonlyhelpsinrebuildingthestateoftheapplication,butalsohelpsinauditinganddebugging.
Messagebrokers,suchasApacheKafka(https://kafka.apache.org/),allowpublishingmessagesdecoupledfromconsumingthemandeffectivelyactasastoragesystemthatmaintainsadurablelogofpublishedevents.Suchsystemsallowtheeventstobecapturedbymultiplesystemsatthesametime,asshownintheprecedingdiagram.Forexample,theeventspublishedbyServiceAandServiceBcanbeconsumedbyServiceC,butatthesametime,theseeventscanbebacked-upinabackupstoreorcapturedinatransactionalstoreforbuildingmachinelearningmodelsorcapturedforreal-timemonitoringandreportingoftheapplicationstateinrealtime.Sincethelogsretainedbythebrokersareimmutableanddurable,applicationsthataredevelopedin-futurecanalsoreplaytheeventstobuildthetemporalstateoftheapplicationandworkonit.
UsingtheCQRSpatternTheCommandQueryResponsibilitySegregation(CQRS)patternappliesthecommand-queryseparation(https://en.wikipedia.org/wiki/Command-query_separation)principlebysplittingtheapplicationintotwoparts,querysideandcommandside.Querysideisresponsibleforgettingthedatabyonlyqueryingthestateoftheapplication,whereascommandsideisresponsibleforchangingthestateofthesystembyperformingcreate,update,ordeleteoperationsonapplicationdatathatmayresultinupdationofoneormoredatabasesusedbytheapplication.AlthoughtheCQRSpatternisoftenusedinconjunctionwiththeevent-sourcingpattern,itneednotbetiedtoeventsandcanbeappliedtoanyapplicationwithorwithoutevents.TheCQRSpatternisrecommendedforapplicationsthatarenotbalancedwithrespecttoreadandwriteloads.
CQS(https://en.wikipedia.org/wiki/Command-query_separation)principlewasdevisedbyBertrandMeyer(https://en.wikipedia.org/wiki/Bertrand_Meyer),whereasCQRStermwasfirstcoinedbyGregYoungaspartofCQRSdocuments(https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf).
TheCQRSpatternfitswellwithmicroservice-basedarchitectureastheentireapplicationissplitintoseparateservices,eachhavingtheirowndatamodelandpublishingthechangesinthestateoftheirdatamodelasevents.Butatthesametime,itisalsochallengingtokeeptheseseparatemodelsconsistent.Thisiswheresagasareuseful,whichsupporteventualconsistency.ThesagaspatternhasbeendescribedinChapter2,MicroservicesArchitecture.
FormoredetailsontheCQRSpatternanditsusage,readtheCQRSarticlebyMartinFowler(https://martinfowler.com/bliki/CQRS.html)andtheClarifiedCQRSarticlebyUdiDahan(http://udidahan.com/2009/12/09/clarified-cqrs/).
IntroductiontoApacheKafkaApacheKafkaisadistributedstreamingplatformthatallowsapplicationstopublishandsubscribetoastreamofrecords.ApacheKafkaisnotjustamessagequeue,italsoallowsapplicationstopublishtheeventsthatarethenstoredbyKafkaasanimmutableloginafault-tolerantway.Itallowstheproducersandconsumersoftheeventstoscalehorizontallywithoutaffectingeachother.SincetheeventsareloggedinthesamesequenceastheyarepublishedwithinKafka,itallowsconsumerstoreplaythelogfromanduptothedesiredpointtoreconstructviewsoftheapplicationstate.
DesignprinciplesKafkaisrunasaclusterofoneormoreserversthatactasmessagebrokers(https://en.wikipedia.org/wiki/Message_broker)inthesystem.Kafkacategorizesthestreamofrecordsundertopicsthatareusedbyproducersandconsumerstoproducerecordsandconsumethem,respectively.Eachrecordconsistsofakey-valuepairandatimestamp.
AtypicalKafkaclusterisshowninthefollowingdiagramalongwiththestructureofaKafkatopicanditspartitions.AtopicisacoreabstractionforastreamofrecordsinKafka.Producerspublishtherecordstoatopicthatcanhavezero,one,ormoreconsumerssubscribedtoit.Eachtopicconsistsofoneormorepartitionsthatareordered,immutablesequenceofrecordsthatisappendedtoacommitlogandstoredonadiskandreplicatedacrossserversforfault-toleranceanddurability.Eachrecordthatispublishedonatopicgetsassignedtoapartitionwithinwhichitisassignedasequentialidentifiernumberthatiscalledtheoffset.Theoffsetuniquelyidentifiesarecordwithinthepartitionandisalsousedasareferencebyconsumerstotracktheirstateofconsumption.Offsetsallowconsumerstoresetthemselvestoanearlierstatetoeitherreplaytherecordsorfast-forwardintimetoskiptherecords.AKafkaclusterretainsallthepublishedrecordsonlyforaconfigurableretentionperiod.Posttheconfiguredretentionperiod,therecordsarenolongeravailableforconsumersirrespectiveofwhethertheywereconsumedearlierornot:
Kafkaproducerspublishdatatooneormoretopicsandhavethefreedomtodecidethepartitionforthepublishedrecords.Kafkaconsumersalwaysjoinwithaconsumergrouplabelthatissharedwithonemoreconsumerthatmaybe
presentononeormoremachines.TheConsumerGroupplaysanimportantroleinthewaytheKafkaclusterdeliversthepublishedrecordstotheconsumers.EachrecordpublishedonaKafkatopicisdeliveredtoonlyoneconsumerwithinaConsumerGroup.Forexample,intheprecedingdiagram,therearetwoconsumergroups—ConsumerGroupAandConsumerGroupBandsixconsumerswiththreeconsumersineachgroup.Anyrecordpublishedonpartitions0,1,2,or3issenttoonlyoneconsumerwithineachgroup.Forexample,recordspublishedtoPartition-0issenttoC1ofConsumerGroupAandC2ofConsumerGroupB.Kafkabalancestheflowofrecordsacrosstheconsumersintheconsumergroups.
ApacheKafkaprovidestotalorderingoverrecordsonlywithinapartition.Usecasesthatrequiretotalorderingovertherecordspublishedonatopicmusthavethetopicconfiguredtohaveasinglepartition.Thisconfigurationalsolimitsthethroughputtoonlyoneconsumerprocessperconsumergroup.
GettingKafkaApacheKafkaisanopen-sourceplatformandcanbedownloadedfromitsdownloadpage(https://kafka.apache.org/downloads).Fortheexamplesinthisbook,downloadandextractthe1.0.0release(https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz)fromoneofthemirrors,asshownhere:
#downloadKafka1.0.0withScala2.11(Recommended)
%wgethttp://redrockdigimark.com/apachemirror/kafka/1.0.0/kafka_2.11-1.0.0.tgz
...
#extractKafkadistribution
%tar-xvfkafka_2.11-1.0.0.tgz
...
#switchtoKafkadirectory
%cdkafka_2.11-1.0.0
ApacheKafkaServersrequireApacheZookeeper(https://zookeeper.apache.org/)forclustercoordination.KafkadistributionpackagesasinglenodeZookeeperinstanceforconvenience;however,itisrecommendedtosetupanexternalZookeeperclusterforproductiondeployments.StartasinglenodeZookeeperwiththedefaultpropertiesfilepackagedwithKafka,asshownhere:
#startZookeeper
bin/zookeeper-server-start.shconfig/zookeeper.properties
....
INFObindingtoport0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
OncetheZookeeperisstarted,eitherswitchtoanewterminalorputtheZookeeperprocessinthebackgroundtogettheaccesstothesameterminal.Next,startaKafkaserverwiththedefaultpropertiesfilepackagedwithKafka,asshownhere:
#startKafkaServer
bin/kafka-server-start.shconfig/server.properties
...
INFOKafkaversion:1.0.0(org.apache.kafka.common.utils.AppInfoParser)
INFOKafkacommitId:aaa7af6d4a11b29d(org.apache.kafka.common.utils.AppInfoParser)
INFO[KafkaServerid=0]started(kafka.server.KafkaServer)
Thedefaultserver.propertiesfilecontainsafixedbroker.idpropertythatissetto0andthelistenersconfiguredtoadefaultportof9092withlog.dirpointingto/tmp.ForasingleKafkaserver,thesesettingsarefine,buttostartmultipleKafkaserversonthesamemachine,thesepropertiesmustbechangedforeachserverto
avoidID,port,andcommitlogdirectoryclashes.Next,createatopicbythenameoftestwithasinglepartition,asshownhere:
#createatopic
%bin/kafka-topics.sh--create--zookeeperlocalhost:2181--replication-factor1--
partitions1--topictest
Createdtopic"test".
#Listandconfirmthattopicwascreated
%bin/kafka-topics.sh--list--zookeeperlocalhost:2181
test
ThecreatetopiccommandrequirestheaddresstotheZookeeperthatisusedbyKafkacluster.SinceZookeeperwasstartedwithdefaultproperties,itwouldhavetakenthe2181portifthatwasfreeonthemachine.Next,startaproducerinanewterminalandpublishsomemessages,asshownhere:
#startanewproducertoproducethemessagesontopic'test'
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topictest
>HelloKafka!
>HelloHelpingHands!
>
ThestartedproducerconnectstotheKafkaserveronport9092;thatis,thedefaultportofKafkaserver.Thestartedproducerisalsoconfiguredtopublishthemessagesonthetesttopicthatwascreatedearlier.Oncetheproducerisstarted,publishthetwomessages,asshownintheprecedingcodesnippet,andstartaconsumerforthesametopicinanewterminal,asshownhere:
#startanewconsumertoconsumethemessagesfromtesttopicfromthebeginning
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-
beginning
HelloKafka!
HelloHelpingHands!
Theconsumerreceivesthepublishedmessagesandprintsthemontheconsole.Now,gobacktotheproducerterminalandpublishonemoremessageandverifythatconsumerreceivesonlythenewmessage,asshownhere:
#publishanewmessage
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topictest
>HelloKafka!
>HelloHelpingHands!
>KafkaWorks!
>
#validatethemessageontheconsumerterminal
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-
beginning
HelloKafka!
HelloHelpingHands!
KafkaWorks!
KafkaManager(https://github.com/yahoo/kafka-manager)isausefultoolthathelpsinmanagingoneormoreKafkaclustersusingasingleweb-basedinterface.
UsingKafkaasamessagingsystemKafkaprovidesbothqueuing(http://en.wikipedia.org/wiki/Message_queue)andpublish-subscribe(http://en.wikipedia.org/wiki/Publish-subscribe_pattern)constructsofamessagingsystembytheconceptofaConsumerGroup.ThemessagespublishedonatopicpartitionarebroadcastedtoaconsumerwithineachConsumerGroupandwithintheConsumerGroup,eachconsumerreceivesthemessagesfromadifferentpartitionofatopic.Therefore,thenumberofconsumerspresentintheConsumerGroupmustnotbemorethanthenumberofpartitionspresentinatopic.Iftheyaremorethanthenumberofpartitions,thentheywillbejustsittingidleandwillonlygetthemessageifoneoftheconsumersfailswithinthegroup.Forexample,todemonstratethemessagingcapabilitiesofKafka,startonemoreconsumerforthetesttopic,asshownhere:#startanotherconsumer%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-beginning
Oncetheconsumerisstarted,anyothermessagethatissentfromtheproducerprocessisreceivedbyboththeconsumers.ThishappensbecauseboththeconsumersareapartofthedifferentConsumerGroupandasperdesign,Kafkawillbroadcastthepublishedmessagesonatopicacrossconsumergroups.SincecurrentsetuphasonlyoneconsumerintheConsumerGroup,bothconsumersreceivethemessage.Next,stopbothoftheconsumerprocessesandstartthemasapartofthesameConsumerGroup,asshownhere:
#startfirstconsumer
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test--from-beginning
#startsecondconsumer
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test--from-beginning
Now,themessagespublishedviaproducerarereceivedbyonlyoneoftheconsumersasboththeconsumersareapartofthetestgroupandthetesttopichasadefaultpartitioncountof1.Tryterminatingthefirstconsumerprocessthatisreceivingthemessagesandsendanewmessagefromtheproducer.Noticethatnow,themessageisbeingreceivedbythesecondconsumerthatbecomesactiveandstartsconsumingthemessagesfromthetopicpartitionwheretheterminated
consumerleftoff.
UsingKafkaasaneventstoreApacheKafkaisnotjustlimitedtoamessagingsystem,itcanalsobeusedasadurablestorageforimmutablerecordsandtobuildastreamingdatapipelineontopofit.Itiswellsuitedforusecasessuchaswebsiteactivitytracking,real-timemonitoring,logaggregation,andprocessingstreamsofdata.AnymessagesthatarepublishedonaKafkatopicarepersistedondiskandreplicatedacrossKafkaserversbasedontheconfigurationforfaulttolerance.SinceKafkaguaranteesthesequenceofmessageswithinatopicpartition,itallowsconsumerstocontroltheirreadpositionirrespectiveofotherconsumersofthesametopic.ThefollowingexampleshowshowtheeventspublishedbyaproduceraremadeavailabletotheconsumersbasedonthetopicandtheassociatedConsumerGroup:
#startproducerfor'test'topic
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topictest
>HelloApacheKafka!
>HelloKafkaEvent!
>
#startconsumerin'test'consumergrouplisteningto'test'topic
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test--from-beginning
HelloApacheKafka!
HelloKafkaEvent!
#consumefrombeginninginadifferentconsumergroup
#sinceitisadifferentconsumergroup,itreplaymessagesthatwere
#publishedearlieraswellastheywerenotcommittedbyanyother
#consumerinconsumergroup'test1'
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test1--from-beginning
HelloKafka!
HelloHelpingHands!
KafkaWorks!
HelloApacheKafka!
HelloKafkaEvent!
#startingaconsumerwithoutthe'from-beginning'flag
#waitsfornewmessagesonlyanddoesnotreplaypreviousmessages
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test1
#startingaconsumerwithoutthe'from-beginning'flag
#waitsfornewmessagesonlyanddoesnotreplaypreviousmessages
#evenforthenewconsumergroup'test2'
%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group
test2
Kafkaconsumerscandecidetheoffsetfromwheretheywishtostartconsumingthemessages.Intheprecedingexample,theconsumerwasstartedwitha--from-beginningflagthattellstheconsumertostartconsumingfromthefirstoffset,andthatiswhyeverytimetheconsumerisstartedwiththisflaginanewConsumerGroup,itwillreplaythereceivedmessagesineachConsumerGroupfromthebeginning,asshownintheprecedingexample.ThereplayofmessagesandoffsetretentionalsodependsontheretentionperiodsetusingKafkaserverproperties.
TheApacheKafkaconfiguration(https://kafka.apache.org/documentation/#configuration)pageprovidesalistofconfigurationparametersforApacheKafkaservers.
UsingKafkaforHelpingHands
TheHelpingHandsapplicationusesApacheKafkatoimplementtheobservermodelandsendasynchronouseventsamongmicroservices.ItisalsousedasaneventstoretocaptureallthestatechangeeventsgeneratedfrommicroservicesthatareconsumedbytheLookupservicetobuildaconsolidatedviewtoserverlookuprequests.TheAlertmicroserviceoftheHelpingHandsapplicationalsoreceivesthealerteventsviatheKafkatopicandsendsanemailasynchronously.
ApacheKafkaincludesfivecoreAPIs:
TheProducerAPIallowsapplicationstopublishstreamsofeventstooneormoretopicsTheConsumerAPIallowsapplicationstoconsumepublishedeventsfromoneormoretopicsTheStreamsAPIallowstransformingstreamsfrominputtopicsandpublishtheresultstooutputtopicsTheConnectAPIallowssupportforvariousinputandoutputsourcestocaptureanddumpstreamofeventsTheAdminClientAPIallowstopicsandservermanagementalongwithotherKafkamanagementoperations
TointegrateKafkawiththeHelpingHandsapplication,theProducerandConsumerAPIswillonlyberequiredtoproducestatechangeeventsandconsumethem.Therequiredtopicscanbecreatedexternallyusingkafka-topics.shscript,asshownintheprevioussection.ToincludetheproducerandconsumerclientAPIs,addtheprojectdependencyofkafka-clients,asshowninthefollowingexample,totheproject.cljfileofthecorrespondingmicroserviceproject:
(defprojecthelping-hands-alert"0.0.1-SNAPSHOT"
:description"HelpingHandsAlertApplication"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
[io.pedestal/pedestal.jetty"0.5.3"]
;;DatomicFreeEdition
[com.datomic/datomic-free"0.9.5561.62"]
;;Omniconf
[com.grammarly/omniconf"0.2.7"]
;;Mount
[mount"0.1.11"]
;;postalforemailalerts
[com.draines/postal"2.0.2"]
;;kafkaclients
[org.apache.kafka/kafka-clients"1.0.0"]
;;logger
[org.clojure/tools.logging"0.4.0"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-
api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
:min-lein-version"2.0.0"
:source-paths["src/clj"]
:java-source-paths["src/jvm"]
:test-paths["test/clj""test/jvm"]
...
:main^{:skip-aottrue}helping-hands.alert.server)
UsingKafkaAPIsAlthoughClojurehasKafkawrappers(https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Clojure)available,thisbookfocusesonusingApacheKafkaJavaAPIsdirectlyinsteadofaClojurewrapper.TocreateaKafkaconsumer,importtherequiredJavaAPIs,asshownhere:
(nshelping-hands.alert.channel
"InitializesHelpingHandsAlertChannelConsumer"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[clojure.tools.logging:aslog]
[helping-hands.alert.config:asconf]
[postal.core:aspostal])
(:import[java.utilCollectionsProperties]
[org.apache.kafka.common.serialization
LongDeserializerStringDeserializer]
[org.apache.kafka.clients.consumer
ConsumerConsumerConfigKafkaConsumer]))
Next,defineafunctioncreate-kafka-consumerthatinitializesaKafkaconsumerandreturnsit,asshownhere:
(defncreate-kafka-consumer
"CreatesanewKafkaConsumer"
[]
(let[props(doto(Properties.)
(.putAll(conf/get-config[:kafka]))
(.putConsumerConfig/KEY_DESERIALIZER_CLASS_CONFIG
(.getNameLongDeserializer))
(.putConsumerConfig/VALUE_DESERIALIZER_CLASS_CONFIG
(.getNameStringDeserializer)))
consumer(KafkaConsumer.props)
_(.subscribeconsumer(Collections/singletonList
(get(conf/get-config[:kafka])"topic")))]
consumer))
TheKafkaconsumerrequiresasetofconfigurationstoconnecttotheKafkaserverandstartconsumingthepublishedmessages.ThisconfigurationcanberetrievedviaOmniconfconfiguration,asdiscussedinthepreviouschapter.Tosettherequiredconfigurationparameter,first,definetheconfigurationparameterforOmniconftopick,asshownhere:
(cfg/define
{:conf{:type:file
:requiredtrue
:verifieromniconf.core/verify-file-exists
:description"MECBOTconfigurationfile"}
:kafka{:type:edn
:default{"bootstrap.servers""localhost:9092"
"group.id""alerts"
"topic""hh_alerts"}
:description"KafkaConsumerConfiguration"}})
Thebootstrap.serversparametercanhavemorethanoneKafkaserverdefinedasacomma-separatedvaluewiththeformatofhost:port.Aspertheprecedingconfiguration,itusesasingleKafkaservertoconnecttothatisrunninglocallyonport9092.Also,ittakesasinputagroup.idthatspecifiestheConsumerGroupfortheconsumerandatopictowhichitsubscribesformessages.Oncetheconsumeriscreated,itcanbeusedtolistenformessagesontheconfiguredtopic,asshownhere:
(defncapture-records
"Consumetherecordsusinggivenconsumer"
[consumerresult]
(whiletrue
(doseq[record(.pollconsumer1000)]
(swap!resultconjrecord))
(Thread/sleep5000)))
Thecapture-recordsfunctiontakesasinputaconsumerthatiscreatedbythecreate-kafka-consumerfunction,asdefinedearlier.Thisfunctionisasamplefunctionthatalsotakesanatom(https://clojuredocs.org/clojure.core/atom)asasecondparameterthatjustcapturesthereceivedmessagesthatcanbereferredtolater.Totakealookatthereceivedmessages,theycanalsobeloggedorpassedontoafunctionasaparametertotakefurtheractiononit.Also,notethatthisfunctionhasanever-endingwhileloop,sothismustbecalledonathreadotherthantheapplicationexecutionthreadtomakesurethattheapplicationexecutionthreadisnotblocked.Totestthefunction,usetheREPL,asshowninthefollowingexample,thatinitializestheconsumerthatconnectstothesameKafkaserverthatwasstartedatthecommandlineusingthekafka-server-start.shscript:
;;initializetherequirednamespaces
helping-hands.alert.server>(require'[helping-hands.alert.channel:aschannel])
nil
helping-hands.alert.server>(require'[helping-hands.alert.config:asconf])
nil
;;initializetheconfigurationforOmniconftopick
helping-hands.alert.server>(conf/init-config{:cli-args[]:quit-on-errortrue})
Omniconfconfiguration:
{:conf#object[java.io.File0x5fa5f9a0"config/conf.edn"],
:kafka
{"bootstrap.servers""localhost:9092",
"group.id""alerts",
"topic""hh_alerts"}}
nil
;;createaconsumer
helping-hands.alert.server>(defconsumer(channel/create-kafka-consumer))
#'helping-hands.alert.server/consumer
;;createanatomtocollecttheresponses
helping-hands.alert.server>(defrecords(atom[]))
#'helping-hands.alert.server/records
;;waitforrecords
helping-hands.alert.server>(channel/consume-recordsconsumerrecords)
Oncetheconsumerislisteningformessages,ontheterminal,startanewproducerforthesamehh_alertstopicthattheconsumerislisteningtousingthekafka-console-producer.shscript,asshowninthefollowingexample.Notethatwedidn'tcreatethetopicbutitisstillavailable.ThereasonisthedefaultconfigurationofKafkaallowsittoauto-createthetopicifitdoesnotexist.Theothertopic,__consumer_offsets,isusedbyKafkatomanagethecommittedoffsets:
%bin/kafka-topics.sh--list--zookeeperlocalhost:2181
__consumer_offsets
hh_alerts
test
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topichh_alerts
>SampleAlert
>SampleAlert1
Publishacoupleofmessagesfromtheproducer,asshownintheprecedingexample,andchecktherecordsatominREPL.Itshouldhavereceivedthemessagesasfollows:
;;C-c-binCIDERbreakstheexecutiontogivebackthecontroltoREPL
;;lookupthepublishedmessages
helping-hands.alert.server>(pprint(map#(.value%)@records))
("SampleAlert""SampleAlert1")
nil
helping-hands.alert.server>
InitializingKafkawithMountInthepreviousimplementation,theKafkaconsumerwascreatedbycallingthecreate-kafka-consumerfunction.Insteadofcreatingaconsumerbycallingthefunctionatruntime,itcanbecreatedandmanagedusingMount,asdiscussedinthepreviouschapter.TouseMount,makesurethattheMountdependencyisaddedtotheproject.cljfile,asshownhere:
(defprojecthelping-hands-alert"0.0.1-SNAPSHOT"
:description"HelpingHandsAlertApplication"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
[io.pedestal/pedestal.jetty"0.5.3"]
;;DatomicFreeEdition
[com.datomic/datomic-free"0.9.5561.62"]
;;Omniconf
[com.grammarly/omniconf"0.2.7"]
;;Mount
[mount"0.1.11"]
;;postalforemailalerts
[com.draines/postal"2.0.2"]
;;kafkaclients
[org.apache.kafka/kafka-clients"1.0.0"]
;;logger
[org.clojure/tools.logging"0.4.0"]
[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-
api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
:min-lein-version"2.0.0"
:source-paths["src/clj"]
:java-source-paths["src/jvm"]
:test-paths["test/clj""test/jvm"]
...
:main^{:skip-aottrue}helping-hands.alert.server)
Next,asshowninthefollowingexample,createanamespaceanddefinethestateoftheKafkaconsumerthatcreatesaKafkaconsumertobeusedbytheAlertsmicroserviceandclosesitwhentheserviceisshutdown:
(nshelping-hands.alert.state
"InitializesStateforAlertService"
(:require[mount.core:refer[defstate]:asmount]
[helping-hands.alert.channel:asc]))
(defstatealert-consumer
:start(c/create-kafka-consumer)
:stop(.closealert-consumer))
Next,addthestartupandshutdownhooksforMountattheapplicationentrypoints,asshowninthefollowingexamplefortheAlertsmicroservice:
(nshelping-hands.alert.server
(:gen-class);for-mainmethodinuberjar
(:require[io.pedestal.http:asserver]
[io.pedestal.http.route:asroute]
[mount.core:asmount]
[helping-hands.alert.config:ascfg]
[helping-hands.alert.service:asservice]))
...
(defnrun-dev
"Theentry-pointfor'leinrun-dev'"
[&args]
(println"\nCreatingyour[DEV]server...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(->service/service;;startwithproductionconfiguration
...)
;;Wireupinterceptorchains
server/default-interceptors
server/dev-interceptors
server/create-server
server/start))
(defn-main
"Theentry-pointfor'leinrun'"
[&args]
(println"\nCreatingyourserver...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(server/startrunnable-service))
NotethatOmniconfconfigurationisinitializedbeforeMountisstartedtomakesurethattheconsumerisabletopicktheconfigurationparametersthatarereadbyOmniconf.OnceMountisinitialized,thekafka-consumerstatecanbeuseddirectlyacrossthenamespaces.Forexample,thefollowingREPLsessionshowshowtoinitializetheAlertserviceindevmodeandusethekafka-consumerstatemanagedbyMountthatisstartedwiththedevinstanceoftheservice:
;;startsserviceindevmode
;;alsoinitializesOmniconfand
;;setstheKafkaconsumerstate
helping-hands.alert.server>(defserver(run-dev))
Creatingyour[DEV]server...
Omniconfconfiguration:
{:alert
{:from"admin@helpinghands.com",
:host"smtp.gmail.com",
:port465,
:ssltrue,
:to"alerts@helpinghands.com",
:user"admin@helpinghands.com",
:creds<SECRET>},
:conf#object[java.io.File0x266875c9"config/conf.edn"],
:kafka
{"bootstrap.servers""localhost:9092",
"group.id""alerts",
"topic""hh_alerts"}}
#'helping-hands.alert.server/server
;;refertotheconsumerstatemanagedbymount
helping-hands.alert.server>(require'[helping-hands.alert.state:refer[alert-
consumer]])
nil
;;createaatomtocapturerecords
helping-hands.alert.server>(defrecords(atom[]))
#'helping-hands.alert.server/records
;;lookformessagestocapture
helping-hands.alert.server>(helping-hands.alert.channel/capture-recordsalert-
consumerrecords)
Now,publishacoupleofmessagesfromthecommand-lineproducer,asshownhere:
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topichh_alerts
>Hello
>HiMount!
>
Thesamemessagesarecapturedwithintherecordsatom,asshowninthefollowingexample.Atthetimeofshutdown,Mountwillstoptheconsumerandcleanuptheconnection:
;;C-c-binCIDERbreakstheexecutiontogivebackthecontroltoREPL
;;lookupthepublishedmessages
helping-hands.alert.server>(pprint(map#(.value%)@records))
("Hello""HiMount!")
nil
helping-hands.alert.server>
IntegratingtheAlertServicewithKafka
TheAlertmicroserviceoftheHelpingHandsapplicationreceivesthealertmessagesovertheKafkatopicthatisusedtosendthealertinanemail.Tosendanemailassoonasthemessageisreceived,itcanbeintegratedwithintheloopthatlooksforamessagepublishedbytheproducer,asshownhere:
(defnconsume-records
"Consumetherecordsusinggivenconsumer"
[consumerresult]
(whiletrue
(doseq[record(.pollconsumer1000)]
(try
(let[rmsg(jp/parse-string(.valuerecord))
msg(into{}(filter(compsome?val)
{:from(conf/get-config[:alert:from])
:to(getrmsg"to"(conf/get-config[:alert:to]))
:cc(rmsg"cc")
:subject(rmsg"subject")
:body(rmsg"body")}))
result(postal/send-message
{:host(conf/get-config[:alert:host])
:port(conf/get-config[:alert:port])
:ssl(conf/get-config[:alert:ssl])
:user(conf/get-config[:alert:user])
:pass(conf/get-config[:alert:creds])}
msg)])
(catchExceptione
(log/error"Failedtosendemail"e)))
(swap!resultconjrecord))
(Thread/sleep5000)))
Intheprecedingfunction,oncethemessageisreceived,itisparsedtogettheJSONwiththekeyssuchasto,cc,subject,andbodythatareusedtocreateanemailandsenditusingthePostallibrary(https://github.com/drewr/postal)thatwasdiscussedinpreviouschapters.Alltheexceptionsarecaughtandloggedforreview.Inthiscase,theproducerpublishesastringifiedJSONwiththerequiredkeys,asshownhere:
%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topichh_alerts
>{"to":"admin@helpinghands.com","subject":"UsageAlert","body":"Usagealertexceeded
threshold100kreq/sec"}
UsingAvrofordatatransfer
ThekeysandvaluespublishedonaKafkatopicmusthaveassociatedSerDes(https://en.wikipedia.org/wiki/SerDes).TheexamplesusedintheprevioussectionusedLongDeserializerforkeysandStringDeserializerfortheKafkaConsumer.Similarly,theKafkaProducerforthecorrespondingconsumerwilluseLongSerializerandStringSerializertopublishthekeyandvalue,respectively.SincemicroservicesmaybewritteninanyprogramminglanguageandmayneedtocollaboratewithotherservicesoverKafkatopics,thelanguage-dependentSerDesisnotagoodoption.
Avro(https://avro.apache.org/)isadataserializationformatthatislanguageagnosticandhassupportformostofthewell-knownprogramminglanguages(https://cwiki.apache.org/confluence/display/AVRO/Supported+Languages).Avrohasitsowndeclarativewayofdefiningtheschema(https://avro.apache.org/docs/current/)thatcanbemappedtothebusinessmodeldescribingtheentity.Oncetheschemaisdefined,themessageisencodedagainsttheschemaattheproducerendandthendecodedattheconsumerendusingthesameschema.Asfarastheschemaisaccessibletobothproducerandconsumer,theycancommunicateviaAvromessagesirrespectiveoftheprogramminglanguagetheyareimplementedin.
AvroClojurelibraryabracad(https://github.com/damballa/abracad)isawrapperoverAvroAPIsthatintegrateswellwiththeapplicationswritteninClojure.Touseabracad,includethedependency[com.damballa/abracad"0.4.13"]intheproject.cljfile.Oncethedependenciesareavailable,theSerDesforAvrocanbedefinedasshowninthefollowingexample,andcanbeusedinsteadofStringSerDesbytheKafkaproducerandconsumer:;;adoptedfromfranzy-avroproject;;https://github.com/ymilky/franzy-avro(deftypeKafkaAvroSerializer[schema]
Serializer
(configure[___])
(serialize[__data]
(whendata
(avro/binary-encodedschemadata)))(close[_]))
(deftypeKafkaAvroDeserializer[schema]
Deserializer
(configure[___])
(deserialize[__data]
(whendata
(avro/decodeschemadata)))(close[_]))
(defnkafka-avro-serializer"AvroserializerforApacheKafka.UseforserializingKafkakeysvalues.
Valueswillbeserializedaccordingtotheprovidedschema.
Ifnoschemaisprovided,adefaultEDNschemaisassumed.
Seehttps://avro.apache.org/
Seehttps://github.com/damballa/abracad"
[schema]
(KafkaAvroSerializer.(orschema(aedn/new-schema))))
(defnkafka-avro-deserializer"AvrodeserializerforApacheKafka.
UsefordeserializingKafkakeysandvalues.
Ifnoschemaisprovided,adefaultEDNschemaisassumed.
Seehttps://avro.apache.org/
Seehttps://github.com/damballa/abracad"
[schema]
(KafkaAvroDeserializer.(orschema(aedn/new-schema))))
SummaryInthischapter,welearnedabouttheimportanceofevent-drivenpatternsformicroservicesandhowwecanuseApacheKafkaasamessagebrokertobuildascalableanddurableevent-drivenarchitecture.Event-drivenarchitecturesarescalable,buttheyareincrediblyhardtodebugforissueswithoutbeingmonitoredinrealtime.Inthenextchapter,wewilllearnhowtosecuremicroservicesanddeploytheminproductionwithareal-timemonitoringsystem.
DeployingandMonitoringSecuredMicroservices
"Thesuccessofaproductiondependsontheattentionpaidtodetail."
-DavidO.Selznick
Microservicesmustbedeployedinisolationandmonitoredforusage.Monitoringthecurrentworkloadandprocessingtimealsohelpstotakeadecisiononwhentoscalethemuporscalethemdown.Anotherimportantaspectofmicroservices-basedarchitectureissecurity.Onewaytosecuremicroservicesistoalloweachoneofthemtohavetheirownauthenticationandauthorizationmodule.Thisapproachsoonbecomesaproblem,aseachmicroserviceisdeployedinisolation,anditbecomesincrediblyhardtoagreeoncommonstandardstoauthorizeauser.Also,inthiscase,theownershipofusersandtheirrolesgetsdistributedacrosstheservices.Thischapteraddressessuchissuesandprovidessolutionstosecure,monitor,andscalemicroservices-basedapplications.Inthischapter,youwilllearnthefollowingthings:
HowtoenableauthenticationandauthorizationformicroservicesHowtouseJSONWebToken(JWT)andJSONWebEncryption(JWE)HowtocreateanauthenticationservicethatworkswithJSONWebTokensHowtocaptureauditlogsandruntimemetricsforreal-timemonitoringHowtodeploymicroservicesusingDockercontainersWhyKubernetesisusefulformicroservices-baseddeployments
EnablingauthenticationandauthorizationAuthenticationistheprocessofidentifyingwhotheuseris,whereasauthorizationistheprocessofverifyingwhattheauthenticateduserhasaccessto.Themostcommonwayofachievingauthenticationisbyaskinguserstospecifytheirusernameandpasswordthatcanthenbevalidatedagainstthebackenddatabaseofusercredentials.
Thepasswordsshouldneverbestoredinplaintextinthebackenddatabase.Itisrecommendedtocomputeaone-wayhashofthepasswordandstorethatinstead.Toresetthepassword,thesystemcanjustgeneratearandompassword,storeitshash,andsharetherandompasswordinplaintextwiththeuser.Alternatively,auniqueURLcanbesenttotheusertoresetthepasswordthroughaformthatcanvalidateauser'sidentityviamethodssuchaspresetquestionsandanswersandone-timepassword(OTP).
Authenticatingtheusersisnotenoughforanapplicationiftheapplicationhasmultiplesecurityboundaries.Forexample,anapplicationmayrequireonlycertainuserstosendnotificationthroughthesystemandpreventallothers.Todoso,theapplicationmustcreateasecurityboundaryforitsresourcesthatisoftendefinedusingrolesthathaveoneormorepermissionsthatcanbevalidatedbytheapplicationbeforeallowingaccesstoitsresourcesandfeatures,suchasnotification.Rolesandpermissionsarethekeyfactorsofauthorizationthatallowanapplicationtocreatemultiplesecurityboundariesforitsresources.
IntroducingTokensandJWTInamonolithicenvironment,authenticationandauthorizationarehandledwithinthesameapplicationusingamodulethatvalidatestheincomingrequestsforrequiredauthenticationandauthorizationinformation,asshowninthefollowingdiagram;thismodulealsoallowstheauthorizeduserstodefinetherolesandpermissionsandassignthemtootherusersinthesystemtoallowthemaccesstothesecuredresources:
Monolithicapplicationsmayalsomaintainasessionstoreagainstwhicheachinstanceofthemonolithicapplicationcanvalidatetheincomingrequestanddetermineavalidsessionfortheuser.Often,suchsessioninformationisstoredinacookiethatissenttotheclientasatokenoncetheuserissuccessfullyauthenticated.Thecookieisthenattachedtoeachrequestbytheclientthatcanthenbevalidatedbytheserverforavalidsessionandassociatedrolestodeterminewhethertoallowordisallowaccesstotherequestedresource,asshownintheprecedingdiagram.
Inamicroservices-basedapplication,eachmicroserviceisdeployedinisolationandmustnothavetheresponsibilityofmaintainingaseparateuserdatabaseorsessiondatabase.Moreover,theremustbeastandardwayofauthenticatingandauthorizingtheusersacrossmicroservices.ItisrecommendedtoseparateouttheauthenticationandauthorizationresponsibilityasaseparateAuthservicethatcanowntheuserdatabasetoauthenticateandauthorizeusers.ThisalsohelpsinauthenticatingtheuseronceviaAuthserviceandthenauthorizingthemto
accesstheresourcesandrelatedservicesthroughothermicroservices.
SinceeachmicroservicemayuseitsowntechnologystackandhavenopriorknowledgeofAuthservice,thereshouldbeacommonstandardtovalidatetheauthenticatedusersacrossMicroservices.JSONWebTokens(JWT)isonesuchstandardthatconsistsofaheader,payload,andsignaturethatcanbeissuedasatokentotheuseraftersuccessfulauthentication.Userscanthensendthistokenwitheachrequesttoanymicroservicethatcanthenvalidateitandgrantaccesstotherequestedresources.
JWTcaneitherhavethecontentencryptedorsecuredusingdigitalsignatureormessageauthenticationcodes.JSONWebSignature(JWS)representsthecontentsecuredwithdigitalsignaturesorMessageAuthenticationCodes(MACs),whereasJSONWebEncryption(JWE)representstheencryptedcontentusingJSON-baseddatastructures.Ifthetokenisencrypted,itcanonlybereadwithakeythatwasusedtoencryptthetoken.ToreadaJWEtoken,servicesmustownthekeythatwasusedtoencryptthetoken.Insteadofsharingthekeyacrossmicroservices,itisrecommendedtosendthetokentotheAuthservicedirectlytodecryptthetokenandauthorizetherequestonbehalfoftheservice.ThismayresultinaperformancebottleneckandsinglepointoffailureduetoeachservicetryingtogetAuthservicefirstforauthorization.Thiscanbepreventedbycachingtheprevalidatedtokensateachmicroservicelevelforaconfigurableamountoftimethatcanbedecidedbasedontheexpirytimeofthe
token.
ExpirytimeisanimportantcriteriawhileworkingwithJWTs.JWTswithaverylargeexpirytimemustbeavoided,asthereisnowayfortheapplicationtologouttheuserorinvalidatethetoken.Anissuedtokenremainsvalidunlessanduntilitexpires.Asfarastheuserownsavalidtoken,theyareallowedtogainaccesstotheserviceswiththeissuedtoken.Topreventtheissueoflogout,oneoptionistoletmicroservicesalwaysvalidateatokenwiththeAuthservicethatmaintainsacacheofuserauthorizationdetailsthatarekeptinsyncwiththeuser'srolesandpermissions.EverytimeanAuthservicereceivesatoken,itcanvalidateitagainstthiscache,andifthereischangeinuserrolesoranyotherproperties,itcaninvalidatethetokenthatwillforcetheusertorequestforanewtoken,andthattokenwillnowhavetheupdatedrolesandtheauthorizationdetails.
Formoredetails,refertoJWTRFC-7519(https://tools.ietf.org/html/rfc7519),JWSRFC-7515(https://tools.ietf.org/html/rfc7515),andJWERFC-7516(https://tools.ietf.org/search/rfc7516).
CreatinganAuthserviceforHelpingHands
TheAuthserviceforHelpingHandscanbebuiltusingthesamepedestalprojecttemplateasthatofothermicroservicesofHelpingHands.Inthisexample,itusesJWEtocreateJWTtokensfortheusers.Tostartwith,createanewprojectwiththedirectorystructureasshowninthefollowingexample;itcontainsanewnamespacehelping-hands.auth.jwtthatcontainstheimplementationrelatedtoJWT—therestofthenamespacesareusedasdescribedintheprecedingchapters.
.
├──Capstanfile
├──config
│├──conf.edn
│└──logback.xml
├──Dockerfile
├──project.clj
├──README.md
├──resources
├──src
│├──clj
││└──helping_hands
││└──auth
││├──config.clj
││├──core.clj
││├──jwt.clj
││├──persistence.clj
││├──server.clj
││├──service.clj
││└──state.clj
│└──jvm
└──test
├──clj
│└──helping_hands
│└──auth
│├──core_test.clj
│└──service_test.clj
└──jvm
12directories,14files
UsingaNimbusJOSEJWTlibraryforTokens
TheAuthserviceprojectwilladditionallyuseaNimbus-JOSE-JWTlibrary(https://bitbucket.org/connect2id/nimbus-jose-jwt/wiki/Home)tocreateandvalidateJSONWebTokensandapermissions(https://github.com/tuhlmann/permissions)librarytoauthorizeusersagainstasetofrolesandpermissions.AddtheNimbus-JOSE-JWTandpermissionslibrarydependencies,asshowninthefollowingproject.cljfile:
(defprojecthelping-hands-auth"0.0.1-SNAPSHOT"
:description"HelpingHandsAuthService"
:url"https://www.packtpub.com/application-development/microservices-clojure"
:license{:name"EclipsePublicLicense"
:url"http://www.eclipse.org/legal/epl-v10.html"}
:dependencies[[org.clojure/clojure"1.8.0"]
[io.pedestal/pedestal.service"0.5.3"]
[io.pedestal/pedestal.jetty"0.5.3"]
;;DatomicFreeEdition
[com.datomic/datomic-free"0.9.5561.62"]
;;Omniconf
[com.grammarly/omniconf"0.2.7"]
;;Mount
[mount"0.1.11"]
;;nimbus-joseforJWT
[com.nimbusds/nimbus-jose-jwt"5.4"]
;;usedforrolesandpermissions
[agynamix/permissions"0.2.2-SNAPSHOT"]
;;logger
[org.clojure/tools.logging"0.4.0"]
[ch.qos.logback/logback-classic"1.1.8"
:exclusions[org.slf4j/slf4j-api]]
[org.slf4j/jul-to-slf4j"1.7.22"]
[org.slf4j/jcl-over-slf4j"1.7.22"]
[org.slf4j/log4j-over-slf4j"1.7.22"]]
:min-lein-version"2.0.0"
:source-paths["src/clj"]
:java-source-paths["src/jvm"]
:test-paths["test/clj""test/jvm"]
:resource-paths["config","resources"]
:plugins[[:lein-codox"0.10.3"]
;;CodeCoverage
[:lein-cloverage"1.0.9"]
;;Unittestdocs
[test2junit"1.2.2"]]
:codox{:namespaces:all}
:test2junit-output-dir"target/test-reports"
:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]
[org.clojure/tools.nrepl"0.2.12"]]}
:dev{:aliases
{"run-dev"["trampoline""run""-m"
"helping-hands.auth.server/run-dev"]}
:dependencies
[[io.pedestal/pedestal.service-tools"0.5.3"]]
:resource-paths["config","resources"]
:jvm-opts["-Dconf=config/conf.edn"]}
:uberjar{:aot[helping-hands.auth.server]}
:doc{:dependencies[[codox-theme-rdash"0.1.1"]]
:codox{:metadata{:doc/format:markdown}
:themes[:rdash]}}
:debug{:jvm-opts
["-server"(str"-agentlib:jdwp=transport=dt_socket,"
"server=y,address=8000,suspend=n")]}}
:main^{:skip-aottrue}helping-hands.auth.server)
CreatingasecretkeyforJSONWebEncryptionTostartwiththeimplementationofJWTwithencryptedclaims,firstcreateaget-secretfunctiontogenerateasecretkeyforencryption.Also,addaget-secret-jwkfunctionthatisusedtocreateaJSONWebKey(https://tools.ietf.org/html/rfc7517)usingthesecretkeygeneratedbytheget-secretfunction,asshowninthefollowingcode:
(nshelping-hands.auth.jwt
"JWTImplementationforAuthService"
(:require[cheshire.core:asjp])
(:import[com.nimbusds.joseEncryptionMethod
JWEAlgorithmJWSAlgorithm
JWEDecrypterJWEEncrypter
JWEHeader$BuilderJWEObjectPayload]
[com.nimbusds.jose.crypto
AESDecrypterAESEncrypter]
[com.nimbusds.jose.jwkKeyOperationKeyUse
OctetSequenceKeyOctetSequenceKey$Builder]
[com.nimbusds.jwtJWTClaimsSetJWTClaimsSet$Builder]
[com.nimbusds.jwt.procDefaultJWTClaimsVerifier]
[com.nimbusds.jose.utilBase64URL]
[java.utilDate]
[javax.cryptoKeyGenerator]
[javax.crypto.specSecretKeySpec]))
(def^:conskhash-256"SHA-256")
(defonce^:privatekgen-aes-128
(let[keygen(KeyGenerator/getInstance"AES")
_(.initkeygen128)]
keygen))
(defonce^:privatealg-a128kw
(JWEAlgorithm/A128KW))
(defonce^:privateenc-a128cbc_hs256
(EncryptionMethod/A128CBC_HS256))
(defnget-secret
"Getsthesecretkey"
([](get-secretkgen-aes-128))
([kgen]
;;mustbecreatediffthekeyhasn't
;;beencreaedearlier.Createonceand
;;persistinanexternaldatabase
(.generateKeykgen)))
(defnget-secret-jwk
"GeneratesanewJSONWebKey(JWK)"
[{:keys[khashkgenalg]:asenc-impl}secret]
;;mustbecreatediffthekeyhasn't
;;beencreaedearlier.Createonceand
;;persistinanexternaldatabase
(..(OctetSequenceKey$Builder.secret)
(keyIDFromThumbprint(orkhashkhash-256))
(algorithm(oralgalg-a128kw))
(keyUse(KeyUse/ENCRYPTION))
(build)))
TheprecedingimplementationshowngeneratesakeyusingtheAES128-bitalgorithm.Thesecretkeygeneratedbytheget-secretfunctionmustbegeneratedonlyonceforthelifetimeoftheapplication.Therefore,itisrecommendedtostoreitinanexternaldatabasethatcanbesharedamongtheinstancesoftheAuthserviceonceitisscaledtomorethanoneinstance.
Nimbus-JOSE-JWTalsosupports256-bitalgorithms.For256-bitalgorithmstowork,JREneedsexplicitJavaCryptographyExtension(JCE)UnlimitedStrengthJurisdictionPolicyFiles(http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html).
Theget-secret-jwkfunctiontakesthesecretkeyasoneofitsinputparametersandgeneratesaJWK,asshowninthefollowingREPLsession;JWKconsistsofaKeyType(kty),PublicKeyUse(use),KeyID(kid),KeyValue(k),andAlgorithm(alg)parametersthataredefinedinJWKRFC-7517(https://tools.ietf.org/html/rfc7517):
;;requirethenamespace
helping-hands.auth.server>(require'[helping-hands.auth.jwt:asjwt])
nil
;;createasecretkey
helping-hands.auth.server>(defsecret(jwt/get-secret))
#'helping-hands.auth.server/secret
;;createaJSONWebKey
helping-hands.auth.server>(defjwk(jwt/get-secret-jwk{}secret))
#'helping-hands.auth.server/jwk
;;dumptheJSONobjectofJWK
helping-hands.auth.server>(.toJSONObjectjwk)
{"kty""oct","use""enc","kid""F5UNJYT4A-GpngZwRMYfs8ZuCKsmRGt08Xo_dMQrY5w","k"
"CvTaCBfdEkAlXfuOnW7pnw","alg""A128KW"}
SinceJWKisjustarepresentationofthesecretkeyinaJSONformat,thesecretkeycanberetrievedfromtheJWKusingautilityfunction,enckey->secret,asshowninthefollowingimplementation:
(defnenckey->secret
"ConvertsJSONWebKey(JWK)tothesecretkey"
[{:keys[kkidalg]:asenc-key}]
(..(OctetSequenceKey$Builder.k)
(keyIDkid)
(algorithm(oralgalg-a128kw))
(keyUse(KeyUse/ENCRYPTION))
(build)
(toSecretKey"AES")))
Theenckey->secretfunctiontakesKeyID(kid)andKeyValue(k)asitsinputtocreatethesecretkeythatissameastheoneusedtocreatethesourceJSONWebKey.ThealgparameterisoptionalandfallsbacktothedefaultAES-128algorithmifitisnotspecified.ThefollowingREPLsessionshowshowtocreateasecretkeyfromJWKgeneratedearlierandvalidatethatitalwaysgeneratesthesameJWK:
;;JSONWebKey(JWK)generatedearlier
helping-hands.auth.server>(.toJSONObjectjwk)
{"kty""oct","use""enc","kid""F5UNJYT4A-GpngZwRMYfs8ZuCKsmRGt08Xo_dMQrY5w","k"
"CvTaCBfdEkAlXfuOnW7pnw","alg""A128KW"}
;;extractthesecretkey
helping-hands.auth.server>(defsecret-extracted(jwt/enckey->secret{:k(.getKeyValue
jwk):kid(.getKeyIDjwk)}))
#'helping-hands.auth.server/secret-extracted
;;generateJSONWebKeythatisexactlysameassource
helping-hands.auth.server>(.toJSONObject(jwt/get-secret-jwk{}secret-extracted))
{"kty""oct","use""enc","kid""F5UNJYT4A-GpngZwRMYfs8ZuCKsmRGt08Xo_dMQrY5w","k"
"CvTaCBfdEkAlXfuOnW7pnw","alg""A128KW"}
helping-hands.auth.server>(.toJSONObjectjwk)
{"kty""oct","use""enc","kid""F5UNJYT4A-GpngZwRMYfs8ZuCKsmRGt08Xo_dMQrY5w","k"
"CvTaCBfdEkAlXfuOnW7pnw","alg""A128KW"}
CreatingTokensThenextstepistodefinethefunctionstocreateandreadJWT.SincetheJWTusedfortheHelpingHandsapplicationusesJWEtoencrypttheclaims,itisOKtoaddbothuserIDandrolesinformationwithinthepayloadthatcanbelaterretrievedfromavalidtokentoauthorizetheuser.
Thecreate-tokenandread-tokenfunctionsshowninthefollowingexampleprovideawaytocreateaJSONWebTokenandreadanexistingone,respectively.Thecreate-tokenfunctionusesautilityfunction—create-payload—tocreatetheclaimsetandthepayloadofJWT.ClaimsetsthatarerelevantforthecurrentexampleareissueTimethatdefinestheepochtimeofwhenthistokenwascreated,expirationTimethatsetsthetimebeyondwhichthetokenwillbeconsideredasexpired,anduserandrolescustomclaimsthatstoretheauthenticatedusernameandtherolesassignedtotheuseratthetimeofissuingthetoken.Formoredetailsontheavailableclaimsetoptions,takealookatJWTRFC-7519(https://tools.ietf.org/html/rfc7519).
(defn-create-payload
"CreatesapayloadasJWTClaims"
[{:keys[userroles]:asparams}]
(let[ts(System/currentTimeMillis)
claims(..(JWTClaimsSet$Builder.)
(issuer"Packt")
(subject"HelpingHands")
(audience"https://www.packtpub.com")
(issueTime(Date.ts))
(expirationTime(Date.(+ts120000)))
(claim"user"user)
(claim"roles"roles)
(build))]
(.toJSONObjectclaims)))
(defncreate-token
"Createsanewtokenwiththegivenpayload"
[{:keys[userrolesalgenc]:asparams}secret]
(let[enckey(get-secret-jwkparamssecret)
payload(create-payload{:useruser:rolesroles})
passphrase(JWEObject.
(..(JWEHeader$Builder.
(oralgalg-a128kw)
(orencenc-a128cbc_hs256))
(build))
(Payload.payload))
encrypter(AESEncrypter.enckey)
_(.encryptpassphraseencrypter)]
(.serializepassphrase)))
(defnread-token
"Decryptsthegiventokenwiththesaidalgorithm
ThrowsBadJWTExceptionistokenisinvalidorexpired"
[tokensecret]
(let[passphrase(JWEObject/parsetoken)
decrypter(AESDecrypter.secret)
_(.decryptpassphrasedecrypter)
payload(..passphrasegetPayloadtoString)
claims(JWTClaimsSet/parsepayload)
;;throwsexceptionifthetokenisinvalid
_(.verify(DefaultJWTClaimsVerifier.)claims)]
(jp/parse-stringpayload)))
ThefollowingREPLsessionshowsthestepstocreateandreadatokenandlaterwaitforittogetexpired.Notetheexceptionthrownbythelibrarythatcanbecapturedtomarktheeventoftokenexpiry:
;;generateanewtokenwiththeuserandroles
helping-hands.auth.server>(deftoken(jwt/create-token{:user"hhuser":roles#
{"hh/notify"}}secret))
#'helping-hands.auth.server/token
;;dumpthecompactserializationstring
helping-hands.auth.server>token
"eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.FiAelEg_R8We8xEF2xRxcC908BCoH1nRYvY3nV_jkqYO8JPp-
QukBw.86-
JKq6cYFH2rtFBOXiA6A.Pxz3ZzBGKX2Cd_sjtYdEwKDltzKQiolWSvrjPbLLGL8NlShcWWEIqkd7NL2WcXHukDa6zS4ANIWnee2hNWUraItqZFEY6N_RhXZVVXQvZJsqzeiueBxvxc1fj1LFUKsyR63oOwLd5ZIIT99ItrqaYPM88enMsjchsXYBJ_Tcb-
WR6R_KirmDBxCVjqFcg7OdWjjcKTP4FcUNIQU9G8fSnQ.pfLyW8ggXV8vQnidytJmMw"
;;readthetokenback
helping-hands.auth.server>(pprint(jwt/read-tokentokensecret))
{"sub""HelpingHands",
"aud""https://www.packtpub.com",
"roles"["hh/notify"],
"iss""Packt",
"exp"1515959756,
"iat"1515959636,
"user""hhuser"}
nil
;;waitfor2mins(expirytimeasperimplementation)
;;tokenisnowexpired
helping-hands.auth.server>(pprint(jwt/read-tokentokensecret))
BadJWTExceptionExpiredJWTcom.nimbusds.jwt.proc.DefaultJWTClaimsVerifier.<clinit>
(DefaultJWTClaimsVerifier.java:62)
EnablingusersandrolesforauthorizationIdeally,theAuthservicemustbebackedbyapersistentstoretokeeptheusers,roles,andthesecretkeyfortheapplication.Forthesakeofsimplicityoftheexample,createasamplein-memorydatabaseinthehelping-hands.auth.persistencenamespace,asfollows:
(nshelping-hands.auth.persistence
"PersistenceImplementationforAuthService"
(:require[agynamix.roles:asr]
[cheshire.core:asjp])
(:import[java.securityMessageDigest]))
(defnget-hash
"CreatesaMD5hashofthepassword"
[creds]
(..(MessageDigest/getInstance"MD5")
(digest(.getBytescreds"UTF-8"))))
(defuserdb
;;Usedonyfordemonstration
;;TODOPersistinanexternaldatabase
(atom
{:secretnil
:roles{"hh/superadmin""*"
"hh/admin""hh:*"
"hh/notify"#{"hh:notify""notify/alert"}
"notify/alert"#{"notify:email""notify:sms"}}
:users{"hhuser"{:pwd(get-hash"hhuser")
:roles#{"hh/notify"}}
"hhadmin"{:pwd(get-hash"hhadmin")
:roles#{"hh/admin"}}
"superadmin"{:pwd(get-hash"superadmin")
:roles#{"hh/superadmin"}}}}))
(defnhas-access?
"Checksforrelevantpermission"
[uidperms]
(r/has-permission?
(->@userdb:users(getuid))
:roles:permissionsperms))
(defninit-db
"Initializestherolesforpermissionframework"
[]
(r/init-roles(:roles@userdb))
userdb)
userdbcontainsthesamplein-memorydatabasethatconsistsofthe:secretkeythatisinitializedtoniland:usersand:rolesthatcontaininformationontheusers
androles,respectively.Roledefinitionfollowstheguidelinesofthepermissionlibraryanddefinestherolesandpermissionsaspertheusageinstructions(https://github.com/tuhlmann/permissions#usage)ofthelibrary.Roleshaveaslash,/,intheirnameandpermissionshaveacolon,:,asdefinedintheprecedingroledefinition.Roledefinitionsarerecursive,andonerolecanencapsulatebothrolesandpermissions.
Theinit-dbfunctionisusedtoinitializethedatabaseandtheroledefinitions.Thehas-access?isautilityfunctionthatcanbeusedtovalidatewhetherausercontainsagivensetofpermissionsornot.ThefollowingREPLsessiondescribestheuseofthehas-access?functionwithanexample:
;;requirethepersistencenamespace
helping-hands.auth.server>(require'[helping-hands.auth.persistence:asp])
nil
;;sincethereisnosecretkeydefine,
;;initializethedatabasewithasecret-key
;;ifitdoesnotexist
helping-hands.auth.server>(let[db(p/init-db)]
;;ifkeydoesnotexist,initializeone
;;andupdatethedatabasewith:secretkey
(if-not(:secret@db)
(swap!db#(assoc%:secret(jwt/get-secret)))@db))
{:secret#object[javax.crypto.spec.SecretKeySpec0xebc150b
"javax.crypto.spec.SecretKeySpec@17ce8"],:roles{"hh/superadmin""*","hh/admin"
"hh:*","hh/notify"#{"notify/alert""hh:notify"},"notify/alert"#{"notify:email"
"notify:sms"}},:users{"hhuser"{:pwd#object["[B"0x1b46ced7"[B@1b46ced7"],:roles
#{"hh/notify"}},"hhadmin"{:pwd#object["[B"0x7b9083e6"[B@7b9083e6"],:roles#
{"hh/admin"}},"superadmin"{:pwd#object["[B"0x64083ac1"[B@64083ac1"],:roles#
{"hh/superadmin"}}}}
;;validatethat`hhuser`hasthe``hh:notify``permission
helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:notify"})
true
;;validatepermissionsthatarenotdefined
helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:admin"})
false
;;validatepermissionsthatareobtainedbyotherrolereferences
helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:notify""notify:email"})
true
TheprecedingexampleexplicitlyinitializesthedatabaseatREPLandsetsthesecretkey.Insteadofexplicitlyinitializingthedatabase,itcanbedoneatthestartupitselfusingmount,asdiscussedinChapter9,ConfiguringMicroservices.Toallowmounttoinitializethestateofthedatabasewiththesecretkeyandmakeitavailableforothernamespaces,definethedatabasestateinthehelping-hands.auth.statenamespace,asfollows:
(nshelping-hands.auth.state
"InitializesStateforAuthService"
(:require[mount.core:refer[defstate]:asmount]
[helping-hands.auth.jwt:asjwt]
[helping-hands.auth.persistence:asp]))
(defstateauth-db
:start(let[db(p/init-db)]
;;ifkeydoesnotexist,initializeone
;;andupdatethedatabasewith:secretkey
(if-not(:secret@db)
(swap!db#(assoc%:secret(jwt/get-secret)))@db))
:stopnil)
Next,enablethestartandstopeventsbyaddingmount/startandmount/stopfunctionstotheserverstartupfunctionsinthehelping-hands.auth.servernamespace,asshowninthefollowingexample:
(nshelping-hands.auth.server
(:gen-class);for-mainmethodinuberjar
(:require[io.pedestal.http:asserver]
[io.pedestal.http.route:asroute]
[mount.core:asmount]
[helping-hands.auth.config:ascfg]
[helping-hands.auth.service:asservice]))
;;Thisisanadaptedservicemap,thatcanbestartedandstopped
;;FromtheREPLyoucancallserver/startandserver/stoponthisservice
(defoncerunnable-service(server/create-serverservice/service))
(defnrun-dev
"Theentry-pointfor'leinrun-dev'"
[&args]
(println"\nCreatingyour[DEV]server...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(->service/service;;startwithproductionconfiguration
...
;;Wireupinterceptorchains
server/default-interceptors
server/dev-interceptors
server/create-server
server/start))
(defn-main
"Theentry-pointfor'leinrun'"
[&args]
(println"\nCreatingyourserver...")
;;initializeconfiguration
(cfg/init-config{:cli-argsargs:quit-on-errortrue})
;;initializestate
(mount/start)
;;Addshutdown-hook
(.addShutdownHook
(Runtime/getRuntime)
(Thread.mount/stop))
(server/startrunnable-service))
CreatingAuthAPIsusingPedestalThenextstepistodefineAPIsfortheAuthservicetoauthenticateandauthorizeusers.Addthe/tokensand/tokens/validateroutestothehelping-hands.auth.servicenamespace,asfollows:
(nshelping-hands.auth.service
(:require[helping-hands.auth.core:ascore]
[cheshire.core:asjp]
[io.pedestal.http:ashttp]
[io.pedestal.http.route:asroute]
[io.pedestal.http.body-params:asbody-params]
[io.pedestal.interceptor.chain:aschain]
[ring.util.response:asring-resp]))
;;Defines"/"and"/about"routeswiththeirassociated:gethandlers.
;;Theinterceptorsdefinedaftertheverbmap(e.g.,{:gethome-page}
;;applyto/anditschildren(/about).
(defcommon-interceptors[(body-params/body-params)http/html-body])
;;Tabularroutes
(defroutes#{["/tokens"
:get(conjcommon-interceptors
`core/validate`core/get-token)
:route-name:token-get]
["/tokens/validate"
:post(conjcommon-interceptors
`core/validate`core/validate-token)
:route-name:token-validate]})
;;Seehttp/default-interceptorsforadditionaloptionsyoucanconfigure
(defservice{:env:prod
::http/routesroutes
::http/resource-path"/public"
::http/type:jetty
::http/port8080
;;Optionstopasstothecontainer(Jetty)
::http/container-options{:h2c?true
:h2?false
:ssl?false}})
TheGET/tokensroutelooksforuidandpwdparametersoravalidauthorizationheadertoprocesstherequest.Iftheuidandpwdparametersarespecifiedandtheyarevalid,aJWTtokenisissuedaspartoftheauthorizationheader.IfanexistingJWTisspecifiedasapartoftheauthorizationheaderintherequest,Authservicereturnstheusernameandtherolesassociatedwithit.
ThePOST/tokens/validaterouteexpectsaformparameter—perms—andavalidauthorizationheaderwithJWTtoauthorizetheuseragainstthegiven
permissions.ThisendpointisusedbyothermicroservicesoftheHelpingHandsapplicationtoauthorizetheuseragainstthepermissionsrequiredbythemicroservicestoprovideaccesstotheresourcesthatitmanages.Sincepermissionsandrolesaredefinedasstrings,administratorscaninitializetheAuthdatabasewithalltheexpectedrolesandpermissionsandassignthemtouserstoallowordisallowaccesstoservicesoftheapplication.
Theinterceptorsusedfortheroutesdefinedintheprecedingcodesnippetareimplementedinthehelping-hands.auth.corenamespace,asshowninthefollowingexample;thevalidateinterceptorpreparesthe:tx-dataparameterwithalltheavailablerequestparametersandalsovalidatesthepresenceofeitheruidandpwdoranauthorizationheader—ifoneofthemdoesnotexist,itreturnsaHTTP400BadRequestresponse:
(nshelping-hands.auth.core
"InitializesHelpingHandsAuthService"
(:require[cheshire.core:asjp]
[clojure.string:ass]
[helping-hands.auth.jwt:asjwt]
[helping-hands.auth.persistence:asp]
[helping-hands.auth.state:refer[auth-db]]
[io.pedestal.interceptor.chain:aschain])
(:import[com.nimbusds.jwt.procBadJWTException]
[java.ioIOException]
[java.textParseException]
[java.utilArraysUUID]))
;;--------------------------------
;;ValidationInterceptors
;;--------------------------------
(defn-prepare-valid-context
"Appliesvalidationlogicandreturnstheresultingcontext"
[context]
(let[params(merge(->context:request:form-params)
(->context:request:query-params)
(->context:request:headers)
(if-let[pparams(->context:request:path-params)]
(if(empty?pparams){}pparams)))]
(if(or(and(params:uid)(params:pwd))
(params"authorization"))
(assoccontext:tx-dataparams)
(chain/terminate
(assoccontext
:response{:status400
:body"InvalidCreds/Token"})))))
(defvalidate
{:name::validate
:enter
(fn[context]
(prepare-valid-contextcontext))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Theget-tokeninterceptorlooksforavaliduidandpwd,andissuesaJWTifauthenticationissuccessful.Iftheuidandpwdarenotpresent,itlooksforavalidauthorizationheaderofaBearertypeand,ifthetokenisvalid,itreturnstheauthenticateduserIDandassignedrolesthatareassociatedwiththeuser:
(defn-extract-token
"Extractsuserandrolesmapfromtheauthheader"
[auth]
(select-keys
(jwt/read-token
(second(s/splitauth#"\s+"))(auth-db:secret))
["user""roles"]))
(defget-token
{:name::token-get
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
uid(:uidtx-data)
pwd(:pwdtx-data)
auth(tx-data"authorization")]
(cond
(anduidpwd(Arrays/equals
(->auth-db:users(getuid):pwd)
(p/get-hashpwd)))
(let[token(jwt/create-token
{:roles(->auth-db:users(getuid):roles)
:useruid}(auth-db:secret))]
(assoccontext:response
{:status200
:headers{"authorization"(str"Bearer"token)}}))
(andauth(="Bearer"(->(s/splitauth#"\s+")first)))
(try
(assoccontext:response
{:status200
:body(jp/generate-string(extract-tokenauth))})
(catchBadJWTExceptione
(assoccontext:response
{:status401:body"Tokenexpired"})))
:else(assoccontext:response{:status401}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Theimplementationofthevalidate-tokeninterceptorshowninthefollowingexampleauthorizestheuserassociatedwiththeJWTsentasanauthorization
headerandaCSVofpermissionsspecifiedasthepermsformparameter:
(defvalidate-token
{:name::token-validate
:enter
(fn[context]
(let[tx-data(:tx-datacontext)
auth(tx-data"authorization")
perms(if-let[p(tx-data:perms)]
(into#{}(maps/trim(s/splitp#","))))]
(if(andauth(="Bearer"(->(s/splitauth#"\s+")first)))
(try
(if(p/has-access?((extract-tokenauth)"user")perms)
(assoccontext:response{:status200:body"true"})
(assoccontext:response{:status200:body"false"}))
(catchBadJWTExceptione
(assoccontext:response
{:status401:body"Tokenexpired"}))
(catchParseExceptione
(assoccontext:response
{:status401:body"InvalidJWT"})))
(assoccontext:response{:status401}))))
:error
(fn[contextex-info]
(assoccontext
:response{:status500
:body(.getMessageex-info)}))})
Totesttheroutes,starttheAuthserviceusingtheleinruncommandorstartitwithinaREPLasshowninthefollowingexample;assoonastheapplicationisstarted,mountkicksinandinitializesasecretkeythatisusedtoissuetokensandalsoreadthemforauthorization:
helping-hands.auth.server>(defserver(run-dev))
Creatingyour[DEV]server...
Omniconfconfiguration:
{:conf#object[java.io.File0x979c2d2"config/conf.edn"]}
#'helping-hands.auth.server/server
helping-hands.auth.server>
Oncetheserverisupandrunning,usecURLtotryoutvariousscenarios,asshowninthefollowingexample.Iftherearenoauthenticationheadersorvalidcredentialsspecified,thevalidateinterceptorwillkickinandmarkitasabadrequest,asfollows:
%curl-i"http://localhost:8080/tokens"
HTTP/1.1400BadRequest
Date:Sun,14Jan201820:49:48GMT
...
InvalidCreds/Token
%curl-i-XPOST-d"perms=notify:email""http://localhost:8080/tokens/validate"
HTTP/1.1400BadRequest
Date:Sun,14Jan201820:50:21GMT
...
InvalidCreds/Token
Ifthespecifiedcredentialsareinvalid,itwillthrowaresponsewithHTTP401Unauthorizedstatus,asshowninthefollowingexample:
%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hello"
HTTP/1.1401Unauthorized
Date:Sun,14Jan201820:53:16GMT
...
%curl-i-H"Authorization:Bearerabc"-XPOST-d"perms=notify:email"
"http://localhost:8080/tokens/validate"
HTTP/1.1401Unauthorized
Date:Sun,14Jan201820:55:35GMT
...
InvalidJWT
Iftheparametersarevalid,endpointsworkasexpected,asshownforthehhuseruser:
%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hhuser"
HTTP/1.1200OK
Date:Sun,14Jan201820:59:48GMT
...
Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-
Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-
-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-
XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw
Transfer-Encoding:chunked
%curl-i-H"Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-
Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-
-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-
XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw""http://localhost:8080/tokens"
HTTP/1.1200OK
Date:Sun,14Jan201821:00:11GMT
...
{"user":"hhuser","roles":["hh/notify"]}
%curl-XPOST-i-H"Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-
Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-
-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-
XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:email"
"http://localhost:8080/tokens/validate"
HTTP/1.1200OK
Date:Sun,14Jan201821:00:38GMT
...
true%
%curl-XPOST-i-H"Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-
Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-
-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-
XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:random"
"http://localhost:8080/tokens/validate"
HTTP/1.1200OK
Date:Sun,14Jan201821:00:49GMT
...
false
%curl-XPOST-i-H"Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-
Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-
-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-
XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:email"
"http://localhost:8080/tokens/validate"
HTTP/1.1401Unauthorized
Date:Sun,14Jan201821:03:55GMT
...
Tokenexpired
AuthservicecanbedeployedinisolationandconnectedthroughtheauthinterceptorofrestoftheservicesoftheHelpingHandsapplicationtoauthorizetheusers.Userscanobtainthetokenbycallingthe/tokensendpointofAuthservicedirectlyandusethesametokentoauthenticateandauthorizethemselveswithotherservices.
Buddy(https://github.com/funcool/buddy)isanotherClojurelibrarythathasaBuddySign(https://github.com/funcool/buddy-sign)librarythatcanalsobeusedtogenerateJSONWebTokens.
MonitoringmicroservicesAmicroservices-basedapplicationishighlyflexibleintermsofdeploymentandscaling.Itconsistsofmultipleservicesthatmayhaveoneormoreinstancesrunningonaclusterofmachinesacrossthenetwork.Insuchahighlydistributedandflexibleenvironment,itisofutmostimportancethateachinstanceofamicroserviceismonitoredinrealtimetogetaclearviewofthedeployedservices,theirperformance,andtocaptureissuesofinterestthatmustbereportedassoonastheyoccur.Sinceeachrequesttoamicroservice-basedapplicationmayspanouttooneormorerequestsamongmicroservices,thereshouldbeamechanismtotracktheflowofrequestsandalsolocatetheareasofbottleneckthatcanbeaddressedbyperformingarootcauseanalysisandoftenscalingtheservicesfurthertomeetthedemand.
Oneofthewaystosetupaneffectivemonitoringsystemistocollectallthemetricsacrosstheservicesandmachinesandstoretheminacentralizedrepository,asshownintheprecedingdiagram.Thiscentralizedrepositorycanthensupporttheanalysisofthecapturedmetricsandhelptogeneratealertsfortheeventsofinterestinrealtime.Acentralizedrepositoryalsohelpsinsettingupthereal-timeviewofthesystemtounderstandthebehaviorofeachserviceanddecidewhethertoscaleituporscaleitdown.Tosetupacentralizedrepositoryfortheapplication,themetricsneedtobeeitherpulledfromalltheservicesandphysicalmachinesorpushedtothecentralizedrepositorybythe
servicesrunningonthephysicalmachines.Bothpushandpullmodelsareusefultosetupaneffectivemonitoringsystemthatservesthesourceoftruthforthestateofthesystemaswellastheperformanceoftheenvironmentthatiscrucialforeffectiveutilizationoftheinfrastructureusedbythemicroservices-basedapplication.
Alltheservicesmustberesponsibletopushthemetricsrelatedtothestateoftheapplicationtoacommonchannel,suchasApacheKafka(https://kafka.apache.org/),onacommontopicthatcanthenbeusedtoaggregatealltheapplication-levelmetricsacrosstheservicesandstoretheminacentralizedrepository.Application-levellogsthatarewrittentothefileonthephysicalserversandtheapplication-levelmetricsthatarepublishedviamediumssuchasJMX(http://www.oracle.com/technetwork/articles/java/javamanagement-140525.html)canbepulledbyanexternalcollectorandlaterpushedtothecentralizedstorage.Tomonitortheperformanceoftheinfrastructure,externalcollectorsmustalsocapturethestatsofthephysicalmachine,includingCPUutilization,networkthroughput,diskI/O,andmore,whichcanalsobepushedtothecentralrepositorytogetaholisticviewoftheresourceutilizationacrosstheservicesoftheapplication.
UsingELKStackformonitoringElasticsearch(https://www.elastic.co/products/elasticsearch),Logstash(https://www.elastic.co/products/logstash),andKibana(https://www.elastic.co/products/kibana),oftenreferredtoasELKStackorElasticStack(https://www.elastic.co/elk-stack),providealltherequiredcomponentstosetupareal-timemonitoringinfrastructuretocapture,pull,andpushtheapplicationandmachine-levelmetricsintoacentralizedrepositoryandbuildamonitoringdashboardforreportingandalerts.ThefollowingmonitoringinfrastructurediagramexhibitswhereeachofthecomponentsoftheELKStackfitin.Collectd(https://collectd.org/)andApacheKafka(https://kafka.apache.org/)arenotapartofELKStack,butELKStackprovidesseamlessintegrationwiththeseoutofthebox:
Collectdhelpsincapturingallthemachine-levelstats,includingCPU,memory,disk,andnetwork.ThecaptureddatacanthenbepulledthroughLogstashandpushedintoElasticsearchtoanalyzetheoverallperformanceandutilizationoftheinfrastructureusedbytheservicesoftheapplication.LogstashcanalsounderstandthestandardsetofapplicationlogsandpulltheloggedeventsfromthelogfilesgeneratedonthemachineandpushittotheElasticsearchcluster.LogstashalsointegrateswellwithApacheKafkaandcanbeusedtocapturethe
applicationstateeventspublishedbytheservicesandpushthemdirectlytoElasticsearch.SinceElasticsearchactsasacentralrepositoryforallthelogs,events,andmachinestats,KibanacanbeuseddirectlyontopofElasticsearchtoanalyzethestoredmetricsandbuilddashboardsthatareupdatedinrealtimeasandwheneventsarriveinElasticsearch.Kibanacanalsobeusedtoperformrootcauseanalysisandgeneratealertsfortheintendedrecipients.
ELKStackisusefulformonitoring,butitisnottheonlyoption.ToolssuchasPrometheus(https://prometheus.io/)canalsobeusedformonitoring.Prometheussupportsadimensionaldatamodel,flexiblequerylanguageandefficienttimeseriesdatabasewithin-builtalerting.
SettingupElasticsearch
TosetupElasticsearch,downloadthelatestversionfromtheElasticsearchdownloadpage(https://www.elastic.co/downloads/elasticsearch)andextractit,asshowninthefollowingexample;thisbookusesElasticsearch6.1.1,thatcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/elasticsearch-6-1-1):#downloadelasticsearch6.1.1tar%wgethttps://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.tar.gz--https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.tar.gzResolvingartifacts.elastic.co(artifacts.elastic.co)...184.73.156.41,184.72.218.26,54.235.82.130,...Connectingtoartifacts.elastic.co(artifacts.elastic.co)|184.73.156.41|:443...connected.HTTPrequestsent,awaitingresponse...200OKLength:28462503(27M)[application/x-gzip]Savingto:‘elasticsearch-6.1.1.tar.gz’
elasticsearch-6.1.1.tar.gz100%[==============================>]27.14M2.38MB/sin21s
...(1.30MB/s)-‘elasticsearch-6.1.1.tar.gz’saved[28462503/28462503]
#extractthedownloadedtarball%tar-xvfelasticsearch-6.1.1.tar.gz...
#makesurethatthesedirectoriesarepresent%tree-L1elasticsearch-6.1.1elasticsearch-6.1.1├──bin├──config├──lib
├──LICENSE.txt├──modules├──NOTICE.txt├──plugins└──README.textile
5directories,3files
AlthoughElasticsearchwillrunstraightoutoftheboxwiththebin/elasticsearchcommand,itisrecommendedtoreviewthefollowingimportantconfigurationsandsystemsettingsforaneffectiveElasticsearchcluster.ThesettingsmarkedasESConfigaregenericforallElasticsearchdeployments,whereastheonesmarkedasSystemSettingarefortheLinuxoperatingsystem.Theenvironmentvariable—$ES_HOME—referstotheextractedElasticsearchinstallationfolder,thatis,elasticsearch-6.1.1forthecommandshownintheprecedingcodesnippet.
Type ConfigLocation ConfigParameter Value
SystemSetting
/etc/security/limits.conf memlock unlimited
SystemSetting
/etc/security/limits.conf nofile 65536
SystemSetting
/etc/sysctl.conf vm.overcommit_memory 1
SystemSetting
/etc/sysctl.conf vm.max_map_count 262144
SystemSetting
/etc/fstab Commentswapconfig -
ESJVM $ES_HOME/config/jvm.options -Xmsand-Xmx 8g,16g,
andsoon
Options
ESConfig
$ES_HOME/config/elasticsearch.yml cluster.name <name>
ESConfig
$ES_HOME/config/elasticsearch.yml node.name <name>
ESConfig
$ES_HOME/config/elasticsearch.yml path.data
Oneormore<pathtokeep
indexes>
ESConfig
$ES_HOME/config/elasticsearch.yml path.logs<pathtolog
directory>
ESConfig
$ES_HOME/config/elasticsearch.yml bootstrap.memory_lock true
ESConfig
$ES_HOME/config/elasticsearch.yml network.host <ip_address>
ESConfig
$ES_HOME/config/elasticsearch.yml discovery.zen.ping.unicast.hosts
Oneormore<ip>:<port>
ESConfig
$ES_HOME/config/elasticsearch.yml discovery.zen.minimum_master_nodes<numberof
nodes>
Notethatsomeofthesystemsettingsshownintheprecedingtablemayrequireasystemrestartforthemtotakeintoeffect.Also,settingslikethatofswapspacemustbedoneonlyifElasticsearchistheonlycomponentrunningonthehost
operatingsystem.Onceallthesettingsareinplace,eachElasticsearchnodecanbestartedusingthefollowingcommand;eachnodewilljointheclusteriftheyhavethesameclusternameandareapartofunicasthostslistthatanodeisallowedtojoin:#changetotheextractedelasticsearchdirectory%cdelasticsearch-6.1.1
#startelasticsearch%bin/elasticsearch[2018-01-15T20:46:35,408][INFO][o.e.n.Node][]initializing......[2018-01-15T20:46:36,328][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[aggs-matrix-stats][2018-01-15T20:46:36,329][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[analysis-common][2018-01-15T20:46:36,329][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[ingest-common]...[2018-01-15T20:46:36,330][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[tribe][2018-01-15T20:46:37,491][INFO][o.e.d.DiscoveryModule][W6r6s1z]usingdiscoverytype[zen][2018-01-15T20:46:37,929][INFO][o.e.n.Node]initialized[2018-01-15T20:46:37,930][INFO][o.e.n.Node][W6r6s1z]starting...[2018-01-15T20:46:38,100][INFO][o.e.t.TransportService][W6r6s1z]publish_address{127.0.0.1:9300},bound_addresses{[::1]:9300},{127.0.0.1:9300}[2018-01-15T20:46:41,164][INFO][o.e.c.s.MasterService][W6r6s1z]zen-disco-elected-as-master([0]nodesjoined),reason:new_master{W6r6s1z}{W6r6s1zTQ96ULo2wq9Tm3w}{ykEtBVl9Sy62mkXOFo892g}{127.0.0.1}{127.0.0.1:9300}...[2018-01-15T20:46:41,196][INFO][o.e.n.Node][W6r6s1z]started[2018-01-15T20:46:41,239][INFO][o.e.g.GatewayService][W6r6s1z]recovered[0]indicesintocluster_state
Notethatthefirstnodethatisstartedisautomaticallyelectedasamasteroftheclustertowhichothernodescanjoin:%curlhttp://localhost:9200
{"name":"W6r6s1z","cluster_name":"elasticsearch","cluster_uuid":"g33pKv6XRTaj_yMJLliL0Q","version":{"number":"6.1.1","build_hash":"bd92e7f","build_date":"2017-12-17T20:23:25.338Z","build_snapshot":false,"lucene_version":"7.1.0","minimum_wire_compatibility_version":"5.6.0","minimum_index_compatibility_version":"5.0.0"},"tagline":"YouKnow,forSearch"}
OncetheElasticsearchnodeisupandrunning,totesttheinstancesendaGETrequesttothedefault9200portonthemachinewhereElasticsearchisrunningusingcURL,asshownintheprecedingexample.Itshouldreturnaresponsestatingtheversionofthenode.Verifythatitistherightversion,thatis,6.1.1,forthisexample.
FormoredetailsontheimportantsettingsofElasticsearchandSystem,takealookattheElasticsearchdocsforImportantSettings(https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html)andSystemSettings(https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html).
TheprecedingconfigurationdiscussesatypicalclusterdeploymentforElasticsearch.ElasticsearchalsoprovidesaconceptofCrossClusterSearch(https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html)thatallowsanynodetoactasafederatedclientacrossmultipleclustersofElasticsearch.
ThestepstakeninthissectionuseElasticsearchtarballtosetupElasticsearchcluster,butElasticsearchprovidesanumberofoptionstosetitupusingbinaries,includingRPMpackage,Debianpackage,MSIpackageforWindows,andaDockerimage.For
moredetails,refertotheinstallationinstructionsathttps://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#install-elasticsearch.
SettingupKibanaKibanaisthevisualizationanddashboardinterfaceforElasticsearch.ItallowsexploringdatausingitsDiscovermodule(https://www.elastic.co/guide/en/kibana/6.1/discover.html)andbuildingreal-timedashboards.Itincludesagoodnumberofvisualizationoptions(https://www.elastic.co/guide/en/kibana/current/visualize.html)thatallowuserstoaggregatedatastoredwithinElasticsearchandvisualizethemusingvariouscharts,suchasline,bar,area,maps,andtagclouds.KibanacanbeusedtobuildthemonitoringdashboardusingthevariousmetricsthatarecapturedwithinElasticsearch.
TosetupKibana,downloadthelatestversionfromtheKibanadownloadspage(https://www.elastic.co/downloads/kibana)andextractitasshowninthefollowingexample;thisbookusesKibana6.1.1,whichcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/kibana-6-1-1):
#downloadKibana6.1.1tar
%wgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.1.1-linux-x86_64.tar.gz
--https://artifacts.elastic.co/downloads/kibana/kibana-6.1.1-linux-x86_64.tar.gz
Resolvingartifacts.elastic.co(artifacts.elastic.co)...54.225.188.6,23.21.118.61,
54.235.82.130,...
Connectingtoartifacts.elastic.co(artifacts.elastic.co)|54.225.188.6|:443...
connected.
HTTPrequestsent,awaitingresponse...200OK
Length:64664051(62M)[application/x-gzip]
Savingto:‘kibana-6.1.1-linux-x86_64.tar.gz’
kibana-6.1.1-linux-x86_64.tar.gz100%[==============================>]61.67M2.06MB/s
in53s
...(1.10MB/s)-‘kibana-6.1.1-linux-x86_64.tar.gz’saved[64664051/64664051]
#extractthedownloadedtarball
%tar-xvfkibana-6.1.1-linux-x86_64.tar.gz
...
#makesurethatthesedirectoriesarepresent
%tree-L1kibana-6.1.1-linux-x86_64
kibana-6.1.1-linux-x86_64
├──bin
├──config
├──data
├──LICENSE.txt
├──node
├──node_modules
├──NOTICE.txt
├──optimize
├──package.json
├──plugins
├──README.txt
├──src
├──ui_framework
└──webpackShims
10directories,4files
Next,configureKibanainstancebysettingthefollowingconfigurationparametersinthe$KIBANA_HOME/config/kibana.ymlfile.Theenvironmentvariable—$KIBANA_HOME—referstotheextractedKibanainstallationfolder,thatis,kibana-6.1.1-linux-x86_64forthecommandshownintheprecedingcodesnippet.
ConfigParameter Value Description
server.port 5601 Defaultserver.host <host_ip> KibanabindstothisIPaddress
server.basePath <base_prefix_URL>Shouldnotendwith`/`.UsedtomapproxyURLprefix,ifany
server.name <name> Displayname
elasticsearch.urlhttp://<es_host>:
<es_port>
ElasticsearchURLtoconnectto;defaultportis9200
kibana.index <name>IndexnameascreatedbyKibanainElasticsearch
pid.file <path_to_pid_file> PIDfilelocationlogging.dest <path_to_log_file> FiletowriteKibanalogs
Onceallthesettingsareinplace,startKibanausingthecommandshowninthefollowingexample;ensurethatElasticsearchisalreadyrunningandaccessibletotheKibananodeontheconfiguredelasticsearch.urlsetting,asdescribedintheprecedingtable.
#changetoextractedkibanadirectory
%cdkibana-6.1.1-linux-x86_64
#startkibana
%bin/kibana
log[16:41:31.409][info][status][plugin:kibana@6.1.1]Statuschangedfrom
uninitializedtogreen-Ready
log[16:41:31.443][info][status][plugin:elasticsearch@6.1.1]Statuschangedfrom
uninitializedtoyellow-WaitingforElasticsearch
log[16:41:31.461][info][status][plugin:console@6.1.1]Statuschangedfrom
uninitializedtogreen-Ready
log[16:41:31.484][info][status][plugin:metrics@6.1.1]Statuschangedfrom
uninitializedtogreen-Ready
log[16:41:31.646][info][status][plugin:timelion@6.1.1]Statuschangedfrom
uninitializedtogreen-Ready
log[16:41:31.650][info][listening]Serverrunningathttp://localhost:5601
log[16:41:31.668][info][status][plugin:elasticsearch@6.1.1]Statuschangedfrom
yellowtogreen-Ready
OnceKibanaisupandrunning,opentheURLhttp://localhost:5601,asloggedintheprecedingmessagestoopenKibanainterface,asshowninthefollowingscreenshot;itshouldshowtheKibanahomepagewithoptionstovisualizeandexploredata:
ThecurrentconfigurationofKibanaallowsuserstoexploreElasticsearchinaclosednetwork.SinceKibanaprovidesfullcontrolovertheunderlyingElasticsearchclusteranddatastoredwithinit,itisrecommendedthatyouenableSSLandalsotheload-balancingoptionasdefinedintheproductionconfiguration(https://www.elastic.co/guide/en/kibana/current/production.html)foruserstoconnectandaccessdashboards.
KibananotonlyallowsuserstoexploredatasetsalreadystoredwithinElasticsearchbutalsosupportsloadingdatasetsdirectlyintoElasticsearchviaitsuserinterface.Tolearnmoreaboutthis,
followtheLoadingSampleDatatutorialofKibana(https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html).
SettingupLogstash
Logstashallowsuserstocollect,parse,andtransformlogmessages.Itsupportsanumberofinput(https://www.elastic.co/guide/en/logstash/current/input-plugins.html)andoutput(https://www.elastic.co/guide/en/logstash/current/output-plugins.html)pluginsthatallowLogstashtocollectlogsfromavarietyofsources,parseandtransformthem,andthenwritetheresultstooneofthesupportedplugins.TosetupLogstash,downloadthelatestversionfromtheLogstashdownloadspage(https://www.elastic.co/downloads/logstash)andextractitasshowninthefollowingcodesnippet—thisbookusesLogstash6.1.1,whichcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/logstash-6-1-1):
#downloadLogstash6.1.1tar
%wgethttps://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz
--https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz
Resolvingartifacts.elastic.co(artifacts.elastic.co)...23.21.118.61,54.243.108.41,
184.72.218.26,...
Connectingtoartifacts.elastic.co(artifacts.elastic.co)|23.21.118.61|:443...
connected.
HTTPrequestsent,awaitingresponse...200OK
Length:109795895(105M)[application/x-gzip]
Savingto:‘logstash-6.1.1.tar.gz’
logstash-6.1.1.tar.gz100%[=================================>]104.71M1.09MB/sin81s
...(1.30MB/s)-‘logstash-6.1.1.tar.gz’saved[109795895/109795895]
#extractthedownloadedtarball
%tar-xvflogstash-6.1.1.tar.gz
...
#makesurethatthesedirectoriesarepresent
%tree-L1logstash-6.1.1
logstash-6.1.1
├──bin
├──config
├──CONTRIBUTORS
├──data
├──Gemfile
├──Gemfile.lock
├──lib
├──LICENSE
├──logstash-core
├──logstash-core-plugin-api
├──modules
├──NOTICE.TXT
├──tools
└──vendor
9directories,5files
ThefollowingtableliststheprimaryconfigurationsettingsthatarerequiredforLogstashandmustbeaddedtothe$LOGSTASH_HOME/config/logstash.ymlfile;theenvironmentvariable—$LOGSTASH_HOME—referstotheextractedLogstashinstallationfolder,thatis,logstash-6.1.1forthecommandshownintheprecedingcodesnippet.
ConfigParameter Value Description
node.name <name>
Nodenametoidentifythenodefromoutputinterface.Goodtohaveasahostname.
path.data <path_to_data>
OneormorepathswhereLogstashanditspluginkeepsthedataforanypersistenceneeds.
pipeline.workers1,2,3,4,andsoon
Workerstoexecutefilterandoutputstages.Ifdeployedonaseparatemachine,setthistonumberofCPUcores.
pipeline.output.workers1,2,andsoon
Numberofworkerstouseperoutputplugininstance.Defaultsto1.
path.config <path_to_config>Locationtofetchpipelineconfigurationformainpipeline.
http.host <host_ip>BindaddressformetricsRESTendpoint.
http.port <host_port>
BindportforthemetricsRESTendpoint.Alsoacceptsranges,suchas(9600-9700),topickthefirstavailableport.
path.logs <path_to_logs> PathwhereLogstashwillkeepthelogs.
Theprecedingtablelistsonlytheprimaryconfigurationparameters;formoredetailsandallthesupportedconfigurationparameters,refertoLogstashSettingsFileguide(https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html).Onceallthesettingsareinplace,testasampleLogstashpipelinethatusesstdin(https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stdin.html)asitsinputplugintoreceivemessagesandstdout(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html)asitsoutputplugintoemitthereceivedmessages,asfollows:
#changetoextractedlogstashdirectory
%cdlogstash-6.1.1
#startlogstashpipelinebyspecifyingtheconfiguration
#atcommandlineusingthe-eflag
%bin/logstash-e'input{stdin{}}output{stdout{}}'
SendingLogstash'slogstologstash-6.1.1/logswhichisnowconfiguredvia
log4j2.properties
[2018-01-15T23:11:02,245][INFO][logstash.modules.scaffold]Initializingmodule
{:module_name=>"netflow",:directory=>"logstash-6.1.1/modules/netflow/configuration"}
[2018-01-15T23:11:02,257][INFO][logstash.modules.scaffold]Initializingmodule
...
[2018-01-15T23:11:03,872][INFO][logstash.runner]StartingLogstash
{"logstash.version"=>"6.1.1"}
[2018-01-15T23:11:04,415][INFO][logstash.agent]SuccessfullystartedLogstashAPI
endpoint{:port=>9600}
[2018-01-15T23:11:06,212][INFO][logstash.pipeline]Startingpipeline
{:pipeline_id=>"main","pipeline.workers"=>4,"pipeline.batch.size"=>125,
"pipeline.batch.delay"=>5,"pipeline.max_inflight"=>500,:thread=>"#<Thread:0x77cbc3e6
run>"}
[2018-01-15T23:11:06,305][INFO][logstash.pipeline]Pipelinestarted
{"pipeline.id"=>"main"}
Thestdinpluginisnowwaitingforinput:
[2018-01-15T23:11:06,413][INFO][logstash.agent]Pipelinesrunning{:count=>1,
:pipelines=>["main"]}
helloworld
2018-01-15T17:41:19.900Zfc-machinehelloworld
2018-01-15T17:41:21.707Zfc-machine
HelloLogstash!
2018-01-15T17:41:28.566Zfc-machineHelloLogstash!
HelloELK!
2018-01-15T17:41:32.255Zfc-machineHelloELK!
HelloHelpingHandsEvents!
2018-01-15T17:41:38.685Zfc-machineHelloHelpingHandsEvents!
Logstashmaytakeafewsecondstostartthepipeline,sowaitforthePipelinerunningmessagetobelogged.Oncethepipelineisrunning,typeamessageintheconsole,andLogstashwillechothesameontheconsoleappendedwiththecurrenttimestampandhostname.Thisisaverysimplepipelinethatdoesnotdoanytransformation,butLogstashallowstransformationstobeappliedonthereceivedmessagesbeforetheyareemittedtothesink.Similartothebasicpipelineshownintheprecedingtest,theLogstashpipelineconfigurationiscreatedforeachpipelinethatisrequiredtobeexecutedbyLogstashtocapture
logs,events,anddata,andstoretheminthetargetsinks.
LogstashpluginsareimplementedprimarilyinRuby(https://www.ruby-lang.org/en/).ThatiswhyallthejobconfigurationfilesforLogstashandtransformationconstructsusesyntaxofRubylanguage.
UsingELKStackwithCollectdCollectdisadaemonthatcanbeconfiguredtocollectmetricsfromvarioussourceplugins,suchasLogstash.AscomparedtoLogstash,Collectdisverylightweightandportable,butitdoesnotgenerategraphs.ItcanwritetoRRDfiles,though,thatneedaRRDTool(https://en.wikipedia.org/wiki/RRDtool)toreadthemandgenerategraphstovisualizetheloggeddata.Ontheotherhand,sinceCollectdiswritteninCprogramminglanguage(https://en.wikipedia.org/wiki/C_(programming_language)),itisalsopossibletouseittocollectmetricsfromembeddedsystemsaswell.
Collectdneedstobebuiltfromsource.First,downloadtheCollectd5.8.0versionandextractthesame:
#downloadCollectd5.8.0tar
%wgethttps://storage.googleapis.com/collectd-tarballs/collectd-5.8.0.tar.bz2
--https://storage.googleapis.com/collectd-tarballs/collectd-5.8.0.tar.bz2
Resolvingstorage.googleapis.com(storage.googleapis.com)...172.217.26.208,
2404:6800:4007:802::2010
Connectingtostorage.googleapis.com(storage.googleapis.com)|172.217.26.208|:443...
connected.
HTTPrequestsent,awaitingresponse...200OK
Length:1686017(1.6M)[application/x-bzip]
Savingto:‘collectd-5.8.0.tar.bz2’
collectd-5.8.0.tar.bz2100%[===============================>]1.61M2.67MB/sin0.6s
...(2.67MB/s)-‘collectd-5.8.0.tar.bz2’saved[1686017/1686017]
#extractthedownloadedtarball
%tar-xvfcollectd-5.8.0.tar.bz2
...
#makesurethatthesedirectoriesarepresent
%tree-L1collectd-5.8.0
collectd-5.8.0
├──aclocal.m4
├──AUTHORS
├──bindings
├──build-aux
├──ChangeLog
├──configure
├──configure.ac
├──contrib
├──COPYING
├──m4
├──Makefile.am
├──Makefile.in
├──proto
├──README
├──src
├──testwrapper.sh
└──version-gen.sh
6directories,11files
Next,installCollectdtoabuilddirectory,asshowninthefollowingexample.Incasetheconfigurescriptrequestsformissingdependencies,installthembeforecontinuingthesetupaspertheFirststepswiki(https://collectd.org/wiki/index.php/First_steps)ofCollectd:
#changetoextractedcollectddirectory
%cdcollectd-5.8.0
#configurethetargetbuilddirectory
#givethefullyqualifiedpathasprefix
#$COLLECTD_HOMEpointstocollectd-5.8.0directory
%./configure--prefix=$COLLECTD_HOME/build
checkingbuildsystemtype...x86_64-unknown-linux-gnu
checkinghostsystemtype...x86_64-unknown-linux-gnu
checkinghowtoprintstrings...printf
checkingforgcc...gcc
checkingwhethertheCcompilerworks...yes
checkingforCcompilerdefaultoutputfilename...a.out
checkingforsuffixofexecutables...
...
#installcollectd
%sudomakeallinstall
...
#verifythebuilddirectories
%tree-L1build
build
├──bin
├──etc
├──include
├──lib
├──man
├──sbin
├──share
└──var
8directories,0files
#owntheentirecollectddirectory
#replace<user>withyourusername
%sudochown-R<user>:<user>.
OnceCollectdisinstalled,thenextstepistoupdatethebuild/etc/collectd.conffilewiththedesiredconfigurationsandplugins.Thefollowingisasamplecollectd.conffiletoenablecpu,df,interface,network,memory,syslog,load,andswapplugins;formoredetailsontheavailablepluginsandtheirconfiguration,refertoCollectdTableofPlugins(https://collectd.org/wiki/index.php/Table_of_Plugins).
#BaseConfiguration
#replaceallpathsbelowwithfullyqualified
#pathtotheextractedcollectd-5.8.0directory
Hostname"helpinghands.com"
BaseDir"/collectd-5.8.0/build/var/lib/collectd"
PIDFile"/collectd-5.8.0/build/var/run/collectd.pid"
PluginDir"/collectd-5.8.0/build/lib/collectd"
TypesDB"/collectd-5.8.0/build/share/collectd/types.db"
CollectInternalStatstrue
#Syslog
LoadPluginsyslog
<Pluginsyslog>
LogLevelinfo
</Plugin>
#Otherplug-ins
LoadPlugincpu
LoadPlugindf
LoadPlugindisk
LoadPlugininterface
LoadPluginload
LoadPluginmemory
LoadPluginnetwork
LoadPluginswap
#Plug-inConfig
<Plugincpu>
ReportByCputrue
ReportByStatetrue
ValuesPercentagefalse
</Plugin>
#replacedeviceandmountpoint
#withthedevicetobemonitored
#asshownbydfcommand
<Plugindf>
Device"/dev/sda9"
MountPoint"/home"
FSType"ext4"
IgnoreSelectedfalse
ReportByDevicefalse
ReportInodesfalse
ValuesAbsolutetrue
ValuesPercentagefalse
</Plugin>
<Plugindisk>
Disk"/^[hs]d[a-f][0-9]?$/"
IgnoreSelectedfalse
UseBSDNamefalse
UdevNameAttr"DEVNAME"
</Plugin>
#reportallinterfaceexceptloandsit0
<Plugininterface>
Interface"lo"
Interface"sit0"
IgnoreSelectedtrue
ReportInactivetrue
UniqueNamefalse
</Plugin>
<Pluginload>
ReportRelativetrue
</Plugin>
<Pluginmemory>
ValuesAbsolutetrue
ValuesPercentagefalse
</Plugin>
#sendsmetricstothisporti.e.
#configuredinlogstashtoreceive
#thelogeventstobepublished
<Pluginnetwork>
Server"127.0.0.1""25826"
<Server"127.0.0.1""25826">
</Server>
</Plugin>
<Pluginswap>
ReportByDevicefalse
ReportBytestrue
ValuesAbsolutetrue
ValuesPercentagefalse
</Plugin>
Oncetheconfigurationfileisinplace,startCollectddaemon,asshownhere:
#startcollectddaemonwithsudo
#someplug-insrequiresudoaccess
%sudobuild/sbin/collectd
#makesureitisrunning
%ps-ef|grepcollectd
anuj272081768001:21?00:00:00build/sbin/collectd
...
#verifysyslogtomakesurethatcollectdisup
%tail-f/var/log/syslog
...
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"syslog"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"cpu"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"df"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"disk"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"interface"
successfullyloaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"load"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"memory"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"network"successfully
loaded.
Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"swap"successfully
loaded.
...
Jan1601:40:01localhostcollectd[28726]:Initializationcomplete,enteringread-
loop.
Next,createaLogstashpipelineconfigurationfile,$LOGSTASH_HOME/config/helpinghands.conf,toreceivethedatafromCollectdusingtheCollectdCodec(https://www.elastic.co/guide/en/logstash/current/plugins-codecs-collectd.html)pluginandsendittoElasticsearchusingitsoutputplugin(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html):
input{
udp{
port=>25826
buffer_size=>1452
codec=>collectd{
id=>"helpinghands.com-collectd"
typesdb=>["/collectd-5.8.0/build/share/collectd/types.db"]
}
}
}
output{
elasticsearch{
id=>"helpinghands.com-collectd-es"
hosts=>["127.0.0.1:9200"]
index=>"helpinghands.collectd.instance-%{+YYYY.MM}"
}
}
Next,runtheLogstashpipelinetoreceivedatafromCollectdprocessoverUDP(https://en.wikipedia.org/wiki/User_Datagram_Protocol)andsendittoElasticsearch.Ensurethatthe25826portspecifiedintheUDPconfigurationabovematchestheportofthenetworkpluginofCollectdconfiguration.BeforerunningLogstash,verifythatElasticsearchandCollectdbotharerunning:
#changeto$LOGSTASH_HOMEdirectoryandrunlogstash
%bin/logstash-fconfig/helpinghands.conf
...
SendingLogstash'slogsto/logstash-6.1.1/logswhichisnowconfiguredvia
log4j2.properties
[2018-01-16T02:03:57,028][INFO][logstash.modules.scaffold]Initializingmodule
{:module_name=>"netflow",:directory=>"/logstash-6.1.1/modules/netflow/configuration"}
[2018-01-16T02:03:57,057][INFO][logstash.modules.scaffold]Initializingmodule
{:module_name=>"fb_apache",:directory=>"/logstash-
6.1.1/modules/fb_apache/configuration"}
...
[2018-01-16T02:03:58,410][INFO][logstash.runner]StartingLogstash
{"logstash.version"=>"6.1.1"}
[2018-01-16T02:03:58,935][INFO][logstash.agent]SuccessfullystartedLogstashAPI
endpoint{:port=>9600}
[2018-01-16T02:04:02,412][INFO][logstash.outputs.elasticsearch]Elasticsearchpool
URLsupdated{:changes=>{:removed=>[],:added=>[http://127.0.0.1:9200/]}}
...
[2018-01-16T02:04:03,777][INFO][logstash.outputs.elasticsearch]NewElasticsearch
output{:class=>"LogStash::Outputs::ElasticSearch",:hosts=>["//127.0.0.1:9200"]}
...
[2018-01-16T02:04:03,883][INFO][logstash.pipeline]Pipelinestarted
{"pipeline.id"=>"main"}
[2018-01-16T02:04:03,960][INFO][logstash.inputs.udp]StartingUDPlistener
{:address=>"0.0.0.0:25826"}
[2018-01-16T02:04:03,997][INFO][logstash.agent]Pipelinesrunning{:count=>1,
:pipelines=>["main"]}
[2018-01-16T02:04:04,030][INFO][logstash.inputs.udp]UDPlistenerstarted
{:address=>"0.0.0.0:25826",:receive_buffer_bytes=>"106496",:queue_size=>"2000"}
OnceLogstashstarts,observeElasticsearchlogsthatshowsthatLogstashhascreatedanewindexbasedonthehelpinghands.collectd.instance-%{+YYYY.MM}patternasconfiguredinLogstash'sElasticsearchoutputplugin.Notethattheindex
namewilldifferbasedonthecurrentmonthandyear.Maintainingatime-basedindexpatternisrecommendedforindexesthatstoretimeseriesdatasets.Itnotonlyhelpsinqueryperformancebutalsohelpsinbackupandcleanupbasedonthedataretentionpoliciesofanorganization.ThefollowingarethelogmessagesthatcanbeobservedinElasticsearchlogfilesforsuccessfulcreationoftherequiredindexforthedatacapturedbyLogstashfromCollectd:
[2018-01-16T02:04:12,054][INFO][o.e.c.m.MetaDataCreateIndexService][W6r6s1z]
[helpinghands.collectd.instance-2018.01]creatingindex,cause[auto(bulkapi)],
templates[],shards[5]/[1],mappings[]
[2018-01-16T02:04:15,259][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]
[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]create_mapping[doc]
[2018-01-16T02:04:15,279][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]
[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]update_mapping[doc]
[2018-01-16T02:04:15,577][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]
[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]update_mapping[doc]
[2018-01-16T02:04:15,712][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]
[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]update_mapping[doc]
[2018-01-16T02:04:15,922][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]
[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]update_mapping[doc]
LetthepipelinerunandstorethemachinemetricscapturedviaCollectd–Logstash–Elasticsearchpipeline.Now,opentheKibanainterfaceinthebrowserusingtheURLhttp://localhost:5601andclickontheSetupindexpatternsbuttononthetop-rightcorner.Itwillautomaticallylistthenewlycreatedindexhelpinghands.collectd.instance-2018.01,asshowninthefollowingscreenshot:
Addtheindexpatternhelpinghands.collectd.instance-*,asshownintheprecedingscreenshot,toincludealltheindexescreatedforthemetricscapturedbyCollectd.ClickontheNextstepbuttonontheright-handsideandselectTimefilterfieldnameas@timestamp,asshowninthefollowingscreenshot:
Next,clickontheCreateindexpatternbutton,asshownintheprecedingscreenshot.ItwillshowthelistoffieldsthatKibanawasabletoretrievefromtheElasticsearchindexmapping(https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).Now,clickonDiscoverintheleft-handsidemenu,anditwillshowthereal-timedashboardofallthemessagesbeingcaptured,asshowninthefollowingscreenshot:
Theleft-handsidepaneloftheDiscoverscreenlistsallthefieldsbeingcaptured.Forexample,clickonhostandpluginfieldstoseethehostsbeingmonitoredandallthepluginsforwhichthedatahasbeencapturedbyCollectdandsenttoElasticsearchviatheLogstashpipeline,asshowninthefollowingscreenshot:
Kibanadashboardallowsbuildingdashboardwithvariousvisualizationoptions.Forexample,totakealookattheCPUutilizationsinceCollectdstartedmonitoringit,performthefollowingsteps:
1. ClickonVisualizeintheleftpaneloftheKibanaapplication2. ClickontheCreateavisualizationbutton3. ClickonLinetochoosethelinechart4. Choosetheindexhelpinghands.collectd.instance-*patternfromthesection
ontheleft5. ClickontheAddafilter+optionbelowthesearchbaratthetop6. Selectfilterasplugin.keywordiscpuandsave7. ClickontheY-axisandchangeAggregationtoAverage8. SelectthefieldasValue9. Next,clickontheX-AxisunderBuckets10. ChoosetheAggregationasDateHistogram11. Itshouldbydefaultselectthe@timestampfield12. KeeptheintervalasAuto13. ClickontheapplychangesplayiconatthetopoftheMetricspanel
14. Itwillshowthechartontheright-handside,asshowninthenextscreenshot
15. Clickonthearrowiconatthebottomofthechartareatobringupthetable
Oncethesestepshavebeenperformed,youshouldbeabletoseetheCPUutilizationovertimeforthelast15minutes(default),asshowninthefollowingscreenshot.Thecreatedvisualization(https://www.elastic.co/guide/en/kibana/current/visualize.html)canbesavedandlateraddedtoadashboard(https://www.elastic.co/guide/en/kibana/current/dashboard.html)tobuildafull-fledgeddashboardtomonitorallthecapturedmetricsinrealtime:
LoggingandmonitoringguidelinesMicroservicesoftheHelpingHandsapplicationmustgeneratebothapplicationandauditlogsthatcanbecapturedbytheELKStack.Applicationlogscanbegeneratedbytheservicesusingthetools.logginglibraryofClojure(https://github.com/clojure/tools.logging)thatlogsamessageinastandardlog4j(https://logging.apache.org/log4j/2.x/)stylesyntax.Logstashworkswellwithmostofthecommonloggingformats(https://www.elastic.co/guide/en/logstash/6.1/plugins-inputs-log4j.html),butitisrecommendedtousestructuredlogginginstead.
StructuredlogsareeasiertoparseandloadwithinacentralizedrepositoryusingtoolssuchasLogstash.Librariessuchastimbre(https://github.com/ptaoussanis/timbre)supportstructuredloggingandalsoallowpublishingthelogsdirectlytoaremoteserviceinsteadofloggingtoafile.Inadditiontostructuredlogging,considerincludingpredefinedstandardtagswithinthelogmessages,asshowninthefollowingtable:
Name Event<service> Thenameoftheservice,suchashelping-hands.alert
<service>-
start
Loggedfromthemainfunctiononcetheapplicationisupandrunning
<service>-
init
Oncetheserviceisupandrunningandithasbeensuccessfullyinitializedwiththerequiredconfiguration
<service>-
stop
Thelaststatementinthemainfunctionbeforetheapplicationexitsorisintheshutdownhook
<service>-
config Usedforconfiguration-relatedmessages<service>-
process Usedforprocessing-relatedmessages<service>-
exception Usedforruntimeexceptionhandlerandrelatedmessages
Tagsareparticularlyusefultofilterthelogmessagesoriginatingfromaservice
ofinterestandalsohelpstofurtherdrilldownthelogsviaspecificstateleveltags,suchaslogmessagesgeneratedatstartup,duringshutdown,orloggedasaresultofanexception.
Inadditiontothetags,itisalsorecommendedtoalwaysuseUTC(https://en.wikipedia.org/wiki/Coordinated_Universal_Time)whilegeneratinglogmessages.Sincealltheselogmessagesareaggregatedinacentralizedrepository,havingdifferenttimezonemakesitchallengingtoanalyzethem,astheywillbeoutofsyncduetothetimezoneofthehostmachinesthatmayberunningindifferenttimezones.
Althoughlogmessagesarequiteusefultodebugtheissuesandprovideinformationregardingthestateoftheapplication,theyaffecttheperformanceoftheapplicationdrastically.So,logjudiciouslyandasynchronouslyasmuchaspossible.Itisalsorecommendedtopublishthelogeventsasynchronouslytochannels,suchasApacheKafka,insteadofloggingtoafilethatrequiresdiskI/O.LogstashhasaninputpluginforKafka(https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html)thatcanreadeventsfromaKafkatopicandpublishittothetargetoutputpluginlikethatofElasticsearch(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html).
Riemann(http://riemann.io/)isanalternativetoELKstack.Itisusedtomonitordistributedsystems,suchastheonesbasedonmicroservices-basedarchitecture.Riemannisincrediblyfastandcanbeusedtogeneratealertsinnearrealtimewithoutoverwhelmingtherecipient,usingitsrollupandthrottleconstructs(http://riemann.io/howto.html#roll-up-and-throttle-events).
UsingELKstacktocollecttheeventsand,atthesametime,streamingLogstasheventsviaRiemannusingLogstashRiemannoutputplugin(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-riemann.html)makesitpossibletogeneratealertsinnearrealtimeandalsousethegoodnessofElasticsearchandKibanatoprovideareal-timemonitoringdashboardfordrill-downanalysis.
DeployingmicroservicesatscaleMicroservicesmustbepackagedasaself-containedartifactthatcanbereplicatedanddeployedusingasinglecommand.Theservicesshouldalsobelightweightwithshorterstarttimestomakesurethattheyareupandrunningwithinseconds.Itisrecommendedtopackagemicroserviceswithinacontainer(https://en.wikipedia.org/wiki/LXC)thatcanthenbedeployedfasterduetoitsinherentimplementationascomparedtosettingupabaremetalmachinewithahostoperatingsystemandrequireddependencies.Packagingmicroserviceswithincontainersalsomakesitpossibletomovefromdevelopmenttoproductionfasterandinanautomatedfashion.
IntroducingContainersandDockerLinuxContainers(LXC)isavirtualizationmethodattheoperatingsystemlevelthatmakesitpossibletorunmultipleisolatedLinuxsystems,alsoknownascontainers,onasinglehostOSusingasingleLinuxKernel(https://en.wikipedia.org/wiki/Linux_kernel).Theresourcesaresharedamongthecontainersusingcgroups(https://en.wikipedia.org/wiki/Cgroups)thatdonotrequirevirtualmachines.SinceeachcontainerreliesontheLinuxKernelofthehostOSthatisalreadyrunning,thestarttimeofcontainersismuchlowerascomparedtoavirtualmachinethatisrunbyaHypervisor(https://en.wikipedia.org/wiki/Hypervisor).
Docker(https://en.wikipedia.org/wiki/Docker_(software))alsoprovidesresourceisolationforthecontainersusingLinuxcgroups,kernelnamespaces(https://en.wikipedia.org/wiki/Linux_namespaces),andunionmountingoption(https://en.wikipedia.org/wiki/Union_mount)thathelpsittoavoidtheoverheadofstartingandmaintainingvirtualmachines.UsingDockercontainerformicroservicesmakesitpossibletopackagetheentireserviceanditsdependencieswithinacontainerandrunonanyLinuxserver.
Althoughitispossibletopackagetheentiremicroservice,includingthedatabasewithinaDockercontainer,itisrecommendedthatyoukeepthedatabaseoutoftheDockercontainer.Reasonbeingthatdatabasesmaynotbetheprimecandidatefordynamicscaleupandscaledownascomparedtotheservicesthemselves.
SettingupDockerDockerisasoftwarethathelpscreateDockerimagesthatcanbeusedtocreateoneormorecontainersandrunitonthehostoperatingsystem.TheeasiestwaytosetupDockeristousethesetupscriptprovidedbyDockerforthecommunityedition,asshownhere:
%wget-qO-https://get.docker.com/|sh
TheprecedingcommandwillsetupDocker,basedonthehostoperatingsystem.DockeralsoprovidesprebuiltpackagesforallthepopularoperationsystemsunderitsDownloadsection(https://www.docker.com/community-edition#/download).Onceinstalled,addthecurrentusertothedockergroup,asshownhere:
%sudousermod-aGdocker$USER
Youmayhavetostartanewloginsessionforgroupmembershiptobetakenintoaccount.Oncedone,Dockershouldbeupandrunning.TotestDocker,trylistingtherunningcontainersusingthefollowingcommand:
%dockerps-a
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
Sincetherearenocontainersrunning,itwilljustlisttheheaders.Totestrunningacontainer,usethedockerruncommand,asshowninthefollowingexample.Itwilldownloadthehello-worldDockerimageandrunitinacontainer:
%dockerrunhello-world
Unabletofindimage'hello-world:latest'locally
latest:Pullingfromlibrary/hello-world
ca4f61b1923c:Pullcomplete
Digest:sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751
Status:Downloadednewerimageforhello-world:latest
HellofromDocker!
Thismessageshowsthatyourinstallationappearstobeworkingcorrectly.
Togeneratethismessage,Dockertookthefollowingsteps:
1.TheDockerclientcontactedtheDockerdaemon.
2.TheDockerdaemonpulledthe"hello-world"imagefromtheDockerHub.
(amd64)
3.TheDockerdaemoncreatedanewcontainerfromthatimagewhichrunsthe
executablethatproducestheoutputyouarecurrentlyreading.
4.TheDockerdaemonstreamedthatoutputtotheDockerclient,whichsentit
toyourterminal.
Totrysomethingmoreambitious,youcanrunanUbuntucontainerwith:
$dockerrun-itubuntubash
Shareimages,automateworkflows,andmorewithafreeDockerID:
https://cloud.docker.com/
Formoreexamplesandideas,visit:
https://docs.docker.com/engine/userguide/
Theprecedingoutputisaresultoftheexecutionofthehello-worldimage.DumpingtheentiresetofstepsthatwereusedbyDockertogeneratethemessageishelpfultounderstandwhatasinglecommandsuchasdockerrundoesattheback.
Theprecedingcommanddownloadsthehello-worldimage,storesitlocally,andthenrunsitwithinacontainer.Theimagejustdumpsamessagewiththestepsthatwereusedtogeneratethemessageontheconsoleandexits.Since,thehello-worldimagewasusedforthefirsttime,itwasnotpresentonthelocalmachine,andthatisthereasonthedockercommanddownloadeditfirstfromremoteDockerRegistry(https://docs.docker.com/registry/).Tryrunningthesamecommandagain,and,thistime,itwilllocatetheimagewithinthelocalmachineanduseitstraightaway,asshowninthefollowingexample.Inthiscase,itjustdumpsthemessageandtheinstallationstepsasanoutputoftheexecutionofthehello-worldimageasbefore:
%dockerrunhello-world
HellofromDocker!
Thismessageshowsthatyourinstallationappearstobeworkingcorrectly.
Togeneratethismessage,Dockertookthefollowingsteps:
1.TheDockerclientcontactedtheDockerdaemon.
2.TheDockerdaemonpulledthe"hello-world"imagefromtheDockerHub.
(amd64)
3.TheDockerdaemoncreatedanewcontainerfromthatimagewhichrunsthe
executablethatproducestheoutputyouarecurrentlyreading.
4.TheDockerdaemonstreamedthatoutputtotheDockerclient,whichsentit
toyourterminal.
Totrysomethingmoreambitious,youcanrunanUbuntucontainerwith:
$dockerrun-itubuntubash
Shareimages,automateworkflows,andmorewithafreeDockerID:
https://cloud.docker.com/
Formoreexamplesandideas,visit:
https://docs.docker.com/engine/userguide/
TolisttheDockerimagesavailableonthelocalmachine,executethedockerimagescommandasshowninthefollowingexample;itlistsalltheavailableimages:
%dockerimages
REPOSITORYTAGIMAGEIDCREATEDSIZE
hello-worldlatestf2a91732366c7weeksago1.85kB
Similarly,tolisttheDockercontainersthatwerecreated,usethedockerps-acommand,asusedearlier.Thistime,itshouldlistthecontainersthatwerestartedusingthehello-worldimage,asshownhere:
%dockerps-a
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
e0e5678ef80ahello-world"/hello"5minutesagoExited(0)5minutesago
happy_rosalind
ea9815d87660hello-world"/hello"8minutesagoExited(0)8minutesago
fervent_engelbart
Formoredetailsontheavailablecommandsandoptions,refertotheDockerCLIcommandsreferenceguide(https://docs.docker.com/engine/reference/commandline/docker/).
FormoredetailsonDockersetupandconfigurationoptions,takealookattheDockerpostinstallationguide(https://docs.docker.com/engine/installation/linux/linux-postinstall/)
CreatingaDockerimageforHelpingHandsHelpingHandsservicescreatedinthepreviouschaptershaveaDockerfilecreatedasapartoftheprojecttemplate.Forexample,takealookatthedirectorystructureoftheAuthservice,asshowninthefollowingexample.TorunwithinaDockercontainer,the::http/hostkeyoftheservicedefinitionwithinthehelping-hands.auth.servicenamespacemustbesettoafixedIPaddressor0.0.0.0tobindtoalltheIPv4addressesavailablewithinthecontainer.
%tree-L1
.
├──Capstanfile
├──config
├──Dockerfile
├──project.clj
├──README.md
├──resources
├──src
├──target
└──test
5directories,4files
ChangethecontentoftheDockerfilefortheAuthservice,asshowninthefollowingexample.Itcopiesboththeconfigdirectoryandthestand-aloneJARfileoftheAuthserviceofHelpingHands.Ifthestand-aloneJARisnotpresentinthetargetfolder,createitusingtheleinuberjarcommandthatwillcreateastand-aloneJARinthetargetdirectoryoftheAuthproject.
FROMjava:8-alpine
MAINTAINERHelpingHands<helpinghands@hh.com>
COPYtarget/helping-hands-auth-0.0.1-SNAPSHOT-standalone.jar/helping-hands/app.jar
COPYconfig/conf.edn/helping-hands/
EXPOSE8080
CMDexecjava-Dconf=/helping-hands/conf.edn-jar/helping-hands/app.jar
Next,createaDockerimageusingthedockerbuildcommand,asshowninthefollowingexample.ThedockerbuildcommandlooksforaDockerfileinthesamedirectorywhereitstartedfrom.IfDockerfileispresentatsomeotherlocation,explicitpathtotheDockerfilecanbespecifiedusingthe-fflag.Formoredetails
onthedockerbuildcommand,refertotheusageinstructions(https://docs.docker.com/engine/reference/builder/#usage).
#buildthedockerimage
%dockerbuild-thelping-hands/auth:0.0.1.
SendingbuildcontexttoDockerdaemon48.44MB
Step1/6:FROMjava:8-alpine
8-alpine:Pullingfromlibrary/java
709515475419:Pullcomplete
38a1c0aaa6fd:Pullcomplete
5b58c996e33e:Pullcomplete
Digest:sha256:d49bf8c44670834d3dade17f8b84d709e7db47f1887f671a0e098bafa9bae49f
Status:Downloadednewerimageforjava:8-alpine
--->3fd9dd82815c
Step2/6:MAINTAINERHelpingHands<helpinghands@hh.com>
--->Runningindd79676d69a4
--->359095b88f32
Removingintermediatecontainerdd79676d69a4
Step3/6:COPYtarget/helping-hands-auth-0.0.1-SNAPSHOT-standalone.jar/helping-
hands/app.jar
--->952111f1c330
Removingintermediatecontainer888323c4cc30
Step4/6:COPYconfig/conf.edn/helping-hands/
--->3c43dfd4af83
Removingintermediatecontainer028df1e03d58
Step5/6:EXPOSE8080
--->Runningin8cf6c15cab9f
--->e79d993e2c67
Removingintermediatecontainer8cf6c15cab9f
Step6/6:CMDexecjava-Dconf=/helping-hands/conf.edn-jar/helping-hands/app.jar
--->Runningin0b4549cf84f2
--->f8c9a7e746f3
Removingintermediatecontainer0b4549cf84f2
Successfullybuiltf8c9a7e746f3
#listtheimagestomakesureitisavailable
%dockerimages
REPOSITORYTAGIMAGEIDCREATEDSIZE
helping-hands/auth0.0.1f8c9a7e746f317secondsago174MB
hello-worldlatestf2a91732366c7weeksago1.85kB
java8-alpine3fd9dd82815c10monthsago145MB
Oncetheimageiscreatedandregisteredusingthespecifiednameandtag,newcontainerscanbecreatedfromthesameimage,asshownhere:
#createanewcontainerfromthetaggedimage
%dockerrun-d-p8080:8080--namehh_auth_01helping-hands/auth:0.0.1
286f21a088dd8b6b6d814f1fb5e4d27a59f46b6d8c474160628ffe72d3de2b56
#verifythatthecontainerisrunning
%dockerps-a
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
286f21a088ddhelping-hands/auth:0.0.1"/bin/sh-c'exec..."5secondsagoUp3
seconds0.0.0.0:8080->8080/tcphh_auth_01
e0e5678ef80ahello-world"/hello"51minutesagoExited(0)50minutesago
happy_rosalind
ea9815d87660hello-world"/hello"54minutesagoExited(0)53minutesago
fervent_engelbart
CheckthelogmessagesgeneratedbyDockertomakesurethatAuthserviceisupandrunning,asshownhere:
%dockerlogs286f21a088dd
Creatingyourserver...
Omniconfconfiguration:
{:conf#object[java.io.File0x47c40b56"/helping-hands/conf.edn"]}
TheAuthservicecannowbeaccesseddirectlyatthe8080portasshowninthefollowingexample.Theportismappedusingthedockerruncommandwith-pflag,asusedearlierwhilecreatingthecontainer.
%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hhuser"
HTTP/1.1200OK
...
Authorization:Bearer
eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.1enLmASKP8uqPGvW_bOVcGS8-
0wtR3AS0xxGolaNixXCSXaY_7LKqw.RcXp4s0397a3M_EB-DyFAQ.B6b93-
1_grZa7HJee6nkcT4LM3gV7QxmR3CIHxX9ngzFqPyyJTcBWvo2N4TTlY4gJYgeNtIyaJsAmvVYCEi7YKyp47bF1wzgFbpjkfVen6y-
580kmf5JqaP2vXQmNpFiVRB6FGGqldnAaDKdBCCrv0HRgGbaxyg_F_05j4G9AktO26hUMfXvmd9woh61Id-
lV4xvRZOcn57X6aH-HL2JuA.hUWvDD6lQWmXaRGYCf3YOQ
Transfer-Encoding:chunked
Tostopthecontainer,usethedockerstopcommand,andtodeletethecontainer,usethedockerrmcommand,asshownhere:
%dockerstophh_auth_01
hh_alert_01
%dockerrmhh_auth_01
hh_alert_01
%dockerps-a
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
e0e5678ef80ahello-world"/hello"54minutesagoExited(0)54minutesago
happy_rosalind
ea9815d87660hello-world"/hello"57minutesagoExited(0)57minutesago
fervent_engelbart
FormoredetailsonhowtocreateeffectiveDockerfile,takealookatthebestpracticesforwritingDockerfiles(https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/).
IntroducingKubernetesContainerizingtheservicesoftheHelpingHandsapplicationallowsthemtobedeployedacrossmultiplemachinesfaster,butscalingthemrequiresmanualeffortandinvolvementoftheDevOpsteamtoscaleitupanddown.Monitoringalltherunningcontainersmayalsobecomeoverwhelmingovertime,astheservicesmayscaletohundredsofinstancesrunningcontainersacrosstheclusterofmachines.Althoughthefailureofanyserviceorcontainercanbealertedtotheteam,butrunningthemmanuallyisatedioustask.Moreover,itisexhaustiveandoftenerror-pronetoestimateandachieveeffectiveresourceutilizationandoptimallybalancethenumberofrunninginstancesofeachservicemanually.
Toavoidsuchmanualtasksandensurethattheconfigurednumberofservicesarealwaysrunningandeffectivelyutilizingtheavailableresources,containerorchestrationenginesarerequired.Kubernetesisonesuchopensourcecontainerorchestrationenginethatiswidelyusedforautomateddeployment,scaling,andmanagementofcontainerizedapplicationssuchastheservicesoftheHelpingHandsapplication.
InaKubernetesdeployment,therearetwokindsofmachines,MasterandNode(previouslyknownasMinions).MasterinstancesarethebrainofKubernetesenginethatmakeallthedecisionsrelatedtothedeploymentofcontainersandalsorespondtovariouseventsoffailuresandnewallocationrequests.Masterinstancerunskube-apiserver(https://kubernetes.io/docs/admin/kube-apiserver/),etcd(https://kuber
netes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/),kube-controller-manager(https://kubernetes.io/docs/admin/kube-controller-manager/),andkube-scheduler(https://kubernetes.io/docs/admin/kube-scheduler/).Theyalsorunkube-proxy(https://kubernetes.io/docs/admin/kube-proxy/)toworkwithinthesameoverlaynetworkasthatofnodes.ItisrecommendedtorunMasteronseparatemachinesthatarededicatedforclustermanagementtasksonly.
Nodes,ontheotherhand,areworkermachinesinaKubernetesclusterandrunsPods.APod(https://kubernetes.io/docs/concepts/workloads/pods/pod/)isasmallestunitofcomputingthatcanbecreatedandmanagedbyKubernetescluster.Itisagroupofoneormorecontainersthatsharenetwork,storage,andacommonsetofspecifications.EachNoderunsaDockerservice,kubelet(https://kubernetes.io/docs/admin/kubelet/),andkube-proxyandismanagedbythemastercomponents.ThekubeletagentrunsoneachNodeandmanagesthepodsthatareallocatedtotheNode.ItalsoreportsthestatusofthepodsbacktotheKubernetescluster.
FormoredetailsonKubernetesMasterandNodecomponents,takealookatKubernetesConceptsdocumentathttps://kubernetes.io/docs/concepts/overview/components/.
Kuberneteshasanin-builtsupportforDockercontainers.ServicesuchasAutomaticBinPacking(https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)foreffectiveutilizationoftheresources,horizontalscaling(https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)toscaletheservicesupanddownandbasedonfactorssuchasCPUusageandSelf-Healing(https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller)toautomaticallyrestartcontainersthatfailisprovidedout-of-the-boxbyKubernetes.
KubernetesalsosupportsServiceDiscoveryandLoadBalancingbyallocatingcontainerstheirownIPaddresses.ItalsoallocatesacommonDNSnameforasetofcontainersthatallowotherexternalservicestojustknowtheDNSnameandusethesametoreachtheservice.KubernetesinternallybalancestherequestsamongtheservicesrunninginthecontainersthatareregisteredwiththeDNSwiththespecifiedname.RollingUpgrades(https://kubernetes.io/docs/tutorials/kubernetes-basics/update-intro/)arealsoprovidedbyKubernetesbyincrementallyupgradingthecontainerswiththenewones.AllupdatesareversionedbyKubernetes,anditallowsrollbacktoanypreviousstableversion.
FormoredetailsonKubernetes,takealookatKubernetestutorialsthatcoverallthebasicfeaturesofKuberneteswithexamplesfromhttps://kubernetes.io/docs/tutorials/.
GettingstartedwithKubernetesThesimplestwaytogetstartedwithKubernetesandrunlocallyonasinglemachineistouseMinikube(https://github.com/kubernetes/minikube).TosetupMinikube,usethefollowinginstallationscript:
%curl-Lominikubehttps://storage.googleapis.com/minikube/releases/latest/minikube-
linux-amd64&&chmod+xminikube&&sudomvminikube/usr/local/bin/
TheprecedingcommanddownloadsthelatestreleaseofMinikubescriptandmakesitavailableonthepathbycopyingittothe/usr/local/bindirectory.Minikubealsorequireskube-ctltointeractwiththeKubernetescluster.Tosetupkube-ctl,usetheinstallationscriptofkube-ctl,asshownhere:
#downloadkube-ctlscript
%curl-LOhttps://storage.googleapis.com/kubernetes-release/release/$(curl-s
https://storage.googleapis.com/kubernetes-
release/release/stable.txt)/bin/linux/amd64/kubectl
#makethescriptexecutable
%chmod+x./kubectl
#makeitavailableonthepath
%sudomv./kubectl/usr/local/bin/kubectl
FormoredetailsonhowtousetheminikubecommandtocreateaKubernetesclusterandkube-ctltointeractwiththeMasteranddeploycontainers,takealookattheMinikubeprojectdocumentation(https://github.com/kubernetes/minikube).
SummaryInthischapter,youlearnedhowtopreparemicroservicesforproduction.WefocusedonsecurityandmonitoringofmicroservicesoftheHelpingHandsapplication.Wealsolearnedhowtodeployandscalemicroservicesusingcontainers.WealsodiscussedorchestrationenginessuchasKubernetesandhowtheyareusefultoorchestratecontainers.Withthischapter,youareallsettobuildyournextbestapplicationusingmicroservices-basedarchitectureanddeployiteffectivelyinproduction.Whatwillyoubuildnext?
OtherBooksYouMayEnjoyIfyouenjoyedthisbook,youmaybeinterestedintheseotherbooksbyPackt:
MasteringMicroserviceswithJava9-SecondEditionSourabhSharma
ISBN:978-1-78728-144-8
Usedomain-drivendesigntodesignandimplementmicroservicesSecuremicroservicesusingSpringSecurityLearntodevelopRESTservicedevelopmentDeployandtestmicroservicesTroubleshootanddebugtheissuesfacedduringdevelopmentLearningbestpracticesandcommonprincipalsaboutmicroservices
Spring5.0Microservices-SecondEditionRajeshRV
ISBN:978-1-78712-768-5
FamiliarizeyourselfwiththemicroservicesarchitectureanditsbenefitsFindouthowtoavoidcommonchallengesandpitfallswhiledevelopingmicroservicesUseSpringBootandSpringCloudtodevelopmicroservicesHandleloggingandmonitoringmicroservicesLeverageReactiveProgramminginSpring5.0tobuildmoderncloudnativeapplicationsManageinternet-scalemicroservicesusingDocker,Mesos,andMarathonGaininsightsintothelatestinclusionofReactiveStreamsinSpringandmakeapplicationsmoreresilientandscalable
Leaveareview-letotherreadersknowwhatyouthinkPleaseshareyourthoughtsonthisbookwithothersbyleavingareviewonthesitethatyouboughtitfrom.IfyoupurchasedthebookfromAmazon,pleaseleaveusanhonestreviewonthisbook'sAmazonpage.Thisisvitalsothatotherpotentialreaderscanseeanduseyourunbiasedopiniontomakepurchasingdecisions,wecanunderstandwhatourcustomersthinkaboutourproducts,andourauthorscanseeyourfeedbackonthetitlethattheyhaveworkedwithPackttocreate.Itwillonlytakeafewminutesofyourtime,butisvaluabletootherpotentialcustomers,ourauthors,andPackt.Thankyou!
Recommended