Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved.|
Microservices,Containers,DatabasesandPersistenceModelsTIP4071
Kuassi MensahDirectorProductManagement,Oracle
PaulParkinsonConsultantMemberTechnicalStaff,Oracle
October23,2018
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SafeHarborStatement
Thefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,timing, andpricingofanyfeaturesorfunctionalitydescribedforOracle’sproductsmaychangeandremainsatthesolediscretionofOracleCorporation.
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Agenda• Concepts:Microservices,Containers,Kubernetes• TransactionModels• MicroservicesPersistenceandDatabases• Demo
3
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Concepts:Microservices,Containers,Kubernetes
4
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
MicroServicesBuildfromScratchorBreakMonolithicappsintoModules• Reducecomplexity• Modulespecificdatamodel• Scaleatmodulelevel• FasterDevelopment– API-First,Events-First,DomainDrivenDesign
• Blue/greendeployments• Idempotent• FaultIsolation
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Containers/Docker• OSvirtualization• Modularity• Reducedsize• Greaterscaling• Isolatedenvironmentforrunningapps,webserver,persistencestore,etc• Consistencyinruntimeenvironments(includingtestingandproduction)• Operationalsimplicity• Productivity• Enableportabilityfromon-premisestotheCloud
Source:Docker
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ContainersOrchestration/KubernetesContainersmustbemanaged(resource),scheduled,loadbalancedandconnectedtoexternalworld
Kubernetes:Greekwordforaship'scaptain• Goal:robustplatformforrunningthousandsofcontainersinproduction.
• Systemforautomatingdeploymentandscalingofcontainerizedapps• Velocity– constantupdates,pause/resumedeployments,versioncontrol
• Immutability– newimageforchanges(maintainshistory,lineage)
• DeclarativeAPI&config– describedesiredstateofsystem,abstracts“how”
• Selfhealing– continuouslymaintainsdesiredstateovertime.
• Autoscaling
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Kubernetes/MicroServicesIdealformicroservices…• Pods,arunnableunitofwork(containerimagesdevelopedbydifferentteamsintoasingledeployableunit);usuallyholds1or2containers• Loadbalancing,naminganddiscoveryisolateonemicroservicefromanother• Namespacesprovideisolationandaccesscontrolsothateachmicroservicecancontrolthedegreetowhichotherservicesinteractwithit.• PlatformservicestellstherestoftheKubernetesenvironmentwhatservices yourapplicationprovides.• Api-registry:tracksofallavailableservicesandtheresourcestheyexpose.
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
KubernetesPlatformServicesonOKE
9
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.| 10
PainPointsAddressed
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
IstioRoutingRules• IstioRouteRulescanbeusedtochangewhichservicereceivesrequests• Thisallowsinvokertopassrequeststodifferentservicewithnocodechangeorredeployment• Routerulescanshifttrafficallatonce,orroutepercentagestonewservices
RouteRule.yaml
kind: VirtualServicemetadata:
name: processnamespace: demo
spec:hosts:- processhttp:- route:
- destination:host: processsubset: v2
11
kind: DestinationRulemetadata:
name: processnamespace: demo
spec:host: processsubsets:- name: v1
labels:version: v1
- name: v2labels:version: v2
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
EventBroker• Addsrules,callbacks,andstructuretovanillamessagebus• Rulesaresimpleandstraightforward• Rulescanbeupdatedtoalterrequestflow• EBcanleverageanymessagebusimplementationunderneath
EventRule.yaml
apiVersion: auraevents.oracledx.com/v1alpha1kind: Rulemetadata:name: rule-process
labels:auraevents.oracledx.com/eventType:
processspec:
action:url: http://process.demo:2222
callbackURL: http://storefront.sf:8888/callback
12
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
EventBroker• CloudEvents standard ExampleCloudEvent
{"cloudEventsVersion":"0.1","eventType":"com.example.someevent","source":"/mycontext","eventID":"A234-1234-1234","eventTime":"2018-04-05T17:31:00Z","comExampleExtension1":"value","comExampleExtension2":{"otherValue": 5
},"contentType":"text/xml","data":"<muchwow=\"xml\"/>"
}
13
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SauronIntegration• EasyintegrationofSauronManagedService• Providesvisualrepresentationofapplicationandsystemmetrics• ProvidesalertingfeaturesviaeMail,Slack,etc.
Prometheusconfig- job_name: 'events-broker'
scrape_interval: 5s
scrape_timeout: 5s
static_configs:
- targets: ['129.213.46.56:30350']
• Businessmetricsbasedonsmartfilteringofeventmetricsindashboard
14
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
MarketplaceProvisioningofATPDBaaS• ServiceBrokeraddsconnectioninfointoclusterviaKubernetesSecrets• ServiceiswrittentoconsumeSecret• Thisallowsservicetobedeployedindifferentenvironmentswithoutanycodechanges,andpickuptheappropriateDBconnectionautomatically
Deployment.yamlenv: - name:DB_ADMIN_USER valueFrom:
secretKeyRef:name:atp-demo-binding
key:user_name - name:DB_ADMIN_PWD valueFrom:
secretKeyRef:name:atp-user-cred
key:password - name:WALLET_PWD valueFrom:
secretKeyRef:name:atp-user-cred
key:wallet_password volumeMounts: - name:creds mountPath:/db-demo/creds
15
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
TransactionsModels
16
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
MicroServices Integration• AnapplicationbecomeanassemblyofMicroServices thatcommunicateamongthemselves• MSintegrationchoices:filesystem,database,RPC,Event• Event-basedIntegration:bestchoice– Asynchronous,loosecoupling– Pub/Sub,Notification– Statetransfer– Immutable– SupportforEventSourcingandCommandQueryResponsibilitySegratation (CQRS)
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
EventSourcingMicroservicesinteractby”sourcingevents”in/outfromtheEventstoreviatheEventBroker• Eventstore(Kafka,AQinthefutureorvanillaOracleDB)isthesinglesourceoftruth:• Producers:logeventsintheEventstorethenpublishanotification• Consumers:notifiedwhenEventsarepublishedthen“read”theEventstore
WebServer Shipping ServiceOrdersService
PublishEvent
ReceiveEvent
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
CommandQueryResponsibilitySegregation- CQRSSeparationofresponsibilitieswithinaboundedcontext(Designdomain)• Commands:makechangestostate/data-- donotquerystate
• Queries:viewstate/data(scalewithMaterializedViews)-- donotchangestate
• OftencombinedwithEventSourcing
WebServer Shipping ServiceOrdersServiceCommand
Query
MaterializedViews
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
BusinessTransactions– RequirementsDistributedTransactionsthatspanmultiplemicroservices– E.g.,Bookingatripinclude:flight,hotel,car,shows
• MicroserviceAcallsBthatcallsC.– TheworkfloworbusinesstransactioncommitsonlywhenCcompletes
• HowdoyouundoA&BshouldCfail?– Traditionaltwo-phaseCOMMITTXisananti-patternwithMicroServices– Eachservicecommitsindividually(locally):howtoensureconsistency?• SAGAPattern• HighconcurrencywithEscrow/Promises
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
BusinessTransactionsConsistencywithSAGAPattern“Asagaisasequenceoflocaltransactions.Eachlocaltransactionupdatesthestate(database)andpublishesamessageoreventtotriggerthenextlocaltransaction inthesaga.Ifalocaltransaction failsbecauseitviolatesabusinessrulethenthesagaexecutesaseriesofcompensating transactions thatundothechanges thatweremadebytheprecedinglocaltransactions.”https://microservices.io/patterns/data/saga.html
Twoapproachesforcompensatingfailedmember(s)ofadistributedTx• Choreography– Theservicesinteractwitheachothertocoordinatewellknown/definedactivity(thedemohasanexample)
• Orchestration– Anorchestratorisusedtocoordinatetheactivity.– Allmessagesarepassedthrough(interceptedandissuedby)thisorchestrator.
• CompensationcansimplybeTxABORT (thiscanbeautomated)
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
BusinessTransactionsConsistencywithEscrow/PromisesEscrow– longrunningbusinesstransactions• Counters(anycolumnwith“commutative”properties)usedforinventoriesmanagement– Stockonhand,money/share/creditofaccount,availableseats,...– manipulatedwithplus/minus-- commutative– Onerequestissufficientamount := amount+/-change
• Queriesreturntwodistinctstates:#availableinstock,#inflow• Benefits:parallelupdate,automaticcompensation(theinversefunction)
Promises(notJava/NodeJSpromises)
• Apromiseisagrantedescrowrequestwithtimelimits(i.e.,On-holdreservation)– Expirationneedscarefulhandlingofraceconditions
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
MicroservicesPersistenceandDatabasesDataModels,Domains,Databases
23
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
PolyglotorMulti-ModelDatabase?Microservicesarchitecturegoals:flexibilityandagility• Polyglotsystem– aspecializeddatabaseforeachmicroservice– mayleadtoanexpensiveanddifficultplatformtooperate
• Multi- modeldatabase(i.e.,AutonomousTransactionProcessing)– EachservicegetsorshareanATPserviceinstance– highavailabilityanddurabilityoptions
Spatial
Graph
Relational
XML UnstructuredData
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OracleAutonomousDatabase
2525
Self-Driving• Automatesallmonitoring,managementandtuning
• Createsmissioncriticalscale-outdatabasewithDR
• Fullcompatibilityenablessimpledatabasemigration
Self-Repairing• Recoversautomaticallyfromanyfailure
• 99.995%uptimeincludingmaintenance,guaranteed
• Elasticallyscalescomputeorstorageasneeded
Self-Securing• Automaticallyappliessecurityupdatesonline
• Secureconfigurationwithfulldatabaseencryption
• SensitivedatahiddenfromOracleorcustomeradmins
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
PDBSharding forMicroservices
• PDBSharding inDB19c– EachPDBcanbesharded acrossmultipleCDBs
• Providesfaultisolationandgeo-distributionformicroservices– LossofanentireCDBmakesonlypartofaPDBunavailable
• Alsoallowseachmicroservices toscaleitsPDBindividually– Moreefficientuseofresourcescomparedtoscalingamonolithicapplication(CDB).
26
Scalability,faultisolationandgeo-distribution Shard-1 Shard-1Shard-1
Shard-2 Shard-2
Shard-3
Recomm
ProductCatalog
ProductCatalog
ProductCatalog
CheckOut
CheckOut
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OnlineShoppingKafkaasEventBrokerandStore
Catalog Cart Order Inventory Shipping
PlaceOrder
BrowseCatalog Product
InCart
KubernetesEventBrokerandEventRules
Kafka/EventStore/EventLog
PublishCloudEvent
ConsumeEvent
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
AQSupportforMicroservices• AQ/ATPasEventstore• QueueoperationsandDMLDataandmessageinthesamelocaltransaction• Sharded Queuesfurnishhighscalability• Transactedsessionguaranteesnolossofmessageandexactlyoncedelivery
• Clientinitiatednotification• Handleslargemessagebacklogs
• Queuelevelaccessprivileges(enqueue,dequeue)supportsCQRS• AQsupportforSAGA-patterndistributedtransactions• HighAvailabilityandDisasterRecovery
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OnlineShoppingKafkaEventBroker&Store+AQMessageproviderservice
Catalog Cart Order Inventory Shipping
PlaceOrder(insertdb+AQmessage)
BrowseCatalog Product
InCart
K8EventBrokerandEventRules
Kafka/EventStore/EventLog
AQEventProviderService
CloudEvent
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
AQEnhancementstoSupportMicroServices• RetentionsupportinSharded Queues• CloudEventssupport• Logcompaction• AutonomousQueuesinDBaaS• ChangeQueryNotificationsupportinATPusingAQsecurenotification• Lightweight,seekable andretroactivesubscribers• Topicauto-creation• KafkaAPI(Java)
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OnlineShoppingAQasEventBrokerandStore
Catalog Cart Order Inventory Shipping
PlaceOrder(insertdb+AQmessage)
BrowseCatalog Product
InCart
AQBroker,Storeand EventRules
Kafka/EventStore/EventLog
AQEventProviderService
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
AsynchronousDatabaseAccessforMicroServicesMicroservicesarchitecturegoals:flexibilityandagility• SynchronousDBAccess(JDBC)– Themicroservicewaitstilldataupdate(DML)iscompleted– Publishestheevent
• AsynchronousDbAccess(ADBA)– ThemicroservicesubmitsaDMLoperation+AQeventpublisgingthenmoveson.a) Create a transaction object b) create/submit a RowCountOperation to insert an orderc) get a CompletionStaged) create another RowCountOperation to publish an AQ event e) session.commitMaybeRollback(trans) f) submit the whole transaction and return
– Hands-onlabandSession@https://bit.ly/2DzV3uT
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
RefactoringMonolithic
• CapturesaSQLTuningSet(STS)whileaSQLworkloadisrunning• ProducesaMapofSQLvsTables• GeneratesaGraph
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
RefactoringMonolithic– BoundedContexts
• Identifynon-conflictingdatamodel• DefineserviceswithinBoundedContextsboundaries
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Demo
35
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OnlineShopping
Catalog Cart Order Inventory Supplier
PlaceOrder(insertdb+AQmessage)
BrowseCatalog Product
InCart
K8EventBrokerandEventRules
Kafka/EventStore/EventLog
AQEventProviderService
CloudEvent
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OnlineShopping- Workflow1. Catalogservice:Graphdatamodel– notimplemented
2. Orderservice:entersanorderintheDBandpublishesanAQmessageinthesamelocalTx
3. Dbeventprovider:picksuptheAQmessagepublishesittotheCloudEventBroker
4. Supplierservice:suppliestheInventoryserviceandpublishesanAQmessageinthesamelocalTx.
5. Inventoryservice:receivesanOrderplaceeventthenpublishesa“lackofinventoryfororderitem”event
6. Orderservice:picksupthateventandconstructsitsstatefromtheeventsourcingmechanismw/oqueryingthedatabase
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Reportoffailedordertoclient(theresultofmissinginventory)…
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OrderinsertedintableandAQmessagesentinsamelocaltransaction(src code)…
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
OrderservicelogsshowinginsertoforderandAQsend
OrderinsertedintableandAQmessagesentinsamelocaltransaction(logs)…
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
AQEventproviderserviceMediatingAQmessagereceivedfromitintoaCloudEventthatispublishedtoeventbrokerforprocessingbasedonrules.Therecanbemultiplerulesforasingleeventtype(eg ordercompletiontriggersemail,delivery,etc.notification)
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
InventoryservicereceivingeventasaresultofruleprocessingoforderplacedCloudeventandthenpublishingeventindicating(lackof)inventoryfororderitem.
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Orderservicereceivinginventoryeventandcancellingorderasaresult(settingstatustofailedandreturningittotheclient).
Choreographysaga:servicescoordinatedirectlywitheachothertomaintaindataintegrityforabusinessactivity.E.g.,the lackofinventory(intheinventoryservice/database)inthiscaseresultsinacancelled/failedorder(intheorderservice/database).
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Supplierserviceinterfacetoinventoryserviceaddsinventory…
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Reportoffailedordertoclient(theresultofinventorynowexisting)…
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Samemessage/workflowandorderisnowsuccessfulasinventoryexists.
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
CQRS(aggregate)queryprovidingdatagleanedfromeventsourcing.Dataissourcedfromeventsreceivedandeachservicebuildsandaccessesthespecificstate/representationitrequireslocallyallowingforCommandQueryResponsibilitySegregation.Theorderserviceorderdetail,eg,ineffecthasinformationfromotherservices’databases(inventoryandsuggesteditems).
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
QandA
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SeeourNextSessions@https://bit.ly/2DzV3uT
60