Distributed Systems - Brown UniversityAll Facebook Data Node A Node B Node C Shard 1 Shard 1 Shard 1 Node D Node E Node F Shard 2 Shard 2 Shard 2 Partition data into shards, maps shards

DistributedSystemsDay13:Distributed Transaction

“ToBeorNottoBe… Distributed ..Transactions”

Summary

• BackgroundonTransactions• ACIDSemantics

• DistributeTransactions• Terminology: Transactionmanager,Coordinator,Participant• TwoPhaseCommit

• AddingIsolationwithLocks:optimisticV.pessimistic• PerformanceIssues

• ConsistencyModels• Serializability VersusLinearizability

https://cloud.google.com/spanner/docs/transactions

https://blog.couchbase.com/optimistic-or-pessimistic-locking-which-one-should-you-pick/

AllFacebookData

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

Partitiondataintoshards,mapsshardstoserverwithconsistenthashing

k1 v1

k2 v2

k3 v3

k4 v4

k5 v5

k0 v0

Hash table

k4 v4

k5 v5

k4 v4

k5 v5

k4 v4

k5 v5

Maintainm

ultiplecopiesForfaulttoleranceandtoreducelatency

ClientssendrequestsToallreplicas

• Replication• Lazy,Passive,Active• Consistencyforasingleshard

• DistributedTransaction• Consistent/Atomicchangetodatain

multipleshards• Multipleshardsè Canusetraditional

replicationtechniques FE

FE

k2 v2

k3 v3

k2 v2

k3 v3

k2 v2

k3 v3

WhatisTransaction?

• Asetofoperationsthatneedtobeperformedtogether.• Example 1:transferringmoneybetweenaccounts• Example 2:shoppingcartcheckout

Read(R)Update(R,$50)Read(T)Update(T,$150)

InitiallyTheoandRodrigohave$100.Goal:Transfer$50fromRodrigotoTheo.

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

Rodrigoisinshard1 Theoisinshard2

WhatisTransaction?

• Asetofoperationsthatneedtobeperformedtogether.• Example 1:transferringmoneybetweenaccounts• Example 2:shoppingcartcheckout

Read(R)Update(R,$50)Read(T)Update(T,$150)

InitiallyTheoandRodrigohave$100.Goal:Transfer$50fromRodrigotoTheo.Ideal:eitherthewhole4operationshappenornonehappenWorstcase:onlyasubsetoccur

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

Rodrigoisinshard1 Theoisinshard2

TransactionBackground

• ACIDSemantics• Atomicity• Consistency• Isolation• Durability

• Transactionsareeasyfortraditionaldatabases• Traditional databases areonasingle serverà failure is``all-or-nothing’’

• Allcomponents ofthetransactionsfail• Distributed transactionsà differentcomponents canfail

• NeedtoprovidesTransactionsemanticswhenonlyasubset ofthecomponents fail

Allornothingsemantics:alloperationssucceedsorfails.

Intermediatestatesarenotexposedtotheoutsideworld(nopartialwritesareexposed)

Resultsofa‘committed’transactionspersistsafterthetransaction(andthroughfailures)

Transitionsfromoneconsistentstatetoanotherconsistentstate

DistributedTransactionsSemantics

• TransactionManager• Serverinchargeoforchestratingthetransaction

• Stepsfortransaction• Client initiates atransaction

• TMgivesclientaTransactionID(TID)• Client submits operationstoTM

• TMrelaysoperationstoreplicas• Client commitstransaction

• TMperformstwophasecommit

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

FE

TransactionManager

DistributedTransactionsSemantics

• TransactionManager• Serverinchargeoforchestratingthetransaction

• Stepsfortransaction• Client initiates atransaction

• TMgivesclientaTransactionID(TID)• Client submits operationstoTM

• TMrelaysoperationstoreplicas• Client commitstransaction

• TMperformstwophasecommit

Client-SideCode:tid =openTransaction();RVal =a.get(tid,Rodrigo);a.update(tid,Rodrigo,RVal - 50);Tval =b.get(tid,Theo);b.update(tid,Theo,Tval+50);

closeTransaction(tid)orabortTransaction(tid)

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

FE

TransactionManager

TransactionManager• TMhandsoutTIDs• TMmanagesandrelays

operationstoReplicaLeaders• TMkeepstrackofallReplicas

involvedinthetransactions

ReplicaManager• Prepareoperations• Storeoperationslocallyinlog• Butdonotcommitoperations

TwoPhaseCommit

TwoPhaseCommit

• ProvidesAtomicityandConsistency• NOTIsolationandDurability

• Assumptions:eachservermaintainsatransactionlog• Transactionlogisstoredinpersistentmemory

• Iffailureà itemsinTransactionLogsurvives

• Terminologychanges:• Coordinator<--- ThetransactionManager• Participantß Leaderofareplica

NodeA

NodeB

NodeC

Shard1

Shard1

Shard1

NodeD

NodeE

NodeF

Shard2

Shard2

Shard2

FE

TransactionManagerCoordinator

Participant Participant

TwoPhaseCommit

• Phase1:• Coordinatorsends requestforvotes• Participants vote

• Phase2:• Coordinatorcountsvotes• Coordinatorinformsoftransactionstatus

• AttheendofPhase2:eitherallparticipantscommitorallabort

ReadytoCommit?

CoordinatorParticipant

Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator

CoordinatorStateDiagramforTwo-PhaseCommit

Abort Commit

Wait

Init

allcommit/commitanyabort/abort

appcommit/votereq

Coordinator

Coordinator pariticpantspariticpantspariticpantspariticpants

Response

Ready?

Vote:CommitorAbort

ParticipantStateDiagramforTwo-PhaseCommit

Abort Commit

Uncertain

Init

votereq/commitvotereq/abort

abort/ack commit/ack

Participant

Coordinator pariticpantspariticpantspariticpantspariticpants

Response

Makeachange

CommitorAbort

StateDiagramsforTwoPhaseCommit

Abort Commit

Wait

Init

Abort Commit

Uncertain

Init


appcommit/votereq votereq/commitvotereq/abort


Coordinator Participant

TwoPhaseCommit

• Phase1:• Coordinatorsends requestforvotes• Participants vote

• Phase2:• Coordinatorcountsvotes• Coordinatorinformsoftransactionstatus

• AttheendofPhase2:eitherallparticipantscommitorallabort

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator

TwoPhaseCommitWithFailures• Whatistheimpactoffailureson2PC?

• 2PCissynchronous• Failure==nodefailureornetworkfailure• Failure-->theprotocolblocks/stalls

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

CoordinatorCoordinatorfails,participantswillbeuncertain(waiting)

Participantfails,coordinatorwillbewaiting

CrashPoints

Abort Commit

Wait

Init

Abort Commit

Uncertain

Init





CrashPoints

Abort Commit

Wait

Init

Abort Commit

Uncertain

Init







• DetectfailureusingTimeouts• IsthismodelSynchronousorAsynchronous?

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator

Coordinatorfails,participantswillbeuncertain(waiting)Participantfails,coordinatorwillbewaiting



• DetectfailureusingTimeouts• CoordinatordetectsparticipantfailureandassumesABORTà Transactionterminates

• ParticipantdetectsCoordinatorfailure• WhyCan’tParticipant automatically ABORT?

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator


ParticipantRecoveryfromCoordinatorFailure

Abort Commit

Uncertain

Init



Participant

• ParticipantinUncertainstate– waitingforcoordinatortosay“commit”or“abort”

• Itdetectsfailureofcoordinator– Usingtimeout

• ParticipantinUncertain state– can’tassumeeitheroutcome

- BADthingshappenifparticipantmakeswrongassumptions

– waitsforcoordinatortorestart- Onrestartcontactcoordinatorforfinaloutcome

ParticipantRecoveryfromCoordinatorFailure

Abort Commit

Uncertain

Init



Participant

• ParticipantinUncertainstate– waitingforcoordinatortosay“commit”or“abort”

• Itdetectsfailureofcoordinator– Usingtimeout

• ParticipantinUncertain state– can’tassumeeitheroutcome

- BADthingshappenifparticipantmakeswrongassumptions

– waitsforcoordinatortorestart- Onrestartcontactcoordinatorforfinaloutcome

CoordinatorRecoveryfromParticipantFailure

• Coordinatorinwaitstate– waitingforparticipanttosay“commit”or“abort”

• Itdetectsfailureofparticipant– Usingtimeout

• Coordinatorassumestheysaid‘no’– Takesnoresponseasanabort– Aborttransaction!!

• IfparticipantFails,Coordinatorcanmakeprogress

Abort Commit

Wait

Init


appcommit/votereq

Coordinator



• DetectfailureusingTimeouts• CoordinatordetectsparticipantfailureandassumesABORTà transactionterminates

• ParticipantdetectsCoordinatorfailure• Theparticipant mustwait forcoordinator• The transaction isstalled!

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator


TwoPhaseCommitandCAPTheorem!

• Duringapartition• Does2PCpickAvailabilityorConsistency?

ReadytoCommit?

Coordinator

Participant

Participant

NetworkPartition

CAPTheorem

• C:Consistency(Linearizable)• A:Availability• P:Partitiontolerance

• Givena“Partition”,youmustpickbetween“Availability”and“Consistency”• PickConsistently:Someclients(notall)canchange“data consistently”• PickAvailability:Allclientscanchangedatabut“inconsistently”

TwoPhaseCommitandCAPTheorem!

• Duringapartition• Does2PCpickAvailabilityorConsistency?

ReadytoCommit?

Coordinator

Participant

Participant

NetworkPartition



• DetectfailureusingTimeouts• CoordinatordetectsparticipantfailureandassumesABORT


ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator

Coordinatorfails,participantswillbewaitingParticipantfails,coordinatorwillbewaiting

CoordinatorassumesABORTTransactionends

ParticipantABORTEventuallytransactionends

ParticipantthatvotedNOcanabortHowever,VotedyescannotABORT



• DetectfailureusingTimeouts• CoordinatordetectsparticipantfailureandassumesABORT


ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant

CountVotes!!!!!!

Coordinator

Coordinatorfails,participantswillbewaitingParticipantfails,coordinatorwillbewaiting

CoordinatorassumesABORTTransactionends

ParticipantABORTEventuallytransactionends

ParticipantthatvotedNOcanabortHowever,VotedyescannotABORT

TwoPhaseCommit:AddingIsolationwithLocks

https://github.com/facebook/rocksdb/wiki/Transactions

https://apacheignite.readme.io/docs/concurrency-modes-and-isolation-levels

PessimisticVersusOptimisticLocking

• Trade-off:concurrencyversusisolationPessimistic:• Getalllocksbeforetransaction• Releasealllocksaftertransaction

• Releaselocksaftercommit/abort• Preventsanyoneelsefromusingthe

dataduringtransaction• Lockspreventread/writeofdata• Locksstopothertransactions

Optimistic• Nolocks• Getacopyofdatabeforetransaction• Aftertransactionchecktomakesuredata

hasnotchanged• IfthedatachangedthenABORT!!!!• Datachangesmeanssomeoneelse

changedthedata

PessimisticVersusOptimisticLocking

• Trade-off:concurrencyversusisolation

Pessimistic

HighlevelofconcurrencyHighthroughput:especiallyifallreadsManyTransactionswillabortifmanywrites

Optimistic

LowlevelofconcurrencyLowperformanceSequentialorderingoftransactions

TwoPhaseCommit:PracticalIssues

PracticalPerformanceIssueswith2PC

• Synchronization:2PCOverheads• Multiple ``rounds’’ofcommunication• Threeroundsofcommunication

• 3(N)messages• Duringthese roundsresourcesarefrozen

ReadytoCommit?


Participant

Abort/Commit?


Participant

Vote[Yes/No]


Participant


• Blocking:2PC• Duringfailureà 2PCcanblock• When2PCblocksà thenothertransactions areunabletoprogress


• Synchronization:2PCOverheads• Multiple ``rounds’’ofcommunication• Threeroundsofcommunication

• 3(N)messages• Duringthese roundsresourcesarefrozen

• Blocking:2PC• Duringfailureà 2PCcanblock• When2PCblocksà thenothertransactions areunabletoprogress

https://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf

DistributedTransactionNoLongerConsideredDead!

https://thenewstack.io/microsoft-orleans-brings-distributed-transactions-to-cloud/

2019:MS’sOrleansDist.TransactiontotheCloud!!!

2007:AvoidDistributedtransactions

https://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf

2012:Google’sSpannerDist.Transaction!!!

DistributedTransactionsandACID

HowdoyougetACIDinDistributedTransactions?• TwoPhaseCommità A+C

• 2PC:atomicandconsistent changefromonestatetoanotherstate

• TwoPhaseCommit+Locksà A+C+I• Locksprovideisolation bypreventingconcurrenttransactionfromaccessingdata

• TwoPhaseCommit+Locks+Logsà A+C+I+D• Logsensurethetransactions persists

ConsistencyModelsRevisited

ConsistencySpectrum

StrictSerializability

Linearizable

Sequential

Causal+

Eventual

WEAKCONSISTENCY

STRONGCONSISTENCY

SLOWERBUTEASYTOPROGRAM

FASTBUTHARDERTOPROGRAM


• Totalorder+FIFO+“Time”à foratransaction• Afteratransactioncommits,allfuturereadswillseecommitteddata

• Requires2PC+pessimisticLocks• Lowperformance:reads/writehavehighlatencyandlowthroughput

• StrictSerializability V.Linearizability• StrictSerializability =Linearizability forTransactions• Linearizability =Totalorder+FIFO+Realtimeforindividualoperations• StrictSerializability =Totalorder+FIFO+Realtimefortransactions(groupsofoperations)


• Totalorder+FIFO+“Time”à foratransaction• Afteratransactioncommits,allfuturereadswillseecommitteddata

• Requires2PC+pessimisticLocks• Lowperformance:reads/writehavehighlatencyandlowthroughput

• StrictSerializability V.Linearizability• StrictSerializability =Linearizability forTransactions• Linearizability =Totalorder+FIFO+Realtimeforindividualoperations• StrictSerializability =Totalorder+FIFO+Realtimefortransactions(groupsofoperations)

Summary

• BackgroundonTransactions• ACIDSemantics

• DistributeTransactions• Terminology: Transactionmanager,Coordinator,Participant• TwoPhaseCommit

• AddingIsolationwithLocks:optimisticV.pessimistic• PerformanceIssues

• ConsistencyModels• Serializability VersusLinearizability

Distributed Systems - Brown UniversityAll Facebook Data Node A Node B Node C Shard 1 Shard 1 Shard 1 Node D Node E Node F Shard 2 Shard 2 Shard 2 Partition data into shards, maps shards

Text of Distributed Systems - Brown UniversityAll Facebook Data Node A Node B Node C Shard 1 Shard 1 Shard 1...

The Shard at London Bridge · 2018-02-14 · 18 TheStructuralEngineer July 2014 The Shard Project focus › The Shard at London Bridge R. Agrawal BA, MSc, DIC, CEng, MIStructE, MIET

Shards - Presentation

Summer Shard, South Petherton, TA13 £780 per month€¦ · Summer Shard, South Petherton, TA13 This lovely two bedroom bungalow located in the popular and quiet Summer Shard development

The Shard Story – Developing The Shard · PDF fileThe Shard Story – Developing The Shard. ... vine Sellar met Renzo Piano for the first time in Ir ... y run at the steel-fabricator’s

A Single Shard Student Sheet - krazykrayon.co.za

11 Shard Price

Search and Visualization Platform Building A Billion ... · A Solr custom “URP” was developed to route tweets to the right by-month shard. It auto creates and deletes shards

Shard London Bridge Tower

Andrey Makarov - The Shard of Lemuria

A Single Shard Power Point Cee

Using Oracle Sharding...10.6.6.2 Adding Shards to a System-Managed SDB 10-22 10.6.6.3 Replacing a Shard 10-26 10.7 Chunk Management 10-29 10.7.1 About Moving Chunks 10-30 10.7.2 Moving

A Single Shard

Oracle Sharding Technical Deep Dive - NYOUGnyoug.org/.../2018/10/Oracle_Sharding_Technical_Deep_Dive_NYOUG.pdf · Oracle Sharding –Compatibility Requirements for Shards Each shard

BUSINESS TERMS & CONDITIONS SHARD CAPITAL PARTNERS LLP€¦ · Shard Capital offers discretionary management services. If you opt to use this service, Shard Capital will manage, on

Single Shard: Chapters 4-6 Vocabulary

Elasticsearch Tutorial - Engage Consulting€¦ · or more replicas. Each node hosts one or more shards, and acts as a coordinator to delegate operations to the correct shard(s)

THE SHARDS -LONDON.pptx

A Single Shard - Reed Novel Studies

Porcelain shards from Portuguese wrecks: Raman

Using Oracle ShardingAdding Shards to a System-Managed SDB 10-26 Replacing a Shard 10-31 Chunk Management 10-34 About Moving Chunks 10-34 Moving Chunks 10-35 About Splitting Chunks

Shard spec brochure july 2011

Efficient MRC Construction with SHARDS

121002 Malmo Shard Presentation

ESGF + DOCKER · ESGF/Docker Solr Cloud Architecture Shard 1 Shard 2 Shard 3 ESGF SOLR CLOUD (Jetty) Shard 1 Shard 2 Shard 3 ESGF SOLR CLOUD (Jetty) Shard 7 Shard 8 Shard 9 •Solr-Cloud

RapidChain: Scaling Blockchain via Full Sharding · "Evict !nodes from the shard!Bounded Cuckoo rule. Experimental Setup!Prototype implemented in Go!Link latency of 100 ms!Node bandwidth

Inter-controller communication in switching SDN architecture...OpenDayLight divides this datastore into shards. Then, subsets of shards are distributed to every cluster member. Shards

Elasticsearch shards, index, filters and queries

Shards of the Throne Rules

Glaze analysis of shards and plates

Cut & Paste Shard Art

Documents

Distributed Systems - Brown UniversityAll Facebook Data Node A Node B Node C Shard 1 Shard 1 Shard 1 Node D Node E Node F Shard 2 Shard 2 Shard 2 Partition data into shards, maps shards

Text of Distributed Systems - Brown UniversityAll Facebook Data Node A Node B Node C Shard 1 Shard 1 Shard 1...