Synchronization is Coming Back But is it the Same? · c Michel Raynal Synchronization is coming back! 44. A NON-BLOCKING timestamp object: deﬁnition • Validity. No two invocations

Synchronization is Coming Back

But is it the Same?

Michel RAYNAL

[email protected]

Institut Universitaire de France

IRISA, Universite de Rennes, France

Dpt of Computing, Polytechnic Univ., Hong Kong

c© Michel Raynal Synchronization is coming back! 1

Motivation

• Synchonization is coming back, but is it the same?

• Lots of new concepts and mechanisms since locks!

• The multicore (r)evolution

• The synchronization world has changed


A recent book on the topic

Concurrent programming:Algorithms, Principles,and Foundations

by Michel Raynal

Springer, 515 pages, 2013

ISBN: 978-3-642-32026-2

6 Parts, composed of 17 ChaptersBalance: algorithms vs foundations


Part I

PART 1: Lock-Based Synchronization

• Chapter 1: The Mutual Exclusion Problem

• Chapter 2: Solving Exclusion Problem

• Chapter 3: Lock-Based Concurrent Objects


Part II

PART 2: On the Foundations Side: the Atomicity Concept

• Chapter 4:

Atomicity: Formal Definition and Properties


Part III

PART 3: Mutex-free Synchronization

• Chapter 5: Mutex-Free Concurrent Objects

• Chapter 6: Hybrid Concurrent Objects

• Chapter 7:Wait-Free Objects from Read/write Registers Only

• Chapter 8:Snapshot Objects from Read/Write Registers Only

• Chapter 9:Renaming Objects from Read/Write Registers Only


Part IV

PART 4: The Transactional Memory Approach

• Chapter 10: Transactional Memory


Part V

PART 5: On the Foundations Side:

from Safe Bits to Atomic Registers

• Chapter 11:Safe, Regular, and Atomic Read/Write Registers

• Chapter 12:From Safe Bits to Atomic Bits:

a Lower Bound and an Optimal Construction

• Chapter 13:Bounded Constructions of Atomic b-Valued Registers


Part V

PART 6: On the Foundations Side:

the Computability Power of Concurrent Objects

• Chapter 14: Universality of Consensus

• Chapter 15: The Case of Unreliable Base Objects

• Chapter 16:Consensus Numbers and the Consensus Hierarchy

• Chapter 17: The Alpha(s) and Omega of Consensus


Summary

• Computing Model: Processes and Concurrent Objects

• Base read/write model, more powerful models

• On the safety side: Linearizability

• On the liveness side: Progress conditions

• Mutex-free concurrent objects

• Comuptability power, consensus number, etc.

• Hybrid concurrent objects

• Conclusion


Part I

MODEL and DEFINITIONS


Asynchronous process model (1)

• Computing model: n sequential processes Π = p1, . . . , pn(Turing machines)

• Timing model: Asynchrony: No upper bound on thetime required to execute a computation step

• Failure model

⋆ No failure⋆ Crash failure


Asynchronous process model (2)

• Process crash: a process behaves according to its spec-ification until it possibly crashes, i.e., halts prematurely(after it has crashed a process is definitely stopped)

• Terminology wrt a run:

⋆ A Correct process is a pi that never crashes

⋆ A Faulty process is a pi that crashes


Asynchronous Base Communication Model (1)

Shared memory provides processes with

• Reliable Read/Write atomic registers

• Reliable Compare&Swap atomic registers

• Atomic means that:

⋆ The operations on a register appear as if they havebeen executed sequentially

⋆ Each operation appears as being executed instanta-neously at some point of the time line between itsstart event and its end event


Atomic register

Real time line

WRITE: 2WRITE: 1

READ: 1

WRITE: 3

Value is 1 3 2

READ: 2

READ: 2

- Lamport L., On interprocess Communication (part 1: Basic Formalism, Part 2:Algorithms), Distributed Computing, 1(2):77-101, 1986

- Herlihy M.P. and Wing J.L., Linearizability: a Correctness Condition for ConcurrentObjects. ACM Trans. on Programming Languages and Systems, 12(3):463-492,1990


Other base (hardware) operations

• Test&set

• Fetch&add

• Compare&swap

• LL/SC

• Swap, Mem-to-Mem SWAP

• etc.


Structural view (Base level)

(could be shared disks, SAN)

p1

pn local memorypi local memory

pi

p1 local memory

Asynchronous shared memoryAbstraction

pn


Part II

On the SAFETY side


SAFETY PROPERTY

LINEARIZABILITY

- Herlihy M.P. and Wing J.M., Linearizability: a correctnesscondition for concurrent objects. ACM Toplas, 12(3):463-492, 1990

- Lamport L., On interprocess Communication (part 1:Basic Formalism, Part 2: Algorithms), Distributed Com-puting, 1(2):77-101, 1986


Concurrent Object: example

An object accessed by concurrent processes

p1 pi pn

Enqueue (v) r ← Dequeue ()


Sequential vs Concurrent (1)

p1

p2

Enq (a)

Enq (c) Deq (a|b|c) ?

Deq (a|b|c) ?Enq (b)

Enq (a) Enq (c) Enq (b) Deq (a)

SEQUENTIAL:

CONCURRENT:

Deq (c)


History (Run, Execution)

p1

p2

Enq (a)



Res (Enq(b))

Inv (Enq(b))History H

A history defines a partial order on the operations

Sequences of events


Two types of Concurrent Objects

• A concurrent object encapsulates a particular synchroproblem

• Concurrent objects with a sequential specification

Queue, file, graph, tree, stack, ...

⋆ Fifo queue: producer/consumer problem

⋆ File (register): Readers/writers problem

• Concurrent objects with a not-sequential specification

Rendezvous Object,Interactive Consistency,Non Blocking Atomic Commit Object, ...


Example: Interactive consistency

Each process pi proposes a value vi

and has to decide a value such that:

• Validity. Let D[1..n] be the vector decided by a process.

∀i ∈ [1..n] : D[i] ∈ vi,⊥ and D[i] = vi if pi is correct

• Agreement. No two processes decide different vectors

• Termination. Every correct process decides


Linearizability

An execution (or “history”) H is linearizable if the opera-tions issued by the processes appear as if they have beenexecuted in some sequential order that respects:

• The seq order in each process

• The seq specification of the object

• The real-time order of the non-overlapping operations



p1

p2

Enq (a)



Enq (b)

Enq (c)Enq (a) Deq (b)

Deq (a)



p1

p2Deq (a|b|c) ?


Enq (a)Enq (c)

Enq (c)

Enq (a)

Enq (b) Deq (c)

Deq (a)


Locality of the Linearizability property (1)

• A property P of a concurrent system is Local if thesystem as a whole satisfies P whenever each individualobject satisfies P

Locality means that, given two objects X and Y , eachsatisfying the property P , the composite object [X,Y ]satisfies the property P

• Linearizability is a local property (consistency criterion)


Locality of the Linearizability property (2)

• Theorem: A history H is linearizable iff, for each objectX, H restricted to the operations on X is linearizable

Locality means that, given two linearizable objects Xand Y , the composite object [X,Y ] is linearizable

• It follows that each object can be implemented indepen-dently from the others (this is a fundamental propertyfrom both theoretical and practical point of views)

• Sequential consistency, some forms of serializability arenot local properties


Sequential Consistency

• An execution (or “history”) H is sequentially consistentif the operations issued by the processes appear as ifthey have been executed in some sequential order thatrespects each process order and the seq specificationof every object

Q.Enq(a)

Q.Deq(b)Q.Enq(b)

Q.Enq(b) Q.Enq(a) Q.Deq(b)

A ”witness” seq history:


Sequential Consistency is not Local

Q.Enq(a)

Q.Enq(b)Q′.Enq(a′)

Q′.Enq(b′) Q′.Deq(b′)

Q.Deq(b)

• Q and Q′ are sequentially consistent

• the whole history H is not sequentially consistent

Impossibility to produce a “witness” sequential history

• See distributed caches


Part III

On the LIVENESS side

PROGRESS CONDITIONS


Two cases

• Reliable systems:

⋆ Lock-based implementations

⋆ Progress conditions

∗ Deadlock-freedom

∗ Starvation-freedom

• Crash-prone systems:

⋆ Mutex-free implmentations

⋆ Associated progress conditions


MUTEX-FREEDOM

• No code is protected by a critical section

R3

O.op1() by p1 O.op1() by p3O.op2(b) by p2

R1 R1 R1R2 R2 R2R3 R3 R3


Abortable object

• If an operation executes alone (concurrency-free con-text), it terminates and returns a correct value

• If an operation executes in a concurrency context, itterminates and returns a correct value or it returns ⊥and has no effect on the object

This definition is different from the one used by Aguilera M.K., Frolund S., Hadzila-cos V., Horn S.L. and Toueg S., Abortable and Query-abortable Objects and theirImplementations. PODC’07, pp. 23-32, 2007


Obstruction-freedom

• If an operation executes alone (concurrency-free con-text), it terminates and returns a correct value

• If an operation executes in a concurrency context,

⋆ it can never termination⋆ if it terminates, its result is correct

Abortability is a stronger notion than obstruction-freedom


Non-blocking

• In a concurrency context, at least one operation termi-nates

• In a failure-free system: Non-blocking = Deadlock-free


Starvation-free

• Any operation invocation terminates

• Failure-prone systems where t process may crash:called t-resilient systems

• Systems where t = n− 1: called wait-free systems

• Difference between wait-freedom and starvation-freedomis failure-free systems


Hierarchy of progress conditions in failure-prone systems

• Obstruction-freedom ≺ Non-blocking ≺ Wait-freedom

• Abortable ≺ Wait-freedom


Part IV

Simple mutex-free implementations

of concurrent objects


Introductory example: the SPLITTER

x processes

right

down

≤ x− 1 processes

≤ x− 1 processes≤ 1 process

stop


Splitter: Specification

• At most one process exits with stop

• At most (x− 1) processes exit with right

• At most (x− 1) processes exit with down

Lamport 1987, Anderson-Moir 1995.


Splitter: WAIT-FREE Implementation

Two shared registers: LAST (init ∀) and DOOR (init open)

procedure direction() % issued by pi %

LAST ← idi;if DOOR = closed then movei← right

else DOOR← closedif (LAST = idi) then movei ← stop

else movei ← downendif endif;return (movei)


Splitter: Proof (very easy)

LAST ← i LAST = iDOOR ← closed

No process pj has modified the atomic register LAST


A NON-BLOCKING timestamp object: definition

• Validity. No two invocations of get timestamp() returnthe same value

• Consistency. Let gt1() and gt2() be two distinct invo-cations of get timestamp().

If gt1() returns before gt2() starts, the timestamp re-turned by gt2() is greater than the one returned by gt1()

• Termination. Obstruction-freedom


A non-blocking timestamp object: base objects

• NEXT : next integer timestamp value (init to 1)

• LAST : unbounded array of atomic registers.

A process pi deposits its index i in LAST [k] to indicateit is trying to obtain the timestamp k

• COMP : unbounded array of atomic Boolean registers(all init to false)

A process pi sets COMP[k] to true to indicate that it iscompeting for the timestamp k (several processes canwrite true into COMP [k])


A non-blocking timestamp object: implementation

operation get timestamp(i) isk ← NEXT ;repeat forever

LAST [k]← i;if (¬COMP [k])

then COMP [k]← true;if (LAST [k] = i)

then NEXT ← NEXT +1; return(k)end if

end if;k ← k +1end repeat

end operation.


A wait-free stack: model and base objects

• Read/write model enriched with Fetch&add and swapoperations

• Base atomic objects

⋆ REG[0..∞): array of atomic registers which containsthe elements of the stack (init to ⊥)

REG[0] sentinel

⋆ NEXT : atomic register that contains the index of thenext entry where a value can be deposited (init to 1)


A wait-free stack: push algorithm

operation push(v) isin← NEXT .fetch&add()− 1;REG[in]← v;return()

end operation.


A wait-free stack: pop algorithm

operation Q.pop() islast← NEXT − 1;for x from last to 0 do

aux← REG[x].swap(⊥);if (aux 6= ⊥) then return(aux) end if

end for,return(empty)

end operation.

wait-freedom vs bounded wait-freedom


Part V

Boosting

IMPLEMENTATIONS


Boosting progress with contention managers

• From Obstruction-freedom to wait-freedom

Failure detector: P (eventually perfect)

• From non-blocking to wait-freedom

Failure detector: ΩX


Boosting progress with contention managers

Contention manager

operationm()operation1()

need help()

stop help()

Obstruction-free

implementation

Failure detector-based


From an obstruction-free to a non-blocking timestamp

operation get timestamp(i) isk ← NEXT ;repeat forever

LAST [k]← i;if (¬COMP [k])

then COMP [k]← true;if (LAST [k] = i)

thenNEXT ← NEXT +1;CM.stop help(i); return(k)

end ifend if;

k ← k +1; CM.need help(i)end repeat

end operation.


From obstruction-freedom to non-blocking

operation CM .need help(i) isNEED HELP [i]← true;repeat x← j | NEED HELP [j]

until (ev leader(x) = i)end repeat;return()

end operation.

operation CM .stop help(i) isNEED HELP [i]← false; return()

end operation.


Part VI

Read/write-based WAIT-FREE

IMPLEMENTATIONS


The (Static) Renaming Problem

• id1, . . . , idn are the initial identities of the processes

• id1, . . . , idn ∈ [0..N − 1], where N >>>>> n

• Each process has to acquire a new name in the set[0..M − 1]

• No two processes can get the same name

• Type of renaming:

⋆ Static renaming: a name is acquired once for all

⋆ Dynamic renaming: names are (acquired; released)∗

• Theoretical result: M ≥ 2n− 1 (Herlihy-Shavit)


Moir-Anderson’s Protocol

• Considers M = n(n+1)/2

• Basic idea: a grid of splitters


Grid of Splitters

0

1

2

4

3

0 1 2 3 4ki

li

0 21 3 4

5 6 7

9 10 11

12 13

14

8


The wait-free algorithm

operation get name(idi)

di ← 0; ri← 0; movei← down;while (movei 6= stop) do

movei← splitter[di, ri].direction();case (movei = right) then ri← ri +1

(movei = down) then di ← di +1(movei = stop) then exit loop

end caseend whilereturn (n× di + ri − (di(di − 1)/2))% the new name is the position [di, ri] in the grid %


A snapshot Object

• Keeps data provided by processes

• When pi invokes store(v) it defines v as its lastdeposited value

• A process invokes snapshot to get the valuesdeposited by the processes

• Everything has to appear as if the operationswere executed instantaneously (at some timebetween their invocation and their termina-tion)


Snapshot operations

snapshot() by pj, ∀jupdate(v) by pi

SM[1] SM[n]

SM[i]


Underlying idea

operation update (v)sni ← sni +1; % local seq number generator %SM[i]← (v, sni) % atomic write %

operation snapshotwhile true doAi← scan; Bi← scan;% double “asynchronous” scan %if (∀j : Ai[j].sn = Bi[j].sn) then return (Ai.val) end if

% Ai.val = [Ai[1].val, . . . , Ai[n].val] %end while


Snapshot: Partial proof (easy)

REG[1]

REG[4]

REG[3]

REG[2]

first scan() second scan() snapshot() operation

time line

bbi[1].sn = a

bbi[3].sn = c

aai[2].sn = b

aai[3].sn = c

bbi[2].sn = b

aai[4].sn = d

aai[1].sn = a = SM [1].sn

bbi[4].sn = d = SM [4].sn

linearization point of the snapshot() operation


How an update can help a snapshot

pi

pj

up1

snap

snap int

up2


Afek et al.s algorithm (1)

operation update (v):help arrayi← snapshot();sni ← sni +1;SM[i]← (v, sni, help arrayi)


Afek et al.s algorithm (2)

operation snapshot:could helpi ← ∅;while true doAi← scan; Bi← scan; % double “asynch” collect %if (∀j : Ai[j].sn = Bi[j].sn)

then return (Ai.val)else for j : 1 ≤ j ≤ n do

if (Ai[j].sn 6= Bi[j].sn) thenif (j ∈ could helpi)

then return (Bi[j].help array)else could helpi ← could helpi ∪ j

end if end ifend for

end ifend while


Snapshot: Proof

update()

successful double scan

update()

pisnapshot()

pjupdate() update()

snapshot()

pk

help array

help array

snapshot()


Part VII

HYBRID

IMPLEMENTATIONS


Definition

An implementation of a concurrent object is

• Mutex-free if it does not use locks

• Static Hybrid if

⋆ It uses locks only for some operations

• Dynamic Hybrid if

⋆ It never uses locks in “favorable circumstances”


Example

• Static hybrid set:

⋆ add() and remove(): lock-based

⋆ belong(): no lock

• Static hybrid stack/queue

⋆ In concurrency-free contexts: no lock and a boundednumber of steps

⋆ In concurrency contexts: locks can be used


Hybridism vs adaptivity

• Hybridism is wrt to lock and concurrency

• Adaptivity is wrt the concurrency degree (cost has todepends on the numb of competing processes only)


Part VII

A MUTEX-FREE

ABORTABLE STACK


Underlying system

• Atomic read/write registers

• Compare&swap objects


Compare&Swap: definition

Compare&Swap is a conditional write

primitive X.C&S(old, new):if (X = old) then X ← new; return(true)

else return(false)end if.


Compare&Swap: use by pi

• Let X = a

• pi reads a from register X

• It then does computation and computes a new value cfor X

• Finally pi wants to assign c to X if and only if X hasnot been modified since it read it

To that end pi issues X.C&S(a, c)


Compare&Swap: the ABA problem

Unfortunately the previous use of Compare&Swap is incor-rect!

• Initially X = a

• At time τ1: pi reads a from X

• At time τ2 > τ1:pj successfully executes X.C&S(a, b) (X = b)

• At time τ3 > τ2:pj successfully executes X.C&S(b, a) (X = a)

• At time τ4 > τ3:pi successfully executes X.C&S(a, b) and erroneously be-lieves that X has not been modified by another processin the interval [τ1..τ4]


Solving the ABA problem

Associate a new sequence number with every X.C&S

• X is now a pair 〈a, sn〉

• At time τ1:pi reads 〈a, sn〉 from X

• At time τ2 > τ1:pj successfully executes X.C&S(〈a, sn〉, 〈b, sn+1〉)

• At time τ3 > τ2:pk successfully executes X.C&S(〈b, sn+1〉, 〈a, sn+2〉)

• At time τ4 > τ3:when pi executes X.C&S(〈a, sn〉, 〈c, sn + 1〉), the writeinto X fails and returns false to pi


Stack operations

• The stack is of size k

• Operation push(v)

⋆ returns full if the stack is full, otherwise

⋆ adds v to the top of the stack and returns done

• Operation pop()

⋆ returns empty if the stack is empty, otherwise

⋆ suppresses the value from the top of the stack andreturns it


Stack representation (1)

• An array STACK [0..k] of atomic registers

• ∀x : 0 ≤ x ≤ k : STACK [x] has two fields

⋆ STACK [x].val contains a value

⋆ STACK [x].sn contains a seq number (used to preventthe ABA problem on this register)

It counts the nb of successful writes on STACK [x]

∀x : 1 ≤ x ≤ k : STACK [x] initialized to 〈⊥,0〉

• STACK [0] always stores a dummy entry (init to 〈⊥,−1〉)


Stack representation (2)

• A register TOP that contains the index of the top ofthe stack plus the corresponding pair 〈v, sn〉

• TOP initialized to 〈0,⊥,0〉

• Both STACK [x] and TOP are modified with Compare&Swap


Principle: laziness + helping mechanism

• A push or pop operation

⋆ updates TOP, and

⋆ leaves to the next operation the corresponding updateof the stack

Hence it helps the previous (push or pop) operationby modifying the stack accordingly

Shafiei N.,Non-blocking Array-based Algorithms for Stacks and Queues.Proc. th Int’l Conference on Distributed Computing and Networking (ICDCN’09),Springer Verlag LNCS #5408, pp. 55-66, 2009


Abortable push: weak push()

operation weak push(v):(index, value, seqnb)← TOP ;help(index, value, seqnb);if (index = k) then return(full) end if;sn of next← STACK [index+1].sn;newtop← 〈index+1, v, sn of next+1〉;if TOP .C&S(〈index, value, seqnb〉, newtop)

then return(done) else return(⊥) end if.


Abortable stack: help procedure

procedure help(index, value, seqnb):stacktop← STACK [index].val;STACK [index].C&S(〈stacktop, seqnb− 1〉, 〈value, seqnb〉).


Abortable pop: weak pop()

operation weak pop():(index, value, seqnb)← TOP ;help(index, value, seqnb);if (index = 0) then return(empty) end if;belowtop← STACK [index− 1];newtop← 〈index− 1, belowtop.val, belowtop.sn+1〉;if TOP .C&S(〈index, value, seqnb〉, newtop)

then return(value) else return(⊥) end if.


From an abortable to a non-blocking stack

operation non blocking push(v):repeat res← weak push(v) until res 6= ⊥ end repeat;return(res).

operation non blocking pop():repeat res← weak pop() until res 6= ⊥ end repeat;return(res).


Part VIII

A Dynamic Hybrid

STARVATION-FREE STACK


From a non-blocking lock to a starvation-free lock (1)

• A non-blocking lock denoted LOCK

• A boolean register FLAG[i ] (initialized to false) thatprocess pi sets to true when it wants to obtain thestarvation-free lock


From a non-blocking lock to a starvation-free lock (2)

operation starvation free lock():FLAG[i]← true;wait ((TURN = i) ∨ (¬FLAG[TURN ]));LOCK .lock().

operation starvation free unlock():FLAG[i]← false;if (¬FLAG[TURN ])

then TURN ← (TURN mod n) + 1end if;LOCK .unlock();


Operation and additional data structures

• Operations strong push() or strong pop()

denoted strong push or pop()

• A boolean register CONTENTION (initialized to false)that is set to true by a process when it executes theunderlying weak operation(par) operation.


The implementation (par = v for push() and ⊥ for pop())

operation strong push or pop(par):if (¬CONTENTION )

then res← weak push or pop(par);if (res 6= ⊥) then return(res) end if

end if;starvation free lock();CONTENTION ← true;repeat res← weak push or pop(par) until (res 6= ⊥);CONTENTION ← false;starvation free unlock();return(res).


Proof

• Lemma 1:If a process pi returns from its strong push(v) or strong pop()invocation, it returns a non-⊥ value

• Lemma 2:If, while executing a strong push(v) or strong pop(), a pro-cess pi reads true from CONTENTION at first line orobtains res = ⊥ at the 2nd line, it eventually obtainsthe lock

• Theorem:Any invocation of strong push() or strong pop() returns anon-⊥ value, and all invocations are linearizable

Moreover, the algorithm is contention-sensitive: any op-eration invoked in a contention-free context is lock-freeand accesses six times the shared memory


When there are failures

• The abortable and non-blocking implementations copewith any number of process crashes

• The contention-sensitive implementation is wait-free IFno process crashes between the invocation of lock() andthe return of unlock()


Part IX

THE fundamental ISSUE


The fundamental problem

• Wait-free implement linearizable concurrent objects inpresence of process crashes

Compose base objects to get “more powerful” objects

Problem: Given two objects A and X, is therea wait-free implementation of A by X in asystem of n processes (prone to crashes)?

• Wait-free = live with respect to any number of processcrashes


Example

• Example: Build a wait-free queue A from read/writeatomic registers in a system where up to n−1 processescan crash

x base[1] x base[k]· · · · · ·

X: t-fault tolerant Compare&Swap object k = t+1

x base[2]

• Is it possible??


Herlihy’s Result

• An object is universal if it can be used to wait-free im-plement any object (that has a seq specification)

• Main result (Herlihy 1991):

Consensus is a universal object

• Any concurrent object (that has a sequential speci-fication) can be built from a consensus object (andread/write registers)

• A Universal construction is an algorithm that, given thesequential specification of an object, constructs a cor-responding concurrent object (from consensus objectsand registers)


The CONSENSUS Problem: Definition

Each process proposes a value and has to decide a valuein such a way that:

• Termination: Every correct process eventuallydecides some value

• Validity: If a process decides v, then v wasproposed by some process

• Uniform Agreement: No two (correct or not)processes decide differently


CONSENSUS in Action! (Concept)

Let X be a consensus object (a single operation Propose(v))

• Consensus is a Synchronization tool:Consensus makes a decision irreversible (only one ac-tion/value succeeds)

x1← X.Propose(a) at t1 and x2← X.Propose(b) at t2we always have x1 = x2 (either a or b) whatever theinvocation times t1 and t2

• Consensus solves Non-determinism:Consensus makes a result UNIQUE

Let f be a non-deterministic function

If x1← X.Propose(f(a)) and x2← X.Propose(f(a))we always have x1 = x2


... But there is a bad news: The Main Result

Fischer-Lynch-Paterson’s Impossibility result (1985)

There is no protocol that solves theconsensus problem in asynchronoussystems (shared memory or messagepassing) that is subject to even a singleprocess crash failure

Fischer M.J., Lynch N.A. and Paterson M.S., Impossibilityof Distributed Consensus with One Faulty Process. Journalof the ACM, 32(2):374-382, 1985.


Herlihy’s Results

• Which objects allow implementing a consensus object?

• Idea: investigate the synchro power of base objects

Each object has a consensus number

The consensus number of X is the largest n for whichX solves consensus among n processes

• FLP ⇒ CN(Read/Write atomic variables)=1Which means that read/write operations are not pow-erful enough to solve consensus in presence of even asingle crash when n > 1

• Which is the synchronization power needed to solveconsensus in a SM asynchronous system?


Herlihy’s Results

• As the synchronization power of read/write atomic vari-ables is too weak to implement consensus, are they syn-chronization primitives powerful enough to solve con-sensus in a SM asynchronous system?

• A few synchronization primitives (synchr objects):

⋆ Test&Set (shared)=[prev ← shared; shared← 1; return (prev)]

⋆ Swap (local, shared)= [local↔ shared]

⋆ Move (shared1, shared2)= [shared1↔ shared2]

⋆ Compare&Swap (shared, old, new)=[prev ← shared; if prev = old then shared ← new fi;return (prev)]


Herlihy’s Results

• Consensus numbers define a hierarchy on the power ofsynch primitives

⋆ CN(Test&Set) = CN(Swap) = CN(Stack)=

CN(Fetch&Add) = CN(Fifo Queue) = . . . = 2

⋆ CN(Move) = CN(Compare&Swap)= +∞


Consensus with Test&Set (2 processes)

• Let shared shared variable init to 0

• Let prefer [0], prefer [1] be two shared variables init to ⊥

Propose (v) = % issued by pi (i = 0 or 1) %

prefer [i]← v;val← Test&Set (shared);case val = 0 then return (v)

val = 1 then return (prefer [1− i])endcase

The “winner” is the first that executes Test&Set(shared)


Consensus with a FIFO Queue (2 processes)

• Let queue be a shared queue init to < 0,1 >(return from an empty queue returns ⊥)

• Let prefer [0], prefer [1] be two shared variables init to ⊥

Propose (v) = % issued by pi (i = 0 or 1) %

prefer [i]← v;val←Dequeue (queue);case val = 0 then return (v)

val = 1 then return (prefer [1− i])endcase

The “winner” is the first process that dequeues 0


Consensus with Compare&Swap (n processes)

• Let shared shared variable init to ⊥

Propose (v) = % issued by pi (i = 1,2, . . . , n) %

Let first be a local variable;first←Compare&Swap (shared,⊥, v)if first = ⊥

then return (v)else return (first)

endif

The “winner” is the first that deposits its v in shared


An Observation

• With Test&Set or FIFO Queue:the shared register used to synchronize (determine awinner) and the shared register used to store the decidedvalue are distinct registers

• With Compare&Swap:the shared register used to synchronize (determine awinner) and the shared register used to store the decidedvalue are the same register


Herlihy’s Hierarchy

Herlihy’s Hierarchy

Read/Write objects

Test-and-Set, Swap, Queue, Stack, ...

Consensus object

Comp-and-Swap, LLCS, Move, ...

CN = 1

CN = 2

CN = +∞


What has been learnt from Herlihy’s Hierarchy

• Test&Set, Swap, Fetch&Add, etc. are synchronizationprimitives that are too weak to implement reliable ob-jects in presence of process crashes

A hierarchy on the power of synchr primitives when ad-dressing fault-tolerance

• Additional synchr power is required to design objectstolerant to process crashes.

Object combination does not always work! (e.g., atomicread/write objects do not allow the construction of moresophisticated objects)

FLP means (here) that fault-masking can be impossibleto achieve when solving non-trivial problems if we do notrely on powerful enough synch primitives


Part X

A UNIVERSAL CONSTRUCTION

- Chandra T. and Toueg S., Unreliable Failure Detectorsfor Reliable Distributed Systems. Journal of the ACM,43(2):225-267, 1996.

- Herlihy M.P., Wait-free synchronization. ACM Toplas,11(1):124-149, 1991.

- Guerraoui R. and Raynal M., Fault-Tolerance Techniquesfor Concurrent Objects. Tech Report # 1667, 22 pages,IRISA, Universite de Rennes 1 (France), December 2004http://www.irisa.fr/bibli/publi/pi/2004/1667/1667.html


The Specification of the Objects

• X an object of type T

• Operations: X .op(param) (returns always a response)

• The type of X is defined by a transition function δ()

⋆ δ(sx,op(param)) is a non-empty set of (sx′, res) pairsdefining all the possible “results” we can obtain whenthe object X is in the state sx

⋆ Each pair (sx′, res) is such that sx′ is a possible newstate of X , and then res is the corresponding resultreturned by op(param)

⋆ If the set has only one pair, the type T is deterministic

Otherwise, it is non-deterministic


Local data structures

• sxi is a local variable containing the state of the objectX as currently known by pi

• next sni[1..n] is a local array local of sequence numbers

next sni[j] (initialized to 1) is the next sequence numberthat, to pi’s knowledge, pj will associate with its nextoperation on X

• propi (a list), execi (a list), resulti (an invocation result)and ki (an integer) are auxiliary variables

Notation:

execi[r] = rth element of the list execi

|execi| = size of the list execi


Shared Base Objects

• LAST OP [1 ..n]: shared array of 1WnR atomic registers

Only pi can write LAST OP [i ]

Each register LAST OP [j ] has two fields:

⋆ LAST OP [j].op: last operation invoked by pj

⋆ LAST OP [j].sn: associated sequence number

⋆ Each entry of the array is initialized to (⊥,0)

• A list of consensus objects CONS [k ] for k = 1,2, . . . (usedsequentially by each process)

A process invokes CONS [k ].propose (v)


Universal construction (1); User interface

when operation X .op(param) is invoked by pi:resulti ← ⊥;LAST OP [i]← (op(param),next sni[i]);wait until (resulti 6= ⊥);return (resulti)


Universal construction (2): Background task

while (true) do

Step 1: Build a proposalpropi ← ǫ; % empty list %for 1 ≤ j ≤ n do

if (LAST OP [j].sn ≥ next sni[j]) thenadd (LAST OP [j].op, j) to propi end if;

end for;

Step 2: Try to commit the proposalif (propi 6= ǫ) then ki← ki +1;

.......end if

end while


Universal construction (3): Background task

Assume first that the type T of the object is deterministic

execi← CONS [ki].propose (propi);let ℓ = |execi|;for r from 1 to ℓ do

(sxi, res)← δ(sxi, exec[r].op);let j = execi[r].proc;next sni[j]← next sni[j] + 1;if (i = j) then resulti ← res end if

end for


The case of non-deterministic types

• Brute force strategy:

Consider a deterministic reduction δ′() of δ()

• A nicer solution:

Use the consensus invocation to solve ordering + singleresult of each operation + single resulting state


Correctness proof: The object is live and safe

• Wait-free property:

Show that each operation invoked by a correct processterminates whatever the behavior of the other processes

This is the liveness property stating that the implemen-tation ensures the operations do terminate

• Linearizability (semantics of the objet):

From an external observer point of view, the operationson the concurrent object occur as if the object wasaccessed sequentially by the processes

This is the safety property stating that the implemen-tation of the object is linearizable


A Variant

“Simplified” version of the construction where the sharedarray LAST OP [1 : n] and the seq numbers are suppressed,propi is a simpe variable (init. to ⊥) and δ(sxi,⊥) = sxi

when operation X .op(param) is invoked by pi:

resulti ← ⊥; propi ← op(param) ;

wait until (resulti 6= ⊥); return (resulti)

while (true) do % Background Task %ki ← ki +1;execi← CONS [ki].propose (propi);(sxi, res)← δ(sxi, execi.op);let j = execi[r].proc;if (i = j) then resulti ← res; propi← ⊥ end if

end while


Wait-free vs Non-Blocking

• The previous construction:

⋆ Satisfies the non-blocking property: if processes pro-pose operations at least one process progress

⋆ Does not satisfy the wait-free property: the progressof a correct process cannot be ensured

• The universal construction algorithm ensures the wait-free property thanks to a helping mechanism

The shared array LAST OP [1 : n] (and the associatedseq numbers) allows a process to propose to the con-sensus instances not only its own operations but all thepending operations

Helping mechanisms: feature of wait-free computing


Part XI

When OBJECTS CAN FAIL

- Jayanti P., Chandra T.D. and Toueg S., Fault-TolerantWait-Free Shared Objects. Journal of the ACM, 45(3):451-500, 1998

- Guerraoui R. and Raynal M., Fault-Tolerance Techniquesfor Concurrent Objects. Tech Report # 1667, 22 pages,IRISA, Universite de Rennes 1 (France), December 2004http://www.irisa.fr/bibli/publi/pi/2004/1667/1667.html


From Process Failures to Object Failures

• Until now: we have considered thatonly processes may fail, objects (atomicregisters) were implicitly assumed reli-able

• From now on: base objects can fail


Object Failure Modes

• Crash failure of an object: after some time alloperations answer ⊥ (responsive failure mode)

• Omission failure of an object: after some timethe answers to some processes are always ⊥

• Arbitrary failure of an object: the answers canbe arbitrary


Failure Mode Hierarchy

• An implementation of a shared object is t-fault tolerantwith respect to a failure mode F (crash, omission, arbi-trary) if the object remains correct and wait-free despitethe occurrence of up to t base objects that fail accordingto F

• Failure mode F is less severe than the failure mode G(denoted F ≺ G) if a protocol that is t-fault tolerant forthe failure mode G is also t-fault tolerant for the failuremode F

• crashes ≺ omissions ≺ arbitrary failures


State Machine Replication

• Replication management using the State Ma-chine approach (Thomas, Lamport-Schneider’sapproach): concurrent RPC-like + votes

• Replication + Sequential iteration on replicasThe type of control structure applied to repli-cas becomes decisive (concurrent=independentvs sequential=dependent)


A Wait-Free t-Omission Tolerant Implementation

t+1 copies of the base object: base cons[1..(t+1)]

sequential traversal of the base consensus objects

procedure PROPOSE v

% est, aux, k: local variables of the invoking process %est← v;for k from 1 to (t+1) do

aux← base cons[k].propose est;if (aux 6= ⊥) then est← aux endif

endfor;return(est)


Graceful Degradation

• A wait-free implementation of a shared object is grace-fully degrading if it never fails more severely than thebase objects it is derived from, whatever the number ofbase objects that fail

• Remark: the “severity” relation on failure modes (≺)involves only the existence of a fault-tolerant protocol(it does not involve the notion of graceful degradation)


Graceful Degradation: Example

• Let us assume that base objects can fail by crashing.If the implementation remains wait-free and correct de-spite the crash of any number of processes and the crashof up to t base objects, then this implementation ist-fault tolerant with respect to the crash failure mode

• If the implementation is wait-free and fails only by crash(if it fails) despite the crash of any number of processesand the crash of any number of base objects, then it isgracefully degrading


A Gracefully Degrading t-Omission Tolerant Impl.

2t+1 copies of the base object: base cons[1..(2t+1)]

procedure PROPOSE v

% V [1..2t+1], est, k: local variables of the invoking process %est← v;for k from 1 to (2t+1) do

V [k]← base cons[k].propose est;if (V [k] 6= ⊥) ∧ (V [k] 6= est) then

est← V [k];V [1..(k − 1)]← [⊥, . . . ,⊥] endif

endfor;if (#⊥(V ) > t) then return(⊥) else return(est) endif


Impossibility for the Crash Failure Mode

• It is impossible to design gracefully degradingt-fault tolerant wait-free implementations forthe crash failure mode (for any object)

• It is possible to design gracefully degradingt-fault tolerant wait-free implementations forthe omission (or arbitrary) failure mode

• Hence (Jayanti-Chandra-Toueg 1998): Com-bining fault-tolerance and graceful degradationis not possible for all failure modes


Conclusion

• Linearizability

• Wait-free computing

• Consensus number, Herlihy Hierarchy

• Universal construction

• Object failure modes

• Process failure vs object failures

• t-Resilient wait-free objects

• Gracefully degrading objects


Conclusion

• “Wait-Free” concept

• Wait-free objects

• Process failure vs Object failures

• Fault-tolerant wait-free objects

• Gracefully Degrading wait-free objects


The other “only slide to remember” !

Asynchrony and failuresdo modify our view of synchronization


Three books

• Taubenfeld G., Synchronization algorithms and concur-rent programming. Pearson Education/Prentice Hall,423 pages, 2006 (ISBN 0-131-97259-6)

• Herlihy M. and Shavit N., The art of multiprocessor pro-gramming. Morgan Kaufmann, 508 pages, 2008 (ISBN978-0-12-370591-4)

• Raynal M., Concurrent programming: algorithms, prin-ciples and foundations. Springer 510 pages, November2012 (ISBN 978-3-642-32026-2)


Documents

Synchronization is Coming Back But is it the Same? · c Michel Raynal Synchronization is coming back! 44. A NON-BLOCKING timestamp object: deﬁnition • Validity. No two invocations