1

Chapter 10Chapter 10

Distributed Distributed AlgorithmsAlgorithms

2

Chapter ContentChapter Content

This and the next two chapters present This and the next two chapters present algorithms designed for loosely-connected algorithms designed for loosely-connected distributed systems that communicate by distributed systems that communicate by sending and receiving messages over a sending and receiving messages over a communications networkcommunications network

Chapter 10 - Chapter 10 - Distributed mutual exclusionDistributed mutual exclusion Chapter 11 -Chapter 11 - Distributed terminationDistributed termination Chapter 12 -Chapter 12 - Consensus (Fault tolerance)Consensus (Fault tolerance)

3

The Distributed Systems The Distributed Systems Model Model

A A nodenode is a physically identifiable object is a physically identifiable object which may contain several concurrent which may contain several concurrent processes. processes.

The internal synchronization among The internal synchronization among processes in a node is accomplished using processes in a node is accomplished using sharedshared-memory primitives -memory primitives

PProcesses in different nodes can rocesses in different nodes can communicate only by communicate only by sending and sending and receiving messagesreceiving messages..

4

AssumptionsAssumptions Each node has a two-way communications Each node has a two-way communications

channel with each other nodechannel with each other node The communications channels are error-freeThe communications channels are error-free The transit times of messages are finite but The transit times of messages are finite but

arbitrary. In particular, messages need not be arbitrary. In particular, messages need not be delivered in orderdelivered in order

Nodes do not failNodes do not fail. . Since mutual exclusion will Since mutual exclusion will be accomplished by a co-operative protocol be accomplished by a co-operative protocol (send and receive messages), nodes should (send and receive messages), nodes should not failnot fail or at least or at least continue executing continue executing processes that send and receive messagesprocesses that send and receive messages if if failedfailed. .

5

Sending and Receiving Sending and Receiving MessagesMessages

Two statements for communication:Two statements for communication: Send(message type, destination [, Send(message type, destination [,

parameters])parameters]) Receive(message type [, parameters])Receive(message type [, parameters])

6

ExamplesExamples Example 1: node 5 sends a “request” Example 1: node 5 sends a “request”

message to node 3message to node 3

Example 2: node 5 sends a “request” Example 2: node 5 sends a “request” message to node 3 and by sending its id message to node 3 and by sending its id to wait for a replyto wait for a reply

7

Bakery AlgorithmBakery Algorithm

In the bakery algorithm, a process wishing to In the bakery algorithm, a process wishing to enter its critical section is required to enter its critical section is required to take a take a numbered ticket whose value is greater numbered ticket whose value is greater than the values of all the outstanding than the values of all the outstanding ticketstickets.. Then it waits until its ticket has the Then it waits until its ticket has the lowest value. lowest value.

The name of the algorithm comes from a The name of the algorithm comes from a similar situation in a bakery which has a ticket similar situation in a bakery which has a ticket dispenser at the entrance. Customers are dispenser at the entrance. Customers are served in order of ascending ticket numbers.served in order of ascending ticket numbers.

8

Bakery AlgorithmBakery Algorithm (Cont.) (Cont.)

The advantage of this scheme is that The advantage of this scheme is that there need be no variable which is both there need be no variable which is both read and written by more than one read and written by more than one process (like the process (like the turnturn in Dekker's in Dekker's algorithm). Thus the bakery algorithm algorithm). Thus the bakery algorithm can be implemented on a wider range of can be implemented on a wider range of architectures than those algorithms architectures than those algorithms requiring atomic load and store to the requiring atomic load and store to the same global variables.same global variables.

9

Bakery Algorithm Bakery Algorithm Choosing : array(1..N) of integer;Choosing : array(1..N) of integer; Number : array(1..N) of integer;Number : array(1..N) of integer;task body Pi istask body Pi is

I : constant integer := ... ; -- Task idI : constant integer := ... ; -- Task idbeginbegin looploop

non_critical_section;non_critical_section;

choosing(I):= 1;choosing(I):= 1; Number(I):= 1 + max(Number);Number(I):= 1 + max(Number); choosing(I):= 0;choosing(I):= 0;for J in 1..N loopfor J in 1..N loop if J /= I if J /= I then then

loop exit when choosing(J) = 0; end loop;loop exit when choosing(J) = 0; end loop; loop exit when Number(J) = 0 or Number(I) < Number(J) loop exit when Number(J) = 0 or Number(I) < Number(J)

oror (Number(I) = Number(J) and I < J);(Number(I) = Number(J) and I < J); end loop;end loop;

end if;end if; end loop;end loop;

critical_section;critical_section;Number(I):= 0;Number(I):= 0;

end loop;end loop;end Pi;end Pi;

10

Explanation of the Explanation of the AlgorithmAlgorithm

The array The array choosingchoosing is used as a flag to indicate other is used as a flag to indicate other processes that a process is in the act of choosing a processes that a process is in the act of choosing a ticket number. Note that, a new number must be ticket number. Note that, a new number must be chosen to enter the critical section every time and a chosen to enter the critical section every time and a number may be used by several processes if they number may be used by several processes if they decide to chose a number at the same time. decide to chose a number at the same time. FurthermoreFurthermore,, notice that each process independently notice that each process independently chose a ticket.chose a ticket.

choosing(I):= 1;choosing(I):= 1;Number(I):= Number(I):= 1 + max(Number);1 + max(Number);choosing(I):= 0;choosing(I):= 0;

max(max(NNumber)umber) compares all process tickets from compares all process tickets from 1..N and the new number is one greater than this 1..N and the new number is one greater than this maximum (which becomes the new maximum)maximum (which becomes the new maximum)

11

Explanation of the Algorithm Explanation of the Algorithm (Cont.)(Cont.)

A process is allowed to enter its CS when it has a A process is allowed to enter its CS when it has a lower ticket number than all other processes. In lower ticket number than all other processes. In case of a tie in comparing ticket numbers, the lower case of a tie in comparing ticket numbers, the lower numbered process is arbitrarily given precedence.numbered process is arbitrarily given precedence.

for J in 1..N loopfor J in 1..N loopif J /= I if J /= I

then then ....

end if;end if; end loop;end loop;

The loop executes for J The loop executes for J = = 1..N, skips over the if 1..N, skips over the if statement when statement when J = I since we compare the ticket J = I since we compare the ticket value of Pi with all other processes.value of Pi with all other processes.

12

Explanation of the Algorithm Explanation of the Algorithm (Cont.)(Cont.)

""loop exit when choosing(J) = 0; end loop;loop exit when choosing(J) = 0; end loop;““ means wmeans wait if J'th process is currently choosing a ait if J'th process is currently choosing a ticket.ticket.

""loop exit when loop exit when Number(J) = 0 or Number(I) < Number(J) Number(J) = 0 or Number(I) < Number(J)

oror(Number(I) = Number(J) and I < J);(Number(I) = Number(J) and I < J);

end loop;end loop;"" ""Number(J) = 0Number(J) = 0"" means that J'th process is not in means that J'th process is not in

its CS and not in the queue of processes by its CS and not in the queue of processes by choosing a new ticket (ie., possibly in its choosing a new ticket (ie., possibly in its non_critical section).non_critical section).

13

Explanation of the Algorithm Explanation of the Algorithm (Cont.)(Cont.)

""Number(I) < Number(J)Number(I) < Number(J)"" means Pi has a lower means Pi has a lower numbered ticket thus Pj should wait for Pi.numbered ticket thus Pj should wait for Pi.

""Number(I) = Number(J) and I < JNumber(I) = Number(J) and I < J"" means that means that Pi and Pj has the same ticket number but as I < J Pi and Pj has the same ticket number but as I < J Pj will wait for Pi according to the tie rule. Pj will wait for Pi according to the tie rule.

For all other processes Pi will stay in the loop For all other processes Pi will stay in the loop until Pj exits from its critical section and until Pj exits from its critical section and make Number(J):= 0. Process Pi enters its CS make Number(J):= 0. Process Pi enters its CS after a complete scan of all processes. after a complete scan of all processes.

14

Distributed Mutual Exclusion Distributed Mutual Exclusion AlgorithmAlgorithm

(Ricart-Agrawala Algorithm)(Ricart-Agrawala Algorithm) Algorithm will be similar to the Bakery algorithm. Algorithm will be similar to the Bakery algorithm. However, since the nodes cannot directly read the However, since the nodes cannot directly read the

internal variables of the other nodes, the comparison of internal variables of the other nodes, the comparison of sequence numbers (tickets) and the decision to enter sequence numbers (tickets) and the decision to enter the CS must be made by sending and receiving the CS must be made by sending and receiving messages. messages.

A node chooses a number, broadcasts its choice to all A node chooses a number, broadcasts its choice to all other nodes (a Request message) and then waits until other nodes (a Request message) and then waits until it has received a confirmation ( a Reply) from each it has received a confirmation ( a Reply) from each other node that the number chosen is now the lowest other node that the number chosen is now the lowest outstanding sequence number. outstanding sequence number.

Ties on the chosen sequence number are resolved in Ties on the chosen sequence number are resolved in favor of the node with the lowest identification number.favor of the node with the lowest identification number.

15

Structure of the AlgorithmStructure of the Algorithm

The structure of The structure of the the algorithm consists of three algorithm consists of three processes:processes:

A A main processmain process which executes the usual which executes the usual loop of protocols enclosing a CSloop of protocols enclosing a CS

A process which receives A process which receives RequestRequest messages messages from other nodesfrom other nodes

A process which receives A process which receives ReplyReply messages messages from other nodesfrom other nodes

16

Main ProcessMain Process

task body Main_Process_Type istask body Main_Process_Type isbeginbegin looploop

Non_critical_section;Non_critical_section;Choose_sequence_number; Choose_sequence_number; Send_request_to_nodes; Send_request_to_nodes; Pre-Pre-

protocolprotocolWait_for_Reply; Wait_for_Reply;

Critical_section; Critical_section; Reply_to_deferred_nodes; Reply_to_deferred_nodes; Post-Post-

protocolprotocol end loop;end loop;end Main_Process_Type;end Main_Process_Type;

17

Request ProcessRequest Process

task body Request_Process_Type istask body Request_Process_Type isbeginbegin looploop

accept Message;accept Message;if decide_to_deferif decide_to_defer

then Defer_Replythen Defer_Replyelse Send_Reply;else Send_Reply;

end if;end if; end loop;end loop;end Request_Process_Type;end Request_Process_Type;

18

Reply ProcessReply Process

task body Reply_Process_Type istask body Reply_Process_Type is

beginbegin

looploop

accept Message;accept Message;

Increment_reply_count;Increment_reply_count;

If last_reply then If last_reply then Wake_Main_Process; end if;Wake_Main_Process; end if;

end loop;end loop;

end Reply_Process_Type;end Reply_Process_Type;

19

SummarySummary A nodeA node,, which receives a which receives a RequestRequest from another node from another node,,

must respond with its sequence number. must respond with its sequence number. The node compares the received number with its own. The node compares the received number with its own.

If the received number has a If the received number has a lower sequence numberlower sequence number then a then a reply is sent immediatelyreply is sent immediately. .

If If notnot, the reply is , the reply is deferreddeferred (give way, submit) (give way, submit). This . This effectively suspends the sending node in its pre-effectively suspends the sending node in its pre-protocol since it will not enter its CS until it has protocol since it will not enter its CS until it has received replies from each of the other nodes.received replies from each of the other nodes.

When replies have been received from each of the When replies have been received from each of the other nodes, the main process proceeds to execute its other nodes, the main process proceeds to execute its critical section. critical section.

The post-protocol consists of sending the deferred The post-protocol consists of sending the deferred replies, which will eventually release suspended nodes.replies, which will eventually release suspended nodes.

20

Details of the Details of the AlgorithmAlgorithm

Since the algorithm is a mutual Since the algorithm is a mutual exclusion algorithm over distributed exclusion algorithm over distributed nodes the communication is done by nodes the communication is done by passing messagespassing messages

The transfer of messages between The transfer of messages between nodes will be based on the nodes will be based on the rendezvous principlerendezvous principle

21

Variables global to a node Variables global to a node (cannot be accessed by other (cannot be accessed by other

nodes)nodes) Number:Number: Integer := 0;Integer := 0; --Sequence number chosen --Sequence number chosen

by the nodeby the node High_Number:High_Number: Integer := 0;Integer := 0; --The highest sequence --The highest sequence

number received so far. Used by the node to choose a higher ticket number received so far. Used by the node to choose a higher ticket numbernumber

Requesting:Requesting: Boolean := False;Boolean := False; --A flag to indicate whether --A flag to indicate whether this node is requesting this node is requesting to enter to enter its CS. This is used to skip the its CS. This is used to skip the comparison of sequence numbers if a node is not contending for the comparison of sequence numbers if a node is not contending for the critical sectioncritical section

Reply_count:Reply_count: Integer := 0;Integer := 0; ----Used to count received Used to count received Reply messages so that the node knows when it is allowed to enter its Reply messages so that the node knows when it is allowed to enter its CSCS

S:S: Semaphore := 1;Semaphore := 1; ----A semaphore to provide mutual A semaphore to provide mutual exclusion during the access to the global variables of a nodeexclusion during the access to the global variables of a node

Wake_up:Wake_up: Semaphore := 0;Semaphore := 0; ----A semaphore to suspend A semaphore to suspend the main process and awaken it when it is the time to enter its CS.the main process and awaken it when it is the time to enter its CS.

Deferred:Deferred: array(1..N) of Boolean ;array(1..N) of Boolean ; ----A data structure A data structure indicating which nodes have had their replies deferred. N is the indicating which nodes have had their replies deferred. N is the number ofnumber of nodes. nodes.

22

Declarations for message Declarations for message transfertransfer

task type Main_Process_Type;task type Main_Process_Type;

task type Request_Process_Type istask type Request_Process_Type is entry Message(Num,ID : in integer);entry Message(Num,ID : in integer);

end Request_Process_Type;end Request_Process_Type;

task type Reply_Process_Type istask type Reply_Process_Type isentry Message;entry Message;

end Reply_Process_Type;end Reply_Process_Type;

Main_Process :Main_Process : array(1..N) of Main_Process_Type;array(1..N) of Main_Process_Type;Request_Process:Request_Process: array(1..N) of Request_Process_Type;array(1..N) of Request_Process_Type;Reply_Process:Reply_Process: array(1..N) of Reply_Process_Type;array(1..N) of Reply_Process_Type;

23

Pre-protocol Procedures Pre-protocol Procedures procedure choose_sequence_number isprocedure choose_sequence_number isbeginbegin

wait(S);wait(S);requesting:= true;requesting:= true; {wants to enter the CS} {wants to enter the CS}number:= high_number + 1;number:= high_number + 1;signal(S);signal(S);

end choose_sequence_number;end choose_sequence_number;

procedure send_request_to_nodes isprocedure send_request_to_nodes isbeginbegin

reply_count := 0;reply_count := 0;for J in 1..N loopfor J in 1..N loop

if J /= I then Request_Process(J).Message(number,I); if J /= I then Request_Process(J).Message(number,I); end if;end if;

end loop;end loop;end send_request_to_nodes;end send_request_to_nodes;

procedure wait_for_reply isprocedure wait_for_reply isbeginbegin

wait(wake_up);wait(wake_up);end wait_for_reply;end wait_for_reply;

24

Post-protocol Procedure Post-protocol Procedure

procedure reply_to_deferred_nodes isprocedure reply_to_deferred_nodes isbeginbegin

wait(S);wait(S);requesting:= false;requesting:= false; {out of CS} {out of CS}signal(S);signal(S);for J in 1..N loopfor J in 1..N loop

if deferred(J) then if deferred(J) then deferred(J):= false;deferred(J):= false;reply_process(J).message;reply_process(J).message;

end if;end if;end loop;end loop;

end reply_to_deferred_nodes;end reply_to_deferred_nodes;

25

Request_process_Type Request_process_Type BodyBody

task body Request_Process_Type istask body Request_Process_Type isreceived_number : integer;received_number : integer;received_id received_id : integer;: integer;decide_to_defer decide_to_defer : boolean;: boolean;

beginbeginlooploop

accept message(num, id : integer) doaccept message(num, id : integer) doreceived_number := num;received_number := num; received_id := id;received_id := id;

end message;end message;high_number:= max(high_number, received_number);high_number:= max(high_number, received_number);wait(S);wait(S);decide_to_defer := requesting anddecide_to_defer := requesting and (number < received_number (number < received_number

or or (number = received_number and I < (number = received_number and I <

received_id) );received_id) );if decide_to_deferif decide_to_defer then deferred(received_id) := truethen deferred(received_id) := true

else else Reply_Process(received_id).message;Reply_Process(received_id).message;end if;end if;signal(S);signal(S);

end loop;end loop;end Request_Process_Type;end Request_Process_Type;

26

Reply_Process_Type body Reply_Process_Type body

task body Reply_Process_Type istask body Reply_Process_Type isbeginbegin

looploopaccept message;accept message;reply_count:= reply_count + 1;reply_count:= reply_count + 1;if reply_count = N-1 if reply_count = N-1 then then

signal(wake_up); signal(wake_up); end if;end if;

end loop;end loop;end Reply_Process_Type;end Reply_Process_Type;

2727

Sample ExecutionSample Execution

NodeNode Sequence #Sequence # SentSents tos to ReceiveReceives froms from StatusStatus

11 2020 2, 3, 42, 3, 4 2, 42, 4 Waiting reply from 3Waiting reply from 3

22 2323 1, 3, 41, 3, 4 44 Waiting reply from 1, Waiting reply from 1, 33

33 1515 1, 2, 41, 2, 4 1, 2, 41, 2, 4 ExecutesExecutes

44 -- --   

Suppose there are 4 nodes. Process 1 chooses sequence number 20, process 2 chooses 23 and process 3 chooses 15. All processes want to enter their critical sections

Time 1: Processes 1, 2 and 3 attempt to enter their critical sections

28

Sample ExecutionSample Execution (Cont.) (Cont.)

NodeNode Sequence #Sequence # SentSents tos to ReceiveReceives froms from StatusStatus

11 2020 2, 3, 42, 3, 4 2, 4 and 32, 4 and 3 ExecutesExecutes

22 2323 1, 3, 41, 3, 4 4 and 34 and 3 Waiting reply from 1Waiting reply from 1

33 -- -- TerminatedTerminated

44 -- --   

Time 2: Process 3 terminates, sends reply to process 1 and 2

Time 3: Process 1 terminates, sends reply to process 2NodeNode Sequence #Sequence # SentSents tos to ReceiveReceives froms from StatusStatus

11 -- -- TerminatedTerminated

22 2323 1, 3, 41, 3, 4 4, 3 and 14, 3 and 1 ExecutesExecutes

33 -- -- TerminatedTerminated

44 -- --   

29

Token Passing AlgorithmsToken Passing Algorithms The previous The previous permission-basedpermission-based algorithm is algorithm is

inefficient if there are a large number of nodes (too inefficient if there are a large number of nodes (too many messages have to be passed)many messages have to be passed)

Furthermore, the algorithm does not show improved Furthermore, the algorithm does not show improved performance in the absence of contention; a node performance in the absence of contention; a node wishing to enter its critical section must always send wishing to enter its critical section must always send and receive messages from all other nodesand receive messages from all other nodes

In token-passing algorithms, permission to enter the In token-passing algorithms, permission to enter the CS is associated with the possession of an object CS is associated with the possession of an object called a called a tokentoken

The algorithm is efficient: only one message is The algorithm is efficient: only one message is needed to transfer the token and its associated needed to transfer the token and its associated permission from one node to another, and a node permission from one node to another, and a node possessing the token can enter its CS any number of possessing the token can enter its CS any number of times without sending or receiving any messagestimes without sending or receiving any messages

30

Ricart-Agrawala Token-passing Ricart-Agrawala Token-passing AlgorithmAlgorithm

Any process which holds a “granted” token Any process which holds a “granted” token (only one) can enter the CS.(only one) can enter the CS.

A token held by a process will not be passed to A token held by a process will not be passed to another node unless it is needed and the another node unless it is needed and the process may enter the CS repeatedlyprocess may enter the CS repeatedly

Two arrays are maintained in each process Two arrays are maintained in each process with an entry for each of the nodes with an entry for each of the nodes Requested arrayRequested array : ticket numbers passed from each : ticket numbers passed from each

node as they make a new request to enter a CSnode as they make a new request to enter a CS Granted arrayGranted array : ticket numbers held by each node : ticket numbers held by each node

the the last time last time it was granted permission to enter it was granted permission to enter the CSthe CS

31

1 2 3 4 51 2 3 4 5 Consider the data structures maintained by node 3 in a Consider the data structures maintained by node 3 in a

five node systemfive node system Requested array is updated when messages arrive, Requested array is updated when messages arrive,

granted array is received as a whole when node was granted array is received as a whole when node was granted the token last timegranted the token last time

Request messages have been received at node 3 from Request messages have been received at node 3 from nodes 2 and 4nodes 2 and 4

If node 3 If node 3 holds the token and is not in the CSholds the token and is not in the CS, the token , the token must be passed to node 2 or 4must be passed to node 2 or 4

If node 3 is in the CS, it must If node 3 is in the CS, it must pass the token when it pass the token when it leaves the CSleaves the CS

Data Structures for RA Token- Data Structures for RA Token- Passing AlgorithmPassing Algorithm

32

Ricart-Agrawala Token-passing Ricart-Agrawala Token-passing AlgorithmAlgorithm

Send the “granted” array with Send the “granted” array with the tokenthe token

33

Ricart-Agrawala Token-passing Ricart-Agrawala Token-passing Algorithm (Cont.)Algorithm (Cont.)

Make a requst to Make a requst to enter the CSenter the CS

Send id and request Send id and request number to all other number to all other nodesnodes

Some process sends Some process sends the tokenthe token

Record request Record request number as the value number as the value of the grantedof the granted

Pass the token to Pass the token to another node if another node if neededneeded

34

Ricart-Agrawala Token-passing Ricart-Agrawala Token-passing Algorithm (Cont.)Algorithm (Cont.)

35

Comparison of the Last Two Comparison of the Last Two AlgorithmsAlgorithms

The original RA algorithm required that a node The original RA algorithm required that a node entering its CS send N-1 request messages and entering its CS send N-1 request messages and receive N-1 replies. The messages are short, receive N-1 replies. The messages are short, containing only a few values of fixed sizecontaining only a few values of fixed size

In the token-passing algorithm, a node need not In the token-passing algorithm, a node need not send messages if it possesses the token; send messages if it possesses the token; otherwise, it needs to send N-1 request messages otherwise, it needs to send N-1 request messages as before, but it only need to receive one token as before, but it only need to receive one token messagemessage

However, the token message is long, containing N However, the token message is long, containing N ticket numbers. If N is large, this can be ticket numbers. If N is large, this can be inefficient, but in practise, the token passing inefficient, but in practise, the token passing algorithm should be more efficient than the RA algorithm should be more efficient than the RA algorithm because there is a large overhead in algorithm because there is a large overhead in sending a message, regardless of its sizesending a message, regardless of its size