Computer Science 425Distributed

Systems(Fall 2009)

Lecture 6Multicast/Group Communication and

Mutual ExclusionSection 15.2.2&12.2

Acknowledgement

• The slides during this semester are based on ideas and material from the following sources:

– Slides prepared by Professors M. Harandi, J. Hou, I. Gupta, N. Vaidya, Y-Ch. Hu, S. Mitra.

– Slides from Professor S. Gosh’s course at University o Iowa.

Administrative

• Homework 1 posted September 3, Thursday– Deadline, September 17 (Thursday)

• MP1 posted September 8, Tuesday– Deadline, September 25 (Friday), 4-6pm Demonstrations

• MP1 Tutorial September 14, Monday– 7pm, 3401 Siebel Center

Few Comments: VTand its Usage (Causal Ordering)

a b

c d

e f

m1

m2

(2,0,0)(1,0,0)

(2,1,0) (2,2,0)

(2,2,2)(0,0,1)

p1

p2

p3

Physical time

P1

P2

P3(1,1,0)

0,0,0

0,0,0

1,0,0 1,1,0

1,0,0

1,1,0

1,1,0

1,1,00,0,0

1,0,0 1,1,0

(1,0,0)

(1,0,0)

(1,1,0) (1,1,0)

Plan for Today

• Group Communication and Group Membership – Group View– Process View– View Synchrony

• Mutual Exclusion – Semaphores review– Coordinator-based algorithm – Token Ring algorithm

Group Communication

• Multicast is also known as group communication because process groups are destinations of multicast messages

• Multicast algorithms rely on groups• Open group: anyone can join (e.g., customers in

Best Buy Store)• Closed group: membership is closed (e.g., class

of 2010)• Before, we assumed membership of groups was

statically defined• Practically, dynamic membership is required

since members can join and leave

Group Membership • Implementation of group communication

– Need group membership service to manage dynamic membership of groups in addition to multicast communication

• Multicast and Group Membership management are strongly interrelated.

• Role of Group Membership– Providing interface for group membership changes

» Operations to create and destroy process groups» Operations to add or withdraw process to or from group

– Implementing failure detector» Operation to detect that a group member crashed or left

– Notifying members of group membership changes» Operation to notify when process was added or left

– Performing group address expansion » Operation to map (expand) group ID to list of process IDs

Group Communication

“Member”= process (e.g., RM) Static Groups: group membership is pre-defined Dynamic Groups: Members may join and leave, as

necessary

Group Send

Address Expansion

Multicast Communication

Group Membership Management

Leave

Fail

Join

Group

Group Views

Group View List of current group members. Maintained by group membership service

View Vp(g) of a Process pIts current knowledge of the membership Each member maintains its own local viewExample: V p.0(g) = {p}, V p.1(g) = {p, q}, V p.2 (g) = {p, q, r}, V p.3 (g)

= {p,r}

New group view is disseminated, throughout the group, whenever a member joins or leaves.Member detecting failure of another member reliable multicasts a

“view change” message (requires causal-total ordering for multicasts)

Inconsistent Views can lead to problems

• Four members (0,1,2,3) will send out 144 emails equally

• Assume that member 3 left the group but only member 2 knows about it

• 0 will send 144/4 = 36 emails (first quarter 1-36)• 1 will send out 144/4 = 36 emails (second quarter

37-72)• 2 will send out 144/3 = 48 emails (last one third 97-

144)• 3 has left, emails 73-96 will never be sent out!

Dealing with Open Groups

• View can change unpredictably, and no member may have exact information about who joined and who left at any given time

• Views and their changes should propagate in the same order to all members

• Current view (of all processes) V0(g) = {0,1,2,3}• Let 1,2 leave and 4 join the group concurrently• This view change can be ordered in many ways:

– {{0,1,2,3}, {0,1,3}, {0,3,4} OR– {0,1,2,3}, {0,2,3}, {0,3}, {0,3,4} OR– {0,1,2,3}, {0,3}, {0,3,4}

• To ensure that every member observes these changes in the same order, changes in the view should be sent via TOTAL ORDER MULTICAST

View Propagation

{Process 0}:

V0(g); V0(g) = {0,1,2,3}send m1,…;V1(g) ; V1(g) = {0,1,3}send m2, send m3, ..;V2(g); V2(g) = {0,3,4}

{Process 1}:

V0(g); V0(g) = {0,1,2,3}send m4, send m5; V1(g); V1(g) = {0,3}send m6,..;V2(g); V2(g) = {0,3,4}

View Delivery Guidelines

• If a process j joins and continues its membership in group g that already contains process i, then eventually j appears in all views delivered by process i

• If a process j permanently leaves group g that contains process i, then eventually j is excluded from all views delivered by process i

Group Views

Requirements for view delivery Order: If p delivers Vi(g) and then Vi+1(g), then no other

process q delivers Vi+1(g) before Vi(g).

Integrity: If p delivers Vi(g), then p is in Vi(g).

Non-triviality: if process q joins a group and becomes reachable from process p, then eventually, q will always be present in the views that are delivered at p.

View-Synchronous Communication

• Rule. With respect to each message, all correct processes have the same view

• m sent in view V → m received in view V

• View Synchronous Communication = Group Membership Service + Reliable multicast

View-Synchronous CommunicationIntegrity: If a process j delivers a

view Vi(g), then Vi(g) must include j.

Validity: If a process k delivers a message m and k є Vi(g) and another process j є Vi(g) does not deliver that message m, then the next view Vi+1(g) delivered by k must exclude j

Agreement: if a correct process k delivers a message m and Vi(g) with kєVi(g) before delivering the next view Vi+1(g), then every correct process jє Vi(g) ∩ Vi+1(g) must deliver m before delivering Vi+1(g).

Example• Let process 1 send out m and

then crash • Possibility 1. No one delivers m,

but each delivers the new view {0,2,3}

• Possibility 2. Processes 0,2,3 deliver m and then deliver new view {0,2,3}

• Possibility 3. Processes 2, 3 deliver m and then deliver new view {0,2,3}, but process 0 first delivers the view {0,2,3} and then delivers m.

• Are these acceptable?

Example: View Synchronous Communication

p

q

r

V(p,q,r)

p

q

r

V(p,q,r)

p

q

r

V(p,q,r)

p

q

r

V(p,q,r)

X

XX

V(q,r)

V(q,r)

V(q,r)

V(q,r)

X

X X

Not Allowed Not Allowed

Allowed Allowed

mmm

m m mm

m

Comment about View Synchrony

• When a new process joins, state transfer may be needed (at view delivery point) to bring it up to date

– “state” may be list of all messages delivered so far (wasteful)– “state” could be list of current server object values (e.g., in a

bank database)

• View Synchrony = “Virtual Synchrony”– Provides an abstraction of a synchronous network that hides

the asynchrony of the underlying network from distributed applications

• Used in ISIS toolkit (NY Stock Exchange)

DISTRIBUTED MUTUAL EXCLUSION

Why Mutual Exclusion?

• Bank Database: Think of two simultaneous deposits of $10,000 into your bank account, each from one ATM.

– Both ATMs read initial amount of $1000 concurrently from the bank server

– Both ATMs add $10,000 to this amount (locally at the ATM)– Both write the final amount to the server– What’s wrong?

Why Mutual Exclusion?

• Bank Database: Think of two simultaneous deposits of $10,000 into your bank account, each from one ATM.

– Both ATMs read initial amount of $1000 concurrently from the bank server

– Both ATMs add $10,000 to this amount (locally at the ATM)– Both write the final amount to the server– What’s wrong?

• ATMs need mutually exclusive access to your account entry at the server (or, to executing the code that modifies the account entry)

Critical section (CS) problem: Piece of code (at all clients) for which we need to ensure there is at most one client executing it at any point of time.

Solutions: Semaphores, mutexes, etc. in local operating systems Message-passing-based protocols in distributed

systems: enter() the critical section AccessResource() in the critical section exit() the critical section

Distributed mutual exclusion requirements: Safety – At most one process may execute in CS at any time Liveness – Every request for a CS is eventually granted Fairness/Ordering (desirable) – Requests are granted in

FIFO order

Mutual Exclusion

Refresher - Semaphores

• To synchronize access of multiple threads to common data structures

• Semaphore is protected variable with two operations

• init: semaphore S=1;• wait(S):

while(1){ // each execution of the while loop is atomic if (S > 0) S--; break; }

• signal(S)S++; // atomic

Refresher - Semaphores

• To synchronize access of multiple threads to common data structures

• init: semaphore S=1;• wait(S):

while(1){ // each execution of the while loop is atomic if (S > 0) S--; break; }

• signal(S):S++;

enter() // enter critical region

Access to resources

exit() // exit critical region

How are semaphores used?

semaphore S=1;

ATM1:wait(S); // enter CSobtain bank amount;add in deposit;update bank amount;signal(S); // exit

extern semaphore S;

ATM2wait(S); // enter CS obtain bank amount;add in deposit;update bank amount;signal(S); // exit

One Use: Mutual Exclusion – Bank ATM example

Distributed Mutual Exclusion

• No shared variables nor facilities supplied by a single kernel can be used

• Require solutions to distributed mutual exclusion based solely on message passing !!

Distributed Mutual Exclusion:Performance Evaluation

Criteria• Bandwidth: the total number of messages sent in each entry and

exit operation.• Delay:

– Client delay: the delay incurred by a process at each entry and exit operation (when no other process is waiting)

– Synchronization delay: the time interval between one process exiting the critical section and the next process entering it (when there is only one process waiting)

• These translate into throughput -- the rate at which the processes can access the critical section, i.e., x processes per second.

(these definitions more correct than the ones in the textbook)

Assumptions

• For all the algorithms studied, we assume

– Reliable FIFO channels between every process pair » Each pair of processes is connected by reliable

channels (such as TCP). Messages are eventually delivered to recipients’ input buffer in FIFO order.

– Processes do not fail.

A central coordinator or leader L

Is appointed or elected

Grants permission to enter CS & keeps a queue of requests to enter the CS.

Ensures only one process at a time can access the CS

Separate handling of different CS’s

Centralized Control of Distributed Mutual Exclusion

Centralized control of Distributed MEOperations (token gives access to CS)

To enter a CS Send a request to L & wait for token.On exiting the CS Send a message to the coord to release the token.Upon receipt of a request, if no other process has the token, L replies with the token;

otherwise, L queues the request.Upon receipt of a release message L removes the oldest entry in the queue (if any) and replies with a

token.

Features: Safety, liveness and order are guaranteed It takes 3 messages per entry + exit operation. Client delay: one round trip time (request + grant) Synchronization delay: one round trip time (release + grant) The coordinator becomes performance bottleneck and single point of

failure.

Processes are organized in a logical unidirectional ring: pi has a communication channel to p(i+1)mod (n+1).

Operations:Only the process holding the token can enter the CS. To enter the critical section, wait for the token. To exit the CS, pi sends the token onto its neighbor. If a process does not want to enter the CS when it receives the token,

it forwards the token to the next neighbor.

Token Ring Approach

P0

P1

P2

P3

Pn

Previous holder of token

next holder of token

current holder of token

Features:Safety & liveness are guaranteed, but

ordering is not.Bandwidth: 1 message per exitClient delay: 0 to (N+1) message

transmissions.Synchronization delay between one

process’s exit from the CS and the next process’s entry is between 1 and N message transmissions.

Summary

• Discussed relation between multicast and group communication

– Important issue of view synchrony – Important for MP1 - please, consider carefully the group membership protocol

and view synchrony issues when designing Chat’s membership service

• Mutual Exclusion – Review semaphore concept from cs241 or other OS class– Semaphores alone will not do the trick of protecting critical regions in

distributed systems– Need much more complex message-passing algorithms to ensure distributed

mutual exclusion

• Next time – More Complex Distributed Mutual Exclusion Algorithms – Leader Election Algorithms