Real And Logical ClocksLecture 7

Coordination and Synchronization• How processes can coordinate and synchronize with each other • Process synchronization:

• Wait for another process to complete some action

• Data synchronization• 2 sets of data should be the same

Time in a conventional OS• Processes can get “system time” via systemcalls

• Such as gettimeofday() in UNIX

• Time-stamps can be used for ordering and coordination• Example: make uses file modified time to decide what actions to run • If the OS is the only time-source, then all processes observe the same time• Is it possible to synchronize all clocks in a distributed system?

Physical Clocks and UTC • Local time-keeping: crystal oscillator inside CPUs triggers timer interrupt in the

OS • Multiple CPUs difference in clock values • Temperature (and other factors) also affect frequency clock skew• Standardization: Universal Coordinated Time (UTC)• Based on atomic clocks • UTC “time” broadcast via radios, satellites, and through phone-numbers

• NIST +1-808-335-4363

• Accuracy is around 0.5ms

Clock Synchronization • Precision: Keep deviation between two clocks on any two machines within a

specified bound

• Accuracy: The difference between all clocks and UTC time is less than \alpha

• Internal synchronization: Keep clocks precise• External synchronization: Keep clocks accurate

NTP: Network Time Protocol • Server B is a time-server with an accurate clock (say, atomic)

• Server A wishes to get it’s clock synchronized with B’s clock • By periodically polling B

• A asks B for the time • But messages face network delays!• Collect 8 pairs of (\theta,\delta) and

choose \theta with lowest \delta • Never set a clock backwards!• Adjust rate of clock-ticks to slow-

down or catch-up to A’s time • Hierarchy of NTP servers

Keeping Time Without UTC• NTP allows for external synchronization with UTC time• Sometimes, internal consistency suffices • Time server polls all other machines, computes average time, and broadcasts it

Logical Clocks

Happened Before Relationship• First fundamental result in distributed computing • Partial order among events

• Process-order, send-receive order, transitivity

• Important that all processes agree on the order of events

Ordering Of Events• Happened-before relation:

1. If a and b are two events that occur in the same process, and a comes before b, then a b

2. If a corresponds to sending a msg, and b is the receipt of that message, then ab

3. Transitive: ab and bc, then abc

• This is a partial ordering of events in a system

Ordering Of Events Example

Partial order Some events will be concurrent

Logical Clocks• How to maintain a global view of system behavior that is consistent with

the happened-before relation? • Approach: assign a timestamp C(e) to each event e, such that:

• If ab, then C(a) < C(b)• C must be monotonically increasing

• How to attach a timestamp to an event when there’s no global clock?• Maintain a consistent set of logical clocks, one per process

Lamport Clocks• Each process 𝑃 maintains a local counter 𝐶 and adjusts this counter 1. New local event, increment 𝐶2. Send a timestamp 𝐶 with each message sent by 𝑃 (i.e., ts(m) = 𝐶 )3. Whenever a message is received by 𝑃 , 𝑃 adjusts its local counter

𝐶 = max{𝐶 ts(m)} . Since this is a new event, increment 𝐶

Lamport Clock Example

Total Ordering of Time-stamps

• It’s possible for 𝐶 (𝑒 ) == 𝐶 (𝑒 )

• I.e., Lamport clock timestamps are not totally ordered• We can break ties based on process-id • For process P_i, the clock value becomes 𝐶 . 𝑖

• For example: 3.2 for process-id==2

Example: Total-ordered Multicast• Two replicas of database• P1 adds $100 to an account (initial: $1000)• P2 increments account by 1%• Replica-1: $1111• Replica-2: $1110

Totally Ordered Multicast• Need to ensure that two update operations are performed in the same order

by all nodes• Actual order of operations is immaterial (add first or interest first) • Totally ordered multicast: All messages are delivered in the same order to each

receiver • Assumption: No failures, and FIFO delivery of messages

Totally Ordered Multicast Algorithm• All messages timestamped with sender’s logical time • Sender sends to all recipients, including itself• When a message is received:

• 1. It is put into a local queue • 2. Queue is ordered based on timestamp • 3. The acknowledgement is multicast (with the receiver’s logical time ofcourse)

Totally Ordered Multicast Algorithm, continued • Message is delivered to application only when:

1. It is at the head of the queue2. All the acknowledgement for that message have been received

Example: Totally Ordered Multicast

San Francisco (P1)1.12.1

New York (P2)

1.22.23.2

Issue mSend m

Recv n

Issue nSend nRecv m3.1

Example: Totally Ordered Multicast• The sending of message m consists of sending the update operation and the

time of issue which is 1.1• The sending of message n consists of sending the update operation and the time

of issue which is 1.2 • Messages are multicast to all processes in the group including itself.

• Assume that a message sent by a process to itself is received by the process almost immediately.• For other processes, there may be a delay.

Example: Totally Ordered Multicast• At this point, the queues have the following:

• P1: (m,1.1), (n,1.2)• P2: (m,1.1), (n,1.2)

• P1 will multicast an acknowledgement for (m,1.1) but not (n,1.2). • Why? P1’s identifier is higher then P2’s identifier and P1 has issued a request• 1.1 < 1.2

• P2 will multicast an acknowledgement for (m,1.1) and (n,1.2)• Why? P2’s identifier is not higher then P1’s identifier • 1.1 < 1.2

Example: Totally Ordered Multicast

• P1 does not issue an acknowledgement for (n,1.2) until operation m has been processed.• 1< 2

• Note: The actual receiving by P1 of message (n,1.2) is assigned a timestamp of 3.1.

• Note: The actual receiving by P2 of message (m,1.1) is assigned a timestamp of 3.2

Example: Totally Ordered Multicast• If P2 gets (n,1.2) before (m,1.1) does it still multicast an acknowledgement for

(n,1.2)? • Yes!

• At this point, how does P2 know that there are other updates that should be done ahead of the one it issued?• It doesn’t;• It does not proceed to do the update specified in (n,1.2) until it gets an acknowledgement from all other

processes which in this case means P1.

• Does P2 multicast an acknowledgement for (m,1.1) when it receives it? • Yes, it does since 1 < 2

Example: Totally Ordered Multicast

San Francisco (P1)1.12.1

3.1

5.1

New York (P2)

1.22.23.24.2

Issue mSend m

Recv n

Issue nSend nRecv mSend ack(m)

Recv ack(m)

Note: The figure does not show a process sending a message to itself orthe multicast acks that it sends for the updates it issues.

Example: Totally Ordered Multicast• To summarize, the following messages have been sent:

• P1 and P2 have issued update operations.• P1 has multicasted an acknowledgement message for (m,1.1).• P2 has multicasted acknowledgement messages for (m,1.1), (n,1.2).

• P1 and P2 have received an acknowledgement message from all processes for (m,1.1).

• Hence, the update represented by m can proceed in both P1 and P2.

Example: Totally Ordered Multicast

San Francisco (P1)1.12.1

3.1

5.1

New York (P2)

1.22.23.24.2

Issue mSend m

Recv n

Issue nSend nRecv mSend ack(m)

Recv ack(m)Process m

Note: The figure does not show the sending of messages it oneself

Process m

Example: Totally Ordered Multicast• When P1 has finished with m, it can then proceed to multicast an

acknowledgement for (n,1.2). • When P1 and P2 both have received this acknowledgement, then it is the case

that acknowledgements from all processes have been received for (n,1.2). • At this point, it is known that the update represented by n can proceed in both

P1 and P2.

Example: Totally Ordered Multicast

San Francisco (P1)1.12.1

3.1

5.1

New York (P2)

1.22.23.24.2

Issue mSend m

Recv n

Issue nSend nRecv mSend ack(m)

6.1Send ack(n)

Recv ack(m)

7.2 Recv ack(n)

Process m

Process nProcess n

Process m

Example: Totally Ordered Multicast• What if there was a third process e.g., P3 that issued an update (call it o) at

about the same time as P1 and P2.• The algorithm works as before.

• P1 will not multicast an acknowledgement for o until m has been done.• P2 will not multicast an acknowledgement for o until n has been done.

• Since an operation can’t proceed until acknowledgements for all processes have been received, o will not proceed until n and m have finished.

State Machine Replication• Totally ordered multicast enables state machine replication• Servers can be thought of deterministic state machines that change state based

on the messages they receive. • Replicating state machines has many benefits: fault-tolerance, performance..

• If a server crashes, then contact other replicas

• With totally ordered multicast, all servers can execute the same operations in the same order

• Totally-ordered multicast is the “holy grail” of distributed computing • Yet we could somehow achieve it? Or did we??