Distributed Systems Concepts and Design Chapter 11: Time and Global States

Distributed Systems Concepts and Design Chapter 11: Time and Global States

Steve Wallis, Raymond Ho, Bruce Hammer

Steve Wallis, Raymond Ho, Bruce Hammer 2Steve Wallis, Raymond Ho, Bruce Hammer 2

11.1: Introduction Time and Global States

Time is a quantity we need to measure accurately Synchronization of Clocks Development Of Algorithms Relative Order Of Events Notion of Physical Time – this results from the need to

know the order in which a pair of events or whether they occur simultaneously

Steve Wallis, Raymond Ho, Bruce Hammer 3Bruce Hammer, Steve Wallis, Raymond Ho 3

11.2: Clocks, events and Process States

Model of interaction between the processes within a distributed system History of process



Clocks Clocks are electronic devices Operating System reads the node’s hardware

clock, scales it and adds an offset to produce a software clock

In general, clock is not completely accurate



Clock skew and clock drift

Computer clocks tend not to be in perfect agreement

Clock skew - Instantaneous difference between the readings of any two clocks is called their skew


11.2: Clocks, events and Process States Clock drift – computer crystal-based clocks count

time at different rates, and so diverge



Coordinated Universal Time Synchronize computer clocks to external source

of highly accurate time Such as atomic clock with drift rate is about one

part in 1013 Output from these atomic clocks known as

International Atomic Time Seconds and years and other time units are rooted

in astronomical time


11.2: Clocks, events and Process States Coordinated Universal Time (UTC)

International standard for timekeeping Based on atomic time UTC signals are synchronized and broadcast regularly

from land-based radio stations and satellites to many parts of the world


11.3: Synchronizing physical clocks External synchronization - For process

accountancy purposes, necessary to synchronize the processes’ clocks with an authoritative, external source of time


11.3: Synchronizing physical clocks Internal synchronization – measure of the interval

between two events occurring at different computers by appealing to their local clocks


11.3: Synchronizing physical clocks Clocks not necessary synchronized internally and

externally Common to define a hardware clock H to be

correct if it drift rate falls with a known bound p>0, mean the error in measuring the interval between real times t and t’ (t’ >t) is bound

Monotonicity – is the condition that a clock C only ever advances


11.3: Synchronizing physical clocks Faulty – A clock that does not keep to whatever

correctness condition apply


11.3: Synchronizing physical clocks Synchronization in a synchronous system

Synchronous System Has minimum transmission time min Also has maximum transmission time max So transmission time be u = (max - min) Optimum bound on clock skew when synchronizing N

clocks is u(1 – 1/N)


11.3: Synchronizing physical clocks Asynchronous System

Found in most distributed systems Message delays are not bounded in their effect No upper bound max on message transmission

delays T trans = min + x, where x>0, x is not known in a

particular case, although a distribution of values may be measurable for a particular installation


11.3: Synchronizing physical clocks Cristian’s method for synchronizing clocks

Probabilistic: Algorithm achieves synchronization only if he observed round-trip times between client and server are sufficiently short compared with the required accuracy


11.3: Synchronizing physical clocks Discussion of Cristian’s algorithm

Cristian method suffers from the problem associated with all services implemented by a single server but the server might fail and thus render synchronization impossible temporarily

Cristian suggested that time should be provided by a group of synchronized time servers and use only the first reply obtained


11.3: Synchronizing physical clocks The Berkeley algorithm

Algorithm from Gusella and Zatti for internal synchronization

Developed for collections of computers running Berkeley UNIX

A coordinator computer to act as the master This computer periodically polls the other

computers whose clocks are to be synchronized, called slaves


11.3: Synchronizing physical clocks Master estimates the local clock times by

observing the round-trip times sent from each slave computer

Master sends the amount by which each individual slaves clock requires adjustment

The algorithm eliminates readings from faulty clocks

Should the master fail, then another can be elected to take over and function as it predecessor


11.3: Synchronizing physical clocks The Network Time Protocol (NTP)

NTP [Mills 1995] defines an architecture for a time service and a protocol to distribute time information over the Internet

Provide a service enabling clients across the Internet to be synchronized accurately to UTC.

NTP employs statistical techniques for the filtering of timing data and discriminates between the quality of timing data from servers


11.3: Synchronizing physical clocks To provide a reliable service that can survive

lengthy losses of connectivity To enable clients to resynchronize sufficiently

frequently to offset the rates of drift found in most computers

To provide protection against interference with the time service, whether malicious or accidetal


11.3: Synchronizing physical clocks NTP service is provided by a network of servers

located across the Internet Primary servers are connected directly to a time

source such as a radio clock receiving UTC Secondary servers are synchronized wit primary

servers The servers are connected in a logical hierarchy

called a synchronization subnet


11.3: Synchronizing physical clocks


11.3: Synchronizing physical clocks The subnet levels are called strata Primary servers occupy stratum 1 at the root Secondary servers are stratum 2, and so on The lowest-level (leaf) serves execute in users’

workstations The synchronization subnet can reconfigure as

servers become unreachable or failures occur


11.3: Synchronizing physical clocks NTP servers synchronize with one another in one

of three modes: Multicast mode – intended for use on a high-speed

LAN, can achieve only relatively low accuracies Procedure-call mode – one server accepts requests

from other computers and reply with its timestamp, it is suitable for higher accuracies than multicast

Symmetric mode – intended for use by the servers that supply time information in LANs and by the higher levels (lower strata) of synchronization subnet to achieve highest accuracies


11.3: Synchronizing physical clocks In all modes, messages are delivers unreliably,

using the standard UDP Internet transport protocol

In procedure-call and symmetric modes, processes exchange pairs of messages

Each message bears timestamps of recent message events




11.3: Synchronizing physical clocks NTP calculates an offset for each pair of

messages sent between two servers Which is an estimate of the actual offset between

the two clocks, a delay, which is the total transmission time for the two messages

If the true offset of the clock at B relative to that at A is o, and if the actual transmission times for m and m’ are t ad t’ respectively then we have:




11.3: Synchronizing physical clocks In general, NTP server engages in message

exchanges with several of its peers with data filtering

NTP applies a peer-selection algorithm to examines the values obtained from exchanges with each of several peers. The output may cause a server to change the peer that it primarily uses for synchronization


11.3: Synchronizing physical clocks NTP employs a phase lock loop model, which

modifies the local clock’s update frequency in accordance with observations of its drift rate


11.4: Logical Time and Logical Clocks Ordering Of Events

For a single process, events are ordered uniquely by times on the local clock

Clocks cannot be synchronized perfectly across a distributed system, so we cannot use general time to find out the order of a pair of events occurring within it.

In the ordering of events, there are 2 obvious points If 2 events occurred at the same process pi (i=1,2,..,N)

then they occurred in the order pi observes them. This is denoted by i


11.4: Logical Time and Logical Clocks Ordering Of Events

Whenever a message is sent between processes , the sending event always occurs before the receiving event.

Happened Before■ This partial ordering was named the ‘happened before’

relation by Lamport ■ Also known as casual ordering or potential casual

ordering■ The ‘happened before’ relation is denoted by


11.4: Logical Time and Logical Clocks Happened Before (contd)

■ Definitions of HB■ For any message m, send(m) receive(m) , where send(m) is

the sending event and receive(m) the receiving event ■ If e, e` and e`` are 3 events where ee`, and e`e`` then ee``

■ HB does not imply actual causality, If event e1 happens before event e2 it does not imply e1 caused e2

■ If 2 events a and b are not related i.e they occur in different processes and there is no chain of messages between them, so neither ab or ba is true, then they are concurrent and denoted as a||b


11.4: Logical Time and Logical Clocks Happened Before (contd)

p1

p2

p3

a b

c d

e f

m1

m2

Physicaltime


11.4: Logical Time and Logical Clocks Logical Clocks

Happened Before ordering can be captured numerically by a mechanism called a logical clock

Invented by Lamport A software counter with no relationship to any physical

clock Each process pi keeps its own Logical Clock Li

This is used to apply the Lamport Timestamp to the event


11.4: Logical Time and Logical Clocks Logical Clocks (contd)

The Lamport Timestamp is denoted by Li(e)

To capture the happened before relation , processes update their Logical Clocks and transmit the values as follows:

LC1: Li is incremented by 1 before each event is issued at process pi : Li = Li + 1

LC2: when process pi sends message m, it appends to the message the value t = Li

On receiving message (m,t) process pj computes


11.4: Logical Time and Logical Clocks Logical Clocks (contd)

Lj = max (Lj, t) and applies LC1 before time stamping the event receive(m).

For the happened before relation it then follows for 2 events a, b that a b => L(a) < L(b)

Note : converse if not true : if L(b) > L(a) then it does not follow ab (they may be concurrent e.g a || b)


11.4: Logical Time and Logical Clocks Logical Clocks

■ Lamport Timestamps

a b

c d

e f

m1

m2

21

3 4

51

p1

p2

p3

Physical time


11.4: Logical Time and Logical Clocks Totally Ordered Logical Clocks

Some pairs of distinct events, generated by different processes, have numerically equivalent Lamport Timestamps

A total order of events – one for which all pairs of distinct events are ordered – can be created by taking into account the identifiers of the processes at which events occur.

If e is the event at process pi, with local Timestamp Ti, then global logical timestamp is denoted as (Ti, i)


11.4: Logical Time and Logical Clocks Vector Clocks

■ Vector Clocks were developed by Mattern and Fidge to overcome the shortcoming of Lamport clocks

■ That is if L(e`) > L(e) we cannot conclude that ee`■ A Vector Clocks for a system of N process is an array of

N integers

■ Each process pi keeps its own Vector Clock Vi , and like Lamport timestamps processes piggy back Vector timestamps to messages they send to other processes.


11.4: Logical Time and Logical Clocks Vector Clocks (contd)

■ Vector Clocks are updated as follows :■ VC1 Vi[j]=0 for i, j=1,2,…,N

■ VC2: Vi is incremented by 1 before each event is issued at process pi : Vi [i] = Vi [i] +1

■ VC3: process pi includes the value t= Vi in every message it sends

■ When a process pi receives a Vector timestamp in a message it sets Vi [j]=max (Vi [j],t[j]) for j=1,2,…,N


11.4: Logical Time and Logical Clocks Vector Clocks

a b

c d

e f

m1

m2

(2,0,0)(1,0,0)

(2,1,0) (2,2,0)

(2,2,2)(0,0,1)

p1

p2

p3

Physical time


11.4: Logical Time and Logical Clocks Vector Clocks (contd)

■ Comparing Vector Timestamps : V = V′ if V[j]=V′ [j] for j=1,2,…,N V ≤ V′ if V[j] ≤ V′[j] for j=1,2,…,N V < V′ if V ≤ V′ V ≠ V′⋀

■ As in Logical Clocks, it follows that : For 2 events e, e′ that e e′ => V(e) < V(e′ )

■ But now the converse IS true If V(e) < V(e′ ) then e e′

Steve Wallis, Raymond Ho, Bruce Hammer 44

11.5: Global States Using Vector Timestamps

Reconstruct the State of a system thru logical sequencing of events across multiple processes

Find out the state of a process Whether a particular property of the system or a

process is true or has been true


11.5: Global States Examples

Distributed Garbage Collection – Are there any references to an object anywhere in the

distributed system? If no, reuse memory

Deadlock Detection Are two processes just waiting for each other to send?

Distributed Termination Detection Has a process terminated or is it merely waiting for a

response? Similar to Deadlock


Figure 11.9Cuts

m1 m2

p1

p2Physical

time

e10

Consistent cutInconsistent cut

e 11

e 12

e 13

e 20

e 21

e 22


11.5: Global States Slicing a distributed system to determine the

system’s state or a process’s state The vector times, as noted above, sequence

the system’s messages and processes Cut 1 is inconsistent because the receipt of the

message by p2 is included but the sending by p1 is not


11.5: Global States Linearization or Consistent Run

An ordering of all the events in a cut or global history that is consistent with happened-before

Determine what state is reachable from sequence of events

Global State Predicate Function that maps from the set of global states of

processing in the system to true or false Termination, deadlock, garbage


11.5: Global States Chandy and Lamport’s ‘snapshot’ algorithm

Goal to record a snapshot of channels and processes at any point in time

Uses a designated collector process Each process records its state and its messages Uses marker messages – initiates the snapshot

Marker receiving rule – obligates a process that has not recorded its state to do so

Marker sending rule – obligates a process to send a marker after they have recorded their state and before they send messages


11.5: Global States (contd)Reachability between states in the snapshot algorithm

Sinit – the global state before the first process recorded its state Ssnap – the constantly-changing global state Sfinal – the global state when the snapshot terminates Upper linearization is Sys; Lower linearization is Sys’ Establishes the reachability relationship

Sinit Sfinal

Ssnap

actual execution e0,e1,...

recording recording begins ends

pre-snap: e'0,e'1,...e'R-1 post-snap: e'R,e'R+1,...


11.6: Distributed Debugging Can we determine whether the system was

stable or transitory in an execution? Practical example – were all the pipes in the

factory open at one time? Uses the snapshot algorithm Uses a central monitor, which lies outside the

system, (Marzullo and Neiger)


11.6: Distributed Debugging a global state predicate Possibly – there is a consistent global state S

through which a linearization of H (history) passes such that (Sis true

Definitely all linearizations of H passes such that (S) is true

Evaluating possibly entails a search through all consistent global states derived from the observed execution


11.6: Distributed Debugging Processes may react to what condition is

being tested for and respond accordingly The monitor must assemble consistent global

states against which it evaluates The monitor uses the vector clock values of

each message to determine the consistent global state.


11.6: Distributed Debugging For possibly ,

the monitor process starts at the initial state and steps through all consistent states reachable from that point, evaluating at each stage. When is true, it stops

For definitely The monitor process must attempt to find a set of

states through which all linearizations must pass, and at each of which is true


Figure 11.16Algorithms to evaluate possibly and definitely


11.6: Distributed Debugging Cost of distributed debugging

Development of code to respond to the snapshot request, with different variables

Processing of the request and messaging cost to the monitor

Cost of comparing all the states of each of the observed processes

Space of storing all the values for comparison


11.6: Distributed Debugging Debugging in a synchronous system

An asynchronous system requires the vector timestamps to determine a consistent global state

A synchronous system can send vector timestamps and physical timestamps to the monitor. The monitor can then determine more accurately

whether events occurred simultaneously, crossed paths, et al.


11.7: Summary Items Covered

The importance of accurate timekeeping for distributed systems

Algorithms for synchronizing clocks The ‘happened before’ relation Logical Clocks and Vector Clocks Concepts of events, local and global histories, local and

global states, consistent states Algorithms of Chandy/Lamport and Marzullo/Neiger


11.7: Summary

‘Time Is An Illusion – Lunchtime Especially So’

(Ref : Adams D – Hitch Hikers Guide To the Galaxy 1977)

Documents

Distributed Systems Concepts and Design Chapter 11: Time and Global States