LECTURE 7
DR. SAMMAN H. AMEEN
Message Passing Models
and
Multicomputer distributed
system
1
PAGE 2
Message-passing
direct network
interconnection
Node Node Node Node
Node
Node
Node
Node
Node Node Node Node
PAGE 3
• Computers that rely on message passing for communication rather than cache coherent shared
memory are much easier for hardware designers to build
• There is an advantage for programmers as well, in that communication is explicit, which means there are
fewer performance surprises than with the implicit communication
• The downside for programmers is that it's harder to port a sequential program to a message- passing
computer, since every communication must be identified in advance or the program doesn't work.
• Cache-coherent shared memory allows the hardware to figure out what data needs to be communicated,
which makes porting easier. There are differences of opinion as to which is the shortest path to high
performance, given the pros and cons of implicit communication, but there is no confusion in the
marketplace today. Multicore microprocessors use shared physical memory and nodes of a cluster
communicate with each other using message passing.
PAGE 4
• Advantages
• Easier to build than scalable shared memory machines
• Easy to scale (but topology is important)
• Programming model more removed from basic hardware operations
• Coherency and synchronization is the responsibility of the user, so the system designer need not worry about them.
• Disadvantages
• Large overhead: copying of buffers requires large data transfers (this will kill the benefits of multiprocessing, if not kept to a minimum).
• Programming is more difficult.
• Blocking nature of SEND/RECEIVE can cause increased latency and deadlock issues.
PAGE 5
• Basic communication constructs/ primitives
• Send(destination, message)
• Receive(source, message)
• Here, source/destination is process name or link mailbox or port
• Issues in message passing communication
• Direct Vs. Indirect
• Blocking Vs. Non‐blocking
• Reliable Vs. Unreliable
• Buffered Vs. Unbuffered
PAGE 6
• Unique process name
• This can be done by concatenating host name and process id to identify the source/destination
• Note that the format for both send and receive primitives are symmetric
• There is a bidirectional communication between source and destination. That means, for every send primitive there is a receive primitive
A B
PAGE 7
p2@machine2
Format is Symmetric,
.
.
Send(p2@machine2, buffer[])....
Communication is bidirectional (forevery Send/Recv, there is Recv/Send
command)
p1@machine2
Recv(p1@machine1, buff[])
.
.
.
.
PAGE 8
• Receive message from unknown source
• Source in Receive primitive is a variable which gets the value (sender id) when the message is received.
• Asymmetric sender and receive primitive. [Only sender need to specify the receiver id and receiver may receive the message from any sender]
• Unidirectional path between sender and receiver [No waiting for specific sender by receiver]
C B
A
PAGE 9
.
.
.
.
.
.variable = p1@machine1, on receipt ofmessage from p1@machine1variable = p2@machine2 on receipt ofmessage from p2@machine2variable = p3@machine3 on receipt ofmessage from p3@machine3.
.
PAGE 10
• This type of communication is used when sending processes are not concerned with the identity of receiving processes and receiving processes are interested only in the message and not the identity of sending processes.
• Example: Multiple clients may request services from one of the multiple servers.
Network
Clients Servers
Any client request may be served by any available server. So sender (client)and receiver (server) concerned onlywith data and not id of each other
PAGE 11
Process
Kernel
Network
When sending the message from source to destinationprocess, the message is passed to sender kernel, whichis then pass over communication network, then toreceiver kernel; finally to the destination process
PAGE 12
• While sending the message, synchronization can take place at one or more of the following points• Between Sender and Sending kernel
• Between Sending kernel and Receiving kernel
• Between Receiving kernel and Receiver
• Based on above synchronization points, we have different kind of semantics under send and receive primitives.
• Buffering space is present in the sender’s kernel, receiver’s kernel and in the communication network. These can be logically combined into a single buffer.
PAGE 13
• Blocking receive:
• Receive has only one semantic, that is the process has to wait till a message is received by the process
RECEIVE
.Recv(......)
.
.
Se
n
d
PAGE 14
• Non blocking:
• The sender process is released after the message has been composed and copied onto sender’s kernel
• This is asynchronous send
• Buffer space is assumed to be unbound
• The sender process is blocked if the sending kernel is not ready to accept the message
SEND
PAGE 15
Blocking send:
– Sender process is released after the message has been transmitted into the communication network
Reliable blocking send:
– Sender process is released after the message has been received by the destination kernel
Explicit blocking send:
– Sender process is released after the message has been received by the destination (receiver) process
Request and reply:
– Sender process is released after the message has been received by the receiver and response is returned to the sender
– Also called Client‐Server communication
SEND
PAGE 16
SEND
Sending Process Receiving Process
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
PAGE 17
SEND
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
NON BLOCKING SEND (till message reaches S kernel)
Sending Process Receiving Process
PAGE 18
SEND
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
BLOCKING SEND (till message reaches n/w)
Sending Process Receiving Process
PAGE 19
SEND
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
RELIABLE BLOCKING SEND (till message reaches R kernel)
Sending Process Receiving Process
PAGE 20
SEND
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
Sending Process Receiving Process
EXPLICIT BLOCKING SEND (till message reaches receiver)
PAGE 21
SEND
SendingKernel
ReceivingKernel
Network
SENDER IS BLOCKED
Sending Process Receiving Process
EXPLICIT BLOCKING SEND (till message reaches receiver)NON BLOCKING SEND (till message reaches S kernel)BLOCKING SEND (till message reaches n/w)RELIABLE BLOCKING SEND (till message reaches R kernel)
endi ern
ving el
SK
EXPLICIT BLOCKING SEND (till message reaches receiver)
ngel
ReceiKern
Network
SENDER IS BLOCKED
REQUEST AND REPLY (till response from receiver reaches back sender)
Sending Process Receiving Process
PAGE 22
• It is the communication endpoint of communication link and is managed by the transport services.
• Sockets are used for communication between processes in heterogeneous domain.
• It is established by socket system call, which returns a descriptor and informs the kernel about the protocol (TCP/UDP) the process is going to use.
• The system call is described by – (protocol, local address, local process/port, foreign address, foreign process/port)
PAGE 23
SERVER
Socket Socket ProtocolProtocol
Local address+ port
To the Local address +port given by the systemBind() Connect()
Ready toacceptconnection
Listen()
Accept()Block untilconnectionfrom client
Data(Request)
Read() Write()
Processrequest Write() Read()
Data(Reply)
CLIENT
PAGE 24
Socket SocketProtocol
Bind() Bind()Block untildata receivedfrom theclient Receive From() Send To()
Send To() Receive From()
SERVER CLIENT
PAGE 25
• Idea is group of processes receive messages from source• If group size = N, communication is broadcasting
• Else (group size < N) communication is multicasting
• N : number of processes in a DS
• The source may belong to the same group of recipients
• The source may or may not belong to the same group of recipients
PAGE 26
• Unanimity:
• A message is delivered either to all processes in the group or not at all (atomic property )
• Termination• The result of a message submission to the broadcast server is known to
the submitted broadcast server within a finite time period
• Uniformity:
• If a message is delivered to a process in a group, then it should be delivered to all the processes in that group
• Order:• BS will deliver the messages to APs in the group based on some
predefined ordering requirements
PAGE 27
• Wide area network (WAN); A WAN connects a large number of computers that are spread over large geographic distances. It can span sites in multiple cities, countries, and continents.
• Metropolitan area network (MAN); The MAN is an intermediate level between the LAN and WAN and can perhaps span a single city.
• Local area network (LAN); A LAN connects a small number of computers in a small area within a building or campus.
• System or storage area network (SAN). A SAN connects computers or storage devices to make a single system.
PAGE 28
• A network channel c=(x,y) is characterized by
• width wc: the number of parallel signals it contains,
• frequency fc: the rate at which bits are transported at each signal
• latency tc is the time required for a bit to travel from x to y.
• A bandwidth of a channel is W= wc * fc.
• The throughput Θ of a network is the data rate in bits per second that network accepts per input port.
• Under a particular traffic pattern, the channel that carries the largest fraction of the traffic determines the maximum channel load γ. Load on the channel can be equal or smaller than channel bandwidth.
• Θ=W/γ
PAGE 29
• Deterministic: The simplest algorithm - for each source, destination pair, there is a single path. This routing algorithm usually achieves poor performance because it fails to use alternative routes, and concentrates traffic on only one set of channels.
• Oblivious: So named because it ignores the state of the network when determining a path. Unlike deterministic, it considers a set of paths from a source to a destination, and chooses between them.
• Adaptive: The routing algorithm changes based on the state of the network.
PAGE 30
• Message: logical unit for internode communication
• Packet: basic unit containing destination address for routing
• Packets have sequencing # for reassembly
• Flits: flow control digits of packets
PAGE 31
• Header flits contain routing information and sequence number
• Flit ( FLow control digITs) length affected by network size
• Packet length determined by routing scheme and network implementation
• Lengths also dependent on channel b/w, router design, network traffic, etc.
PAGE 32
• 100 Gigabit Ethernet (100GbE) and 40 Gigabit Ethernet (40GbE) are groups of computer networking technologies for transmitting Ethernet frames at rates of 100 and 40 gigabits per second (100 and 40 Gbit/s), respectively. The technology was first defined by the IEEE 802.3ba-2010 standard.
• InfiniBand (abbreviated IB) is a computer network communications link used in high-performance computing featuring very high throughput and very low latency. It is used for data interconnect both among and within computers. As of 2014 it is the most commonly used interconnect in supercomputers.
PAGE 33
• Latency is an element that contributes to network speed.
• The term latency refers to any kind of delay typically incurred in processing of network data.
• A low latency connection is one that generally experiences small delay times, while a high latency connection generally suffers from long delays.
PAGE 34
In the context of parallel computing, granularity is the
ratio of communication time over computation time.
Fine grain parallelism is characterized by seemingly
more communications as the relative computation time
is shorter. Coarse grain parallelism, then, is
characterized by seemingly fewer communications with
much longer computation time.
Load balance is easier to achieve with fine grain parallelism
because small tasks depend less on the operating system,
interrupts and so on. Coarse grain parallelism, on the converse,
makes it harder to predict when any given task will terminate,
therefore making it harder to assign tasks for optimal usage of
the multiple processors.
Fine grain parallelism requires more synchronization overhead
due to the need to communicate data and synchronize tasks
among processors. Therefore, the fewer communications in
coarse grain parallelism reduces overhead.