Simple Load Balancing

CS550Operating Systems

Announcements

• Project will be posted– TBA

• This project will use the client-server model and will require implementation of load balancing

• We will review square matrix multiplication as an example of load balancing

Parallel decomposition and load balancing

• Parallel decomposition - how do we divide up work and do work in parallel.

• Load balancing - how do we distribute this work so that it is best divided between resources (CPUs/nodes for cluster computing)

• For addition of square matrices A and B, we could divide the work by rows and try to allocate the same number of rows to each thread or process.

Example

• Assume we have two N by N matrices, A and B• Assume we have K threads or processes where

K < N.• We can divide up the work as follows:– Allocate N / K rows to each process or thread and

perform addition.– Note that N / K is an integer division.

Example

• Given two 16*16 matrices A and B, divide the work between 4 threads

• So, each thread should calculate 16 / 4 = 4 rows of addition.

What if things don't divide up evenly?

• Given two 1024 * 1024 matrices, divide the work between 5 threads.

• Using the previous formula, we would allocate 1024 / 5 = 204 rows to each thread to perform addition.

• But 204 * 5 = 1020. What about the other 4 rows?• Notice that 1024 % 5 gives the remainder. We must

allocate the rest of the work to at least one of our existing threads.

• We could try to balance the work again if necessary.

Other Load Balancing Problems

• Finding pi• Numerical Integration• Particle Physics Simulations• Computational Chemistry• Weather• Air Traffic Control

• Sometimes dividing work (load balancing) can be very difficult – see IEEE and ACM journals.