Parallelization: Area Under a Curve

AUC: An important task in science

• Neuroscience– Endocrine levels in the body over time

• Economics– Discounting: the fee incurred for delaying a

payment over time• Pharmacokinetics and clinical pharmacology,

machine learning, medicine, psychiatry and psychology, chemistry, environmental science, fisheries and aquatic sciences, and many others

Newton and Leibniz Calculus

• Integration: exact AUC for simple curves• Problems:– Doesn’t work for many curves, even simple ones– Doesn’t work in a reasonable number of steps

• Solution: approximation

Bernhard Riemann’s methods

• Domain of the function is segmented into the widths of shapes (rectangles in this case)

• Height of the rectangle is the y-value of the function for some x-value in the segment

• Sum of the areas of the rectangles is the approximate total area

The left Riemann sum

• Rectangle height = y-value of the left-most x-value of the rectangle width

How do we approximate?

• Small number of rectangles: by hand or calculator

• Beyond a small number of rectangles: single computer

• Even more: parallel processing• More and more: cluster computer

Parallelism

• Concurrency (doing things at the same time)• Multiple flows of execution (virtual entities

that perform computations) working on the same problem

• Distributed Memory• Shared Memory• Hybrid

Flows of execution

• Processes – Distributed memory– Must communicate (message passing)

• Threads– Shared memory

• Processes with threads– Hybrid

Parallel hardware

• Multiple cores on a single compute node– Shared memory

• Multiple compute nodes sharing a network– Distributed memory

• Multiple compute nodes with multiple cores sharing a network– Hybrid

What are some standards?

• Message Passing Interface (MPI) – distributed memory/message passing

• OpenMP – shared memory• MPI/OpenMP - hybrid

How to approach the parallel algorithm

• Identify data structures – constants and variables

• Determine the process and thread locality of the data structures

• Analyze the interactions of the data structures• Lay out the algorithm (a step by step set of

instructions)

Considerations

• Assume hybrid parallelism– Distributed memory, shared memory, or serial can

be refined from hybrid• Hybrid on 1 thread per process is just distributed

memory• Hybrid on 1 process is just shared memory• Hybrid on 1 thread and 1 process is just serial

• The entire code is executed by each process• Threads only execute code for which they are

spawned

How do we visually represent the data structures?

How do we represent the data structures in writing?

What are the scopes of the data structures?

How do the data structures interact?

What is the order of interactions?

How does this look in pseudocode?

MPI functions

MPI_Reduce in more detail

(Fortran only) Store any error codes in this variable

OpenMP regions