CCACommon Component Architecture

Manoj Krishnan

Pacific Northwest National Laboratory

MCMD Programming and Implementation Issues

CCACommon Component Architecture

2

Motivation

• The challenges in developing large-scale applications are …– Addressing complexity

• Improve productivity

– Scaling to massive number of processors• How applications can exploit the massive amount of

parallelism available in teraflop and petaflop-scale systems

CCACommon Component Architecture

3

Multilevel Parallelism in Computational Chemistry: Our Approach

• Proposed solution to improve scalability– Increase granularity of computation => improve the overall

scalability.– Exploitation of multiple levels of parallelism (MLP)

• Instead of execution entire application on the full set of processors, assign parts of application to appropriately-sized subsets of processors

• Many apps qualify

– Challenge: Difficult to implement

• Use advanced tools to address programming complexity• Common Component Architecture (CCA)

• Global Arrays (GA) shared-memory programming model

• Objective: To demonstrate how CCA and GA can be used together to address requirements of real scientific applications

CCACommon Component Architecture

4

Technology

• Technologies for exploiting multiple level parallelism– Global Arrays (GA) shared-memory programming

model• High level parallel data management abstractions

– Common Component Architecture (CCA)• Component technology for HPC applications• Hiding complexity• Enables composition of software modules written in

different languages and programming styles

Driver

Gradient GradientGradient Gradient

Energy EnergyEnergy

EnergyEnergyEnergy

Energy

Energy

Energy Energy Energy Energy

CCA

QM

CCACommon Component Architecture

5

Multiple Component Multiple DataModel

• Introducing Multiple Component Multiple Data– i.e. multiple program multiple data (MPMD) model in

context of CCA– instantiating components on subgroups of processors – create a dynamic environment to partition

computational resources and manage them to execute the overall application effectively

• Facilitate dynamic behavior of the application itself for example – Resizing processor groups based on memory

requirements or scaling characteristics– swapping components based on numerical or

computational performance

CCACommon Component Architecture

6

Numerical Hessian Example

0.1

1

10

100

0 32 64 96 128 160 192 224 256 288

Processors

Tim

e (

ho

urs

)one-level (native)

• Numerical Hessian Algorithm– determination of energy

second derivatives through numerical differentiation of gradients, which may in turn be obtained from numerical differentiation of energies

• Multiple gradient calculations– Each gradient has multiple

energy calculations

• limited scalability• Not effectively utilizing variable

degrees of parallelism

Gradient

Energy

Hessian

CCACommon Component Architecture

7

Numerical Hessian Scalability - I

0

20

40

60

80

100

120

1 2 4 8 16 32 64 128

Processors

Para

llel E

ffic

ien

cy

Single Energy CalculationSingle Energy Calculation

0.1

1

10

100

0 32 64 96 128 160 192 224 256 288

Processors

Tim

e (

ho

urs

)

one-level (native)

two-level (groups)

• Single energy calculation does not scale beyond 4 processes*

• Two-level Parallelism– Native parallel code – Energy

level– group-based energy

calculations at gradient level• using GA processor groups

QM Gradient

Energy

Energy

Energy

0

10

20

30

40

50

60

70

16 32 64 128 256

Processors

% o

verh

ead native-groups

CCACommon Component Architecture

8

Multilevel Parallelism

• Combining SPMD and MPMD paradigms

• MCMD – Multi Component Multiple Data• MPMD + Component

• The MCMD Driver launches multiple instances of NWChem QM components on subsets of processors (CCA)

• Each NWChem QM (gradient) component does multiple energy computations on subgroups (GA)

MCMD Hessian DriverMCMD Hessian DriverGoGocProps

cPropsModelFactory

ModelFactoryNWChem_QM_1NWChem_QM_1

ModelFactory

ModelFactory

cPropscProps

Param PortParam Port

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

NWChem_QM_0NWChem_QM_0

ModelFactory

ModelFactory

cPropscProps

Param PortParam Port

NWChem_QM_2NWChem_QM_2

ModelFactory

ModelFactory

cPropscProps

Param PortParam Port

NWChem_QM_nNWChem_QM_n

ModelFactory

ModelFactory

cPropscProps

Param PortParam Port

Driver

Gradient GradientGradient Gradient

Energy EnergyEnergy

EnergyEnergyEnergy

Energy

Energy

Energy Energy Energy Energy

CCA

QMGradient

Energy

Energy

Energy

CCACommon Component Architecture

9

Multiple Component Multiple Data (CCA’s MCMD Model)

MCMD DriverMCMD DriverGoGocProps

cPropsModelFactory

ModelFactory

BuilderBuilder

Builder ServiceBuilder ServiceBuilder

BuildercPropscProps

QM_0QM_0ModelFact

ory

ModelFactory

cPropscProps

Parameter Parameter

QM_0QM_0ModelFact

ory

ModelFactory

cPropscProps

Parameter Parameter

QM_0QM_0ModelFact

ory

ModelFactory

cPropscProps

Parameter Parameter

QM_0QM_0ModelFact

ory

ModelFactory

cPropscProps

Parameter Parameter

MCMD Driver• Create new components • Create processor groups• Assign processor groups to

components• Connect components • Collect results

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

EnergyEnergy

Collect Results

CCACommon Component Architecture

10

Numerical Hessian Scalability - II

0.1

1

10

100

0 32 64 96 128 160 192 224 256 288

Processors

Tim

e (

ho

urs

)

one-level (native)

two-level (groups)

three-level (groups + CCA)

Application efficiency improved 10x times on 256

CPUs

Driver

Gradient GradientGradient Gradient

Energy EnergyEnergy

EnergyEnergyEnergy

Energy

Energy

Energy Energy Energy Energy

CCA

QM

• Three-level ParallelismThree-level Parallelism• Energy-Level

– Native parallel code• Gradient-Level

– group-based single energy calculations using GA groups

• Hessian Level– Task-based gradient

calculations using CCA

CCACommon Component Architecture

11

Potential Applications Relevant To This Approach

• Molecular Dynamics• Monte Carlo

– Growth nucleation

• Numerical Hessians– Vibrational spectra

• Optimization techniques– Simulated annealing with local optimization

• Nudged Elastic Band methods– Determine reaction path for kinetic rates

• Trajectory simulations

CCACommon Component Architecture

12

MCMD Programming

• Multi-level parallelism– Nested parallel decomposition– Possibly multiple levels of parallelism– Multiple parallel simulations are run concurrently in

a coupled fashion, exchanging data at boundaries or perhaps even within volumes.

CCACommon Component Architecture

13

MCMD Services

• Develop MCMD services to support MLP– Creating and management of processor groups

• CCA Represenation for Groups id, membership

– Mapping of component to groups and their coordination• Coordination of concurrent and nested SCMD/MCMD tasks

– Communication between groups– Dynamic reconfiguration– Handling termination of processor groups, components

• MCMD as a service or a component ?

CCACommon Component Architecture

14

Activities

• Year 1:– Develop a model to express Multi-level parallelism through

processor groups– Requirements gathering and design of flexible dynamic

multi-level parallelism model– Coordinate & interact with other initiatives (ongoing)

• Year 2– Define a CCA Standard way of specifying and translating

processor group membership and mapping between components

• Year 3, 4, 5.– …

CCACommon Component Architecture

15

Implications of MCMD for CCA model

• Model for Applications with Multi-Level Parallelism – Important

• Process group abstraction – compatible with MPI, PVM, GA, GAS languages, HPCS languages (?)– MPI as default ? Group translators– How to address threaded components? OpenMP?

Pthreads? Processor group for a threaded component?

• Group-awareness to CCA and a CCA way of naming groups– i.e. multi-level parallelism at the CCA level/BuilderService

CCACommon Component Architecture

16

Implications of MCMD for CCA Implementations

• Processor group management• Run-time configuration

– At run-time, user should be able to blow-up connections, create components and assign groups

– Swapping components, ..

• Mapping communicators• Overlapping/Disjoint processor groups

CCACommon Component Architecture

17

Summary - Found MCMD Effective

• Implemented a flexible, multi-level software architecture for computational chemistry applications – Exploits variable levels of parallelism – A order of magnitude of performance improvement

• Hiding complexity and enabling better s/w composition

• MCMD model has potential for addressing scalability in future large scale systems

• More work is needed in CCA infrastructure and s/w to take advantage for larger class of apps

– Facilitate dynamic groups• Make MCMD easier to adopt for apps