Charm++

Migratable Objects + Active Messages + Adaptive Runtime=

Productivity + Performance

Laxmikant V. Kale‡Anshu Arya, Nikhil Jain, Akhil Langer, Jonathan Lifflander, Harshitha Menon

Xiang Ni, Yanhua Sun, Ehsan Totoni, Ramprasad Venkataraman‡, Lukasz WesolowskiParallel Programming Laboratory

Department of Computer Science, University of Illinois at Urbana-Champaign‡{kale, ramv}@illinois.edu

1. SUMMARYCharm++ is an elegant, general-purpose parallel program-

ming model backed by an adaptive runtime system [1]. Thiscombination yields portable performance and a spectrumof real-world productivity benefits that have been demon-strated in production applications. Our submission to thisyear’s HPC Challenge (Class II) comprises six benchmarksthat demonstrate these practical benefits available to appli-cation developers today. These include: (1) interoperabil-ity with MPI (2) automatic, dynamic load balancing (3) ef-fortless checkpointing and restart on a different number ofprocessors, and (4) application progress in the presence offailing processes. The submission also explains how over-decomposed, message-driven, migratable objects enhance theclarity of expression of parallel programs and also enable theruntime system to deliver portable performance. Our code-size (line counts) and a summary of the performance resultscan be seen in Table 1.

1.1 Submitted Benchmarks• Required

– Global FFT

– Random Access

– Dense LU Factorization

• Optional

– Molecular Dynamics

– Sparse Triangular Solver

– Adaptive Mesh Refinement

1.2 Demonstrated Capabilities• Automatic communication-computation overlap

• Asynchronous, nonblocking collectives

• Interoperability with MPI

• Automated, dynamic load balancing

• Checkpointing to save application state

• Resilience to failing processes

• Modular, yet composable parallel libraries

2. PROGRAMMING MODELWe describe relevant aspects of the Charm++ program-

ming model in order to set the context for explaining thebenchmark implementations.

2.1 Salient FeaturesThe Charm++ programming model has several funda-

mental characteristics that enable high performance whileproviding high productivity.

2.1.1 Object-basedParallel programs in Charm++ are implemented in an

object-based paradigm. Computations are expressed in termsof work and data units that are natural to the algorithm be-ing implemented and not in terms of physical cores or pro-cesses executing in a parallel context. This immediately hasproductivity benefits as application programmers can nowthink in terms that are native to their domains.

The work and data units in a program are C++ objects,and hence, the program design can exploit all the benefits ofobject-oriented software architecture. Classes that partici-pate in the expression of parallel control flow (chares) inheritfrom base classes supplied by the programming framework.They also identify the subset of their public methods thatare remotely invocable (entry methods). This is done in aseparate interface definition file described in subsection 2.2.Charm++ messages can also be C++ classes defined by theprogram. Any work or data unit that is transmitted acrossprocesses has to define a serialization operator, called pup()for Pack/UnPack. This allows the transmission of complexobjects across processes during execution.

Chares are typically organized into indexed collections,known as chare arrays. Chares in an array share a type, andhence present a common interface of entry methods. A singlename identifies the entire collection, and individual elementsare invoked by subscripting that name. Code can broadcasta message to a common entry method on all elements of anarray by invoking that method on the array’s name, withoutspecifying a subscript. Conversely, the elements of an arraycan perform asynchronous reductions whose results can bedelivered to an entry method of the array itself or of anyother object in the system.

In essence, Charm++ programs are C++ programs where

Productivity Performance

Code C++ CI Benchmark Driver Total Machine Max Performance HighlightSubtotal Cores

Required Benchmarks

1D FFT 54 29 83 102 185 BG/P 64K 2.71 TFlop/sBG/Q 16K 2.31 TFlop/s

Random Access 76 15 91 47 138 BG/P 128K 43.10 GUPSBG/Q 16K 15.00 GUPS

Dense LU 1001 316 1317 453 1770 XT5 8K 55.1 TFlop/s(65.7% peak)

Additional Benchmarks

Molecular Dynamics 571 122 693 n/a 693 BG/P 128K 24 ms/step (2.8M atoms)BG/Q 16K 44 ms/step (2.8M atoms)

AMR 1126 118 1244 n/a 1244 BG/Q 32k 22 steps/sec, 2d mesh,max 15 levels refinement

Triangular Solver 642 50 692 56 748 BG/P 512 48x speedup on 64 coreswith helm2d03 matrix

Table 1: Performance and line count summaries for each benchmark. ‘C++’ and ‘CI’ refer to Charm++ code necessary tosolve the specified problem. ‘Driver’ refers to additional code used for setup and verification. All numbers were generatedusing David Wheeler’s SLOCCount. Performance data is simply a summary highlight. Detailed performance results can befound in the following sections

interactions with remote objects are realized through aninheritance and serialization API exposed by the runtimeframework.

2.1.2 Message-DrivenMessaging in Charm++ is sender-driven and asynchro-

nous. Parallel control flow in Charm++ is expressed in theform of method invocations on remote objects. These in-vocations are generally asynchronous, i.e., the control re-turns to the caller before commencement or completion ofthe callee, and thus no return value is available. If data needsto be returned, it can flow via subsequent remote invocationby the callee on the original caller, be indirected through acallback, or the call can explicitly be made synchronous.

These semantics immediately fit well into the object-basedparadigm. Each logical component (chare) simply uses itsentry methods to describe its reactions when dependencies(remote events or receipt of remote data) are fulfilled. Itis notified when these dependencies are met via remote in-vocations of its entry methods. Upon such invocation, itcan perform appropriate computations and also trigger otherevents (entry methods) whose dependencies are now fulfilled.The parallel program then becomes a collection of objectsthat trigger each other via remote (or local) invocations bysending messages. Note that these chares do not have toexplicitly expect a message arrival by posting receives. In-stead, the arrival of messages triggers further computation.The model is hence message-driven.

Its worthwhile to note that the notion of processes / coreshas not entered this description of the parallel control flowat all. This effectively separates the algorithm from anyprocessor-level considerations that the program may haveto deal with; making it easier for domain experts to expressparallel logic.

2.1.3 Over-decomposedDivorcing the expression of parallelism (work and data

units) from the notion of processes / cores allows Charm++programs to express much more parallelism than the avail-able number of processors in a parallel context. The funda-mental thesis of the Charm++ model is that the applica-tion programmer should over-decompose the computationsin units natural to the algorithm, thereby creating an abun-dance of parallelism that can be exploited.

2.1.4 Runtime-AssistedOnce an application has been expressed as a set of over-

decomposed chares that drive each other via messages, thesecan be mapped onto the available compute resources andtheir executions managed by a runtime system. The pro-gramming model permits an execution model where the run-time system can:

• maintain a queue of incoming messages, and deliverthem to entry methods on local chares.

• overlap data movement required by a chare with entrymethod executions for other chares.

• observe computation / communication patterns, andmove chares to balance load and optimize communi-cation.

• allow run-time composition (interleaving) of work fromdifferent parallel modules.

A list of advantages of the Charm++ programming andexecution model can be found at http://charm.cs.illinois.edu/why/.

2.2 Interface DefinitionsThe Charm++ environment strives to provide a conve-

nient, high-level interface for parallel programmers to usein developing their applications. Rather than requiring pro-grammers to do the cumbersome and error-prone bookkeep-ing necessary to identify the object and message types andassociated entry methods that make up their program, Charm++provides code generation mechanisms to serve this purpose.When building a Charm++ program, the developer writesone or more interface description files (named with a .ci ex-tension), listing the following:

1. System-wide global variables set at startup, known asread-only variables,

2. Types of messages that the code will explicitly con-struct and fill, specifying variable-length arrays withinthose messages, and

3. Types of chare classes, including the prototypes oftheir entry methods.

Interface descriptions can be decomposed into several mod-ules that make up the overall system, generated declarationsand definitions for each of which will be placed in a separatefile (named with .decl.h and .def.h extensions, respectively).A Charm++ program’s starting point, equivalent to main()in a C program, is marked as a mainchare in its interfacedescription. The runtime starts the program by passing thecommand-line arguments to the mainchare’s constructor.

2.3 Describing Parallel Control FlowOne effect of describing a parallel algorithm in terms of in-

dividual asynchronous entry methods is that the overall con-trol flow is split into several pieces, whose relationship canbecome complicated in any but the simplest program. Forinstance, one pattern that we have observed is that objectswill receive a series of similar messages that require someindividual processing, but overall execution cannot proceeduntil all of them have been received. Another pattern issome entry methods must execute in a particular sequence,even if their triggering messages may arrive in any order.To facilitate expressing these and other types of higher-levelcontrol flow within each object, Charm++ provides a no-tation known as Structured Dagger [2]. Structured Daggerlets the programmer describe a sequence of behaviors in anobject that can be abstractly thought of as a local task de-pendence DAG (from which the name arises).

The basic constructs for describing parallel control floware drawn from the traditional control-flow constructs in C-like languages: conditional execution based on if/else, andfor and while loops. The constructs operate much like com-mon sequential C or C++ code. In addition to basic Cconstructs, the when clause indicates that execution beyondthat point depends on the receipt of a particular message.Each when clause specifies the entry method it is waiting for,and the names of its argument(s) in the body of the clause.A when clause may optionally specify a reference number ornumerical expression limiting which invocations of the asso-ciated entry method will actually satisfy the clause. A refer-ence number expression appears in square brackets betweenan entry method ’s name and its argument list. Definitionsfor entry methods named in when clauses are generated bythe interface translator. Though not used in this submission,

the overlap construct describes divergent control flow, akinto a task fork/join mechanism specifying that a sequenceof operations must complete, but not an ordering betweenthem. Because a chare is assigned to a particular core atany given time, these operations do not execute in parallel,avoiding many problems encountered by concurrent code.

The various constructs are implemented in generated codeby a collection of message buffers and trigger lists that havenegligible overhead comparable to a few static function calls.If the body of a method is viewed as a sequential thread ofexecution, a when can be seen as a blocking receive call.However, rather than preserving a full thread stack andexecution context, only the current control-flow location issaved, and control is returned to the scheduler. Local vari-ables to be preserved across blocks are encoded as membervariables of the enclosing chare. In contrast, thread contextswitches that provide the same behavior with a less sophis-ticated mechanism can be much more costly. Charm++does provide mechanisms to run entry methods in separatethreads when blocking semantics are needed for other pur-poses. Also, unlike linear chains of dependencies in threads,Structured Dagger allows one to express a DAG of depen-dencies between messages and local computations.

2.4 Demonstrated Capabilities

2.4.1 Automated Dynamic Load BalancingAs HPC hardware architectures and applications become

more complex, and execution scales continue to grow larger,load imbalances will intensify further. Making decisions suchas when to balance load, and what strategy to use are al-ready quite challenging in the context of large applications.Multiple application runs may be needed to tune for the bestvalues for all the parameters that control the application’sexecution and utilization profiles. Charm++ has a maturemeasurement-based load balancing framework, which relieson the principle of persistence [3]. During application exe-cution, the load balancing framework continuously monitorsthe performance profile, and on invocation, it redistributesthe load without any requirement from the application. Wehave demonstrated beneficial load-balancing strategies forseveral scientific applications.

The latest addition to the Charm++ load balancing frame-work is the Meta-Balancer, which automates decision mak-ing related to load balancing: when to invoke the load bal-ancer and what load balancing strategy to use. Meta-Balanceruses a linear prediction model on the collected informationabout the application, and predicts the time steps (or it-erations) at which load balancing should be performed foroptimal performance. By observing the load distribution,the idle time on processors and the volume of communicationduring an application run, it also identifies a suitable balanc-ing strategy. This relieves application writers from conduct-ing theoretical analysis and multiple test runs to tune theload balancing parameters for the best performance. Giventhe increased complexity of applications and systems, such aanalysis may not even be feasible. Meta-Balancer has beenshown to be effective for various real-world applications andbenchmarks [4], and has been used to improve the perfor-mance of LeanMD in this submission.

2.4.2 Checkpointing Application StateFor HPC applications running for days and hours, it is

MPI

Charm++Time...

P(1) P(2) P(N-1) P(N)

...

P(1) P(2) P(N-1) P(N)

...

P(1) P(2) P(N-1) P(N)

(a) Time Sharing (b) Space Sharing (c) Combined

Figure 1: Supported modes of interoperation betweenCharm++ and MPI

difficult to get the allocations needed at one time. Charm++provides support to checkpoint the application state to diskfor split execution. The application can even be restartedusing the checkpoint file on a different number of processorsthan the original run (e.g., a 10000 processors run can becheckpointed and restarted on 9000 processors). To performcheckpointing, the user only needs one extra line of code:

CkStartCheckpoint(”log”,callback)

Here “log” is the filesystem path to the checkpoints. call-

back is called when the checkpoint (or restart) is complete.

2.4.3 Tolerating Process FailuresApplications running on large scale machines may face a

high failure frequency as the number of components in everygeneration of supercomputers keeps increasing. It will be-come increasingly difficult for applications to complete exe-cution without any fault tolerance support. Charm++ offersa double in-memory fault tolerance mechanism for applica-tions running on unreliable systems.

The in-memory based checkpointing algorithm helps re-duce the checkpoint overhead in comparison to the file sys-tem based checkpointing scheme by storing the checkpointin memory. It therefore avoids any congestion caused bysimultaneous access to the file system.

In the scheme, periodically at application synchronizationpoints, two copies of checkpoints are stored; one in the lo-cal memory of the processor and the other in the memoryof a remote processor (the “buddy” processor of the localprocessor). Charm++ automatically detects failures usinga periodic hearbeat, and chooses a new processor to replacethe crashed processor. This new processor can either be apreviously unused spare processor, or a currently used pro-cessor that is already part of the execution. The most recentcheckpoint is then copied to the chosen replacement proces-sor from the buddy of the crashed processor. Thereafter,every processor rolls back to the last checkpoint, and theapplication continues to make progress. The fault tolerancemechanism provided by Charm++ can be used by addingonly one line of code:

CkStartMemCheckpoint(callback)

callback is called when the in-memory checkpoint(or restart)is complete. To enable this feature, the syncft option shouldbe added to the configuration when compiling Charm++.

1 //MPI Init and other basic initialization2 { optional pure MPI code blocks }34 //create a communicator for initializing Charm++5 MPI Comm split(MPI COMM WORLD, peid%2, peid, &

newComm);6 CharmLibInit(newComm, argc, argv);78 { optional pure MPI code blocks }9

10 //Charm++ library invocation11 if(myrank%2)12 fft1d(inputData,outputData,data size);1314 //more pure MPI code blocks15 //more Charm++ library calls1617 CharmLibExit();18 //MPI cleanup and MPI Finalize

Figure 2: Sample code showing a Charm++ library(FFT1D) being used from an MPI program

We inject failures into the application by randomly select-ing a process and forcing it to become completely silent andunresponsive. The “failed” process simply ceases to commu-nicate with any other process in the execution. All applica-tion and runtime state on this process is lost to the rest ofthe execution. The runtime system, upon failing to receiveheartbeat signals from the “failed” process, flags it as a pro-cess failure and commences recovery protocol. The silencingfunctionality is encapsulated behind a CkDieNow() functionthat is called on the randomly selected failing process.

2.4.4 Interoperating with MPICharm++ provides support for multi-paradigm program-

ming where only a subset of application modules are writ-ten in Charm++, while the remaining use MPI (or MPI +OpenMP). Any legacy MPI application can use a libraryimplemented in Charm++ in the same way that it uses anyexternal MPI library. This also provides a gradual migra-tion path to existing applications that wish to adopt theCharm++ programming model.

Charm++ supports interoperability in three different modesas shown in Figure 1. (1 ) In the time sharing mode, thecontrol transfers between Charm++ and MPI in a coordi-nated manner for all the processors in the execution. (2 ) Inthe space sharing mode, a set of processors always executeCharm++ code, while the rest execute the MPI program. (3) In the general case, time sharing and space sharing can beused simultaneously by dividing the MPI ranks using MPIcommunicators during initialization, and then, making callsto Charm++ libraries as desired.

A simple interface function is added to the Charm++library to permit invocation from MPI. In Figure 2, wepresent example usage of a one-dimensional FFT librarywritten in Charm++, from an MPI program. The state-ments of interest are the additional calls to CharmLibInitand CharmLibExit, which are required once at the begin-ning and end of execution, respectively. After such initial-ization, Charm++ libraries can be invoked at any time usingthe interface methods.

2.4.5 Nonblocking Collective Communication

256 512 1024 2048 4096 8192 16384 32768 6553610

1

102

103

104

Cores

GF

lop/

s

BG/PBG/QSerial FFT limit (BG/P)

Figure 3: Performance of Global FFT on IBM’s BG/Q andBG/P

Sizes BG/P BG/Qm Cores GFlops Cores GFlops

327682 256 19.4 128 43.5460802 512 39.9 256 72.5655362 1024 69.1 512 133.8921602 2048 229.2 1024 220.7

1310722 4096 284.1 2048 325.51802242 8192 454.4 4096 634.12621442 16384 867.9 8192 974.73604482 32768 1430.8 16384 2306.25242882 65536 2708.9 - -

Table 2: Performance of Global FFT 1D on Intrepid(IBM BG/P) and Vesta (IBM BG/Q)

Charm++ supports a number of collective communicationpatterns using high performance implementations. Somecollectives, such as broadcast, are expressed by overload-ing basic operations on proxies. Broadcast, as an exam-ple, is expressed using the same function invocation syntaxas point-to-point sends, with the distinction that the entrymethod is invoked on a collection instead of a single chare orprocessing element. The intuitive expression of broadcastsis an example of the productivity benefits of Charm++.

For more complex cases, Charm++ provides communica-tion libraries which support the most commonly used col-lectives: broadcasts, reductions, multicasts, and many-to-many/all-to-all. All collective communication in Charm++is naturally asynchronous and non-blocking, and overlapswith other computation and communication that may in-volve the same cores or objects.

Some Charm++ libraries are more subtle than the typi-cal collective calls of a message passing library. One of thesespecialized libraries, which we found useful in improving per-formance and elegance of two of our benchmarks (FFT andRandom Access) is Mesh Streamer. Mesh Streamer is alibrary for accelerating all-to-all communication. By aggre-gating small units of communication into larger messages,and routing these messages over the dimensions of a virtualmesh comprising the processing elements involved in the run,Mesh Streamer improves fine-grained all-to-all communica-tion performance. The library is asynchronous and highlyflexible in supporting communication patterns that are notfully all-to-all. As an example, the various processing ele-ments participating in the collective are free to send differ-ent numbers of messages, with varying message sizes, as partof the same streaming step. Processing elements may evenavoid participating in the collective completely. Further,many-to-many exchanges are also supported, and a singlestreaming step can even support numerous localized many-to-many exchanges in the same streaming step, and combineintermediate messages which are not part of the same many-to-many operation. In many ways, Mesh Streamer typifiesthe flexibility with which the Charm++ programming modelcan handle dynamic communication patterns which do notbelong to commonly accepted patterns.

The library also permits clean expression of communica-tion in the application. For example, in the random accessbenchmark, the receipt of the update messages from remote

processors is expressed as follows:

1 void process(const T &key) {2 CmiInt8 localOffset = key & (localTableSize − 1);3 // Apply update to local portion of table4 HPCC Table[localOffset] ˆ= key;5 }

The code snippet looks like it is receiving individual up-dates one at a time. However, the library handles the dataaggregation and routing completely invisibly under the ap-plication, producing very succinct application code.

3. GLOBAL FFT

3.1 PerformanceRuns were performed on Intrepid (IBM’s Blue Gene/P)

and Vesta (Blue Gene/Q). Performance scales well up to atleast 64K cores of Blue Gene/P, in large part due to the useof our message aggregation and software routing commu-nication library, Mesh Streamer (also used in the RandomAccess benchmark). ESSL was used to perform serial FFTs.Figure 3 and table 2 summarize the results.

3.2 ImplementationOur implementation of Global FFT takes input size N

and performs a complex 1D FFT on an NxN matrix wheresubsequent rows are contiguous data elements of a doubleprecision complex vector. Three all-to-all transposes are re-quired to perform the FFT and unscramble the data. All-to-all operations are done via a general Charm software rout-ing library, Mesh Streamer, and external libraries (FFTWor ESSL) perform serial FFTs on the rows of the matrix.

3.2.1 Interoperable with MPIThe FFT library submitted as part of this code can also in-

teroperate with programs written in MPI, and the library inCharm++. The general concept, and the simplicity of use ofinteroperation in Charm++ was described in Section 2.4.4.In order to use the FFT library with MPI, one should buildCharm++ using MPI as the base layer. For more details,one can refer to the README and the interface code thathas been submitted as part of this submission for example

0.0625

0.25

1

4

16

64

128 512 2K 8K 32K 128K

GU

PS

Number of cores

43.10

Perfect ScalingBG/PBG/Q

Figure 4: Performance of random access on Intrepid (IBMBG/P) and Vesta (IBM BG/Q)

Cores BG/P BG/QGUPS Mem(TB) GUPS Mem(TB)

128 0.11 0.031 0.242 0.062256 0.21 0.062 0.403 0.125512 0.40 0.125 0.751 0.2501K 0.71 0.250 1.327 0.5002K 1.38 0.500 2.360 1.0004K 2.49 1.000 4.623 2.0008K 4.43 2.000 8.204 4.000

16K 8.34 4.000 15.001 8.00032K 15.13 8.00 - -65K 26.94 16.00 - -

128K 43.10 32.00 - -

Table 3: Performance of random access on Intrepid (IBMBG/P) and Vesta (IBM BG/Q)

usage. Note that the same library code has been used byboth the drivers (in MPI and in Charm++).

3.3 VerificationThe benchmark code self-validates by executing an inverse

FFT and the residual is printed.

4. RANDOM ACCESS

4.1 Productivity

4.1.1 Communication LibrariesCharm++ contains libraries for improving network com-

munication performance for various scenarios. For this bench-mark we use the Mesh Streamer library, further describedbelow, for optimizing all-to-all communication on small mes-sages.

4.1.2 Automatic Topology DeterminationThe Charm++ Topology Manager automates the task of

determining physical network topology for a partition as-signed to a job. We use it to provide topology informationfor Mesh Streamer.

4.2 PerformancePerformance results on IBM BG/P and BG/Q are pre-

sented in Figure 4 as well as in Table 3. We achieve over 35GUP/s when running on 128K cores of BG/P.

4.3 ImplementationWe used a Charm++ group to partition the global table

across the nodes in a run. A group can be thought of asa chare array with one chare per Processing Element (PE),which corresponds to either a core or a thread in the system.For example, on Blue Gene/P, we ran using a single PE percore, while on Blue Gene/Q, we ran in a mode employingone PE per thread, and up to 4 threads per core. Eachelement of the group allocates its part of the global table,generates random update keys, and sends the updates to theappropriate destination.

The small size of individual update messages in this bench-mark made it prohibitively expensive to send each item asa separate message. To improve performance, we used aCharm++ message aggregation and routing library calledMesh Streamer. Mesh Streamer routes messages over an N-dimensional virtual topology comprising the PEs involved inthe run. The topology needs to be specified when creatingan instance of the library. To facilitate the specification ofthe virtual topology, we used the Charm++ Topology Man-ager library, which provides an interface to determine thenetwork topology for the current run. In typical cases, wefound that using virtual topologies with dimensions match-ing the network representation of the current run led to goodperformance.

In the context of Mesh Streamer, each processor is lim-ited to sending and receiving messages from a subset of theprocessors involved in the run. When determining whereto send a particular data item (in our case table update),Mesh Streamer selects a destination from among its peersso that data items always make forward progress toward thedestination. Items traveling to the same intermediate des-tination are combined into larger messages. This approachachieves improved aggregation and lower memory utiliza-tion compared to schemes which aggregate separately foreach destination on the network.

Mesh Streamer allows specifying a limit on the numberof items buffered by each local instance of the library. Weset the buffering limit to 1024 to conform to the benchmarkspecifications.

4.4 VerificationVerification is done by performing a second phase with the

same updates as in the timed run. In the absence of errors,this has the effect of returning the global table to its initialstate. To verify correctness, we check the state of the tableto ensure its final state is the same as when the benchmarkis started. In our implementation memory is distributedamong the members of the group. Each region of memoryis only accessed by one group member, making the imple-mentation thread-safe. As such, our implementation doesnot allow for even the small number of errors permissible

according to the benchmark specification.

5. DENSE LU FACTORIZATION

5.1 ProductivityCharm++ is a general and fully capable programming

paradigm, and hence our LU implementation does not em-ploy any linear algebra specific notations. Our implementa-tion is very succinct and presents several distinct benefits.

5.1.1 Composable LibraryThe implementation is presented as a modular library that

can be composed into client applications. By composition,we imply both modularity during program design and seam-lessness during parallel execution. Factorization steps canbe interleaved with other computations from the applica-tion (or even with other factorization instances of the samelibrary!) on the same set of processors.

5.1.2 Flexible Data PlacementThe placement of matrix blocks on processes is completely

independent of the main factorization routines; is encapsu-lated in a sequential function, and can be modified withminimal effort. We have utilized this ability to explore novelmapping schemes, demonstrating that deviating from a tra-ditional block-cyclic distribution can increase performancefor modern multicore architectures [5].

5.1.3 Block-Centric Control FlowFor block sizes on which dgemm calls perform well, we typ-

ically need hundreds of blocks assigned to each processorcore to meet the memory usage requirements. Such over-decomposition is also necessary for load balance. By elevat-ing these over-decomposed, logical, entities to become theprimary players in the expression of parallelism, Charm++enables a succinct representation of the factorization logic.Additionally, Structured Dagger allows the control flow foreach block to be directly visible in the code in a linear styleeffectively independent of other activity on the same proces-sor.

5.1.4 Separation of ConcernsThe factorization algorithm has been expressed from the

perspective of a matrix block. However, processor-level con-siderations (e.g, memory management) are implemented asseparate logic that interacts minimally with the factoriza-tion code. This demonstrates a clear separation of concernsbetween application-centric domain logic and system-centriclogic. Such separation enhances productivity of both appli-cation domain experts and parallel systems programmers.It also allows easier maintenance and tuning of parallel pro-grams.

5.2 PerformanceWe have scaled our implementation to 8064 cores on Jaguar

(Cray XT5 with 12 cores and 16GB per node) by increasingproblem sizes to occupy a constant fraction of memory (75%)as we increased the number of cores used. We obtain a con-stant 67% of peak across this range. We also demonstratestrong scaling by running a fixed matrix size (n = 96, 000)from 256 to 4096 cores of Intrepid (IBM Blue Gene/P with4 cores and 2GB per node). The matrix size was chosen tojust meet the requirements of the spec (occupying 54% of

0.1

1

10

100

128 1024 8192

Tota

l T

Flo

p/s

Number of Cores

Theoretical peak on XT5Weak scaling on XT5

Theoretical peak on BG/PStrong scaling on BG/P

Figure 5: Weak scaling (matrix occupies 75% of memory)from 120 to 8064 processors on Jaguar (Cray XT5). Strongscaling (n = 96, 000) from 256 to 4096 processors on Intrepid(IBM BG/P).

memory) at the lower end of the strong scaling curve (256cores). Our results are presented in Figure 5. Extensiveexperiments with LU are an expensive proposition as theamount of computation increases as n3 (where n is the ma-trix size). We fully expect the implementation to scale tomuch larger partitions and demonstrate high performanceon multiple architectures.

5.2.1 Adaptive LookaheadOur implementation provides completely dynamic, memory-

constrained lookahead so that panel factorizations are over-lapped as much as memory usage limits will allow. In keep-ing with its library form, applications can choose to restrictthe factorization to use only a fraction of the available mem-ory.

5.2.2 Asynchronous CollectivesCharm++ collective operations are also asynchronous like

its other messaging semantics and can be overlapped withother work. For example, this allows asynchronous pivotidentification reductions to be overlapped with updating therest of the sub-panel. Masking such latencies allows thisimplementation a wider choice of data placements.

5.3 ImplementationWe use a typical block-decomposed algorithm for the LU

factorization process. Our focus in this effort has been lesson choosing the best possible factorization algorithm thanon demonstrating productivity with a reasonable choice.

The input matrix of n × n elements is decomposed intosquare blocks of b×b elements each. We delegate underlyingsequential operations to an available high performance linearalgebra library, typically a platform-specific implementationof BLAS and perform partial pivoting for each matrix col-umn.

5.3.1 Data DistributionMatrix blocks are assigned to processors when the library

starts up, according to a mapping scheme, and are not re-assigned during the course of execution. Charm++ facil-itates the expression and modification of the data distri-

bution scheme by encapsulating the logic into a simple se-quential function call that uses the block’s coordinates tocompute the process rank it should be placed on:

process rank = f(xblock, yblock) (1)

This is a standard feature in Charm++ and is availableto all indexed collections of chares (chare arrays). This al-lows library users to evaluate data distribution schemes thatmay differ from the traditional two-dimensional block-cyclicformat.

5.3.2 Asynchrony and OverlapIn our implementation, each block is placed in a message-

driven object, driven by coordination code written in Struc-tured Dagger [2]. The coordination code describes the mes-sage dependencies and control flow from the perspective ofa block. Thus, every block can independently advance asit receives data and we avoid bulk synchrony by allowingprogress in the factorization when dependencies have beenmet. With many blocks per processor, overlap is automati-cally provided by the Charm++ runtime system.

5.3.3 Block SchedulerOur solver implements dynamic lookahead, using a dy-

namic pull-based scheme to constrain memory consumptionbelow a given threshold. To implement the pull-based scheme,we place a scheduler object on each processor in additionto its assigned blocks. The scheduler object maintains alist of the blocks assigned to its processor, and tracks whatstep they have reached. Within the bounds of the memorythreshold, it requests blocks from remote processors that areneeded for local triangular solves and trailing updates. Anearlier technical report [6] describes the dependencies be-tween the blocks and how the scheduler object uses this tosafely reorder the selection of trailing updates to execute.We include this in our submission to demonstrate that theprogramming model allows separation of concerns in a par-allel context.

5.4 VerificationOur LU implementation conforms fully to the spec and

passes the required validation procedures for all the resultspresented here. We have supplied a test driver with thelibrary that generates input matrices, invokes the library forthe factorization and solves, and validates the results whilealso measuring performance. Performance and validationstatistics are printed at the end.

6. MOLECULAR DYNAMICSLeanMD is a molecular dynamics simulation program writ-

ten in Charm++. This benchmark simulates the behaviorof atoms based on the Lennard-Jones potential, which isan effective potential that describes the interaction betweentwo uncharged molecules or atoms. The computation per-formed in this code mimics the short-range non-bonded forcecalculation in NAMD [7, 8] and resembles the miniMD ap-plication in the Mantevo benchmark suite [9] maintained bySandia National Laboratories.

The force calculation in Lennard-Jones dynamics is donewithin a cutoff-radius, rc for every atom. In LeanMD, thecomputation is parallelized using a hybrid scheme of spatialand force decomposition. The three-dimensional (3D) sim-ulation space consisting of atoms is divided into cells of di-

1024

2048

4096

8192

16384

32768

1024 2048 4096 8192 16384 32768

Spee

dup

Number of cores

LeanMD Speedup on BlueGene/Q (2.8 million atoms)

ms/step44.2ms/step

IdealCharm++

Figure 6: Scaling of LeanMD for the 2.8 million atoms sys-tem on Vesta (IBM BG/Q)

mensions that are equal to the sum of the cutoff distance, rcand a margin. In each iteration, force calculations are donefor all pairs of atoms that are within the cutoff distance.The force calculation for a pair of cells is assigned to a dif-ferent set of objects called computes. Based on the forcessent by the computes, the cells perform the force integrationand update various properties of their atoms – acceleration,velocity and positions.

6.1 ProductivityOur implementation of LeanMD takes only 693 lines of

code while offering capabilities that match some of the pro-duction molecular dynamics applications. In comparison,miniMD from the Mantevo benchmark suite, which nurturessimilar objectives of representing real applications, requiresjust under 3000 lines of code [9] but does not offer many ofthe capabilities of LeanMD.

Below, we present several Charm++ features that havebeen exploited in LeanMD that significantly improve pro-grammer productivity without sacrificing performance andin some cases, such as load balancing, lead to performanceimprovements.

6.1.1 Automatic Load Balancing with Meta-BalancerCharm++’s fully automated measurement-based load bal-

ancing framework enables strong scaling with minimal effortfrom the application programmer. It is critical in an appli-cation like LeanMD, which can suffer from substantial loadimbalance because of the variation in sizes of computes re-sulting from a spherical cutoff. To enable automatic loadbalancing decisions using Meta-Balancer, the user simplyspecifies a flag, +MetaLB, at the command line, and therun-time system will automatically identify a suitable loadbalancing period. Measurement-based load balancing is wellsuited for applications where the recent past is a predictorof the near future. In such situations, load balancing deci-sions taken by Meta-Balancer based on such predictions areoften correct. This works well for LeanMD because atomsmigrate slowly and hence the load fluctuations are gradual.

6.1.2 In-disk and in-memory CheckpointingApplications that perform physical simulation are exe-

cuted for a long period (tens of hours to a few days). In sucha scenario, providing fault-tolerance mechanisms are criticalfor sustained execution. LeanMD has been used to demon-

10

100

1000

10000

2k 4k 8k 16k 32k 64k 128k

Tim

e pe

r st

ep (

ms)

Number of cores

Performance on Intrepid (2.8 million atoms)

No LBPeriodic LB

Meta LB

10

100

1000

10000

1k 2k 4k 8k 16k 32k

Tim

e pe

r st

ep (

ms)

Number of processes

Performance on BlueGene/Q (2.8 million atoms)

No LBHybrid LB

Figure 7: Performance of LeanMD for the 2.8 million atoms system on Vesta (IBM BG/Q) and Intrepid (IBM BG/P)

32

64

128

256

512

1024

2048

64 128 256 512 1024

Ela

psed

tim

e (s

)

LB Period

Elapsed time vs LB Period (BlueGene/P)

8k cores16k cores32k cores

64k cores128k cores

Figure 8: Execution time of LeanMD: variation in LB Periodfor 2.8 million atoms system on Intrepid (IBM BG/P)

Cores No LB (s) Periodic LB (s) Meta-Balancer (s)

8k 666 504 41316k 336 260 27732k 171 131 13164k 122 104 100128k 73 54 52

Table 4: LeanMD application time with various load balancingstrategies

0

10

20

30

40

50

60

2048 4096 8192 16384 32768

Tim

e(m

s)

Number of processes

LeanMD Checkpoint Time on BlueGene/Q

2.8 million1.6 million

0

20

40

60

80

100

120

140

160

2048 4096 8192 16384 32768

Tim

e(m

s)

Number of processes

LeanMD Restart Time on BlueGene/Q

2.8 million1.6 million

Figure 9: Checkpoint and restart of LeanMD for a 2.8 million atoms system on Vesta (IBM BG/Q)

strate two important checkpointing mechanisms provided byCharm++ - checkpointing to disk for split execution andcheckpointing to memory for fault tolerance support. Asmentioned earlier, selection of a checkpointing scheme inCharm++ is as simple as making a different function call.

6.1.3 Dense and Sparse Chare ArraysIndexed collection of objects in Charm++ provide an el-

egant and easy to understand abstraction for representingdissimilar but related work units. Different phases and com-putation in an application can be assigned to different charearrays. Cells are a dense 3D chare array that represent aspatial decomposition of the 3D simulation space. They areresponsible for sending positions and collecting the forces fortheir atoms. Computes, on the other hand, form a sparse6-dimensional array of chares. Such a representation makesit convenient for a pair of cells with coordinates (x1, y1, z1)and (x2, y2, z2) to use a compute with coordinates (x1, y1,z1, x2, y2, z2) to calculate forces for their atoms.

6.1.4 Ability to run variable size jobs: Charm++ enables users to run applications on any num-

ber of cores without any restrictions on the size or shape ofthe processor partitions that are used. It also allows an ap-plication to checkpoint for a particular number of processors,and restart on a different number of processors. Addition-ally, it provides the freedom to choose a convenient numberof work units, depending on the simulation, independent ofthe number of cores the application is run on. Note thatthis freedom does not come at the expense of performancewhich either improves or follows the general trend seen inthe more restricted environment.

6.1.5 Communication LibrariesCharm++ provides a set of communication libraries which

supports efficient multicast and reduction operations. Ineach iteration of LeanMD, the atoms contained in a cell aresent to every compute that needs them. Also, the resul-tant forces on atoms in a cell are obtained by a summationof the forces calculated by each compute that received thoseatoms. LeanMD exploits the ability of the runtime to gener-ate efficient spanning trees over arbitrary sets of processors.In Charm++, creation of such spanning trees is performeddynamically using asynchronous message passing withoutimposing a global barrier.

6.2 PerformanceWe present performance numbers for LeanMD on two ma-

chines: 1) Vesta - an IBM Blue Gene/Q installation at ANL,and 2) Intrepid - an IBM Blue Gene/P installation at ANL.LeanMD was run using a molecular system that consist of2.8 million atoms distributed in a uniformly random mannerin the space.

We experimented within a range of LB periods (50, 200,500, . . ., 2000) to find the period at which periodic loadbalancing gives the best performance. Figure 8 shows theapplication run time using these LB periods on various corecounts. If the load balancing is done very frequently, theoverheads of load balancing overshoots the gains of load bal-ancing, and results in bad performance. On the other hand,if load balancing is done very infrequently, the load imbal-ance in the system reduces the gains due to load balancing.

In Table 4, a comparison of total execution time of LeanMD

for the following three cases is presented - without load bal-ancing, best periodic load balancing and Meta-Balancer. Weobserve that in all the cases Meta-Balancer either improvesthe best performance obtained by periodic load balancing,or matches it. Meta-Balancer successfully identifies thatthere is an initial load imbalance in LeanMD and hence callsload balancing at the very beginning. Thereafter, the fre-quency of load balancing decreases as the change in loadsof individual chare is slow. Given the superiority of Meta-Balancer, all load balancing decisions were automated usingMeta-Balancer in the remaining experiments.

Figure 7 presents performance results for execution ofLeanMD on Vesta and Intrepid. On Vesta, time per stepdecreases linearly as the process count is increased. Thespeed up for the runs on Vesta is shown in Figure 6. Thelinear decrease in the time per step to 44 ms/step for a 2.8million atoms system on just 16384 is a result of accurateautomated decision making by Meta-Balancer. On Intrepid,the scaling results are shown from 2K cores to 128K cores.It is to be noted that although LeanMD does not scale lin-early on BG/P for large core counts, the benefits of loadbalancing are always seen. On 128K cores, performing loadbalancing reduces time per step from 36 ms to 24 ms, whichis a performance gain of 33%.

Figure 9 shows the checkpoint and the restart time for a2.8 million atoms and a 1.6 million atoms system on Blue-gene/Q. For the 2.8 million atoms system, as the numberof processes increase from 2K to 32K, the checkpoint timedecreases by 25% from 43 ms to 33 ms. Such a drop isobserved because the checkpoint data size per process de-creases as the number of processes is increased, and it over-shadows the communication overheads. Note that the timeto checkpoint decreases slowly after 8K cores when the syn-chronization overheads becomes the main bottleneck.

The restart time, which is measured from the time a fail-ure is detected to the point where the applications is recov-ered and is ready to continue execution, is shown in Fig-ure 9. Since several barriers are used to ensure consistencyuntil the crashed processor is recovered, the complexity ofthe recovery process is O(logP ) (due to the use of reductiontree for synchronization). The restart time increases from66 ms on 4K cores to 139 ms on 32K cores for the 2.8 mil-lion system. Note that the efficiency of the restart processis partly due to the fact that the double in-memory faulttolerance protocol allows the application to restart from thelast checkpoint in the local memory. Hence, an applicationcan restart without a full stop, and thus job turn-aroundtime and a new job submission are avoided.

6.3 ImplementationIn the Charm++ implementation the computation is par-

allelized using a hybrid scheme of spatial and force decom-position. The three-dimensional (3D) simulation space con-sisting of atoms is divided into cells of dimensions that areequal to the sum of the cutoff distance, rc and a margin.In each iteration, force calculations are done for all pairs ofatoms that are within the cutoff distance. The force calcula-tion for a pair of cells is assigned to a different set of objectscalled computes.

At the beginning of each time step, every cell multicaststhe positions of its atoms to the computes that need themfor force calculations. Every compute receives positions fromtwo cells and calculates the forces. These forces are sent back

l11l21 l22

l33l43 l44

l54 l55l66l76 l77

l81 l82 l83 l88l91 l92 l93 l99

Figure 10: Sparse matrix divided into blocks.

to the cells which update other properties of the atoms. Ev-ery few iterations, atoms are migrated among the cells basedon their new positions. Structured Dagger is used to controlthe flow of operations in each iteration and trigger depen-dent events. The load balancing framework is invoked peri-odically after a certain number of iterations to redistributecomputes and cells among the processors. In addition, thefault tolerance scheme performs a periodic checkpointing. Inthe submitted version, the parallel control flow is describedin the run functions of each chare in leanmd.ci. The reduc-tion for forces computed by computes is in physics.h.

6.4 Specification and VerificationFor a pair of atoms, the force can be calculated based on

the distance by,

~F =

(A

r13− B

r7

)× ~r (2)

where A and B are Van der Waals constants and r is thedistance between the pair of atoms. Table 5 lists a set ofparameters and their values used in LeanMD.

Parameter Values

A 1.1328× 10−133

B 2.23224× 10−76

Atoms per cell 700

Cutoff distance, rc 26 A

Cell Margin 2 ATime step 1 femtosecond

Table 5: Simulation details for LeanMD

The benchmark computes kinetic and potential energyand uses the principle of conservation of energy to verifythat the computations are stable. Users can choose to runthe benchmark for as many timesteps as desired, and verifi-cation statistics are printed at the end.

7. SPARSE TRIANGULAR SOLVERSolution of sparse triangular systems of linear equations is

a performance bottleneck in many methods for solving moregeneral sparse systems, such as many iterative methods withpreconditioners. It is notoriously resistant to parallelism,however, and existing parallel linear algebra packages ap-pear to be ineffective in exploiting much parallelism for thisproblem. We chose this benchmark to show the ability ofour system for challenging sparse linear algebra computa-tions. The algorithm we implement is described in detail

1 // if this chare has some diagonal part of matrix2 if (onDiagonalChare) {3 // schedule the independent computation with lower priority4 serial { thisProxy[thisIndex].indepCompute(...) }5 // ”while” and ”when” can happen in any order6 overlap {7 // while there are incomplete rows, receive data8 while (!finished) {9 when recvData(int len, double data[len], int rows[len])

10 serial {if(len>0) diagReceiveData(len, data, rows);}11 }12 // do serial independent computations scheduled above13 when indepCompute(int a) serial {myIndepCompute();}14 }15 // if chare doesn’t have diagonal part of matrix16 } else {17 // wait for x values18 when getXvals(xValMsg∗ msg) serial {nondiag compute();}19 // while there are incomplete rows, receive data20 while (!finished) {21 when recvData(int len, double data[len], int rows[len])22 serial {nondiagReceiveData(len, data, rows);}23 }24 }

Figure 11: Parallel flow of our triangular solver in StructuredDagger

elsewhere [10], and a brief summary follows. The matrix isdivided into blocks of columns, as shown with blue, greenand grey colors in Figure 10. Each block is analyzed to findits independent rows for computation. If there are denseregions below the diagonal section (purple and orange inFigure 10), they are divided into new blocks. Each diago-nal block starts the computation with its independent partsand waits for required messages from the left. Nondiagonalblocks wait for the solution values from their correspond-ing diagonal block, and then start their computation (andreceipt of other messages). This flow is expressed in theStructured Dagger code of Figure 11.

7.1 ProductivityUsing different features of Charm++, we improve both

performance and productivity, comparing to the state-of-the-art packages such as SuperLU DIST. We implement a

0.001

0.01

0.1

1

10

1 2 4 8 16 32 64 128 256 512

Solu

tion

Tim

e (s

)

Number of Cores

slu_webbase-1Mslu_helm2d03

slu_hoodslu_largebasis

SuperLU_largebasisSuperLU_webbase-1M

SuperLU_helm2d03SuperLU_hood

640µs

Figure 12: Performance of sparse triangular solver on In-trepid (IBM BG/P) for different sparse matrices. Perfor-mance is also competetive with SuperLU DIST

more complicated algorithm with only 692 total SLOCs,comparing to 879 SLOCs of the triangular solver in Su-perLU DIST. For better load balance and overlap of com-munication and computations, we divide the columns intomany blocks (more than the number of processors) and mapthem in round-robin fashion to processors. This perfor-mance optimization feature is essential for this solver andmanaging the blocks manually is burdensome for the pro-grammer. Thus, virtualization in Charm++ improves pro-ductivity significantly for this implementation. In addition,having parallel units independent of the processors and cre-ating them dynamically allows the solver to adapt to thestructure of the input matrix (create new blocks for densernonzero regions) and benefit from more parallelism. Fur-thermore, message-driven nature of the algorithm matchesperfectly to the Charm++ programming paradigm, so theoverall flow is easily expressed in Structured Dagger withhigh performance and less programming cost. As an ex-ample, doing the same message-driven computation in MPIwould require MPI Iprobe calls that cost both performanceand productivity. Also, easy usage of priorities improves theperformance with minimal programming effort.

7.2 PerformanceFigure 12 compares the performance of our implementa-

tion against the triangular solver of SuperLU DIST pack-age for different matrices on a Blue Gene/P. As shown,we achieve much better performance and scaling for all thecompared cases. For example, SuperLU DIST is about 18.5times slower than the best serial performance on 64 cores formatrix slu helm2d03, whereas our solver achieves a speedupof more than 48. SuperLU DIST uses a simple 2D decom-position approach for parallelism, which is inefficient. Ourscaling and performance is made possible by a better algo-rithm and its effective implementation in Charm++.

7.3 ImplementationIn the Charm++ implementation, column blocks are as-

signed to elements of a Chare array and mapped in round-robin fashion into processors. The denser regions (withmany nonzeros) below the diagonal sections are also dividedinto new blocks and new Chare elements are created forthem. Different Chares start their independent computationbut with a lower priority than the messages they receive,with the intention of accelerating the critical path. Notethat since there are multiple Chares per processor, a Charemay receive some messages before starting its independentcomputation, while other Chares are using the processor. Ifthere are some subdiagonal Chares corresponding to a di-agonal Chare, it will multicast its results for them (usingCkMulticast library for faster communication). These mes-sages have the highest priority in the system, since theyenable new computations and accelerate the critical path.

7.4 VerificationWe verify the solution by computing a residual as de-

scribed by the HPL specification.

8. ADAPTIVE MESH REFINEMENTTraditional AMR algorithms phrase the design in terms

of processors that contain many blocks. Instead, we pro-mote blocks to first-class entities that act as a virtual pro-cessor. As the mesh is refined or coarsened in AMR, the

number of blocks will change and traditional algorithms re-quire synchronization at these points. However, to enhanceproductively, we abstract the blocks as a collection that dy-namically expands and contracts over time. Refinement de-cisions can then be local to each block and propagated asfar as algorithmically required so blocks are always withinone refinement level of their neighbors. Because remeshingoccurs only at discrete points in the simulation time, insteadof using collective communication that is proportional to thedepth of the recursive refinement, we use a scalable termina-tion detection mechanism built into our runtime to globallydetermine when all refinement decisions have been finalized.Previous collective methods require O(d) rounds of collec-tive communication, where d is the refinement and consumeO(P ) memory per processor to store the results. By utiliz-ing termination detection, we consume a negligible amountof memory and communicate no data. Besides terminationdetection, blocks execute completely asynchronously, com-municating only with neighboring blocks when required.

Traditional AMR implementations store the quad-tree in-stance on each process consuming O(P ) memory and takingO(logN) time for neighbor lookup. We organize the blocksinto a quad-tree but address each block by their locationin the tree using bit vectors to represent quadrants recur-sively. It requires only O(#blocks/P ) memory per processand O(1) lookup time. It also frees the programmer fromhaving to know where the block lies; instead, the underly-ing runtime system manages the physical locations of eachblock and provides direct, efficient communication betweenthem. The runtime system can then redistribute the blocksperiodically without any change to the logic.

8.1 ImplementationIn our implementation, the collection of blocks is orga-

nized as a chare array that is indexed by a custom array in-dex. This index is a bit vector used to describe the block’s hi-erarchical location in the quad tree. More information aboutthe algorithm and implementation is available [11]. The par-allel control flow is expressed succinctly for the remeshingalgorithm which consists of two phases: reaching the consen-sus on granularity of each chare in phase 1, and performingthe actual refinement or coarsening in phase 2.

8.2 PerformanceWe present strong scaling results for a finite-difference

simulation of advection. A first-order upwind method in2D space was used for solving the advection equation (fig-ure 13). The benchmark attains efficiencies of 99%, 95%,65%, 55% at 2k, 8k, 16k and 32k ranks, respectively (fig-ure 14a) when the dynamic depth of the mesh is allowedto vary from 5 to 15. In order to evaluate our remeshingalgorithm, we graph the distribution of remeshing latencies(Figure 14b) of each block. The remeshing latency is thetime spent in remeshing without any computation overlap,calculated as the duration between the last processor start-ing remeshing and the beginning of the next time step.

9. CONCLUSIONCharm++ is a general-purpose parallel programming paradigm

capable of high performance, and suitable for a wide spec-trum of parallel algorithms. Over the years it has presentedabstractions and semantics that are have evolved as general-izations of successful domain-specific solutions. The bench-

Figure 13: AMR benchmark: example simulation of a circular fluid advected by a constant velocity field. From left to right,the figure displays the fluid density after 2, 572, and 1022 iterations respectively. The white squares demonstrate how themesh is evolving over time.

0.1

1

10

100

256 512 1k 2k 4k 8k 16k 32k

Tim

este

ps /

sec

Number of Ranks

IBM BG/Q Min−depth 4IBM BG/Q Min−depth 5

(a) Timesteps per second strong scaling on IBM BG/Q witha max depth of 15.

0

5

10

15

20

25

30

35

16 32 64 128 256 512 1k 2k 4k 8k 16k 32k

Rem

eshi

ng L

aten

cy T

ime

(ms)

Number of Ranks

Depth Range 4−9Depth Range 4−10Depth Range 4−11

(b) The non-overlapped delay of remeshing in milliseconds onIBM BG/Q. The candlestick graphs the minimum and max-imum values, the 5th and 95th percentile, and the median.

Figure 14: Performance results for the AMR benchmark.

marks presented here do not use any domain-specific lan-guages tailored to the problem. However, it is possible toenvision domain-specific languages or targeted, parallel ab-stractions deployed atop this general model for further pro-ductivity benefits. Our implementations should underscorethe productivity impact of the programming model and thebenefits of the approaches we have discussed in section 2.

10. READING ORDERWe suggest the following reading order, as a progressive

introduction to the various features of Charm++ used.

Global FFT.

1. main.ci – contains control flow for the driver

2. main.cc – contains serial code and initialization routine

3. fft.ci – contains control flow for FFT calculation

4. fft.cc – contains all serial code for doing an FFT

5. data.cc – contains the the data initializer, container,and residual calculation routine

Molecular Dynamics.

1. leanmd.ci – contains the parallel control flow; focus onrun() function of each chare.

2. Cell.cc – contains important functions of Cell; focus onsendPositions() and updateProperties().

3. Compute.cc – interact() function that does the forcecomputation

4. Main.cc – start point of application; invokes run()

Sparse Triangular Solver.

1. sparse solve.ci – contains the parallel control flow; focuson start() function.

2. sparse solve.C – reads and analyzes input matrix, ini-tializes solver and verifies solution

Random Access.

1. randomAccess.ci – contains chare, group, and entrymethod declarations

2. randomAccess.cc – contains the driver, rng, table up-date sends / receives, and verification code

Dense LU Factorization.

1. lu.ci – Control flow for the factorization and solve pro-cess is described here, starting from the steps thatevery block executes, then proceeding to the factor-ization steps taken by blocks in different active panelpositions (above diagonal, below diagonal, on diago-nal). The methods used during solving and startupfollow. Implementations of sequential methods calledfrom lu.ci can be found in order of reference in lu.C.These include the data copying and sending used inpivoting, and routines to set up BLAS calls.

2. mapping.h – Some data distributions available for usewith the factorization/solver library. These can beset independently of the algorithm’s computation andcommunication logic.

3. benchmark.C – Setup and validation code.

4. scheduler.C – Logic to adaptively fetch remote data andschedule block computations while remaining withinmemory constraints. Also tracks when latency-sensitiveactive panel work is occurring in order to defer bulktrailing updates.

AMR.

1. advection.ci – contains the parallel control flow for theremeshing algorithm

2. Advection.C – contains the methods that describe thelocal refinement propagation algorithm and decision-making state machine

3. QuadIndex.C – contains the data type for indexing theparallel collection via a hierarchical bit vector

AcknowledgementsResults presented here were obtained via experiments on In-trepid (IBM BG/P ALCF), Vesta (IBM BG/Q ALCF), andJaguar (Cray XT5, OLCF). We used resources of the Ar-gonne Leadership Computing Facility at Argonne NationalLaboratory, and the Oak Ridge Leadership Computing Fa-cility at Oak Ridge National Laboratory. These were sup-ported by the Ofın ↪Ace of Science of the Department of En-ergy under contracts DE-AC02-06CH11357 and DE-AC05-00OR22725, respectively.

11. REFERENCES[1] L.V. Kale and S. Krishnan. CHARM++: A Portable

Concurrent Object Oriented System Based on C++.In A. Paepcke, editor, Proceedings of OOPSLA’93,pages 91–108. ACM Press, September 1993.

[2] L. V. Kale and Milind Bhandarkar. StructuredDagger: A Coordination Language for Message-DrivenProgramming. In Proceedings of Second InternationalEuro-Par Conference, volume 1123-1124 of LectureNotes in Computer Science, pages 646–653, September1996.

[3] Laxmikant V. Kale. Some Essential Techniques forDeveloping Efficient Petascale Applications. July 2008.

[4] Harshitha Menon, Nikhil Jain, Gengbin Zheng, andLaxmikant V. Kale. Automated load balancinginvocation based on application characteristics. InIEEE Cluster 12, Beijing, China, September 2012.

[5] Jonathan Lifflander, Phil Miller, RamprasadVenkataraman, Anshu Arya, Terry Jones, andLaxmikant Kale. Mapping dense lu factorization onmulticore supercomputer nodes. In Proceedings ofIEEE International Parallel and DistributedProcessing Symposium 2012, May 2012.

[6] Jonathan Lifflander, Phil Miller, RamprasadVenkataraman, Anshu Arya, Terry Jones, andLaxmikant Kale. Exploring partial synchrony in anasynchronous environment using dense LU. TechnicalReport 11-34, Parallel Programming Laboratory,August 2011.

[7] Abhinav Bhatele, Sameer Kumar, Chao Mei, James C.Phillips, Gengbin Zheng, and Laxmikant V. Kale.Overcoming scaling challenges in biomolecularsimulations across multiple platforms. In Proceedingsof IEEE International Parallel and DistributedProcessing Symposium 2008, April 2008.

[8] Chao Mei, Yanhua Sun, Gengbin Zheng, Eric J.Bohm, Laxmikant V. Kale, James C.Phillips, andChris Harrison. Enabling and scaling biomolecularsimulations of 100 million atoms on petascale machineswith a multicore-optimized message-driven runtime. InProceedings of the 2011 ACM/IEEE conference onSupercomputing, Seattle, WA, November 2011.

[9] Michael A. Heroux, Douglas W. DoerınCer, Paul S.Crozier, James M. Willenbring, H. Carter Edwards,Alan Williams, Mahesh Rajan, Eric R. Keiter,Heidi K. Thornquist, and Robert W. Numrich.Improving performance via mini-applications.Technical report, Sandia National Laboratories,September 2009.

[10] Ehsan Totoni, Michael T. Heath, and Laxmikant V.Kale. Structure-adaptive parallel solution of sparsetriangular linear systems. October 2012.

[11] Akhil Langer, Jonathan Lifflander, Phil Miller,Kuo-Chuan Pan, , Laxmikant V. Kale, and PaulRicker. Scalable Algorithms for Distributed-MemoryAdaptive Mesh Refinement. In Proceedings of the 24thInternational Symposium on Computer Architectureand High Performance Computing (SBAC-PAD 2012).To Appear, New York, USA, October 2012.