NIH Resource for Macromolecular Modeling and Bioinformatics Beckman Institute, UIUC Charm++ Hits and Misses A NAMD Perspective

NIH Resource for Macromolecular Modeling and Bioinformaticshttp://www.ks.uiuc.edu/

Beckman Institute, UIUC

Charm++ Hits and MissesA NAMD Perspective

James PhillipsBeckman Institute, University of Illinoishttp://www.ks.uiuc.edu/Research/namd/



National Center for Supercomputing Applications



Beckman InstituteUniversity of Illinois at

Urbana-Champaign

Theoretical and ComputationalBiophysics Group



Computational Microscopy

Ribosome: synthesizes proteins from genetic information, target for antibiotics

Silicon nanopore: bionanodevice for sequencing DNA efficiently



NAMD: Scalable Molecular Dynamics37,000 Users, 1700 Citations

Blue Waters Target Application

2002 Gordon Bell Award

Illinois Petascale Computing Facility

PSC LemieuxATP synthase Computational Biophysics Summer School

GPU Acceleration

NCSA LincolnNVIDIA Tesla



NAMD: Practical Supercomputing• 37,000 users can’t all be computer experts.

– 8800 have downloaded more than one version.

– 1700 citations of NAMD reference papers.

• One program for all platforms.– Desktops and laptops – setup and testing

– Linux clusters – affordable local workhorses

– Supercomputers – free allocations on TeraGrid

– Blue Waters – sustained petaflop/s performance

• User knowledge is preserved.– No change in input or output files.

– Run any simulation on any number of cores.

• Available free of charge to all.

Phillips et al., J. Comp. Chem. 26:1781-1802, 2005.



• Spatially decompose data and communication.• Separate but related work decomposition.• “Compute objects” facilitate iterative, measurement-based load balancing system.

NAMD Hybrid DecompositionKale et al., J. Comp. Phys. 151:283-312, 1999.



NAMD Overlapping Execution

Objects are assigned to processors and queued as data arrives.

Phillips et al., SC2002.



Shallow valleys, high peaks, nicely overlapped PME

green: communication

Red: integration Blue/Purple: electrostatics

turquoise: angle/dihedral

Orange: PME

94% efficiency

Apo-A1, on BlueGene/L, 1024 procs

Charm++’s “Projections” Analysis too

Time intervals on x axis, activity added across processors on Y axisl



System Noise ExampleTimeline from Charm++ tool “Projections” http://charm.cs.uiuc.edu/



The NAMD-PPL Partnership

• NIH-funded collaboration since 1992– C++ was new and poorly understood– Charm++ was born (from Chare kernel)– PVM was popular (original NAMD target)– MPI was new

• Symbiotic relationship– PPL provides tools and expertise– NAMD provides feedback and use cases



“Parallel Programming” Lab

• MPI is not “parallel programming”– Limited domain: “Message-Passing Interface”– Concurrent execution, yes, but not programming– Just sequential programs exchanging data– Lockstep execution

• What is “parallel programming”?– Pthreads, OpenMP, Intel TBB - threads and shared data– CUDA, OpenCL - many many threads and shared data– Charm++ - message-driven execution



Triumphs of MPI

• Keep the easy things easy

• Make the easy things standard

• Make promises implementations can keep– If runs with no buffers, will run with any size

• Simple model for libraries

• Designed for reliability at scale



Limitations of MPI

• Must deal with process rank and count

• Procedural programming only

• Just a library, no runtime

• Adding more idioms over time– One-sided get/put– Best idiom can vary by platform



Triumphs of Charm++

• Driven by deep application collaborations

• Support for message-driven programming– Also support other models like AMPI

• Principle of persistence

• Make hard things possible

• Details are for tuning, not implementation

• Stay alive and relevant despite MPI



Limitations of Charm++

• Make easy things cumbersome

• Limited guarantees on resource usage

• Many idioms to learn (and document)

• Hard to debug without program counter– Why is it hanging? What didn’t happen?



Little-used Charm++ in NAMD

• Arrays

• Parameter marshalling

• Chare migration

• Reductions

• Multicasts with nokeep messages

• Node groups

• Multiphase shared arrays



Under-appreciated Charm++

• Groups (ye olde branch-office chares)

• Quiescence detection

• Cooperative threads

• Structured dagger (SDAG)

• Message priorities

• Standalone mode



NAMD Development Priorities

• Reliability and Maintainability– Avoid features likely to break– Keep non-performance-sensitive code simple

• Performance– Specialized NAMD load balancers– Roll our own broadcast and reduction trees



Charm++ Idioms in NAMD

• Working solutions to old problems

• Some driven by specific requirements

• Some result from inertia

• Newer solutions may not be better



Top-level Tcl Interpreter

• Tcl used to parse config file on pe 0

• “run nsteps” or eof calls scheduler

• NAMD startup reads files and runs

• After last step do QD to exit scheduler

• Scheduler returns to Tcl interpreter

• Tcl parses script until next “run”, repeat

• E.g., used to implement replica exchange



Startup Phases

• In parallel after reading input data on pe 0

• Used to data, construct and link chares

• Phases separated by quiescence detections

• Simplest way to ensure side effects done

• Allows diagnosis of:– What operation is crashing or hanging– What operation is using memory or time



Flow-Controlled Broadcasts

• Need to send data at startup that approaches size of available memory on the node.

• Not enough memory to create a message.

• Set up stream, broadcast series of small messages - used for molecule at startup.

• New version uses SDAG to limit messages in flight - used to distribute grid force data.



Patches, Proxies, and Computes

• Patches store atomic data

• Patches send data to proxies on other pes

• Compute objects reference local proxies

• Enqueued when referenced proxies ready

• Compute objects write results to proxies

• Proxies send results when computes done

• Patch continues when all proxies return



Controller and Sequencers

• Non-chare objects with cooperative threads

• Code looks like traditional “outer loop”– Global controller receives reductions, prints

energies, propagates global dynamic variables, broadcasts global variables to sequencers

– Sequencers (one per patch) receive broadcasts, drive atomic force calculation and integration

• Suspend until results ready, then resume



Manager Groups

• Most objects are not chares - predates arrays• Managers are like Charm++ array manager:

– Store objects of a given type on a pe– All message code is in manager class– Can aggregate and optimize communication:

• Patch to proxy communication uses trees

• Atom migration aggregated per pe

– Computes like PME have their own managers



Pencil Arrays in PME

• PME is FFT-based full electrostatic method• Charge grid aggregated on each PE• Arrays used for thick pencils in x,y,z dims• SDAG used to express communication

– Grid to z to y to x to y to z to grid

• Not over-decomposed!– At most one pencil of each type per PE– On enough PEs, x & z on different PEs from y to

overlap sending and receiving– Could use groups, but arrays code is easier to read.



Coordinated Message Priorities

• Computes enqueued as messages

• Priorities based on critical path– Earlier step before later step– Earlier PME stages before later stages– Computes for remote patches before local

• Modules must be coordinated!– All priorities defined in Priorities.h– Diagnose and confirm using Projections



Coordinated Object Placement

• Patches don’t move

• Distribute patches across machine– Pay attention to topology– Place computes to minimize communication

• Keep PME off of patch PEs

• Keep PME near referenced patches

• Keep computes off of PME PEs



Grainsize Tuning

• Maximum patch size is based on cutoff• Ideally one or more patches per processor

– To double, split in x dimension (two-away x)– Then split in y dimension (two-away x,y)– Then split in z dimension (two-away x,y,z)– Number of computes grows much faster!

• Hard to automate completely– Also need to select number of PME pencils



Best New Charm++ for NAMD

• SMP and physical node awareness, affinity

• Posix shared memory for data sharing

• Network topology awareness

• Non-MPI support for high-speed networks

• CkDirect for RDMA-capable networks

• Mpirun startup for non-mpi builds

• Distributed hybrid load balancer



NAMD Wishes for Charm++

• Make large bulk data transfers trivial– As easy as MPI synchronous send/recv

• Support procedural idioms better– Apply algorithms to data like STL

• Support functional idioms better– Maintain a parallel stack and return to caller




• Make SDAG look more like threads– Generate support code for nested calls– Put atomic bodies in def.h, not decl.h

• Compartmentalize quiescence detection– Only track message traceable back to caller

• Support node-level execution better– Execute chare entry on any pe in node– Compare priorities between pe and node queues




• Document and promote special features– Ensure use beyond target application– Avoid solving same problem twice

• Catalogue Charm++ design patterns– Huge design space to choose from– Many ways to solve the same problem– Which is preferable under what situation?



Thanks, and keep up the good work!

James PhillipsBeckman Institute, University of Illinoishttp://www.ks.uiuc.edu/Research/namd/

Documents

NIH Resource for Macromolecular Modeling and Bioinformatics Beckman Institute, UIUC Charm++ Hits and Misses A NAMD Perspective