CRPC

Center for Research on ParallelComputation

Goal: to help develop the science and technology infrastructureneeded to make scalable parallel computing truly usable—at leastas usable as conventional supercomputers.

A National Science Foundation Science and Technology Center - Rice, Caltech, Argonne, Los Alamos, Tennessee, Syracuse, UT Austin - Affiliated Sites: Boston, Drexel, ICASE, Illinois, Indiana, Maryland

Challenges: 1. Develop parallel programming methods and algorithms 2. Develop software and languages for parallel computers 3. Improve science and engineering education 4. Increase participation by minorities and women 5. Transfer knowledge to industry and research community

CRPC

What is the Problem?

• Sequential Algorithms Usually Do Not ParallelizeEasily

- numerical optimization- complex discretizations in simulation- irregular problems

• Programming is Difficult- Programmer must manage many details

management of parallelism, communication, and synchronization debugging and performance tuning porting to different parallel machines

- Automatic parallelization does not work!

CRPC

Relationship to Competitiveness

• Increasing the size of the parallel computing market- easy-to-use parallelism = inexpensive performance upgrades

scalable from PC to supercomputer

• Improving the practice of science and engineering- easy-to-use parallelism = better and faster

product designs health care applications environmental simulations services on the information highway etc.

CRPC

Research Organization

• Software- Fortran Parallel Programming System (Kennedy)- Parallel Paradigm Integration (Chandy)

• Parallel Algorithms- Physical Simulation (Meiron/Wheeler)- Linear Algebra (Dongarra)- Optimization and Automatic Differentiation (Dennis)

• Applications- Direct Participation in Applications (All)

15 Grand Challenge Projects Keck Center for Computational Biology Geophysical Parallel Computation (Wheeler)

- Coordination (Fox)

CRPC

Major Center Themes

• Extensions to Conventional Languages- High Performance Fortran and C++- Class libraries for scientific computation- Programming support tools and libraries

• Overcoming Impediments to Parallelization- Parallelization of irregular problems- Scalable parallel I/O- Parallel computation in engineering

physical simulation, multidisciplinary design optimization

• Knowledge Transfer- Libraries, templates, and archetypes- National HPCC Software Exchange

scalable distribution system based on World Wide Web and Netlib- Retooling supercomputer users for parallelism

• New Computing Paradigms- Parallel computation on heterogeneous networks- Web-based computing

CRPC

Research Accomplishments

• Software- Development and implementation of new parallel programming

languages (e.g., Fortran D, PVM, Fortran M, CC++)- Establishment of cross-industry language standards (High

Performance Fortran, Message-Passing Interface)

• Algorithms- Development of new algorithms for solving science and engineering

problems on parallel computers (e.g. Parallel Direct Search)- Delivery of these algorithms in software libraries and reference

implementations (e.g., ScaLAPACK, Templates, Archetypes)

• Applications- Numerous collaborations with application groups:

Aerodynamics, Astrophysics, Biology, Chemistry, Climate, Energy, Environment, Manufacturing, Telecommunications...

CRPC

Parallel Computation: What We Did Wrong

• Focused Too Much on Architectures- if we build it, they will come — NOT

• Failed to Identify Software as the Problem Early Enough- users will not move from easy programming to hard programming

without due cause- many standard algorithms do not parallelize well

• Underestimated the Software Problem- vectorization took 10 years- parallelization is much more complicated

• Failed to Ask Non-Research Users What They Must Have- industry and independent software vendors must have a

machine-independent, standard programming interface (e.g. PVM, MPIor HPF) to protect their programming investment

CRPC

Architecture-Independent Parallel Programming

• Definition- a programming language, its compilers, and run-time system support

architecture-independent parallel programming if compiled code ≅ hand code for same algorithm for each target architecture

• Strategies- Portable libraries

PVM, Linda- Software distributed shared memory (DSM)

simulate DSM using system paging mechanisms- Standard interfaces

MPI- Standard languages

HPF, HPC++

CRPC

Evolution of HPF

• Research on Compiling for Distributed Memory- distribution specified by programmer- Fortran D (CRPC)- Vienna Fortran (Vienna)- CM Fortran (Thinking Machines)

• Informal Standardization Effort- HPFF Forum founded Supercomputing 91

industry initiated, chaired by Kennedy- HPF 1 available July 1993- HPF 2 begun January 1995- HPF 2 available January 1997

• Implemented by Compiler Vendors

CRPC

HPF Goals

• Support for Scalable Parallel Systems- scaling from one to thousands of processors

• Focus on Data Parallelism- parallelism through subdivision of data domain

• Machine Independent Programming Support- object program achieves performance comparable to hand-coded

MPI on each target machine on the same algorithm

• High Level of Abstraction- implicit generation of communication

CRPC

HPF Strategy

Fortran 90

HPF

IBMSP-2CM-2

HP/ConvexSPP2000

Data DistributionDirectives

SequentialMachine

Free

CRPC

HPF Language

• Fortran 90 + Data Distribution

• Explicit Parallelism- Fortran 90 array statements

• Implicit Parallelism- "owner computes"

• Extended looping and aggregate assignment- FORALL (single-statement and block)- DO INDEPENDENT

!HPF$ TEMPLATE D(256,256)

!HPF$ ALIGN A(I,J) WITH D(I+1,J)

!HPF$ DISTRIBUTE D(BLOCK,CYCLIC)

virtual processor array

align data elements

map to processors

CRPC

HPF Sample Program

REAL A(1023,1023), B(1023,1023), APRIME(511,511)!HPF$ TEMPLATE T(1024,1024)!HPF$ ALIGN A(I,J) WITH T(I,J)!HPF$ ALIGN B(I,J) WITH T(I,J)!HPF$ ALIGN APRIME(I,J) WITH T(2*I-1,2*J-1)!HPF$ DISTRIBUTE T(BLOCK,BLOCK)

!HPF$ INDEPENDENT, NEW(I) DO J = 2, 1022 ! Multigrid Smoothing Pass (red-black relaxation)!HPF$ INDEPENDENT DO I = MOD(J,2), 1022, 2 A(I,J) = 0.25*(A(I+1,J) + A(I+1,J) + A(I,J-1) + A(I,J+1)) + B(I,J) END DO END DO

!HPF$ INDEPENDENT, NEW(I) DO J = 2, 510 ! Multigrid Restriction!HPF$ INDEPENDENT DO I = 2, 510 APRIME(I,J) = 0.05*(A(2*I-2,2*J-2) + 4*A(2*I-2,2*J-1) + & & A(2*I-2,2*J) + 4*A(2*I-1,2*J-2) + 4*A(2*I-1,2*J) + & A(2*I,2*J-2) + 4*A(2*I,2*J-1) + A(2*I,2*J)) END DO END DO

! Multigrid convergence test ERR = MAXVAL( ABS(A(:,:)-B(:,:)) )

CRPC

Status

• Bad News- Acceptance of language has been slow in coming

MPI has achieved penetration much faster compilers complex and slow to mature performance was initially disappointing additional features needed

• Good News- Compilers now available for every HPC platform- Applications beginning to emerge- Promising benchmark results

CRPC

HPF Commercial Interest

• Announced HPF Products

• Announced HPF Efforts

• Interested

ACELahey

NAGnCUBE

EPCSGI

Tera

Applied Parallel ResearchCDACCray ResearchDigital EquipmentFujitsuHitachiHPIBMIntel

MeikoMotorolaNA SoftwareNECPacific Sierra ResearchPortland GroupSunTranstech

CRPC

HPF Usage

• Installations- PGI reports over 100 site licenses

• Applications- 2 Grand Challenge projects using HPF- 10 Major Collaborations with PGI (8000-25,000 lines)- NCSA: PGI on SGI is migration platform for CM-5- European-funded applications (extension of Europort)- CRS4 Seismic Migration Code "GeoComp"- Amoco: Reservoir Modeling

• Benchmarks- NAS Benchmarks: PGI compiler within 20-50% of MPI on Class A

benchmarks- Cornell HPF Study: IBM HPF compiler outperforms hand-coded MPI on

a financial application

CRPC

Other Center Accomplishments

• Education- National Retooling Project for supercomputer users- Computational Science and Engineering degree programs and courses- Summer research programs for undergraduates

emphasis on minorities and women- Computational science awareness courses for K-12 teachers

• Outreach- Parallel Computing Research newsletter- Meetings and workshops on issues in parallelism- Advanced parallel computing facilities

• Technology and Knowledge Transfer- High Performance Fortran and MPI Forums- Parallel computing enabling technologies (PCE-Tech)

templates for algorithm distribution InfoMall: job creation through small business

- National HPCC Software Exchange (NHSE)- DoD Modernization Collaboration

programming environments and training (PET)

CRPC

National HPCC Software Exchange

• Library based on World-Wide Web and Netlib- http://www.netlib.org/nhse/home.html

• Road map- encyclopedia designed by experts- help in locating right software for the job

• Software review- unreviewed- registered

checked by librarians- reviewed

NHSE Review: an electronic review journal

• Support for limited distribution- cryptographic methods

CRPC

NHSE Components

• Distribution via the World Wide Web- Discipline-oriented repositories- Common infrastructure

cataloging, naming and authentication, etc. repository in a box

- HPCC-specific search

• Electronic Review Journal: NHSE Review

• Network-based Road Map

• Research and development- Advanced network-based software distribution mechanisms

• Outreach and technology transition- To the HPCC user community and industry

• Measurement

CRPC

NHSE Review: an Electronic Journal

Goal: to provide, in electronic form, a compendium ofcomparative articles covering subcollections of NHSE software.These comparative articles would be designed and written tohelp users find the right software for their needs.

• An Electronic Review Magazine- Articles by acknowledged experts- Each article covers a collection of related software systems- Model: review articles in PC and Mac magazines

• First Issue:- Review of Cluster Management Software- Overview of Recent Supercomputers- http://nhse.cs.rice.edu/NHSEreview/96-1.html

• Articles become an integral part of the Road Map

CRPC

Technical Details

• Review Articles Invited- noted experts in specific subareas- examples: Coddington, Dongarra, Fox, Pancake, Presberg- reviewed by three referees

• Archival but HTML Only- Specific, citable publication date- Links to software home pages and appropriate sections of Road Map- Links from author and NHSE portfolio pages to Review

• Moderated User and Author Feedback- Users who wish to report experience or contradict review assertions- Authors who wish to update information about software mentioned in

a review (or respond to review statements)- Follow-up surveys of users who have downloaded reviewed software

CRPC

National Retooling Project

• Packaged courses- advanced topics in parallel computation- for use by supercomputer center staff trainers

• Materials- all slides in html on WWW- on-line multimedia presentations and books

• Topics to date- problem classification and mapping to

technology- MPI- Fortran M- templates for sparse linear algebra- High Performance Fortran- parallel numerical optimization- metacomputing technologies

CRPC

Computational Science and Engineering Research

CRPC

State and Regional Centers

Users

Parallel computingcourses

Parallel ComputingTechnologies

Leverage is the Key

Retooling Strategy

NSF Metacenter

CRPC

Major Collaborations

• Scalable I/O Initiative- National consortium to attack the I/O problem

funded by multiple federal agencies origin in the Concurrent Supercomputer Consortium (CSCC) led by Caltech with significant CRPC contribution

- Language, compilers, and run-time libraries

• Defense Modernization Program- Collaborating on proposals for Major Shared Resource Centers

(MSRCs)- Successes on three of four announcements

CEWES (Vicksburg, MS) Wright Patterson ASC (Dayton, OH) Army Research Lab (Aberdeen, MD)

- Programming environment and training (PET)

CRPC

NSF Partnerships

• Supercomputer Center Recompetition- Hayes Report: replace centers with partnerships

lead center with high-end facilities mid-level (regional) centers research groups

• CRPC Partnerships- Illinois NCSA (Larry Smarr)

Kennedy, Reed, Stevens will coordinate CS research effort Rice will be regional center Tapia will lead program for under-represented groups Fox will lead graduate education program

- UC San Diego (Sid Karin) collaboration on Scalable I/O Messina will be Chief Architect, coordinate partners Texas will be a regional center Tapia will help design education programs

CRPC

InstitutionalOversightCommittee

TechnicalSteering

Committee

ExternalAdvisory

CommitteeDirectorKennedy

Executive DirectorTorczon

Assoc. Dir. External RelationsEl-Messidi

Assoc. Dir. BusinessPowell

Fortran SystemsKennedy

Paradigm IntegrationChandy

Linear AlgebraDongarra

OptimizationDennis

FacilitiesMessina

Differential EquationsMeiron/Wheeler

ApplicationsFox

ArgonneStevens

Los AlamosWhite

CaltechKeller

Education/MinoritiesTapia

Executive Committee

Management

CRPC

The Future

• Completion of current projects- Integration of Fortran D with Fortran M/CC++- Support for irregular problems- Libraries and templates- Engineering parallel computation- Scalable I/O

• Continuing areas of emphasis- Education, outreach, knowledge transfer- Participation in applications

• New themes- Parallel computing on heterogeneous networks- Problem-solving environments- Increasing emphasis on research capitalization