1

Dr. Frederica DaremaSenior Science and Technology Advisor

NSF

Future Parallel Computing Systems – what to remember from the past

RAMP WorkshopFCRC 2007

2

Overview

• Systems Modeling and Analysis is important– Design cycle and support the runtime– measurements an important (static and runtime)– performance, dependability, reliability, power

management, security, ….

• Multicore systems – New opportunities and something from the past– What we learned from shared memory systems of

the early-mid80’s

3

Authenication

/

Authorization

Fault Recovery

Services

Distributed Systems Management

Distributed, Heterogeneous, Dynamic, AdaptiveComputing Platforms and Networks

DeviceTechnology . . .

CPUTechnology

Visualization

Scalable I/OData Management

Archiving/Retrieval

Services

Collaboration Environments

Distributed Applications

MemoryTechnology

Two Components of CSR Program:

•* (AES) Advanced Execution Systems•* (SMA) System Modeling and Analysis

Prog.Models

Libraries

Tools

Compilers

Advanced Execution Systems

Parallel and Distributed Operating

Systems

Syst e

m M

od

el in

g a

nd

An

al y

sis

4

System Modeling and Analysis (SMA)

Seeks to develop methods and tools for modeling, measuring, analyzing, evaluating, and predicting the performance and correctness of complex

computing and communications systems SMA emphasizes the development of methods and tools for modeling,

measuring, analyzing, evaluating, and predicting the performance and correctness of complex computing and communications systems

Topics of Interest• Hardware and Software modeling

– methods tools and measurements, providing multimodal, hierarchical or multilevel modeling and analysis capabilities of such systems;

– methods that describe components of the system, but also the system as a total, and enable assessment of the effects of individual hardware and software layers and components of these systems;

– ability to describe the system in multiple levels of detail (characteristics and time-scales);

– combine different methods of describing components and layers

5

System Modeling and Analysis (SMA)

Topics of Interest (cont’d)• Novel modeling and measurement approaches

– Develop capabilities to describe, analyze and predict the behavior of the components as well as the systems; Analysis and prediction due to changes in the application, system software, hardware; multilevel approaches and multi-modal approaches

• Performance Frameworks – combine tools in “plug-and-play” fashion – multiple views of the system

Emphasis on Multidisciplinary Research (across sub-areas of CS)

6

Multiple views of the systemThe applications’ view

Authenication/ Authorization

DependabilityServices

Distributed Systems ManagementDistributed, Heterogeneous, Dynamic, Adaptive

Computing Platforms and Networks

DeviceTechnology . . .CPU

Technology

VisualizationScalable I/O

Data ManagementArchiving/Retrieval

ServicesOther Services . . .

Collaboration Environments

Distributed Applications

MemoryTechnology

Application

Models

OSScheduler

ModelsArchitecture /

Network Models

MemoryModels

IO / FileModels

. . . Languages

LibrariesTools

Compilers

7

Authenication/ Authorization

DependabilityServices

Distributed Systems Management

VisualizationScalable I/O

Data ManagementArchiving/Retrieval

ServicesOther Services . . .

Collaboration Environments

Distributed Applications

Distributed, Heterogeneous, Dynamic, AdaptiveComputing Platforms and Networks

DeviceTechnology . . .CPU

TechnologyMemory

Technology

Application Models

Architecture /Network Models

MemoryModels

OSScheduler

Models

IO / FileModels

. . . Languages

LibrariesTools

Compilers

Multiple views of the systemThe Operating Systems’ view

8

DynamicallyLink

&Execute

The NGS Program developsTechnology for integrated feedback & control Runtime Compiling System (RCS) and Dynamic Application

CompositionApplication

Model

Application Program

ApplicationIntermediate

Representation

CompilerFront-End

CompilerBack-End Performance

Measuremetns&

Models

DistributedProgramming

Model

ApplicationComponents

&Frameworks

Dynamic AnalysisSituation

LaunchApplication (s)

Distributed Platform

Ada

ptab

leco

mpu

ting

Syst

ems

Infr

astr

uctu

re

Distributed Computing Resources

MPP NOW

SAR

tac-com

database

firecntl

firecntl

alg accelerator

database

SP

….

9

Advanced Execution Systems (AES) (a component of the Computer Systems Research

Program)(CSR Program)

Seeks to create systems software to facilitate the development and runtime support of complex applications executing on large,

heterogeneous high-end computing and grid platforms AES emphasizes runtime compiling systems and application

composition systems interface with the underlying operating systems services and incorporating systems modeling and analysis methods

and tools. Topics of Interest• Novel Compiler Technology that go beyond the standard static notion of a

compiler – for example by embedding a portion of the compiler in the runtime and

endowing the system with resource awareness and adaptive mapping capabilities; 

– new compiler techniques for determining functional and data dependencies across multiple levels of memory hierarchy and across platforms;

– mechanisms for matching an application’s resource needs to underlying resources when both are changing as the application executes

10

Advanced Execution Systems (AES)

Topics of Interest• Programming models and tools

– expressing application partitioning across distributed, heterogeneous computing platforms; application-level checkpointing and recovery

• Application composition system (ACS) technology– constructing applications to fit the available resources and to

adapt to changes in the underlying execution environment; – methods for automatically selecting application components; – creating knowledge bases for application components;

interfacing with the underlying computing platform models to determine suitable application components;

– and developing appropriate application component libraries and interfaces so the run-time portion of the RCS can link to such libraries.

A number of projects are developing: systems modeling methods and performance frameworks

&runtime-compiler and application composition systems

Projects can be found in the proceedings of the

Next Generation Software(Workshop Series & PI-Meetings)

I organize every year in conjunction with IPDPS

The RAMP effortis very important

in enabling the modeling capabilities needed for present and future

computer systems and environments

Can lead to a valuable standard Performance

Framework usable by the broader

academic community and industry

13

With respect to the Emerging Multicore Systems:

Things to remember from the past• Coherent caches – issues of scalability to large numbers of processors• On-chip networks:

– Bus-based communication – contention - & scalability issues– Cross-bars - can have contention for 100s procs (let alone 100Ks of

procs ) – Consider multistage switches– Consider combining networks

• Test&Set/Compare& Swap - Fetch&op (e.g Fetch&Add) for synchronization

• Shared data – Synchronization (place in a “fast” shared memory - cache)– Other application shared data (place in shared memory – stage in

cache) – Past experiments showed that even if 99% of data are shared –

accesses to the shared data were ~20%• Memory wall - remains challenge; reducing latency remains challenge

– Hardware – pipelining– Software – compilers/prefetching

• Temporary cacheable shared data (compiler enabled)• Performance Tools - Parallel tracing – parallel simulations