Dr. Francine Bermanlibraries.ucsd.edu/chronopolis/_files/presentations/ACM-SDSC.pdf• Uses adaptive mesh refinement (AMR) to provide high spatial resolution in 3D – The Santa Fe

UCSD

SAN DIEGO SUPERCOMPUTER CENTER

Fran Berman

Dr. Francine Berman

Director, San Diego Supercomputer Center

Professor and High Performance Compu8ng Endowed Chair, Department of Computer Science and Engineering, UC San Diego

UCSD


Fran Berman

Why Cyberinfrastructure-enabled Research Matters: 21st Century Problems Require 21st Century Solutions

"Let us be the generation that reshapes our economy to compete in the digital age. Let's set high standards for our

schools and give them the resources they need to succeed. … let's invest in

scientific research, and let's lay down broadband lines through the heart of inner cities and rural towns all across America."

Barack Obama

Will current levees hold in a large‐scale earthquake?

What is the impact of Global

Warming?

What therapies can be used to cure or control

cancer?

Can financial data be used to accurately predict market outcomes?

What plants work best for biofuels?

UCSD


Fran Berman

Cyberinfrastructure is the foundation for modern research and education

Ifinfrastructureisrequiredforanindustrialeconomy,thenwecouldsaythat

cyberinfrastructureisrequiredforaknowledgeeconomy.”

NSFFinalReportoftheBlueRibbonAdvisoryPanelon

Cyberinfrastructure

Cyberinfrastructure: theorganizedaggregateofinformaFontechnologiescoordinatedtoaddressproblemsinscienceand

society.

•  Cyberinfrastructure components:

–  Digitaldata

–  Computers

–  Wirelessandwirelinenetworks

–  Personaldigitaldevices

–  ScienFficinstruments

–  Storage

–  SoJware

–  Sensors

–  People…

UCSD


Fran Berman

SDSC’s Challenge: Accelerate Research and Education using Cutting-Edge Tools as the IT

landscape changes

Late ‘80’s ‐ early 90’s

AddressGrandChallengeProblemsusingSupercomputers

Emerging technology: supercomputers

EvolvingITlandscape:

Mid‐90’s – mid 00’s

IntegratediverseinformaFontechnologiestosolvemodernchallenges

Emerging technology: Grids

Mid 00’s – present

Linkresearchersto“unlimitedresources”inawaythatfacilitatestheirusetocreateusefulinformaFonand

newknowledge

Emerging technology: Clouds

UCSD


Fran Berman

Today’s SDSC

Resources •  SDSC has ~250 research,

technology, and IT staff with expertise in data use, management, and preservation, high performance computing, software tools, domain science, and education

•  SDSC’s data center is one of the largest academic data centers in the world with 36+ PBs capacity

•  SDSC is home to the San Diego Network Access Point (SDNAP)

•  SDSC’s computers are architected to support data-intensive applications

Research and Education

•  In 2007, SDSC was home to over 110 research projects involving researchers at UCSD and throughout academia.

•  SDSC hosts hosted over 100 separate community digital data sets and collections for sponsors such as NSF, NIH, and the Library of Congress.

•  SDSC staff and collaborators published scholarly articles in a spectrum of journals including Cell, Science, Nature, Journal of Seismology, Journal of the American Chemical Society, Journal of Medicinal Chemistry, Nano Letters, PLoS Computational Biology, and many others

•  SDSC provided training and oversight for practica for over 200 students in 2007

UCSD


Fran Berman

Today’s Presentation

•  Cyberinfrastructure-enabled Applications at SDSC –  Personalized Medicine –  Simulating the Universe 1 Billion Years after the Big Bang

•  SDSC Cyberinfrastructure Resources and Services –  Triton

–  Chronopolis

•  Cyberinfrastructure for the Next Decade

UCSD


Fran Berman

Promoting Good Health

•  How can we use information technologies to promote good health?

–  Better monitoring

–  Better diagnosis

–  Better cures

–  Better therapies

–  Better health infrastructure

UCSD


Fran Berman

Personalized Medicine

•  Current “$1,000” gene testing services examine only < 0.1% of the complete human genome

•  SDSC’s Allan Snavely and his group working with medical researchers to develop high performance computational techniques to efficiently analyze the entire genome

Current genotyping services only look

for localized variations (“SNPs”)

in a very small portion of the

genome

True personalized medicine will require broad

examination of the full genome of tens

of thousands of people

Human genome consists of 3,200,000,000 “base pairs” of data requiring massive computation for full analysis

UCSD


Fran Berman

Using Dynamic Programming to Compare Genetic Sequences

•  Algorithms such as Smith-Waterman, used for genetic sequence comparison, are instances of dynamic programming

•  Large search space structured into a succession of stages, such that

1.  the initial stage contains trivial solutions to sub-problems,

2.  each partial solution in a later stage can be calculated by recurring a fixed number of partial solutions in an earlier stage and

3.  the final stage contains the overall solution.

•  Data stored in 3B+ X 3B+ base-pair matrix

UCSD


Fran Berman

Genomic Analysis at Scale Snavely’s Group employing a mulG‐pronged

approach

1.  Create new algorithms that can efficiently calculate structural similariGes (the“architecturaldifference”)betweentwosequencesinparallelonFghtly‐coupledHPCarchitectures

2.  Create new algorithms that tolerate high latency, high processor count cloud environments

3.  Create new approaches for querying and storing very large databases

Berm

an

form

ulaG

on

Jin

form

ulaG

on

UCSD


Fran Berman

Next Generation Cosmology

UCSD


Fran Berman

Evolving the Universe from the “Big Bang”

now then dump1 dump2 dump3 dump4 dump5

Composing simulation outputs from different timeframes builds up light-cone volume

Slide modified from Mike Norman

UCSD


Fran Berman

now then dump1


The Universe’s First Billion Years after the “Big Bang”

•  ENZO simulates the first billion years of cosmic evolution after the “Big Bang”

•  Key period which represents

–  A tumultuous period of intense star formation throughout the universe

–  Synthesis of the first heavy elements in massive stars

–  Supernovae, gamma-ray bursts, seed black holes, and the corresponding growth of supermassive black holes and the birth of quasars

–  Assembly of first galaxies

UCSD


Fran Berman

ENZO Simulations What ENZO does:

•  Calculates the growth of cosmic structure from seed perturbations to form stars, galaxies, and galaxy clusters, including simulation of

–  Dark matter –  Ordinary matter (atoms) –  Self-gravity –  Cosmic expansion

•  Uses adaptive mesh refinement (AMR) to provide high spatial resolution in 3D

–  The Santa Fe light cone simulation generated over 350,000 grids at 7 levels of refinement

–  Effective resolution = 65,5363

Formation of a galaxy cluster


Level 0 Level 1 Level 2

AMR

UCSD


Fran Berman

Enzo Data Volumes •  20483 simulation (2008)

–  8 gigazones X –  16 fields/zone X –  4 bytes/field –  = 0.5 TB/output X –  100 outputs/run –  = 50 TB

•  40963 simulation (2009) –  64 gigazones X –  16 fields/zone X –  4 bytes/field –  = 4 TB/output X –  50 outputs/run –  = 200 TB

UCSD


Fran Berman

Greater Simulation Accuracy Requires More Computing and Generates More Data

ENZO at Petascale (10^15) ●  Self-consistent radiation-hydro

simulations of structural, chemical, and radiative evolution of the universe simulates from first stars to first galaxies

Computer Science challenges ●  Parallelizing the grid hierarchy metadata for millions of subgrids distributed

across 10s of thousands of cores ●  Efficient dynamic load balancing of the numerical computations, taking

memory hierarchy and latencies into account ●  Efficient parallel “packed AMR” I/O for 100 TB data dumps ●  Inline data analysis/viz. to reduce I/O

Slide modified from Mike Norman, image by Robert Harkness

UCSD


Fran Berman

Verifying Theory with Observation

•  James Webb Space Telescope, coming in 2013 will probe the first billion years of the universe – providing observations of unprecedented depth and breadth

•  Data will enable tight integration of observation and theory, and will enable simulations to approach realistic complexity

•  Analysis of petascale data sets will be essential for validating model

Slide modified from Mike Norman, image by Robert Harkness

UCSD


Fran Berman

SDSC Cyberinfrastructure Resources and Services

UCSD


Fran Berman

SDSC’s new Triton Resource provides unique resources to drive research

and serve as gateways to the larger resource

environment

ComputeCapacityEnvts.

Large‐ScaleStorage

Integratedcapability

environments

COMPUTE (more FLOPS)

DAT

A (m

ore

BY

TES

) Home, Desktop

Applications

DATA OASIS PSDAF

Shared Resource Cluster

DigitalLibraries

Na8onal facili8es

Clouds

Super‐computers

Community Repositories

Data‐intensive applica8ons

Data‐ and Compute‐

intensive HPC Applica8ons

Compute‐Intensive HPC Applica8ons

The focus of SDSC’s resources and services is to

provide an integrated environment that

empowers users to move beyond their local boundaries

to further research and education goals

Enabling Ideas

UCSD


Fran Berman

Triton Resource Draft Configuration Will go into Production by Summer 2009

UCSD High Bandwidth Research Network

LargeMemoryPDAF

• 256/512GB/Node• 9TBTotal• 112GB/sec• 9TF

x32 Triton Compute Cluster Resource Cluster

• 24GB/Node• 6TBTotal• 256GB/sec• 20TF

Data Oasis LargeScaleStorage• 2–4PB• 75–150GB/sec• 3000–6000disks

UCSD/UC Research Labs

UCSD


Fran Berman

What will we do with the Data Oasis? –  Faculty Terabytes – Archival disk storage for UCSD faculty

data –  Storage Services: Backups, DB hosting, Project leverage –  Research Data Services: Data visualization, database

development, portals, GIS, data mining, statistical analysis, etc.

–  Chronopolis preservation services

The Triton Resource: Data Oasis

•  Data Oasis will be used as storage for

–  PDAF, Triton Compute Cluster, UCSD black boxes, in situ instruments and clusters, UC colleagues, key projects

•  Administered by SDSC and the UCSD Libraries

Shared Research Cluster

PDAF

•  Integrated •  Expandable •  Efficient

Data Oasis High Performance Disk Storage

ComputeCapacityEnvironme

nts


Integratedcapabilityenvironme

nts


DAT

A (m

ore

BY

TES

)

Home, Desktop Applications

DATA OASIS PDAF

Triton Compute Cluster

UCSD


Fran Berman

Triton’s Expandable Triton Compute Cluster

•  Triton Triton Compute Cluster is a launch system that provides a seed for expandable “condo-style” core computational facilities

•  Full suite of centralized support services (24X7 operations, security, networking, SW, storage, support, training, documentation, system administration)

ComputeCapacity

Environments


Integratedcapabilityenvironmen

ts


DAT

A (m

ore

BY

TES

)

Home, Desktop

Applications

DATA OASIS PDAF


“Condo” facilities are green and expandable

•  Economies of scale for capital purchase and operations

•  Whole is greater than the sum of small parts – can aggregate across machine for “capability” runs during idle periods

UCSD


Fran Berman

The Triton Resource: PDAF

PDAF overview: • Large-Memory Data-intensive cluster • 8 x 512GB + 20 x 256 GB nodes

•  8 AMD 8380 Shanghai Processors at 2.5GHz. 4 X 10Gbit Myrinet.

•  Sun x4600M2

•  4 Nodes will have large local storage for DBs (2 @ 7.6TB, 2@ 24TB)

•  9 TB Total • Linux based

• Energy efficient

• High speed network to Data Oasis

•  PetascaleDataAnalysisFacilitysupportsSDSCResearchfocus

•  CampusplanninggroupofusersdeterminedpracFcalbalancing(compute,memory,storage,networkconnecFvity)ofTriton

•  Configuredforanalysis,modeling,simulaFonofverylargedatasets

ComputeCapacityEnvironme

nts


Integratedcapabilityenvironme

nts


DAT

A (m

ore

BY

TES

)

Home, Desktop

Applications

DATA OASIS PDAF


UCSD


Fran Berman

SDSC Overall “Architecture”

Projects Chrono‐polis

BIRN GEONGreen‐Light

CAMERAXD

PlanningHPWREN Road‐Net

OceanObservingIniFaFve

CAIDATera‐Grid

Swami Etc.

Exper8se, Services

Long‐termdata

preser‐vaFon

Co‐locaFonServices

Storageservices

Datauseservices

R&Dservices(workflow,data

mining,workbenches,portals,etc.)

ConsulFngservices

HPC,data,cloud,cyberinfrastructure,domainexperFse

Hardware Triton Resource

DataOasis,PDAF,CondoclusterUC

SharedResourcesProject

ResourcesTeraGrid cluster

Stay tuned

Facili8es UC/UCSD /Project– supported FaciliGes

(efficientpower/cooling,24*7monitoring,networking,faciliFesmgt,resourcehosFng,physicalsecurity,etc.)

UCSD


Fran Berman

Data Cyberinfrastructure for Long-term Preservation

Projects Chrono‐polis

BIRN GEONGreen‐Light

CAMERAXD

PlanningHPWREN Road‐Net

OceanObservingIniFaFve

CAIDATera‐Grid

Swami Etc.

Exper8se, Services

Long‐termdata

preser‐vaFon

Co‐locaFonServices

Storageservices

Datauseservices

R&Dservices(workflow,data

mining,workbenches,portals,etc.)

ConsulFngservices

HPC,data,cloud,cyberinfrastructure,domainexperFse

Hardware Triton Resource

DataOasis,PDAF,CondoclusterUC

SharedResourcesProject

ResourcesTeraGrid cluster

Stay tuned

Facili8es UC/UCSD /Project– supported FaciliGes

(efficientpower/cooling,24*7monitoring,networking,faciliFesmgt,resourcehosFng,physicalsecurity,etc.)

UCSD


Fran Berman

What is Chronopolis?

Chronopolis Long-term Preservation Service and Resources

Data Grid supporGng a Long‐term PreservaGon Service

Data Providers (User‐focusedrepositories,libraries,etc.)

Data Users

Who is Chronopolis? Chronopolis is being developed by a

national consortium led by SDSC and the UCSD Libraries (UCSDL) and funded by the Library of Congress.

Initial Chronopolis nodes include:

•  SDSC/UCSDL at UCSD

•  University of Maryland Institute for Advanced Computer Studies (UMIACS)

•  National Center for Atmospheric Research (NCAR) in Boulder, CO

UCSD Libraries

UCSD


Fran Berman

Current Chronopolis Collections March2009

Collections from Data Providers:

•  Inter-university Consortium of Political and Social Research – preservation copy of all collections including 40 years of social science data and Census (8 TB)

•  California Digital Library – political and government web crawls, Web-at-risk collection (5 TB)

•  SIO Explorer – data from 50 years of research voyages (1 TB)

•  NCSU Libraries -- State and local geospatial data (6 TB)

http://chronopolis.sdsc.edu

UCSD


Fran Berman

Inside Chronopolis

•  Sites linked by main staging grid where data is verified for integrity, and quarantined for security purposes.

•  Collections independently pulled into each system.

•  Manifest layer provides added security for database management and data integrity validation.

•  Benefits –  Each collection copy

independently managed –  Collections available from

each site? –  High reliability

Chronopolis Data Grid supporGng Long‐term PreservaGon Service

NCAR

SDSC Core Center

Archive

Pull Pull

Pull

Push

UMD

Copy 1

Copy 2 Copy 3

Tape

Grid Brick Disks

Data Providers (CDL,NCSU,ICPSR,SIO,…)

SDSC Staging

Grid

MCAT

Grid Brick Disks

MCAT

MCAT

Manifest Management MCAT DB Multiple Hash Verifications

UCSD


Fran Berman

SDSC and the UCSD Libraries – a Unique Organizational Relationship

•  SDSC and the UCSD Libraries working together to support the entire data life cycle – from curation and ingest to storage and preservation

•  Joint SDSC/UCSDL team has worked with National Science Foundation, NARA, and the U.S. Library of Congress to pioneer integrated approaches to digital data cyberinfrastructure

In addition to Chronopolis, SDSC/UCSDL worked with the Library of Congress on a pilot to develop distributed preservation solutions for valued collections like the Prokudin-Gorskii photographs of the Russian Revolution

UCSD


Fran Berman

Cyberinfrastructure for the Next Decade

UCSD


Fran Berman

Significant Trends in Cyberinfrastructure

•  Datamanagement

•  Datamininganddatause

•  DatavisualizaFon

•  ProgrammingEnvironments

•  VirtualizaFon

•  ApplicaFons

•  Algorithms

Green IT

MulGcore and Extreme HPC

Long‐term PreservaGon

Data Mining

Small‐scale devices, Sensors

•  Performance

•  Power

•  Efficiency

UCSD


Fran Berman

Cyberinfrastructure will need to support

both a broader set of use cases and a

greater set of limiting factors

Power Economics

Regulation, Policy

Low-cost physical environments for

Cyberinfrastructure have become increasingly important as systems

scale

The Next Decade will see Increased Constraints

UCSD


Fran Berman

The Next Decade will see Increased Complexity

Scale Globalization

Interoperability More advanced applications and more capable systems

will require an unprecedented degree of

integration, interoperability, coordination, and protections

UCSD


Fran Berman

The Next Decade will see Increased Opportunity

Cyber-physical systems

Data Information Knowledge

Social networks, Mass communication

Boundaries between domains are blurring,

providing greater opportunities for innovation,

agility, and synergy, and presenting new challenges

for Cyberinfrastructure

UCSD


Fran Berman

SDSC Focus will be to Continue Working with Researchers at the Cutting Edge

Next Generation Biology Workbench Parkinson’s

Research

UCSD


Fran Berman

Thank you

www.sdsc.edu

Documents

Dr. Francine Bermanlibraries.ucsd.edu/chronopolis/_files/presentations/ACM-SDSC.pdf• Uses adaptive mesh refinement (AMR) to provide high spatial resolution in 3D – The Santa Fe