CyberShake Study 15.4Technical Readiness Review

Study 15.4 Scientific Goals

• Calculate a 1 Hz map of Southern California

• Produce meaningful 2 second results for the UGMS• RotD50 and RotD100 at 2, 3, 4, 5, 7.5, 10 seconds• Contour maps

• Compare 0.5 Hz and 1 Hz hazard maps

• Use Graves & Pitarka (2014) rupture generator with regular spaced hypocenters

• 336 sites (10 km mesh, points of interest, “gap” sites)• Run 14 UGMS sites first

• Produce 1 Hz seismograms which could be combined with BBP high-frequency seismograms

Study 15.4 Technical Goals

• Run CyberShake across Blue Waters and Titan• SGT and post-processing workflows on Blue Waters• SGTs only on Titan

• All SGTs to be calculated on GPUs

• Will measure CyberShake application makespan• Equivalent to the makespan of all of the workflows

• (All jobs complete) – (first workflow submitted)• Includes hazard curve calculation time• Includes system downtime, workflow stoppages

• Compare performance of Blue Waters and Titan

• Compare 1 Hz performance to previous studies

Performance Enhancements

• Pegasus cleanup used to decrease temp storage• Should avoid Study 14.2 problem of running out of

scratch space

• Parallel version of AWP reformat code• 65% reduction in runtime

• DirectSynth for post-processing• Single job for entire post-processing• Reads in SGTs across nodes, communicates via MPI• Reduces 1 Hz post-processing CPU-hours by 75%

• MD5 sums in post-processing workflow moved out of critical path

Proposed Study sites (336)

Green sites are the 50 new “gap” sites

Study 15.4 Data Products

• CVM-S4.26 Los Angeles-area hazard maps• RotD100 2, 3, 4, 5, 7.5, 10 sec• RotD50 2, 3, 4, 5, 7.5, 10 sec• Geometric mean 2, 3, 5, 10 sec

• Hazard curves for 336 sites, at 2, 3, 5, 10s

• 2-component seismograms for all ruptures (~160M)

• Peak amplitudes in DB• Geometric mean at 2, 3, 5, 10 sec• RotD100, RotD50 at 2, 3, 4, 5, 7.5, 10 sec

Study 15.4 Notables

• First 1 Hz hazard maps

• First study with RotD50 and RotD100 calculated

• First study to use OLCF Titan

• First study with Graves & Pitarka (2014) rupture generator with uniformly spaced hypocenters

• First study with 200 m rupture grid point spacing

• First study with source filtered at a different frequency than the simulation frequency

Study 15.4 Parameters

• 1.0 Hz deterministic• 100 m spacing• dt=0.005 sec• nt=40000 timesteps

• CVM-S 4.26• Vs min = 500 m/s

• UCERF 2

• Graves & Pitarka (2014) rupture variations• 200 m rupture grid point spacing

• Source filtered at 2.0 Hz

Changes to SGT Software Stack

• UCVM 14.3

• SGTs• Only using AWP-ODC-SGT GPU version, on 800 nodes

per component

• PostAWP• Changed to parallel version of AWP reformatting, 65%

speedup• Reduced read sizes to avoid issue with Titan filesystem• Separated MD5 sum calculation into separate job

• Handoff• Modified handoff job from Study 13.4 to provide interface

between Titan SGTs and Blue Waters post-processing

Changes to PP Software Stack

• Rupture generator V3.3.1• Extraction & Synthesis

• Created new job, DirectSynth• Single job for all seismogram synthesis, PSA, and

RotD calculation• Set of SGT handler processes read in SGTs• Set of workers work on synthesis, request SGTs

from SGT handlers• Data products are sent to master, which writes to

filesystem• Will use 1024 SGT handlers, 2560 workers per job

• Additional database jobs added for RotD data• Checks, insertions, curve calculations

Changes to Workflows• Post-processing workflows simpler

• Only 1 job for extraction and synthesis• MD5 sum out of critical path, but will abort workflow if fails

• Auto-submit cron tool on shock enhanced• List of sites to execute passed to cron job

• Will start with 14 UGMS sites• Will repeat first 2 sites (CCP, COO) on Titan and Blue Waters for

verification• Maintains constant # of workflows• When more are needed, selects next site(s),

creates/plans/runs the workflow• Now supports SGT, PP, and full workflows• Dynamically assigns workflows to remote resources

Workflow HierarchyIntegrated Workflow (one per site per model)

PreCVM(creates volume)

Generate AWP Workflow

AWP Workflow

PP Pre Workflow

PP Main Workflow DB workflow

Post-processing workflow (Blue Waters & shock)

SGT workflow (Blue Waters or Titan)

Handoff(Titan/BW interface)

336 workflows for Study 15.4

AutoSubmit cron job on shock.usc.edu

Distributed Processing

• Pegasus 4.5.0 RC, HTCondor 8.2.8, Globus 5.2.5

• Cron job on shock.usc.edu creates/plans/runs SGT, PP, and full workflows

• Jobs submitted to Blue Waters via GRAM

• Results staged back to shock, DB populated, curves generated

• Jobs submitted to Titan using pilot jobs• Cannot submit jobs to Titan directly, due to security

Titan Distributed ProcessingTitanshock.usc.edu

monitor_daemon.pyCondor queue

1. Every 5 minutes, monitor daemon queries Condor queue on shock.2. If there are more Titan SGT workflows in the queue than sets of jobs in the Titan

queue, a new set of pilot jobs are submitted, with qsub dependencies.3. These jobs start up Condor processes which call back to the shock Condor collector

and can be assigned work.

(1)

SGT workflow, Titan

SGT workflow, BW

SGT workflow, Titan

PP workflow, BWBatch queue

(2)

Pre SGT pilot jobSGT pilot jobPost SGT pilot jobPre SGT pilot jobSGT pilot jobPost SGT pilot jobCondor

collector

(3) condor_master

Computational Requirements

• Per site: ~3720 node-hrs • SGTs: depends on execution site (~50%)

• Titan = 2110 node-hrs / 63,300 SUs• Blue Waters = 1760 node-hrs / 30,200 SUs• More expensive for Titan because of padding in pilot jobs and

different node-hrs -> SU conversion• PP: 1880 node-hrs / 60,200 SUs (~50%)

• Computational time:• Titan (SGTs): 355K node-hours / 10.7M SUs• Blue Waters: 928K node-hours

• SGTs: 275K GPU node-hrs, 21K CPU node-hrs• PP: 632K CPU node-hrs

• Titan has 104M SUs remaining

• Blue Waters has 5.3M node-hrs remaining

Storage Requirements

• Titan• Purged: 526 TB (for SGTs and temp data)

• Blue Waters• Delayed purge: 506 TB (for Titan SGTs)• Purged: 526 TB SGTs + 9 TB data products

• SCEC• Archived: 9.1 TB (seismograms, PSA, RotD)• Database: 268 GB (Geom @ 4 periods, RotD @ 6)• Temporary: 608 GB (workflow logs)• Shared SCEC disks have 171 TB free

Metrics Gathering

• Monitord for workflow metrics• Will run during workflows, since DirectSynth

dramatically cuts the number of tasks

• Python scripts to calculate standard metrics

• Cronjob on Blue Waters• Core usage over time• Jobs running and idle

• Pilot monitor process on Titan• Core usage• Jobs running and idle

• Will use start and end of workflow logs to perform makespan measurement

Monitoring Tools

• Will use Study Manager to track progress• Hosted on northridge.usc.edu• Tracks number of runs in each state• Estimates completion time based on velocity

• Run Manager tracks status of individual runs

• If errors, will dig into individual logs

Estimated Duration

• Limiting factors:• XK node queue time

• 800 XK nodes is 19% of Blue Waters• Titan -> Blue Waters

• If throughput is very high, transfer could be bottleneck• USC HPC downtime for ~1 week in April

• Estimated completion is 12 weeks (11 running + 1 downtime)• Based on same node availability as Study 14.2

• Planning to request reservation on Blue Waters

• Planning to request high priority on Titan

Personnel Support• Scientists

• Tom Jordan, Kim Olsen, Rob Graves

• Technical Lead• Scott Callaghan

• Job Submission / Run Monitoring• Scott Callaghan, David Gill, Phil Maechling

• NCSA Support• Omar Padron, Tim Bouvet

• Titan Support• Val Anantharaj

• USC Support• John Yu, John Mehringer

• Workflow Support• Karan Vahi, Gideon Juve

Risks• Queue times on Blue Waters for XK nodes

• Will try to dynamically assign SGT jobs to resources

• Unforeseen complications with Titan pilot jobs

• Globus toolkit upgrades on NCSA• Globus upgraded on shock and tested• Waiting on new Condor release for shock

• Congestion protection events (network overloaded)• If triggered consistently, will need to limit number of

post-processing workflows

• Scott goes on leave before study is complete• Difficult to run study under other accounts at this time

Action Items

• Upgrade UCVM on Blue Waters to 14.3.0

• Notify OLCF, NCSA, and USC of study

• Request reservation on Blue Waters

• Request increased priority on Titan

• Develop an approach for continuity of study during Scott’s leave

• Test single workflow, multiple sites approach

Thanks for your time!