INCITE 2015 Project Name: High Frequency Physics-Based Earthquake System Simulations (Year 1 of 2)

PI: Thomas H. Jordan

Co-PI(s): Jacobo Bielak, Carnegie Mellon University, Po Chen, University of Wyoming,

Yifeng Cui, San Diego Supercomputer Center, Philip Maechling, Southern California Earthquake Center,

Kim Olsen, San Diego State University, Ricardo Taborda, University of Memphis

ALCF Project Name: GMSeismicSimOLCF Project Name: GEO112

Performance Period: January 1 - June 2015 (Q1 and Q2)

Quarterly Update: Q2 - 2015

Report on Project Milestones Provide status on each of your project’s simulations milestones as outlined in your

original proposal

Year 1 Milestone Descriptions MilestoneAchievement Status

M1 Use full 3D tomography and comparative validations using to improve existing California velocity models for use in high frequency wave propagation simulations at 0.2Hz

Achieved. Used Mira to calculate five iterations of Central California Model. The improved 3D model now available to ground motion modelers through UCVM.

M2 Run high frequency forward simulations using alternative material attenuation (Q) and seismic velocity models (CVMs). Compare the impact of material properties, topography, and models including spatial variability (heterogeneities) and soft-soil deposits (or geotechnical layers) on 4Hz+ simulations by simulating forward events using alternative models and comparing results among synthetics and with data.

Started, not completed. AWP-ODC and Hercules code branches have tested these physics. Currently running baseline 4Hz simulations without these physics with good agreement at 4Hz among 3 wave propagation codes with a simple velocity structure.

M3 Run high frequency forward simulations using alternative approaches to include the effects of off-fault and near-surface plastic deformation. Compare the impact of alternative plasticity

Started, not completed. AWP-ODC and Hercules code branches have tested these physics. Currently running baseline 4Hz simulations

models (linear-equivalent, 3D+1D hybrid, full 3D plastic) on 4Hz+ simulations by simulating forward events and comparing the results among synthetics and with empirical relationships and data.

without these physics with good agreement at 4Hz among 3 wave propagation codes with a simple velocity structure.

M4 Calculate a 1.0Hz CyberShake Hazard curve. Use updated CVMs, source models, and codes to calculate a higher frequency CyberShake hazard curve

Achieved. Used Titan and Blue Waters to calculate a CyberShake 1Hz Los Angeles area probabilistic seismic hazard model based on 336 site-specific hazard curves.

Year 2 Milestone Descriptions ObjectiveM5 Use full 3D tomography and comparative

validations using to improve existing California velocity models for use in high frequency wave propagation simulations at 0.5Hz

Not started

M6 Run high frequency forward simulations using alternative material attenuation (Q) and seismic velocity models (CVMs). Compare the impact of material properties, topography, and models including spatial variability (heterogeneities) and soft-soil deposits (or geotechnical layers) on 8Hz+ simulations by simulating forward events using alternative velocity models and comparing the results.

Not started

M7 Run high frequency forward simulations using alternative approaches to include the effects of off-fault and near-surface plastic deformation. Compare the impact of alternative plasticity models (linear-equivalent, 3D+1D hybrid, full 3D plastic) on 8Hz+ simulations by simulating forward events and comparing the results among synthetics and with empirical relationships and data.

Not started

M8 Calculate a 1.5Hz CyberShake Hazard curve. Use updated CVMs, source models, and codes to calculate a higher frequency CyberShake hazard curve

Not started

List major accomplishments thus far in CY2015. Please include scientific and computational details of simulations undertaken, including images if possible.

Our 2015-2016 INCITE proposal defined the following four objectives:

O1: Improve the resolution of dynamic rupture simulations by an order of magnitude and investigate the effects of realistic friction laws, geologic heterogeneity, and near-fault stress states on seismic radiation.

O2: Extend deterministic simulations of strong ground motions to 10 Hz for investigating the upper frequency limit of deterministic ground-motion prediction.

O3: Compute physics-based Probabilistic Seismic Hazard Attenuation (PSHA) maps and validate those using seismic and paleo-seismic data.

O4: Improve 3D earth structure models through full 3D tomography using observed seismicity and ambient noise.

During the first six months of our current allocation, SCEC researchers have worked towards these objectives in three main ways: (a) Define and improve a new central California 3D velocity model using full 3D tomography, (b) Produce a comprehensive, physics-based hazard model for the Los Angeles region valid up to seismic frequencies of 1 Hz, and (c) Extend realistic earthquake simulations above the 1-Hz frequency barrier by incorporating new aspects of earthquake physics. Our INCITE project activities in Q1 and Q2 mark substantial progress towards these goals including the following significant accomplishments:

a) We defined a central California 3D velocity model, that we call Central California Area (CCA), and used full 3D tomography computational methods to validate and improve the 3D seismic velocity model using both observed moderate earthquakes and ambient seismic noise observations.

b) We completed a 1-Hz urban seismic hazard model for the Los Angeles region (Figure 1). The new model, which comprises more than 300 million synthetic seismograms sampling the Uniform California Earthquake Rupture Forecast, was computed from a new high-resolution image of crustal structure derived using full-3D tomography (CVM-S4.26). It will be registered into the USGS Urban Seismic Hazard Mapping Project, and the results will be submitted for use in the 2020 update of the Recommended Seismic Provisions of the National Earthquake Hazards Reduction Program.

c) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan supercomputer using GPU-optimized finite-difference and finite-element codes that include frequency-dependent attenuation, small-scale near-surface heterogeneities, tomography, and a nonlinear dissipation in the near-fault and near-surface regions. These simulations set the stage for the ground motion prediction modeling at frequencies beyond 1 Hz.

Accomplishments led by Dr. Yifeng Cui in the development of GPU-enabled wave-propagation codes were recognized with NVIDIA’s 2015 Global Impact Award.

Major Project Accomplishments:

a) We defined a central California 3D velocity model, that we call Central California Area (CCA), and used full 3D tomography computational methods to validate and improve the 3D seismic velocity model using both observed moderate earthquakes and ambient seismic noise observations.

SCEC research team used ALCF Mira to produce a 3D seismic velocity model for Central California that we call Central California Area (CCA). We have used Mira to perform the computational and data intensive stages of our full 3D tomographic (F3DT) computational method. The purpose of the F3DT for central California is to improve the crustal 3D seismic velocity model for central California. An accurate 3D velocity model is essential input needed for accurate deterministic earthquake wave propagation simulations.

We have further improved the computational efficiency of our F3DT workflow on ALCF Mira and are now applying F3DT to Central California and statewide. As of 1 July, 2015, we have carried out 5 F3DT iterations for Central California and 3 F3DT iterations for the statewide California. Our improved Central California velocity model provides substantially better fit to over 12,000 seismic waveforms at frequencies up to 0.2 Hz and shows interesting small-scale structures in the upper to mid crust that agree with local geology and other independent geophysical evidence. Our latest statewide velocity model significantly improves the fit to over 27,000 waveforms at frequencies up to 0.1 Hz, and it has revealed new structural features in the mid to lower crust that are consistent with our understanding of the geotectonic development in California. More F3DT iterations will be carried out for both Central California and statewide. Gradual improvements in our velocity models have allowed us to incorporate an increasing volume of observed seismograms into our F3DT workflow, which is allowing us to resolve finer structural details with higher accuracy.

The map below shows the bounding box of this new velocity model. The bounding corners for the CCA model are: -120.0000, 33.3999, -122.9500, 36.6000, -118.2962, 39.3548, -115.4454, 36.0403. The starting model is defined using a 500m grid spacing and we use trilinear interpolation in between the grid points when constructing meshes. The model covers depths down to 50km.

The purpose of the F3DT for central California is to improve the crustal velocity model in central California for more accurate ground motion predictions. Our initial model is based on the updated Community Velocity Model for Southern California, CVM-S4.26 [Lee et al., 2014], and other existing velocity models for northern California [Xu et al., 2013]. Our study area is located on the edges of the two models, where the data coverage is poor for the two models. Our advances in this full 3D tomography work during this performance period include the following:

The 5th Full-3D tomography (F3DT) iteration for Central California crust

During the 2015 Q2, we have completed the 5th iteration for the Central California crustal F3DT. In this iteration, we used the available ambient noise Green’s functions (ANGFs) among the broadband and short-period stations to invert the velocity model. We applied two bandpass filters to the ANGF waveforms to separate the high (0.1~0.18Hz) and low (0.03~0.1Hz) frequencies sources. In our 5th iteration, more than 12,000 waveform windows have been picked and more than 59,000 frequency dependent measurements have been made for the Central California tomographic inversion (Figure 1). To evaluate the waveform improvement, we measured the difference between observed uk ( t) and its corresponding synthetic ~uk (t) waveforms within the time window [tk, tk'] by the relative waveform misfit (RWM) statistic, defined by the integral

RWM k=∫tk

tk'

[uk (t )−~uk (t ) ]2dt

√∫t kt k'

uk ( t )2dt∫tk

tk'

~uk ( t )2dt

Here, the time [tk, tk'] for the kth waveform window runs from the starting and end time of the window. After our 5 tomographic inversions, the sum of RWM has reduced more than 28% when compare to that of the initial model (Figure 2). In addition, the iteratively inversions have reduced the variance of frequency dependent group delay measurements (dtg) by over 35% relative to the starting model (Figure 3). The perturbations in velocities have begun to heal the velocity artifacts inherited from starting model (Figure 4). Many features revealed in the model are consistent with independent geophysical observations in Central California, including controlled-source tomography, gravity anomalies, and the locations of active faults.

Full-wave centroid moment tensor (CMT) inversion in an updated 3D velocity model for earthquakes in Central California

To include the earthquake recordings in the next iteration, we have applied a full-wave Central Moment Tensor (CMT) inversion algorithm to more than 200 earthquakes recorded in Central California (Figure 5). The procedure relies on the use of receiver-side Green tensors (RGTs), which comprise the spatial-temporal displacements produced by the three orthogonal unit impulsive point forces acting at the receiver. We have constructed a RGT database for more than 180 broadband stations in Central California using the updated Central California F3DT to reduce the potential errors in velocity structures. In our CMT inversion method, we implement the Bayesian inference on our measurements. An important advantage of the Bayesian approach is that, instead of a single best solution, the complete posterior probability density on the sample space is obtained, which allows formal estimation of the uncertainties associated with the derived source parameters. More robust earthquake source parameter estimates are critical for both geologic interpretation of active faults and seismic hazard analysis.

Validation of Central California F3DT using earthquake recordings not used in previous inversions

To validate the updated Central California F3DT, we tested the waveform predictions of the model using the earthquake recordings not used in previous tomographic inversions. More than 11,500 three-component broadband seismograms with signal-to-noise ratio (SNR) larger than 4 has been used in this validation (Figure 6). All synthetic and observed seismograms were band-pass filtered using a Butterworth filter with corners at 0.02 Hz and 0.2 Hz. We measured RWM between an individual observed seismogram uk ( t) and its corresponding synthetics ~uk (t) of the initial model (CCA00) and the updated model (CCA05). The time window for the waveforms are from the first arrival to the end of the main surface wave group, so that RWM measures the net waveform difference across all of the main phases on the seismograms. In those comparisons, the synthetics computed using the updated velocity model (CCA05) provide better fit to observed seismograms at frequencies below 0.2 Hz than those computed using the initial model (CCA00) (Figure 6).

Figure 1. Distribution of all frequency dependent measurements for the 5th iteration at different frequencies for ambient noise Green’s functions.

Figure 2. The histograms of RWMs for ambient noise Green’s functions for the initial model (CCA00) and the updated model (CCA05).

Figure 3. The histograms of frequency dependent group delay measurements (dtg) for ambient noise Green’s functions for the initial model (CCA00) and the updated model (CCA05).

Figure 4. Shear wave (S wave) velocity at (top) 2 km, (middle) 10 km, and (bottom) 20 km depths in (left) the initial model CCA00, (middle) the 5th iteration model CCA05, and (right) the perturbations. The color bar on the lower right corner of each plot shows the range of the color scale with red indicating relatively slow S wave velocities and blue indicating relatively fast S wave velocities. Black solid lines show major faults in our study area.

Figure 5. The CMT solution for earthquakes analyzed in Central California. Yellow triangles indicate the locations of broadband stations. Red dots indicate the epicenters of those earthquakes. The box indicates our study area. Major faults in this area are plotted in black solid lines. The background color shows topography.

Figure 6. The histograms of RWMs for earthquake recordings not used in current tomographic inversion for the initial model (CCA00) and the updated model (CCA05).

b) We completed a 1-Hz urban seismic hazard model for the Los Angeles region (Figure 1a-1b).

SCEC's research team used the OLCF Titan and NCSA Blue Waters supercomputers to perform CyberShake Study 15.4 (initiated in April, 2015). This computation doubled the maximum seismic frequency represented in the Los Angeles urban seismic hazard model, from 0.5 Hz to 1 Hz. Seismic hazard curves were derived from large ensembles of seismograms at frequencies below this maximum for 336 surface sites distributed across the Los Angeles region. This new probabilistic model uses refined earthquake rupture descriptions through revisions to the conditional hypocenter distributions and the conditional slip distributions. This seismic hazard calculation used the CVM-S4.26 3D velocity model, which was validated and improved using ALCF Mira, as the best available southern California 3D velocity model. The CS15.4 model provides new seismic hazard information of interest to broad impact customers of CyberShake, including seismologists, utility companies, and civil engineers responsible for California building codes. The new model, which samples the complete Uniform California Earthquake Rupture Forecast, will be registered into the USGS Urban Seismic Hazard Mapping Project (http://earthquake.usgs.gov/hazards/products/urban/).

The GPU-based anelastic wave propagation AWP-ODC software was used to run CPU-based post-processing calculations that synthesized over 300 million seismograms. In Study 15.4, SCEC utilized approximately 200 pilot jobs to run CyberShake tasks on Titan resources. Over 80% of the node-hours burned on Titan were from jobs which ran on 25% or more of the machine. Approximately 200 TB of SGT data was transferred from Titan to Blue Waters automatically as part of the workflow. On Titan, the accelerated calculations of the GPU Strain Green Tensor (SGT) implementation is 6.3 times more efficient than the CPU implementation, which saved us 2 million node-hours over the course of the study.

Our GPU development was recognized with NVIDIA’s 2015 Global Impact Award. “The full three-dimensional treatment of seismic-wave propagation has the potential to improve seismic hazard analysis models considerably, and that is where the accelerating technology is particularly helpful at this moment,” said Thomas Jordan, director of SCEC. “With GPU computing power we’re gaining insight as to how the ground will move in high-risk areas, and how we can better plan for the aftermath of a major event.”

c) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan

The SCEC finite-element wave propagation solver, Hercules, which integrates an efficient octree-based hexahedral mesh generator with an explicit FE formulation, has been optimized on Titan this year achieving near perfect strong and weak scaling. Its GPU capabilities are currently being used in verification and validation studies for the 2014 Mw 5.1 La Habra earthquake on Titan, to

test the accuracy of the code compared to other codes, and to examine how close the predicted ground motions are to observations.

We have implemented non-associated Drucker-Prager nonlinear rheology following the return map algorithm in the scalable AWP-ODC code, and we have used this code to model ground motions from the M7.8 ShakeOut scenario source description. This work accounts for the limited strength of crustal rocks; i.e., to simulate the absorption of rupture energy by permanent rock deformation. Our results suggest that this nonlinear behavior could reduce previous simulation-based predictions of expected ground motion velocity in the Los Angeles basin during a large-magnitude event on the southern San Andreas Fault by 30 to 70 percent. Nonlinear material response occurs in soft soils near the surface, typically reducing high-frequency (> 1 Hz) shaking that controls damage to low- and mid-rise buildings. Our simulations show that nonlinear response in crustal rocks may also reduce the amplitudes of long-period surface waves that pose a hazard to high-rise buildings, implying less destruction than previously anticipated. Although more research will be needed to quantify the impact of these findings on damage and casualty estimates for future large-magnitude earthquakes on the San Andreas Fault, the study pioneers more accurate earthquake scenarios based on better representations of the nonlinearity in the Earth's crust.

We have implemented realistic attenuation structure (frequency-dependent Q, or Q(f)) in the GPU-based AWP-ODC code (Withers et al., 2015). Tests using the 2008 Mw 5.4 Chino Hills earthquake indicate that Q(f) generally fits the strong motion data better than for constant Q models for frequencies over 1-Hz, which becomes more and more important as the distance increases from the fault. We also found that media heterogeneity reduces the within-event variability to that for observations and is thus important to characterize the ground motion.

Realistic ground-motion simulations require highly accurate crustal structural models. A significant portion of the awarded computational resources was used to construct full-3D, high-resolution crustal seismic velocity models in the Central California region and also in the statewide California through full-3D seismic waveform tomography (F3DT) (Lee et al. 2014ab). F3DT represents the latest development in seismic tomography techniques. Its application to seismic data recorded in Southern California has yielded a new community velocity model for the region, CVM-S4.26, which has unprecedented resolution of crustal structure. CVM-s4.26 is the 3D structural model used in the CyberShake 15.4 study.

Impact of Research: The San Andreas fault system is prone to major earthquakes, yet Los Angeles has not experienced a major quake since its urbanization in the early twentieth century. Data for the region are available from smaller quakes, but such information doesn’t give emergency officials and structural engineers the information they need to prepare for a quake of magnitude 7.5 or bigger. CyberShake Study 15.4 represents a major milestone in physics-based PSHA for Southern California. The performance of the code and improved workflow management, combined with the new physics it models (e.g., fault roughness, small-scale heterogeneities, frequency-dependent attenuation, near-surface nonlinearities), take physics-based seismic hazard analysis to a new level and pioneer the use of Petascale heterogeneous computing resources for ground motion simulations used in building engineering design and evaluation.

The reduction of peak velocities in our models caused by mostly shallow, near-fault nonlinear effects may have important implications for the scaling of ground motion intensities between surface-rupturing and buried earthquakes. Our nonlinear simulation results show that nonlinearity in the fault zone is important even for conservative values of cohesion, suggesting that current simulations based on a linear behavior of rocks are over-predicting the level of ground motion in the Los Angles sedimentary basins during future large earthquakes on the southern San Andreas Fault, and possibly for other large earthquake scenarios. This will have far-reaching implications on earthquake emergency planning scenarios that are based on ground motions predictions, such as the damage scenario of the 2008 Great California ShakeOut. The addition of statistical models of near-surface small-scale heterogeneities has enabled us to capture the “within-event” variability of earthquakes more accurately, providing models that can be used to improve physics-based seismic hazard analysis.

References:Lee, E.-J., P. Chen, T. H. Jordan, P. B. Maechling, M. A. M. Denolle, and G. C. Beroza (2014), Full-3-D tomography for crustal structure in Southern California based on the scattering-integral and the adjoint-wavefield methods, J. Geophys. Res. Solid Earth, 119(8), 6421–6451, doi:10.1002/2014JB011346.

Xu, Z., P. Chen, and Y. Chen (2013), Sensitivity Kernel for the Weighted Norm of the Frequency-Dependent Phase Correlation, Pure Appl. Geophys., 170(3), 353–371, doi:10.1007/s00024-012-0507-3.

Project ProductivityPrimary Publications –

1. Isbiliroglu, Y., R. Taborda and J. Bielak (2015) Coupled soil-structure interaction effects of building clusters during earthquakes. Earthquake Spectra. Vol. 31, No. 1, 463-500, Feb 2015.

2. Donovan, J. (2015), Forecasting Directivity in Large Earthquakes in Terms of the Conditional Hypocenter Distribution, PhD Thesis, University of Southern California, 154 pp.

3. Jordan, T. H. (2015), An effective medium theory for three-dimensional elastic heterogeneities, Geophys. J. Int., submitted Mar 29, 2015.

4. Lozos, J., K. B. Olsen, J. Brune, R. Takedatsu, R. Brune, and D.D. Oglesby (2015), Broadband ground motions from dynamic models of rupture on the northern San Jacinto fault, and comparison with precariously balanced rocks, Bull. Seismol. Soc. Am., 105. (in press), doi: 10.1785/0120140328.

5. Olsen, K. B. and R. Takedatsu (2015), The SDSU Broadband Ground-Motion Generation Module BBtoolbox Version 1.5, Seism. Res. Letter, 86, 1, 81-88.

6. Roten, D., K. B. Olsen, Y. Cui, and S. M. Day (2015), Quantification of fault zone plasticity effects with spontaneous rupture simulations, to be submitted to Workshop on Best Practice in Physics-Based Fault Rupture Models for Seismic Hazard Assessment of Nuclear Installations, Vienna, Austria, Nov 18-20.

7. Shaw, J. H., A. Plesch, C. Tape, M. P. Suess. T. H. Jordan, G. Ely, E. Hauksson. J. Tromp, T. Tanimoto, R. Graves, K. Olsen, C. Nicholson, P. J. Maechling, C. Rivero, P. Lovely, C. M. Brankman, and J. Munster (2015), Unified Structural Representation of the southern California crust and upper mantle, Earth Planet. Sci. Lett., 415, 1-15, doi:10.1016/j.epsl.2015.01.016.

8. Withers, K. B., K. B. Olsen, S. M. Day (2015). Memory-efficient simulation of frequency dependent Q, Bull. Seismol. Soc. Am., in revision.

Presentations –

1. Seismological Society of America Meeting 2015 - Lee, E., Thomas, H. J., Chen, P., Maechling, J. P., Boué, P., Denolle, M., Beroza, G., & Eymold, W. K. (2015) Full-3D Tomography of Crustal Structure in Central California. Abstract and presentation, 2015 SSA Annual Meeting.

2. Blue Waters 2015 Symposiu - SCEC presented the results at the annual Blue Waters symposium, include the CyberShake calculation, an example of a SCEC, NSF Blue Waters, and INCITE research collaborative effort. Links to the presentation are posted on a SCEC wiki at: http://scec.usc.edu/scecpedia/Blue_Waters_Symposium_2015

3. NSF Software Infrastructure for Sustained Innovation 2015 PI Meeting: - SCEC members presented software descriptions and research results related to our Full 3D tomography (F3DT) and Unified Community Velocity Model (UCVM) work using INCITE resource at a Feb 2015 NSF Software Infrastructure for Sustained Innovation (SI2) meeting

Jan 2015. The NSF SI2 program currently provides research funding for software infrastructure including F3DT, UCVM, AWP-ODC, and Hercules used on our SCEC INCITE research activities. More details are available via the following link to the meeting website: https://share.renci.org/SI2PI2015/Lists/SI2PI2015Posters/View_01.aspx

Secondary Journal Covers, Awards, Honors, Popularizations

In April 2015, SCEC worked with an INCITE and DOE team to create a display kiosk describing how SCEC researchers were able to use INCITE resources to perform seismic hazard research for use in the 2015 Science Bowl. Our primary DOE point of contact for that activity was Carolyn Lauzon, ALCC Program Manager Advanced Scientific Computing Research U.S. Department of Energy.

Technical Accomplishments – Please list technical accomplishments such as development of reusable code resulting in a new tool, new algorithm design ideas or programming methodologies, formal software releases, etc.

We continued to develop our workflow capabilities on Titan to support our CyberShake computational goals. We used Pegasus-WMS, Condor DAGManager, Globus-based workflows to organize and automate our CyberShake SGT workflows and data transfer on Titan.

Other, for example: Simulation results used in outreach initiatives/students graduated or postdocs deployed; Journal Covers; Awards/Honors –

Highlights – the center creates (concise, short, highly visible) bi-weekly center highlights to submit to DOE—is your project ready, willing, and able to contribute a highlight?

Center Feedback Please answer as applicable: Has the support received from the following been beneficial to

your project team? Cite examples if possibleo User Assistance Centero Scientific Computing Groupo Visualization and Analysis Team

Any additional feedback from your project team for the ALCF?

We received beneficial support from ALCF user support group during Q1 regarding migration of existing project data sets on Mira, under a previous allocation (Equake_SS), to project directories under our current INCITE allocation (GMSeismicSim). The ALCF User support group gave us appropriate options (delete existing data, migrate existing data to new allocation), and plenty of advance notice. They also worked around our on-going simulations on Mira, so that the migration occurred without impacting our new simulations.

We also received helpful support from the OLCF User assistance group. Before we began our CyberShake production simulations using Titan, the User assistance team participated on a

telecom with our SCEC research group to review our simulation plans. They provided useful feedback that helped us make better use of Titan during these runs.

Code Description and Characterization Name and provide a description of the primary codes used by your project What are the typical production run sizes that your team plans to undertake in the coming

year?

During Q2, we performed a large-scale production seismic hazard calculation that we call CyberShake Study 15.4 using Titan and Blue Waters. In this production workflow, we calculated 1Hz CyberShake probabilistic seismic hazard curves for 336 site locations. The run took 38 days, considerably less than the 84 days we estimated. The 38 day makespan includes 13 days of downtime. This CyberShake Study 15.4 was a cooperative and collaborative technical accomplishment for the INCITE Leadership class computing organization at OLCF and the NSF Track 1 computing organization at NCSA, and SCEC, working with California seismic hazard organizations on an important seismic hazard data product.

For this study, we ran over 4000 jobs on both Blue Waters CPU and GPU nodes and on Titan GPU nodes and used about 642,000 node-hours on Blue Waters and 426,000 node-hours on Titan, a total of 888,000 node hours. All CyberShake 15.4 required calculations were defined in a scientific workflow prior to starting, ensuring repeatability of the study. Study jobs were submitted to standard user queues on Blue Waters and Titan.

For CyberShake 15.4, it averaged 2640 node-hours/site, which is an increase of 5.8x over the 455 node-hours/site required for CyberShake Study 14.2. We expect to have more details performance metrics for this study in the next few months, after we processes existing workflow performance logs gather during the run.

This is the first 1Hz CyberShake hazard model calculation ever performed. 1Hz calculations have been out of reach until recent performance improvements in our wave propagation code, development of a GPU version of the code, and performance improvements in automation of our large many-task computing post-processing. Previous CyberShake studies were done at 0.5Hz. Doubling the frequency to 1.0Hz requires SGT calculations that do 16x as much work, and post-processing calculations that do 50x. So, an actual increase of only 5.8x to perform a site calculation actually represents a very impressive efficiency gain for this kind of heterogeneous, end-to-end, scientific calculation.

What languages and libraries (scientific, I/O, etc.) are used in each code? If possible and useful, please indicate which of the following algorithmic motifs appear in each

of your major production codes.

Several of our research group members have used both ALCF and OLCF systems in small or medium scale simulations in order to evaluate some aspects of our research code on INCITE systems. However, at this time we expect that our F3DT and CyberShake simulations, the two main computational projects discussed in this progress report, to use most of our computing time in the next few months. These two computational projects both make use of different versions of

our AWP-ODC finite difference wave propagation software. For both F3DT and CyberShake, our SCEC research group is working, during Q2, to increase the number of nodes used by our codes during our production runs. Our codes scale well, and, with what we believe will be some minor modifications, we are working to increase the nodes used to 20% or more of more of Mira and Titan during standard production runs.

Code Name

Dense Linear Algebra

SparseLinearAlgebra

Monte Carlo

FFTs Particles Structured Grids

Unstructured Grids

AMR

AWP-ODC

Yes

AWP-ODC-GPU-SGT

Yes