INCITE 2015 Project Name: High Frequency Physics-Based Earthquake System Simulations (Year 1 of 2)

PI: Thomas H. Jordan

Co-PI(s): Jacobo Bielak, Carnegie Mellon University, Po Chen, University of Wyoming,

Yifeng Cui, San Diego Supercomputer Center, Philip Maechling, Southern California Earthquake Center,

Kim Olsen, San Diego State University, Ricardo Taborda, University of Memphis

ALCF Project Name: GMSeismicSimOLCF Project Name: GEO112

Performance Period: Q1- 2015 (Jan 2015 - March 2015)

Quarterly Update: Q1 - 2015

Report on Project Milestones: Provide status on each of your project’s simulations milestones as outlined in your

original proposal.

Year 1 Milestone Descriptions MilestoneAchievement Status

M1 Use full 3D tomography and comparative validations using to improve existing California velocity models for use in high frequency wave propagation simulations at 0.2Hz

Achieved. Used Mira to calculate four iteration of Central California Model now available to ground motion modelers through UCVM.

M2 Run high frequency forward simulations using alternative material attenuation (Q) and seismic velocity models (CVMs). Compare the impact of material properties, topography, and models including spatial variability (heterogeneities) and soft-soil deposits (or geotechnical layers) on 4Hz+ simulations by simulating forward events using alternative models and comparing results among synthetics and with data.

Started, not completed. AWP-ODC and Hercules code branches have tested these physics. Currently running baseline 4Hz simulations without these physics.

M3 Run high frequency forward simulations using alternative approaches to include the effects of off-fault and near-surface plastic deformation. Compare the impact of alternative plasticity models (linear-equivalent, 3D+1D hybrid, full 3D plastic) on 4Hz+ simulations by simulating forward events and comparing the results among synthetics and with empirical relationships and data.

Started, not completed. AWP-ODC and Hercules code branches have tested these physics. Currently running baseline 4Hz simulations without these physics.

M4 Calculate a 1.0Hz CyberShake Hazard curve. Use updated CVMs, source models, and codes to calculate a higher frequency CyberShake hazard curve

Achieved. Used Titan and Blue Waters to calculate 32 1Hz CyberShake hazard curves.

Year 2 Milestone Descriptions ObjectiveM5 Use full 3D tomography and comparative validations using to

improve existing California velocity models for use in high frequency wave propagation simulations at 0.5Hz

Not started

M6 Run high frequency forward simulations using alternative material attenuation (Q) and seismic velocity models (CVMs). Compare the impact of material properties, topography, and models including spatial variability (heterogeneities) and soft-soil deposits (or geotechnical layers) on 8Hz+ simulations by simulating forward events using alternative velocity models and comparing the results.

Not started

M7 Run high frequency forward simulations using alternative approaches to include the effects of off-fault and near-surface

Not started

plastic deformation. Compare the impact of alternative plasticity models (linear-equivalent, 3D+1D hybrid, full 3D plastic) on 8Hz+ simulations by simulating forward events and comparing the results among synthetics and with empirical relationships and data.

M8 Calculate a 1.5Hz CyberShake Hazard curve. Use updated CVMs, source models, and codes to calculate a higher frequency CyberShake hazard curve

Not started

List major accomplishments thus far in CY2014. Please include scientific and computational details of simulations undertaken, including images if possible.

During Q1 of our INCITE Project Year 1, out computational group made significant progress in two project areas. (1) Using Mira, we have used our full 3D tomography computational method to produce a 3D seismic velocity model for Central California that we call Central California Area (CCA). (2) Using Titan, we calculated our first 1Hz CyberShake hazard curves.

(1) Using Mira, we have used our full 3D tomography computational method to produce a 3D seismic velocity model for Central California that we call Central California Area (CCA)

During Q1, we have made rapid progress using our full 3D tomography (F3DT) computational techniques to produce a 3D seismic velocity model for Central California. As of 1 April 2015, we have completed the 4th F3DT iteration of a new central California crustal seismic velocity model.

The map below shows the bounding box of this new velocity model. The bounding corners for the CCA model are: -120.0000, 33.3999, -122.9500, 36.6000, -118.2962, 39.3548, -115.4454, 36.0403. The starting model is defined using a 500m grid spacing and we use trilinear interpolation in between the grid points when constructing meshes. The model covers depths down to 50km.

The purpose of the F3DT for central California is to improve the crustal velocity model in central California for more accurate ground motion predictions. We have applied our F3DT techniques to central California and completed four iterations (fig. 1). Our initial model is based on the updated Community Velocity Model for Southern California, CVM-S4.26 [Lee et al., 2014], and other existing velocity models for northern California [Xu et al., 2013]. Our study area is located on the edges of the two models, where the data coverage is poor for the two models. Therefore, the velocity model in central California still needs further improvements.

Figure 1. Shear wave (S wave) velocity at (top) 2 km, (middle) 10 km, and (bottom) 20 km depths in (left) the initial model CCA00, (middle) the 4th iteration model CCA04, and (right) the perturbations. The color bar on the lower right corner of each plot shows the range of the color scale with red indicating relatively slow S wave velocities and blue indicating relatively fast S wave velocities. Black solid lines show major faults in our study area.

In this tomographic model, we first used the available ambient noise Green’s functions (ANGFs) to invert the central California crustal velocity model. The advantages of using ANGFs in seismic

tomographic inversion are three-folded. First, the source and receiver locations of ANGFs are known. Second, the use of ANGFs could improve the data coverage in our study area. Third, the new velocity model improved through the ANGFs dataset could be used for earthquake source inversions to reduce the structure errors in latter tomographic inversion processes. Currently, the ANGF dataset includes available station pairs among stations in different seismic networks, including temporal broadband stations in the Incorporated Research Institutions for Seismology (IRIS) and permanent broadband and short-period stations in northern California and southern California. Figure 2 shows the estimated data coverage of our current data.

Figure 2. The data coverage estimates of two ANGF datasets. The model is separated into 3 km by 3 km cells and the color represents the number of crossing path in each cell. (a) The data coverage of ANGFs in the 3rd iteration. (b) The ANGF data coverage of 4th iteration. In the 4th iteration, the ANGF dataset includes the empirical Green’s functions among broadband and short-period stations and therefore the data coverage is significantly improved in our study area.

In our tomographic inversions, we applied two bandpass filters to the waveforms of ANGFs to separate the high (0.1~0.18Hz) and low (0.03~0.1Hz) frequencies sources. In our 4th iteration, more than 13,000 waveform windows with usable signal-to-noise ratios were used for tomographic inversion. In this tomographic inversion, we adopted the scattering-integral (SI) method for kernel calculations and a Gauss-Newton algorithm, LSQR method, for inversion due to the good initial model and the high efficiency of the inversion method. For ANGFs, we stored the strain fields of receiver-side strain Green tensors and the strain fields were decimated by a factor of 2 in each spatial dimension and a factor of 10 in time. Since we were using only the vertical component of ANGFs in our inversion, only one finite-difference simulation with a vertical point force at the virtual source location was needed for each virtual source.

In our 4th update, we still gained about 13% of reduction on summary of relative waveform misfit (RWM) and about 8% of reduction on variance of phase-delay time measurements (fig. 3). Figure

4 shows examples of waveform improvements on ambient noise Green’s functions cross our study area. As illustrated in fig. 1, the perturbations have reduced the velocity gradients between the northern and southern California initial models and begun to heal the velocity artifacts inherited from starting model. At the upper crust (2 km), the Vs in the Great Valley reduced more than 10% respect to the initial model. In the middle crust (~ 20km), the Vs in the Sierra Nevada region increased about 5~7% with respect to the initial model. The perturbations also enhance the velocity gradients across faults, such as the San Andreas Fault at 20 km. Many features revealed in the model are consistent with independent geophysical observations in central California, including controlled-source tomography, gravity anomalies, and the locations of active faults.

Figure 3. Distribution of relative waveform misfits and phase delay times between observed and synthetic waveforms for the ambient noise Green’s functions (ANGFs) used in the 4th iteration. The upper row shows the measurements between observed and synthetic waveforms of the 3rd iteration model and the lower row shows that of the 4th model.

Figure 4. Examples of waveform improvements in our starting CCA model. The map shows the source and receiver stations and the source-receiver paths are represented by green lines. In waveform comparisons, the data seismograms are in black and the synthetics are in red.

To help researchers access the CCA 3D velocity model, we have registered the CCA starting model, and each of the four resulting F3DT updated models into SCEC Unified Community Velocity Model (UCVM) software framework. Now registered into UCVM, CCA models can be queried and used to build velocity meshes for use in ground motion simulation. Figures 5 shows Vs values at three depths from the CCA starting model. Figure 6 shows Vs values at the same three depths from the CCA-I4 (the fourth iteration of the CCA model) as

Figure 5: These three horizontal slices were also taken at 0m, 500m, and 200m depths from the starting model.

Figure 6: These three horizontal slices generated from UCVM that show the fourth iteration CCA model at the surface level (0m depth), 500m depth, and 2km depth.

(2) Using Titan, we calculated our first 1Hz CyberShake hazard curves.

In Q1, we used Titan to calculate Strain Green’s Tensors (SGT) using AWP-ODC-GPU code. We have developed workflows that submit SGT jobs and transfer the resulting SGT-data to Blue Waters, which we use for a many-task computing (MTC) phase of our research calculations.

Our CyberShake research now involves broad impact users including utilitiy companies (PG and E) and civil engineers developing California building codes. To meet the needs of these users, CyberShake calculations must be expanded in two dimensions. First, earthquake simulations must simulate higher frequency ground motions. Second, the geographical region for which CyberShake seismic hazard models are calculated must be expanded.

The F3DT work, performed on Mira, and described above, will produce a 3D seismic velocity model for central California. This will enable SCEC to extend our CyberShake calculations to more geographical regions in California, extending our results from southern California to include Central California. We have not yet begun CyberShake calculations for Central California, but our INCITE F3DT work is preparing the necessary earth structure model needed for that work.

SCE researchers have been working to run CyberShake simulations at higher frequencies. During Q1, we double the maximum simulated CyberShake frequency from 0.5Hz to 1Hz. These new higher frequency CyberShake simulations will provide seismic hazard information at the higher frequencies of interest to civil engineers. SCEC researchers have worked for months to verify and validate our 1Hz CyberShake simulations, and by the end of Q1 2015, our research group has now used INCITE resources to calculate valid 1Hz CyberShake simulations.

Figure 7 provides one example (of nearly 30 completed seismic hazard curves) of a 1Hz seismic hazard curve. One of our INCITE computational goals for Q2 2015 is to complete a Los Angeles

regional seismic hazard model at 1Hz. We expect these results to be reviewed at a May 2015 meeting with a civil engineering group developing recommendations for California Building codes.

Figure 7: A seismic hazard curve for a site (Century City Plaza) in Los Angeles calculated using Titan during Q1 2015 showing the probabilities this site will experience ground motions that exceed Peak Spectral Accelerations at 2 Seconds Period (PSA2.0). CyberShake seismic hazard estimates for PSA 2 Seconds are a new result that was not previously available from the lower frequency CyberShake results calculated previously at 0.5Hz.

References:

Lee, E.-J., P. Chen, T. H. Jordan, P. B. Maechling, M. A. M. Denolle, and G. C. Beroza (2014), Full-3-D tomography for crustal structure in Southern California based on the scattering-integral and the adjoint-wavefield methods, J. Geophys. Res. Solid Earth, 119(8), 6421–6451, doi:10.1002/2014JB011346.Xu, Z., P. Chen, and Y. Chen (2013), Sensitivity Kernel for the Weighted Norm of the Frequency-Dependent Phase Correlation, Pure Appl. Geophys., 170(3), 353–371, doi:10.1007/s00024-012-0507-3.

Project ProductivityPrimary Publications – Not yet

Presentations – Group members presented software descriptions and research results related to our Full 3D tomography (F3DT) and Unified Community Velocity Model (UCVM) work using INCITE resource at a Feb 2015 NSF Software Infrastructure for Sustained Innovation (SI2) meeting Jan 2015. The NSF SI2 program currently provides research funding for software infrastructure including F3DT, UCVM, AWP-ODC, and Hercules used on our SCEC INCITE research activities.More details are available via the following link to the meeting website:https://share.renci.org/SI2PI2015/Lists/SI2PI2015Posters/View_01.aspx

Secondary Journal Covers, Awards, Honors, PopularizationsNo

Technical Accomplishments – Please list technical accomplishments such as development of reusable code resulting in a new tool, new algorithm design ideas or programming methodologies, formal software releases, etc.

We continued to develop our workflow capabilities on Titan to support our CyberShake computational goals. We are using our Pegasus-WMS, Condor DAGManager, Globus-based workflows to organize and automate our CyberShake SGT workflows and data transfer on Titan. The performance and impact on Titan of these initial smaller-scale workflows will be evaluated in order to determine whether we can run our full-scale workflows on Titan.

Other, for example: Simulation results used in outreach initiatives/students graduated or postdocs deployed; Journal Covers; Awards/Honors –

No

Highlights – the center creates (concise, short, highly visible) bi-weekly center highlights to submit to DOE—is your project ready, willing, and able to contribute a highlight?

No

Center Feedback Please answer as applicable: Has the support received from the following been

beneficial to your project team? Cite examples if possibleo User Assistance Centero Scientific Computing Groupo Visualization and Analysis Team

Any additional feedback from your project team for the ALCF?

We received beneficial support from ALCF user support group during Q1 regarding migration of existing project data sets on Mira, under a previous allocation (Equake_SS), to project directories under our current INCITE allocation (GMSeismicSim). The ALCF User support group gave us appropriate options (delete existing data, migrate existing data to new allocation), and plenty of advance notice. They also worked around our on-going simulations on Mira, so that the migration occurred without impacting our new simulations.

We also received helpful support from the OLCF User assistance group. Before we began our CyberShake production simulations using Titan, the User assistance team participated on a telecom with our SCEC research group to review our simulation plans. They provided useful feedback that helped us make better use of Titan during these runs.

Code Description and Characterization Name and provide a description of the primary codes used by your project What are the typical production run sizes that your team plans to undertake in the

coming year? What languages and libraries (scientific, I/O, etc.) are used in each code? If possible and useful, please indicate which of the following algorithmic motifs appear

in each of your major production codes.

Several of our research group members have used both ALCF and OLCF systems in small or medium scale simulations in order to evaluate some aspects of our research code on INCITE systems. However, at this time we expect that our F3DT and CyberShake simulations, the two main computational projects discussed in this progress report, to use most of our computing time in the next few months. These two computational projects both make use of different versions of our AWP-ODC finite difference wave propagation software. For both F3DT and CyberShake, our SCEC research group is working, during Q2, to increase the number of nodes used by our codes during our production runs. Our codes scale well, and, with what we believe will be some minor modifications, we are working to increase the nodes used to 20% or more of more of Mira and Titan during standard production runs.

Code Name

Dense Linear Algebra

SparseLinearAlgebra

Monte Carlo

FFTs Particles Structured Grids

Unstructured Grids

AMR

AWP-ODC

Yes

AWP-ODC-GPU-SGT

Yes