Accelerators at ORNL - Application Readiness, Early Science, and Industry Impact

http://energy.ornl.gov/

Experiences using

Accelerators at ORNL

Application Readiness, Early Science, and Industry Impact

John A. Turner

Group Leader Computational Engineering and Energy Sciences

Chief Computational Scientist Consortium for Advanced Simulation of

Light-Water Reactors (CASL)

HPC User Forum

Sept 15-17, 2014, Seattle, WA

2 http://energy.ornl.gov/

Overview

• U.S. DOE Leadership Computing Program

• OLCF-3: The Titan Project

• Application Readiness and Early Science on Titan

• The Consortium for Advanced Simulation of Light-Water

Reactors (CASL) – a U.S. DOE Innovation Hub

– connection to Titan Application Readiness and Early Science

• Industry Impact


What is the U.S. DOE Leadership

Computing Program?

• Collaborative DOE Office of Science program at ORNL and ANL

• Mission: Provide the computational and data resources required to solve the most challenging problems.

• 2-centers / 2-architectures to address diverse and growing computational needs of the scientific community

• Highly competitive user allocation programs (INCITE, ALCC).

• Projects receive 10x to 100x more resource than at other generally available centers.

• LCF centers partner with users to enable science & engineering breakthroughs (Liaisons, Catalysts).


Three primary ways for access to LCF

Distribution of allocable hours

60% INCITE 5.8 billion core-hours in

CY2014

Up to 30% ASCR Leadership Computing

Challenge

10% Director’s Discretionary

Leadership-class computing

DOE/SC capability

computing

INCITE seeks computationally intensive, large-

scale research and/or development

projects with the potential to

significantly advance key

areas in science and

engineering.


Demand for INCITE resources outstrips

supply with 3x more time requested than

available – Number of projects remains flat

2004 2010 2005 2009 2006 2007

2004

2008 2011 2013 2012


200 cabinets

4,352 ft2 (404 m2)

8.2 MW peak power

SYSTEM SPECIFICATIONS:

• 27.1 PF/s peak performance • 24.5 GPU + 2.6 CPU

• 17.59 PF/s sustained perf. (LINPACK)

• 18,688 compute nodes, each with: – 16-Core 2.2 GHz AMD Opteron 6200 CPU

– NVIDIA Tesla K20x GPU (2688 “cores”)

– 32 GB DDR3 + 6 GB DDR5 memory

• 710 TB total system memory

• 32 PB parallel filesystem (Lustre)

• Cray Gemini 3D Torus Interconnect

• 512 Service and I/O nodes

ORNL’s “Titan” Hybrid System:

Cray XK7 with AMD Opteron + NVIDIA Tesla processors

• Throwing away 90% of available performance if not using GPUs

Throwing away 90% of available performance if not using GPUs


Center for Accelerated Application

Readiness (CAAR)

• We created CAAR as part of the Titan project to help prepare applications for accelerated architectures

• Goals: – Work with code teams to develop and implement strategies for

exposing hierarchical parallelism for our users applications

– Maintain code portability across modern architectures

– Learn from and share our results

• We selected six applications from across different science domains and algorithmic motifs


Early Science Challenges for Titan

WL-LSMS Illuminating the role of material disorder, statistics, and fluctuations in nanoscale materials and systems.

S3D Understanding turbulent combustion through direct numerical simulation with complex chemistry. .

NRDF Radiation transport – important in astrophysics, laser fusion, combustion, atmospheric dynamics, and medical imaging – computed on AMR grids.

CAM-SE Answering questions about specific climate change adaptation and mitigation scenarios; realistically represent features like precipitation patterns / statistics and tropical storms.

Denovo Discrete ordinates radiation transport calculations that can be used in a variety of nuclear energy and technology applications.

LAMMPS A molecular dynamics simulation of organic polymers for applications in organic photovoltaic heterojunctions , de-wetting phenomena and biosensor applications


CAAR Plan

• Comprehensive team assigned to each app – OLCF application lead

– Cray engineer

– NVIDIA developer

– Other application, local tool/library developers, computational scientists

• Single early-science problem targeted for each app – Success on this problem is ultimate metric for success

• Particular plan-of-attack different for each app – WL-LSMS – dependent on accelerated ZGEMM

– CAM-SE– pervasive and widespread custom acceleration required

• Multiple acceleration methods explored – WL-LSMS – CULA, MAGMA, custom ZGEMM

– CAM-SE– CUDA, directives

– Two-fold aim

• Maximum acceleration for model problem

• Determination of reproducible acceleration path for others


Effectiveness of GPU Acceleration?

OLCF-3 Early Science Codes -- Performance on Titan XK7

Titan: Cray XK7 (Kepler GPU plus AMD 16-core Opteron CPU)

Cray XE6: (2x AMD 16-core Opteron CPUs) *Performance depends strongly on specific problem size chosen

Application Cray XK7 vs. Cray XE6

Performance Ratio*

LAMMPS* Molecular dynamics

7.4

S3D Turbulent combustion

2.2

Denovo 3D neutron transport for nuclear reactors

3.8

WL-LSMS Statistical mechanics of magnetic materials

3.8


Application Power Efficiency of the Cray XK7

WL-LSMS for CPU-only and Accelerated Computing

• Runtime Is 8.6X faster for the accelerated code

• Energy consumed Is 7.3X less

o GPU accelerated code consumed 3,500 kW-hr

o CPU only code consumed 25,700 kW-hr

Power consumption traces for identical WL-LSMS runs

with 1024 Fe atoms on 18,561 Titan nodes (99% of Titan)


0

20

40

60

80

100

120

0 3,000 6,000 9,000 12,000 15,000 18,000

Se

con

ds

Nodes

Titan Sweep Performance

CPU

GPU

Denovo SN

Acceleration

• SWEEP kernel re-written in C++ & CUDA, runs on CPU or GPU

• Refactored SWEEP is in mainline code

• Titan: SWEEP speedup of 6-7x, Denovo speedup ~3.8x

• Scaling to over 200K cores with opportunities for increased parallelism on GPUs

• Refactored code 2x faster on Cray XT5 (CPU only)

Full Denovo run, CPU vs. GPU sweeper,

CPU+GPU vs. CPU only

CPU vs. GPU sweeper, Titan, Kepler K20x


Increasing Usage of GPUs

As measured by ALTD against linked libraries

INCITE 2013

INCITE 2014


Results and Impact

• Advanced numerical methods

• Increased coupling of physics

• Increased use of mechanistic models of lower length-scale phenomena

• Large-scale software development – geographically-dispersed, multi-institutional

Computational Science Areas

Objectives and Strategies

Consortium for Advanced Simulation

of Light-Water Reactors (CASL)

• Advance understanding of key reactor phenomena

• Improve performance of today’s commercial power reactors

• Evaluate new fuel designs to further enhance safety margins

• DOE Innovation Hub on Modeling & Simulation for Nuclear Energy Systems

• Develop “Virtual Reactor” to assess fuel design, operation, and safety criteria

• Virtual Environment for Reactor Applications (VERA)


CASL was the first DOE

Innovation Hub

Core partners

Oak Ridge National Laboratory

Electric Power Research Institute

Idaho National Laboratory

Los Alamos National Laboratory

Massachusetts Institute of Technology

North Carolina State University

Sandia National Laboratories

Tennessee Valley Authority

University of Michigan

Westinghouse Electric Company

Contributing Partners

ASCOMP GmbH

CD-adapco

City College of New York

Florida State University

Imperial College London

Rensselaer Polytechnic Institute

Texas A&M University

Pennsylvania State University

University of Florida

University of Wisconsin

University of Notre Dame

Anatech Corporation

Core Physics Inc.

G S Nuclear Consulting, LLC

University of Texas at Austin

University of Texas at Dallas

University of Tennessee – Knoxville

Pacific Northwest National Laboratory

A Different Approach

• “Multi-disciplinary, highly collaborative teams ideally working under one roof to solve priority technology challenges” – Steven Chu

• “Create a research atmosphere with a fierce sense of urgency to deliver solutions.” – Kristina Johnson

• Characteristics – Leadership – Outstanding, independent, scientific

leadership – Management – “Light” federal touch – Focus – Deliver technologies that can change the

U.S. “energy game”


CASL Test Stands provide early

deployment to industry.

• Early deployment into industrial environment for testing, use, and adoption of VERA to support real-world LWR applications

• Status of initial deployment to core industry partners – WEC: Deployment during June 2013; focus on VERA

simulation of AP1000 first core startup

– EPRI: Deployment Dec 2013; fuel performance

– TVA: Deployment in progress; lower plenum flow anomaly

• Early Test Stand deployment is already producing dividends for CASL and users – Better code installation processes

– Input processing for heterogeneous cores

– Reductions in user problem setup times

– Core tilt analysis

– Analysis of new design features (e.g., tungsten rods)

Westinghouse VERA Test Stand

VERA deployed on Westinghouse computer cluster and

employed for a high-impact industrial application: • AP1000 PWR start-up core physics tests simulation

VERA-CS simulations performed on a dedicated

Westinghouse computer cluster where VERA has

been deployed by a CASL-Westinghouse team.

The graphic below shows the computer arrangement

and communication to automate VERA updates from

the ORNL central repository as new capabilities are

being added.

The Westinghouse VERA build is operational and

exercised by Westinghouse personnel.

Pictorial of the AP1000 plant with the fission rate distribution predicted

by VERA during one of the startup physics tests measurements. It will

be possible to compare the VERA predictions against measured data

when the first AP1000 unit will come on line in 2015 (Sanmen, China).

All the AP1000 units under construction will feature the same startup

core modeled by VERA.

The AP 1000 features an

advanced first core with

several enrichments and fuel

heterogeneities which allows

to quickly achieve

equilibrium cycle after fuel

shuffle and reload.

The advanced first core is a

major economic advantage

since it reduces the number

of transition cycles before

equilibrium but it also poses

challenges to the simulation.

VERA high-fidelity physics

provided an ideal match for

simulating this challenging

core and gain confidence in

the start-up predictions

AP1000 Monte Carlo model shown. Thanks to VERA common input, it has been possible

to generate a complete AP1000 core model within a compact and intuitive input.

AP1000 Advanced First Core Model


Timeline for CASL Westinghouse (WEC)

Test Stand

• early 2013: Test Stand discussion

• April 2013: Scope proposed by WEC

• June 2013: VERA deployment at WEC

• July-Nov 2013: Technical analysis

• Jan 2014: Analysis completed and

documented (Mar 2014)

Enhanced confidence in AP1000 PWR start-up predictions • High-quality benchmarks for code comparison

• Expanded application of VERA to an advanced core

• Feedback from WEC to guide future developments

• Framework for VERA build and update


Approaches to neutronics balance accuracy and

computational requirements. As part of Titan Early

Science, we compared 3 methods.

Method Attributes Code Cross

Sections

Energy Scat-

tering

Lang. Scalability

Simplified PN

(SPN)

Cartesian

mesh, Lin.

Syst.

Insilico pin-

homogen-

ized

multigroup PN C++ linear solver-

dependent

Discrete

Ordinates (SN)

Cartesian

mesh,

Wavefront

Denovo pin-

homogen-

ized

multigroup PN C++ >200,000

Method of

Characteristics

(MOC)

unstruct.

mesh, Ray

tracking

MPACT subgroup multigroup PN Fortran in testing

Monte Carlo CAD,

particle

tracking

Keno-IV exact continuous exact Fortran few hundred

Monte Carlo CAD,

particle

tracking

Shift exact continuous exact C++ >200,000


Execution

Goals

Evaluation of Shift: VERA Continuous-Energy Monte Carlo

Quarter-Core Zero Power Physics Test

• Awarded 60 million core-hours on Titan (worth >$2M) as part of Titan Early Science program

• AP1000 model created and results generated for reactor criticality, rod worth, and reactivity coefficients

• Identical VERA Input models used for Shift, SPN, and SN – dramatically simpler than KENO-VI input model

• Compare fidelity and performance of Shift against Keno, SPN, and SN (Denovo)

• Generate high-fidelity neutronics solution for code comparison of solutions for predicting reactor startup and physics testing

Results • Some of the largest Monte Carlo calculations ever performed

(1 trillion particles) have been completed – runs used 230,000 cores of Titan or more

• Excellent agreement with KENO-VI

• Extremely fine-mesh SN calculations, which leverage Titan’s GPU accelerators, are under way AP1000 pin powers

MB AO AO

MB MA


Acceleration efforts are underway or in

initial phases for other VERA components

• Hydra-TH (CFD, unstructured finite volume) – Performance analysis, code optimization and scaling efforts

have improved performance and scaling for both single phase and multiphase (MPI only)

– Collaboration with NVIDIA has been under way for over a year to exploit hybrid parallelism (MPI + threads) • Incorporating NVIDIA's AMG GPU lib, AmgX, into Hydra

(https://developer.nvidia.com/amgx)

– Thread other parts of the code (OpenMP, OpenACC, CUDA)

• MPACT (neutronics, Method of Characteristics) – Fortran, so OpenACC is most appropriate path forward

– NVIDIA staff stationed full-time at ORNL identified to assist team

• Shift (neutronics, Monte Carlo) – Core Exnihilo kernels, including Shift, have been extracted and

are being released as an open-source mini-app (Profugus)

– NVIDIA staff have been identified to assist in Shift acceleration

– Currently awaiting export control ruling for release


Questions?

e-mail: [email protected]

23 Managed by UT-Battelle for the Department of Energy

The research and activities described in this

presentation were performed using the

resources of the Oak Ridge Leadership

Computing Facility at the at Oak Ridge National

Laboratory, which is supported by the Office of

Science of the U.S. Department of Energy under

Contract No. DE-AC0500OR22725.


Supplemental


Problem 7: VERA Analysis of Watts

Bar 1 Hot Full Power Simulation

Purpose • First large-scale coupled multi-physics model of operating PWR reactor using

Components of VERA

• Features resolved based on dimensions and state conditions of Watts Bar Unit 1 Cycle 1 geometry for fuel, burnable absorbers, spacer grids, nozzles, core baffle

Common to both approaches: • Common input used to drive all physics codes

• Subchannel thermal-hydraulics in coolant (COBRA-TF)

• Rod-by-rod heat conduction in fuel (COBRA-TF)

• Temperature and density dependent multigroup neutron cross sections

Insilico/Denovo SPN ,SCALE/XSProc cross sections • >1M unique material (fuel/coolant/internals) regions resolved

• 17.5 hours on Titan using 18,769 cores (330,000 core-hours)

MPACT 2-D (MOC) / 1-D (nodal diffusion), subgroup cross sections • >2.8B unknowns for multi-group scalar flux

• 12 hours on Eos using 2,784 cores (33k core-hrs)

• now <4 hours on Eos using 2,784 cores (11k core-hrs)

30x reduction in core-hr requirement >3x reduction in memory

need another factor of 5-10x for overnight cycle

Fission Density Profile

in Reactor Core


Computational resources

are unprecedented.

• Compute cycles and total memory will continue to increase – Core counts will continue to double every 1.5-2 yrs.

– Next OLCF system will have 5-8x more memory than Titan

• Cost is difficult to generalize, but cost for 1 TF/s has steadily dropped... – 1997: $55M (ASCI Red)

– 2002: <$1M

– 2012: <$5k (~200 TF/s for $1M)

– 2020: $100? (~10 PF/s for $1M)

• As we discuss challenges, also consider opportunities...

http://www.extremetech.com/extreme/171678-intel-unveils-72-core-

x86-knights-landing-cpu-for-exascale-supercomputing

http://www.enterprisetech.com/2013/11/18/

ibm-embraces-nvidia-gpus-acceleration/


Let’s enjoy a bit of optimism...

• Compute cycles and total memory will continue to increase

– Core counts will continue to double every 1.5-2 yrs.

– Next OLCF system will have 5-8x more memory than Titan

• Cost is difficult to generalize, but 1 TF/s has steadily dropped

– 1997: $55M (ASCI Red)

– 2002: <$1M

– 2012: <$5k (~200 TF/s for $1M)

– 2020: $100? (~10 PF/s for $1M)

• As we discuss challenges, also think about opportunities for CASL codes

– CE Monte Carlo with 8T particles

– multiphase CFD for ¼-core

http://www.extremetech.com/extreme/171678-intel-unveils-72-core-

x86-knights-landing-cpu-for-exascale-supercomputing

htt

p:/

/ww

w.e

nte

rprisete

ch.c

om

/2013/1

1/1

8/

ibm

-em

bra

ces-n

vid

ia-g

pus-a

ccele

ration/


Similarly for coming NERSC systems...

Extracted from 10/8/2013 presentation by Jack Deslippe, NERSC User

Services titled “Preparing Your Applications for Future NERSC Systems”


NERSC-8 will use Knights Landing

http://www.anandtech.com/show/8217/intels-knights-landing-coprocessor-detailed


Effective node-level parallelism is essential.

• Future performance is mainly from node improvements. – Number of nodes is not increasing dramatically.

• 70-80% of developer time spent restructuring code – 1-2 person-years for codes to make Jaguar to Titan

transition

– required regardless of the type of processors – work is in exposing additional parallelism on the node and minimizing data movement

– tools/compilers can help, but cannot do this type of refactoring automatically

– also largely independent of threading model

– however, it pays off for other systems —refactored codes often run significantly faster even on CPU-only (Denovo 2x, CAM-SE >1.7x, S3D 2x)

• Each code team must make its own choice of threading model (see next slide)

• Abundance of flops can make previously infeasible or inefficient models and implementations viable

CPU Accelerator (GPU,Phi)

Optimized for sequential multitasking • Optimized for many

simultaneous tasks

• 10 performance per socket

• 5 more energy-efficient systems

Distilled from 2014 presentation to AASTCS2: Exascale Radio Astronomy

Jack Wells, OLCF Director of Science, ORNL


Programming models for node-level parallelism

Must design for MPI+X, where X is one of the thread options (OpenMP, OpenCL, CUDA, OpenACC, pthreads, TBB, etc.)

CUDA

TBB OpenMP

CAPS

PGI

OpenCL

pthreads

CUDA

TBB

OpenMP

CAPS OpenCL

pthreads

OpenACC X

2010 Today homogeneous

multicore

hybrid / accelerators

(GPUs, PHI, FPGA)

MPI

PGAS

PGI MPI

PGAS


Some VERA components are being

modified for multicore / hybrid

systems.

VERA

Hydra-TH COBRA-TF

Thermal-Hydraulics

Fuel Performance

Peregrine

Shift

Neutronics

Insilico

Chemistry

MAMBA Common

Input / Output

front-end & back-end

(workflow / analysis)

Trilinos

DAKOTA

MOOSE

PETSc

Solvers / Coupling /

SA / UQ

libMesh

STK

DTK

Geometry / Mesh /

Solution Transfer

ANC

Interoperability

with External

Components

Star-CCM+

RELAP5

RELAP7

MPACT

Others TBD (e.g. ANSYS)


Of the two primary solver libraries used

by VERA, Trilinos is more aggressive

about preparing for multicore / hybrid.

• In Trilinos, distributed linear algebra objects are implemented in the Epetra and Tpetra packages

– Epetra was the original implementation and is production stable (MPI only)

– Tpetra is the new implementation – templated, namespaces, STL, and builds on Kokkos (MPI + threads/vectors)

• Kokkos implements node-level threaded and vector kernels

– both sparse and dense

– interoperable with vendor-optimized libraries

– working closely with NVIDIA on CUDA-based implementations

• VERA components using Tpetra will benefit with no modifications

– however, this will be difficult to leverage in Fortran

• less familiar with multicore/hybrid efforts in PETSc

– some operator and solver-specific CUDA and OpenMP implementations have been published

Trilinos info from June 2013 Intel Software Tools Spring Technical Webinar by Mike Heroux, SNL (https://software.intel.com/en-us/articles/intel-software-tools-spring-technical-webinar-series)

VERA has been used to gain

unique insights in the AP1000

core behavior and advanced

operational features

3D core VERA Monte-Carlo

simulations tracking 1 trillion particle

histories performed on 240,000 cores

of the OLCF Titan

• A 1-year per simulation for a typical

Monte-Carlo code vs. 3-hr per

simulation thanks to VERA massively

parallel Monte-Carlo

• Simulations performed as part of a 60

Million Core hours Early Science Award

– arguably the largest computational

effort in the history of commercial

nuclear energy

MB AO AO

MB MA

Predicted power distribution for an AP1000 Monte Carlo simulation

with multiple control rod banks inserted (AO, MA, and MB in the

picture). This simulation emulates the bank configuration occurring as part

of the AP1000 MSHIM™ operational strategy, where control rod banks are

used in synergy to provide flexible load following capabilities and reactor

operation while minimizing soluble boron variations and related effluents.

High-Fidelity 3D Core

Power Prediction

Recognitions for the Westinghouse VERA

Test Stand

• Mission-oriented team

effort received IDC’s HPC

Innovation Excellence

Award for “innovative HPC

applications with

significant benefits to

industry and general

public”

– Early Science Award 60-

Million core hour allocation on

OLCF Titan

– Nuclear Engineering

International invited article,

May 2014 Issue

Several members of the

Westinghouse-CASL team proudly

display the HPC award (from L to

R): Jess Gehin, Oak Ridge National

Laboratory; Fausto Franceschini,

Westinghouse Electric Company;

Andrew Godfrey, Oak Ridge

National Laboratory; and John

Turner, Oak Ridge National

Laboratory. The team receiving the

award also includes Tom Evans

(ORNL) and the SHIFT development

team

• Gained unique

insights in the

AP1000 first core and

operational features

• Reinforced

confidence in

Westinghouse in-

house predictions

• Performed

breakthrough

simulations

supporting advanced

plant operation

Other VERA Applications at Westinghouse

After successful application to the AP1000 PWR, VERA is currently being

applied for the analysis of other challenging cores by Westinghouse

Reg. # - # of Pyrex Rods

R3 00 R3 00 R3 00

R3 00 R3 00 R2 00 R2 08 R2 00 R3 00 R3 00

R3 00 R3 08 R1 00 R2 12 R1 00 R2 12 R1 00 R3 08 R3 00

R3 00 R3 08 R1 00 R2 12 R1 00 R2 16 R1 00 R2 12 R1 00 R3 08 R3 00

R3 00 R1 00 R2 12 R1 00 R2 16 R1 00 R2 16 R1 00 R2 12 R1 00 R3 00

R3 00 R2 00 R2 12 R1 00 R2 16 R1 00 R1 08 R1 00 R2 16 R1 00 R2 12 R2 00 R3 00

R3 00 R2 08 R1 00 R2 16 R1 00 R1 08 R1 00 R1 08 R1 00 R2 16 R1 00 R2 08 R3 00

R3 00 R2 00 R2 12 R1 00 R2 16 R1 00 R1 08 R1 00 R2 16 R1 00 R2 12 R2 00 R3 00

R3 00 R1 00 R2 12 R1 00 R2 16 R1 00 R2 16 R1 00 R2 12 R1 00 R3 00

R3 00 R3 08 R1 00 R2 12 R1 00 R2 16 R1 00 R2 12 R1 00 R3 08 R3 00

R3 00 R3 08 R1 00 R2 12 R1 00 R2 12 R1 00 R3 08 R3 00

R3 00 R3 00 R2 00 R2 08 R2 00 R3 00 R3 00

R3 00 R3 00 R3 00

P P P P

GT GT GT GT GT GT

P GT GT P P P P P

IT GT IT GT

GT GT

P GT GT P P P P P

GT GT GT GT GT GT

P P P P

P P

P GT P

P P P P

IT GT

GT

P P P P

P GT P

P P

The example shows the application of VERA to the 2-loop Krsko reactor. The core loading pattern is shown on the left, the

16x16 fuel lattice is shown in the middle with the detailed (sub-pin) fuel representation from VERA-CS shown in the blown up

views. This level of detail in the core physics characterization is not available in the current industrial core simulators

3D Power Distribution for Krsko Core

VERA Monte Carlo

(Shift)

VERA Deterministic

(VERA-CS)

k ∆k RMS ∆P Max ∆P Hot Spot

3D Core All-

Rods-Out 1.00051 -72 0.5% 1.9% -0.1%

Delta power

(VERA-CS vs.

Shift) Showing

excellent

agreement

between

VERA-CS and

reference Shift

Monte-Carlo

results

3D Power

Distribution

(Shift) Showing well-

behaved low-

power peaks

core power

profile and the

effect of

burnable

poisons

Exceptionally Low 3D-Core Power

Uncertainty (Shift) Uncertainty, inherent in the stochastic approach

to neutron transport, can be reduced by

increasing the number of particle histories. Shift

calculations with up to 2x1012 particle histories

have been performed on 240,000 cores of Titan.

3D core results from VERA-CS showing outstanding agreement with the

Monte-Carlo prediction. The Root-Mean-Square (RMS) delta power

between the two codes calculated on a 1.5 million cells geometry is 0.5%,

with a 0.1% difference at the “hot spot” (largest power location). The hot-

spot calculation uncertainty in current simulators is 5%.