Jinwon Kim and Paul Ramirezand

RCMES Science and IT Teams led byDuane Waliser (JPL), Science Leader

Chris Mattmann (JPL), IT Leader

Regional Climate Model Evaluation System (RCMES):

Combining Observations & IT to Establish Core Climate Model Assessment Capabilities

NCPP Workshop, Boulder, Colorado, August 2013

RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities

• Model evaluation is a key step in assessing climate model fidelity and in turn the (un)certainty of climate change impacts.

• Systematic experimentation and evaluation of GCMs have been undertaken for some time (e.g., AMIP, CMIP, CFMIP), while that for RCMs (e.g., NARCCAP, CORDEX) being more recent and less mature.

• NASA can provide critical and unique observational data to facilitate RCM evaluations and thus make key contributions to impact assessment process, e.g., the National Climate Assessment (NCA).

• RCMES was developed to facilitate model evaluations via easy access to observational data, especially from satellite remote sensing, and software tools for calculating and visualizing evaluation metrics.

Raw Data RCMED(Regional Climate Model Evaluation

Database)

RCMET(Regional Climate Model Evaluation

Tool)

Metadata

Data TableData TableData TableData TableData TableData TableCommon Format,

Native grid,Efficient

architecture

Cloud Database

Extractor for

various data

formats

TRMM

MODIS

AIRS

CERES

ETC

Soil moistur

e

Extract OBS data

Extract model data

Userinput

Regridder(Put the OBS & model

data on the same time/space grid)

AnalyzerCalculate evaluation

metrics & assessment model input data

Visualizer(Plot the metrics)

URL

Users' own

software for

specific evaluation

and analysis

Data extractor(Binary or netCDF)

Model dataOther Data Centers

(ESGF, DAAC, ExArch Network)

Assessment

modeling

RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities

(http://rcmes.jpl.nasa.gov; Powered by Apache Open Climate Workbench)

• RCMES is flexible and open source software• Can utilize multiple (distributed) data sources.• Easily transferrable to local platforms (Linux, Mac-OS)• Open source (via Apache)

RCMED Datasets: Satellite retrievals, Surface analysis, Reanalysis, Assimilations* MODIS (satellite cloud fraction): [daily 2000 – 2010]

* TRMM (satellite precipitation): 3B42 [daily 1998– 2010]

* AIRS (satellite surface + T & q profiles) [daily 2002 – 2010]

* CERES and GEWEX-SRB radiation – Surface and Top of the atmosphere

* NCEP CPC Rain Gauge analysis (gridded precipitation): [daily 1948 – 2010]

* CRU TS 3.1: precipitation, Tavg, Tmax, Tmin [monthly means, 1901 – 2006]

* University of Delaware precipitation and temperature analysis

* Snow Water Equivalent over Sierra Nevada Mts [monthly 2000-2010]

* NASA MERRA Land Surface Assimilation & pressure-level data [daily, 1979-2011]

* ERA-Interim (reanalysis): [daily 1989 – 2010]

* AVISO sea-level height [1992-2010]

* (In progress) CloudSat atmospheric ice and liquid, Satellite-based snow (Himalayas), ISCCP cloud fraction, Fine-scale SST, etc.

RCMET Metrics:* Bias (e.g. seasonal means or variance)

* RMS error (e.g. interannual variability)

* Anomaly Correlation (spatial patterns of variability)

* PDFs (likelihoods, extremes and their changes)

* Taylor Plots & Portrait Diagrams (overall model performance)

* Statistical Tests

* User-defined regions (e.g. water shed, desert, sea, political)

• Datasets and metrics are continuously updated.

Not Illustrated HereArctic & Antarctic Domains

RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities

RCMES is being (will be) used for model evaluation in a number of regional climate experiment regions via worldwide collaboration

• N. America – NARCCAP• WCRP CORDEX Regions

• Ongoing/tested: Africa, South Asia, Middle East – N. Africa, Australia• Under arrangement: Caribbean, South America, East Asia, Arctic

NARCCAP Multi-decadal Hindcast Evaluation: Surface Insolation

Considerable biases with large-scale spatial structures exist in surface insolation fields.

Figure. Surface insolation biases against the GEWEX SRB data

Model performance varies according to regions and seasons.

Figure. Evaluation of the mean & interannual variability in various regions within the conterminous US.

Bias (% SRB) Interannual variability (% SRB)

Kim et al., 2013, J. Climate, 26, 5698-5715.

RCMES' capability to handle multiple model- and/or/both observation datasets allows to visualize model performance as well as uncertainties in the model evaluation due to observational data.

Figure. Evaluation of the annual-mean climatology from multiple RCMs and observations against the equal-weight multi-observation ensemble. The figure shows that there exist noticeable uncertainties among widely used observations.

Model Evaluation and Observational Uncertainties

Kim et al., 2013, J. Climate, 26, 5698-5715.

Observational UncertaintyTRMM, CPC, CRU, UDEL, GPCP

Construct watershed-mean met data for hydrology model

*Transferring the gridded climate model data onto a watershed is the first step in assessing water resources using a bulk hydrology model that runs on watershed-mean met data.

1. Overlay model grid over the watershed area.2. Calculate the percentage of each grid box contained within the watershed

area. This is the weighting factor for calculating the area-mean meteorological data.

3. Calculate the watershed-mean value of a variable P using the weights

The shaded area is the 0.5o-resoln RCM grid boxes that are entirely or partially included in the basin.

The Sacramento River basin

Map the watershed area onto the RCM domain (Kim et al. 2000, J. Hydromet.)

Area mapping

Figure. Calculation of area-mean data for an irregularly-shaped watershed from gridded climate model data.

Evaluation of extreme eventsAnalysis of long-tailed temperature PDFs in NARR and NARCCAP RCMs:

Loikith et al., 2013: Geophys. Res. Lett., in press.

• Variance/skewness of daily temperature are related with weather extremes.

• These PDF properties can also be related with regional processes• e.g., Positive tail in the LA basin is related with the Santa Ana winds

• Utilize these PDF characteristics to evaluate RCMs in simulating extreme events

*RCMES can be also used to evaluate multiple GCMs for various regions using multiple observation data.

*Can be useful, for example, in identifying GCMs most suitable for downscaling.

Fig. July precipitation bias: CMIP5 present-day Ens & NARCCAP RCM Ens

CMIP5 GCMs - Observation NARCCAP RCMs - Observation

Regional Evaluation of GCMs

Summary• The Regional Climate Model Evaluation System (RCMES) was developed to

facilitate model evaluation via easy access to key observational datasets especially from satellite remote sensing data sets.

• RCMES is being used in evaluation studies for several regional climate experiments including NARCCAP and multiple CORDEX domains.

• RCMES includes calculation of a number of metrics for model evaluation.

• RCMES is continuously improved with additional datasets, more efficient database scheme, and evaluation metrics.

• RCMES is useful for a wide range of users:• Can be obtained and installed as a Virtual Machine image

• http://rcmes.jpl.nasa.gov/training/downloads• Deployable to user laptops and server machines with ease• Source Code at Apache Open Climate Workbench

• http://climate.incubator.apache.org/• Command line interface • Browser based interface• API to construct your own evaluations

Ongoing and Future Efforts• Links to ESGF

• Enable users to import data directly from ESGF for processing.

• Made-to-order system• Distribution of RCMES in a virtual machine package as specified by users.

• Link climate model data with assessment models• Preparation of met forcing data from climate model data, bias correction of met

forcing data for specific assessment models.

• Performance-based variable weighting model ensemble construction

• Data processing and metrics• Handling of very large datasets.

• Develop metrics for evaluating PDF characteristics (e.g., variance, skewness)

• Cluster analysis

• Collaborative Development through Apache Open Climate Workbench• Open source effort based on a well known open source software community

and team expertise at Apache

RCMES: Evaluation of Multi-model Hindcast in the CORDEX Africajkim@atmos.ucla.edu; duane.e.waliser@jpl.nasa.gov; chris.a.mattmann@jpl.nasa.gov

Terrain of the evaluation domain and the annual precipitation climatology Evaluation of the spatial variability of the annual mean precipitation

RMSE (% Annual mean) Correlation coefficients

Evaluation of the simulated precipitation annual cycle

Kim, J., D. Waliser, C. Mattmann, C. Goodale, A. Hart, P. Zimdars, D. Crichton, C. Jones, G. Nikulin, B. Hewitson, C. Jack, C. Lennard, and A. Favre, 2013: Evaluation of the CORDEX-Africa multi-RCM hindcast: systematic model errors. Clim. Dyns, DOI 10.1007/s00382-013-1751-7.Project supported by:NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002), American Recovery and Reinvestment Act.NSF: ExArch (1125798), EaSM (2011-67004-30224)

RCMES: Precipitation Evaluation of Multi-model Hindcast in the CORDEX South Asia – Indian subcontinent

Indian subdomain

The hindcast and evaluation domain

Hindcast domain

% of Observation Ensemble

Uncertainties in the Observations

mm/dayObservations

RCMs

Model biases in simulating spatial variations in the annual-mean climatology

R01R04R05R06

R01 R02

R03 R04

Annual cycle evaluation for the northern mountainous region

Supported by: NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002); NSF: ExArch (1125798)

Global Climate Projections

RegionalDownscaling

DecisionSupport

RCMES – Regional Climate Model Evaluation System

Sponsor Acknowledgements• NSF G8 Initiative: ExArch Project• NASA: National Climate Assessment (NCA)• NASA: Adv. Information. Systems Technology (AIST)• NASA: Comp. Modeling Alg. & Cyberinfras. (CMAC)

Selected Publications• Crichton et al., IEEE Software, 29, 63-71.

• Hart et al., ICSE 2011 Workshop Software Engineering for Cloud Computing - SECLOUD, Honolulu, HI, May 2011, 43-49, ISBN: 978-1-4503-0582-2, doi: 10.1145/1985500.1985508.

• Kim et al., 2013a, J. Climate, 26, 5698-5715.

• Kim et al., 2013b, Climate Dynamics, DOI 10.1007/s00382-013-1751-7.

• Loikith et al.,2013, Geophys. Res. Lett., in press.

• Mattmann et al., 2013, Earth Sci. Informatics, doi 10.1007/s12145-013-0126-2.

• Waliser et al., 2012, Tech. input for consideration to the 2013 Us National Climate Assessment, 19pp.

• Whitehall et al., 2012, WMO Bulletin, 61, 29-34.plus a number of IT papers and conference papers

http://rcmes.jpl.nasa.gov/