AgMIP SSA MeetingAccra, Ghana

12 September, 2012

Importing and translating crop model data

AgMIP IT ProductsWebsites

Public site www.agmip.org (under renovation) Research site research.agmip.org AgMIP toolshed tools.agmip.org (under construction)

Tools Climate tools Crop model data translation tools Web and desktop interfaces for modeling tools Open source repositories github.org/agmip

Databases Crop Experiment / Sentinel site (operational) Crop model outputs (operational) Crop scenarios (conceptual) Gridded data (conceptual)

AgMIP harmonized databases

Model data stored in a consistent format

Filters allow import of data from various model-ready files and other data sources

Translators allow export of data to various model-ready formats

Web-based data searchhttp://test.agmip.org/ace-ui2/#mappage

Guide for Regional Integrated Assessments HandbookItem 3: Assemble Existing Data from

Experiments and Calibrate Crop ModelsItem 5: Assemble Data and Calibrate Crop

Models for Regional Yield Simulations and Assessments

Item 10: Archive Data and Analyses Results for Integrated Assessments

Types of dataData rich field experiments for calibration

of cropping system parameterssufficiently complete to parameterize crop

modelsSurvey data – evaluation of baseline vs

future conditionsmay be incompleteassumptions based on local knowledge of

agricultural practices

Steps to data preparation and translation

1. Collect raw data2. Using spreadsheet templates, transform the

data to harmonized units and variables – weather, soils, and field / management / metadata

3. Save data in csv format files4. QuadUI – desktop application for data

translation to DSSAT and APSIM formats 5. View assumptions and errors (repeat steps 2-

5 as needed)6. Run models7. Collect output data in ACMO.csv format for

input to economic models

Step 1 – Collect raw dataMachakos data as an example (Survey.xls)

Step 2 – Prepare crop model data3 ways to input data

DSSAT format (use Xbuild and other DSSAT tools to prepare crop model files). These can be translated directly into AgMIP harmonized format.

Use spreadsheet templates to fill in data with proper ICASA variable names and units.multiple templates will allow different kinds of data

User Interface for data entry – under development

Step 2 – Survey data template

Step 2 – Survey data template (continued)

Excel spreadsheet for calculations, conversions, date formatsOne line per site for metadata, field, and

management dataseparate csv file for soil dataseparate csv file for weather dataseparate csv file for regional assumed

management data (initial soil conditions, fertilizer rates, planting conditions)

See sample template (Machakos_import_2.5.xlsx)

ICASA Master Variable ListHarmonized database uses ICASA Master

Variable List to interpret data definitions and units.

See ICASA_Master_Variables_10Sep2012.xlsx or online (always up-to-date) at https://docs.google.com/spreadsheet/pub?key=0AiUhvGN3ZVgXdGJud3lDNThFSnlGWGlkTjgxWXRwbWc&output=htmlliving document – go to web version for the

latestadding variables to spreadsheet templates:

use code_display column from ICASA list

Step 3 - Save csv format filesSeparate csv file for each data type:

soilweathermanagement / field / metadata (initial

conditions, observed data)regional assumptions

Beware of date formats when you re-open a csv file with Excel!

Zip all csv files together for import with QuadUI

Step 4 - QuaDUIQuick and Dirty User InterfaceTemporary desktop application for

conversion of spreadsheet templates or DSSAT-format files to AgMIP harmonized format, APSIM or DSSAT formats.

Near future: web-based application to upload and archive data, search and query, download in multiple formats.

Step 4 – QuaDUI (continued)Open by double-clicking the

quadui.bat file and selecting the zip file and location for output files.

Step 4 – QuaDUI (continued)Select format(s) for conversion• DSSAT• APSIM• JSON

Currently working on additional translators for:• STICS• Aquacrop• InfoCrop• EPIC• ORYZA

• CropSyst• Salus• WOFOST• RZWQM• Need to add SARRAH

translator!• Hit “Convert” button

Step 5 – QuaDUI troubleshootingLook at crop model files for notes on

assumptions made by the translatorsLook at QuadUI log file (where the QuadUI

jar file is located) for notes on invalid variables

Application requires that headers are spelled exactly as in code_display of ICASA Master Variable List (not case-sensitive)

Variables must appear in correct file (e.g., no weather data in soils file)

Step 6 – Run ModelsDSSAT – bat file provided which will run the

experiments created.APSIM – use APSIM UI

Step 7 - Crop model outputsMachakos_ACMO.csv file – format for

harmonization of output from crop models.

Spreadsheet template for sentinel site dataUFGA8201_MZX_trt1.xlsx

Legend:

Symbol Location Description! Column 1 Comment

#Column 1, headers

Summary data variable names

%Column 1, headers

Series data variable names (time series or soil layer data)

Unique id (1, 2, A, B, …)

First field of data

Relational data requires unique id to show relationships between summary and series sections of data

*First field of data

Complete data (one line per experiment)