3SAQS Technical WorkshopApril 30th, 2015

Data WarehouseCurrent and Future Development

Intermountain West Data Warehouse

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

IWDW Overview

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

IWDW Overview

DW Development Update• Project Management– Q1 2015

• Data Requests; Inventory Management; Data Delivery• Documentation

• Data & Information Management– Q2 2015 through 2016

• Visualization tool development (modeling I/O & analysis)• Modeling platform review• Expanded documentation and “easy-to-reference” information

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

Data Request Summary – pg. 1Request Requestor Data

Supporting Docs

RequestSubmitted

3-State Steward

Action

Direct Interest Studies for WSAQS Cooperators

Methane/VOC Control Sensitivity Study

Tammy Thompson CIRA

2008 Emissions Scope of Work Yes

Emissions Inventory Data Summary for 2008

Tammy Thompson CIRA

2008 Emissions None Yes Mike Barna Filled 4/24/2015

Source Apportionment Modeling for Nitrogen in Greater Yellowstone Area (GYA)

Tammy Thompson CIRA

2011 Emissions(unreleased)

Project Plan Yes Mike Barna

National Park AQRVTodd McDonnell E&S Environmental Chemistry

2011 Emissions(unreleased)

None Yes

Routes to Sustainability for Natural Gas Development and Water and Air Resources in the Rocky Mountain Region - Sensitivity Analysis

Shannon Capps University of Colorado, Boulder

2008bBase year Platform (partial)

Modeling Protocol and Timeline

YesKevin Briggs

Filled 4/24/2015

Comparison of source contributions to nitrogen deposition at Class 1&2 areas in the Western U.S.

Tammy Thompson CIRA

2008b Emissions and CAMx inputs

None No

WRAP TSS Emission Tool Updates – 2011a emissions displays

Cassie Archuleta Air Resource Specialists

2011a Emissions (unreleased)

None No Tom Moore

Inventory--categories and methodology consistent with previous WRAP inventories for regional haze from 2004 and 2008 (reasonable progress)

Daniel Bon CDPHE2008b Base year Platform

None (may be taken care under Archuleta request above)

Yes Tom Moore

Will fulfill via TSS after update w/2011 emissions

Data Request Summary – pg. 2

Request Requestor DataSupporting

DocsRequest

Submitted3-State

StewardAction

NEPA Studies

Hiawatha Regional Energy Development Project (HREDP) Environmental Impact Statement

Ralph Morris ENVIRON

2008b NEPA Platform

None No Charis Tuers

Converse County Development Environmental Impact Statement (EIS)

Chao-Jung Chien AECOM

2008b NEPAPlatform

None YesRyan McCammon

Fulfilled3/18/2015

Outside Requests (applicability to WSAQS Cooperators not clear)

Colorado Oil and Gas Sensitivity Analyses

James Wilkinson Golder Associates Inc.

2008b NEPA Platform

None Yes

Update of a USFS Fire handbook Suresh Ratnam University of Southern California

Base2020_08a None No

Improving Ozone modeling in Uintah Basin

Huy Tran Utah State University

2008b Emissions and utilities

NoneYes incomplete

Analysis of FRAPPÉ and DISCOVER-AQ 2014 campaigns

Gabriele Pfister NCAR

2011a Emissions and utilities(unreleased)

None Yes

Responding to Data Requests

• Process needs for operating the Data Warehouse noted so far:– the identities, roles, and responsibilities of

cooperating agency “stewards” need to be clarified– the necessary level of documentation and

applicability to the overall goals of the cooperators; and

– the form of a “release” of a platform and its component data, including the associated use recommendation.

Pending Project Proposals with no data requests

Information or Data Request Proposal

Requestor Start Date DataRequest

SubmittedFulfillment Approach Action

Emissions QA 2011 Base A Gail Tonnesen EPA Region 8

10/1/2014 None No Internal project report completed  

Ozone spatial plots for 2011 MPE Gail Tonnesen EPA Region 8

10/16/2014 None NoDisplay capability on DW in

development 

PEM integration with SMOKE/CMAQ

Private consultant 11/11/2014 None No Cancelled  

2011a Modeling Sensitivities Zac Adelman UNC-IE 9/1/2014 None NoConducted as internal DW modeling study analysis

 

PMDETAIL Ralph Morris ENVIRON

12/1/2014 None NoPending, will utilize files from DW and PMDETAIL project

 

2020 Future Case Sensitivity Mike Barna National Park Service, Air Resource Division

1/23/2015 None No Pending  

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

Image Browser• http://views.cira.colostate.edu/tsdw/Tools/ImageBrowser.aspx?pathid=TsdwPlotsRoot

• Displays large image sets delivered by data provider

• Possible Next Steps– Improve navigation (fewer mouse clicks)– Display meta data– Browse multiple image sets side by side

Model-To-Obs Tool• http://views.cira.colostate.edu/TSDW/modeling/modelobscomparison_iwdw.aspx

• Compare model values to observed values by network

• Possible Next Steps– Platform selection– Additional filtering capabilities

Emissions Review Tool

• http://views.cira.colostate.edu/tsdw/Emissions/ReviewTool.aspx

• View modeled emissions by source category, state, county, and SCC

• Possible Next steps– Refactor data base to correctly label and

aggregate parameters and source categories– Expand SCC visualization capabilities

Dynamic Geospatial Model Visualization• http://views.cira.colostate.edu/tsdw_Dev/Maps/ModelDataGIS.aspx• Displays model output data with overlaid monitor data

Dynamic Geospatial Model Visualization• ArcSDE (Spatial Database Engine) with SQL Server– Stores/Serves rasters of model data– Stores/Serves raster attributes– Data can be accessed remotely by ArcGIS users

• ArcServer– Provides web services which render data stored in

SDE– Web Services consumed by web applications such as

the data warehouse

Dynamic Geospatial Model Visualization

• Advantages– Map Projection– Uniform color scale– Monitor data overlay– Ease of combining multiple features– Sharing data with other GIS users

Dynamic Geospatial Model Visualization

• Disadvantages– More Pre-processing– Data duplication– Infrastructure heavy (ArcServer, ArcSDE)

Dynamic Geospatial Model Visualization

• Possible Next steps– Parameterize, platform, species, layer,

time window– Additional Monitor data– Additional features• Cities• Roads

– Automate preprocessing, execute on demand

Source Apportionment Tool• http://ims.environcorp.com/aqms_test• Developments/Status

• Received prototype from Environ• Database up and running

• Possible Next steps• Deploy prototype user interface on CIRA

systems• Integrate user interface with main warehouse

system• Integrate database with main warehouse DB• Automate data ingestion

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015

Data Request Form• Automated FTP fulfillment– One click FTP user creation and data access– Create/Update FTP user account– Enable FTP access to requested data without

duplication of data– Two Successful deliveries via FTP– Image Browser uses FTP fulfillment to serve

images

Data Request Form• http://views.cira.colostate.edu/tsdw_dev/DataRequest/PlatformSelection.aspx

• Component Selection– Flattened tree– More descriptive information– Alternative to category tree

Data Request Form• Possible Next Steps– Integrate component selection with request form– Possibly separate project proposal from data

request– Continue to evolve approval and stewardship

policies and procedures– Integrate data release status and/or evaluation

level– Notification email clean-up

Category Management• Create/Remove/Edit

– Data Categories– File Items in category– Meta Data

• Browse file system• Data Management Reports

– Missing data• Possible Next steps

– Additional reporting as needed– Better display of category/file meta data parameters– File/Category history

Inventory Management• MD5 checksums stored in relational database• History of file meta data

– file size– mod time– checksum

• Data Recovery tool• Next Steps

– Incorporate detailed meta data scraper– Automation of Checksum computation– Reporting

• Report changes to data requestors• File level data status

Forum & Email Notifications• http://views.cira.colostate.edu/tsdw/Forum/

Outline• Update - DW Overview & Development • Data Request Status• Visualization tools

– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site

• Functionality• Data Request Form

• Prototype data selection form• Category Management

• Inventory Management• Forum & email notifications

• Wrap-up and action items

3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015