New Tools for a Better Air Quality System Data Flow EN 2012 May 31, 2012

New Tools for a BetterAir Quality System Data Flow

EN 2012May 31, 2012

• Air Quality System (AQS) flow automation enhancements

• Exchange Network Services Center• AQS Plugins for OpenNode2 and EN-Node• Q&A / Discussion

Topics to Cover

• AQS submissions via a Node or Node Client used to require extra manual steps

• More difficult to access AQS submission logs and reports

• Most users preferred to use EPA’s non-Exchange Network interface called CDX-Web

Jumping Through Hoops

Can I interest you in an AQS data flow?

• Today, files submitted through the Exchange Network are automatically processed by AQS

• Exchange Network Nodes can download the submission log and reports

• New tools for submitting AQS data via the Network• Support for AQS submissions through CDX Web is being

discontinued

Fewer Flaming Hoops…More Puppies

Perhaps this is more to your liking?

• AQS data flow (v2.2a) now uses the Exchange Network Header

• The Header file contains information about the submission data file– Security Token– Exchange Network and AQS User IDs– Payload type: 1) XML or 2) FLAT– AQS Load Options

• Final Processing Step• Stop on Error

How Does it Work?

The submitter defines the desired level of automation:

1. Stage: Submission file is uploaded to the AQS server and verified to be a valid AQS file type, but with no subsequent processing.

2. Load: The file is loaded into the AQS database and its contents validated. Sample measurements are left at the status “Pre-production”; all other data types (Site information, Monitor information, and QA information) are set to the status “Production”.

3. Post: All valid sample measurement data for the file is set to the status “Production”.

– Post is the default setting

AQS Load Options

• Exchange Network Services Centerhttps://enservices.epa.gov/login.aspx –Automatically creates EN Header–Supports manual submissions to AQS–Replacing CDX-Web for AQS submissions

New Submission Option #1

Node Plugins for OpenNode2 and EN-Node

•Customized for easy integration with 2 widely used air monitoring systems

– OpenNode2 - .NET – AirVision and DR-DAS– OpenNode2 - Java – DR-DAS– EN-Node - .NET – AirVision– EN-Node – Java - AirVision

•Automatically create EN Header•Support submissions to AQS and data publishing•In testing. Production-ready versions likely available in June for free download from EN website.

New Submission Option #2

Show and Tell

• Exchange Network Services Center

• EN-Node Plugin– AirVision

• OpenNode2 Plugin– AirVision & DR-DAS

EN-Node AirVision Plugin

• AirVision Plugin Features 3 Operations Submit AQS data to CDX Publish AQS data (Solicit) Publish AQS data (Query)

1

2

3

EN-NodeAQS

Monitor

AirVision

AirVisionPlugin

ENDS

CDXEPA AQS

2

1

3

Submit

SolicitQueryEN Browser

• URL: http://code.google.com/p/en-node2/downloads/list• File: AirVisionPlugIn.zip

Installation File & Contents

File Name Purpose Installation1. Six (6) DLL files Plugin software Copy to EN-Node

file directory

2. InsertAQSHeaderTemplate.sql DB scripts to install AirVision Plugin in EN-Node DB

Execute scripts

3. Task_SubmitAQSXMLData_DataWizardConfig.xml

Submit AQS data to CDX

Copy to EN-Node file directory

4. Query_GetAQSXMLData_DataWizardConfig.xml Publish AQS data (Query)

Copy to EN-Node file directory

5. Solicit_GetAQSXMLData_DataWizardConfig.xml Publish AQS data (Solicit)

Copy to EN-Node file directory

• What is Configuration?– Business logics used by EN-Node to complete the data flow – Config. data saved in an external BEPL file (Business Process Execution Language) – Each Operation has its own BEPL file – Configuration tools

• EN-Node Admin Console• BEPL XML file provided with the Plugin installation file

• How to Configure:Step 1: Create a Domain (AirVision)Step 2: Register 3 operations under AirVision1. Submit AQS data to EPA2. Publish AQS (Query)3. Publish AQS (Solicit)

Step 3: Test PluginStep 4: Publish AQS data at ENDS or EN Browser

Configure AirVision Plugin

Plugin Configuration Options

Data Needed to Configure AirVision Plugin

Parameter Required? Value Example

EPAURL Yes URL of the EPA Endpoint in which to submit data

https://testcdxnode64.epacdxnode.net/cdx-enws20/services/NetworkNode2

NAASUserID Yes User Name EnfotechNAASUserPassword Yes Password enfopassword

AirVisionURL Yes URL of the AirVision Node http://localserver/AirVision.WebService

FileName No File Name Airvision_Submission

AQSXMLSchemaVersion No Version Number 2.2

CompressPayload No True/False TrueDurationCode No Number in hours 1

StartTime Yes Date in the format YYYY-MM-DD 2011-10-01

EndTime Yes Date in the format YYYY-MM-DD 2011-12-31

SendOnlyQAData No True/False FalseSendRATransactions No True/False TrueSendRBTransactions No True/False TrueSendRDTransactions No True/False TrueSendRPTransactions No True/False True AuthorName No Author Name John SmithOrganizationName No Organization Name Org1 Inc.DocumentTitle No Document Title AQS SampleAQSUserID No AQS User Identifier EnfotechAQSScreeningGroup No AQS Screening Group EnfotechAQSSchemaVersion No Schema version of document 2.0

AQSPayloadTypeProperty No Format in which data is presented XML

AQSLoadOptions No Load/Operation LoadPayloadOperation No Load/Operation LoadAgencyCode No Agency Identification Code 001SiteCode No Site Identification Code 210ParameterCode No Parameter Code 46324

• AirVision End Point• EPA AQS End Point• Organization Name• NAAS ID & PWD• Plugin file location

Minimum Data Needed

Before Configuration…

Important!• Open 3 DataWizardCinfig.xml files with text editor• Enter Plugin file location

• Create AirVision domain – Choose web services (Submit, Download, Query, Solicit, Notify) allowed for

the domain.

• Establish “Administrator” accounts to setup/modify this domain.

Step 1: Create Domain

• Register Operation(s)1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml

Step 2: Register Operation(s)

Define•EPA Endpoint URL•Validation rule (if any)•Data viewing stylesheet (if any)

Define•Task schedule

• Register Operation(s)1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml

Step 2: Register Operation(s)

1. Test Submit AQS to EPA– Test: Enter Start Date and End Date to a

date earlier than the current date will execute task after the user clicks save.

– Track submission status CDX will return “Completed” or “Failed” View Download Data Viewer

Step 3: Test Operation(s)

Authentication

Query Request

2. Test AQS Data Query– Authenticate – Input Start Date and End Date for AQS

monitoring date– Query will return data in XML

Step 3: Test Operation(s)

Authentication

Solicit Request

Get Status

Call Download

3. Test AQS Solicit– Authenticate – Solicit (Input AQS monitoring date)– Get Status– Download AQS data (in XML)

Step 3: Test Operation(s)

• Manual data input to ENDS:– Test: https://ends2.epacdxnode.net– Production: https://ends2.epa.gov

• XML Data Upload to ENDS:– Auto Pull from ENDS: GetServices – Manual XML Upload to ENDS

Step 4: Publish AQS Data at ENDS/ENB

ENDS: Service/DEDL Registration

Option A: Manual

Option B: Automated

(Published via GetServices)

(Node submits DEDL to EPA)DEDL XML

ENDS XML

Node Admin

Data Provider

Exchange Network

Node

EPA Service Registry (ENDS)

(Automate your Node to transfer registration info to EPA’s Registry)

Stores registry of all services, input parameters, pick lists, etc.

B1

B2

Node Admin Registers Services (ENDS)

Node Admin Registers DEDL

A1

A2

EN Browser

• Test: https://ends2.epacdxnode.net/Node2WS.svc • Production: https://ends2.epa.gov/Node2WS.svc • Dataflow: ENDS_v20• Operation: Service Refresh

• Implement Only the Operation(s) you need• Minimum data you need to configure Plugin

– AirVision End Point– EPA AQS End Point– Organization Name– NAAS ID & PWD– Plugin file location

• Useful resources– Plugin: URL: http://code.google.com/p/en-node2/downloads/list– Data Publication Guide:

http://www.enbrowser.net/enbrowser/ClientBin/HowToPublishData.pdf – ENDS & FCD: http://www.exchangenetwork.net/2011/07/15/exchange-

network-discovery-service-ends/

• Support– enfoTech, Lijie Xu: lijie_xu@enfotech.com

Useful Tips for EN-Node AirVision Plugin

Show and Tell:OpenNode2 AQS v2.2 Plugin

• http://code.google.com/p/opennode2• .NET OpenNode2

– AirVision – DrDAS

• Java OpenNode2– DrDAS

OpenNode2 AQS v2.2a Plugins

OpenNode2 AQS Submit Workflow

OpenNode2

AirVisionOr

DrDASWeb

Service

AirVisionOr

DrDASWeb

Service

EPA CDX

Node

EPA CDX

Node

1. Request Data

2. Return Data

3. Submit Data

1. On a set schedule (or manually triggered), request data from AirVision or DrDAS• Parameters set to filter data request

2. AQS XML data is returned to OpenNode23. Relay the data to EPA CDX/AQS system

• Optionally, header is added• Optionally, notifications can be sent

Get Status/Download Workflow

OpenNode2

EPA CDXEPA CDX

Status Response1. Get Status

Files2. Download Request

3. Notifications

1. On a set schedule (or manually triggered), check status of outstanding submissions

2. If CDX returns “completed” or “failed”, download processing reports from CDX• Do nothing if status is “pending”

3. Optionally, send notification emails to agency staff1. Mail sent for all submissions where status

changes from “pending” to “completed” or “failed”

• “Publishing” makes data available for any authorized partner to come and query from your node– Exposed as Query and/or Solicit services

• No additional setup needed to make these services available from OpenNode2– Setting up the services in Node Admin tool makes

them available with a simple checkbox

Publishing AQS Data from OpenNode2

Setting up the AQS Plugin/Flow

• Very similar to setting up any other OpenNode2 data flow

• Using the Node Admin interface:1. Create the AQS flow2. Upload the plugin (contains all workflow logic)

3. Configure the Services (exposes plugin logic)a) Service 1: Get data from AirVision/DrDAS for submission to EPAb) Service 2: Check status of submission and download results

4. Create the Schedules (automates workflow)

… and that’s it for the OpenNode2 portion

OpenNode2 AQS Service Setup

Flow 1Service 1Service 2

Flow 2Service 1Service 2

View in Node Admin of AQS flow with services configured:

AQS “GetXmlData” Service Detail

Note: AirVision settings shown DrDAS settings very similar

AirVision Schedule Detail

DrDAS Schedule Detail

Parameter Value

FileGenerationPurposeCode “AQS”

BeginDate Desired monitoring period start date

BeginTime “00:00”

EndDate Desired monitoring period end date

EndTime “23:59”

DataValidityCode “V”

DataApprovalIndicator “Y”

IncludeMonitorDetails “N”

IncludeEventData “FALSE”

SchemaVersion “2.2”

Kurt Rakouskas301.531.5186kurt@exchangenetwork.net

Questions?