LEADS/EMS SYSTEM ADMINISTRATION

2 - 1 IPS MeteoStarFebruary 22, 2007

LEADS/EMSLEADS/EMS

SYSTEM SYSTEM ADMINISTRATIONADMINISTRATION


DIRECTORY STRUCTURE

• LEADS/EMS Typically Uses Two File Spaces (Disks) For Its Installation

• Many Of The Machine Directory Structures Are The Same – LEADS1, DSR, And WWW Have A Common Directory Structure

• The CFEP And NOAAPort Have Different Directory Structures

• The NOAAPort Machine Is Not Covered In This Section Of The Training Material


DIRECTORY STRUCTURE (Cont.)

Refer to:

“Appendix A – Directory Structures”


DATA FLOW


DATA FLOW (Cont.)

Refer to:

“Appendix B – LEADS-EMS Data Flow”


SCHEDULED TASKS

• Equivalent To Unix Cron Jobs

• Scheduled Tasks Is The Microsoft Windows Utility Used To Schedule An Operation To Be Executed At Periodic Intervals

• All Tasks Run Under The “leadsadmin” account

• There Are Tasks Scheduled For Periodic Execution On The LEADS1 and CFEP Machines

• Most Scheduled Tasks Can Be Manually Invoked As Needed– Certain Tasks May Need To Be Invoked As New Sites Are Brought On-line

– Data Reloads May Necessitate This As Well

• Scheduled Tasks Can Also Be Temporarily Disabled

• This Is Done With The Scheduled Tasks Utility


CFEP SCHEDULED TASKS

• CommandedTimeSync – Runs Every Minute

– Looking For A Web Commanded Time Sync Of A Single CAMS

• ModemPolling – Runs Every Six Hours

– Poll Data Loggers With Telephone/Modem Connections

• InternetPolling – Runs Every Fifteen Minutes

– Poll Data Loggers With Internet Connections

• SendZenoFiles – Runs Every Fifteen Minutes

– Send Data Sets Collected From All Polled Sites To The LEADS1 Machine For Ingest Processing

• TimeSyncs – Runs Twice A Week

– Send Time Synchronization Updates To All Data Loggers


LEADS1 SCHEDULED TASKS

• Build Starts – Runs Every Three Hours

– Scans The Hourly Database And Creates An Auxiliary File Containing Information About Each Parameter Monitored

• BuildPreviousYearSummaries – Runs Each Sunday

– Forces A Rebuild Of The Various Summary Files For The Previous Year (This Is Necessary In Case Someone Has Modified Any Data. Through Manual Validation For Example.)

• Create Loss – Runs Once A Day

– Scans The Hourly Database And Creates An Auxiliary File Noting The Number Of Flags Recorded For Each Monitor

• Create1Hour – Runs Every Hour

– Scans The Hourly Database And Creates An Auxiliary File Containing The Maximum Ozone Averages Recorded For Each Site For Each Day


LEADS1 SCHEDULED TASKS (Cont.)


– Scans The Hourly Database And Creates Auxiliary Data Files Containing Daily Averages For Select Parameters

– Currently Configured To Create Daily Averages For SO2, PM-2.5, PM-10, And O3

• Create30Minute – Runs Every Seven Hours

– Scans The Hourly Database And Creates Auxiliary Files Containing Running 30-Minute Averages Of SO2 And H2S


– Scans The Hourly Database And Creates Auxiliary Files Containing Rolling 8-Hour Averages Of O3 And CO

• CreateAQIMaps – Runs Every Hour

– Constructs The AQI Map Image – Uses Many Of The Auxiliary Data Files



• CreateExceed – Runs Every Hour

– Scans The Hourly Database And Creates An Auxiliary File Containing Any Ozone Averages That Exceed 125 PPB

• CreateOzoneMaps – Runs Every Hour

– Creates The Maps Of Current Ozone Levels

• EPA AirNow Export – Runs Every Hour

– Scans The Hourly Database And Constructs An Input File For The EPA AIRNow System, Then FTPs This File To The AIRNow Site

• Load Calibration Reports – Runs Every Six Hours

– Update The Ancillary SQL Database With Calibration Run Results Harvested From The Data Stream



• Missing Data Summary – Runs Every Fifteen Minutes

– Scans The Hourly Database Looking For Any Sites That Have Not Had Data Harvested For The Last Two Hours

– Constructs An Auxiliary File Enumerating Any Sites With Problems

• Sync AutoRestart – Runs Every Five Minutes

– If A Unison Synchronization Problem Is Reported, Rename The DANGER.README File In Order To Allow Synchronization To Continue

• Sync Files from CFEP – Runs Every Fifteen Minutes

– Synchronize The Contents Of The “//CFEP/leads/logs/” Directory To “//LEADS1/leads/logs/CFEP/” And To “//DSR/leads/logs/CFEP/”

– These Files Are Automatically Created By The CFEP Processing Software And Contain System Information



• Sync Files to DSR – Runs Every Fifteen Minutes

– Synchronize The Contents Of The “//LEADS1/dsrweb/” Directory To “//DSR/dsrweb/”

– Synchronize The Contents Of The “//LEADS1/leads/data/pdcs/” Directory To “//DSR/leads/data/pdcs/”

– Syncrhonize The Contents Of The “//LEADS1/leads/data/config/” Directory

– To “//DSR/leads/data/config/”

• Sync Files to WWW – Runs Every Hour

– Synchronize The Contents Of The “//LEADS1/dsrweb/” Directory To “//DSR/dsrweb/”

– Synchronize The Contents Of The “//LEADS1/leads/data/pdcs/” Directory To “//DSR/leads/data/pdcs/” (Excluding “…/pdb/five-minute/”)

– Syncrhonize The Contents Of The “//LEADS1/leads/data/config/” Directory

– To “//DSR/leads/data/config/”



• Sync Remote Commands to CFEP – Runs Every Minute

– If A Web-Commanded Time Sync Has Been Requested, Synchronize The Command to The CFEP


LOGS AND LOG ANALYSIS

• Important LEADS Log Files Include

– CFEP Error Logs – Use The CFEP Error Log Web Page To Peruse These

– Comms Log Files – Use The Comms Report Web Page To Peruse These

– r_script.debug – A Log File Of r_script Messages Maintained In The /leads/data/tmp Directory On The CFEP

• CAMSxx.backup – Not A Log File, But Backup Data Files Maintained In The /leads/data/backup Directory On The CFEP


• This page intentionally left blank


SYNCHRONIZATION

• Independently Of The Real-Time Operational Data Flows Of The LEADS/EMS System There Is A Set Of Data Exchange Activities Which We Group Under The Category Of "Synchronization“

• When We Refer To Synchronization We Are Referring To The Maintaining Of Certain Data Sets In A 'Close Enough To The Same' State On Multiple Discrete Computers (I.E., Identical But With A Time Lag)

• There Are Three Main Sets Of Data Which Are Being Maintained In Such A 'Synchronized' Condition


COMMS RESULTS

• Results Of Communications With The Data Loggers In The Field

– Note That The “Results Of Communications” Is Quite Different Than The “Content Of Communications”

– This Is The Information About Success, And Failure, Of Just Doing The Talking – Not The Gathered Sensor Data Itself

– This Data Consists Of The Log Files Accumulated By The Communications Front End Processor (CFEP) System

– The //CFEP/leads/logs/ Directory Is Periodically Synchronized To Both The LEADS1 And The DSR Systems


PROCESSED DATA

• Processed Field Sensor/Monitor Data

– This Is The Stored Sensor Data After Real-time Calculations Made By The LEADS/EMS System As Well As Any Changes Made By Data Validators Or Other System Users With Authority To Make Such Changes

– This Data Set Is Further Segregated Into Two Main Components:• Five-minute Data

• One-hour Data

– To Enable Access To The Extensive Set Of Web Enabled Reports The //LEADS1/leads/data/pdcs/ directory Hierarchy Is Synced To Both The Internal (DSR) And External (WWW) Web Servers

• With The Exception That The .../five-min/ Directory Hierarchy Is Excluded From The Sync To The WWW Machine In The Interest Of Reducing Network And CPU Overhead Since The Five-minute Data Reports Are Not Provided To The Public


WEB PAGES AND SUPPORT DATA

• Static And Dynamic Web Pages And Supporting Data

• All Web Page Modifications Are Performed On The Leads1 System And Then Replicated (Sync'ed) To The DSR And WWW Systems

• There Are Three Files (INDEX.HTML, EXTERNAL.CFG, And INTERNAL.CFG) Which Are Excluded From This Synchronization Step

– These Files Configure The Behavior Of The System As Either An Internal Access Only DSR System Or As A Public Access WWW System


SYNCHRONIZATION MECHANISM

• The Mechanism Used To Perform These Data Synchronization Operations Is The Unison Application Which Resides In /Leads/Bin, The Assorted Command Files ( *.Cmd ) Invoked Via Shortcuts From "Scheduled Tasks" And The Profile Definition Files (*.Prf) In "C:\documents And Settings\leadsadmin\.unison\“ Which Specify The Detailed Synchronization Configurations

• All Operations Are Performed Under Account "leadsadmin" And, Therefore, Target Directories Must Be Accessible Via A Shared/Mapable Directory With Modify/Read/Write Access Permissions Granted To "leadsadmin"

• Errors In The Synchronization Process Are Captured In Files In Directory "C:\documents And Settings\leadsadmin\" Named DANGER.README (Or Very Similar Names) With An Incrementing Number In The Event Of The Detection And Auto Error Recovery Of Multiple Failures

• Auto Error Recovery/Resume Operations Are Handled By A Scheduled Task Driven Script Which Renames DANGER.README

• This Allows Synchronizing To Resume But Does Not Clean Up Failed Sync Issues


SYNC SUMMARY

• CFEP -> Leads1 | log files for analysis and reporting• CFEP -> DSR | log files for analysis and reporting

• Leads1 -> CFEP | CAMS TimeSync Request

• Leads1 -> DSR | /leads/config• Leads1 -> DSR | /leads/data/pdcs• Leads1 -> DSR | /dsrweb

• Leads1 -> WWW | /leads/config• Leads1 -> WWW | /leads/data/pdcs• Leads1 -> WWW | /dsrweb


SYSMON

• Provides A Method Of Determining Which LEADS Processes Are Running On The Local LEADS1 Machine

• Divided Into Process Groups• Green Circle Means Process Is Running, Yellow Is In A Wait

State, And Red Indicates The Process Is Not Currently Running• User Can Start Or Stop Process Groups, As Well As Perform A

Reinitialization Of The System By Pressing The “Master Re-read Config” Button To Trigger The Master Process To Re-read The SYSTEM.CFG Configuration File




REMOTE DESKTOP CONNECTION

• The Manual Validation Tool Is Always Run On The LEADS Primary Server

• To Provide Access For Users, A Remote Desktop Connection Is Used

• The RDC Provides A Mechanism Under Windows So That Users Can Run An Application Remotely And Have The Display Presented On Their Local Computer


RDC (Cont.)

Refer to:

“Appendix C – TS and RDC for ManVal”


SITE INITIATION FORM

• Data Collection Begins With The Real World

• If The Data Logger Is Misconfigured, Then Reality Is Misperceived And Misreported

• Likewise, If The LEADS/EMS System Is Misconfigured (Does Not Match The Data Logger Configuration), Then Reality Is Misperceived And Misreported

• The Site Initiation Form (SIF) Is Used To Record The Configuration Of A Monitoring Site


SITE INITIATION FORM (Cont.)

• The SIF Attempts To Capture All The Relevant Information Required To Define A Site, Collect Data, And Report Data To The Various End Users

• The Template Has Defaults For The Most Common Channel Assignments, Method Codes, And Units Codes As Well As Standard Data Flag Settings

• The Contents Of The SIF Are Transferred To The Various LEADS/EMS Configuration Files


SIF TEMPLATE

Please Refer To:

“Appendix D – SIF TEMPLATE”




SIF FIELDS

• Mandatory Fields Are Denoted With A “”

– Without The Information From These Fields, The Site Cannot Be Configured Within LEADS/EMS

Site Name

– A Descriptive Name For The Site; Primarily Seen On The Various Web Pages

AQS Number

– The 9-Digit EPA-Style Site Identifier

– The First Two Digits Are The State Code

– The Next Three Digits Are The County Code

– The Last Four Digits Are The Site Identifier; These Are Used To Differentiate Between The Sites Located In The Same County

– The AQS Number Is A Primary Identifier For Data Storage


SIF FIELDS (Cont.)

• Project Manager

– The Name Of The Project Manager Overseeing This Site

SIF Applicability Date

– The Date The Site Is Expected To Start Collecting Data With The Configuration Described On The Form

• Street Address

– The Physical Street Address Of The Monitoring Site

Data Logger ID

– One To Four Digits; Allowable Range Is 1 To 9998

– For Convenience Used As A Shorthand Reference To The AQS Number

– Many Data Logger ID’s Can Map To One AQS Number


SIF FIELDS (Cont.)

• Site Operator

– The Name Of The Person Responsible For Day-To-Day Site Maintenance

• Site Phone Number

– If There Is A Phone At The Site, Enter The Number Here

– If All Sites A Network Connected, This Is Optional Since All The Sites Use The Internet For Data Collection; If There Are Any Sites Added That Are Serviced By Standard Phone Lines And Modems, This Field Becomes Mandatory As It Is The Number Dialed For Data Collection

• City

– The Name Of City Where The Monitoring Site Is Located

• Zip Code

– The Zip Code For Where The Monitoring Site Is Located


SIF FIELDS (Cont.)

Time Zone

– Time Zone The Monitoring Site Is Located In

– All Data Is Collected And Stored In UTC (Greenwich Mean Time); Many EPA Submissions And Web-Based Reports Use Local Standard Time; This Is The Key To Translating Between UTC and Local Standard Time

• Data Validator

– The Name Of The Person Responsible For Data Validation At The Site

IP Address (Port For Serial Adapter)

– Mandatory For Internet Connections

– This Is The Internet Address Used For Data Collection


SIF FIELDS (Cont.)

• County Name (Code)

– The Code Should Match The Middle Portion Of The AQS Number

Area Description (Code)

– This Is A Discriminator Used For Mapping Ozone And Reporting The Air Quality Index

– Due To The General Geographical Layout Of Monitoring Networks And The Close Spacing Of Sites In Large Metropolitan Areas, It Is Necessary To Have Some Way Of Discriminating Which Map Background To Use

– Most Networks Can Be (Arbitrarily) Divided Into Logical Areas


SIF FIELDS (Cont.)

• Region (Code)

– The EPA Region The Monitoring Site Is Located In

Responsible Entity

– This The Entity That Operates And/Or Validates The Monitoring Site

– This Becomes Very Important When Outside Entities Are Brought Onto The System

• Location Method

– The Method Used To Determine The Site Latitude And Longitude

– Should Include The Datum And Any Other Critical Information


SIF FIELDS (Cont.)

• Contact Person

– The Name Of The Contact Person Representing The Responsible Entity

Lat(itude) (Deg:Min:Sec)

– The Latitude Of The Monitoring Site In Degrees:Minutes:Seconds

– The EPA Rules For Monitoring Site Locations Correspond To An Accuracy Of About One Arc-Second

– North Is Considered Positive; All Sites In The Continental US Have Positive Latitudes

• Office Phone Number

– The Office Phone Number Of The Contact Person That Represents The Responsible Entity


SIF FIELDS (Cont.)

Lon(gitude) (Deg:Min:Sec)

– The Longitude Of The Monitoring Site In Degrees:Minutes:Seconds

– The EPA Rules For Monitoring Site Locations Correspond To An Accuracy Of About One Arc-Second

– West Is Considered Negative; All Sites In The Continental US Have Negative Longitudes

• Cell Number

– The Cell Phone Number Of The Contact Person That Represents The Responsible Entity

Elevation (Meters)

– The Elevation Of The Monitoring Site In Meters

– This Is Used To Determine The Standard Barometric Pressure Of The Site Which Is In Turn Used For Automated Met Data Processing


SIF FIELDS (Cont.)

• Site Photos (Date)

– The Date The Site Photos Were Taken

– The EPA Desires That Site Photos Be Updated Periodically (Especially If Instruments Are Added Or Removed); This Is A Convenient Reminder Of When The Last Photos Were Taken

• Comments

– Any General Comments About The Site

The Remainder Of The Form Describes The Data Captured By The Data Logger

– These Are Crucial To Proper Processing And Display Of The Data

– Mistakes Or Errors In This Part Will Require Extraordinary Measures To Correct – Especially Once A Large Quantity Of Data Is Collected


SIF FIELDS (Cont.)

Data Triplet

– This Is The Output Channel Configured On The Data Logger

– Each Parameter Transmitted By The Data Logger Must Have A Unique Output Channel

– It Is Strongly Recommended That Standard Output Channels Be Used Across The Data Loggers – This Greatly Simplifies Field Support

• Parameter Name

– A Mnemonic To Help With What Is Configured On The Data Triplet

– The Parameter Code Fully Defines What The Parameter Is, But Is Much Harder To Remember


SIF FIELDS (Cont.)

Parameter Code

– The Five-Digit LEADS Parameter Code

• Generally The LEADS Parameter Code Is Identical To The EPA Parameter Code

– Every Parameter MUST Have A Parameter Code Defined

• Sometimes A Pseudo-Parameter Code Must Be Used – For Parameters That Are Not Registered With The EPA

– Data Is Stored In The Various LEADS/EMS Databases And Auxiliary Files Based On Parameter Code – This Makes The Parameter Code A Primary Data Identifier

– The Following EPA Web Site Can Be Used To Find Parameter Code Information (As Well As Other Information)

• http://www.epa.gov/ttn/airs/airsaqs/manuals/codedescs.htm


SIF FIELDS (Cont.)

EPA AQS Method Code

– The Three-Digit EPA Method Code That Describes How The Instrument Makes Its Measurements

– This Information Is Mandatory For Data Submittal To The EPA

– Each Instrument Manufacturer That Has Received An EPA Equivalency Code For A Particular Instrument Will Place A Sticker On The Instrument Specifying The Reference And Equivalence Code Of That Particular Instrument

– The Last Three Digits Of The Equivalency Code Are The Method Code

– Unfortunately, Method Codes Are Not Unique – You Will Find The Same Three-Digit Method Code Applied To Different Instruments Measuring Radically Different Parameters

– The Following EPA Web Site Can Be Used To Search For Equivalency Codes

• http://www.epa.gov/ttn/amtic/criteria.html


SIF FIELDS (Cont.)

EPA AQS POC

– The Parameter Occurrence Code

– The POC Is A Convenient Method For Tracking Multiple Measurements Of A Single Parameter At One Site

– Each Additional Instrument That Measures A Specific Pollutant At A Monitoring Site Is Assigned A New POC

– For Example, The First Ozone Monitor At A Site Will Have A POC Of 1; If A Second Ozone Monitor Is Installed, It Will Have A POC Of 2

– Sometimes, An Instrument May Have Been Present At A Site In The Distant Past; Often, A New POC Will Be Assigned In This Case

– The POC Should NOT Be Used To Track Instrument Swaps– The POC Is The Only Way To Differentiate Between The Same Data

Measured By Multiple Instruments At A Site– The POC Is A Primary Identifier For Data Storage In The LEADS/EMS

Databases And Auxiliary Data Files


SIF FIELDS (Cont.)

• Monitor Manufacturer

– In Combination With A Specific Monitor Model, The Equivalency Code (If One Has Been Assigned By The EPA) Can Be Determined

• Monitor Model

– In Combination With A Monitor Manufacturer, The Equivalency Code (If One Has Been Assigned By The EPA) Can Be Determined

Monitor Output Range And Units

– The Physical Output Range Of Each Instrument Installed At A Monitoring Site

– This Information Is Used To Program The Conversion Of Raw Measurements To Engineering Units

– Some Conversions Are Programmed Into The Data Logger, Others Are Done Centrally


SIF FIELDS (Cont.)

Corresponding Measurement Range And Units– What The Raw Measurements Are Expected To Be Converted Into

– These Descriptions Will Greatly Influence Calibration Limits (For Calibrated Parameters)

– The Expected Instrument Range Can Also Be Used To Establish Upper And/Or Lower Limit Checks

Measured Units Code– The Three-Digit LEADS Units Code That Describes The Units The Particular

Parameter Is Measured In

• Generally The LEADS Units Code Is Identical To The EPA Units Code

– This Information Is Mandatory For Data Submittal To The EPA

– The LEADS/EMS Database Uses Implicit Units – Each Parameter Is Expected To Always Be Measured In The Same Units At Every Site

– There Is Leeway In Setting Up What Units You Measure In, But They Have To Remain Constant Across Sites (And Time)


SIF FIELDS (Cont.)

• The Remainder Of The Fields Are Data Flags That Influence Processing And Display Of The Data

AQI

– A Flag Indicating Whether Or Not The Parameter Should Be Included In The Air Quality Index Calculation

– Some Sites Do Not Meet Siting Criteria And Should Be Excluded (i.e., Sites Located Near A Strong Point Source Which Causes Measurements To Always Be Higher Than Surrounding Ambient Conditions)

– Many Of The Web Pages And Background Processes Check This Flag

– Only Certain Parameters Go Into The Air Quality Index

• Ozone, Carbon Monoxide, Sulfur Dioxide, PM-10, And PM-2.5• Nitrogen Dioxide Could, But It Is Normally Not Included

– The Background Processing And Web Pages Default To NOT Include A Parameter In The AQI Calculations (And Some Image Processing)


SIF FIELDS (Cont.)

AQS

– A Flag Indicating Whether Or Not The Parameter Is Expected To Be Reported To The EPA AQS Database

– This Should Influence The Data Validators To Emphasize Validation Of Data That Is Going To Be Reported To The EPA Over Data That Is Not

– The Web Pages And Other Processes Default To A Value Of “No – Do Not Expect To Send This To The EPA AQS Database”

– Currently, This Is Used Only As A Data Descriptor (No Actual Processing Is Controlled Based On This Flag); Future Processing May Use This Flag As A Control


SIF FIELDS (Cont.)

AIRNow

– Controls Whether Or Not A Parameter Is Automatically Exported To The EPA AIRNow Web Site

– Only Certain Parameters Are Exported To AIRNow

• Ozone• PM-10• PM-2.5

– The Following Met Parameters Can Be Exported In Support Of Homeland Defense

• Resultant Wind Speed• Resultant Wind Direction• Outdoor Temperature (10 Meter)• Relative Humidity• Solar Radiation• Barometric Pressure• Precipitation

– The Background Processing And Web Pages Default To NOT Export A Parameter To AIRNow


SIF FIELDS (Cont.)

Public

– A Flag Indicating Whether Or Not The Parameter Is Available To The General Public

– Certain Parameters Such As Internal Temperatures, Test Probes, Etc. Should Never Be Exposed To The General Public

– The Web Pages And Background Processing Default To Block Any Parameter From Public Consumption

Regulatory

– Many Of The Web Pages Have The Capability To Add Embellishments Describing Whether Or Not A Particular Instrument Should Be Considered Regulatory

– This Flag Controls This

– The Web Pages Default To Showing Any Instrument As Non-Regulatory


SIF FIELDS (Cont.)

Experimental

– A Flag Indicating Whether Or Not An Instrument Is Considered Experimental

– Under Certain Conditions, It May Be Desired To Install An Instrument At A Site That Is Experimental In Nature

– This Flag Will Trigger Embellishments On The Web Pages Indicating That A Particular Instrument Is Experimental

– Background Processes Will Exclude Any Instrument That Is Flagged As Experimental From Certain Averages And Indices

– The Web Pages And Background Processing Default To Assume That An Instrument Is NOT Considered Experimental


HOW TO ADD A SITE

• Adding A Site Involves Configuration Of The Datalogger, The Automatic Gas Calibrator, And The LEADS/EMS Software

• This Section Primarily Covers The Configuration Of The LEADS/EMS Software

– It Is Strongly Recommended That Common Configurations Be Used For The Data Loggers And Calibrators – This Reduces Errors And Eases Maintenance

• The First Step Is To Completely Fill Out A Site Initiation Form

• LEADS/EMS Is Configured Almost Exclusively From The Site Initiation Form


BASIC DATA LOGGER SETUP

• Configure The Input Channels To Cover The Physical Instruments At The Site

• Configure Any Processing (Such As Met, Dewpoint Calculation, Etc.)

• Configure The Output Channels

• The Data Logger Configuration MUST Match The Site Initiation Form, Or The LEADS/EMS Configuration Will Be Incorrect

• Errors Can Be Corrected After The Fact, But This Process Is Time-Consuming And Erroneous Data Is Displayed Until Corrections Are Made


MORE DATA LOGGER SETUP

• A Time Sync Should Be Issued To The Data Logger After It Is Installed At The Site

• Immediately After The Time Sync, All Existing Data On The Data Logger Should Be Erased

• From This Point, The Data Logger Is Ready To Collect, Store, And Transmit Data

– The Data Logger Can Store Between One To Three Weeks Of Data, Depending Upon The Number Of Instruments

– It Is Not Imperative That Communications Are Set Up Immediately, But Is Highly Recommended Since You Want The Initial Time Sync


BASIC CALIBRATOR SETUP

• It Is Highly Recommended That All The Automatic Gas Calibrators Should Be Configured Identically

• This Should Be Done Prior To Installation At The Site

• Sequence Definitions Must Match The LEADS System Implementation (This Is NOT Configurable In LEADS)

• The Duration Of The Sequence Levels Must Match The Settings In The Zeno Channel Configuration File

• The Level Concentrations Must Match The Settings In The Calibration Limits File

• The Automatic Calibration Checks Will Fail And Reject Data If There Is A Mismatch In Any Of These


LEADS/EMS NEW SITE SETUP

• Site Setup Cannot Really Begin Until The Site Initiation Form Is Completely Filled Out

• There Are Several Distinct LEADS/EMS Components That Will Require Configuration Changes To Add A New Site

• These Include

– The LEADS Primary Server

– The Comms Front End Processor

– An Ancillary SQL Database

– The Web Pages

• Efforts Should Begin With The LEADS Primary Server


UPDATE EPASITE.DAT

• Add A New Entry To The EPASITE.DAT File

• This Is An ASCII Text File Located In The /leads/data/pdcs/pdb/sites Subdirectory

– There Is A File Header Describing The Various Fields

– Study Existing Lines And Follow Their Formatting And Spacing

– It Is Recommended That You Copy An Existing Line And Change It To Match The New Site

– Of Particular Note Is The Barometric Pressure Entry – This Is Used By The Ingest Software To Perform Automated Checks Against The Readings From A Pressure Transducer (If One Is Installed At A Site) – This Should Correspond To The Standard Atmospheric Pressure Based On Site Elevation

• Changes To This File MUST Be Made On The LEADS Primary Server


EPASITE.DAT

#STATE,COUNTY,SITE,REGION,LATTITUDE,LONGITUDE,ELEVATION,LST_OFFSET,MONITORS,DESCRIPTION

#

#STATE: EPA State Code

#COUNTY: EPA County Code

#SITE: EPA Site Code

#REGION: Region Number

#LATITUDE: (-)DDMMSS (positive north) do not use +

#LONGITUDE: (-)DDDMMSS (positive east) do not use +

#ELEVATION: Site elevation in meters (signed) <-- NOT USED BY LEADS, BUT USED BY WEB PAGES

#LST_OFFSET: Number added to GMT to get LST (Signed )

#MONITORS: Mnemonic for type data monitored, (Z)eno, (S)aroad, (A)irs, Auto(G)C <-- MAY HAVE MULTIPLE MONITORS

#MILLIBARS_AVG: Average atmospheric pressure in millibars

#SPRING_START: Start of spring season, month and day (mmdd)

#SPRING_TEMP: Spring average temperature (degrees Farenheit)

#SUMMER_START: Start of summer season, month and day (mmdd)

#SUMMER_TEMP: Summer average temperature (degrees Farenheit)

#FALL_START: Start of fall season, month and day (mmdd)

#FALL_TEMP: Fall average temperature (degrees Farenheit)

#WINTER_START: Start of winter season, month and day (mmdd)

#WINTER_TEMP: Winter average temperature (degrees Farenheit)

#DESCRIPTION: Site description so you don't have to remember only EPA numbers <-- NOT USED BY LEADS, but used by web pages

#

32,003,0043,1,360623,-1151512,735,-8,Z,928.4,0321,99.9,0621,99.9,0921,99.9,1221,99.9,Paul_Meyer


UPDATE CAMSSITE.DAT

• Add A New Entry To The CAMSSITE.DAT File







CAMSSITE.DAT

#NUMBER,START_DATE,END_DATE,STATE,COUNTY,SITE

#

#END_DATE: 0 if START_DATE is current

#

#If there are multiple CAMS or TAMS co-located, make sure

# to define the primary CAMS first in the file - this is

# the site that will be used by the web pages and other

# utilities to map an EPA site back to a CAMS number

#

0043,20061025000000,0,32,003,0043,0,0,0

0071,20060905000000,0,32,003,0071,0,0,0

0072,20060724000000,0,32,003,0072,0,0,0

0073,20061005000000,0,32,003,0073,0,0,0

0075,20060518000000,0,32,003,0075,0,0,0

2002,20060503000000,0,32,003,2002,0,0,0


UPDATE ADDRESS.DAT

• Add A New Entry To The ADDRESS.DAT File





• Information In This File Is Used Exclusively By The Web Pages



ADDRESS.DAT

#STATE,COUNTY,SITE,ADDRESS,CITY,ZIP CODE

#

#STATE: EPA State Code

#COUNTY: EPA County Code

#SITE: EPA Site Code

#ADDRESS: Street Address

#CITY: City

#ZIP CODE: Zip Code

#

# Note: no commas are allowed in any of the fields

#

32,003,0043,4525 New Forest Dr.,Las Vegas,89147

32,003,0071,7701 Ducharme Dr.,Las Vegas,89145

32,003,0072,3525 N. Valdez St.,Las Vegas,89129

32,003,0073,333 Pavillion Center Dr.,Las Vegas,89144

32,003,0075,6651 West Azure Ave.,Las Vegas,89130

32,003,2002,1301 East Tonopah (B),North Las Vegas,89030


UPDATE REGION.DAT

• Add A New Entry To The REGION.DAT File, If Necessary





• Information In This File Is Used Exclusively By The Web Pages



REGION.DAT

# REGION.DAT

#

# If a large number of sites are installed, the site selection

# list becomes too long to easily find the sites of interest.

# Use this file to establish regions. The site/region relationship

# is defined in EPASITES.DAT. The region numbers used here and in

# EPASITES.DAT must match.

#

# Format:

# region name,full region name, region number, include region

#

# NOTE: Leave no spaces between string fields and commas

#

All Regions,All Regions, 0, 1

Clark County,Clark County, 1, 1

Clark Test,Clark - Region 99, 99, 1


UPDATE METRO.DAT

• Add A New Entry To The METRO.DAT File, If Necessary




• Information In This File Is Used Exclusively By The Web Pages And Background Processes

• This File Defines The Names Of The Metro Areas Used For

Report Grouping Purposes



METRO.DAT

0,Not in a Metro Area

1,Greater Las Vegas

2,Mesquite

3,Jean

4,Boulder City

5,Apex


UPDATE SITEINFO.DAT

• Add A New Entry To The SITEINFO.DAT File





• Information In This File Is Used Exclusively By The Web Pages And Background Processes

• This File Maps An EPA Site To A Metro Area For Report Grouping Purposes



SITEINFO.DAT

# State, county, site, metro area

32,003,0043,1

32,003,0071,1

32,003,0072,1

32,003,0073,1

32,003,0075,1

32,003,2002,1


CREATE A NEW CHANNEL CONFIG

• Create A New Zeno Channel Definition File – This File Maps Output Channels From The Data Logger To Parameters

• These Are ASCII Text Files Located In The /leads/data/pdcs/pdb/sites Directory

• Copy The ZCxxx.DAT Template File To ZCnnn.DAT, Where nnn Is The New CAMS Number – This MUST Be Done On The LEADS Primary Server

– Please Note The File Naming Convention Used• If The CAMS Number Is One Or Two Digits, It Is Padded With Leading Zeros To Three Digits,

Otherwise It Is Not Padded


– Edit The New ZCnnn.DAT File You Just Created

– Change The File Contents To Match The Site Initiation Form Output Channel Definitions

– Set The Effective Date To The Start Data From The Site Initiation Form


ZENO CHANNEL CONFIG FILE

Please Refer To:

“Appendix E – ZCnnn.DAT Contents”


SET INTERNAL TEMP LIMITS

• If This Site Has Internal Temperature Readings That Require Different Upper And Lower Limits From The System-Wide Defaults, Set New Upper And Lower Limits

• This Is An ASCII Text File Located In The /leads/data/pdcs/pdb/sites Directory

• Edit The OVERRIDE_FLAG.DAT File And Add A New Line For The New Internal Temp Parameter(s)


– Edit The File Contents To Match The Site Initiation Form Output Channel Definitions

– Set The Effective Date To The Start Data From The Site Initiation Form

• The Contents Of This File Are Used Exclusively By The Web Pages And Background Processes



OVERRIDE_FLAG.DAT

# Use this file to define individual upper and lower limits for internal temp

#

# setting either the upper or lower limit to -999 causes the web pages to ignore and not color code

# if a site does not have upper and lower limits set in this file, the system-wide default limits are used

# (for whatever parameter you might have interest in)

# not all parameters have color-coding on the web pages, so you would not expect all parameters to show up

# in this file

# this file was initially intended to provide color-coding override capability for internal station temperature

#

# EPA Site, param, poc, start_date_utc, stop_date_utc, lower_lim, upper_lim

#48_465_0003,62107,3,20040804130000,0,32,110


CREATE A NEW AUDIT TRAIL FILE

• The Audit Trail Files Reside In The /leads/data/pdbs/audit Subdirectory

• A New Audit Trail File Is Not Automatically Created By Manual Validation – You Must Manually Create One

• Copy The Template File “00_000_0000_template” To “SS_CCC_NNNN” – This MUST Be Done On The LEADS Primary Server

– Where SS Is The Two-Digit State Code

– Where CCC Is The Three-Digit County Code

– Where NNNN Is The Four-Digit Site Number (Zero-Padded If Necessary)

• This File Does Not Have Any Entries To Start With, It Just Needs To Exist – Copying The Template Will Create A File With Just A Header Section


CREATE CALIBRATION LIMITS

• The Calibration Limit Files Reside In The /leads/data/pdbs/calibdata Subdirectory

• The Contents Of This File Are Used By The Ingest Software During Calibration (Span, Or Span-Zero) Processing

• If A Calibration Limits File Does Not Exist For A Site, Automatic Processing Of Quality Checks Will Fail

• It Is Recommended That You Copy An Existing CLxx_xxx_xxxx.DAT File To CLnn_nnn_nnnn.DAT

– Ideally, The Existing Site Should Have Nearly The Same Instrumentation As The New Site

• If The New Site Does Not Have Any Calibrated Parameters, This File Is Not Necessary


CLnn_nnn_nnnn.DAT

• This Is An ASCII Text File

• Once You Have A Copy For The New Site, Edit The File And Make Any Changes Required

– Changes Will Be Necessary If The New Site Uses A Different Full-Scale Instrument Range

– Changes Will Be Necessary If The New Site Uses Different Calibration Levels

– Changes Will Be Necessary If Different Limits Are Desired



CALIBRATION LIMITS FILE

Please Refer To:

“Appendix F – CL_nn_nnn_nnnn.DAT Contents”


CREATE CALIBRATION DATA FILE

• The Calibration Data Files Reside In The /leads/data/pdbs/calibdata Subdirectory

• The Contents Of This File Are Used By The Ingest Software During Calibration (Span, Or Span-Zero) Processing

• If A Calibration Limits File Does Not Exist For A Site, Automatic Processing Of Quality Checks Will Fail

• Copy The CDxx_xxx_xxxx.DAT Template File To CDnn_nnn_nnnn.DAT

• This New File Will Be Empty And Will Be Populated By The LEADS/EMS Software As Cals, Spans, And Span-Zeros Are Processed

• If The New Site Does Not Have Any Calibrated Parameters, This File Is Not Necessary


SQL AUXILIARY DATABASE

• There Is An Auxiliary SQL Database That Is Used To Warehouse Calibration Details

• Certain Database Tables Must Be Updated When New Sites Or Monitor Are Added

• It Is Highly Recommended That You Use A Tool Such As PGAdmin III For All Database Maintenance

• The Tables That Must Be Modified Include

– Table “site”

– Table “cams”

– Table “cams_monitor” (Only If Calibrated Parameter(s) Are Present)

• Changes To The SQL Database MUST Be Made On The LEADS Primary Server


WEB CONFIGURATION

• It May Be Necessary To Modify One Or More Web Page Configuration Files As New Sites Are Added

• These Files Are Plain ASCII And Are Actually Perl Scripts That Are Included By Almost Every Web Page

• The Web Page Configuration Files Reside In The /dsrweb/cgi-bin/lsr Subdirectory

• Making Changes To These Files Can Result In Crippling Damage To The Web Pages – It Is Recommended That These Files Only Be Changed By Someone Familiar With Perl (Especially The Capability Of Perl To Run In A Debug Mode To Check For Typos And Grammatical Errors)


POTENTIAL WEB CHANGES

• There Are Three Main Web Configuration Perl Scripts

– config_info.pl

– param_config.pl

– site_config.pl

• These Files Normally Do Not Need To Be Tweaked

• Extraordinary Circumstances Such As Adding A New Parameter Not Supported By The Web Pages, Or Installing A Site In A New Area May Require Changes To One Or More Of These Files

• Certain Processing Scripts That Perform Specialized Processing (Such As The Script That Creates The Current Ozone Map – Named aqi_alert.pl), May Also Require Attention


POTENTIAL WEB CHANGES (Cont.)

• The Clickable Image Where One Can Select A Site May Need To Be Updated To Reflect A New Site – Associated With This Image Are Two “.map” Files That Define Clickable Areas On The Image

– The Clickable Map Images And Associated “.map” Files Reside In The /dsrweb/reports/graphics/clickable Subdirectory

• If Site Photos Are Needed, They Will Need To Be Placed In The /dsrweb/reports/site_graphics Subdirectory (Currently, There Are Site Photos For All Known Clark County Sites)

– Be Sure Follow The Correct Naming Convention

• The Site Photo Files Are Named 32003NNNN_<d>.gif

• Where NNNN Is The Site Number (Zero-Padded If Necessary) And <d> Is the Direction Of The Photo; Overall Site Photos Are Designated With A Direction of “o” (For Overall)


POTENTIAL WEB CHANGES (Cont.)

• If A New Street-Level Map Is Required, It Should Be Generated And Placed In The /dsrweb/reports/site_graphics/maps Subdirectory

• Currently, There Are Street-Level Maps Already Generated For All Known Sites

– Be Sure Follow The Correct Naming Convention

• The Street-Level Map Files Are Named SSCCCNNNN_map.gif

• Where SS Is The State Identifier

• Where CCC Is The County Identifier

• Where NNNN Is The Site Number

• Changes To The Web Files MUST Be Made On The LEADS Primary Server


WEB IMAGE SPECIFICATIONS



CFEP CONFIGURATION

• Normal Site Additions On The CFEP Require The Creation Of Two Files And Modification Of Two Existing Command Files

• The First File Created Controls How The Process r_script Communicates With A Data Logger To Collect Data

• The Second File Created Controls How r_script Sends Time Sync Commands To The Data Logger

• Pertinent Files To Change On The CFEP Reside In The /leads/bin Subdirectory


DATA COLLECTION SCRIPT

• Create A New sailer Script For Data Collection

– Copy An Existing sailer.scr.xxx To sailer.scr.NNN

• Where NNN Is The Zeno ID Number (I.E., The CAMS Number)

– Change The IP Address And Any Other Comm Settings As Needed

• IP Address

• IP Port#

• COMM Port#

• Data Logger ID

• Direct Network Or Relayed Configuration

• Number Of Records To Retrieve

• Add A Reference To sailer.scr.nnn in The File modem1.cmd


TIME SYNC SCRIPT

• Create A New sailer Script For Time Syncs

– Copy An Existing camsxxxtimeset.scr To camsNNNtimeset.scr

• Where NNN Is The Zeno ID Number (I.E., The CAMS Number)

– Change The IP Address And Any Other Comm Settings As Needed

• IP Address

• IP Port#

• COMM Port#

• Data Logger ID

• Direct Network Or Relayed Configuration

• Add A Reference To camsNNNtimeset.scr in The File TimeSyncs.cmd


r_script CONFIGURATION

• It May Be Necessary To Modify The r_script Configuration File (Named RSCRIPT.CFG) – If Specialized Processing Such As Data Record Replication Is Required

• This File Is Plain ASCII And Is A Perl Script That Is Included By r_script

• RSCRIPT.CFG Resides In The /leads/bin Subdirectory On The CFEP

• Making Changes To This File Can Result In Crippling Damage To The Data Collection Mechanism – It Is Recommended That These Files Only Be Changed By Someone Familiar With Perl (Especially The Capability Of Perl To Run In A Debug Mode To Check For Typos And Grammatical Errors)


AFTER DATA IS RECEIVED

• Once Data Is Ingested And Stored In The LEADS Database, There Is A Final Setup Step That Must Be Performed

• There Are A Series Of Flags That Control Web Page Displays And Various Background Processes

• Use The Flag Editor Web Page To Set The Public, AQI, Precision Report, AIRNow, Regulatory, And Experimental Flags

• There Must Be Data Present From The Site Before It Is Available For The Flag Editor Interface


DATA FLAGS

• Public Flag – A Flag That Can Be Applied To Any Parameter Measured

– If This Flag Is Not Set Or It Is Set To Not Public, The Parameter Is Blocked From All External Web Pages

• AQI Flag – A Flag That Can Be Used To Include Or Exclude Parameters From The AQI Report And AQI Map

– Only Specific Parameters Go Into The AQI Report

– If This Flag Is Not Set Or It Is Set To No, The Parameter Is Excluded From The AQI Report


DATA FLAGS (Cont.)

• Precision Report Flag – A Flag That Controls Which Reporting Entity The Precision Report For A Parameter Is Included Under

– Only Specific Parameters Have Associated Precision Reports

– If This Flag Is Not Set Or It Is Set To No, The Parameter Is Not Included In Any Precision Report

• AIRNow Flag – A Flag That Can Be Used To Include Or Exclude Parameters From Automatic Export To The EPA AIRNow Web Site

– Only Specific Parameters Go To AIRNow

– If This Flag Is Not Set Or Is Set To No, The Parameter Is Blocked From Automatic Transmittal To AIRNow


DATA FLAGS (Cont.)

• Regulatory Flag – A Flag That Controls Web Page Display Of A Flag Indicating Whether Or Not The Instrument Is Considered Regulatory

– Only Specific Parameters Can Be Considered Regulatory

– If This Flag Is Not Set Or Is Set To No, The Web Pages Will Flag The Parameters As Non-Regulatory

• Experimental Flag – A Flag That Can Be Applied To Any Parameter That Triggers The Web Pages To Indicate The Instrument Reading Is Experimental

– Experimental Readings May Not Reflect Actual Conditions

– If This Flag Is Not Set Or Is Set To No, The Instrument Is Not Considered Experimental (Normal Operations)


CONTACT MANAGER

• As A Final Finish-Out, You May Want To Update The Contact Manager

• You Need To Wait Until Data Has Been Collected From The Site – Then It Will Appear In The Field Operator And Data Validation Additional Data Forms




VERIFICATION OF A NEW SITE

• There Are Several Web Pages That Can Be Used To Verify That A Site Is Configured Correctly

– All Of These Web Pages Read, Format, And Display Many Of The Configuration Files You Edited Or Created

• EPA Site File

• CAMS Definitions

• Zeno Channels

• Cal Limits

• NEG/MUL Definitions

– Step Through Each Of These And Verify That The Display Matches What You Think It Should


MORE VERIFICATION

• Check The Comms Report Web Page To Make Sure Calls Are Being Made To The Site

– Each Site Should Be Contacted Once Every Fifteen Minutes

– The Comms Report Can Help You Isolate Connectivity Issues

• Check The Missing/Unexpected Data Report

– Be Sure To Check The Unexpected Data Option

– This Can Help You Identify Channels That Are Not Configured Correctly


EVEN MORE VERIFICATION

• Check The Contents Of The /leads/data/backup Directory On The CFEP

– If Data Is Being Retrieved From The Site, There Should Be A File Named “CAMSxx.backup” That Contains A Copy Of All The Raw Records Retrieved From The Data Logger

– If This File Exists And Is Updating, Data Is Being Retrieved From The Site

• Check The CFEP Error Log Web Page

– Make Sure There Is Not A Time Problem On The Data Logger

– The System Will Reject Older Data (Data That Should Have Already Been Retrieved)


FINI

This Concludes System Administration

Documents

LEADS/EMS SYSTEM ADMINISTRATION