Presented by: Steve S. Di Lonardo & James C. Alair September 15 th , 2009 West Point, New York

Presented by: Steve S. Di Lonardo & James C. Alair September 15th, 2009 West Point, New York

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Water Information System for the Management and Storage of Time Series

Data

Outline Challenges associated with time series data WISKI software as a solution Improvements to NYC DEP time series data

quality and access? Case studies Other potential

applications

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Challenges of time-series data

Government & environmental agencies produce large amounts of data • Requires the establishment of information

systems to acquire and store data• Limited or no built-in QA/QC capabilities• Often results in data silos • Limited data sharing / access between groups

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Time series data As part of DEP efforts to improve and

facilitate the availability of time series data:• Pilot study using WISKI software• Database manager software• Designed by Kisters North America Inc.

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

WISKI database manager• Infrastructure to automatically IMPORT, PROCESS,

COMPUTE, STORE, and REPORT all time series data

• Unlimited sites, stations, parameters and data storage (only limited by hardware)

• Based on MS-SQL, Oracle, etc….• Scalable from a single user to hundreds of users

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

WISKI database manager Data file importing (current and

archival)• All major data loggers formats• Telemetry data acquisition systems• SCADA systems• XML, ZRXP, & Comma delimited files • User defined time-series data

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Who uses WISKI?

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

• Surface water (temperature)• Ground water (salinity) • Meteorological (precipitation)• Water quality (O2)• Stream discharge (flood management)• Any time-series data!

Applications – Types of Data

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

How/Where does WISKI fit at DEP?

LIMS – lab/field data SAS – historical lab/field data SCADA – real-time data

(operations) WISKI

• Compatible with LIMS, SCADA, SAS, ARCGIS

• Organization of high frequency data files into a manageable user-friendly format

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

WISKI software at DEP• Purchased by NYC DEP (2005)

– Used primarily by EOH Hydrology - Kensico streams

• Advancements 2007 - Present– Kensico Streams– USGS gauge stations– Weather stations – Keypoints – Reservoir Elevation/Capacity

• Water Security Initiative (EPA Grant)– Integral part of keypoint data acquisition and storage

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

• Access via BWS intranet to all DEP employees

• Automated data imports (excel, html, asci)• Reduce need for data requests• Data available in a working format

WISKI as a software solution

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

NYC Watershed & Distribution

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Data ComputationsWISKI Data Management System

• Data Conversions – By Formula– Through Rating

Tables• Creates Summary

Data – Min, Max, Mean, Total – Daily, Monthly, &

Yearly

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

WISKI Data Reports

Standard Reports• Year End Reports• Monthly Reports• Rating Curve Reports• Discharge Measurement Summary• User-defined Reports

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

• Time and user stamped data QA/QC• Original imported time-series data cannot be

edited• Altered data maintains history• User-defined plausibility flags (eg. Turbidity)• Restrict sensitive data to certain user groups

WISKI Data Management System

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Tabular presentation of the Flags

Comments and Remarks

Graphical presentation of the Flags

WISKI Data Management System

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Case Studies & Potential Applications at NYC DEP

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Case Study: Sewage Spill Warning & Prevention

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Water Budgets / Storm Monitoring

Site NameAll Flow

(cfs)Base Flow

(cfs)Storm flow

(cfs)Base Flow

%Storm Flow

%MB-1 10,284,699 3,109,881 7,465,052 30.2% 72.6%E9 49,745,086 14,065,627 35,679,459 28.3% 71.7%

E10 11,986,940 2,595,844 9,272,129 21.7% 77.4%

E11 12,840,648 4,407,686 8,432,962 34.3% 65.7%

Sum 221,367,437 76,834,411 144,752,056 34.7% 65.4%

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Event Detection

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Operational Shutdown Monitoring• Reservoir elevation

• Discharge and turbidity

• Plausibility checks for out of range values

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Flood forecasting•Create a flood forecasting system using

existing gage data•Establish a web-based warning network for

towns

Flood watch

Flood warning

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

• Nodes: represent catchments, junctions or recording sites

• Links: define how the information flows in the model

Data Modeling

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Data analysis - Trends

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Water Meter Information• Efforts have been made by NYC to

automate water meter data acquisition

• WISKI could interface with the data stream and auto-generate reports

• Provide a historical database for data analysis

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Summary• WISKI Software represents a state of the

art tool for dealing with any time series data• Does not require special infrastructure –

can interface with any data source• Significant improvement in QA/QC & data

access• Success requires collaboration

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality

Acknowledgements

Kisters North America Kerri Alderisio Lorraine Janus DEP Management

New York City Department of Environmental ProtectionBureau of Water Supply

Water Quality