Barbara Minsker, U of Illinois, Urbana, IL David Maidment, U of Texas, Austin, TX August 30, 2014

The WATERS (WATer and The WATERS (WATer and Environmental Research Environmental Research

Systems Network) Network: Systems Network) Network: A Joint CLEANER and CUAHSI A Joint CLEANER and CUAHSI

VentureVenture

Barbara Minsker, U of Illinois, Urbana, Barbara Minsker, U of Illinois, Urbana, ILIL

David Maidment, U of Texas, Austin, David Maidment, U of Texas, Austin, TXTX

April 21, 2023April 21, 2023

The Need…and Why Now?Nothing is more fundamental to life than water. Not only is water a basic need, but adequate safe water underpins the nation’s health, economy, security, and ecology.

NRC (2004) Confronting the nation’s water problems: the role of research.

● Water use globally will triple in the next two decades, leading to increases in erosion, pollution, dewatering, and salinization.

● Major U.S. aquifers (e.g., the Ogallala) are being mined and the resource consumed.

● Only ~55% of the nation’s river and stream miles and acres of lakes and estuaries fully meet their intended uses; ~45% are polluted, mostly from diffuse-source runoff.

● From 1990 through 1997, floods caused more than $34 billion in damages in the U.S.

● Of 45,000 U.S. wells tested for pesticides, 5,500 had harmful levels of at least one.

● Fish consumption advisories are common in more than 30 states because of elevated mercury levels (source: mostly fossil fuel combustion; mercury is a neurotoxin).

WATERS Network Grand Challenge from the November Joint CUAHSI/CLEANER Workshop

How are water quantity, quality, and related earth system processes affected by natural and human-induced changes to the environment?

• How do we detect and predict the effects of natural phenomena and human activities on the quantity, distribution, and quality of water across a range of scales?

• How do we manage, engineer, and adapt to aspects of the urban water cycle to achieve sustainable water use and availability for humans and ecosystems?

• How do hydrologic, biologic, geomorphic and chemical transformations in the atmosphere, surface and subsurface affect water quality over multiple space and time scales?

CLEANER Grand Challenge CLEANER Grand Challenge QuestionsQuestions (from All-Hands Meeting, Sept. 20-21, 2005)(from All-Hands Meeting, Sept. 20-21, 2005)1.1. How do we detect and How do we detect and

predict waterborne predict waterborne hazards in real time?hazards in real time?

2.2. How do we predict the How do we predict the effects of human effects of human activities on the quantity, activities on the quantity, distribution, and quality of distribution, and quality of water?water?

3.3. How do we improve water How do we improve water cycle engineering cycle engineering management strategies management strategies to provide water quantity to provide water quantity and quality to sustain and quality to sustain humans and ecosystems?humans and ecosystems?

WATERS Network WATERS Network InfrastructureInfrastructure• These questions cannot be adequately These questions cannot be adequately

addressed without observatory infrastructure:addressed without observatory infrastructure:– Network of field sites (15-20 nationwide)Network of field sites (15-20 nationwide)

• Large spatial scale to study complex environmental Large spatial scale to study complex environmental systemssystems

• Equipped with state-of-the-art sensors and other Equipped with state-of-the-art sensors and other instrumentationinstrumentation

• Technical staff to help with experiments and ITTechnical staff to help with experiments and IT

– Linked to national community via Linked to national community via cyberinfrastructurecyberinfrastructure• Computer hardware, networks, and software Computer hardware, networks, and software

• Creates a “collaboratory” for interdisciplinary Creates a “collaboratory” for interdisciplinary teams in different universities to collaborate teams in different universities to collaborate on large-scale researchon large-scale research

SYNTHESIS

CA

dx)x(r)x(Q

Server

Workstation

domainDNS

Disk array

Investigators

Education & Outreach

Science & ApplicationsCommunities MEASUREMENT FACILITY

& Sensor Development

OBSERVATORIES & Environmental Field

Facilities

CYBERINFRASTRUCTURE& Modeling

WATERS Network WATERS Network InfrastructureInfrastructure

The WATERS NetworkWATERS Network will feature: (a) sites with gradients across the range of human impacts (b) where possible, co-location of minimally impacted sites with other EO field sites (c) nested watersheds ranging from local catchments to major river basins to improve understanding of environmental processes across scales

WATERS Network CI WATERS Network CI PlanningPlanning• CLEANER Project Office (CLEANER Project Office (

http://cleaner.ncsa.uiuc.eduhttp://cleaner.ncsa.uiuc.edu; Minsker PI, ; Minsker PI, Jerry Schnoor & Chuck Haas co-PIs) Jerry Schnoor & Chuck Haas co-PIs) – Cyberinfrastructure Committee (Chairs Jeanne Cyberinfrastructure Committee (Chairs Jeanne

Van Briesen & Tom Finholt) is creating a CI Van Briesen & Tom Finholt) is creating a CI program plan in collaboration with CUAHSIprogram plan in collaboration with CUAHSI

• Two groups are creating CI demonstrations Two groups are creating CI demonstrations for WATERS Network:for WATERS Network:– CUAHSI Hydrologic Information System project CUAHSI Hydrologic Information System project

(Maidment, PI)(Maidment, PI)– NCSA Environmental CI Demonstration (ECID) NCSA Environmental CI Demonstration (ECID)

project (Minsker and Jim Myers co-leads)project (Minsker and Jim Myers co-leads)– We have proposed a draft common We have proposed a draft common

environmental CI architectureenvironmental CI architecture

http://cleaner.ncsa.uiuc.edu/

Environmental CI Architecture: Environmental CI Architecture: Research ServicesResearch Services

Create Hypo-thesis

Obtain Data

Analyze Data &/or Assimilate into Model(s)

Link &/or Run Analyses &/or Model(s)

Discuss Results

Publish

Knowledge Services

Data Services

Workflows & Model Services

Meta-Workflows

Collaboration Services

Digital Library

Research Process

Supporting TechnologyIntegrated CI ECID Project Focus

HIS Project Focus

Knowledge ServicesKnowledge Services• Help users find information they need Help users find information they need

quickly and effectivelyquickly and effectively

• Includes information from:Includes information from:– Archives of community documents, data, Archives of community documents, data,

workflows, models, collaboration transcripts, workflows, models, collaboration transcripts, etc.etc.

– Metadata, including automatically generated Metadata, including automatically generated metadata collected from other users’ activities metadata collected from other users’ activities and relationships among the metadata and relationships among the metadata (“knowledge networks”)(“knowledge networks”)

– Web crawls to create dynamic databases on Web crawls to create dynamic databases on web sites & topics of interest to the communityweb sites & topics of interest to the community

Knowledge Services Knowledge Services (cont’d.)(cont’d.)• Includes both “information push” and “pull”Includes both “information push” and “pull”

– Information push – make referrals to users based Information push – make referrals to users based on their interests and preferenceson their interests and preferences

– Information pull – standard user searchesInformation pull – standard user searches

• To provide comprehensive knowledge To provide comprehensive knowledge services, provenance of all user activities services, provenance of all user activities must be stored in metadatamust be stored in metadata– History and origin of all products (data, workflows, History and origin of all products (data, workflows,

documents, etc.) stored in a flexible and documents, etc.) stored in a flexible and expandable metadata scheme expandable metadata scheme

Knowledge Services: ECID Knowledge Services: ECID Technology DevelopmentTechnology Development

• Metadata harvesting Metadata harvesting from from allall CI activities CI activities enables comprehensive enables comprehensive knowledge servicesknowledge services– Using RDF & Kowari to log Using RDF & Kowari to log

“provenance” (source & “provenance” (source & links) of all objects in links) of all objects in “triples” (subject, object, “triples” (subject, object, property)property)• E.g., graph shows that E.g., graph shows that

data2 came from data1 data2 came from data1 using a meta-workflow using a meta-workflow

Knowledge Services: ECID Knowledge Services: ECID Technology DevelopmentTechnology Development• CI-KNOW (CI Knowledge Networks on the CI-KNOW (CI Knowledge Networks on the

Web) uses metadata to make referrals to Web) uses metadata to make referrals to usersusers– Built with social networking algorithms (Contractor)Built with social networking algorithms (Contractor)

James

Topic A

Jose

Sophia

Topic B

Topic C

Document A

Document B

Document CRobert searches for

cont

ains

auth

or

auth

or

author

author

author

contains

contains

contains

con

tains

Knowledge Network

James

Topic A

Jose

Sophia

Topic B

Topic C

Document A

Document B


cont

ains

auth

or

auth

or

author

author

author

contains

contains

contains

con

tains

James

Topic A

Jose

Sophia

Topic B

Topic C

Document A

Document B


cont

ains

auth

or

auth

or

author

author

author

contains

contains

contains

con

tains

James

Topic A

Jose

Sophia

Topic B

Topic C

Document A

Document B


cont

ains

auth

or

auth

or

author

author

author

contains

contains

contains

con

tains

Knowledge Network


Create Hypo-thesis

Obtain Data



Discuss Results

Publish

Knowledge Services

Data Services


Meta-Workflows


Digital Library

Research Process

Supporting TechnologyIntegrated CI

CUAHSI Hydrologic Information SystemCUAHSI Hydrologic Information System

Continental US Scale (coast to coast data coverage, HIS-USA)

1:500,000 scale

Regional Scale (e.g. Neuse basin)

1:100,000 scale

Watershed Scale (e.g. Eno watershed )

1:24,000 scale

Site Scale (experimental site level)

Site scale Mul

tisca

le d

ata

deliv

ery

A multiscale web portal system for accessing, querying, visualizing, and publishing water observation data and models for any location or region in the United States

Point Point Observation Scale (gage, sampling location)

North American Scale (e.g. North American

Regional Reanalysis of climate)

1:1,000,000 scale

NWIS

ArcGIS

Excel

Storet

NCAR

Observatories

Ameriflux

NCDC

LTER

Matlab

Access SAS

Fortran

Visual Basic

C/C++

CUAHSI Web Services

Some operational services

CUAHSI Web ServicesCUAHSI Web Services

Web ServicesLibrary

Web application: Data PortalYour application• Excel, ArcGIS, Matlab• Fortran, C/C++, Visual Basic• Hydrologic model• …………….

Your operating system• Windows, Unix, Linux, Mac

Internet Simple Object Access Protocol

Direct and Indirect Web Direct and Indirect Web ServicesServices

• DirectDirect web service web service– The data agency provides The data agency provides direct queryingdirect querying

abilityability into its archives through SOAP or into its archives through SOAP or OpenDAP OpenDAP (NCDC)(NCDC)

• IndirectIndirect web service web service– CUAHSI constructs a “web page mimic” CUAHSI constructs a “web page mimic”

service, housed at SDSC, that service, housed at SDSC, that programmatically mimics the manual useprogrammatically mimics the manual use of of an agency’s web pages an agency’s web pages (USGS, Ameriflux)(USGS, Ameriflux)

Observation Site FilesObservation Site Files

Ameriflux Towers Automated Surface Observing System

USGS NWIS Stations Climate Reference Network

Observation Site Map for USObservation Site Map for US

USGS NWISASOSClimate Research NetworkAmeriflux

+ others…….

Neuse Basin with all pointsNeuse Basin with all points

NWIS Streamflow and Water Quality

NWIS Groundwater

ASOS

NARR

Ameriflux

NWIS Streamflow and Water Quality

ASOSNARR

Ameriflux

Filtered Site MapFiltered Site Map

HydroObjects LibraryHydroObjects Library

• CUAHSI has CUAHSI has developed a developed a HydroObjects HydroObjects Library with web Library with web service wrappersservice wrappers that know where that know where to access each to access each web service and web service and how to interpret how to interpret its outputits output

User Application(Excel, ArcGIS, …..)

HydroObjects Librarywith web service wrappers

for NWIS, Ameriflux, NCDC, …

Direct or Indirect web services

Web data

Transfer of research resultsTransfer of research results• CUAHSI web services for NWIS were CUAHSI web services for NWIS were announcedannounced

at a cyberseminar on at a cyberseminar on Friday Oct 28Friday Oct 28• On On Wednesday Nov 2Wednesday Nov 2, Jason Love, from a private , Jason Love, from a private

firm, RESPEC, in Sioux Falls, South Dakota, firm, RESPEC, in Sioux Falls, South Dakota, posted on the posted on the EPA Basins list server:EPA Basins list server: ““Occasionally Occasionally one comes across something that is worth sharing; the one comes across something that is worth sharing; the CUAHSI Hydrologic Information Systems - Web Services CUAHSI Hydrologic Information Systems - Web Services Library for NWIS is a valuable tool for those of us interested in Library for NWIS is a valuable tool for those of us interested in rapidly acquiring and processing data from the USGS, e.g., rapidly acquiring and processing data from the USGS, e.g., calibrating models and performing watershed assessments.”calibrating models and performing watershed assessments.”

• He provided a tutorialHe provided a tutorial on how to use the services on how to use the services from Matlab (which CUAHSI had not developed)from Matlab (which CUAHSI had not developed)

• Technology transfer took Technology transfer took less than 1 week!less than 1 week!


Create Hypo-thesis

Obtain Data



Discuss Results

Publish

Knowledge Services

Data Services


Meta-Workflows


Digital Library

Research Process


Series and FieldsSeries and Fields

Features

Point, line, area, volumeDiscrete space representation

Series – ordered sequence of numbers

Time series – indexed by timeFrequency series – indexed by frequency

Surfaces Fields – multidimensional arrays

Scalar fields – single value at each locationVector fields – magnitude and directionTensor fields – several vectorsRandom fields – probability distribution

Continuous space representation

Digital WatershedDigital WatershedHydrologic Observation

Series

GeospatialData

Remote SensingFields

Weather and ClimateFields

Digital Watershed

A digital watershed is an overlay of observation series and fields on a geospatial framework

to form a connected database for a hydrologic region

ArcGIS ModelBuilder Application for Automated Water Balancing

Fields Series

Geospatial


Create Hypo-thesis

Obtain Data



Discuss Results

Publish

Knowledge Services

Data Services


Meta-Workflows


Digital Library

Research Process


Meta-WorkflowMeta-Workflow• Many of the observatory efforts involve:Many of the observatory efforts involve:

– Studying complex environmental systems that Studying complex environmental systems that require coupling analyses or models of different require coupling analyses or models of different components of the systems, often created by components of the systems, often created by different people, different people, • E.g., water flows drive contaminant transport in multiple E.g., water flows drive contaminant transport in multiple

media, and both affect ecological flora and faunamedia, and both affect ecological flora and fauna

– Real-time, automated updating of analyses and Real-time, automated updating of analyses and modeling that required diverse toolsmodeling that required diverse tools• E.g., spreadsheets, scripts, GIS tools, modelsE.g., spreadsheets, scripts, GIS tools, models

• “ “Meta-workflow” tools enable Meta-workflow” tools enable heterogeneous workflows to be coupled and heterogeneous workflows to be coupled and run on the desktop, on remote servers, or run on the desktop, on remote servers, or across the Computational Gridacross the Computational Grid

CyberIntegrator: ECID Project’s CyberIntegrator: ECID Project’s Meta-Workflow ToolMeta-Workflow Tool


Create Hypo-thesis

Obtain Data



Discuss Results

Publish

Knowledge Services

Data Services


Meta-Workflows


Digital Library

Research Process


Collaboration ServicesCollaboration Services• Environmental observatories will require:Environmental observatories will require:

– Tools for easy remote communication, such as:Tools for easy remote communication, such as:• Wikis for collaborative editing (integrated with word Wikis for collaborative editing (integrated with word

processors)processors)• Instant messenger and chat roomsInstant messenger and chat rooms• Voice over IP and videoconference connectionsVoice over IP and videoconference connections• Screen sharingScreen sharing

• Numerous technologies exist, but are not:Numerous technologies exist, but are not:– Easily integrated with data and analyses for Easily integrated with data and analyses for

technical discussionstechnical discussions– Scalable to large groups across different Scalable to large groups across different

platformsplatforms– Integrated with knowledge services to support Integrated with knowledge services to support

collaborative knowledge sharing within & collaborative knowledge sharing within & across communitiesacross communities

CyberCollaboratory is the CyberCollaboratory is the ECID project’s prototype ECID project’s prototype collaboration service. To collaboration service. To check it out, create an check it out, create an

account ataccount at

http://cleaner.ncsa.uiuc.edu/chttp://cleaner.ncsa.uiuc.edu/cybercollabybercollab

http://cleaner.ncsa.uiuc.edu/cybercollab

http://cleaner.ncsa.uiuc.edu/cybercollab

A New Paradigm for A New Paradigm for ResearchResearch• WATERS Network will create a new WATERS Network will create a new

paradigm for environmental researchparadigm for environmental research– Shared infrastructure at large scalesShared infrastructure at large scales– Interdisciplinary teams collaborating remotely Interdisciplinary teams collaborating remotely

to address complex environmental issuesto address complex environmental issues• New paradigm will enable improved New paradigm will enable improved

understanding & management of large-understanding & management of large-scale natural environmental systemsscale natural environmental systems

• Cyberinfrastructure can create a Cyberinfrastructure can create a nationwide nationwide knowledge networkknowledge network for for environmental researchersenvironmental researchers– Supports all research & education, regardless Supports all research & education, regardless

of their focusof their focus

AcknowledgmentsAcknowledgments• Contributors:Contributors:

– CLEANER project office & planning grant CLEANER project office & planning grant teamsteams

– NCSA ECID teamNCSA ECID team– CUAHSI HIS teamCUAHSI HIS team

• Funding sources:Funding sources:– NSF grants BES-0414259, BES-0533513, NSF grants BES-0414259, BES-0533513,

and SCI-0525308, EAR-0413265and SCI-0525308, EAR-0413265– Office of Naval Research grant N00014-Office of Naval Research grant N00014-

04-1-043704-1-0437

Documents

Barbara Minsker, U of Illinois, Urbana, IL David Maidment, U of Texas, Austin, TX August 30, 2014