Metadata, Provenance and Web Service for Spatial Analysis -- the case of spatial weights

Metadata, Provenance and

Web Service for Spatial

Analysis

--the case of spatial weightsLuc Anselin, Sergio Rey, Wenwen Li

GeoDa CenterSchool of Geographical Sciences and Urban

PlanningArizona State University

•Some Specific Project Goals

• Integrate and sustain a core set of composable, interoperable, manageable, and reusable CyberGIS software elements based on community-driven and open source strategies

•Challenge

•most current spatial analysis/spatial econometrics software written for single CPU

•rethink and rewrite analytical, algorithmic and processing facilities to integrate into a cyberinfrastructure

•address lack of interoperability

•Spatial Econometrics Workbench

•framework for supporting spatial econometric research in a cyberscience era (Anselin and Rey, IJGIS 2012)

•Leverage PySAL and CyberGIS

•Support scientific workflow

•PySAL

•open source library of Python routines for spatial analysis: geocomputation, spatial weights, spatial autocorrelation, spatial econometrics, regionalization

•http://pysal.org

•hosted on github

•PySAL Progress Report

•current version is 1.6 (7th release)

•3.5 years of on-time bi-annual releases

•20,000+ downloads (10,000 in 2012)

•recognized in open source scientific community - Anaconda

Anaconda for big data analytics

•Migrating to CyberGIS

•performance = need for parallelization + refined algorithms

•interoperability = provide functionality as web services

•replicability: need for metadata and provenance tracking

•Example: Spatial Weights

•includes spatial data source, type of weights (e.g., contiguity, distance), any standardization or manipulation (e.g., higher order)

•Lack of Interoperability

•different implementations

•no standards

•duplication of efforts

•hinders interoperability and workflow chaining

•Example: Weights Formats in PySAL

Example: PySAL spgregwhat do we know about south_k6.gwt and

south_ep_k20.kwt

•Conceptual Framework

•separate data source from operations

•data source: polygon or coordinate files with standard metadata (projection, origin, etc.)

•operations: weights metadata

weights vocabulary

weights metadata structure (wmd)

•Web service implementation(OGC WPS)

•wraps PySAL weights module

•(re)creates weights object from information in wmd file

•makes weights object available as a file

Workflow

Weights Parser

Dispatcher

Output

wmd file(json)

Weights

Metadata

Illustration

•Generate Weights from Shapefile

•NAT.shp available on server

•output format = gal

•Get Request

• http://spatial.gdta.asu.edu/cgi-bin/wps.cgi?request=Execute&service=WPS&version=1.0.0&identifier=weights_ws&status=false&datainputs=[outputformat=gal;metadata={"input1":{"type":"shp","uri":"http://toae.org/pub/NAT.shp"},"weight_type":"rook","transform":"O", "parameters":{"p":2,"k":4}}]

metadata input

Server Response

Sample gal output

http://spatial.gdta.asu.edu/wpsoutput/e66df128-14ed-11e3-bde9-0050455c0671.gal

metadata (wmd) file

http://spatial.gdta.asu.edu/wpsoutput/e66df128-14ed-11e3-bde9-0050455c0671.wmd

Performance Evaluation•How does PySAL scale when the

amount of input data increases?

•Is the overhead of web service framework acceptable?

•How does the web service framework scale in handling massive concurrent requests?

Scale-up vs. Scale-out solution

•Scale-up

•High-end computer

•Configuration• Processor 2 x 2.93 GHz Quad-Core Intel Xeon

• Memory 16 GB 1066 MHz DDR3 ECC

• Software Mac OS X Lion 10.7.4 (11E53)

•Scale-out:

•Web server cluster

Web Server Cluster

•Performance •experiment using grid layout for

N = 10,000 to N = 100,000

•rook contiguity and k nearest neighbors (k = 4)

•input shape files on server in Utah, web service on server at ASU

•Experiment 1

•Timing: average over 5 experiments

•web server overhead, data transfer and computation

•explore effect of data size

time for rook and KNN contiguity

•Experiment 2

•Scalability of web service framework

•High-end computer (8-cores)

•Cluster (4 computing nodes, each has 2-core)

•Total processing time

•Speed up

Total processing time

Speed-up

•Experiment 3

•Scalability of the cluster by adding more computing nodes

•Average response time

•128 concurrent requests

•Dataset: 10,000 polygons

Scalability - cluster

Next Steps

•Towards a Standard

•refine specification: flexible, expandable, deal with edge cases

•improve performance (parallelization)

•implement seek operations on distributed files

•interoperability with other software

Thank you!

Metadata, Provenance and Web Service for Spatial Analysis -- the case of spatial weights

Documents

Provenance: Information for Shared Understanding€¦ · Provenance: Information for Shared Understanding. Approved for Public Release: ... Useful provenance information is obtained

Provenance) Whata%Concept!% · Provenance) Whata%Concept!% Reporton)the) iTRUST) Interdisciplinary)Workshop)on) Provenance) May)14,)2015))

Provenance Semirings

Tufts University - Spatial Statistics: What is it and … · Web viewSpatial Regression adds spatial weights into a regression analysis to include space into the model. This tutorial

Provenance Situations: Use Cases for Provenance on Web ...gil/slides/Provenance-WebArchitecture-10-28-10.pdf · W3C Provenance XG October 28, 2010 2 Provenance and Web Architecture:

Geochemical Discriminant for Provenance Characterization and … · 2016-06-14 · Geochemistry, Shale, Provenance ... Provenance, Paleogeography, Dahomey Embayment 1. Introduction

An Introduction to Spatial Regression Analysis in Rlabs.bio.unc.edu/.../documents/anselinintrospatregres.pdf7 Spatial Weights The Moran I test statistic for spatial autocorrelation

Confession #1 Provenance and Causality Provenance-based Belief

Spatial Regression Analysis in R A Workbook · 2016-06-28 · spatial weights in spdep, Anselin 2003a), and the “Regression” tutorial (An introduction to spatial regression analysis

2 Modeling the Spatial Structure of Areal Unitsese502/NOTEBOOK/Part...2.1 Spatial Weights Matrices To model spatial relations between areal units, we now let n denote the number of

Metadata Provenance Tutorial Part 2: Interoperable Metadata Provenance

Open Provenance Model Tutorial Session 7: Open Provenance Model Vocabulary

University of Dundee Estimation of the Spatial Weights

PROVENANCE STUDIES AND SEDIMENTOLOGY OF …ijsit.com/admin/ijsit_files/PROVENANCE STUDIES AND SEDIMENTOLO… · PROVENANCE STUDIES AND SEDIMENTOLOGY OF OGWASHI-ASABA FORMATION, ANAMBRA

The Provenance of Provenance in Germanic Areas

YesWorkflow: More Provenance Mileage from Hybrid Provenance Models and Queries

Ontology-driven Provenance Management in eScience: An ...raghavam.github.io/files/Provenance-ODBASE2009.pdf · effective analysis of provenance information requires well-defined query

Provenance Projects

Geochemical provenance and spatial distribution of ... · PDF fileThe dental and skeletal fluorosis is irreversible and no treatment exists. ... analysis of the other hydrogeochemical

Karma Provenance Framework v2 Provenance Challenge Workshop/GGF18