Restful Patterns

8/4/2019 Restful Patterns

1/71

Hosted and Sponsored by

Gold Sponsor

Copyright 2011 Open Geospatial Consortium

OGC RESTful Services

Common Patterns78th OGC Technical CommitteeBoulder, Colorado (USA)

Pat Cappelaere

September 22d, 2011


2/71

OGC RESTful ServicesCommon PatternsPat Cappelaere

Panagiotis (Peter) A. Vretanos

Volker Mische

Version 14

9/21/2011

With feedback from:

Arnulf ChristiSimon CoxJohn HerringCarl ReedRaj SinghPeter SchutDon Sullivan


3/71

OGC TC Boulder Sept 22, 2011

3

This presentation is Not about WhatRESTis

The current standards need to evolveand meet the needs of a large andgrowing community,

This is What We are Hoping RESTcanDoforOGCServices

Not about SOA vs. ROA either


4/71


Agenda

Part 1

Business Case

ROA

CRUD

Custom REST Actions

Resources, Patterns & Facades

Benefits

4

Part 2

Multi-faceted Example

Asynchronous Transactions Resource Representation

Computational Processes

Discovery

Caching (Etags)

Pubsub & Notifications Data & Activity Feeds

RESTful SecurityBackups Languages & Genomics WFS Queries, SPS, WPS, WCPS, WCS, WMS, SOS examples


5/71


What we want to see are very simple standards,and some of the OGC standards are verycomplicated. If I had one plea to that community itwould be to simplify standards, otherwise people

wont adopt and deploy them and put them intopractice.

5

Jacqueline McGlade

Executive Director European Environment Agency

And Newest OGC Board Member

Business Case: What Some Customers Want


6/71


7/71


Whats in Demand Right Now? 7

Node.js = Javascript on server-side (using asynchronous eventing)

This will have a big impact to new web service development

Upcoming NewDevelopments

Act as Enablers

for Web Services

We need to pay attention to this!

Javascript -> JSON and -> Async Services


8/71


OGC ROA Business Case

OGC Language Has Naturally Evolved TowardsIncreased Complexity Over The Years

And Entering A Period Of Modernization Of LanguageTowards Simplification

We Can Take A Step Back And Look Across All Services

Because We Have Learned Something

We Have A Unique Chance At Harmonization Of Services

And Make It Simpler, Faster To Learn And Cheaper To Use

8


9/71

Opportunity to Use Consistent Patterns Across ALLServices

No New Operations or Verbs/Words

(such asGetCapabilities, SubmitTask, ExecuteProcess,

ProcessCoverage, DescribeProcess, DescribeData,GetStatus)

Focus on Core Concerns


Resource Oriented Architecture9

There are still some semantics issues and that terms

and definitions for OGC usage need to be

finalized but not a topic for this presentation

And Cross Cutting Concerns

Caching (Etags) Pubsub and Notifications

Discovery

Search

Security...


10/71


Resource Oriented Architecture

Everything is a Resource (and has a URL)

What is a Resource?

Something that a user can create[, read,

update, delete] Use Simple HTTP methods ONLY: POST, GET,

PUT, DELETE

Focus on User and its access to the data(not operations)

Keep it Simple

GOAL: Client Implementation in a few hours

(Or NeoGeographers and other datamashers will move on to something else)

10

OBVIOUS

This is not ROA vs. SOA but given ROA


11/71


Custom REST Actions

Still possible

At collection level

/sps/tasks/completed

At resource level

/sps/tasks/1/complete

/sps/tasks/1/edit or new

Could be used for

Halt, resume, abort

11

Lets start thinking in URLs, resources, and APIs Geoffrey Grosenbach, peepcode.com

Theres good URL design, theres bad URL design, and theres everything in

between its subjective -- Kyle Neath, warspire.com

NOT OBVIOUS


12/71


OGC Resource Patterns

WPS

Processes

SPS

Tasks

Feasibilities

WFS

Features

Queries

WCPS

Processes

Algorithms

WfCS

Workflows

WCS Coverages

WMS Maps

We need common patterns to build our services

SOS

Observations

Pubsub/WNS

Notifications

Subscriptions

12

Core Concerns


13/71


From Simple OGC Services To Multi-facetedApplications

13

WFS

SPS

WMS WPS

WCS WCPS

WNS

WCS

SPS

WMS

WMS WfCS

WCS

WPS

WNS SOS

WMS

WFS WCTS

Users can now look at data different

ways using Facades

We need harmonization and common patterns..


14/71


Benefits

Faade APIs can be used for OGC services (WFS,WPS, WCPS, WfCS, SPS)

NO New WORDS/VERBS to be defined

(No GetCapabilities, SubmitTask, ExecuteProcess,ProcessCoverage, DescribeProcess, DescribeData,GetStatus)

Consistent across services

Simpler To Learn

Faster to Use

Therefore Cheaper for this growing market

14

A Simplified Experience!


15/71


What we want to see are very simple standards,and some of the OGC standards are verycomplicated. If I had one plea to that community itwould be to simplify standards, otherwise people

wont adopt and deploy them and put them intopractice.

15

Jacqueline McGlade

Executive Director European Environment Agency

And Newest OGC Board Member

Simpler To Meet Some Customer Expectation


16/71


Agenda

Part 1

State of the World

ROA CRUD

Custom REST Actions

Resources, Patterns & Facets

Benefits

16

Part 2

Multi-faceted Example

Asynchronous Transactions Resource Representation

Computational Processes

Discovery

Caching (Etags)

Pubsub & Notifications Data & Activity Feeds

RESTful SecurityBackups Languages & Genomics WFS Queries, SPS, WPS, WCPS, WCS, WMS, SOS examples

ROA has to address:


17/71


From Stovepipe Services to

Facades

17

WFS

SPS

WMS WPS

WCS WCPS

WNS

WCS

SPS

WMS

WMS WfCS

WCS

WPS

WNS SOS

WMS

WFS WCTS

Users can now look at data different

ways using Facades

Leveraging harmonization and common patterns..

http://en.wikipedia.org/wiki/Facade_pattern


18/71


Modis Flood Extent Server

User Stories

User requests flood extent for tomorrow(coincidental with an EO-1 imaging)

SPS Facade

User requests flood extent overlay

WFS Facade

User request flood extent from last 15 days

WCS Time series

User requests prediction for next 8 days

WPS Facade

18

Facade: Provide a unified

interface to a set ofinterfaces in a

subsystem. Facade

defines a higher-level

interface that makes

the subsystem easier

to use. [GoF, p185] Wrap a complicated

subsystem with a

simpler interface.

API Must Be Consistent


19/71


Facade Implementation

Separation of Concerns

Core Concerns

Specialized Resources of Interest to Users Optional but Recommended Cross-Cutting Concerns

Security, Logging, DIscovery, Search, Pubsub

Use HTTP Operations

Result as an Atom/JSON Feed with links to Data

19


20/71


WCPS Faade Design Example 20

Core Concerns Cross Cutting Concerns

Optional But Recommended

WCPS Faade API

Logging Security

/

/WCPS


21/71


WFS Faade Design Example 21

Core Concerns Cross Cutting Concerns

Optional But Recommended

WFS Faade API

Logging Security

FeaturesFeatureType1FeatureType2..

/

/WFS


22/71


Data Interchange Protocol

HTML (of course, if we deal with users directly)

XML (May not be the easiest approach in many cases)

JSON (Emerging as de-facto standard)

Smaller footprint & easier to work with than XML

Javascript Native on Client Side

Javascript is becoming a language of choice on the serverside as well (Node, V8 engine)

Atom Because of browser native support and readers availability

Also Used for Notifications

Easy to extend -> Common Wrapper for Data Products

OGC Examples: GeoSync

22

About JSON feeds and XML feeds

Common Patterns


23/71


Alternate Representations Pattern

Using request headers (if you are a machine)

Accept

Accept-Encoding

Implicit using uri extension /geoactivities.json

/tasks.atom

Explicit using parameter

/geoactivities?fmt=json

/geoactivities?output=json

/geoactivities?alt=json

23Common Patterns


24/71


We seem to have gotten bogged down in queries, which arejust a simple example of the more general case of information

created by a computational process, where the computationalprocess is written/created by the client.

Raj Singh 9/9/2011

24

Computational processes created by client could be as a result of

executing user-defined WCPS algorithms, WPS-T processes, WfCS

workflows and/or WFS stored queries.

[Note: A simpler case exists when processes are hard-coded in the server.

WPS processes and SPS Tasks for example. No need for user uploads]

OGC Computational Processes


25/71


26/71


27/71


Step 2 - Execute

User creates a Handle by POSTing to relevantcollection

>POST /Handles

{ name: myHndl,

what: myThing,

Param1: xyz}

System returns 201, created and location URL

27

Handle CollectionsReplace with:

tasks

processes

transactions

jobs

This pattern applies to simpler case when processes are hard-coded (WPS

and SPS)

User wants to execute desired query/algo/process/task

What I want to execute

With opt parms


28/71


Step 3 Poll / Notify

User can then get a representation of thatresource in some format and poll for status

>GET /Handles/2345.json

{ name: myHndl,

what: myThing,

param1: xyz

status: started }

Or get notified via Pubsub when data isavailable

28

Not Recommended

Recommended


29/71


Step 4 Get Data

User can then get access to data when complete

>GET /Handles/2345.json

{ name: myHndl,

what: myThing,

param1: xyz

status: completed,

link: http://example/product.atom }

29

Resource Representation in

JSON format (but could be XML,

GML)

Link to actual data products

with additonal metadata,

mime-typesWARNING: This is the case where resource

representation is not the data itself


30/71


WFS Queries, WCPS Algorithms, WfCS

Workflows and Transaction Pattern30

Query or

Algorithm or

WorkflowPOST

GET1

Handle (or Transaction)

POST

GET2StatusLink

Atom feed

LinkLink

PNG

KML

Attrs

Create

Submit

For Execution

3 Wait for NotificationsGet Results4

This is NOTjust a User Interface

Only 2 resources involved!

Examples


31/71


So, Complex TransactionalProcesses

2 POST and 1 GET

[First POST may be optional for hardcoded processes]

31


32/71


Simple Queries.

1 GET[See OpenSearch]

32

More OGC Specific Examples in Backups


33/71


Common Discovery Pattern

Discovery Options

http OPTIONS / HEAD methods are weak (and require

many roundtrips) The Approach that the Google Discovery Service uses

is a better option to tie methods and representations

Client Libraries exist for many languages

JSON Schema necessary for Discovery & Validation

Natural to Javascript Client/Server sides (See Slide7)

Machine Readable as well

Examples

33

What I POST What I GET or PUT


34/71


Auto-Discovering the Discovery

Documents

Auto-Discovering the OpenSearch Description Documents Example What Description Document?

Value: Custom Search Within the Browser

Common Browser Experience

Auto-Discovering the Discovery Service Use HTML Page Headers!

https://portal.opengeospatial.org/files/?artifact_id=44919

Auto-Discovering PuSH Servers (See Annotations) Use Feed Links

Examples

34


35/71


ETags

For Efficiency / Cache Control

Dont send data that has not changed

For Data Integrity

Dont try to update data that has changed

Examples:

HTTP Header

Feed

35

HTTP/1.1 200 OK

Date: Fri, 30 Oct 1998 13:19:41 GMTServer: Apache/1.3.3 (Unix)

Cache-Control: max-age=3600, must-revalidate

Expires: Fri, 30 Oct 1998 14:19:41 GMT

Last-Modified: Mon, 29 Jun 1998 02:28:12 GMT

ETag: "3e86-410-3596fbbc"

Content-Length: 1040

Content-Type: text/html

Foo2006-01-23T16:25:00-08:00http://www.example.com/myFeed

Jo March


36/71


37/71


38/71

OGC TC Boulder Sept 19th 2011

38

Send an POST request to http://geobliki.superfeedr.com

hub.mode:! publishhub.url: ! http://geobpms.geobliki.com/geoactivities.atom

PING for OGC Services

Send an POST request to http://superfeedr.com/hubbub, with HTTP Basic

Authentication and the following params :

hub.mode: ! subscribe or unsubscribehub.verify: ! sync or asynchub.callback: ! http://domain.tld/your/callbackhub.topic: ! http//feed.you.want.to/subscribe/tosuperfeedr.digest: true if you want to receive daily digest notifications.

SUBSCRIBE/UNSUBSCRIBE for Clients

Simple API


39/71


40/71


40Notifications via Email via Chat


41/71


42/71


[Geo]Activity Feeds

We need an acceptable Geo Extension for location

Activity = Subject + Verb + Object/Target + Location

For Microblogging and Notifications

The Activity Streams format has already been adopted by

BBC, Gnip, Google Buzz, Gowalla, IBM, MySpace, Opera,

Socialcast, Superfeedr, TypePad, Windows Live, YIID, and

many others.

42

Links:

http://activitystrea.ms/

http://groups.google.com/group/activity-streams

Lets go beyond data feeds


43/71


Activity Example

43


44/71


Security

You cannot design web services withoutaddressing security issues

It has to be built-in!

But not obtrusive

Using concepts that the user is familiar with

44


45/71


Distributed Identity We may need to access users from other

organizations

You may need to establish trusted relationships

with other Identity Providers

Users need to be authenticated (sometimes withtwo or three factor authentication) and havevarious access control rights

Users need to be able to delegate their authoritiesto workflows

Consumer Applications have their Identity

45


46/71


46

Organization 1

Thats


47/71


47

Organization 1

Organization 2

Trust

Relationship

Organization 3

More Realistic Scenario!


48/71


OpenID / OAuth Hybrid

Users and Consumers Have OpenIDs

Consumers need to register with the OP (and

publish their Public Key)

User Information, Access Control Privileges andKeys are managed by the OP using SimpleRegistration and Attribute Exchange

48


49/71


THANK YOU!

Pat Cappelaere

[email protected]

Panagiotis (Peter) A. Vretanos

[email protected]

Volker Mische

[email protected]

49


50/71


BACKUPS

50

Languages & Genomics WFS Queries, SPS, WPS, WCPS, WCS,

WMS, SOS examples


51/71


How Many Specifications does OGC have?

How Are Specifications Being Developed?

What is The Total Number of Pages?

What is The Total Number of SOAP Verbs?

How Has This Evolved Over The Years?

How Many Schemas Do We need to Master For ASignificant Deployment?

51

Status Check


52/71


52Similar Language Evolution or

Genomes for that matter

http://www.nature.com/embor/journal/v7/n8/full/7400756.html#B4

One would need to know about 22,000 characters to attain 99.99%

coverage of ancient Chinese


53/71


Evolution Phase 2

Modern Chinese

Only 4,600 characters are needed to attain 99.99% coverage

53

The current trajectory of OGC is unsustainableToo many schemas operationsspecs

We need to modernize the approach

This is What I am Hoping RESTcanDoforOGCServices


54/71


55/71


55

Se curi ty

Client

WCPS

FacadeGeoActivitiesAlgori thms

Push

Serve rProcesses

POST

201

Client

GET

200

WCPS POST / GET Processes (to execute specific algorithm and get data)

Subscri ber

Client

Pro cess Comp letedPING

Notification

Security Transactions with OpenID Server ignored

Text


56/71


56

Se curi ty

Client

WCPS

FacadeGeoActivitiesQueri es PublisherFeatures

POST

201

Client

GET

200

WFS POST / GET Stored Queries

Su bscri ber


WFSFacade

WFS POST / GET J b ( ifi d d )


57/71


57

Se curi ty

Client

WCPS

FacadeGeoActivitiesFeatures

Push

Serve rJobs

POST

201

Client

GET

200

WFS POST / GET Job (to execute specific query and get data)

Su bscri ber

Client

Job Comp leted

PING

Notification


WFSFacade


58/71


WFS Query Example

User creates the query >POST /wfs/queries

{ name: myQuery,

query=SELECT ?parcel ?hwy

WHERE { ?parcel rdf:type my:CommercialParcel .?hwy rdf:type my:Street .?parcel ogc:touches ?hwy

}

}

System returns 201 created, location /wfs/query/fbs4769

>GET /wfs/query/fbs4769.json to get it back in json format (or usePUT/DELETE to edit/delete)

{ name: myQuery,

query=SELECT ?parcel ?hwyWHERE { ?parcel rdf:type my:CommercialParcel .

?hwy rdf:type my:Street .?parcel ogc:touches ?hwy }

}

58

GeoSPARQL

example

But could be WFSQuery, SQL, XQL,

XQUERY


59/71


Next steps

Execute the query

>POST /wfs/jobs

{ query: myQuery

output: kml }

System returns 201 created, location /wfs/jobs/169

Poll for status (or rather wait for notification)

>GET /wfs/jobs/169.json

{ query: myQuery

output: kml,

created_at: 9/10 2011

status: processing }

59


60/71


61/71


62/71


63/71


SPS Feasibilities (Story #1)

Submit the feasibilities request

>POST /sps/feasibilities

{

latitude: -89.1,

longitude: 112.34 }

System returns 201, created, location: /sps/feasibilities/539 (for example)

Poll for status and eventually get the data (or get notified)

>GET /sps/feasibilities/539.json

{ ..

latitude: -89.1, longitude: 112.34,created_at: 9/10 2011 10:30:02,

updated_at: 9/10 2011 10:50:02

status: completed,

link: http://www.example.com/sps/feasibilities/products/2456.atom}

63

a 1 POST, 1 GET as well

Example for illustration purpose. Will

depend on actual asset to task


64/71


Feasibilities feed

Feed could return

Not feasible if it cant be done given inputs

Or N entries Each entry could contain:

Link to prepopulated form with additional

parameters that might be needed for a task

request.

User could then simply click on chosen

feasibility to submit task (HATEOAS)

Machine could just get parameters back and

use micro-template to POST request

64


65/71


66/71


67/71


WCPS Examples

User Stories User adds/edits custom algorithms using some representation

User deletes algorithms

User executes algorithms User gets data

>POST /wcps/algorithms Some resource entry in some format (JSON, XML)

>DELETE /wcps/algorithms/:id

>POST /wcps/processes Some resource entry in some format

>GET /wcps/processes/:id.:fmt?

67


68/71


WCS Examples

User Stories

User lists existing coverages

User retrieves specific coverage User search for specific coverage

>GET /wcs/coverages

>GET /wcs/coverages/:id

>GET /wcs/coverages/:filter [OpenSearch pattern]

68


69/71


WfCS Examples

User Stories User adds/edits workflows using some representation

User deletes workflows

User executes workflows

User gets data

>POST /wcps/workflows Some resource entry in some format (JSON, XML)

>DELETE /wcps/workflows/:id

>POST /wcps/processes Some resource entry in some format

>GET /wcps/processes/:id.:fmt?

69


70/71


WMS Example

User Stories:

User retrieves a particular map for location and zoom

level

> GET /wms/map/:mapid/:filter?

70

OpenSearch Pattern


71/71


SOS Example

User Stories

User retrieves list of observation

User retrieve specific observation

User searches for an observation

>GET /sos/observtions

>GET /sos/observations/:id

>GET/sos/observations/:filter [OpenSearch Pattern]

71