Real-World RESTful Service Development Problems and Solutions

Masoud Kalali @MasoudKalali

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Speaker

• Masoud Kalali

• Works at ORACLE

• Software engineer, author, blogger…

• @MasoudKalali

Program Agenda

Introduction

Request and Response….

A little bit of security

Performance matters!

More on Request and Response

1

2

3

4

5

Introduction…To jog our memories

CachingRate

LimitingAuthz Logging

RESTfulService

Endpoint

Micro Services

Content Neg.

HTTP Request

HTTP Response

HTTP Response Status

PaginationHATEOAS

CDN

EIS

Request and Response….Content Negotiation

Flexible Request and Response - The problem

• Produce multiple format of the representation

– JSON

– XML

– HTML

• Produce multiple variation of the representation

– For privileged user compared to non privileged user

– For older or newer client versions

– For different type of clients

Flexible response types

Flexible Request and Response

• Produce multiple representation of the same resource

– Based on HTTP headers, accept and media-type (custom/standard)• List of standard media-types: http://www.iana.org/assignments/media-types/media-types.xhtml

• Custom domain oriented media-types, e.g application/vnd.com.oracle.paas-service+json

– Using URI pattern e.g:• http://api.domain.com/v1.1/customers/1/orders.xml

• http://api.domain.com/v1.1/customers/1/orders.json

– Agent driven• Client decide what representation to choose

• Server sent different representations (300 or 406 status and required headers)

– Combination of all

Content Negotiation

Flexible Request and Response – The solution

• On the server side

–@Produce• Should have matching media-type with request accept header

• Non matching results in 406 - Not Acceptable

–@Consume• Should have matching media-type with request content-type header

• Non matching results in 415 - Unsupported Media Type

• On the client side– Set correct content-type

– Set expected accept header

Content Negotiation: JAX-RS and flexible response types

Request and Response….Resource Versioning

Resource Versioning: The Problem

• Request and or response evolves to be incompatible

– Some attributes are removed from request payload or required in the payload

– Some attributes are added or removed or changed semantically in the response

• Business semantic evolves to become incompatible– E.g the default is no longer set to be a small coffee cup

– The provisioned service, in a PAAS environment, has caching included by default

– Etc.

How to evolve a resource?

Resource Versioning: Solution alternative

• Version added to resource (URI) (Facebook and Twitter)• /v1.1/resource/path/orders

• /resource/path/orders?version=1.1

• Client is locked to the version

• Easy to maintain in the server side

• Requires all or none migration of the clients

• Problems like linked resource address stored in client side…

How to evolve a resource?

Resource Versioning: Solution alternative

• Version is negotiated as part of the request using accept header

– Accept=application/vnd.com.oracle.paas.service-v1.1+json

– Accept=application/vnd.com.oracle.paas.service-v2.1+json

• Server code need to handle all supported previous versions– Same endpoint handling all versions and combination of them

• Easy for clients to use what thy want

• GitHub uses this format

How to evolve a resource?

Resource Versioning: Solution alternative

• No versioning at all!

– There is one interface that evolve every e.g 2 years• Clients must migrate to the new API

• Old api will be shutdown

– Easy to maintain

– Easy to migrate

• Hard to convince clients to migrate before the cut-off

How to evolve a resource?

Request and Response….Validation

Validating Request: Problem

• When and how to validate the requests?

– Are parameters in expected format

– Is the payload Json as it is supposed to be?

• When and how to verify the requests?– Are the parameters meaningful? E.g is there any order with that ID

– Is the payload in expected JSON schema?

Validation before action!

Validating Request: Solution

• Validation goes with versioning and content negotiation

• Unified validation patterns across the codebase

– Codified response format

– Unified response body

• Use annotations and standard validation as much as possible– Supports common media-types

– Unified with custom coding/template being added

Validation before action!

Request and Response….Exception Handling

Exception Handling

• Please don’t send back stack trace

– Use an ExceptionMapper as Provider as last line of cleanup!

– Unless in development environment (with some considerations)

• Codify all failures and include cause/action in the response– Use right http status code

– Add application level code for codifying issues

– Add human understandable message

There are always unforeseen corner cases!

More on Request/Response

• Response status codes– There are more than 200, 404 and 500!

• Unified response style– Predictability of response body– Application specific error code– Human readable message– cause/solution

• debugability and traceability– debugified response – ECID– Unified Logging (Facebook Scribe, Google Dapper, Apache Kafka)

How should a resource respond to a request?

A little bit of securityAuthenticationAccess controlAuditing

Authentication

• Authentication enabled for all resources

• Happens before any other validation

• Exclude the resource patterns that requires no authentication

• No access without validating the authentication token

Know who is requesting a resource

Access control

• Happens after detecting a valid authentication

• Requests are easy to check

– Unless column level checks are required, can be done in simple filter

– Column level access control can be done using media types

• Happens before any validation– unless params are being used as part of the access check

• JAX-RS 2.1 is going to support declarative security

Check who can access a resource

Auditing/ Access logs

• Keep a rich access log

– Depending on the server you use

• Include the usual who, when, what

• Try using W3C Extended Log File Format if supported by server

• Configure security realm for logging

• Think ahead about incident detection/isolation, etc.

Keep record of security incidents!

Performance matters!Caching

Problem: Don’t hit the servers too often

• Don’t reproduce the same response twice

• Don’t use bandwidth whenever possible

• Counter network reliability issues

• Counter server downtimes

Being stingy with the resource usage is OK!

Solution: Caching

• Local Cache

• Proxy Cache

• reverse-proxy (cache)

• Server(Application) Level Cache

Being stingy with the resource usage is OK!

Caching

• Use HTTP caching features

• Client aware of provided caching support

• Server evaluate caching related headers

• intermediately hops

• Types of Caching Headers

– Absolute Caching Headers

– Conditional Caching Headers

Application Level Cache

Caching

• From Server side:

– Cache-Control and directives

– Last-Modified

Absolute Caching

HTTP/1.1 200 OK Content-Type: application/jsonCache-Control: private, max-age=86400Last-Modified: Mon, 08 Sep 2014 16:30:00 GMT

• private

• public

• no-cache

• no-store

• max-age (overrides Expires)

Cache Control Directives

curl -v -X GET -H "If-Modified-Since:Mon, 08 Sep 2014 15:08:27 GMT" http://.../simple/1

Caching

• From client side send headers:

– If-None-Match = "If-None-Match" ":" ( "*" | 1#entity-tag )

• At the server side produce headers:

– Etag, when last-modified is hard to determine or not accurate enough

Conditional

Caching

• Can be used for conflict resolution

• Cache on GET request

• Invalidate cache on PUT, POST or DELETE

• Periodically purge cache entries

• Cache invalidation, eviction is not deterministic

• Give http://www.jboss.org/resteasy a try

– Provides @Cache and @NoCache

– Extension to JAX-RS

More on Caching

Performance matters!Partial Updates & HTTP PATCH

Problem: Wasting processing and bandwidth when updating

• Updates are not usually full representation mutation

• No need to process the whole representation to update one attribute

• No need to re-persist the whole representation on change

Only update what needs to be updated!

Solution: Partial Updates & HTTP PATCH

• Partial Updates with PUT

• Partial Updates with POST

• Partial updates with PATCH

• JSON Patch is the future

Only update what needs to be updated!

Partial Updates & HTTP PATCH

• Partially update a JSON document

• Works with HTTP PATCH

• Requires special body syntax and directives

JavaScript Object Notation (JSON) Patch, RFC 6902

PATCH /coffee/orders/1234 HTTP/1.1Host: api.foo.comContent-Length: 100Content-Type: application/json-patch[{“op”:"replace", ”path”: "/status", "value": "COMPLETED"}]

Partial Updates & HTTP PATCH

• Supports six operations in the payload

– op : can be “add”, “replace”, “move”, “remove”, “copy” or “test”

• Another three attributes to describe the op

– path: Location of the target attribute in the JSON document

– value: The new value to be added or to replace another

– from: (Only for move op) specifies the source location

JavaScript Object Notation (JSON) Patch, RFC 6902

Performance matters!Asynchronous And long running jobs in REST

Problem: Serving more requests on the serverside

• Fixed server side resources

• Heavyweight container threads

• Not keeping container resource longer than needed

Don’t keep unnecessary resources for where not needed!

Solution: Asynchronous And long running jobs in REST

• On the serverside:

–@Asynchronous: Annotate a sub-resource as Asynchronous

– AsyncResponse: Provides results and actions on the running request• setting timeout

• registering callbacks

• resume, cancel suspended request processing

• updating the response

–@Suspended: To inject a suspended AsyncResponse into a sub-resource parameter

Don’t keep unnecessary resources for where not needed!

Asynchronous And long running jobs in REST

• On the serverside:

– CompletionCallback: Async response processing completion callback• Response processing completed and sent to client

• Response processing failed with exception

– ConnectionCallback: Client server connection events callback• Client is disconnected abruptly (before or during writing back the response)

Server side callbacks

Asynchronous And long running jobs in REST

@GET @Produce(“application/json”)

@Asynchronous

public void getOrder(@Suspended AsyncResponse ar, String orderId) {

final String result = prepareResponse(orderId);

ar.resume(result)

}

Some small sample code

Future<Coffee> future = client.target(“/coffees/orderId").request().async().get(Coffee.class);try {

Coffee coffee = future.get(30, TimeUnit.SECONDS);} catch (TimeoutException ex) {

//}

Client Code:

Server Code:

** Alternative to Future is using InvocationCallback to get called when response is back

Few more topicsHATEOAS

Problem: Being perfect according to REST maturity model

– Letting the resource graph describe itself

– Describing the resource graph by name

– Removing dependencies on fixed resource addresses

– Letting clients have easier understanding of the API

How to solve:

{"order": "120""links": [ {"rel": "self", "href":"http://api.dom.com/orders/120"}, {"rel": "cancel", "method":"delete", "href":"http://api.dom.com/orders/120"}, {"rel": "shippingDetails", method: "get", "href":"http://api.dom.com/orders/120/shipping"}]

}

RESTful design maturity levels (Richardson Maturity Model)And the maturity model

Level 0: XML RPC/Remoting – One resource and one verb (POST) for everything!

Level 1: Multiple resources, not correct use of verbs

Level 2- Proper use of HTTP verbs

Level 3- HATEOAS – Resource graph discovery

Mat

uri

ty G

row

th

HATEOAS

• REST maturity model– The glory comes when you are at HATEOAS level!

• Why to` use HATEOAS– Describe the resource graph by name– Remove dependencies on fixed resource addresses– Let the resource graph describe itself– Let clients have easier understanding of the API

• When to use HATEOAS– When one resource has some dependent resources– What is allowed and what is not (logic)– What is accessible to a user and what is not (security)

Let the resource graph describe itself

JAX-RS and HATEOAS

• No standard API so far but the following to the rescue – Link– UriInfo– UriBuilder

• JAX-RS extensions and Jeresy specific APIs– Use https://code.google.com/p/jax-rs-hateoas/ extension– Use Jeresy specific API

• Use Spring MVC

• Use RESTeasy

• It is not end of the world if you don’t!

Java/EE HATEOAS support out of the box

Few more topicsLocalization/Internationalization

Localizations and Internationalization

• One time used messages (not persisted for retrieval)

– Decide the locale using accept-language header

– Produce the right locale representation

• Messages would be used by multiple times(persisted messages)– Codify every text and set of argument

– Let a last layer filter decide how to process the message code and formatting params

– Let the last layer filter decide what to be the date format and , currency should be

– Let the last layer filter rewrite the response replacing the templates

– Keep every message in a memory cache

Different locale getting the right format and style of messages

Few more topicsUsage Throttling

Problem: Abusing the API, categorizing the users

– Better resource allocation and management

– Prevent and handle abuse

– Provide better support for premium users

Keep tap on how many request one user can send!

Solution: Resource Throttling

• Why use Rate Limiting?

– Prevent and handle abuse

– Provide better support for premium users

• How Rate Limiting works?– Servlet filter sits in front

– User user tokens or IP addresses as identifiers

– Use in-memory cache for book keeping

– Prevents requests reaching endpoints when limit reached

– Reset/update cached counters when needed

Keep tap on how many request one user can send!

Throttling Patterns

• Response status, headers

– HTTP 429 Too Many Requests error code

– Retry-After header

– X-RateLimit-Limit: ###

– X-RateLimit-Remaining: ###

– X-RateLimit-Reset: EPOCH_SECONDS

• Descriptive response in the requested media type

Headers and response

Throttling Patterns

• Client side

– Use caching

– keep tap on number of requests

– Pay attention to headers

– No brainless loops (polling)

• Server side

– Support caching (etags and max-age)

– provide streaming endpoints when possible (feeds, news, public data)

Best practices

Few more topicsREST and plug-ability and extensibility

MicroServices…

• Same good old modularity with with distributed interfaces

• Plus with communication over HTTP, mostly!

• Plus communication using message passing (services, events, event bus)

• Plus with independent SLDC

• Plus being on different platforms/use different data-store/etc.

–Whatever tools, language, framework that fits better!

How far?

MicroServices

• Advantages– Simplicity of development and maintenance– Isolation of requirements and fulfillments– Scale-up and Scale-down, scale-out and scale-in– More focused HR managements– Etc.

• Disadvantages– Operations Overhead– High level of DevOps, RE Skills Required– Complexities of a Distributed System– Etc.

Pros and Cons

Comments, Questions?

• http://tools.ietf.org/html/rfc6902• http://tools.ietf.org/html/rfc6901• http://resteasy.jboss.org/• https://jersey.java.net/documentation/latest/• http://tools.ietf.org/html/rfc6585• http://tools.ietf.org/html/rfc5789• http://martinfowler.com/articles/richardsonMaturityModel.html• http://www.slideshare.net/bhaktiks/real-world-restful-service-development-problems-and-solutions

• CCL photos used in slides:• https://www.flickr.com/photos/treehouse1977/2892417805/• https://www.flickr.com/photos/treehouse1977/2892417805/• https://www.flickr.com/photos/essjay/165928100/• https://www.flickr.com/photos/jforth/4413370462/• https://www.flickr.com/photos/sakalak/8737872379/• https://www.flickr.com/photos/jbparrott/8980026600• https://www.flickr.com/photos/pentadact/36593493/• https://www.flickr.com/photos/jasohill/4442279347/• https://www.flickr.com/photos/mdsharpe/5075953655• https://www.flickr.com/photos/chuqvr/8329512894/• https://www.flickr.com/photos/longo/2684733921• https://www.flickr.com/photos/_davor/14757399908

Resources