Web Clients for Ruby and

What they should be in the future

Toru Kawamura@tkawa

RubyKaigi 2016

@tkawaToru Kawamura

• RESTafarian inspired by Yohei Yamamoto (@yohei)

• Technology Assistance Programmer at SonicGarden Inc.Programmer at PlayLife Inc.(on the side)

• Co-organizer of Sendagaya.rb(Regional Rubyist Community & Every Monday Meetup) https://sendagayarb.doorkeeper.jp/

• Facilitator of RESTful-towa (“What is RESTful”) Workshop (Monthly, next 2016-09-13 in Omotesando) https://rubychildren.doorkeeper.jp/

I’m going to talk about

• Human-driven client written in Ruby that accesses a Web API

• A part in server-side app that accesses a Web API is also a client

• The idea of “Web Client” gem

• The thoughts and the findings from creating this gem

I don’t want a client that is…

• Rigid because of being tightly coupled

• Hard to reuse because of too much dedication

I want a client that is…

• Adaptable to change because of being decoupled

• Easy to reuse because of versatility

I want a client that is…

• Adaptable to change because of being decoupled

• Easy to reuse because of versatility

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT

to Building Adaptable Web APIs in RailsRubyKaigi 2014

–Johnny Appleseed

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT

Many clients are built from human-readable documentation

GET /v1/statuses?id=#{id} GET /v1/statuses?id=#{id}

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT

GET /v2/statuses/#{id} GET /v1/statuses?id=#{id}×Need to rewrite code

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT

{ uber: { version: "1.0", data: [{ url: "http://www.ishuran.dev/notes/1", name: "Article", data: [ { name: "articleBody", value: "First note's text" }, { name: "datePublished", value: null }, { name: "dateCreated", value: "2014-09-11T12:00:31+09:00" }, { name: "dateModified", value: "2014-09-11T12:00:31+09:00" }, { name: "isPartOf", rel: "collection", url: "/notes" }, {

• API changes should be reflected in clients

• It is good to split up explanations of the API and embed them into each API response

• A lot of assumptions about the API make a tight coupling

Because of Coupling

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT Decoupling in a example:

FizzBuzzaaS• by Stephen Mizell

http://fizzbuzzaas.herokuapp.com/http://smizell.com/weblog/2014/solving-fizzbuzz-with-hypermedia

• Server knows how to calculate FizzBuzz for given number (<= 100)

• Server knows what the next FizzBuzz will be

• Client wants all FizzBuzz from one to the last in orderhttp://sef.kloninger.com/posts/

201205fizzbuzz-for-managers.html

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT Coupled client

• Every URL and parameter is hardcoded

• Duplicates the server logic such as counting up

"/v2/fizzbuzz/#{i}"

(1..1000)

(1..100).each do |i| answer = HTTP.get("/v1/fizzbuzz?number=#{i}") puts answer end

HYP

ERM

EDIA

: TH

E M

ISSIN

G E

LEM

ENT Decoupled client

• No hardcoded URLs

• Client doesn’t break when changing URLs / the restriction

root = HTTP.get_root answer = root.link('first').follow puts answer while answer.link('next').present? answer = answer.link('next').follow puts answer end Link ‘next’ is the key

I want a client that is…

• Adaptable to change because of being decoupled

• Easy to reuse because of versatility

HTTP Clients for Ruby

• Standard equipment libraries

• net/http

• open-uri

http://bit.ly/RubyHTTPClients

HTTP Clients for Ruby

• Feature comparison by nahi

• 「大江戸HTTPクライアント絵巻」Oedo

RubyKaigi 01 (2011-04-10)

• http://regional.rubykaigi.org/oedo01/

• “net/http has various derivatives and alternatives because of old-style API and simple structure”

Web API is easy to use• We can use one right away with net/http or other HTTP client

• HTTP has the uniform interface

• We can also use it with web browser or curl

• That’s why Web API becomes popular

• → Do you really use net/http or other HTTP client in your app?

There are so many gems dedicated to each Web API

• google-api-client, aws-sdk, octokit, twitter, koala, … (looks similar inside)• Pros

• The gem provides classes corresponding to data types of Web API

• The gem can support detailed specs dedicated to the Web API

• Using classes and method calls, you can write a code with less thinking of Web API

• Cons• The way of use differs depending on the gem

• You have to read a gem’s documentation instead of API’s

There are so many gems dedicated to each Web API

• What if you are on the side of providing a gem?

• You have to re-design an interface different from the Web API

• You have a lot of trouble creating multi-language library if you need

• Some client library reproduce the same class/method structure as in server-side

• It has CRUD mappings in HTTP communication

• But I think it would be better for such a complex API to use RPC

• Web API should be easy for everyone to use!

What makes us produce so many dedicated gems?

• Difference between JSON structure of each Web API

• Handling dedicated error, more detailed than 4xx

• Gap between calling API once and performing a function

Gap between calling API once and performing a function

• We want to perform a function provided by Web API, rather than just call it

• Fetch current data, then update old one if it exists

• Fetch the past 1000 records using the API that returns 100 records limited at once

• In human-driven client, they rarely accomplish their goal in single API call

• A Client app is made up of many functions (or microservices)

Gap between calling API once and performing a function

• How does a client decide what API to call next?

• allow the user to choose or choose by itself from options

• The options are hardcoded in a gem

• The gem defines some classes and methods, which are statically mapped on APIs

• The options should depend on what “state” the client is in

State management• HTTP client doesn’t have a state

• App have a state

• What screen the app is in now

• What screen the app came from

• What does the app show/select now

• In a classic web app, an app state is represented by the current URL

State transition on Web API• App have a state for deciding what API

to call next

• It is better for HTTP client to have such a state

• and get close to web browser that makes state transition in a way to follow a link

• It depends on the app how faithful the screen reflects the transition ”RESTful Web APIs” p.11 Figure 1-7

I want a client that is…

Adaptable to change because of being decoupled

Easy to reuse because of versatility

Capable of state management

= Web Clients*

* definition in this talk

Consider in terms of implementation layers for client & server

Framework

App Server

HTTP Client

Client App

Web API App

Request Response

Rack

Web API AppFramework

App Server

HTTP Client

Client App

Request Response

• Rack provides an interface between web server and ruby app

• An object based on Rack interface is called “Rack App”

• Web app built on Sinatra/Rails is also a Rack App

rack_app = Proc.new do |env| [ '200', {'Content-Type' => 'text/html'}, ['A barebones rack app.'] ] end Rack::Handler::Puma.run rack_app

by Christian Neukirchen

Rack App requirements

• An object that responds to the call method,

• Taking the env hash as an argument,

• Returning an array with three elements:

• HTTP status code

• Hash of response headers

• Array filled with response body

rack_app = Proc.new do |env| [ '200', {'Content-Type' => 'text/html'}, ['A barebones rack app.'] ] end Rack::Handler::Puma.run rack_app

Rack Middleware• Between the server and the framework, Rack Middleware can customize the

request/response and process data to your applications needs

• Rack::URLMap, to route to multiple applications inside the same process

• Rack::CommonLogger, for creating Apache-style logfiles

• Rack::Static, for serving static files in specific directories

• Rack::Reloader, Rack::ContentLength, Rack::Auth::Basic, Rack::MethodOverride, …

Rack Middleware requirements• A class that takes the other Rack App,

then instantiates a wrapped Rack App class FooMiddleware def initialize(app) @app = app end

def call(env) # do something in request res = @app.call(env) # do something in response res end end

http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/concepts.html#wsgi-middleware

Rack Middleware structure

wrapped_app = Rack::Builder.new do use Rack::ContentLength use Rack::CommonLogger use FooMiddleware run rack_app end.to_app

Rack::Handler::Puma.run wrapped_app

Rack::ContentLengthRack::CommonLogger

FooMiddleware

rack_app

RackRack Middleware

FrameworkWeb API App

App Server

HTTP Client

Client App

Request Response

Faraday

• Faraday is an HTTP client library that provides a common interface over many adapters (such as net/http)

• and embraces the concept of Rack Middleware when processing the request/response cycle

by Rick Olson, Zack Hobson

Faraday Middleware• Mechanism for customizing a request/response like Rack Middleware

• url_encoded, to encode parameters into x-www-form-urlencoded in request

• authorization, to add an auth token to request header

• json(ParseJson), for converting JSON of response body into Hash

• follow_redirects

• http_cache

• rack-compatible, to use a rack middleware as a faraday middleware(experimental)

Faraday Middleware requirements

• Very similar to Rack Middleware

• processing response in on_complete block

class BarMiddleware def initialize(app) @app = app end

def call(env) # do something in request @app.call(env).on_complete do |res_env| # do something in response end end end

Faraday::Request::AuthorizationFaradayMiddleware::ParseJson

BarMiddleware

Faraday::Adapter ::NetHttp

Faraday Middleware structure

conn = Faraday.new('https://api.github.com') do |b| b.request :authorization b.response :json b.use BarMiddleware b.adapter Faraday.default_adapter end

res = conn.get('/')

Adapter corresponds to Rack App

RackRack Middleware

FrameworkWeb App / Web API

App Server

FaradayFaraday Middleware

Adapter

Client App

Request Response

RackRack Middleware

FrameworkWeb App / Web API

App Server

FaradayFaraday Middleware

Adapter

Client App

Request Response

Build a gem not as a whole but as a Faraday Middleware

• Reusable

• Respect a common interface

Implemented Middleware

• https://github.com/tkawa/faraday-hypermedia

• faraday-navigation

• faraday-link-extractor

faraday-navigation

• Allow us to go back/forward using a history like a common web browse

• Allow us to follow a link

• And fill in parameters of URL just like an HTML form field

• Link Header from RFC 5988 (Web Linking)

• Link-Template Header from Internet-Draft(draft-nottingham-link-template-01; expired)

• URI Template from RFC 6570

Link/Link-Template HeaderLink: <https://api.github.com/users/tkawa/repos?page=2>; rel="next" Link-Template: <https://api.github.com/search{?q}>; rel="search”

faraday-link-extractor

• Extract links in each kind of Web API and translate them into Link/Link-Template header

• LinkExtractorCJ (Collection+JSON)

• LinkExtractorGithub (GitHub)

Extract Links into Header(in the case of GitHub)

{ "login": "tkawa", "id": 562433, "url": "https://api.github.com/users/tkawa", "followers_url": "https://api.github.com/users/tkawa/followers", "following_url": "https://api.github.com/users/tkawa/following{/other_user}", ... }

Link: <https://api.github.com/users/tkawa>; rel="self", <https://api.github.com/users/tkawa/followers>; rel="followers" Link-Template: <https://api.github.com/users/tkawa/following{/other_user}>; rel="following”

url/*_url treated as a link

history = Faraday::Hypermedia::History.new conn = Faraday.new(url: 'https://api.github.com') do |b| b.use :navigation, history b.request :authorization, ‘bearer', token b.response :json b.response :link_github b.adapter Faraday.default_adapter end

res = conn.get('/'); history.pp_current_links res = conn.get('navigation:link?rel=current_user') res = conn.get('navigation:link?rel=repos') res = conn.get('navigation:link(2)?rel=item') res = conn.get('navigation:back') res = conn.get('navigation:link?title=hypermicrodata') history.fill_in_template_params(number: 1) res = conn.get('navigation:link?rel=pulls')

⭐

⭐

Demo/

current_user

/user

/users/tkawa/repos

/repos/tkawa/activerecord-endoscope

/repos/tkawa/hypermicrodata

/repos/tkawa/hypermicrodata/pulls/1

repos

item#2 back

title=hypermedia

pulls

https://asciinema.org/a/85363

• Make it decoupling

• Tight-coupling over a boundary between client and server makes it hard to change

• Taking advantage of Ruby, dynamic processing lead to decoupling

• Enable to Reuse

• Clip the app/domain-specific part

• Designing along with standards including RFC, we can use general-purpose library

• Build single-function component based on combinable interface such as Faraday Middleware

Conclusion