Performance Testing REST APIs

Copyright © 2012 Constant Contact Inc. 1

Performance Testing REST APIsSelenium Conference 2013Jason Weden @jweden

Principal Quality Engineer

Constant Contact, WebServices


Who We Are

What do we do? Expose our internal web services externally to developers and

partners via an API for their own applications.

http://developer.constantcontact.com Fun Digression: Very easy to get up and running with signing into API using

Mashery and using our console to hit the REST API

http://developer.constantcontact.com/


Problem Domain

System Under Test: HTTP REST APIs Hitting the REST API with concurrent connections Defining your Performance Test approach Getting & Sharing results – performance metrics


Lesson 1: Be DRY

Why in the heck repeat yourself? You already have the logic in your functional tests for REST calls so reuse

that logic for your performance tests http://techblog.constantcontact.com/api/what-lego-blocks-teach-us-ab

out-rest-api-testing/ Reuse your data-seeding (e.g. for tests that have GET calls) Tools like LoadRunner and Jmeter have their place but

They’re creating blocking connections that reduce scalability You are wet – you’re rewriting logic you already wrote in your functional

tests You confine the creation, execution, and general knowledge of the

performance testing arena to one or more toolsmiths

http://techblog.constantcontact.com/api/what-lego-blocks-teach-us-about-rest-api-testing/

http://techblog.constantcontact.com/api/what-lego-blocks-teach-us-about-rest-api-testing/


Lesson 2: Choose Carefully

Choose your http client library with care Non-blocking Scale-up Seriously consider Akka (if on the JVM)

Non-blocking but with high throughput Scales to multiple processors and prevents need to deal with shared state More on this later

Other Examples on the JVM of non-blocking http clients http://spray.io (Scala) – Spray – Akka-based http://www.eclipse.org/jetty/ (Java) -- Jetty 9

https://github.com/timboudreau/netty-http-client (Java) -- Netty

http://spray.io/

http://www.eclipse.org/jetty/documentation/current/http-client.html

http://www.eclipse.org/jetty/

https://github.com/timboudreau/netty-http-client


Lesson 2: Choose Carefully (cont.) – Our Choice…

Written in Java but takes advantage of Akka for concurrency Integrated into test framework for everyone to use 3rd Party libraries

Apache HttpClient library Apache HttpCore library Akka concurrency framework Apache Commons Math3 library

High performance Easy to use programmatic API – small learning curve

Developers can go in and create performance tests Useful for any http functional OR performance testing

Even functional tests have performance metrics logged


Lesson 2: Choose Carefully (cont.) – Our Choice…

Found errors & significant bottlenecks Comparison with JMeter and LoadRunner

Preliminary tests show faster than JMeter (even with our 8 threads vs JMeter's 1500)

All performance tests can be checked into source control No dependency on toolsmiths for HTTP performance testing Performance testing experience shared across the team Likely faster than JMeter and LoadRunner -- still to be

scientifically determined Ability to tweak our home-grown solution vs. closed source

purchased solutions. (e.g. lots of concurrency properties to tweak)


Lesson 3: Devise a Performance Test Plan

What is your SLA? (e.g. avg. response time, avg. requests/sec) What kind of performance test? Options:

Profile of production performance. Do the following concurrently: X amount of POSTs to endpoint 1 Y amount of GETs to endpoint 2 (data seeding needed) Z amount of DELETEs to endpoint 3 (data seeding needed)

One type of Request: x amount of GETs Longevity – run test above for x amount of time

What metrics will be recorded (client vs server)? How will results be captured and reported? Have sign-offs on your test plan (e.g. in a wiki section)


Lesson 4: Capture Perf. Test Results in a Database

Capture: Performance Metrics

Client: min/max/avg response time, standard deviation, avg requests/second, percentiles, request failures

Server: cpu, memory, open db connections, thread count, consider app. profiling

Test tool used and version Test name and description Freeform comments Concurrency level In what environment was test performed (server hostnames) When it was performed Version of Test tool and application under test

Allows for posterity Allows for trending


Lesson 5: Run Performance Tests Continuously

Lobby for a dedicated performance environment

Run tests in non-production CI environments if only for trending relative measurements

Run daily


Lesson 6: Create Visualizations for Perf Test Results

Charts, Charts, Charts: Have a webpage dashboard Preferably having ability to query or filter to obtain customized results

Charts, Charts, Charts: Email results of each performance test run

Visualization Research: Splunk w/ DB Connect D3: http://shop.oreilly.com/product/0636920026938.do

Fun Digression: playing with this at: http://codepen.io

http://shop.oreilly.com/product/0636920026938.do

http://codepen.io/


References – The end…

http://akka.io/

http://commons.apache.org/math/

http://hc.apache.org/httpcomponents-client-ga/

Slides available at: https://www.slideshare.net/jweden/rest-performance-testing

http://akka.io/

http://commons.apache.org/math/



https://www.slideshare.net/jweden/rest-performance-testing

https://www.slideshare.net/jweden/rest-performance-testing

Technology

Performance Testing REST APIs