Integrating Faban with Dockerfor Performance Benchmarking

[Demonstration Paper]

Vincenzo FermeFaculty of Informatics

University of Lugano (USI)vincenzo.ferme@usi.ch

Cesare PautassoFaculty of Informatics

University of Lugano (USI)cesare.pautasso@usi.ch

ABSTRACTReliability and repeatability are key requirements in per-formance benchmarking ensuring the trustworthiness of theobtained performance results. To apply a benchmark tomultiple systems, the reusability of the load driver is essen-tial. While Faban has been designed to ensure the relia-bility of the performance data obtained from a benchmarkexperiment, it lacks support for ensuring that the system un-der test is deployed in a known configuration. This is whatDocker, a recently emerging containerization technology, ex-cels at. In this demo paper we present how we integratedFaban with Docker as part of the BenchFlow framework tooffer a complete and automated performance benchmark-ing framework that provides a reliable and reusable envi-ronment, ensuring the repeatability of the experiments.

KeywordsFaban, Docker, Performance Benchmarking, Repeatability

1. INTRODUCTIONBenchmarking is a well established practice for discovering

application’s performance pitfalls and bottlenecks [4]. A keyrequirement in performance testing is ensuring the reliabil-ity of the experiments [3], since important decisions will betaken based on the obtained results. Many tools for reliableperformance testing exist. One of them is Faban1, a free andopen source performance workload creation and executionframework, largely used in industry standard benchmarks,such as the ones released by the Standard Performance Eval-uation Corporation (SPEC)2. The growing complexity ofmodern distributed applications requires to automate theirdeployment with tools, such as the ones provided by theDocker ecosystem3. Most of the available performance test-

1http://www.faban.org2https://www.spec.org3https://www.docker.com

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

ICPE’16 March 12-18, 2016, Delft, Netherlandsc© 2016 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-4080-9/16/03.

DOI: http://dx.doi.org/10.1145/2851553.2858676

InstanceDatabase

DBMSFaban Drivers

ContainersServers

DATA TRANSFORMERS

ANALYSERS

Performance Metrics

Performance KPIs

Web Service Te

st E

xecu

tion

Ana

lyse

s

Faban+

Microservice

Minio

harnessAdapters

Data Mappers

CO

LLECT

OR

S

MONITORS

Figure 1: BenchFlow Framework

ing tools, do not provide a well defined and standard mecha-nism to integrate the deployment of the System Under Test(SUT) into the performance testing and benchmarking pro-cess. This is key to reduce the cost of automating the bench-marking. Additionally, this integration is necessary to en-hance the repeatability [3] of the performed experiments.Additionally, given the growing complexity of the targeteddeployment environment, it has become important to pro-vide lightweight and non-invasive means to define customperformance data to be collected on the SUT side.

With BenchFlow4 [2] we aim at solving the aforemen-tioned limitations, as well as enhancing the reusability ofthe load drivers. In this demo paper we discuss the func-tionality and the architecture of the framework, and presenta walk-through use case to demonstrate its capabilities.

2. THE BENCHFLOW FRAMEWORKThe BenchFlow framework, presented in Fig.1, is deployed

in Docker containers and builds on top of Faban, to providereliable performance benchmarking, and Docker, to ensurerepeatability and reusability. The load drivers, defined byexploiting the Faban framework, are executed by the Har-ness, and provide the infrastructure needed to define thesimulated users and their interaction with the SUT. TheBenchFlow framework defines adapters between the loaddrivers and the SUTs. That way the same load drivers canbe reused to load different systems by defining the mappingbetween the abstract interaction defined in the generic loaddrivers, and the actual implementation for each SUT. The

4http://www.benchflow.inf.usi.ch

129

adapters guarantee the reusability of the same driver for dif-ferent types of performance test [4].

The benchmark life cycle starts with SUT’s deploymentby exploiting the Docker containerization technology. Thusit provides for the replicability of the experiments by en-suring the SUT is always deployed in the exact same ini-tial state. While containerization technologies introducesome overhead on system’s performance, a recent reliableperformance analysis of Docker [1] indicated that, if care-fully configured, Docker reaches near-zero overhead. Dur-ing the performance test execution, the BenchFlow frame-work monitors the experiment’s execution state to gatherresource utilization (e.g., CPU, RAM, Network) data, us-ing the lightweight monitors of the Docker stats API. Whenthe performance test execution is complete, a set of collec-tors gather the raw performance data from the distributedinfrastructure on which the performance test has been per-formed, and send them to a central storage service, e.g.,Amazon S3, or Minio5. We rely on a storage service, sincethe data have to be efficiently accessed from the componentscomputing the metrics. To abstract from the different dataformats that different SUTs might have, the performancedata are then transformed to a canonical meta-model. Af-ter the transformation, the performance data are stored ina Cassandra6 database (DB), and the performance metricsand KPIs are computed. Cassandra is a powerful DB forstoring and accessing performance data from the service thatcomputes metrics on top of them. The computation is per-formed by relying on Apache Spark7, a fast, general-purposeengine for large-scale data processing. Before the computa-tion of the metrics is triggered, the BenchFlow frameworkchecks the logs collected from the SUT to identify executionerrors and validate the experiment.

The orchestration of the performance test execution, thedata collection, and the performance data analysis, is del-egated to Apache Kafka8, a publish-subscribe messagingframework. We have introduced this state-of-the-art frame-work to decouple the benchmark execution managed by theFaban Harness, from the performance metrics computation,and thus pipeline the gathering of performance data withthe corresponding analytics, which can be performed offline.

3. GOALS OF THE DEMONSTRATIONThe BenchFlow framework is currently used for bench-

marking Workflow Management Systems, and has been suc-cessfully applied in different experiments [2]. In the demowe will present a walk-through use case that shows Bench-Flow framework’s capabilities, both from the perspective ofperformance researchers and performance testers. Duringthe demo, the framework will be pre-installed on multipleservers, in a dedicated, reliable and controlled environment,which should be accessible over VPN from the conferencevenue. We will go through the end-to-end performance testprocess, by defining, submitting and monitoring the perfor-mance test, describing the SUT deployment definition, andaccessing the automatically calculated performance metrics.

Defining the performance test: the framework sim-plifies the definition of load drivers’ behavior with Faban,

5https://www.minio.io6http://cassandra.apache.org7http://spark.apache.org8http://kafka.apache.org

through load driver definition descriptors. We will define aperformance test, involving a distributed system of multi-ple microservices and their DBs, by means of a benchmarkdefinition file, where we can define all the performance testparameters (e.g., a load test). The framework will take careof translating the performance test definition to the actualformat used internally by Faban. Describing the SUTdeployment definition: the deployment is performed byrelying on the Docker Compose tool. This ensures that eachSUT deployment configuration (e.g., different amounts ofRAM) and its initial state can be precisely described andexecuted obtaining the exact same initial conditions for theexperiment. Paired with Docker Swarm9 it is possible to au-tomate SUT’s deployment on a distributed infrastructure.Submitting and monitoring the performance test:we will submit the performance test we have previously de-fined, and monitor its execution status. The test will last 1minute and will be repeated 3 times. Accessing the auto-matically calculated performance metrics: when thebenchmark execution is complete and the performance dataare ready to be explored, we will visualize them at the endof the demo.

During the demo, we aim to demonstrate the flexibilityand the simplicity of using the BenchFlow framework. Oncedefined by means of the BenchFlow benchmark definitionfile and the Docker Compose deployment definition file, theexperiments can be replicated multiple times in a fully au-tomated way to obtain reliable and verified results.

4. CONCLUSIONS AND FUTURE WORKThe BenchFlow framework greatly simplifies and acceler-

ates the definition and execution of reliable, repeatable andreusable performance tests. It does so by integrating twopowerful technologies: Faban and Docker, and building ontop of their functionality to provide a complete frameworkfor automated performance test execution over a distributedenvironment. The next planned steps concern enabling thesimplified definition of additional performance test types(e.g., spike testing), as well as providing stronger perfor-mance test validation and means for performance test resultanalysis and exploration. Moreover we also plan to collabo-rate with the ICPE and the SPEC community, to drive theframework’s development and strengthen its functionality.

5. ACKNOWLEDGMENTSThis work is partially funded by the Swiss National Sci-

ence Foundation with the BenchFlow - A Benchmark forWorkflow Management Systems (Grant Nr. 145062) project.

6. REFERENCES[1] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An

updated performance comparison of virtual machinesand linux containers. Technical report, IBM, July 2014.

[2] V. Ferme, A. Ivanchikj, and C. Pautasso. A frameworkfor benchmarking BPMN 2.0 workflow managementsystems. Proc. of BPM ’15, pages 251–259, 2015.

[3] K. Huppler. The art of building a good benchmark.TPCTC 2009, pages 18–30. Springer, 2009.

[4] I. Molyneaux. The Art of Application PerformanceTesting: From Strategy to Tools. O’Reilly, 2nd edition,2014.

9https://www.docker.com/docker-swarm

130