Soasta | CloudBees webinar 11/12/2015

Continuous Performance Testing and Continuous Delivery with JenkinsNovember 12, 2015Tom Chavez, SOASTA @TomChavezBrian Dawson, CloudBees @brianvdawson

2© 2015 SOASTA Inc.- All rights reserved.

In This Webinar

TODAY’S PRESENTERS

Tom Chavez: SOASTA, Sr. Product Marketing ManagerBrian Dawson: CloudBees, DevOps Practitioner

TWO COMPANIES, A SINGLE SOLUTION

SOASTA: The leader in Mobile and Cloud Testing

CloudBees: The Enterprise Jenkins Company

Together, we offer a complete cloud-based service:

Continuous Performance Testing & Continuous Delivery with Jenkins


Poll Question

What best describes your status with Continuous Integration / Continuous Delivery (CI/CD)? (single answer)

a)Practicing CI

b)Practicing CD

c)Implementing CI/CD

d)Determining if CI/CD is a good option

e)What is CI/CD?


Agenda

• Why should we performance test earlier? • Building tests and preparing the test environment• Connecting your performance tests to your CI (Jenkins)• Establishing and managing to a performance baseline• Execute tests in parallel with CD pipelines built and executed

with Jenkins Workflow• Q&A

Questions: Submit in the question box during event


Why Should We Load Test Earlier?

6© 2015 SOASTA, Inc.-All rights reserved.


7© 2015 SOASTA, Inc.-All rights reserved.


Okay. Why DON’T we load test earlier?

Copyright © 2014 SOASTA, Inc. - All rights reserved.


Why DON’T We Load Test Earlier?

“Because it’s hard.”o Tools require coding skills we don’t have (C, VB?)o We don’t have a scalable test environmento We don’t have hardware for load generationo It’s a different toolo There are too many other tests to do




“Because it takes too long.”o Scripting takes forevero Results are slowo Fast setup of environments is impossibleo We’re dependent on IT or other teamso We’re too agile





“Because no one ever told us to.”o There are no “performance coverage” requirementso There isn’t a performance baseline for referenceo There isn’t a defined performance targeto No one is managing to a performance metric





“Because no one ever told us to.”o There are no “performance coverage” requirementso There isn’t a performance baseline for referenceo There isn’t a defined performance targeto No one is managing to a performance metric

Until Now.


Poll Question

What do you see holding back a more iterative load testing approach? (check all that apply)

a) Skills gaps

b) Lack of test environment(s)

c) Slow test development

d) Test tools limitations

e) 3rd Party Dependencies

f) Undefined Performance Requirements

g) Lack of Management Push

h) Other


Load Testing Has Been an Elite Event

SThe Issue:

• High expectations from tests run with diminishing time windows

• Code-level performance bugs found late set entire project back

• Separate team and schedule conflict with delivery goals

• Major issues escape to production

o pecialty skills and tasks are a legacy barrier

Iteration 1

Automated unit tests

Iteration 2

Automated unit & functional tests

Iteration 3

Regression test suite

Iteration n

Regression test suite

Load & Performance Testing

Deploy

Issues?


Continuous Load TestingValidate performance piece by piece

Iteration 1Automated unit

tests

Deploy

Production perf tests

Iteration 2Automated unit & functional tests

Iteration 3Regression test

suite Incl:

Iteration nRegression test

suite Incl:

End to End Production Scale Test

Small perf test Small perf testsPerf smoke tests Perf smoke tests

The Solution:• Run incremental performance tests automatically with each iteration• Establish a trend line for performance early• Developers find code and architecture issues early• Performance teams focus on end-to-end infrastructure and user related testing• Minimize production escapes and implement production testing

Development/Test team Performance/Ops teamDevOps


Continuous Performance (for Web & Mobile)The Path to a Performance Culture

I JUST MADE A CODE CHANGE

CloudTest runs load tests via JenkinsDaily (or more frequent) updates keep

teams “performance focused”

Identify bottlenecks & inform teams,

just like unit and functional

tests in Jenkins

Jenkins launches suitable cloud-based test environment

Manage with a new metric!


Poll Question

What types of Load Testing are you doing today? (Single Answer)

a) Non-Web Protocols (not mobile)

b) Primarily web protocols

c) Primarily mobile web traffic

d) Both web and mobile

e) All web, mobile & non-web protocols

17© 2015 SOASTA, Inc. All rights reserved.

Enabling Continuous Performance Testing with a Jenkins CI/CD Pipeline

Brian DawsonTwitter: @brianvdawson

Email: [email protected]


Why are we here today?

Fundamentally we are talking about......

18

DeliveringBetter

SoftwareFaster!

Concept-to-Customer

High Quality• Performance• Feature• Function

RapidlyEliminating WasteMarket Innovation

High Value


How Do You Deliver Better Software Faster?

19

Prod

RapidlyRepeatedly

Reliably

ProdDev



20

RapidlyRepeatedly

Reliably

ProdDev



21

“DevOps” Has Great Promise

“DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.”[1]

[1] http://en.wikipedia.org/wiki/DevOps

DevOps



22

“DevOps” Has Great Promise

“DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.”[1]

[1] http://en.wikipedia.org/wiki/DevOps

DevOps

But You Can’t “Do” DevOps. DevOps an outcome.


You Can “Do” Continuous Delivery

The application of automation and orchestration technologies and principles of Lean Manufacturing to the software delivery lifecycle.

23



24

DevOpsThe “What”

27© 2015 SOASTA Inc.- All rights reserved.27

DevOps

The “How”

The “What”

Continuous Integration

AgileIteration

Continuous Delivery


SDLC CoverageFeature, Non-Functional, Function, Operational

Integration and A

utomation


DevOps

The “How”

The “What”


AgileIteration

Process&

Practices

Continuous Delivery

CI and Continuous Delivery (CD) become the foundation for a DevOps transformation.



!Robin Thicke “Blurred Lines”

“Today, meaningful integration happens less in source control and more frequently in a runtime environment… if you want to quickly verify that the work of two developers is not

conflicting, you need to run API tests, as well as some functional tests scenarios. That means

bringing the various services into an integration test environment, and validating runtime functionality at a pace similar to the classic CI build loop ”

— Steve Smith 2014

Continuous Integration and todays technologies...

The line between Continuous Integration and Continuous Delivery has blurred


CI, CD and Feedback

Did the

Build

break?

Is it fast?

Does it

deploy?Is the “need” met?


So, How do you “Do” Continuous Delivery?


Automation is the Key

32Photo courtesy of Steve Jurvetson via Flickr


Continuous Delivery (CD) Applies Automation to the Application Lifecycle

ProdDev BuildCommit Test Stage Deploy

Feedback Loop


CD Starts with Continuous Integration (CI): Automating Application Build and Test


ProdDev

Automation Engine

Developer

Source Code

Control System

Compile/BuildCode

Scan TestsCodeCommit

Results Report

BuildCommit Test


Jenkins is the #1 Automation Platform Used for CI


Developer

Source Code

Control System

Compile/BuildCode


Results Report

BuildCommit Test


Jenkins is Also Your Key to Continuous Delivery

Developer

Source Code

Control System

Compile/BuildCode


Results Report

BuildCommit Test Stage Deploy

Deploy Prod

Deploy Stage

Complex Delivery Pipelines

Delivery of App and Config

Continuous Delivery

ProdDev


Have you met Jenkins?

#1 CI/CD Automation Engine

• Open Source

• Highly Extensible

About 8 years old

Easy to install/use

1000+ plugins

Widely adopted

• 100K+ installations


Over 1000 Jenkins Plugins for Tool Integration

Integrated Development Environments (IDEs), editors/compilers/debuggers, code review and collaboration tools.

Source code control systems and related asset management tools.

Automated API-based testing frameworks.

Release management and release automation software

Automated API-based testing frameworks, static code analysis tools.

Load Testing Tools and Security Scanning technology

Automated and manual UAT tools

Release management, release automation, and change control software.

Environment and Configuration Management SoftwareDevelop

Commit

Build & Integrate

Unit Tests

Release(to Test)

AutomatedRegression Tests

Release(to Prod)

User Acceptance Tests

Load & Security Tests


Jenkins is the Hub of the CD Ecosystem

On-premise or in the cloud

Plug-ins for all your tools

RunDeployStageTestBuildCommit


Are You Using Today?

42

Chances are someone in your organization is.



How Do You Manage Jenkins and Testing at Enterprise Scale?


About CloudBeesJenkins founder, Kohsuke Kawaguchi, CloudBees CTOWe optimize Jenkins for the enterprise

• Jenkins Operations Center

• Jenkins OSS Support

• Enterprise-class plugins

• Commit code and support the Jenkins Community

• Jenkins training and consulting

• Jenkins User Conferences (JUC) and newsletters

…we are the Jenkins Enterprise experts!


CloudBees Offerings


CloudBees Jenkins Enterprise

47

Professional support

High availability

Large installations

Security

Optimized utilization

Continuous Delivery


How Can CloudBees Jenkins Enterprise Help with Your CD Strategy?

48

Workflow Stage ViewProvides deep insight into the execution of your workflow stages so you can detect problems and performance issues

Manage complex “level-of-detail” testing pipelines

Jenkins WorkflowCreated by CloudBees to provide Jenkins with a way to define flexible delivery pipelines that allow for a rich set of new delivery scenarios

Manage and share “testing-as-code”

Dynamic Environment CreationSpin up new test/deployment environments automatically as part of your delivery pipeline

Deployment Tracking and VerfiicationTrack the history of your deployments – including who kicked off a deployment and what code changes were included in it

Create pipelines that leverage verification tests to determine the quality of a new deployment

Jenkins AnalyticsMonitor and analyze performance across all your Jenkins instances

Monitor, measure and manage adoption of testing, CI and CD best practicesRestart builds, tests, and deployments where they

breakLeverage workflow checkpoints to restart delivery pipelines without a full (expensive) rebuild

...central source of orchestration to bridge dev, test, and ops alike

And more…


Jenkins Workflow and Workflow Stage View Enable Full Continuous Delivery Pipelines

Continuous Delivery

ProdDev

Developer

Source Code

Control System

CodeCommit


CloudBees Jenkins Workflow Stage View

Restartable Checkpoints

Artifact Downloads


Monitoring, Visualization and Analytics

Jenkins Workflow Stage View Visibility into Pipeline status

Jenkins AnalyticsCross-Team Build Analytics

… among many other features, tailored for enterprise organizations using Jenkins @ scale


Version control

systems

Build & CI Systems

SysAdminsDevOps

Test @ Scale with w/Docker Containers and Cloud Support

Docker Hub

Docker Engine

Developer

$ docker

Benefits• Frictionless movement

from dev > test• Dev + Ops

collaboration in Hub

Test, Stage or Production


Additional Jenkins Services from CloudBees

53

Jenkins Certified Training• Delivered via CloudBees partners

Jenkins Consulting• Delivered via CloudBees Professional Services Team

Jenkins Newsletter• Community news and expert tips

Jenkins User Conferences• Learn from other community members


Continuous Performance Testing

o Demo:

• JBoss Ticket Monster Overview

• SOASTA Cloud Test: Setting up a load test up for unattended runs

• SOASTA TouchTest: Setting up a mobile functional test for unattended runs

• Setting up Continuous Performance Testing

• Jenkins Test Data:

• Reviewing Performance Trending Data: Track performance of each build

• Reviewing Performance Pass/Fail Data: Failing

• Reviewing Functional Test Resultsat are we going to show

What are we going to show


The SOASTA Platform

Performance Analytics

A Continuous Approach: End-to-End Digital Performance Management


SOASTA Jenkins Plug-in

o Open Source: online at jenkins-ci.org

o Automatically available on CloudBees DEV@cloud

o Complete control over CloudTest/TouchTest Automation from within your Jenkins builds:

o Totally integrated into Jenkins environment

https://wiki.jenkins-ci.org/display/JENKINS/SOASTA+CloudTest+Plugin+for+Jenkins

Questions?Ask in Question window


Resources

Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:

twitter.com/SoastaIncfacebook.com/cloudtest

Knowledge Center• White Papers• Webinar Recordings• Case StudiesCloudLink Community• Support• Tutorials• Video

Contact [email protected] us:

twitter.com/cloudbeesfacebook.com/CloudBees

www.SOASTA.com www.CloudBees.comTry it @ developer.cloudbees.com

Documentation• CloudBees Jenkins Platform• Workflow in Jenkins

White Papers:• Making the Leap from Continuous Integration to Contin

uous Delivery• The Business Value of Continuous Delivery

http://www.soasta.com/cloudtest/

mailto:[email protected]

http://www.cloudbees.com/

mailto:[email protected]

http://www.SOASTA.com/

http://www.CloudBees.com/

http://developer.cloudbees.com/

http://documentation.cloudbees.com/docs/platform-intro/index.html

https://github.com/jenkinsci/workflow-plugin/blob/master/README.md

https://pages.cloudbees.com/Website_CI_To_CD_eBook.html?lsd=WebsiteForm_MakingTheShifteBook

https://pages.cloudbees.com/Website_CI_To_CD_eBook.html?lsd=WebsiteForm_MakingTheShifteBook

https://pages.cloudbees.com/Website_BizValue-Continuous-Delivery.html?lsd=BizValue-Continuous-Delivery

http://www.soasta.com/cloudtest-lite-download/

Continuous Performance Testing and Continuous Delivery with JenkinsNovember 12, 2015Tom Chavez, SOASTA @TomChavezBrian Dawson, CloudBees

Internet

Soasta | CloudBees webinar 11/12/2015