Embed Size (px)
Citation preview
May 28, 2015
South Florida Agile Association – Fort Lauderdale, FL
“It’s not called Continuous Integration for Nothing!”Dan Boutin – Senior Product Evangelist
[email protected] (404) 304-9529@DanBoutinSOASTA
© 2014 SOASTA. All rights reserved. 2
In This Discussion Today
Agenda: • SOASTA Introduction• The need for “Continuous”• The “Big 3”• CI with mobile and performance testing
© 2014 SOASTA. All rights reserved. 3
SOASTA – Performance Analytics
100 BillionUser Experiences Tested
10 MillionTests Performed
5 MillionFailures Avoided
Actual CloudTest view
© 2014 SOASTA. All rights reserved. 4
In This Discussion Today
Agenda: • SOASTA Introduction• The need for “Continuous”• The “Big 3”• CI with mobile and performance testing
© 2014 SOASTA. All rights reserved. 5
CI Offers Speed of DeliveryWhat’s not to like?
© 2014 SOASTA. All rights reserved. 6
CI Offers Speed of Delivery
o Small Batch Sizes
What’s not to like?
© 2014 SOASTA. All rights reserved. 7
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
What’s not to like?
© 2014 SOASTA. All rights reserved. 8
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
What’s not to like?
© 2014 SOASTA. All rights reserved. 9
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
What’s not to like?
© 2014 SOASTA. All rights reserved. 10
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
What’s not to like?
© 2014 SOASTA. All rights reserved. 11
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
What’s not to like?
© 2014 SOASTA. All rights reserved. 12
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
What’s not to like?
© 2014 SOASTA. All rights reserved. 13
CI Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration
What’s not to like?
© 2014 SOASTA. All rights reserved. 14
In This Discussion Today
Agenda: • SOASTA Introduction• The need for “Continuous”• The “Big 3”• CI with mobile and performance testing
© 2014 SOASTA. All rights reserved. 15
What’s Been Missing?
© 2014 SOASTA. All rights reserved. 16
So, let’s break down DevOps – What is it, really?
o It’s not what’s within the silos
o It’s about the relationships
A typical silo’d organization lacks end-to-end integration
Performance Engineering & Continuous Integration-Full Lifecycle
Operational Configuration
Continuous Integration touchpoints
ADLCQA/Test Pre-Production
Production
QA/Test Environment
Operational Production
Environment
Application Development Environment
Development Test
Configuration management
Release management
Change management
Performance management
CI Focus Areas – The assembly line Hand-offs
Key Processes
© 2014 SOASTA. All rights reserved. 19
Software is simple: It’s Manufacturing & BOM
o Bill of Materials (BOM!)
o The “Big 3” => Change, Configuration & Release
• That is your software assembly line conveyor belt
© 2014 SOASTA. All rights reserved. 20
So, what?
• We can only test as fast as the conveyor belt allows.
• If the conveyor belt is moving slow, testing is delayed.
• The Big 3 drives testing cycle.
o Why is CI critical?
• Speed
• New features, functions, etc. need to get to the customer, ASAP!
© 2014 SOASTA. All rights reserved. 21
So, let’s break that down…
o CI is a key component of Performance Engineering
o Performance Engineering spans the silos
BIG 3
Testing
ApplicationDevelopment Life Cycle
ITOperations
CapacityManagement
Performance
Management
PerformanceEngineering
© 2014 SOASTA. All rights reserved. 22
Laying the FoundationA factory streamlines for quality and speed
Does it work as designed?
Does it perform under
extreme circumstance
s?
Approve materials
Real Drivers
What happens in actual conditions?
Assembly Line
© 2014 SOASTA. All rights reserved. 23
Laying the FoundationCI streamlines for continuous quality
Functional validation
Performance,reliability,scalability tests
Real Users
Build
What happens in actual conditions?
Unit-tested code
© 2014 SOASTA. All rights reserved. 24
In This Discussion Today
Agenda: • SOASTA Introduction• The need for “Continuous”• The “Big 3”• CI with mobile and performance testing
© 2014 SOASTA. All rights reserved. 25
Continuous QualitySOASTA’s Lifecycle
© 2014 SOASTA. All rights reserved. 26
Continuous QualitySOASTA’s LifeCycle
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
© 2014 SOASTA. All rights reserved. 27
Continuous QualitySOASTA’s Lifecycle
Run iterative load tests via Jenkins
Spin up suitable, consistent test environments in the
cloud or the lab
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
© 2014 SOASTA. All rights reserved. 28
Continuous QualitySOASTA’s Lifecycle
Run iterative load tests via Jenkins
Mobile? Run functional automation on real devices for functional & performance validation
Spin up suitable, consistent test environments in the
cloud or the lab
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
© 2014 SOASTA. All rights reserved. 29
Continuous QualitySOASTA’s Lifecycle
Run iterative load tests via Jenkins
Daily (or more frequent) updates keep teams quality focused
Embedded failure details & bottleneck data isolate issues
Mobile? Run functional automation on real devices for functional & performance validation
Spin up suitable, consistent test environments in the
cloud or the lab
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
© 2014 SOASTA. All rights reserved. 30
Continuous QualitySOASTA’s Lifecycle
Run iterative load tests via Jenkins
Daily (or more frequent) updates keep teams quality focused
Embedded failure details & bottleneck data isolate issues
Manage to a performance
baseline
Mobile? Run functional automation on real devices for functional & performance validation
Spin up suitable, consistent test environments in the
cloud or the lab
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
© 2014 SOASTA. All rights reserved. 31
Continuous QualitySOASTA’s Lifecycle
Use real user data toiterate dev & test plans
Run iterative load tests via Jenkins
Daily (or more frequent) updates keep teams quality focused
Embedded failure details & bottleneck data isolate issues
Manage to a performance
baseline
Mobile? Run functional automation on real devices for functional & performance validation
Spin up suitable, consistent test environments in the
cloud or the lab
I JUST MADE A CODE CHANGE
Jenkins runs any unit tests as usual
SOASTA – Branch Build from CloudTest
SOASTA – Test Results
SOASTA – Drill Down (Test Failures)
SOASTA – UI Testing Bundle Results
SOASTA – UI Testing Failure Drill down into SOASTA CloudTest from Jenkins
SOASTA – Successful regression test results
SOASTA – Release Promotion
© 2014 SOASTA. All rights reserved. 40
Continuous Quality for Continuous Delivery
o SOASTA’s Process Metrics• 3 Main Products
• (2) Builds per day of each product (Dev branch + Customer branch)
• 8,000 tests executed per build
• For Mobile (TouchTest), Over 300 fully-automated tests each for iOS & Android
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 41
The Path to Continuous Testing
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 42
The Path to Continuous Testing
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 43
The Path to Continuous Testing
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 44
The Path to Continuous Testing
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 45
The Path to Continuous Testing
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
SOASTA - Takeaways
© 2014 SOASTA. All rights reserved. 46
CONFIDENTIAL – Not for Distribution
Final TakeawayWhy is CI important?
Revenue
Brand
Competitive advantage
© 2014 SOASTA. All rights reserved. 47
We will close with some Mobile Trivia
@DanBoutinSOASTA
© 2014 SOASTA. All rights reserved. 48
Questions?
Dan Boutin – Senior Product [email protected]
Mobile (404) 304-9529@DanBoutinSOASTA
SOASTA TouchTest
• FREE DownLoad: http://goo.gl/6CErMN
Visit our Blog: http://goo.gl/TA2HKP
Questions? @DanBoutinSOASTA
May 28, 2015
South Florida Agile Association – Fort Lauderdale, FL
“It’s not called Continuous Integration for Nothing!”Dan Boutin – Senior Product Evangelist
[email protected] (404) 304-9529@DanBoutinSOASTA
