Upload
nowsecure
View
283
Download
2
Embed Size (px)
Citation preview
Shifting left:Continuous testing for better
app quality & security
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Connect
Twitter: @NowSecureMobile | @GuerrillaQA
—
Subscribe to #MobSec5, our weekly mobile security news digest
http://mobsec5.nowsecure.com/
—
Web: nowsecure.com | guerrillaqa.com
Steven WinterFounder & Chief Strategist, GuerrillaQA
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Andrew HoogCEO & Co-founder, NowSecure
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Contents
● Why deploy more quickly?
● Going fast, achieving quality, & saving money
● What now? Must do’s!
● Continuous testing in practice
● Q & A
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Why deploy mobileapps more quickly?
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
The business value of more frequent deployments
Happier customersNew features / improvements increase customer satisfaction & lead to faster
realization of revenue from new features.
Fix defects fasterIdentifying flaws earlier & shortening
the feedback loop leads to less expensive, faster fixes.
Reduce riskSmaller deployments include fewer
things that can go wrong, & those failures are easier to fix
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Pain experienced as a result of infrequent releases
Dissatisfied customersApp users churn due to their perception
that the developer is not responsive with improvements & new features.
Slower reaction timeImprovements & fixes take longer to be released, are more expensive, &
leave customers dissatisfied longer.
High-risk, complex deploysMonolithic releases include more
dependencies & potential failures resulting in more expensive & time-consuming fixes.
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
What does the ideal look like?
Company Deploy Frequency Deploy Lead Time
Amazon 23,000 / day minutes
Google 5,500 / day minutes
Netflix 500 / day minutes
Facebook 1 / day hours
Twitter 3 / week hours
Typical enterprise Once every 9 months Months or quarters
Kim, Gene. "Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win" 2014.
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
How frequently are others deploying?
https://blog.newrelic.com/2016/02/04/data-culture-survey-results-faster-deployment/
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Where are you in your journey? First steps
Automate what testing you can
Take advantage of Continuous Integration
Shift security & performance testing left
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
You can go fast, achieve quality, & save money
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Earlier testing & remediation prevents technical debt
Requirements / Architecture
Coding Integration /Component
Testing
System /Acceptance
Testing
Production / Post-Release
Source: National Institute of Standards & Technology
The cost for fixing vulnerabilities is
30x higher after an app has been deployed
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
The cost of fixing a P1 mobile bug in production Case Study
Team Hours
Detection & communication 20
Verification 16
Fix 40
Build, test, certify the fix 60
Customer acceptance 40
Post-publish verification 20
Total hours 196
As well as
● Loss of client & app user confidence
● Negative app ratings
● Derailment of feature development & release
$35KIn total costs
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Automation pays for itself with repeatability
Manual Testing
Automated Testing
Releases
Time / Effort
Time Savings
© Copyright 2015 NowSecure, Inc. All Rights Reserved. Proprietary information.
Development / Integration Staging Production
Dev TeamVersion Control
Build & Unit Tests
Automated Acceptance
TestsRelease
User Acceptance
Tests
Check-in
Check-in
Check-in
Trigger
Trigger
Trigger
Trigger
Trigger Approval
Approval
Feedback
Feedback
Feedback
Feedback
Feedback
Feedback
Engineer QA DevOps
© Copyright 2015 NowSecure, Inc. All Rights Reserved. Proprietary information.
Development / Integration Staging Production
Dev TeamVersion Control
Build & Unit Tests
Automated Acceptance
TestsRelease
User Acceptance
Tests
Check-in
Check-in
Check-in
Trigger
Trigger
Trigger
Trigger
Trigger Approval
Approval
Feedback
Feedback
Feedback
Feedback
Feedback
Feedback
Engineer QA DevOps
Shift security & performance Testing to the left
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
CI + CT = !!
Continuous Integration+
Continuous Testing=
Productivity multiplier
● Sets the stage for “set-it-and-forget-it” deployment
● Deliver higher quality code at lower risk in less time
● “Parallelizes” testing
○ Security, regression, performance, etc.
○ Simultaneously
● Repurpose test scripts
○ Write once
○ Use everywhere
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
What now? Must do’s!
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
1. Agree & commit to improving
Must do’s!
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Must do’s!
1. Agree & commit to improving2. Plan testing & automation scripting up front
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Must do’s!
1. Agree & commit to improving2. Plan testing & automation scripting up front
3. Agree on test coverage
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Must do’s!
1. Agree & commit to improving2. Plan testing & automation scripting up front
3. Agree on Test Coverage4. Measure, measure, measure
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Must do’s!
1. Agree & commit to improving2. Plan testing & automation scripting up front
3. Agree on Test Coverage4. Measure, measure, measure
5. Plan for test script maintenance
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Continuous testing in practice
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Case study: Value realized in just a few hours
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Steven’s experience at scaleFrom 4 months to Nightly
© Copyright 2016 NowSecure, Inc. All Rights Reserved. Proprietary information.
Scaling your automated testing based on maturity
Small Medium Enterprise
● Leverage open-source tools
● Build CI environment
● Create a basic smoke test
● Expand test coverage
● Leverage cloud platform services
● Plug security & performance
testing into CI
● Create smoke tests for each
feature (not the entire app)
● Prioritize by feature’s
success / risk
● Pick the top three & go!
Let’s talk
NowSecure+1 312.878.1100
@NowSecureMobilewww.nowsecure.com
GuerrillaQA+1 415.763.TEST
@GuerrillaQAwww.guerrillaqa.com
Subscribe to #MobSec5 - a collection of the week’s mobile news that matters - http://mobsec5.nowsecure.com/