EMBRACING DEVOPS TO IMPROVE VELOCITY AND SECURITY
February 22, 2018
1
ROB CLYDE, CHAIR ISACA BOARD OF DIRECTORSCISM, NACD BOARD LEADERSHIP FELLOWMANAGING DIRECTOR, CLYDE CONSULTING LLCEXECUTIVE CHAIR WHITE CLOUD SECURITYBOARD DIRECTOR, TITUSEXECUTIVE ADVISOR TO BULLGUARD AND HYTRUST
REMEMBRANCE…AND THANKS…
Robert StroudCGEIT, CRISC2014-2015 ISACA Board Chair2015-2018 ISACA Board Director
Industry leader…Trusted colleague…Mentor to many…
And most importantly…friend.
2
WHY DEVOPS
Source: Robert Stroud; Xebia Labs
A REAL LIFE EXAMPLE OF DELAYING VELOCITY
Security Compliance Release Management
Software Development Life CycleSource: Robert Stroud; Xebia Labs
“we have to implement DevOps as it’s the only way to deliver the speed, security,
velocity and quality our customers demand”Fortune 500 CEO
Source: Robert Stroud; Xebia Labs
Copyright © 2018 Information Systems Audit and Control Association, Inc. All rights reserved.
Placeholder slide for video
WHY DEVOPS
Efficiency - Faster time to market
Predictability - Lower failure rate of new releases
Reproducibility – Version everything
Maintainability - Faster time to recovery
Image from - dev2ops.orgSource: Robert Stroud; Xebia Labs
DEVOPS: A TIMELINE
8
Source: Robert Stroud; Xebia Labs
Source: Robert Stroud; Xebia Labs
DEVOPS FOR EVERYONE!
“Successful product delivery with DevOps has many different engaged stakeholders – from highly technical to business oriented“
DEV
ReleaseMgmt
QA
Business
OPSCompliance
Mgmt
Security
WHO DOES DEVOPS (BETTER SAID: WHO DOESN’T)
Source: Robert Stroud; Xebia Labs
DEVOPS ADOPTION WAVE
Hitting the Scalability Wall
Initial success with team of “rock
stars”
Attempts to go wide, run into
trouble
Data-Driven Continuous Improvement &
Involvement
DevOps at Enterprise ScaleA Leap of Faith
Skills
Software
Scaling
Security & Compliancy
CI/CD is our silver bullet and will solve all our problems!
Our team has an increase in productivity by 80%!
We’re shipping 3X more often, let’s roll this out more widely
Our IT heroes can do this
This is how we will become a modern IT enterprise
This is cool, no more manual steps. We can automate everything
Lets build compliancy into the pipeline
Let’s replace our big testing phases at the end with continuous testing
Let’s use data to drive our improvement cycle at scale
More teams and more roles are included
We also have to think about deployments further than Dev & Test
Let’s simplify our application architecture to speed up
All parties can be involved, even the auditing team
Can I redesign my security & compliancy process to speed up delivery?
Why don’t the new teams get it?
We need a plan to manage this transformation at scale
Why is our governance department so upset?
Only the real techies can do the magic
If we start scripting all our applications, it will become a nightmare
We need to make this work for our current business applications
Can we benefit from the cloud?
Let’s get more engineers to keep up
Source: Robert Stroud; Xebia Labs
CONTINUOUS INTEGRATION
Source: Robert Stroud; Xebia Labs
CONTINUOUS DELIVERY
Source: Robert Stroud; Xebia Labs
CONTINUOUS DEPLOYMENT
Source: Robert Stroud; Xebia Labs
DEVOPS, AGILE, ETC.
CODE BUILD INTEGRATE TEST DEPLOY OPERATERELEASE
AGILE DEVELOPMENT
CONTINUOUS INTEGRATION
CONTINUOUS DELIVERY
CONTINUOUS DEPLOYMENT
DEVOPS
Source: Robert Stroud; Xebia Labs
96xFaster mean time to recover from downtimeThat means high performers recover inless than an hour instead of several days.
5xas likely that changes will succeedThat means high performers’ changes fail 7.5% of the time instead of 38.5%.
Source: Robert Stroud; Xebia Labs
More frequentCode deployments
46xThat’s the difference between multiple times per day and once a week or less.
Faster lead time from commit to deploy
440xThat’s the difference between less than an hour and more than a week.
Source: Robert Stroud; Xebia Labs
MANULIFE/JOHN HANCOCK: BACKGROUND
19
Manulife/John Hancock offers a variety of financial services: Insurance, Mutual Funds, Asset and Wealth management, Private and Commercial Banking, Commercial Mortgages, Real Estate
Founded in 1862 as John Hancock Mutual Life Insurance company in Boston, Massachusetts, USA
Acquired by Manulife Financial (Toronto, Canada, founded 1887) in 2004
Named after a famous US Founding Father and signer of the Declaration of Independence
Acknowledged as a one of the best known American and Canadian brands
34,000 employees
20+M customers
Increasing & improving cadence of delivery and productivity across the various construction and hosting technologies in the portfolio
Enable true transformation to modern software development practices across a varied portfolio
Integrated security and code quality scanning for all technologies
Leveraging existing and new automation from build, test, deploy to more visible and accountable operations
Standardized management of regions & provisioning with test data & self-service infrastructure management
Efficiency with Scale - utilizing common pipeline tech stack solutions in partnership with other Manulife divisions
Insights, measurements & visibility on activities for continuous improvement
Establish an environment where building, testing and releasing software can be
done rapidly, frequently and reliably while maximizing predictability, efficiency,
security and maintainability of all of the applications in the
Enterprise Portfolio
DevOps PipelineOur Journey and Mission
Emerging Capabilities
Accelerated Delivery Resource Locations
SCO
PE
APP
RO
AC
H Deliver to the 5-year roadmap with constant evaluation for required changes. Address change in an Agile manner.
ALI
GN
MEN
T Resources will be engaged & dedicated to the Pipeline team, ready to assist wherever needed to aid Accelerated Delivery.
Infrastructure as Code
Accelerated Environment Provisioning
Database code back
out and governance
Fast database cloning
Self service environment provisioning
Identify emerging needs for new tools
New toolEnablement
Tool Support and Maintenance
Existing automation supportComplex new Automation
Enablement
New tool Adoption
New Tool R&D
Our Services
Drive new Technical and Service Capabilities
Offshore: India
Onshore: Boston
Our
Mis
sionBy providing DevOps Technical Leadership across the US Division and
some Global areas, our IT Ops/Accelerated Delivery Pipeline Team contributes to our BU IS Partners' ability to deliver & implement high-quality products through multiple DevOps centric Capabilities, DevOps & Accelerated Delivery Techniques.
Offshore: Manila
Capabilities to deliver any new tool, or support any existing tool in the Accelerated Delivery pipeline.
DEVOPS PIPELINE TEAM
JOHN HANCOCK DEVOPS PIPELINE STACK
Monitoring
Code Security
Plan Code
PPMSource Code Mgmt
Build Test OperationalRelease
Unit Test
Environment Provisioning & Infrastructure as Code
Test Data Management
Code Quality
Continuous Deployment
Acceptance Test (Stage)
Deployment
Continuous Integration
CI Toolfor Salesforce
Artifact Mgmt
MS PowerShell
ALMBuild Automation
Object Level Integration
Application Monitoring
System Test SpritzALM
Release Orchestration – Application Release Automation
22
DB Deploy
Security Monitoring
Windocks
DEV(SEC)OPS
23
DEVOPS TO DEVSECOPS
Can we really call it ‘disruption’ anymore if it’s a chronic occurrence?
Change is accelerating
DevOps brought speed, agility, quality and security to the innovation/change process
DevSecOps increased the presence of security as an organizational concern
24
EMPHASIS ON BUSINESS VELOCITY GETS EQUAL EMPHASIS ON SECURITY
Source: Robert Stroud; Xebia Labs
SECURITY IS IN CRISIS
100: 10: 1 Dev: Ops: Sec
There is an inequitable distribution of labor in IT.
Source: Robert Stroud; Xebia Labs
SECURITY KNOWS THERE IS A PROBLEM
Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly something is wrong. The root of the problem is twofold: we’re protecting the wrong things, and we’re hurting productivity in the process.
-Thinking Security, Steven M. Bellovin, 2015
Source: Robert Stroud; Xebia Labs
SECURITY’S NEW CADENCE
Agile and Security meet
Etsy Security Culture in a Fast-paced Dev Shop (deploy code 25 times/day)
Enabling the Paved Road at Netflix (originated Microservices movement)
“many security teams [still] work with a worldview where their
goal is to inhibit change as much as possible”
Source: Robert Stroud; Xebia Labs
Copyright © 2018 Information Systems Audit and Control Association, Inc. All rights reserved.
Placeholder slide for video
DEVOPS IS A CULTURE CHANGE
Creating Awareness
Leadership workshops
Utilizing communication vehicles
Sharing articles (Agile product ownership and The Phoenix Project)
Working Differently
Agile
Product focus
Value delivery
One team
Automation
Speed with quality
Leadership
Experiment, learn culture
Key Considerations
Successful companies act like software companies
Add value to business through software
Get value to market quickly
Business and Technology “work as one”
CHANGE MUST BE EMBRACED ACROSS THE ENTERPRISE
Source: Robert Stroud; Xebia Labs
SECURITY-FIRST, PRODUCT-FOCUSED APPROACHES
Source: Robert Stroud; Xebia Labs
TRANSFORMATION TO PRODUCT TEAMS AND SECURITY
IntegratedProductTeams
Communities of Practice
Servant-Leadership
LOBCMO
CIO
Source: Robert Stroud; Xebia Labs
SECURITY AND COMPLIANCE INTEGRATED
DevCDCI
Prod
QA
UATBuild
Public ComponentRepositories
Source Control DeployRepository
DevelopersSource: Robert Stroud; Xebia Labs
Version Control System
Build
Test
code
Infrastructure-as-Code tools
Artifact Repo
CONTINUOUS INTEGRATION = CONTINUOUSLY HEALTHY CODE: SECURE AND AUDITABLE
Source: Robert Stroud; Xebia Labs
IdeaCustomervalue
Control points
Versionedsource repository
Codeenvironconfigstests
Continuousintegration and testing
Artifactrepository
“Built”artifacts
Backlog
Releasedecision
Releaseautomation
Vendors
Opensource
Developers,Enterprise Architects,
testers, ops, and security
Enterprise Architecture,
developers, Ops,QA, and Security
INTEGRATION OF SECURITY, AUDIT AND CONTROLS: HIGHER VELOCITY AND QUALITY, GREATER TOTAL SECURITY
Source: Robert Stroud; Xebia Labs
DEVSECOPS CORE PRINCIPLES
DevSecOps is the extension of the DevOps culture for the inclusion of Security:
Design for the Worst CaseTest for Security across the PipelineAbandon the AppSec Training Fallacy
35
ASSUME ZERO TRUST
OWASP Top 10 -2017Source: Robert Stroud; Xebia Labs
DESIGNING FOR THE WORST CASE
36
Bulkhead Pattern
Evil User Stories
Threat Modelling
Mozilla Rapid Risk Assessments
DON’T CODE FOR THE HAPPY PATH
http://legacy17.sela.co.il/?CategoryID=552&ArticleID=221&Page=2Source: Robert Stroud; Xebia Labs
TEST FOR SECURITY ACROSS THE PIPELINE
Adversity Testing
Security as CodeInvolves developersSame pattern as “Infra as Code” affectedTest driven development (TDD)
Security TestingStatic application Security Testing (SAST)
Dynamic Application Security Testing (DAST)
Interactive Application Security Testing (IAST)
37
AUTOMATED TESTING EARLY, OFTEN
Source: Robert Stroud; Xebia Labs
TRANSFORM THE APPSEC TRAINING FALLACY
38
Humans write Code
Error Rate
Automation
Instrumentation
Code Hygiene
OWASP Dependency Check
PRACTICE NOT THEORY
OWASP Top 10 -2017Source: Robert Stroud; Xebia Labs
Source: Robert Stroud; Xebia Labs
dev test uat prod[Cloud] Orchestration
Stack Middleware NoSQLPaaS
Containers OS OS
OS IaaS
Network Servers DB / Storage Security
SOFTWARE DEFINED DATA CENTER / CLOUD
RELEASE ORCHESTRATION
Agile Backlog Management
Prov
isio
ning
/ C
onfig
urat
ion
SecurityITSM
CMDB
plan
Project Management
IssueTracking
ALM
DEPLOYMENT AUTOMATION
CONTINUOUS DELIVERY ECOSYSTEM IS A GREAT FIRST STEP IN DEVSECOPS
code
SCM
Code Analysis
build
Continuous Integration
CentralizedRepository
test
Test Tooling
Test Visualization
releaseChatOps /
Collaboration
Email/phone/ Excel
operate
BI /Monitoring
Logging
Source: Robert Stroud; Xebia Labs
SECURITY, GOVERNANCE, AND CONTROLS INTEGRATED WITH CONTINUOUS DELIVERY AND BUSINESS AS USUAL
Integrate security into sprint planning and reviews
Test Driven Development
Security use cases Fuzzing Load Testing
Automated scanning Active log monitoring Rescan for vulnerabilities
Static code analysis
Dynamic code analysis
Patching Dependency
tracking
Audit and compliance data delivered in real time Source: Robert Stroud; Xebia Labs
@mik_kersten project2product.org
UNDERSTAND FLOW TO FOCUS
Source: Robert Stroud; Xebia Labs
@mik_kersten project2product.org
UNDERSTAND YOUR DEVELOPMENT FOCUS
Source: Robert Stroud; Xebia Labs
1.
Run pipeline locally
2.
Integrate quickly and often
3.
Practice test driven development
4.
Keep changes small
5.
Get continuous feedback
KEY PRINCIPLES
7.
Have a fast pipeline
8.
Automated unit testing
9.
Trunk based development
6.
Decomposition
Source: Robert Stroud; Xebia Labs
PILOT TO LEARN FOR SUCCESS, THEN SCALE
Initial CI pipeline implemented
CI pipeline used by 5 volunteer Java/Front End teams
Benefits made visible and demonstrated to senior management
Senior management decision to transition organizationally
Check out project from SCM
Developer triggers build
Build project and execute unit tests
Code quality scan
Publish Deployable
artifact
Source: Robert Stroud; Xebia Labs
ORGANIZATION EXAMPLE FOR COMMUNITIES OF PRACTICE
Requirements Team
Software Logistics Team
Application Deployment Support
Team
Test Tooling Team
Application Monitoring Team
Change & Configuration Management Team
Portfolio Management Team
Application Logging Team
Implement Tooling Upgrades
Implement New Tools
Enhance and Improve CI/CD Pipelines
Implement New CI/CD Pipelines
Handle User Management
Support Agile Teams
Conduct Incident & Problem Management
Mainframe Modernization
Pipelines Team CICD Metrics Team
Source: Robert Stroud; Xebia Labs
BUILD & DELIVERY PIPELINE
Acceptance environment
(ET)
Production environment
(PRD)
Test environment
(ST)
Zero touch platforms
Deployment
Build
Static secure code
Package
Develop
Source code
Build &Unit
Tests
Code quality scans
ContinuousIntegration
Build artifacts
Continuous Delivery
Test data mgmt
ATAF Test suites
Release management
Source: Robert Stroud; Xebia Labs
“Stop Valueless Tool Fights”
Product DeliveryValue Stream
BYOBD“BRING YOUR OWN BUILD/DEPLOY”
DEVSECOPS: STANDARD CI PIPELINES AND BUILD BREAKERSSHIFT SECURITY LEFT – IT BECOMES PART OF BAU – RATHER THAN AN AFTERTHOUGHT
Dependency check
Check out project from SCM
Developer triggers build
Build project and execute unit tests
Code quality scan
Secure coding scan
Publish Deployable
artifact
N
Y
Source: Robert Stroud; Xebia Labs
SECURITY: PART OF THE COMPLETE PIPELINE
Integrate security into sprint planning and reviews
Test Driven Development
Security use cases Fuzzing Load Testing
Automated scanning Active log monitoring Rescan for vulnerabilities
Static code analysis
Dynamic code analysis
Patching Dependency
tracking
Audit and compliance data delivered in real time Source: Robert Stroud; Xebia Labs
REALISED BENEFITS
Test environment uptime improved
Improved code quality & secure coding
Improved cooperation across stakeholders
Improved time to market
Improved development processes
Source code mgt
Build & Unit test
Code quality review
PackageDevelop Compo-nent mgt
Deploy Release tests (ET) Deploy
Continuous integration
Continuous delivery
Continuous deployment
Prod checksDeploy Test (ST)
Zero touch platforms
Code push flow Deployment flowBuild, QA and package flow
x5 deployments to UT x4 deployments to ET+40% successful Builds -100% Package creation time -100% Testing time
We never thought it would be possible to develop, test
and deploy something completely in one sprint
Doubled velocity after 1 sprint
containing CICD improvements only
From 4 Internet Banking releases to 18 releases per
year
Core review times have been shortened and
violations when merging are being prevented
Changes are being rolled out as soon
as they are available
Increased velocity
Private Banking International team
reduced build from 5 hours to 5 minutes
First continuous deployment realised by
identity access mgmtteam
Release times halved for teams using XL
Release
Source: Robert Stroud; Xebia Labs
Copyright © 2018 Information Systems Audit and Control Association, Inc. All rights reserved.
EXECUTION
Trust
Learning culture
Integrate teams – critical to success
Integrate security risk and compliance tools into your toolchains
Feedback loops
Automate, automate, automate
Source: Robert Stroud; Xebia Labs
Security, audit and compliance teams should be working closely with product teams