Upload
others
View
1
Download
1
Embed Size (px)
Citation preview
Business Value ofCI, CD, & DevOpsScaling Up to Billion User
Global Systems of Systems Using End-to-End AutomationDr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFe
Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf
Author Background Gov’t contractor with 32+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe
2
Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 150 talks to 12,000+ people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at five Washington, DC-area universities
Today’s Whirlwind Environment
3
OverrunsAttritionEscalationRunawaysCancellation
GlobalCompetition
DemandingCustomers
OrganizationDownsizing
SystemComplexity
TechnologyChange
VagueRequirements
Work LifeImbalance
InefficiencyHigh O&MLower DoQVulnerableN-M Breach
ReducedIT Budgets
81 MonthCycle Times
RedundantData Centers
Lack ofInteroperability
PoorIT Security
OverburdeningLegacy Systems
ObsoleteTechnology & Skills
Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
Global Project Failures
4Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trill
ions
(US
Dolla
rs)
Expenditures Failed Investments
0% 20% 40% 60% 80% 100%
28%
34%
29%
35%
32%
33%
27%
28%
29%
49%
51%
53%
46%
44%
41%
56%
55%
52%
23%
15%
18%
19%
24%
26%
17%
17%
19%
2000
2002
2004
2006
2008
2010
2012
2014
2015
Year
Successful Challenged Failed
Software in U.S. DoD Systems
Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.
No. of software-intensive systems is growing 80% of US DoD functions performed in software Major driver of cost, schedule, & tech. performance
5
Requirements Defects & Waste
6
Sheldon, F. T. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20.Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.Goffin, K., & Mitchell, R. (2005). Innovation management: Strategy and implementation. London, UK: Palgrave-Macmillan.Chedalawada, A. (2012). Lean-agile overview. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
Requirements defects are #1 reason projects fail 80% of requirements exist only as tacit knowledge 65% to 95% of explicit requirements are never used
Other 7%
Requirements47%
Design28%
Implementation18%
Defects
Always 7%
Often 13%
Sometimes16%
Rarely19%
Never45%
Waste
Large Traditional Projects
7
Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DE
FEC
TS
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
SIZE
Size vs. Productivity
PR
OD
UC
TIV
ITY
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
SIZE
Size vs. Change
CH
AN
GE
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
SIZE
Size vs. SuccessS
UC
CE
SS
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
SIZE
Today’s Global Marketplace Most of world’s population connected to Internet Systems must support billions of simultaneous users New approaches are needed to scale to global market
8Kemp, S. (2016). Digital in 2016: We are social's compendium of global digital, social, and mobile data, trends, and statistics. New York, NY: We Are Social, Inc.
What is Agility? A-gil-i-ty (ә-'ji-lә-tē) Property consisting of quickness,
lightness, and ease of movement; To be very nimble The ability to create and respond to change in order to
profit in a turbulent global business environment The ability to quickly reprioritize use of resources when
requirements, technology, and knowledge shift A very fast response to sudden market changes and
emerging threats by intensive customer interaction Use of evolutionary, incremental, and iterative delivery
to converge on an optimal customer solution Maximizing BUSINESS VALUE with right sized, just-
enough, and just-in-time processes and documentationHighsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
9
What are Agile Methods?
10
People-centric way to create innovative solutions Product-centric alternative to documents/process Market-centric model to maximize business value
Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.orgRico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf
Customer Collaboration
Working Systems & Software
Individuals & Interactions
Responding to Change
valuedmore than
valuedmore than
valuedmore than
valuedmore than
Contracts
Documentation
Processes
Project Plans
Frequent comm. Close proximity Regular meetings
Multiple comm. channels Frequent feedback Relationship strength
Leadership Boundaries Empowerment
Competence Structure Manageability/Motivation
Clear objectives Small/feasible scope Acceptance criteria
Timeboxed iterations Valid operational results Regular cadence/intervals
Org. flexibility Mgt. flexibility Process flexibility
System flexibility Technology flexibility Infrastructure flexibility
Contract compliance Contract deliverables Contract change orders
Lifecycle compliance Process Maturity Level Regulatory compliance
Document deliveries Document comments Document compliance
Cost Compliance Scope Compliance Schedule Compliance
Courage
How do Lean & Agile INTERSECT?
11
Agile is naturally lean and based on small batches Agile directly supports six principles of lean thinking Agile may be converted to a continuous flow system
Womack, J. P., & Jones, D. T. (1996). Lean thinking: Banish waste and create wealth in your corporation. New York, NY: Free Press.Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. New York, NY: Celeritas.Reagan, R. B., & Rico, D. F. (2010). Lean and agile acquisition and systems engineering: A paradigm whose time has come. DoD AT&L Magazine, 39(6).
Economic View
Decentralization
Fast Feedback
Control Cadence& Small Batches
Manage Queues/Exploit Variability
WIP Constraints& Kanban
Flow PrinciplesAgile Values
CustomerCollaboration
EmpoweredTeams
IterativeDelivery
Respondingto Change
Lean Pillars
Respectfor People
ContinuousImprovement
Customer Value
Relationships
Customer Pull
Continuous Flow
Perfection
Value Stream
Lean Principles Customer relationships, satisfaction, trust, and loyalty Team authority, empowerment, and resources Team identification, cohesion, and communication
Lean & Agile Practices
Product vision, mission, needs, and capabilities Product scope, constraints, and business value Product objectives, specifications, and performance As is policies, processes, procedures, and instructions To be business processes, flowcharts, and swim lanes Initial workflow analysis, metrication, and optimization Batch size, work in process, and artifact size constraints Cadence, queue size, buffers, slack, and bottlenecks Workflow, test, integration, and deployment automation Roadmaps, releases, iterations, and product priorities Epics, themes, feature sets, features, and user stories Product demonstrations, feedback, and new backlogs Refactor, test driven design, and continuous integration Standups, retrospectives, and process improvements Organization, project, and process adaptability/flexibility
NetworkComputer
Operating SystemMiddlewareApplications
APIsGUI
How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success
12Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional1 2 3 Faster
Early ROI
Lower Costs
Fewer Defects
Manageable Risk
Better Performance
Smaller Attack Surface
Late
No Value
Cost Overruns
Very Poor Quality
Uncontrollable Risk
Slowest Performance
More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction
MINIMIZES MAXIMIZES
JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes
Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
13
Methods to “scope” project, product, or system “Key” is smallest possible scope with highest value Reduces cost, risk, time, failure, & tech. obsolescence
Barely Sufficient Design
INCREASES TESTABILITY, QUALITY, RELIABILITY, SECURITY, MORALE, MAINTAINABILITY, & SUCCESS
Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.
MINIMUM
MARKETABLE FEATURE- MMF -
AdvantageDifferenceRevenueProfitSavingsBrandLoyalty
MINIMUMVIABLE PRODUCT
- MVP -
GoalProcessFeaturesPrioritiesStory MapArchitecture
STORY MAPOR IMPACT MAP
- SM or IM -
GoalActors ImpactsDeliverablesMeasuresMilestones
VISIONSTATEMENT
- VS -
For <customer>Who <needs it>The <product> Is a <benefit>That <customer>Unlike <other>Ours <different>
MICRO-SERVICE- MS -
PurposeAutomatedUnique IndependentResilientEcosystemConsumer
14
Capability #1
● Feature 1● Feature 2● Feature 3● Feature 4● Feature 5● Feature 6● Feature 7
Capability #2
● Feature 8● Feature 9● Feature 10● Feature 11● Feature 12● Feature 13● Feature 14
Capability #3
● Feature 15● Feature 16● Feature 17● Feature 18● Feature 19● Feature 20● Feature 21
Capability #4
● Feature 22● Feature 23● Feature 24● Feature 25● Feature 26● Feature 27● Feature 28
Capability #5
● Feature 29● Feature 30● Feature 31● Feature 32● Feature 33● Feature 34● Feature 35
Capability #6
● Feature 36● Feature 37● Feature 38● Feature 39● Feature 40● Feature 41● Feature 42
Capability #7
● Feature 43● Feature 44● Feature 45● Feature 46● Feature 47● Feature 48● Feature 49
1
2 3
4
5 6
7
8 9
10
11 12
13
14 15
16
17 18
19
20 21
Evolving “Unified/Integrated” Enterprise Data Model
“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)
ETL
A A
B C
D E F
G H I J K
A
B C
D E F
A
B C
D E
A
B C
D
A
B C
A
B
“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server
“Leased” DWA/HPC/Cloud Services
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
ETL ETL ETL ETL ETL ETL
Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.
(for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total)
Organize needs into capabilities, features, and stories Prioritize features, group releases, and initiate sprints Develop minimum set of features with highest value
Agile Systems Development
Release
Release
Release
ReleaseMMF- or -MVP
Five Keys to Enterprise Agility
15Kim, G., Debois, P., Willis, J., & Humble, J. The devops handbook: How to create world-class agility, reliability, and security in technology organizations. Portland, OR: IT Revolution Press.
Everything begins with lean & agile principles Next step is smaller portfolio & simpler designs Final step is modular interfaces & E2E automation
Models of AGILE METHODS
16
Agile methods spunoff flexible manufacturing 1990s Extreme Programming (XP) swept the globe by 2002 Today, over 90% of IT projects use Scrum/XP hybrid
Use Cases
Domain Model
Object Oriented
Iterative Dev.
Risk Planning
Info. Radiators
Planning Poker
Product Backlog
Sprint Backlog
2-4 Week Sprint
Daily Standup
Sprint Demo
Feasibility
Business Study
Func. Iteration
Design Iteration
Implementation
Testing
Domain Model
Feature List
Object Oriented
Iterative Dev.
Code Inspection
Testing
Release Plans
User Stories
Pair Programmer
Iterative Dev.
Test First Dev.
Onsite Customer
Cockburn, A. (2002). Agile software development. Boston, MA: Addison-Wesley.Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.Stapleton, J. (1997). DSDM: A framework for business centered development. Harlow, England: Addison-Wesley.Palmer, S. R., & Felsing, J. M. (2002). A practical guide to feature driven development. Upper Saddle River, NJ: Prentice-Hall.Beck, K. (2000). Extreme programming explained: Embrace change. Reading, MA: Addison-Wesley.
CRYSTAL METHODS- 1991 -
SCRUM- 1993 -
DSDM- 1993 -
FDD- 1997 -
XP- 1998 -
Reflection W/S Retrospective Quality Control Quality Control Continuous Del.
Basic SCRUM Method
Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.
Created by Jeff Sutherland at Easel in 1993 Product backlog comprised of prioritized features Iterative sprint-to-sprint, adaptive & emergent model
17
18
Numerous models of lean development emerging Based on principles of lean thinking & just-in-time Include software, project, & product management
Poppendieck, M., & Poppendieck, T. (2003). Lean software development: An agile toolkit for software development managers. Boston, MA: Addison Wesley.Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. New York, NY: Celeritas.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.Olsen, D. (2015). The lean product playbook: How to innovate with minimum viable products and rapid customer feedback. Hoboken, NJ: John Wiley & Sons.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
LEAN SOFTWARE- 2003 -
LEAN PRINCIPLES- 2009 -
LEAN KANBAN- 2010 -
LEAN PRODUCTS- 2015 -
LEAN ENTERPRISES- 2015 -
Create Value
Eliminate Waste
Amplify Learning
Late Decisions
Deliver Fast
Empower Team
Build-in Integrity
See the Whole
Economic View
Manage Queues
Use Variability
Small Batches
WIP Constraints
Flow Control
Fast Feedback
Decentralize
Visualize
Limit WIP
Manage Flow
Use Policies
Quality Focus
Lead Times
Improvement
Reduce Variation
Target Customer
Market Needs
Market Value
Min. Viability
Prototype
User Experience
Market Testing
Improvement
Measure Risks
Uncertainty
Marketing
Improvement
Value & Flow
Lean Engineering
Experimentation
Bus. Alignment
Models of LEAN METHODS
Basic KANBAN Method
Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Adapted to IT by Dave Anderson in 2006 Activities, buffers, queues, WIP limits, tasks, etc. Lean, JIT pull/demand system leading to high quality
19
Models of AGILE PROJECT MGT.
20
Dozens of Agile project management models emerged Many stem from principles of Extreme Programming Vision, releases, & iterative development common
Prioritization
Feasibility
Planning
Tracking
Reporting
Review
Visionate
Speculate
Innovate
Re-Evaluate
Disseminate
Terminate
Scoping
Planning
Feasibility
Cyclical Dev.
Checkpoint
Review
Envision
Speculate
Explore
Iterate
Launch
Close
Vision
Roadmap
Release Plan
Sprint Plan
Daily Scrum
Retrospective
Thomsett, R. (2002). Radical project management. Upper Saddle River, NJ: Prentice-Hall.DeCarlo, D. (2004). Extreme project management: Using leadership, principles, and tools to deliver value in the face of volatility. San Francisco, CA: Jossey-Bass.Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.Highsmith, J. A. (2010). Agile project management: Creating innovative products. Boston, MA: Pearson Education.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
RADICAL- 2002 -
EXTREME- 2004 -
ADAPTIVE- 2010 -
AGILE- 2010-
SIMPLIFIED APM- 2011 -
Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
Created by Mark Layton at PlatinumEdge in 2012 Mix of new product development, XP, and Scrum Simplified codification of XP and Scrum hybrid
21
Basic AGILE PROJECT MGT. Method
22
Numerous models of agile portfolio mgt. emerging Based on lean-kanban, release planning, and Scrum Include organization, program, & project management
Schwaber, K. (2007). The enterprise and scrum. Redmond, WA: Microsoft Press.Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.Larman, C., & Vodde, B. (2008). Scaling lean and agile development: Thinking and organizational tools for large-scale scrum. Boston, MA: Addison-Wesley.Ambler, S. W., & Lines, M. (2012). Disciplined agile delivery: A practitioner's guide to agile software delivery in the enterprise. Boston, MA: Pearson Education.Thompson, K. (2013). cPrime’s R.A.G.E. is unleashed: Agile leaders rejoice! Retrieved March 28, 2014, from http://www.cprime.com/tag/agile-governanceSchwaber, K. (2015). The definitive guide to nexus: The exoskeleton of scaled scrum development. Lexington, MA: Scrum.Org
Models of AGILE FRAMEWORKS
ESCRUM- 2007 -
SAFe- 2007 -
LESS- 2007 -
DAD- 2012 -
RAGE- 2013 -
SPS- 2015 -
Product Mgt
Program Mgt
Project Mgt
Process Mgt
Business Mgt
Market Mgt
Strategic Mgt
Portfolio Mgt
Program Mgt
Team Mgt
Quality Mgt
Delivery Mgt
Business Mgt
Portfolio Mgt
Product Mgt
Area Mgt
Sprint Mgt
Release Mgt
Business Mgt
Portfolio Mgt
Inception
Construction
Iterations
Transition
Business
Governance
Portfolio
Program
Project
Delivery
Product Mgt
Program Mgt
Sprint Mgt
Team Mgt.
Integ Mgt.
Release Mgt
Created by Dean Leffingwell of Rally in 2007 Knowledge to scale agile practices to enterprise Hybrid of Kanban, XP release planning, and Scrum
23
Basic SAFE 4.0 Method
Leffingwell, D. (2017). SAFe reference guide: Scaled agile framework for lean software and systems engineering. Boston, MA: Pearson Education.
24
Numerous models of lean-agile testing emerging Based on principles of lean & agile one piece flow Include software, hardware, system, & port. testing
TDD- 2003 -
CI- 2006 -
BD- 2008 -
CD- 2011 -
DEVOPS- 2012 -
User Story
Acc Criteria
Dev Unit Test
Run Unit Test
Write SW Unit
Re-Run Unit Test
Refactor Unit
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Analyze Feature
Acc Criteria
Dev Feature Test
Run Feature Test
Develop Feature
Re-Run Feature
Refactor Feature
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerization
Deployment
Monitor & Supp
Models of AGILE TESTING
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Barker, K., & Humphries, C. (2008). Foundations of rspec: Behavior driven development with ruby and rails. New York, NY: Apress.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Huttermann, M. (2012). Devops for developers: Integrate development and operations the agile way. New York, NY: Apress.
What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components
25Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
AGILE TESTING- Early Incremental Testing -
TRADITIONAL TESTING- Late Big Bang Integration Testing -
Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at TimeCode is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports
Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One TimeCode Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports
Code Automatically DeployedLate Defects Freeze Projects
Thousands of TestsContinuously Executed
No More Late BigBang Integration
User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur
26Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
BuildIntegration
Server
VersionControlServer
BuildScripts
UsesWatches
BuildStatus
ProvidesDeveloper A
Developer B
Developer C
CommitsChanges
CommitsChanges
CommitsChanges
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,Efficient, & Repeatable
Constant ReadinessState & CM Control
Lean, Waste Free, Low WIP,No Deadlocked Test Queues
Rapidly & SuccessfullyDev. Complex Systems
Agile Testing—Workflow
BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated
27Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
Agile TDD consists of seven broad practices Document test criteria, tests, software units, etc. Include refactoring, verification, optimization, etc.
28
Practice
User Story
Acc Criteria
Dev Test
Run Test
Dev Unit
Rerun Test
Refactor Unit
Description
Read story, analyze meaning, ask questions, and clarify understanding
Identify, verify, and document acceptance criteria for each user story
Design, develop, code, and verify automated unit test for user story
Run automated unit test to verify that it fails the first time (sanity check)
Design, develop, code, and verify the software unit to satisfy user story
Rerun automated unit test to see if code satisfies automated unit test
Refine, reduce, and simplify code to remove waste and optimize performance
PRACTICES—Test Driven Develop.
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
BASIC—Behavior Driven Develop. Term coined by Dan North in 2006 Consists of writing feature tests before design Ensures all system features are verified and validated
29Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.
Agile BDD consists of seven broad practices Document test criteria, tests, syst. features, etc. Include refactoring, verification, optimization, etc.
30
Practice
Feature
Acc Criteria
Dev Test
Run Test
Dev Feature
Rerun Test
Refac Feature
Description
Read feature, analyze meaning, ask questions, and clarify understanding
Identify, verify, and document acceptance criteria for each feature
Design, develop, code, and verify automated feature test for feature
Run automated feature test to verify that it fails the first time (sanity check)
Design, develop, code, and verify the feature software to satisfy feature
Rerun automated feature test to see if code satisfies automated feature test
Refine, reduce, and simplify code to remove waste and optimize performance
PRACTICES—Behavior Driven Dev.
Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.
ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system
31Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
ALL DEVELOPERS RUN PRIVATE BUILDS
DEVELOPERS COMMIT CODE TO VERSION CONTROL
INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
REPORTS AUTOMATICALLY GENERATED & REVIEWED
Agile CI consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.
32
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
PRACTICES—Continuous Integration
Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline
33Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.
CoQ
• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK
ENTERPRISE—Continuous Delivery
Agile CD consists of seven broad practices Automated acceptance, load, performance, etc. Include packaging, pre-production test, C&A, etc.
34
Practice
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Description
Frequently generating system images for pre-production testing & checkout
Frequently performing automated system & user acceptance testing
Frequently performing automated system load, stress, & capacity testing
Frequently performing automated system user & technical performance testing
Frequently performing automated pre-production tests prior to final deployment
Frequently performing automated system certification & accreditation tests
Frequently generating product images for pre-deployment testing & checkout
Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.
PRACTICES—Continuous Delivery
Created by Patrick Debois of Jedi BVBA in 2007 Collaboration of developers & infrastructure people Goal to automate the deployment to end-user devices
35Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
GLOBAL—Development Operations
Agile DevOps consists of seven broad practices Automated sys admin, CM, building, monitor, etc. Include virtualization, containerize, deployment, etc.
36
Practice
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerize
Deployment
Monitor & Supp
Description
Frequently performing automated system administration tasks, i.e., scripting
Frequently performing automated infrastructure config. mgt./version control
Frequently performing automated system and server host builds and config.
Frequently performing automated system, server, & net virtualization services
Frequently performing automated software and Microservices containerization
Frequently generating final end-user system & software images for distribution
Frequently performing automated metrics collection & deployment monitoring
Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.
PRACTICES—Development Operations
Agile Testing Metrics—Example
37Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
38
Traditional vs. Agile Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Agile Testing—Workflow—Cont’d
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing
39
Agile Testing—CI Statistics
40
Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.
TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 x
Regression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
Agile Testing—CD Statistics
Assembla went from 2 to 45 releases every month 15K Google developers run 120 million tests per day 30K+ Amazon developers deliver 136K releases a day
41Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
62x FasterU.S. DoD
IT Project
3,645x FasterU.S. DoD
IT Project
Agile Testing—DevOps Statistics
Activity Def CoQ DevOps Economics Hours ROIDevelopment Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%
Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%
Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%
Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%
"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%
Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%
Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a
42
Agile testing is orders-of-magnitude more efficient Based on millions of automated tests run in seconds One-touch auto-delivery to billions of global end-users
Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com
Agile Testing—DevOps CoQ
43
Agile Tools“Periodic Table of Testing Automation”
XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.
Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention
44Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
MICRO ADJUSTMENTS- Focused Impact Tuning-
MACRO ADJUSTMENTS- Wide Impact Tuning-
Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test LibrariesReduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware
In-Memory CompilationParallelize Test RunsPre-Install Test LibrariesRemove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process
Agile Testing—Scaling Practices
Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing
45
Technical BarriersOrganizational BarriersDevelopers don’t want to test
· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing
Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture
··
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing—Common Barriers
Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project
46Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
What’s the Bottom Line?“Agile Testing Done Early & Often”
Agile TestingTraditional TestingDramatically reduces risks
· Automates manual processes· Instant verification & validation· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability
Late defect discovery· Low quality software· Poor project visibility· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure
··
Key Agile SCALING POINTERS One must think and act small to accomplish big things Slow down to speed up, speed up ‘til wheels come off Scaling up lowers productivity, quality, & business value
47Rico, D. F. (2014). Dave's Notes: For Scaling with SAFe, DaD, LeSS, RAGE, ScrumPLoP, Enterprise Scrum, etc. Retrieved March 28, 2014 from http://davidfrico.com
EMPOWER WORKFORCE - Allow workers to help establish enterprise business goals and objectives.
ALIGN BUSINESS VALUE - Align and focus agile teams on delivering business value to the enterprise.
PERFORM VISIONING - Frequently communicate portfolio, project, and team vision on continuous basis.
REDUCE SIZE - Reduce sizes of agile portfolios, acquisitions, products, programs, projects, and teams.
ACT SMALL - Get large agile teams to act, behave, collaborate, communicate, and perform like small ones.
BE SMALL - Get small projects to act, behave, and collaborate like small ones instead of trying to act larger.
ACT COLLOCATED - Get virtual distributed teams to act, behave, communicate and perform like collocated ones.
USE SMALL ACQUISITION BATCHES - Organize suppliers to rapidly deliver new capabilities and quickly reprioritize.
USE LEAN-AGILE CONTRACTS - Use collaborative contracts to share responsibility instead of adversarial legal ones.
USE ENTERPRISE AUTOMATION - Automate everything with Continuous Integration, Continuous Delivery, & DevOps.
Conclusion Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality
48Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdfRico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdfRico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance
Books on Agile Testing Thousands of textbooks on agile methods Include requirements, design, coding, test, etc. Continuous Integration, Delivery, & DevOps best
49
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Kim, G., Debois, P., Willis, J., & Humble, J. The devops handbook: How to create world-class agility, reliability, and security in technology organizations. Portland, OR: IT Revolution Press.
Dave’s PROFESSIONAL CAPABILITIES
50
SoftwareQuality
Mgt.
TechnicalProject
Mgt.
SoftwareDevelopment
Methods
Strategy &Roadmapping
SystemsEngineering
Cost Estimates& Scheduling
Acquisition &Contracting
OrganizationChange
Lean, Kanban,& Six Sigma
Modeling &Simulations
Big Data,Cloud, NoSQL
WorkflowAutomation
Metrics,Models, & SPC
BPR, IDEF0,& DoDAF
DoD 5000,TRA, & SRA
PSP, TSP, &Code Reviews
CMMI &ISO 9001
InnovationManagement
Statistics, CFA,EFA, & SEM
ResearchMethods
EvolutionaryDesign
Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.
STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.
● Data mining. Metrics, benchmarks, & performance.● Simplification. Refactoring, refinement, & streamlining.● Assessments. Audits, reviews, appraisals, & risk analysis.● Coaching. Diagnosing, debugging, & restarting stalled projects.● Business cases. Cost, benefit, & return-on-investment (ROI) analysis.● Communications. Executive summaries, white papers, & lightning talks.● Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.
PMP, CSEP,FCP, FCT
ACP, CSM,& SAFE
33 YEARSIN IT
INDUSTRY