Photograph © Tom Poppendieck
Mary [email protected] 1
Copyright©2016 Poppendieck.LLC
2Photograph © Tom Poppendieck
Business Software:
Why One Server?
Monolithic & Slow to changeERP Database IntegrationOn a Single Server
CAP Theorem:Partitioned databasesmust choose between:
AvailabilityConsistency
Photograph © Tom Poppendieck
Copyright©2016 Poppendieck.LLC
3
[c. 2001]
Copyright©2016 Poppendieck.LLC
4
Get a bigger computer.
Copyright©2016 Poppendieck.LLC
5
Get more computers. Google Hardware c. 1999
Search the entire InternetInstantly
Response:Horizontal ScalingBreak files into small piecesMake three copies of each pieceSpread the pieces across many servers
Manage the pieces with software
Google’s Problem:
Copyright©2016 Poppendieck.LLC
6
Amazon ~ 2001Handle a gazillion transactionsAll at once
What Amazon Did:Break transactions into servicesReplicate bottleneck services
Each service owned by a semi‐autonomous “two pizza” team
Option 2:Scale Out
Option 1:Scale Up
Copyright©2016 Poppendieck.LLC
7
2001
2003Google File System Paper
2004 Google MapReduce Paper
2006‐2012 Hadoop Grows Up
Doug Cutting, joined by Mike Cafarella
Web Crawler
Copyright©2016 Poppendieck.LLC
8
Photograph © Tom PoppendieckCopyright©2016 Poppendieck.LLC
9
Pinkham moves to South AfricaAsked to pursue project thereAssembles and leads a teamDevelops EC2 in 2 years
EC2 Launches in 200610 Years Later: The Cloud is cheaper, more stable, more secure, and more expandable than most on‐premises data centers.
Autonomous Services Autonomous Service Teams Independent Deployment
Much Stress in Operations
Chris Pinkham (VP Infrastructure) Proposes self‐service deploymentfor development teamsMaybe sell the capability?
Time Passes….
Photograph © Tom PoppendieckCopyright©2016 Poppendieck.LLC 10
The Evolution of IT1996 2016
Running on any available set of
physical resources(public/private/virtualized)
Assembled by developers using best available
services
Thin app on mobile, tabletThick, client‐server app
on thick client
Well‐defined stack:‐ O/S‐ Runtime‐ Middleware
MonolithicPhysical
Infrastructure
www.docker.io11
Scale Out
Technology Platform
Scale Up
EnterpriseArchitecture
Federated Architecture
Copyright©2016 Poppendieck.LLC 12
Copyright©2016 Poppendieck.LLC 13
The GripenCost Per Hour Compared to Competitors
Designing for scale: Resilient Architectures:Big Data systems are inherently distributed; their architectures must explicitly handle:
partial failuresconcurrencyconsistencyreplication communications latencies
Replicate data to ensure availability in the case of failure. Design components to be
statelessreplicatedtolerant of failures of dependent services
Distribution, Data, Deployment: Software Architecture Convergence in Big Data SystemsIan Gorton, John Klein, Software Engineering Institute Carnegie Mellon University, May 2014
“Big data applications are pushing the limits of software engineering …. It is essential that the body of software architecture knowledge evolves to capture this advanced design knowledge for big data systems.”
Photograph © Tom Poppendieck
Copyright©2016 Poppendieck.LLC
14
Photograph © Tom PoppendieckCopyright©2016 Poppendieck.LLC
15
For Complex SystemsThis does not work
Copyright©2016 Poppendieck.LLC
16
For Complex SystemsThis works
Copyright©2016 Poppendieck.LLC
17
Photograph © Tom Poppendieck
Copyright©2015 Poppendieck.LLC
18
Specification as Code
Functional Layer of ApplicationFixture
TestTool
Specification[by Example]
Designers SMEs Testers Engineers
ConfigurationManagement Copyright©2016
Poppendieck.LLC19
Photograph © Tom Poppendieck
TestReport
Feature: User trades stocks Scenario: User requests a sell before close of trading
Given I have 100 shares of MSFT stock And I have 150 shares of APPL stock And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Then I should have 80 shares of MSFT stock And I should have 150 shares of APPL stock And a sell order for 20 shares of MSFT stock should have been executed
From Martin Fowler.com
Consumer Provider
https://github.com/realestate-com-au/pact
Request Response
Mock
Does the Response
behave the same as the
Mock??
Request Response
PACT
Moc
khttp://martinfowler.com/articles/consumerDrivenContracts.html
Mock
Beth Skurrie
Martin Fowler
Copyright©2015 Poppendieck.LLC 20
Copyright©2016 Poppendieck.LLC
21
Acceptance test driven development processCross‐functional teams include Product, QA and OpsAutomated build, testing, db migration, and deploymentIncremental development on the mainline with continuous integrationSoftware always production ready, or everything stops until it isDeploy constantly, release by switch
Slide content credit: Jez Humble
Software Engineering for the Cloud
Copyright©2016 Poppendieck.LLC
22Photograph © Tom Poppendieck
IBM’s Agile Principles
Clarity is more important than CertaintyCourse Correction is more important than PerfectionSelf‐directed Teams work better than Control StructuresCollective Wisdom outweighs Individual Insights
Full Stack Teams of 8‐10 do the whole job: design, build, deployLeadership Role:
Build Strong TeamsCreate Clarity of PurposeCreate a Productive EnvironmentInspire People to do Great ThingsMeasure (only) what matters
Copyright©2016 Poppendieck.LLC 23
Jeff Smith, IBM CIO
Copyright©2016 Poppendieck.LLC
24
Photograph © Tom Poppendieck
Photograph © Tom Poppendieck
Online Experimentation at Microsoftby Kohavi , Crook, & Longbotham
Presented at KDD 2009(Knowledge Discovery & Data Mining)
http://exp‐platform.com/expMicrosoft.aspxCopyright©2016 Poppendieck.LLC
25
Photograph © Dustin Poppendieck
Standish Group Study Reported at XP2002 by Jim Johnson, Chairman
Copyright©2016 Poppendieck.LLC
26
Delivery Problem Solving
Copyright©2016 Poppendieck.LLC 27
Start with SignalsFocus on ProblemsPlan with Hypotheses
Do MultipleExperiments
Use Data to Decide
Not Requirements Not FeaturesNot Estimates
Not a Backlogof Stories
Don’t Guess ata Solution
Signals /Patterns
Problem Statement
HypothesisExperiment
Analysis & Conclusion
Copyright©2016 Poppendieck.LLC
28
To The Product MindsetFrom The IT Mindset*
Copyright©2016 Poppendieck.LLC 29
Focus on CustomersEntrepreneur Problem‐solving Engineering TeamSuccess = Delighted CustomersTough Tradeoffs are Made Often,Based on Market RealitiesProfit Center Mentality:Spend Money to Make Money
Focus on “The Business”Project ManagerOrder‐taking Development TeamSuccess = Cost, Schedule, ScopeTough Tradeoffs are Made During the Planning ProcessCost Center Mentality:Constantly Reduce Costs
*Thanks to Marty Cagan
Photograph © Tom Poppendieck
Our competitive advantage is Digital Productivity.
Winning requires simpler organizations. Change requires new business models that are leaner, faster, more decentralized. Complex centralized bureaucracies areobsolete. Annual processes are too slow.Push capability to local teams who are empowered to take risks.If you are joining GE in your 20’s you’re going to learn to code. It doesn’t matter whether you are in sales, finance or operations. You may not be a programmer, but you will know how to code.
Copyright©2016 Poppendieck.LLC
30
The Industrial Internet of ThingsUse Data to Increase Asset Productivity
Critical Skills: Science, SecurityCritical Challenge: Attract and challenge talented people
Copyright©2016 Poppendieck.LLC
31
Federated ArchitectureTests as SpecificationContinuous DeliveryFault ToleraceConstantly Evolving ProductsProblemsHypothesesFull Stack TeamsOutsource Infrastructure
Don’t doMonolithic ArchitectureTesting at the EndPeriodic Releases Five NinesProjectsFeaturesEstimatesITOutsource Development
Do
Copyright©2016 Poppendieck.LLC
32
Photograph © Tom Poppendieck 33
Mary [email protected]