Embed Size (px)
Citation preview
Tuning our Agile ProcessUsing Systems Engineering
Mike Day
Director of Software Quality
January 27, 2011
Who Am I…
BackgroundUndergraduate Degrees in Business Administration, Finance, Economics
Industries include Manufacturing, Non-Profit, Health Care, Telco
Masters Degree in Information Systems
Industries include Telco, Open Source, Enterprise and SaaS development
Practicing Agile for 10+ yearsConsulting/Agile Coaching - late 90s/early 2000s
Ping Identity Agile Quality Team development
PassionsSystems Dynamics, Data Mining and Modeling, and the Growth and Leadership of Highly Performant Teams
Who is Ping Identity
THE leader in Secure Internet Single Sign-OnPingFederate – On-Premise, Enterprise SSO solution
PingConnect – On-Demand SSO solution for SaaS and the Cloud
~ 600 Customers Worldwide – growing at 1+ per day
> 40% of Fortune 100 Companies
100+ Global Partners
Founded in 2002 – Privately OwnedStrong Cash position – Debt Free
Consistently Meeting or Exceeding Financial Goals
Headquartered in Denver
Offices in Boston, Istanbul, London, Moscow & Vancouver
Worldwide sales and services
Define:Systems Engineering
Systems Engineering an interdisciplinary field of engineering that focuses on how complex engineering projects should be designed and managed over the life cycle of the project. Issues such as logistics, the coordination of different teams, and automatic control of machinery become more difficult when dealing with large, complex projects. Systems engineering deals with work-processes and tools to handle such projects, and it overlaps with both technical and human-centered disciplines such as control engineering, industrial engineering, organizational studies, and project management.
System Dynamics an approach to understanding the behaviour of complex systems over time. It deals with internal feedback loops and time delays that affect the behaviour of the entire system
Agile @ Ping
Plan
Releases
Product
Owners
ReleasesIterateIterate
Customers
Define
System
Product
Planning
V2
V1
?Accept? ?
Inception Inception Transition Transition
IterateIterateIterateIterate IterateIterate IterateIterate
FeaturesFeatures FeaturesFeatures FeaturesFeatures FeaturesFeatures
IterateIterate
IterateIterate
IterateIterate
IterateIterate
Hardened FeaturesHardened FeaturesIntegrated ProductIntegrated ProductRolloutRollout
• Business analysis thinking-throughoutBusiness analysis thinking-throughout• Requirements Elaboration throughoutRequirements Elaboration throughout• Requirements Matrix-built progressivelyRequirements Matrix-built progressively• Supplemental SpecificationSupplemental Specification• In-Flight Agile Dashboard drives feature priority throughoutIn-Flight Agile Dashboard drives feature priority throughout
Elaboration and ConstructionElaboration and Construction
QueueQueue
Agile @ Ping
• 2006: 26 Releases
• 2007: 28 Releases
• 2008: 22 Releases
• 2009: 40 Releases
• 2010: 39 Releases
• & Low Field Defects (94% DDP)
• Agile From Initial Dev Team Staffing 7 Years • Distributed Agile: Teams/Individuals in 4 Countries• Fixed Timeboxes < 3% slips ~12 days/7 years
FlywheelExpectations
Agile Pain Points… Throughout the Years
Management and MitigationQuality DebtRisk
Team Development and DynamicsVelocityDelivery to a time-boxed cadenceCommitment to Quality
Systems Engineering
Quality Debt Management
Key Drivers for AnalysisDelays in Addressing DefectsFeature Development vs. Defect CorrectionTeams are DifferentMinimum Quality Criteria
Data Mining and Modeling Addressing ConcernsEngineering DashboardsQuality Debt PaintballReliability Growth ModelingDefect Lifetime AnalysisDefect Detection Percentage
Engineering Dashboard
Quality Debt Paintball
Reliability Growth Modeling
Defect Lifetime Analysis
82%
Defect Detection Percentage
162 Total Field Escapes since July 2007
Risk Management and Mitigation
Key Drivers for AnalysisExtremely Feature Rich Product OfferingExecution of More with LessTeams are DifferentMinimum Quality Criteria
Data Mining and Modeling Addressing ConcernsUnderstanding the Product LongtailRisk Mitigation ModelingCode Change TreeMaps
Understanding the Product Long tail
Risk Mitigation Modeling
Risk Calculcation Table
Ping FactorsCustomer Risk
(Impact of Failure)Quality Risk
(Probability of Failure)Complexity Dev
Whitneys / Wus'
Complexity SQE
Whitneys% Testing Complete
Weighted Risk
RemainingUsage
FrequencyBusiness Criticality
Defect Prone Areas
Changed Areas
Code Complexity
TOTAL RISK
Weights Weights 10 3 5 10 3
Info Only Not Used in Calculations
Used in Risk Calucation as Multiplier
Based on Matrix Completion
SQE Complexity / % Testing Remaining
Scoring to be provided by TPM
Scoring to be provided by TPM
Analysis of Prior Defect History and Defects from the Field
Perforce Change List analysis, TreeMap and Structure 101
Cyclomatic Complexity and Unit Test Code Coverage Statistics
Customer Risk
Quality Risk
Features / Risks Addressed Reference ID IDP and SP Adapter 3 2 0% 200% 3 3 5 2 2 39 51 3978Reference ID Attribute Serialization using JSON 3 6 10% 540% 1 3 1 5 2 19 61 6258.6Reference ID Implement Client Cert Auth 4 3 0% 300% 1 1 1 2 1 13 28 1092Reference ID Modify SP Adapter Advanced Configuration 3 3 0% 300% 1 1 1 2 1 13 28 1092Reference ID Pass authentication policy to the Idp application via query/post rather than REF 3 3 0% 300% 3 3 3 3 3 39 54 6318Reference ID Sample Code 3 0Support Sharepoint 2010 3 3 0% 300% 5 5 1 2 1 65 28 5460Sharepoint 2010 Interop 3 0Core Provisioning Logging to DB 3 1 0% 100% 3 1 1 1 1 33 18 594Configurable XML Signature Algorithm 3 1 0% 100% 3 1 1 1 1 33 18 594Google Package Google OpenID Idp Integration Kit 3 7 0% 700% 5 5 4 4 1 65 63 28665SFDC Customer Portal Connector 3 3 0% 300% 1 1 1 2 1 13 28 1092SFDC Partner Portal Connector 3 3 0% 300% 1 1 1 1 1 13 18 702SFDC CRM Connector 3 3 0% 300% 1 1 1 2 1 13 28 1092SFDC IdP Adapter 3 3 0% 300% 3 3 1 3 1 39 38 4446SFDC Support CRM SAML 3 4 0% 400% 1 1 1 2 1 13 28 1456SFDC Verify Idp Adapter with Protals 3 2 0% 200% 1 1 1 2 1 13 28 728Provisioning SDK 4 5 0% 500% 2 2 3 2 2 26 41 5330Social Networking Adapter 3 1 0% 100% 3 3 1 1 1 39 18 702
Risk Mitigation Modeling (Cont.)
Code Change TreeMaps
Understanding and Improving Team Velocity
Key Drivers for AnalysisEnterprise Software Development != SaaS DevelopmentExecuting more FrequentlyTeams are DifferentMinimum Quality Criteria
Data Mining and Modeling Addressing ConcernsEngineering DashboardComplexity Point Modeling
Engineering Dashboard
Complexity Point Modeling
Complexity Point Modeling
What’s Next… Systems Engineering @ Ping
Analysis of Adoption of Product OfferingsFrequencyDevelopment to First Adoption Temporal MapUnderstanding Cost/BenefitModeling ROIDefining Level of Effort on Incremental Work
Test Asset ManagementConsolidation and Reporting of all Automated TestsBringing All Tests under Single Management UmbrellaMonitoring Test Effectiveness and Coverage
Wrap it Up…
