Upload
darleen-ray
View
217
Download
0
Embed Size (px)
Citation preview
BSPIN Agile Practice Benchmarking Report 2012
Highlights
BSPIN Agile Practices Benchmarking Report October 2012
BSPIN Agile Practice Benchmarking Report 2012
Introduction• Agile Practice Benchmarking:
BSPIN conducted a study during 2011-2012 to benchmark the Agile practices to provide insights on the effective ways of adopting Agile in small to large organizations. 12 Organizations presented their Agile experience across 3 Panel discussions Agile Practice benchmarking survey was conducted in July 2012
• Designing of Questionnaire, Consolidation, Analysis of feedback was done in coordination with Agile coach and Senior Consultant from Improvalue Consulting, Bangalore (www.improvalue.com).
• Objectives :– Explore new areas like:
• Domains where Agile has been adopted. Agile Practices in use.• Benefits and Barriers encountered by organizations on their Agile journey• Metrics Reported (Project and Product) • Coexistence/Compatibility with other standards (CMMi) and practices (Lean)• Best Practices
– Focus is on IMPLEMENTATION aspects of AGILE
• Outcome:– Learn from industry experience– Enable informed decisions
2
BSPIN Agile Practice Benchmarking Report 2012
Agenda• Demographics - Agile Spread• Usage of Agile Practices• Estimation Techniques• Automation, Tools• Agile Metrics• Agile – Coexistence with other Methodologies/Models• Benefits and Barriers• Challenges Faced, Best Practices• Conclusions/Recommendations
3
BSPIN Agile Practice Benchmarking Report 2012
4
• 94 responses from 75 Organizations • More participation from Small and Medium Organizations
(compared to other surveys)• Product and Service Organizations covered equally• 98% of Large Organizations and 77% of Small an d Medium
Organizations have external quality certifications• More than 50% of the respondents have been using Agile for
more than 2 years• Panel discussions – 12 Organizations shared their Agile
Experience ( 6 Large and 6 Small-Medium)
Demographics
BSPIN Agile Practice Benchmarking Report 2012
How Agile adoption is supported?
Training Methods used:
Support to Agile
5
• This section was answered by 72% of the organizations
• As the organization gains more experience on Agile, Agile becomes a part of QMS
• Only 6% of the organization indicated “Self Management by teams” with no other support.
• This was answered by 90% of respondents
• About 40% of the organization had more than one method of training
• About 20% of the organization indicated only “In-house” with no other support.
BSPIN Agile Practice Benchmarking Report 2012
Agile Spread
6
• SCRUM is the mostly adopted methodology
• Increase in usage of KANBAN - – Many Organizations evolved from
manufacturing background where LEAN is adopted
• Agile experience of the organization has no major effect on the methodology chosen
• Nearly 70% of the organizations are using 1 or 2 methods
• Of the 7% using more than 4 methodologies, 3% are small organizations
BSPIN Agile Practice Benchmarking Report 2012
Agile Spread
ABREVIATIONS(1) BFS – Banking and Financial Services (2) IA – Industrial Automation (3) A,A,A – Avionics, Aviation, Aerospace (4) SI – Systems Integration
7
• Agile has been used in almost all domains including safety critical domains like Pharma, Avionics and other Embedded Systems.
• Telecom domain which is mostly hardware dependent has reported highest percentage of Agile implementation.
• More than 75% of the respondents have reported the implementation of ‘Distributed Agile’
• Agile has been used in most of the project types – including Testing and Systems integration
Others Domains Reported:•OS, Middleware, Kernel•Platform development •Control Systems
• Product Integration• Media and Publication, Education• Travel, Energy and Storage
BSPIN Agile Practice Benchmarking Report 2012
Usage of Agile Practices
8
• Core Agile (Scrum) Practices have been well adopted.• Engineering Practices usage (Refactoring, Test Driven Development) needs to be strengthened.
82% have release cycles <= 4 weeks
• 48% of the respondents use 2 to 3 weeks release cycles.
BSPIN Agile Practice Benchmarking Report 2012
Agile Practices Vs Agile Experience
9
• Core Agile practices are embraced right from the beginning
• There is a significant increase in the usage of Open Work Areas (compared to other surveys)
• Some of Scrum practices take more time to stabilize
• Organizations can have increased rigor early in the Agile adoption for better benefits.
• The Engineering practices strengthen over time
• Refactoring and TDD need more focus for effective Agile adoption
BSPIN Agile Practice Benchmarking Report 2012
Agile Estimation Techniques
10
• About 53% of the respondents use "User Stories" estimation at proposal/bid stage
• About 80% of the respondents are using one or more of the following estimation techniques: i. User Stories estimationii. WBSiii. Guesstimation
• Organizations are taking more than 2(two) years to move away from Guesstimation to WBS which needs attention.
Usage of traditional estimation techniques at proposal/bid stage and User Story Estimation/ WBS at iteration stage seems to be in vogue
BSPIN Agile Practice Benchmarking Report 2012
Extent of Automation
11
• Only 76% of the respondents answered this section
• Automation of unit testing is low, inline with low adoption of TDD
• Automation of code reviews is low, as evident from low adoption rate of Refactoring
• Automation done in later stages of testing and quality compromised- missing out the essence of Agile
• The automation level increases with Agile experience of the organization
• The size of the organization had no impact on the level of automation
Organizations have shared multiple Tools : For Agile Project Management and Engineering.
BSPIN Agile Practice Benchmarking Report 2012
Tools Used
12
Requirements
• Caliber RM
• DOORS
• Greenhopper
• JIRA
• RALLY
• Rational Requisite Pro
• Red Mine
• ScrumWorks
• Sharepoint
• TFS
• VersionOne
• Caliber RM
• DOORS
• Greenhopper
• JIRA
• RALLY
• Rational Requisite Pro
• Red Mine
• ScrumWorks
• Sharepoint
• TFS
• VersionOne
Code Reviews
• CheckStyle
• Eclipse plugins
• FxCop
• IBM AppScan
• Rational Software Analyzer
• SONAR
• StyleCop
• TFS
• Toad
• xTest
• xUnit
• CheckStyle
• Eclipse plugins
• FxCop
• IBM AppScan
• Rational Software Analyzer
• SONAR
• StyleCop
• TFS
• Toad
• xTest
• xUnit
Testing
• HP Quality Center
• HP Quick Test Professional
• JIRA
• LoadRunner
• MS VS
• Selenium
• Silk
• VSTS
• Winrunner
• xTest
• xUnit
• HP Quality Center
• HP Quick Test Professional
• JIRA
• LoadRunner
• MS VS
• Selenium
• Silk
• VSTS
• Winrunner
• xTest
• xUnit
Defect Tracking
• Bugzilla
• TeamForge
• Clearquest
• HP Quality Center
• JIRA
• Rational ClearQuest
• TFS
• Bugzilla
• TeamForge
• Clearquest
• HP Quality Center
• JIRA
• Rational ClearQuest
• TFS
Project Management
• CollabNet
• Green Hopper
• HP-Project and Portfolio Management
• JIRA
• Mingle
• Rally
• Scrum Works
• TFS
• Version One
• XPlanner
• CollabNet
• Green Hopper
• HP-Project and Portfolio Management
• JIRA
• Mingle
• Rally
• Scrum Works
• TFS
• Version One
• XPlanner
Not a complete list. Widely used tools are reported here.
BSPIN Agile Practice Benchmarking Report 2012
Traditional Practices with Agile
13
• 56% of the respondents reported:– Upfront collection and finalization of
requirements – Upfront architecture
• This practice was predominantly observed in Product companies when compared to Service Providers
BSPIN Agile Practice Benchmarking Report 2012
Management Metrics
14
Other Metrics Reported:
Schedule: Schedule Variance, Sprint Stretch Factor
Effort: Effort Variance,
Scope: Commitment Accuracy, (De)Scoping Index,Requirements Stability, Scope burn down
Quality: Product Metrics, Review Efficiency, Defect Turnaround Time, Defect Resolution Productivity, Defect Removal effectiveness
Testing: Test Yield, Test case pass percentage
• 90% of the respondents have answered this question
• The usage of metrics increases with Agile experience to measure effectiveness
Recommend BSPIN to conduct ‘Agile data Benchmarking’ to obtain industry benchmark
BSPIN Agile Practice Benchmarking Report 2012
Engineering Metrics
15
• Monitoring Product Metrics is fairly good except for the Code related activities
• Measuring Code complexity is key for refactoring – needs more focus
• Measuring Code coverage is key for quality – needs more attention
• There seems to be a steady improvement as the organization gains more experience on Agile
• However, Code Complexity and Code Coverage metrics needs more focus
BSPIN Agile Practice Benchmarking Report 2012
Agile - CMMi Coexitence
16
• This question was answered by 72% of the respondents
• 68% of the respondents think that Agile and CMMi can coexist. Of these 60% are from organizations that have ISO or CMMi certification.
• 72% of the respondents reported that their Agile projects were also being audited (as per the corresponding certification process)
What does the Survey Report?
Reasons cited for “No” and “not Sure” :i.Different focus (People centric Vs Process centric)ii.Agile is project management, engineering framework and CMMi is a process improvement frameworkiii.It is compatible till L3 except IPM, OPD
Reasons cited for “Yes”::i.Agile does not throw away the processes. It is a mature way of looking how efficient the processii.CMMi prescriptions can be applied to any methodology.
BSPIN Agile Practice Benchmarking Report 2012
CMMi on Agile
17
What does CMMi v1.3 say?
To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PP, PPQA, RD, REQM, RSKM, TS, and VER.. . .Even if you are not using an Agile approach, you might still find value in these notes.. . .Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008].
To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PP, PPQA, RD, REQM, RSKM, TS, and VER.. . .Even if you are not using an Agile approach, you might still find value in these notes.. . .Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008].
Possible reason for higher maturity organizations to agree that Agile and CMMi are
compatiblePractitioner Groups of CMMi and Agile can integrate the practices
to adopt the best of both
Reference: Section-5: Interpreting CMMI When Using Agile Approaches – CMMI-DEV, V1.3 by SEI
BSPIN Agile Practice Benchmarking Report 2012
Coexistence with other models Integrating LEAN with Agile
• 33% of the respondents reported using LEAN techniques with Agile
• Most organizations (26%) have adopted KANBAN
Integrating Agile with other Life Cycle Models
• 46% of the respondents reported using Agile best practices in other LCMs
• Agile practices predominantly used:
18
BSPIN Agile Practice Benchmarking Report 2012
Benefits of Agile adoption
19
Key Benefits:
•Better management of Requirement and Priority changes, Risks
•Improved time-to-market and customer satisfaction
BSPIN Agile Practice Benchmarking Report 2012
Benefits Perceived – Role wise
20
• Benefits perceived by organizations increase with Agile experience, inline with practices adopted and metrics collected/analyzed
• Ability to handle risks and improved customer satisfaction were rated high by Agile team and Agile Coach
• Managing changing priorities and better project visibility were rated high by senior management and process professionals
BSPIN Agile Practice Benchmarking Report 2012
Barriers to Agile adoption
21
Agile Coaches/Consultants and Senior Management to focus on methodical transition to AGILE
These observations are in line with the world wide survey reports by other organizations
BSPIN Agile Practice Benchmarking Report 2012
Barriers Perceived – Role wise
22
• Barriers reported by Senior Management and Process Professional are much higher than Agile Coach/Consultant and Agile Team
• The size and Agile experience of the organization had no significant impact on the barriers
0%
10%
20%
30%
40%
50%
60%Project technical complexity / size
Lack of focus among PM / PMO
Lack of onsite - offshorecollaboration
Agile cannot handle Maintenance
Lack of Skilled Resources
Resistance to change
Agile produces Insufficientdocumentation
Lack of tools to support shortiterations
Management not interested
Budget constraints
Senior Management Process Professional Agile Coach Agile Team
BSPIN Agile Practice Benchmarking Report 2012
Benefits and Barriers - Impact
23
• Time-to-market• IT and Business Collaboration • Handle risks earlier/ better
• Time-to-market• IT and Business Collaboration • Handle risks earlier/ better
• Manage changing requirements• Better Team-morale• Better collaboration
• Manage changing requirements• Better Team-morale• Better collaboration
• Manage changing priorities• Better project visibility• Customer/End-user satisfaction
• Manage changing priorities• Better project visibility• Customer/End-user satisfaction
• Project technical complexity / size• Lack of onsite - offshore collaboration• Lack of tools/ Automation for short iterations
• Project technical complexity / size• Lack of onsite - offshore collaboration• Lack of tools/ Automation for short iterations
• Lack of coordinated transition to agile• Lack of focus among PM/PMO Groups• Resistance to change
• Lack of coordinated transition to agile• Lack of focus among PM/PMO Groups• Resistance to change
• Agile cannot handle Maintenance• Lack of skilled resources• Agile produces Insufficient documentation
• Agile cannot handle Maintenance• Lack of skilled resources• Agile produces Insufficient documentation
Team
ManagementCustomer
BSPIN Agile Practice Benchmarking Report 2012
• Can think of Hybrid model based on the scope clarity.
• Redefine/Customize the traditional metrics.
• Handholding by Agile coach. Mandatory Agile skill training.
• Stakeholders to come together and decide.
• Policies need to be flexible to have Team benefits
• Address the dependency issues in Daily stand-up meetings. Effective Scrum of Scrums
• Senior management to focus on infrastructure needs.
• Teams to look into non-functional aspects during planning meetings. Detail planning of testing effort for manual testing
• Benchmark within the organization . Participate in external Forums.
• Breakdown stories for ease of development. Help customers articulate User Stories and Product Backlog.
• Usage of Tools
• Enhance People Agile skills; especially on Engineering side.
• Fixed Price Model is a challenge, T&M have worked well.
• Collection of Traditional Metrics in Agile Projects.
• Moving away from Command & Control; Self Organizing Teams
• Scope creep and sign-offs.
• Compensation / Appraisal related issues due to no individual Heroes
• Communication/ Collaboration challenges between the distributed sub-teams.
• Infrastructure needs for Continuous integration. Teams Not exposed to usage of tools and automation.
• Frequent release cycles in sprints thus not enough time for testing. Very less focus on Non functional Test. Some legacy application testing cannot be fully automated.
• No predefined performance benchmarks.
• User story articulation by different stakeholders.
• Focus on Maintainability and code quality.
• Not all teams will be empowered or trained for ‘Test Driven Development’. Need External Support.
Challenges ExpressedChallenges Faced Possible Solution
24
BSPIN Agile Practice Benchmarking Report 2012
• Record the retrospection and use some Agile tools for tracking them.
• Stake holder involvement in defining test scenarios.
• More encouragement from management to adopt Test Driven and Behaviour Driven Development.
• Invest on tools and infrastructure to run automated regression suite on every check in
• Representative at onsite for each offshore team: Overlap of the key people from offshore with onsite team.
• Slightly heavy onsite team for better coordination.
• Good build plan. Report invocation from Build.
• Teams based on functionality rather than skills
• Buddy Testing from developer machine to save the deployment time.
• All the team members who are part of particular sprint should be part of the scope and estimation. • Measurement of code complexity for quantification.
• Use of Agile training from time to time, participation in Agile conferences and Forums.
Best Practices Reported
25
BSPIN Agile Practice Benchmarking Report 2012
Conclusions and Recommendations
26
• Agile as a methodology is well established - more than 50% of respondents have used it for past 2 years and included it in QMS. Scrum is the most widely used methodology followed by Kanban
• Agile usage is spread across multiple domains and project types; Over 75% of the respondents have confirmed using Distributed Agile
• While Management practices (predominantly from SCRUM) have been well adopted, the adoption of engineering practices (predominantly from XP) needs more rigor
• Usage of metrics and automation level increased with Agile experience• Coexistence of Agile with other industry standards (like ISO and CMMi) has been agreed upon
NEXT STEPS• Data Benchmarking of Agile projects• Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both
Bibliography•CMMi for Development, Version 1.3 by SEI•“2011 State of Agile Development Survey Results” by VersionOne•“Agile adoption in India Survey Report – 2011” by ThoughtWorks•“Agile India 2012 Survey Results” by Valtech
BSPIN Agile Practice Benchmarking Report 2012
Discussions
THANK YOU
27