The leading pioneer in GPS technology
Copyright © 2008 NavCom Technology, Inc.
Prioritizing Process Improvement and Prioritizing Process Improvement and Ignoring the RatingIgnoring the Rating
Bruce Pedersen
12 June 2008
SEPG Europe 2008
Copyright © 2008 NavCom Technology, Inc.2
Case StudyCase Study
This is a case study of a small high technology software company that decided to strategically apply the benefits of CMMI process improvement to meet immediate business goals and ignore the rating. It will be shown that for this company, applying just 3 of the Level 2 process areas (PPQA, M&A, CM) struck the right balance in achieving cultural buy-in, allowing efficient utilization of available resources, all while reaching a significant return on investment.
Copyright © 2008 NavCom Technology, Inc.3
TopicsTopics
Background What is the problem? Management Speaks Determining the Best Approach Implementation Strategy Did all of this Work Make a Difference? Next Steps
Copyright © 2008 NavCom Technology, Inc.4
BackgroundBackground
NavCom Technologies Inc. is a John Deere company that supplies the enterprise with high precision Global Positioning System (GPS) navigation solutions for integration into the farm tractor auto-track system
Software releases are transmitted to another John Deere company, Intelligent Vehicle Systems (IVS), who performs system integration of the GPS component with other components on the farm tractor
IVS also operates the customer service center and manages complaints from the farmers about tractor operation
IVS attempts to separate out and communicate the GPS related farmer “bug reports” to NavCom as well as “bug reports” generated during system test
Increasingly high number of software defects are being reported back to IVS from the farmer
The software release quality issue has become visible to farming customers around the world and action has to be taken to correct the problem
NavCom is directed by John Deere corporate to “fix” the software release quality problem
Copyright © 2008 NavCom Technology, Inc.5
Who is NavCom Technology?Who is NavCom Technology?
125 Employees located at the Torrance, California facility
35 software engineers supporting six concurrent GPS Software Development projects
Average GPS Product development cycle 6 months to 2 years
Process improvement has been on-going for less than two years
Established in 1992, NavCom Technology, Inc, a John Deere Company, is a leading provider of advanced GPS receivers
NavCom provides John Deere with a centimeter level GPS position solution that is integrated into the farming tractor Autotrack system (40,000 units in the field)
Autotrack combined with terrain compensation provides straight line operations on hillsides or rough terrain “hands free”
For more information visit: http://navcomtech.com/
Copyright © 2008 NavCom Technology, Inc.6
GPS at Work for John DeereGPS at Work for John Deere
GPS
StarFireCommunications
Satellite
DGPSCorrections
Galileo
WAAS
EGNOSGlonass
MSAS
Copyright © 2008 NavCom Technology, Inc.7
Software Bundle Releases Software Bundle Releases
Software releases are tied to farming seasons
Two bundle releases (summer/winter) are made per year
Major activity by both NavCom and IVS in pre-bundle release testing (see right)
NavCom performs an accelerated software development life cycle
After bundle release, reported software bugs from both IVS and farmers fixed and saved for the next bundle release
Software Engineering
Test
Product Verification & Validation
Bug Reports from both IVS and Farmers
Analysis and Implement Bug
Fixes
NavCom IVS
IVSNavCom
Feature requests by the farmer are also saved for a bundle releases
Show stopper bugs found after a bundle release are fixed and bundle updates made
Copyright © 2008 NavCom Technology, Inc.8
What’s the Problem?What’s the Problem?
Software Defect Visualized
NavCom- Reactive software development culture (i.e. firefighting)
Bug fixes and schedule agreed to over the phone without requirement traceability
Feature requests taken over the phone for next release- Lack of engineering test resources (personnel and tools)
available to apply the necessary pre-release software engineering test rigor
IVS - System Integrator- Lack of GPS expertise to accurately report bug details
necessary for bug environment recreation at NavCom - Lack of system test capabilities to isolate bugs caused by GPS
vs. other integrated tractor component sources General
- Poor communication path between NavCom and the IVS (very little paper trail)
Priority differences based on different business models Trust
Copyright © 2008 NavCom Technology, Inc.9
Management SpeaksManagement Speaks
Management defines business goal related to software releases- “Significantly reduce the total number of software defects
released to the customer in delivered software” Choose a methodology that will minimize
- Cost to Implement- Business Disruption- Cultural Impact- Time for improvement visibility
Noticeable reduction in customer reported defects Make recommendations for what’s needed to apply additional
software engineering test rigor- Internal infrastructure changes- Additional test tools
There is no corporate or customer pressure for:- A particular methodology- Schedule target for a particular methodology level or rating
Copyright © 2008 NavCom Technology, Inc.10
Determining the Best ApproachDetermining the Best Approach
What methodology?- What improvement scope? - What representation?
What implementation strategy?- What Organizational infrastructure changes are needed
to support the initiative?- What software engineering infrastructure changes?- What additional test tools are needed?
Copyright © 2008 NavCom Technology, Inc.11
What Methodology?What Methodology?
The CMMI model selected for implementation- CMMI used by other John Deere divisions- World wide software industry acceptance of the CMMI
model and its best practices- CMMI model flexibility
Tailorable for unique business casesSelectable representation staged vs. continuous to
better fit company objectives- The “I” in CMMI allows phased integration of Systems
and Hardware into the process improvement scope- Management directives can be satisfied using CMMI
Copyright © 2008 NavCom Technology, Inc.12
What Improvement Scope?What Improvement Scope?
Software only with future integration of systems and hardware engineering
Product development software projects in scope of CMMI process improvement- Research and development software projects out of
scope Test group in scope Hardware only projects out of scope Maintenance software projects to use portions of
the CMMI model that makes sense (e.g. peer reviews, test)
Copyright © 2008 NavCom Technology, Inc.13
What Representation?What Representation?
Representation selected should allow process improvement focus to be directly tied to the company business goal- “Significantly reduce the total number of software
defects released to the customer in delivered software” Representation selected should also take into
consideration Management directives- Cost to Implement- Business Disruption- Cultural Impact- Time for improvement visibility (noticeable reduction in
customer reported defects)
Copyright © 2008 NavCom Technology, Inc.14
Representation ComparisonRepresentation Comparison
Provides freedom to select the order of improvement that best meets the organization’s business objectives
Increased visibility into the capability achieved within each individual process area
Quick wins increase buy-in Comparison across the
divisions can only be made on a process area by process area bases (no maturity rating)
A relatively new approach which has limited data to demonstrate its ROI
Provides a predefined sequence of improvement areas based on proven path of successive levels each serving as a foundation for the next
Provides visibility at the maturity with limited visibility at the process area level
Permits easy comparison across divisions due to single maturity rating
Builds on a relatively long history of use that builds on case studies and data that demonstrates proven ROI
Continuous Representation Staged Representation
Copyright © 2008 NavCom Technology, Inc.15
Representation SelectionRepresentation Selection
The continuous representation chosen- Satisfies Management directives- No customer pressure for CMMI capability
level rating- Organic process improvement growth works
for NavCom
Mapping of process areas to business goals imperative- Process Areas chosen that focuses on the immediate business
goals- 5 year plan created to phase in future process areas to support
projected business goals- Plan updated every year to reflect changing business strategy
CMMIBusiness
Goals
Copyright © 2008 NavCom Technology, Inc.16
Implementation StrategyImplementation Strategy
Process Area Selection Establish a Capability Level Profile Define the current project processes Establish an organizational infrastructure that will
support continuous process improvement- Involve practitioners early on to foster process ownership
Develop plans and processes that will enable selected process area support of the current project processes
Pilot processes and obtain practitioner feedback for constant improvement
Measure improvement progress using metrics
Copyright © 2008 NavCom Technology, Inc.17
Process Area SelectionProcess Area Selection
Select Process Areas based on:- Direct contribution to business goal- Provide foundation for future process area integration- Consider dependencies between process area’s and generic practices - Ease of implementation - complexity- Amount of support processes in place already at NavCom
Choose the desired capability level for each of the selected Process Areas (1-5)- The continuous model allows additional capability levels to be achieved
on focused process areas
Construct a capability level profile- Demonstrate process improvement progress to management- Guide process improvement activities
Process improvement can be conducted on individual PA’s at different rates Resources can be focused on lagging PA activities
Copyright © 2008 NavCom Technology, Inc.18
Process Area Selection (cont)Process Area Selection (cont)
The process areas that meet the selection criteria are a subset of the categorized “Basic Support” process areas:- Process and Product Quality Assurance (PPQA)- Configuration Management (CM)- Measurement & Analysis (MA)
Peer Reviews and Test were also selected from the Verification process area (Specific Goals 2 & 3) to directly support “reducing the number of defects in software releases”
Copyright © 2008 NavCom Technology, Inc.19
Process Area MappingProcess Area Mapping
Minimize MA PPQA CM
Cost to Implement
Med Med Low
Business Disruption
Med Low Low
Cultural Impact Med Low Low
Time for ROI Visibility
Low Med Med
PP PMC SAM MA PPQA CM
Management Products
Code Product
Practitioners
Project Managers
Level 2 PAs
Management Requirements
VERPeer Reviews and partial implementation of testing
REQM
Copyright © 2008 NavCom Technology, Inc.20
Process Area DescriptionProcess Area Description
PPQA – provide staff and management with objective insight into processes and associated work products via independent auditing
CM – establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits
MA –develop and sustain a measurement capability that is used to support management information needs
Peer Reviews and Test – ensure that selected work products meet their specified requirements (VER process area)
Note: Test refers to system test prior to Product Verification and Validation (PV&V) testing
Copyright © 2008 NavCom Technology, Inc.21
Establish a Capability Level Profile Establish a Capability Level Profile
Determine what capability level to target
for each selected process area
Establish a method for overlaying the
current achievement profile to the target- Determination of current % complete report to
Management- Used for appraisal readiness check
Use target staging to establish a path of process improvement to be followed by the organization
Capability Level Profile
0
1
2
3
4
5
PPQA CM MA
Cap
abil
ity
Lev
el
Copyright © 2008 NavCom Technology, Inc.22
Capability Level ProfileCapability Level Profile
0
1
2
3
4
5
Cap
abili
ty L
evel
PPQA CM MA
Target Profile
AchievementLevel
AchievementLevel
AchievementLevel
Target Profile
Target Profile
GG3 Institutionalize a Defined Process
Copyright © 2008 NavCom Technology, Inc.23
Define the Current Project ProcessDefine the Current Project Process
The Current Project Process defines the way a project develops software today
Current Project Process is unique to each project (Level II)
Provides awareness to project practitioners of gapsin current unmanaged project software process- Written processes not consistently followed
(e.g. skipped in schedule crunch)- Unwritten processes verbally communicated by peers- Non-Existent processes made-up by individuals to do their job
Starting point for application of selected CMMI Process Area support
Opens up communication path between the process group and project practitioners- Flow chart used to communicate current process at the 50,000 foot level- Allows further drill down in focused process areas using non-CMMI
jargon or concepts
Copyright © 2008 NavCom Technology, Inc.24
PA Support of Current Project ProcessPA Support of Current Project Process
Configuration items and change requests
Current Project ProcessInformation Needs
Measurements and analysis
Baselines and audit reports
Quality and noncompliance issues
Standards and procedures
Defect Reports
Peer Reviews & Testing
PR + Test PPQA
MACM
L2
L2
L2
L3
Fosters practitioner process ownership in helping the process group define needed processes
Identifies areas were application of process improvement could be focused for near term ROI
Allows groups supporting the project to have a voice
Can be used for training
new project personnel
Copyright © 2008 NavCom Technology, Inc.25
Establishing an Organizational Process Establishing an Organizational Process Improvement InfrastructureImprovement Infrastructure
SPI Sponsor selected (Director of Systems) Software Process Improvement group formed
- SPI Manager- PPQA Lead Auditor- Process Engineer - Administrator (training records, PAL administrator, etc.)
SEPG formed with representation from each project and functional group (7 members)- SPI Manager Chair- Program Management- Software Development- Test
Technical Working Groups managed by SEPG to accomplish work 3 member reserve audit team established to support the PPQA
Lead- Made up of project practitioners- Reserve auditor training performed by PPQA Lead- Auditor performs cross project audits only (not own project)
Copyright © 2008 NavCom Technology, Inc.26
Focused Efforts at Cultural Buy-InFocused Efforts at Cultural Buy-In
Early participation by practitioners in defining processes foster process ownership
Cross functional representation in the SEPG
- Membership rotation scheme introduced to expose a greater % of the practitioners
to process improvement activities Tool deployed to facilitate practitioners process
improvement suggestions Monthly published process improvement
newsletter that features SPI topics and progress SCAMPI B and C appraisals conducted every 6 months
- Focus process improvement efforts towards a near term goal- Align culture to the appraisal process
Periodic Workshops held with individual project to discuss new process deployment and lessons learned
Training program initiated to teach new processes to project personnel
Copyright © 2008 NavCom Technology, Inc.27
Process Creation and DeploymentProcess Creation and Deployment
The SPI Group creates plans and processes for direct support of projects defined process- Project practitioners participate in defining the current project
process- Practitioners participate in content review of project plan/process
documents The SPI Group creates workshops to familiarize project
practitioners on new processes New processes are piloted first and practitioner feedback
used for process improvement updates Metrics are defined to measure process effectiveness Lessons learned are managed and used for process
updates
Copyright © 2008 NavCom Technology, Inc.28
What Else Do We NeedWhat Else Do We Need
Test – additional rigor focused on finding defects before release to the customer- Formalize testing using
historical test scenarios- Dedicated group formed to
perform software engineering test function
Tools –leverages process improvement advantage- Artifact storage- Automated CM- Bug tracking- Central repository for
process improvement documents (PAL)
Copyright © 2008 NavCom Technology, Inc.29
Establishing a Test OrganizationEstablishing a Test Organization
A Software Engineering Test Group (SETG) was formed with a charter to formalize the software pre-release engineering test process of the organization- SETG Lead- Three Test Engineers
In parallel, two tasks were conducted to support the SETG- A PC based GUI tool was developed to automate regression
testing against baseline and new releases GUI allows test engineer to select and control the battery of tests to
be run Automatic report generation and plotting used for analyzing test
results- Regression library created to furnish regression test scenarios
that have been created to solve problems during the past seven years
Scenarios generated to recreate conditions that would reproduce reported bugs
Copyright © 2008 NavCom Technology, Inc.30
What Test Tools are Needed?What Test Tools are Needed?
CM tool (Microsoft Team Foundation Server) selected to help software development teams communicate and collaborate more effectively- Streamlines the implementation of the CM process area across
the organization- Enforces CM on both source code and documents via predefined
rules- Provides a central repository for process documents, templates
(Process Assets Library) and artifacts supporting the appraisal- Integrated CMMI methodology component for “head start” in
process definition and implementation - Integrated work item tracking and reporting (e.g. bug tracking)- Extensive suite of “out of the box” reports for tracking progress
and metrics mining (e.g. defect phase containment, density)
Copyright © 2008 NavCom Technology, Inc.31
Did All of This Work Make a Difference?Did All of This Work Make a Difference?
Copyright © 2008 NavCom Technology, Inc.32
Total Software Reported Defects by Bundle
0
10
20
30
40
50
60
70
80
90
100
SSB05 WSB06 SSB06 WSB07 SSB07 WSB08
Def
ects
CustomerReportedTotal Defects
10.3%
17.7%
9.3% 8.1%
7.1%
1.6%
NavComMajor GPS Technology
Release
SSB = Summer SW Bundle, WSB = Winter SW Bundle
NavCom’s SETG On-Line
IVS Test Group
Revamped
Copyright © 2008 NavCom Technology, Inc.33
Customer Defect Metric ExplanationCustomer Defect Metric Explanation
The total reported defects “green column” are what the system integrator (IVS) reported back to NavCom - Customer reported defects are included in the green column
More feature requests and bug fixes are included in the winter bundles thus generating more defects
With each bundle release comes increased complexity from newly implemented GPS features and capabilities
A major technology upgrade to increase both GPS accuracy and reliability was made in the Winter Software Bundle 07
Copyright © 2008 NavCom Technology, Inc.34
Bottom LineBottom Line
The drop in customer reported bugs between the Winter and Summer bundle 06 is due to the activation of the SETG and early adoption of peer reviews- SETG able to detect complex navigation related bugs via simulation
The dramatic increase of defect detection by IVS in SSB06 was due to the addition of testing resources and training by NavCom on effective GPS testing techniques
Although NavCom’s defect metric database is still immature, indirect evidence indicates that NavCom’s cultural change has started and defect detection and correction is getting better
NavCom’s cultural change is rippling over to IVS causing them to also get better at detecting defects and reporting back accurate test results
The farmer is benefiting from all of this by reducing the number of defects seen in the field from 17.7% to 1.6% in two years
Shown is a downward trend in total software defects released to the customer
Copyright © 2008 NavCom Technology, Inc.35
Next StepsNext Steps
Align the process improvement 5 year plan to the newly created 5 year strategic business plan
for NavCom
Based on lessons learned over the past two years,select the next group of process areas for future implementation- Requirements Management- Technical Solution- Project Planning
Continue refining the metrics database to give better visibility to defect detection effectiveness
Continue laying the infrastructure foundation to allow seamless
transition to level 3 process areas
Copyright © 2008 NavCom Technology, Inc.36ConfidentialCopyright 2007 - Deere & Company NavCom Technology, Inc.NavCom Technology, Inc.
Questions?
Contact Information: Bruce Pedersen [email protected]