Embed Size (px)
Citation preview
CONTINUOUS IMPROVEMENT
You Can’t Improve What You Can’t See
DevOps East 2018 Renaissance Seaworld, Orlando, FL
November 6, 2018
Brandon Carlson
Thanks for joining me at DevOps East 2018. I hope you enjoyed the talk as much as I enjoyed putting it together. The insights we gained while experimenting with the ideas in this document were astounding and we keep finding more and more uses for this type of thinking. This stuff has really changed the way I think about continuous improvement. My hope is to inspire you to look at continuous improvement differently and try some new ideas of your own. If these pages do happen to help and you’ve got some success stories of your own to share, please reach out to me at [email protected], I’d love to hear them.
Sincerely,
Brandon Carlson, Lean TECHniques, Inc.
2
CONTENTS 4 6 11 16 20Continuous
Improvement Today
Big Data To The Rescue?
Case #1: Slow Laptops
Case #2: Oops! I forgot!
Case #3: Where are the bugs?
Continuous improvement takes many forms, you’ve probably used them all.
While the traditional improvement activities are valuable, maybe there’s another way.
Learn how one team used the data exhaust from local developer builds to make the case for new laptops.
You’ve probably experienced a developer forgetting to change a file at deployment time. Here’s how you can help prevent it in the future.
Do you know where the bugs are hiding in your codebase? You can, the data is all there.
24 27 32 37 42Case #4:
Commitment!Case #5: Death
by TestsCase #6: Go
Faster!Case #7:
Regression Test Woes
Conclusion
Your teams are trying their best to provide accurate sprint commitments, but what can your past tell you about your future?
You’re investing in test automation, but are the test providing the feedback you need?
Have you ever asked your teams to go faster? What’s really getting in their way?
There never seems to be enough time for full regression testing. Can you be a bit smarter about how you use your tester’s time?
Summary and contact information for any questions regarding the content.
4CONTINUOUS IMPROVEMENT TODAY
While continuous improvement is a
trendy word in today’s work
environment, the reality is that it has
been around all along and takes on
many forms:
Root-Cause Analysis
Post-mortem
Retrospectives
Kaizen Events
4
You’ve been there, done that, when it comes to continuous improvement initiatives. In this section, we’ll discuss some pros and cons of these “classic” approaches to improvement and common anti-patterns when using them.
“What's in a name? That which we call a rose by any other name would smell as sweet.”
— William Shakespeare
For the purposes of this document, we will refer to all improvement processes, meetings, ceremonies, etc. within our organizations as “retrospectives” in order to convey the intent of reflecting on what has happened and
making changes to improve our chances o f success in the future.
Retrospectives: The Good, Bad, and Ugly
Without a doubt, any organization that has adopted a continuous improvement mindset
CONTINUOUS IMPROVEMENT TODAY
5will be better equipped to succeed long term than an organization that hasn’t. If you ’ve eve r a t t ended a we l l r un retrospective you know how great it feels to systematically and regularly improve your process. Things get easier, more robust, or even down right eliminated in the process (although this is more rare an occurrence than we’d like).
Early on things come easy for the teams, improvements are obvious and easily tackled. Items like “We seem to be taking too long to plan and the task breakdowns seem redundant, let’s try to eliminate task breakdowns for a sprint” are common early on and, while successful, get us started down the path of subjective measures. As time goes on, the retros become virtually ineffective as takeaways like “We should communicate more” become the standard. Using data and metrics can significantly improve your improvement initiatives.
“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” – Jim Barksdale, former CEO of Netscape Communications Corporation 5
Big Data
“Data is the new science. Big Data holds the answers.” Pat Gelsinger, CEO — VMware
Big Data
IT MAY BE THE NEW HOTNESS, BUT YOU DON’T HAVE TO DIVE HEAD FIRST INTO THE WORLD OF BIG DATA IN ORDER TO MAKE USE OF SOME OF ITS CONCEPTS. IN FACT, NEXT WE’LL DISCOVER A KEY CONCEPT IN BIG DATA THAT CAN PROVE EXTREMELY USEFUL TO US IN OUR QUEST FOR CONTINUOUS IMPROVEMENT.
Data Exhaust“Data exhaust refers to the data generated as trails or information byproducts resulting from all digital or online activities. These consist of storable choices, actions and preferences such as log files, cookies, temporary files and even information that is generated for every process or transaction done digitally. This data can be very revealing about an individual, so it is very valuable to researchers and especially to marketers and business entities.”
http://www.techopedia.com/definition/30319/data-exhaust
Data exhaust plays a key role in Big Data, and you can see the results in action over at Google, who used the data exhaust created by its users to develop one of the most advanced spell checkers in the world.
Google kept track of searches followed by searches with similar words spelled differently. If the first search yielded no subsequent clicks, but the second did, Google was able to determine that the first was probably a misspelling of the second, and added it as an alternative way to find the content being searched.
DATA EXHAUST
9Does data exhaust pertain only to online activities? Of course not, let’s look at another industry that has been using your data exhaust for a long time: Financial Services
Once upon a time, when you used your bank card, your bank didn’t think much of it. Those were the days before fraud was so prevalent, and one of the first anti-fraud techniques relied on your data exhaust. By using location i n f o r m a t i o n f r o m t h e merchants you v i s i ted mos t frequently, they were able to harvest your d a y t o d a y s p e n d i n g behavior and identify patterns, flagging transactions that don’t fit your profile. This is your data exhaust in a physical environment.
Finally, consider payments via wearable devices. With motion sensors and location aware technology, data such as your arm swing could be recorded and be used as additional biometric data for authentication since a thief is unlikely to have your same gait.
What does this notion of “Data Exhaust” have to do with our continuous improvement initiatives?
Every day we develop software, execute on projects, and interact with each other in ways that leave an exhaust trail behind. By harnessing the natural exhaust generated by your activities, you can bring greater visibility to your work. This increased visibility leads to questions that previously went unasked, accelerating learning and providing your team insights you may not have discovered previously.
Follow along in this document and see just how teams have taken advantage of this data exhaust in order to create improvements by capturing what was already happening on a daily basis and using it to inform decisions that weren’t previously visible.
The examples shown here are just the tip of the iceberg. There are many more uses waiting to be discovered. Use these ideas to spur your own thinking about where improvement opportunities are hiding and what you could uncover if you look in the right places.
DATA EXHAUST MEETS CONTINUOUS IMPROVEMENT
10
Slow Laptops?
Case #1
How we helped one group of teams leverage data exhaust to improve the performance of their laptops
TECHNOLOGY REFRESH
12machines. In doing so they could most likely justify the cost of replacing them. The challenge became just how to do that. After a bit of discussion they pointed out that, as a part of their continuous integration practices, each developer ran the build multiple times per day, and that build times could be used as a relative proxy for laptop performance.
The next question became how to track the builds. Could they rely on simply writing down the build times each time someone ran the build or was there another way?
You’ve probably been in a retrospective when the development team pointed out that “the machines are too slow.” This is a common complaint among developers, and the reason is simple. As your product grows, so do your build and test times. In this retrospective, the team decided to do something about it and they put together a proposal for new machines for the development team, taking it to the infrastructure team for approval only to hear “New machines for your team are not in budget for this year. They are on a 3-year upgrade cycle and we will upgrade them when it’s your team’s turn.”
What do you do in this situation? Escalate to management? Give up on new machines?
This team did something completely different. They turned to their data exhaust in order to build a stronger business case.
Proposal 2.0
The team got together after this failed first attempt and took a closer look at how they could more effectively make their case. They decided that what they really needed to do was show to the organization the true cost of the slow
TECHNOLOGY REFRESH
13The original idea was to write the data to a file on the individual developer’s desktop. These files could then be e-mailed to a team member for aggregation and analysis.
One developer had another idea, however, and that was to connect it to the central Jenkins server as a “parameterized job”, allowing access to it from a URL. This job simply took the values from the parameters and appended them to a CSV file every time the job was executed. Setup took about 15 minutes and the Windows batch file was updated to include a WGET to the Jenkins job, including the computer name as an identifier.
Capturing Data Exhaust
They decided that the build times were already being tracked by the build tool, but in such a way that it eventually scrolls off of the top of the screen, never to be seen again. All that was needed was a way to “screen scrape” the build times to get an accurate representation of the developers’ idle time waiting on builds without the need for them to try to manually track it.
“Screen Scraping” the build times was a bit of a chore, so the team decided on a much simpler approach: Wrapping the build script in a simple Windows batch file. The batch file was very basic, simply retrieving the current time at the beginning of the script, running the build, and capturing the time after the build ran. If every developer used this batch file when they ran the builds, the data necessary to make their case would be captured, but then what to do with it?
Centralizing The Data
Now that the team was running the build script that captured the intended information, they needed to figure out what to do with it.
TECHNOLOGY REFRESH
14than a simple memory upgrade. The team subsequently changed its strategy.
Success
The new strategy was to max out the memory of the development machines rather than the old strategy of wholesale replacement of the development machines. From a cost benefit perspective this was easier for the organization to swallow and was quickly acted on. The team had machines that were fast enough and the solution was easy on the organization’s pocketbook.
Once the updated build script was in place, the team was capturing build times for every local developer build and began the next step in their business case.
Analyzing the Build Times
After they had collected a sprint or two’s worth of information, they could return to the business case for new computers. Upon inspection of the data, however, a surprise jumped out at them. They noticed that some of the machines were significantly faster than others when running local builds. Their curiosity peaked, one of the team leads took the next step of collecting more information about each of the computers found in the data. They captured things like OS versions, CPUs, Disk Drives, and installed RAM. As it turned out, things like having a solid-state drive ended up having much less effect on build times than did other factors such as the amount of RAM installed on the machine. They found that simply upgrading the RAM in the machines would significantly increase the machines’ performance!
As you are aware, purchasing a new machine is significantly more expensive
TECHNOLOGY REFRESH
15Not only did this information feed directly into their business case for the technology refresh, it gave the team the insight hadn’t previously been aware of: That memory played more of a part of their laptop performance than any other factor. Armed with this additional information, they were able to come to the table with a more affordable solution to the team’s laptop woes that was quickly approved and implemented.
Outcomes
This team had a problem with slow machines, but were unable to easily articulate the cost of the problem to the organization. They decided that the build times were a good approximation of overall machine performance and found a way to harness the data exhaust that was generated each time a developer ran the build on their local machine.
Oops! I Forgot…
Case #3
Often things get overlooked in
the process of developing software. Here’s a data-driven
approach you can take to reduce
errors of omission.
17OOPS! I FORGOT
Has this ever happened in your organization? You come in late at night for the release and it crashes and burns? Everyone is working at a frenzied pace in order to resolve the issue, only to find that one of the developers overlooked changing a configuration file. This type of thing happens all too much when we rely so much on memory and documentation .
Preventing Omissions
Agile processes use frequent and open communication combined with practices such as standup and pair programming to generate and leverage the collective wisdom of the team to improve quality and reduce errors. Unfortunately, even with these practices, we make mistakes and omissions are a common occurrence.
How then does a team ensure that these errors and omissions are prevented? If you are like many organizations, you turn to one of several techniques for addressing the issue, several of which are listed below:
• Additional Processes
• Training
• Role-based Solutions
Each of these solutions, while valid, have some down sides. Take the case of adding additional processes as an example. All too often these processes include strict sign-offs and rules for how the work should be done. This limits individual autonomy (which often reduces team engagement) and slows down the flow of work
OOPS! I FORGOT
18through your development processes because of increased process overhead.
For another example, consider training. We often turn to some type of training when things go wrong with the hopes that we won’t see a repeat of the incident in the future. While this is a good option, a major challenge with training is that it isn’t timely, meaning you get the information long before you have a need for it. When the time comes to use it, it is often forgotten.
Finally, teams occasionally turn to Role-based solutions such as requiring a member of a more specialized group/role to make these sorts of changes. Unfortunately for this solution, it suffers from as many problems as the “additional process” solution described previously.
If people and process slow things down and remove individual autonomy, leading to less engaged employees and training isn’t timely enough, what can you do in this situation to reduce errors?
Before we get into one solution to this problem we need to take a detour into e-commerce, because it was the inspiration for this clever use of data exhaust.
Personalized Recommendations
If you’ve ever shopped online, it’s pretty likely that you’ve used Amazon.com. It’s also pretty likely that whenever you scroll down a bit on a product page you see some friendly little help from Amazon about what you may want to purchase in addition to your current item. This helpful widget shows you items that are typically purchased together and helps you, as
19buyer find additional items that you may need (of course it’s all in the name of helping you, the customer, out. They’re not just up-selling you).
As it turns out, Amazon is using the data exhaust generated by people adding items to their shopping cart to identify related items that you may also want, ensuring you don’t forget anything.
Turning to Version Control Once Again
Just like a shopping cart, your commits bundle related items together in terms of file changes. Mining this commit history can teach you all kinds of things about your codebase, including omitted files.
By writing a plugin for Git, you can analyze the code history and learn what changes are typically packaged together and, exploit that knowledge of the system just in time, when the developer is getting ready to commit some code without making the appropriate changes to other files.
OOPS! I FORGOT
Where are the bugs?
Case #2
How one team leveraged data exhaust to visualize buggy code in their system and improve quality
In many test automation initiatives, teams look to metrics such as code coverage to help them understand how well their code is tested as well as to ensure progress is being made towards their goal. On the surface, that seems like a decent metric, telling you what code has been tested and what code needs some automation around it. Unfortunately, this model treats all code as equal which, in theory, makes sense, but in practice breaks down on first contact with legacy code. For example, if a certain piece of code has been in place for two years and has never changed, does it need tests? What about code that is frequently changed? Should it require higher test coverage? How much time should be spent testing this code?
Shining A Light On Bugs
Bugs like to hide. Everyone on the team intuitively knows
where the bugs are, but they’re hard to see. One
team wanted to find a way to make them more visible in the code so that they could take action on it, getting it cleaned up to prevent more
DEFECT DENSITY HEATMAP
21from creeping in. How do you turn up the contrast on these bugs, making them stand out in the code? The team hypothesized that if they could identify what code changes were made during the process of fixing defects, then they would be able to highlight error-prone areas of the code in order to make it stand out more. This was a tricky problem to solve, because code changes look largely the same whether they were made to fix a bug or add new features. After some discussion, the team zeroed in on an alternative to examining the code: Looking at commit comments.
By looking through commit history for individual files, they decided they could look for key words like “Fixed” or “Fixing” within the commit messages that would give them a clue to the nature of the work being performed with that commit. While this would work, it’s a bit fuzzy and didn’t provide the precision the team was hoping for, they wanted something better.
Traceability To The Rescue
This organization did work with the federal government and, as a result, had a strong compliance program. The compliance program in place required that a ticket be
opened for all code changes performed and each commit message required a ticket number or the code would be rejected by the continuous integration (CI) environment. From a governance perspective, this enabled full traceability between the code and the requirements.
Fortunately, it was exactly this code traceability that gave the team the precision they were looking for in a solution. Every commit could be traced directly back to the Jira ticket that initiated the change in the first place.
Collecting The Data
Now that the team had the information it needed linking code changes to tickets,
they could get to work creating their visualization. They started by
writing a script that would parse o u t t h e c o m m i t l o g s ,
identifying each unique ticket and all of the files
that were changed for that ticket. After the process was run, each individual code file had a list of ticket numbers that had been associated
DEFECT DENSITY HEATMAP
22with that file. This became the first data point that the team was interested in: “Code Churn”. Code that was associated with higher numbers of tickets changed more often than code associated with fewer tickets.
Although that information was useful, it didn’t help improve the quality situation in any way. Another step was necessary to realize that goal and that was to take advantage of a little bit of additional information found in Jira.
To do this, they added some additional functionality to the script that made a web service call to Jira, requesting the details for each of the tickets associated with a given file. This provided them with additional information, namely the “classification” of the ticket: “Defect” or “Feature”. Armed with this information, the team combined the two pieces of information. This combined information provided the team with not only “Code Churn”, but “Defect Churn”, in other words, of the code changes made, how many of them were defects? None? All of them? The only thing the team needed now was some way to make this information a little more accessible to everyone on the team by turning the light on and shining it down on the buggy code.
Visualizing The Information
While there may be many forms of visualization for this type of information, the team chose to use a tag cloud using the code filenames. Files that changed for more unique tickets (say, 10) were shown larger than files that changed relatively rarely (say, 1). This made highly modified files stand out more than files that were modified less often. What it didn't do, however was shine the light on the bugs in a useful way.
To solve this problem, the team turned to color. To do this, they transitioned the color of the filename in the tag cloud from green to red based on the ratio of defects to feature
changes. For example, if a given file changed for five tickets and none of
those tickets were defects, the code was considered “clean” and
colored green in the tag cloud. By contrast, if the same file
was changed for five tickets and all five tickets
were defects, the code w a s c o n s i d e r e d “buggy” and colored red. This gave the “ h e a l t h ” o f individual code
DEFECT DENSITY HEATMAP
23
files at a glance and, when combined with the size of the file in the tag cloud, could help make informed testing and quality decisions nearly instantaneous: Big and green? You’re probably ok, big and red? Tread lightly and test more.
Outcomes
Leveraging the data exhaust created by the traceability process, the team was able to make informed decisions about their code at a glance and, since the heatmap was generated from the VCS, it was an accurate, constantly evolving, picture of the system.
Commitment! Case #4
One group discovered how simplicity can trump elaborate planning
Team commitments are a cornerstone in Agile Software Development using Scrum and because of this they are taken pretty seriously. Teams often times spend a fair amount of effort trying to nail down exactly what they can and cannot commit to by taking things like vacations and production support into consideration during forecasting, only to find out they still got it wrong in the end.
Often teams turn to the estimation process itself or tracking time more precisely in order to take control of their commitments.
THE COMMITMENT CONUNDRUM
25Unfortunately, these approaches rarely work and a simpler approach was adopted by one team that we worked with.
Commitment Vs. Forecast
The notion of commitment is becoming more antiquated and less often used in Agile since often times roadblocks are encountered that are outside of the control of the delivery team. You can’t commit to something you can’t directly control right? That’s why we’ve moved to using the word “Forecast”. In a forecast, we are stating our prediction, but leaving open the idea that uncertainty still exists about whether the team will be able to deliver on that goal.
One set of teams within a large organization w e r e n o t r e l i a b l y m e e t i n g t h e i r commitments (only about 40% of the time), which made pro jec t t rack ing and forecasting very unreliable. The leadership team needed something better, but wasn’t sure how to find it.
Sometimes Age is More Than Just A Number
Just because an idea is old, doesn’t mean it’s not useful. One of the original concepts in XP falls right in line with the Forecasting
metaphor: Yesterdays Weather
The idea of Yesterday’s Weather is simple: If you want to know what today’s weather will be like, take a look at yesterday, it will likely be similar. In software, we can take what our velocity last sprint (or, if you want to get fancy, an average of the last few sprints) and that number will be a decent approximation of what can be accomplished this sprint.
In order to test the use of Yesterday’s Weather, the teams turned to its history that was gradually being collected over the course of
THE COMMITMENT CONUNDRUM
26a year. They went to their time tracking tool (they were using Rally) and downloaded a year’s worth of commitments and actuals as a CSV file. This allowed them to use their historical data and run a couple of what-if scenarios to see how Yesterday’s Weather would stand up to reality.
Surprising Results
Using some good old fashioned Excel wizardry, the team updated all “Committed” values in the spreadsheet to those taken from the previous sprint and looked at the updated “Say/Do” ratio. It went up significantly, to over 70%. This confirmed that, while it still didn’t get the teams to the desired 80-90%, this simple solution got them much further than any of their existing efforts and it was easier on the team to boot. With no guesswork about availability, sprint planning sessions were more concise and the teams were more predictable than before.
The Past As Data Exhaust
Using their unused but recorded historical information, these teams identified & reintroduced an old rule of thumb, improving their planning process.
Case #5
Death By Tests
Data exhaust floating in the CI environment can help your team identify poor tests
DEATH BY TESTS
28
Sick Tests
Death by tests usually occurs after your automation practice has been in place for a while and you’re building up quite a large suite of tests. You’ve probably experienced the symptoms of this slow death in your organization. For example, have your tests experienced any of the following?
• Take too long to run (hours vs. minutes)
• Intermittent test failures
• Failing tests that don’t lead to defects
Everyone knows that test automation is a critical component in building a strong IT organiza t ion . Unfor tuna te ly, many organizations cannot answer a follow-on question: “How effective are your tests?” Can you?
Feedback Loops
Arguably the most important outcome of writing automated tests is in the feedback they provide. This feedback comes in the form of increased confidence in the current state of the codebase. If the tests are green, we are more confident that the code is behaving as expected. If they are red, we are more confident that the code is not working as expected. As with all feedback, in order for it to be effective, it has to be timely. One of the most popular test strategies out there, testing through the UI with tools such as Selenium/WebDriver or WATIR, does not scale well when it comes to providing timely feedback. These tests are typically long running & filled with instability. (For more, see “Test Automation: Yes you need it and yes you’re doing it wrong!” booklet)
DEATH BY TESTS
29in many organizations. These tests are commonly identified using the phrase “just run it again” and lead to false positives. If the test is not repeatable, your team will eventually lose confidence in the test suite and, for all practical purposes, it will be ignored. Tests such as these are poisonous to your automation culture and they should be improved or eliminated.
Failing Tests That Don’t Lead To Defects
The final symptom we will be talking about today is also very common in many organizations. The purpose of a test is to
Long Running Tests
This first symptom of Death by Tests happens for one of a couple reasons. First, the test suite itself grows to a point that the suite takes hours to run and causes the developer (or development team) to wait for the feedback. Since we don’t like to sit around for hours at a time (unless there’s a foosball table in the break room of course) we tend to start to ignore the tests or run them less often than we would otherwise, disconnecting the feedback from the event (code change).
While, on the whole, suites can get larger and run longer, they are often the victim of a few long-running tests that take up the bulk of the time. In this case, separat ing out the long running tests cases from the shor te r ones ( in to “ fa s t feedback” and “slow feedback” suites) can help your team get the feedback they desire and keep your longer running tests.
Intermittent Test Failures
Tests that fail intermittently is another common symptom found
DEATH BY TESTS
30from your “fast build”. This increases the timeliness of your feedback loop, ensuring your tests continue to get run.
Looking at the same information as a baseline but sprinkling in a little bit of information from the version control history (also found in the logs of most build servers) you can identify tests that went from red to green without any code changes being made in between. Flagging these tests allows your team to see the offending tests and improve or eliminate them.
Finally, there are more than a few tests out there that are always being updated, but never lead to defects. Using the same data,
ensure that the code behaves in such a way that it matches the requirements. Any time a test fails because the code doesn’t match the requirements, it indicates a defect. If the test fails and it is determined that fixing the test is in order, then test maintenance can become burdensome. You can see this type of situation make itself visible in a number of ways, and often it involves UI changes.
Identifying Bad Tests
You know the symptoms of sick tests, but what do you do about it? How do you identify them? Modern organizations often use build servers such as Jenkins or Bamboo as a part of their Continuous Integration process and this is a great source of data exhaust. Mining the log files can uncover almost all the information you need to identify these symptoms.
Take the slow tests as an example. Each test run typically includes test timings in the log output for that test. By selecting a threshold (say, 15 seconds), you can use the logs to get average test times for each test, flagging slow tests for exclusion
DEATH BY TESTS
31
you look for tests that were red, the test changed, and then back to green. These are brittle tests, and they need to be updated or eliminated. If the test has never actually caught any defects and requires routine maintenance, is it worth keeping?
Staying Alive
Using the data exhaust floating around in your CI environment can help you treat your sick tests and maintain a healthy test suite.
• LearnerReceivesCertificate [Brittle, Slow] • InstructorLaunchILT [Brittle] • AdministratorLearningPlanProgressReport [Slow] • AdministratorSearchForCannedContent [Brittle]
• LearnerTextSearch [Slow] • AdministratorAutoAssignmentOfGroups [Slow] • AdministratorAutoAssignmentOfCourses [Slow] • AdministratorAddUser
Case #6
How one team leveraged the data exhaust from the most unlikely of places to improve their whole team’s productivity.
Go Faster!
GO FASTER!
33
Options
As the team explored its options for improving their velocity, it became apparent to them that managing interruptions would be a key component in improving their overall velocity.
H o w d o e s y o u r t e a m d e a l w i t h interruptions? This team, like many others came up with several solutions, including:
• Require all questions for the team to be routed through the Scrum Master
• Set aside one team member each sprint to deal with interrupt-driven work
• Sequester the team in another area of the building or a conference room so nobody can find them
As you know, there are other options, and these three seem to be some of the more frequent solutions to the problem. If you look closely however, only one of these solutions addresses the root cause of the
issue. Sequestering the team, will do a pretty good job of minimizing
the interruptions (as long as e-mail and IM are turned off to the
outside world of course) but it is a big step, and makes it hard to work with the team when there are critical issues and
Velocity. It’s a hot topic in the industry and you typically can’t get through a Sprint retrospective without hearing about it. I’m guessing you’ve heard this question a time or two as well:
“How can we increase our velocity?”
It’s a common question asked by a number of teams and leaders, and it’s difficult to answer. Do we need additional staff? How do you justify it? Are there too many interruptions? How do you quantify that? One team had those questions, and by examining and visualizing their own data exhaust, they were able to tell a story about their productivity that was hard to ignore.
GO FASTER!
34through the Scrum Master. Again, this didn’t help solve the problem, although the Scrum Master would have the ability to prioritize the work and push back on some of the interruptions. In addition, they felt that people outside of the team already had relationships with the team members they were reaching out to on a regular basis and trying to get them to change their behavior would be difficult.
Education As An Option
The team wasn’t particularly fond of any of their options at this point since most of them don’t address the root cause of the problem so they turned to a different approach. What if educating the leadership team about the true cost of the interruptions would cause them to take action and reduce their demands for increased velocity? They decided to try it.
In oder to tell a story to management, they first needed to make it visible. Their first
idea was to “track interruptions”, but as anyone who has tried to track their
time this way knows, it can be burdensome and it’s easy to
forget to write interruptions down. They were hoping for something less error
prone, so the turned to the data exhaust for answers.
reduces the quality of the interactions with supporting team members such as business stakeholders, project/product managers, and SMEs.
After deciding that moving the team around wasn’t a good option, the team discussed the issues associated with the other options, starting with setting aside one “firefighter” on the team each sprint.
This option didn’t appeal to everyone on the team since, as a newer team, there were still quite a few knowledge silos to be broken down. The team felt that this style of managing the work would not technically reduce interruptions by much, and may make matters worse (although it may have a positive impact on cross training) since the “firefighter” would likely need to ask the specialist for assistance.
Finally they discussed funneling all q u e s t i o n s
GO FASTER!
35After recording this information for a couple of sprints, one of the developers grabbed the log files and wrote a script to start processing them. They decided that a “distraction” was defined as something that took them away from their IDE for more than 15 minutes at a time. This gave them some wiggle room for those impromptu team conversations or whiteboard sessions, but was short enough to provide useful insight. When it was all done processing, the team had a simple timeline showing when developers were coding an when the weren’t.
Eye Opening
While it was admittedly scary to record this information, it was also very informative. Looking at the results, the development team realized that they only spent 2-2.5 hours per day on average actually coding! They were even shocked! Even worse, the 2.5 hours wasn’t consecutive, it was broken into 15-30 minute chunks throughout the
day. Intrigued by this and how to respond when they showed the information to the leadership
team, they decided to cross reference it with one other piece of
information laying around in their systems…
Hands On Keyboard Time
As the team discussed what was value-added time and what was not, a pattern came to light. They felt that when they spent longer times away from their IDEs, the less productive they were. The interruptions came in the form of drop-in questions and meetings. They decided that the best way to show it was to do something drastic: Install a Key Logger into their IDE that would watch their every move.
The team was using WebSphere and Rational Application Developer as a platform and knew it was based on the Eclipse platform. They found an open source key logger online named Fluorite. After it was installed, it recorded a log of all of their activities within their IDE, effectively capturing the exhaust they generated during their day-to-day development.
GO FASTER!
36
About That Key Logger…
After their experiment was complete and improvements were made, there was no longer a reason to capture the data exhaust generated from IDE usage so the team deleted it. Nobody denied the non-invasive utility of it, however, it allowed them to capture more information at a fine grained level than they could have done on their own.
Scheduled Interruptions
By cross-referencing team members’ Outlook calendars with the information they collected, they quickly realized that the organizations meeting focused culture was a major black hole for their time. Armed with t h i s a d d i t i o n a l i n f o r m a t i o n , t h e y approached their leadership team with the charts.
This information was then presented to the leadership team and they ran with it, re-evaluating meetings and working to reduce the team’s involvement in non-critical discussions. They also set rules such as meetings with developers should be held in the afternoon, to provide the teams with more focused time.
Case #7
Regression Testing can’t keep up with the pace of development.
…or can it?
REGRESSION TEST WOES
38accounting integration.” Other examples we commonly hear from teams include statements such as “We didn’t change anything that could materially impact the user management functionali ty, we probably don’t need to spend much time on that.”
Leveraging the knowledge of your team is crucial to making good decisions about how they use their time but, as you know, our memories are fallible and don’t tell us the whole story. What we need is to somehow augment that expertise with something that doesn’t easily forget things, and we can do that using some more of our data exhaust.
Think back to your last release. Did you have enough time to do proper regression testing? If you’re like many organizations, the answer is a resounding “No!”
What do you do in this situation? Delay the release in order to ensure you complete your testing? We’d like to believe that delaying the release to ensure quality is a viable option, but schedule and market demands typically win out over quality in the short term. What your organization probably does is take a pragmatic approach to testing, ensuring that you use your limited time in the best possible way by testing the most important functionality first.
Identifying Testing
Often times, identifying required testing takes a “risk-based” approach. The team will take a look at the changes being made for the upcoming release and have an open dialog around the risks and concerns the team has for each area of the system. We typically hear examples such as: “We made s o m e p r e t t y s i g n i fi c a n t changes to the billing module, so let’s make sure we test renewals and
REGRESSION TEST WOES
39plan from it. Unfortunately, while the technology exists to make this possible, it is not exactly practical and tracking it can slow down your test environments.
There is another, albeit less precise, way to accomplish the same thing though, and that’s the combination of ticket traceability and your version control system as we saw in the Defect Density Heatmap case study.
Ch-Ch-Ch-Changes
One of the interesting aspects of your code is that code that interacts and works together in some way has a tendency to be changed together. You can use this property
Augmenting People
Even with the best intentions, people make mistakes and overlook things. The question becomes, how can you reduce the opportunity for mistakes while still leveraging the knowledge of your people?
Once again, the data exhaust generated out of the version control system can be harnessed to paint a powerful picture about testing risk, that is, if you’re looking for it.
Prioritizing Testing
Deciding what to test essentially involves understanding how different aspects of the system fit together to form the whole and using that information to prioritize your testing effort, focusing on the riskiest areas first.
The thing is, your systems already know about which components work with other components, if you can track their run-time code paths. Using this information you could definitively identify related features (features that execute the same code) and build your test
REGRESSION TEST WOES
40
Making The Connection
Once again, the VCS is your friend. By getting your hands dirty and writing some code, you can pull your commit history and identify what code changes are associated to what tickets that were submitted (assuming traceability mentioned in the heat map case study). Once you have the information about what code changes were made for each ticket, you can now look for other tickets that change the same code. The details of how to write the code are beyond the scope of this document but feel free to contact us and we can provide some guidance.
to find code related to the existing change.
Consider a simple example: imagine TaxBracketResolver.java changed for the “Run Payroll” story and the “Prepare Estimate” story. Because each story required changes to the same code, then it is fairly safe to assume that these two stories are related and should both be tested if any code in this file changes.
How about a more complex example? Using the previous example as a baseline, let’s now assume that, in addition to TaxBracketResolver.java, the “Run Payroll” story also required changes to FicaCalculator.java, as well as PayStub.java. In addition, consider a third story, “Pay Commissions” that also required the developers make changes to TaxBracketResolver.java and PayStub.java. You can reasonably assume that, of the three stories, “Run Payroll” and “Pay Commiss ions” a re more strongly correlated and some extra time should be taken to test both of those features. If time permits, your team can spend the rest of it testing the “Prepare Estimate” feature since there were fewer overlapping code changes.
REGRESSION TEST WOES
41
There is an additional side benefit to this tool that the team never expected when it was built: Training. By showing the relationships between features in the system, team members new to the application were able to quickly discover related features within the system that they may have not been aware of before, but needed tested when changes were made. This had the added effect of growing the team’s overall knowledge of the system as a whole.
The Results
After building the relationships, the team generated a Venn diagram that showed the historical relationships between the different features within the system. While the results are not perfect, they provided insight never before seen by the team and was considered an enabler to more effective testing.
Data exhaust is everywhere. You just have to look for it and capture it. We believe that once you get the hang of looking for it, it’s relatively easy to observe and you’ll see it everywhere too. In this document we showed you seven cases where we spotted it and used it to our advantage. We used it to:
• Build a business case for laptops
• Ensure files aren’t forgotten
• Identify buggy code
• Set more realistic commitments
• Improve test feedback
• Identify interruptions
• Improve regression testing
After reading this we hope you will be able to identify and leverage data exhaust to your advantage within your organization. When you do, we’d love to hear about it.
If you have any questions about any of the content of this document please do not hesitate to reach out to us at:
E-Mail: [email protected]
Phone: 515.669.5597
Web: http://leantechniques.com
CONCLUSION
42
