ebook-Understanding-CT-in-Agile-World

DevOps Guide

Understanding Continuous Testing

in an Agile World

SOASTA: Performance Is Everything 2

IntroductionThere are two concepts that are fundamental to the modern development life cycle:

1. The role of user experience in agile development is critical to the success of testing.

2. Continuous testing is critical to the success of agile development.

In the next few pages, we will introduce and explain these two concepts. We’ll also dig deep into the process of continuous integration (CI) and the key processes that make up the overall CI process.

We’ll discuss the relationships and process flows between change management, configuration management, and release/build management —and how the CI process, when coupled with a solid performance engineering discipline across the product lifecycle, can result in a better user experience for your web and mobile applications.

We’ll also cover the “conveyor belt” of the application lifecycle, including explanation of the “Big Three” processes that support overall CI strategy.

“The problem with a lot of people who aren’t familiar with load testing is they think it’s just a bandage you put on right before launch. That is not the way to look at it.”

TOM POIRIERIT Manager

IDEXX Laboratories


In an agile world, user experience is key

Gathering 100% of your user data is no longer a “nice to have”: it’s a “need to have”. That’s because analyzing real user data lets you ask questions such as:

• Are users using your application in the way in which it was intended?

• Or have they found a shortcut or alternate path that was not considered during initial UX design and testing as a mainline path, but now must be added to the test coverage matrix on the functional side, as well as analyzed on the performance side?

CASE STUDY: IDENTIFYING AND OPTIMIZING A COMMON USER PATH LED TO INCREASED CONVERSION RATEOne major retailer discovered that the most common and shortest user experience path for women customers was directly accessing the shopping cart and checking out. In the sunburst chart below, real user measurement data collected from SOASTA mPulse is visualized to show the most common conversion paths that users of this site followed when they completed a transaction.

Once they looked at the data, it was easy to understand why the short cart-to-checkout path was so popular. The retailer found that its women customers were savvy shoppers: they put items in their carts to see what the final prices would be, and then went off-site (typically through a different browser tab) to access competing retailers’ sites and compare prices. If this major retailer

PA

RT

1


offered the best price, then customers returned to the site to resume the purchase process and complete their transactions.

Once the retailer began to test performance in this scenario, it took advantage of these insights by offering discounts when shoppers returned back to the site. This increased the site’s conversion rate, which equated to increased revenue.

TAKE THE GUESSWORK OUT OF USER EXPERIENCE: KNOW YOUR PERFORMANCEBy measuring real user experiences of your application in this manner, you can quickly and clearly see how your users are using your website or application.

• What are their click-through paths to conversion?

• Which click-through path leads to the most revenue?

• What should your test coverage model be?

• What are the true edge cases?

DOES YOUR TESTING STRATEGY ENCOMPASS THESE FIVE BEST PRACTICES?

UX and performance testing with real-user data is a game-changer. Your testing strategy should encompass these five best practices:

1. Develop an in-depth application performance strategy that encompasses real-world scenarios and real-world data.

2. Analyze your application back-end metrics to get a fuller picture of the sometimes-invisible factors that affect performance.

3. Analyze your front-end performance metrics to get a truer understanding of your end user’s experience with your application.

4. Tie your application’s performance with revenue-generating or cost-generating transactions.

5. Identify the “hot zones” on your sunburst chart where you can have a fast impact to most effectively improve your website or application performance.

If you are a practicing agile shop, these activities and best practices are a must-have for your UX — and your performance team.

PA

RT

1


Redefining a healthy development life cycle

Now that you understand why it’s crucial to focus on user experience throughout the testing process, let’s talk about why testing needs to be built into the entire development life cycle.

THE HISTORIC PROBLEM: TOO MANY SILOSTraditionally, the software development life cycle has fallen into three silos:

But this dated thinking does a serious disservice to all three silos. While software development and monitoring belong in silos, testing does not.

And neither do the key supporting processes that make up the conveyor belt.

Testing is an integral part of these “Big Three” support processes:

• Change

• Release

• Configuration Management

These are called “The Big Three” because they are the foundation for product development speed and continuity. Coincidentally, they also form what is arguably the baseline for DevOps.

HISTORIC

� SOFTWARE DESIGN

� CODING

� CHANGE

� RELEASE

� CONFIGURATION MANAGEMENT

Development

� CHANGE

� RELEASE


QA/ TestingPre-Production

� IT OPERATIONS

� MONITORING

� CHANGE

� RELEASE


Production

PA

RT

2


DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.

These three processes should persist throughout the development life cycle, as they are the foundation of each silo.

These processes are called “supporting processes” in the US Commercial Software Standard, ISO 12207. They’re “supporting” because they support each silo, rather than being found in just one silo. (As an example, design or code is found in the development silo only, not in operations.)

So a (simplified) life cycle diagram could look more like this:

Herein lies the problem in many organizations: each silo owns their own piece of the process, but no one silo owns the hand-off from the other.

THE SOLUTION: ENABLE BETTER HAND-OFFS BETWEEN SILOSAs discussed, the testing silo contains pieces that should be spread across the other two silos. Testing should be considered a supporting process.

Testing needs to start in the development silo in fact, in most organizations, it already does with unit testing.

TODAY

SOFTWARE DESIGN

CODING

PERFORMANCE MANAGEMENT CHANGE MANAGEMENT RELEASE MANAGEMENT CONFIGURATION MANAGEMENT

Development QA/ TestingPre-Production

IT OPERATIONS

MONITORING

Production

Performance Engineering & Continuous Integrationspan the full life cycle

PA

RT

2


It should also extend into the operations silo. You should be testing in production to see how your application really works, and how your users are seeing it. You can only do that in production.

WHAT SHOULD A HEALTHY DEVELOPMENT LIFE CYCLE LOOP LOOK LIKE?Now that we have identified the key pain points across the lifecycle, how should a painless (well, relatively painless) life cycle look? Building on the diagram in the previous section, a more detailed breakdown of the processes within each silo could look like this:

Let’s walk through how this life cycle works with a quick scenario:

1. A requirement change (could be a feature request, a bug fix, etc.) leads to a code change.

2. This leads to a “checkin” using a CI solution.

3. This then kicks off a test suite (usually, but not always, from the cloud), where the tests are “run” iteratively (hence the runners).

4. Tests ensure that the platform/device coverage is automated, results are provided (“the truth”), issues resolved, a baseline established, and feedback from real users in the production environment.

5. Then we start the process all over again with a new stream (if agile) or release cycle.

This is how SOASTA’s engineering organization builds our software. With several thousand tests per day and hundreds of release cycles and feature streams, this automation cycle is the best practice that keeps the conveyor belt moving along.

Development (sandbox)

UNIT SCALABILITY

COMPONENTSCALABILITY

PERFORMANCEBASELINE

INTEGRATIONTESTING

FAILOVER

NETWORK

LATENCY

LIVE E2ETESTS

STRESSTESTING

SOAK TEST(reliability)

CAPACITY

DATABASETESTING

PERFORMANCEREGRESSION(smoke tests)

END TO ENDW/ SERVICE

VIRTUALIZATION

3RD PARTYSERVICES

Testing (staging) Operations (prod)

PA

RT

2


Takeaways

1. REMEMBER THAT CONFIGURATION MANAGEMENT IS MORE THAN VERSION CONTROLWithout version control, there is no reliable way to know what a given unit of work contains. At any point in the development-testing-release process, the first debugging question should always be “What changed?” A version control system helps answer that question — but it’s not the whole story.

Version control typically covers just the code under development, whereas configuration management covers the entire process that includes not only software, but also hardware, tests, documentation, connection pool settings, other configuration files, and more. It identifies every end-user component and tracks every proposed and approved change to it from Day 1 of the project to the day the project ends.

Configuration management also ensures reproducible builds. It removes the waste of manually assembling code. Reproducibility is a key component of safety: you can’t safely deliver frequent changes if you don’t know what you’re releasing.

Few things are more frustrating to hear than this: “It worked on my machine.” Environmental inconsistency is the primary cause of this situation. Manual system configuration exacerbates consistency problems, and it’s common for developers to be running different versions of an SDK. They also often test against system software that differs from the software running in the integration environment. That environment, in turn, doesn’t match production.

Fortunately, this problem can be solved using configuration automation and by treating infrastructure as code. A single set of configuration scripts can be used to provision development, testing, and production environments. Consistent deployment of configuration changes across large numbers of environments and machines becomes as simple as checking in a configuration script change. This is one of SOASTA’s internal best practices. It should be yours, too.

A sound configuration management process can ensure that the functionality and performance and physical attributes are known, and managed, across the lifecycle. A good user experience (UX) starts here, whether you are developing with an agile, lean, waterfall or other process. This is the foundation for the “Big Three”, and a key part of any solid DevOps environment.

PA

RT

3


2. UNDERSTAND CHANGE MANAGEMENTThis one seems simple, and it is if done right. This process involves the identification of potential changes to an application, no matter where it is in its lifecycle from day one to sunset.

In reality, potential changes should only originate from one of two places:

• the business owner/customer who requests that the application be developed (e.g. submitting a requirement), and a bug which can originate from an end-user through a ticketing system (e.g. a problem report or a feature request), or

• internally from QA during any of the different testing phases for any and all functional and performance tests that should be executed continuously across the development lifecycle.

3. KNOW THE DIFFERENCE BETWEEN RELEASE MANAGEMENT AND BUILD MANAGEMENTThese two terms are often used interchangeably, but they mean very different things.

The standard definition of a release is “a set of approved changes or features approved for the application”. This may consist of several change requests that have several new features being added, as well as several problem report resolutions that impact features being added/changed in this particular release.

A build, on the other hand, is typically an incremental set of requirements/changes/problem resolutions, that in the CI process world are released and tested incrementally throughout each testing process in the conveyor belt (e.g., functional testing and performance testing in Dev, Test/QA and in preproduction/staging).

4. REALIZE THAT CI BUILD FAILURES MAY BE RELATIVELY QUICK AND EASY TO FIXAs much as possible, however, these failures should be avoided. Development teams should have the goal of checking in complete, correct code. The further “shift left” in the lifecycle you find a bug, the less costly it is to fix. Finding bugs prior to check-in is less costly still than finding them during a continuous integration build.

The best way to catch bugs prior to check-in is to have developers and testers work together, rather than to treat testing as a post-coding activity. Short-lived feature branches are ideal for this purpose. Developers and testers can share, review and run each other’s code. By running the test suite locally, they can minimize the likelihood of discovering bugs later in the lifecycle.

PA

RT

3


5. FOCUS AND FIX THE PROCESS FIRSTYour tools should support your processes, not the other way around. SOASTA has helped hundreds of clients implement a CI process cycle into their organizations with the goal to improve bottom line performance with speed to market, quality improvement, and a focus on performance engineering across the lifecycle all while keeping a laser focus on performance.

6. THE VALUE OF RUNNING TESTS DEPENDS ON THE QUALITY AND COMPLETENESS OF THE TESTSIn a continuous integration environment, tests are code just like any other code. Story planning needs to account for test development time. Development teams need to pay as much attention to specifying needed tests as they do to specifying needed functions. Test specifications need to account for both nonfunctional (e.g. performance testing) and functional validation.

7. ACHIEVING HIGH-SPEED CYCLE TIMES REQUIRES AUTOMATION OF MORE THAN JUST THE TEST ITSELFSOASTA’s internal approach is particularly conducive to compressing the SDLC of web and mobile applications through the use of CI tools and best practices involving the key processes above, all the while coupled with a solid performance engineering process.

When combined with Jenkins/Hudson, for example, it is possible to automate the entire process from build through test and into reporting and diagnostics. Results are displayed in a common interface and automated regression testing can be done completely hands off. This alone would not necessarily obviate the need for any manual testing, but it does make automation, maintenance, and reusability accessible to developers and testers to achieve speed with a quality focus.

PA

RT

3

We help the top 20 online retailers continuously monitor, test, and optimize their web and mobile

applications. We can help you, too.

ADOPTING CONTINUOUS INTEGRATION?

Then you need continuous testing.

Sign up for CloudTest Lite IT’S FREE!

SOASTA: Performance Is Everything

Documents

ebook-Understanding-CT-in-Agile-World