2TestCompleteby SMARTBEAR

Strategizing Your Mobile App Testing

ContentsIntroduction ....................................................................................................3

How to Develop a Web and Mobile App Testing Strategy .............................4Testing your way to success ......................................................................................4How to get started with your mobile app testing strategy .........................................5

The Benefits of Emulated vs. Real Device Mobile App Testing .....................6Advantages and disadvantages of each approach ...................................................7

The Most Common Cross-Platform Mobile App Development Challenges ...8How to confront the cross-platform challenges ..........................................................9The first step – goal-setting........................................................................................9

Why You Should Build an Automated Mobile App Testing Matrix ................10There are no quick fixes........................................................................................... 11

Conclusion ...................................................................................................13It’s a lot of effort, but it’s worth it...............................................................................13

3TestCompleteby SMARTBEAR

Introduction

If you’re like most app developers, you spend a great deal of effort try-ing to create apps that work the first time and every time on as many platforms as possible. For instance, your team is developing an app that needs to have access to a database on a PC, on a mobile phone and on the web. Testing that database will require approaching testing it in three different ways. For the sake of this eBook we won’t go into the conventional way of testing a desktop application or the nuances of test-ing a web application but rather look at mobile app testing.

That’s not as easy as it looks. With tens of thousands or hundreds of thousands of lines of code, launching a successful app can be very challenging. To make matters worse, mobile app users have high expec-tations. In fact, one study found that 60% of users will abandon your app if it doesn’t load within 3 seconds.

As if that weren’t challenging enough when an app doesn’t work proper-ly, users are just a thumb click away from giving it a bad rating or shar-ing their feelings about it on Twitter, Facebook, LinkedIn or Google+.

The bottom line is that there are a lot of landmines out there. Which makes a difficult job even more challenging.

That’s why we’ve created this guidebook. In it, you’ll see a variety of tips and techniques you can use to help you navigate the landscape of app testing across different platforms.

Here’s just some of what you’ll find on the following pages:

¿ How to Develop a Mobile App Testing Strategy

¿ The Benefits of Emulated vs. Real Device Mobile App Testing

¿ The Most Common Cross-Platform Mobile App Development Challenges

¿ Why You Should Build an Automated Mobile App Testing Matrix

4TestCompleteby SMARTBEAR

If you’re looking for a step-by-step roadmap on how to successfully test your way to success, then this guidebook is just what you’re looking for. It provides everything you need to set-up, launch and manage a suc-cessful web and mobile app testing program of your own.

So, with that in mind, let’s get started.

How to Develop a Web and Mobile App Testing Strategy

With so much demand and so much competition, mobile app testing has never been more important. Indeed, the testing phase is often seen as a necessary evil, sandwiched between the creative process of inception and the excitement of taking a new product to market. How-ever, drilling down on the positives that testing brings make this step anything but evil.

According to Flurry Analytics, “the U.S. consumer spends an average of 2 hours and 38 minutes per day on smartphones and tablets. 80% of that time is spent inside apps and 20% is spent on the mobile web.”

Given the growth of the industry as well as the time consumers are spending inside apps, developers should have every excuse they need to develop a testing strategy that ensures what they place in front of the consumer operates correctly the first time.

Testing your way to success

How many times have you read a story about an app that has been pulled because it failed on a massive scale? The truth is, proper testing means the difference between: “Hey, I love that app you guys cre-ated,” and “Aren’t you the guys who had to pull your app off the market because it kept failing?”

In 1978 when Federal Express made the promise “When it Absolutely, Positively has to be there overnight,” you can be sure they did so only

5TestCompleteby SMARTBEAR

after testing and fine-tuning their process and services. They did it so well that when people send something overnight they frequently say they are going to “Fedex” it, even when using another carrier.

A company’s name does not become part of society’s vernacular by hoping what they offer works. Instead, it becomes part of society’s ver-nacular by testing their way to success.

How to get started with your mobile app testing strategy

First, you should have a very specific statement of what you want the app to do, as well as what you are not designing it to do.

That sounds painfully obvious, but think back to those brainstorming sessions when all ideas were fair game. At some point everyone on the team needs to be crystal clear on exactly what the end product will be.

Next, you’ll need to review a series of questions prior to executing your plan. What follows is a list of some of the key questions to consider as you develop your app testing strategy. By asking these questions now, you’ll ensure a successful launch in the future.

¿ What functionality do we need to include in the final version of this product?

¿ What are some of the most common bugs in products like these?

¿ What kind of testing will be part of the development process?

¿ Will we be using in-house teams? Or outsourced partners?

¿ Will we be using emulators? Actual devices? Or both?

¿ What role will agile testing play in the development process?

At some point everyone on the team needs to be crystal clear on exactly what the end product will be.

6TestCompleteby SMARTBEAR

¿ How will we test across platforms?

¿ Were will the app roll out? In other words, what role does geography play in the final deliverable?

¿ At what point will we conduct real world tests?

The questions above are not all encompassing, but they should give you a sense of how to think about your mobile app testing strategy. The most important thing is to start with the end in mind. By focusing on the final outcome, you can work backwards from there and create a road-map that starts with the initial concept, then moves through develop-ment and testing and ends with a successful launch of the product.

Pre-launch testing of any mobile application is a crucial step before going to market. Testing eliminates the chances your product will have to be recalled, not to mention the public embarrassment that goes with a defective app. And by testing your app thoroughly before it goes to market, you communicate to the consumer that your brand is one they can trust, as well as one they can buy from again.

According to International Data Corporation, by 2015, 182.7 billion mobile apps will be downloaded. That’s a 1600% increase from the 10.7 billion apps downloaded in 2010. What mobile application development company wouldn’t want in on that?

While there are a number of OS’s out there, and more coming, there are two predominant ways that de-velopers can test a mobile app. You can go with device testing or use an emulator program. At first glance, you might feel the need to choose just one avenue or the other, but the truth is that using both approaches is the best route to success.

The Benefits of Emulated vs. Real Device Mobile App Testing

You can go with device testing or use an emulator program.

7TestCompleteby SMARTBEAR

Advantages and disadvantages of each approach

One of the key advantages of device testing is that you’re holding it in your hand and can see what works and what doesn’t work. You can test sign-up, login, data handling, connection speed and error messages on the fly.

You also find out if the app is as easy to use on one device as it is on another. Is navigation on a touchscreen as easy as a device with a QWERTY keyboard? Is the user experience on a Samsung Galaxy S4 circumscribed by the smaller screen of an iPhone 5? If one phone has a screen that doesn’t deliver the same resolution as another, is there a significant drop in experiential quality?

The challenge with device testing is that if you want 100 tests, you’re going to have to do them one at a time. Or get another 99 testers, all looking for the same things you are. It is also worth asking, will every single person testing the device use the same standards as you do?

Of course, to test on all platforms is going to require the acquisition of a lot of handsets. Not a deal breaker, but not cheap. As such, some mobile app developers prefer to use emulators or simulators to test the operation of an app.

Most operating systems have a readily available SDK that allows for creation of apps. As such, multiple emulators can be programmed to run doing much of the same work as a device tester, but faster and at greater volume. It does not require the same amount of device hard-ware, either.

Additionally, an emulator can extract data in real time and refresh re-ports as it runs, so the information an app development team is working with can be quickly accessed.

8TestCompleteby SMARTBEAR

This is not to say emulators are the alpha and omega of mobile app testing. Unless the software is programmed down to the finest nuance of human behavior there are things a person will catch that an emula-tor won’t. Likewise a computer or computer program can’t walk across the room if something looks suspect, tap you on the shoulder and ask a question. As such, most developers are going to use a combination of testing methods to get the best results.

Ensuring the success of a mobile app in this marketplace is too impor-tant to make your testing an either/or proposition. Since neither is inher-ently superior, as judged by the number of developers who rely on both, it would be shortsighted for anyone to place all their testing in just one.

So if someone should find out you’re in mobile app development and ask which testing method – device or emulator is better – the answer is “yes.”

The Most Common Cross-Platform Mobile App Development Challenges

We live and operate in a competitive environment – one in which we’re constantly asked to decide between competing choices. Are you Mac or PC? Office Max or Office Depot? Paper or plastic? We get used to the feeling that we have to choose either one or the other.

This goes out the window when it comes to developing mobile apps, because in the mobile app universe we want to play on all stages. While there might be some who want to only work on iOS or Android, most of us are not willing to block off those revenue streams.

That said, it’s worth spending a few minutes thinking about whether de-veloping an app that goes across all platforms is good for your business. For example, suppose your app is just right for a phone with a touch-screen UI. There is still an audience out there for the “old fashioned” QWERTY/non-touchscreen device. Are you going to go after them? For

9TestCompleteby SMARTBEAR

example, Temple Run on a Blackberry Curve might not be as fun as it is on a Samsung Galaxy.

It’s a conversation worth having internally with the development team. But for now let’s assume you’re going all in – iPhone, Android, Black-berry, Windows, et al. What hurdles await?

How to confront the cross-platform challenges

While all platforms offer an SDK for developers to work with, we hope no one in your development group will be surprised at the specificity that goes with each. Keep in mind that as you develop an app going across all platforms, each of those companies is battling for market share. So not only is your app going to need a framework designed for each OS, it is also going to have to be imagined to accept updates as they occur.

Don’t assume the SDK will be the panacea. It might be wise to be more app-centric than device-centric at this step. By that we mean how do you want you app to look and perform regardless of the device? Be prepared for updates as they happen – and any smartphone user knows they happen on a regular basis.

Make sure your development team doesn’t get too attached to just one framework. One member of the U.S. Marine Corp. said it best; we should all be “Semper Gumby” – always flexible. Quite often going from one OS to another is going to mean learning how to use some new de-velopment tools and master new skills. Being the best iOS app creator may sound like a nice title, but there are a significant number of people with money to spend who don’t have iPhones.

The first step – goal-setting

This is a good time to step away from development and talk about your goals. It’s reasonable to assume that any competent developer can create an app for a specific phone. However, if you are a company that

10TestCompleteby SMARTBEAR

hopes to develop apps that go across several platforms you’re going to want to build an infrastructure that isn’t thinking one dimensionally. Before a company ventures too far into the arena, management needs to make sure its teams are going to have the resources to work across platforms from the beginning.

Another hurdle is to ensure the UI experience is going to translate cleanly across all platforms you choose to go after. The end game should be that no matter what device is being used, the experience is exactly the same. Nothing can create a buzzkill around an app as the news that it works great on one platform but not another.

The bottom line. As mentioned previously, a wise investment in device and emulator testing is a must. According to Gartner Research world-wide revenue from app stores will increase this year by 62%, bringing the total industry revenue to $25 billion dollars. That’s an audience with too many choices to tolerate glitches.

And finally think about this. A considered, well-funded investment of time and money developing App A that will shine across all desired platforms is going to give you the framework for App B, App C, etc. Making that investment, in the short term, may look expensive for just one mobile application but those same resources can be brought to the table again and again. So if you have the ability to reuse the code, design or data-base you’ve made your future a lot brighter.

Why You Should Build an Automated Mobile App Testing Matrix

Creating cross-platform mobile development protocols that allow you to play in virtually any OS sandbox is a great idea, but the execution of that is getting increasingly difficult to manage. There are a score of different platforms, OS protocols, form factors and carrier/manufacturer custom-izations that have to be considered.

11TestCompleteby SMARTBEAR

Now layer over that mobile web differences, native apps, or the notori-ous “all of the above” and you can see that creating an effective, com-prehensive and omnibus testing matrix is a marathon. Maybe two.

There are no quick fixes

If the notion that there is not a quick fix for executing across platforms bothers you, keep in mind that the companies creating proprietary op-erating systems for their proprietary devices are trying to clobber each other. If they had their way they would run the competition out of town.

Understandably they are not staying awake nights, prostrate with worry over how easy it may be that an app you created for their device can easily be translated to the competition.

Previously, we discussed the merits of device testing vs. emulator test-ing, coming out strongly in favor of both. In the case of the former we don’t think there’s anything that can replace having the device in hand and actually working with the app. However, having a great app testing framework is also going to require automated abilities as well, for sev-eral reasons:

¿ It ensures that application features work as designed on the selected mobile devices – not to mention performance on various networks and bandwidths.

¿ It tracks stability of the application on desired mobile devices and networks.

¿ It reduces how long it takes to get it market ready.

¿ The increasing number of mobile applications platforms can be factored in.

¿ It gives real-time results on an ongoing basis, simplifying the data retrieval process.

12TestCompleteby SMARTBEAR

Further, the testing matrix should have the flexibility to go from object testing to element testing. As such, your testing should address three main areas:

First, the number of companies out there making devices, as well as the number of devices your app needs to be available to. This means atten-tion must be paid to rendering differences and run times, with the latter variable between handsets manufactured by the same company.

Second, what are the network types that are going to be brought into play? Within those networks how much customization is present? How do the speeds vary?

Third is to look at the devices and conduct a SWOT (strengths, weak-nesses, opportunities, threats) analysis. Some apps may be best used on a device with a large, high-resolution touchscreen. An app that allows the user to reserve a hotel room probably will be wonderful regardless of how much visual real estate a specific mobile phone offers.

And while there is not a one-size-fits-all testing solution coming down the pike that would bring about a huge cost savings when it comes to a superior app testing matrix, consider that once you have one in place it will facilitate your work on future applications going forward.

True, testing your app across all the mobile platforms is never going to be easy as 1-2-3, but setting up, updating and maintaining a mobile app testing matrix isn’t about today. It’s about setting up systems that allow app developers to more easily reach broad audiences with their wares. It’s about building your company’s image in the marketplace as a go-to resource for consumers, no matter which mobile device they may have at any given moment.

13TestCompleteby SMARTBEAR

It’s a lot of effort, but it’s worth it

If this sounds like a lot of effort, consider this from The Wall Street Journal:

The question is not whether you should make a substantive investment in a superior testing matrix; the question is why wouldn’t you give your mobile apps every possible chance to succeed?

Conclusion

With a better understanding of mobile app testing and the different ap-proach needed as opposed to desktop and web applications, you now have the tools needed to test on all platforms. For web and desktop ap-plications TestComplete is a great tool to automate the testing process for you. Try a 30 day no-cost trial of TestComplete today and see the power of fully automated testing.

“The odds of striking gold in the apps business are quite long. While there are more than 800,000 mobile apps available in Apple Inc.’s App Store, only 80 of them generated more than $1 million in revenue during the fourth quarter, according to research firm Distimo. Reaching a few hundred downloads a day used to get an app on a Top 10 or 20 list a few years ago. It now requires several thousand, so larger apps companies—such as Electronic Arts Inc., EA Walt Disney Co., Zynga Inc. ZNGA and Rovio Entertainment Ltd.—tend to dominate the top-seller lists. Only 2% of the Top 250 publishers for iPhone apps in the U.S. App Store are newcomers, again according to Distimo.”

Conclusion

About SmartBear Software

More than one million developers, testers and operations profession-als use SmartBear tools to ensure the quality and performance of their APIs, desktop, mobile, Web and cloud-based applications. SmartBear products are easy to use and deploy, are affordable and available for trial at the website. Learn more about the company’s award-winning tools or join the active user community at http://www.smartbear.com, on Facebook or follow us on Twitter @smartbear and Google+.

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915+1 978.236.7900 www.smartbear.com ©2013 by SmartBear Software, Inc. Specifications subject to change. SB-TC-053113-WEB