Mobile Research & Survey Design Best Practices Guide

www.apptentive.com @apptentive

Why Do Mobile Research?

Mobile research is a powerful and cost-effective tool for taking the guesswork out of your mobile strategy.

For a mobile-first company, customer satisfaction is the single most important driver of revenue. But before you can hope to delight your mobile customers,you need a mechanism for dissecting their experience with your app and give them a voice to express suggestions and critiques.

Consider the following questions that mobile research can address: ➢ How do customers like your app? ➢ Are there any pain points in the customer journey? ➢ Is there an existing demand for the feature you want to introduce in your next

update?

Why Do Mobile Research?

Mobile research allows you to:

➢ Open up communication with your mobile customers

➢ Identify customer segmentation trends and pain points

➢ Improve mobile retention and customer loyalty by asking customers what they’d like to see in your app, and acting on those insights

➢ Circumvent negative app store ratings by addressing customer issues before they make their way to a public channel

➢ Extend your sample size – if done right, mobile surveys have dramatically greater response rates than web, phone, or email surveys

Getting Started

So how do you begin to design a survey and integrate it into your app?

Start by asking yourself these three questions that we will go over in this presentation:

➢ What are you looking to learn? What are your research questions?

➢ What sort of data provides this insight and what should you be asking to get this data?

➢ Who is your audience and how will you reach them?

What Are You Looking To Learn?

Before designing your mobile research instrument, take a step back and consider your objective for doing this research. Is your research purely exploratory or do you have a specific research question you’re looking to address?

Your sampling criteria and survey design should align with your objective – asking the right questions and getting them in front of those in the best position to answer them.

Research questions can be attitudinal, behavioral, demographic, or technical.

Attitudinal: How do new users like your app?

Behavioral: How do users interact with your app? What are the most common use cases?

Demographic: Which age bracket is your app most popular with?

Technical: How can this app be improved?

What Sort of Data Provides These Insights?

Allow what you want to learn from your research and what data you want to collect and analyze to dictate the questions you ask.

Some question types are inherently better suited for particular cases.

For example, consider using an open-ended question if you’re looking to open up communication and welcome new suggestions.

Likewise, use a closed-ended question with a limited number of predefined response choices if you’re looking for more specific information on a particular attribute of the app or customer journey.

What Sort of Data Provides These Insights?

Survey questions can be classified as:

Open-ended (i.e.: Providing a textbox for the respondent to type his or her answer) ➢ Exploratory in nature ➢ Less likely to result in bias from leading questions/response choices ➢ Provides qualitative responses similar to a focus group ➢ Can be time-consuming to answer, particularly when using a mobile device,

leading to lower response rates

Closed-ended (i.e.: Multiple choice questions and rating scales with pre-defined response choices (see next slide)) ➢ Can provide both qualitative and quantitative responses ➢ Questions typically take less time to answer and experience higher response

rates ➢ Questions and response construction requires more care remove bias

Mixed (i.e.: A multiple choice question with a fill-in ‘Other’ option) ➢ Allows room to write in answers that were not considered when the survey was

designed – may uncover new customer needs/sentiments

Types of Closed-Ended Questions

Dichotomous Multiple Choice Likert Scale Semantic Differential

Example Do you like this app? -Yes -No

How many times a week do you use this app? - 0-1 - 2-4 - 6-7 - 8+

I would recommend this app to a friend. -Strongly agree -Agree -Undecided -Disagree -Strongly disagree

How would you describe your experience with this app? Good 1-2-3-4-5-6-7 Bad

Uses Can be used to direct respondents to different follow-up questions based on their information

Gathering generalized data in a way that takes very little time away from the customer experience

Allows for operationalizing and quantifying customer attitudes and perceptions around an

attribute of your mobile app

Precautions Respondents may not see the question as black and white and desire more choices

Multiple choice options commonly experience overlap. I.e.: How many times a week do you use this app? (a) 0-2 (b) 2-4 (c) 4-6 (d) 6+

Both (a) and (b) are valid choices for a customer who uses the app twice a week.

Commonly leads to a central tendency bias, where respondents are hesitant to choose an option at either extreme, and instead select a middle-of-the-road response. As a result, the

deviation in responses may understate the deviation in actual opinions.

Who Is Your Audience?

As you begin to think about mobile research, first consider the criteria for the sample from whom you want to collect the research from and how you will reach them within the app.

Common mobile research samples include:

Opt-in (no sampling criteria)

Pros: ➢ Survey can be easily integrated as a link in your app’s navigation with no third

party tools or prompts

Cons: ➢ Can lead to selection bias as those who opt-in may not share fundamentally

different views/attitudes than those who do not opt-in ➢ Opt-ins have the lowest response rate and the survey link can be difficult for

potential respondents to find

Who Is Your Audience?

Random sample – Randomly choose a percentage of your mobile customers to survey

Pros: ➢ Diversity of responses makes the data highly representative of your overall

audience ➢ Surveys can be prompted even if mobile analytic capabilities do not collect

information on customers and customer activity

Cons: ➢ Sample may be too broad to address narrow research questions that require a

high level of familiarity with the app ➢ Responses may be too generalized to uncover trends based on customers’

familiarity with the app, device used to access the app, etc.

Who Is Your Audience?

New Users – prompted during the first hour or so of using the app

Pros: ➢ Allows you to collect information, unbiased by existing loyalty, that can be used to

improve the customer experience Cons: ➢ Requesting customer information from first-time users may create a negative

initial experience with the app

Loyal Users – prompted the n-th time a customer opens the app or a few months after installing and regularly using the app

Pros: ➢ Can be used to uncover which features loyal users find the most valuable / what is

bringing them back to your app ➢ Can be used to gather suggestions from those already familiar with those apps –

and customers can be further delighted if those suggestions are acted on Cons: ➢ Responses may have an upward bias and have difficulty capturing equally

important negative experiences with the app

Who Is Your Audience?

Event-Based Targeting – survey prompted at pre-specified ‘mobile moments,’ i.e.: • The third time a customer uses the Search function • The first time a customer shares content via the app • After a customer updates to a new version • After a customer uses a new/beta feature

After 3 Searches

After a customer Reserved a Table

Pros: ➢ Can be used to address narrow, feature-specific

research questions ➢ Can be used to refine and beta test new versions

and rollouts ➢ Respondents inherently have familiarity with the

app event they’re providing feedback on

Cons: ➢ Hardest to integrate without third-party mobile

engagement solutions

Mapping Your Plan

What are you looking to learn?

What sort of data provides this insight?

What questions will provide the data you need?

Who is your audience?

How will you reach them?

How can this app be improved?

Mix of quantitative and qualitative

How would you rate the following features: [rating scale]

Do you have any suggestions for how we could improve this app? [open-ended response]

Returning users who have opened your app 3+ times

Prompt on next log-in

Best Practices: Mobile Research Design

When designing your mobile research instrument, do: ➢ Design with mobile in mind

➢ Keep questions brief and concise

➢ Make questions optional and allow customers to opt out at any time

➢ Aim to address your research objective with as few questions as possible

➢ Limit the number of options for multiple choices

➢ Break the questions up so that only one or two appear at a time

➢ Provide an ‘Other’ field with a textbox for fill-in answers to your multiple choice questions if you suspect that some respondents may have answers you had not previously considered

➢ Add an option for ‘Don’t Know’ or ‘Not Applicable’ for questions that some respondents may not be able to answer

➢ Pre-test your survey internally to identify any weaknesses and ambiguity

Best Practices: Mobile Research Design

And don’t: ➢ Create overlap in multiple choice responses. All responses should be

mutually exclusive

➢ Present rating scales with large matrices of options or questions ones that require scrolling on a mobile screen

➢ Create vague responses that are open to the interpretation of the respondent (i.e.: If asking about use frequency, give tangible options like ‘twice a week’ and ‘once a month’ rather than ‘often’ or ‘rarely’

➢ Frame questions in a way that leads the respondent or creates bias (i.e.: “Why do you like this app?”)

➢ Request personal information at the start of the survey as this may lead to lower response rates. If you need this information, make the questions optional and move them to the end of the survey

Best Practices: Survey Integration

For best results integrating your research instrument within your app, we recommend you:

➢ Use an in-app survey rather than directing mobile customers to a web survey so as to not detract from the customer experience. If you are using a web survey, be upfront about asking customers to leave the app for an external link.

➢ Use event-based targeting that isn’t intrusive. Don’t immediately ask new users to take a survey, and only ask customers to take your survey once rather than asking each time they load an event.

➢ Integrate the survey with your existing customer analytics to allow you to target the responses against your audience segmentation to uncover trends based on loyalty, device, etc. without having to ask customers to fill out additional questions.

Additional Resources

We hope this guide has helped kick start a conversation around why it’s important to do mobile research and how to design an effective survey instrument.

For additional tips, we recommend SurveyMonkey’s Guide to Smart Survey Design:

http://www.slideshare.net/ROhl/surveymonkey-smart-survey-design-guide

As well as these related posts on the Apptentive blog:

➢ The Importance of Mobile Feedback

➢ Why Your Mobile App Needs In-App Surveys

➢ 5 Tips to Writing Effective Surveys for Mobile Apps