PDMA Event - Creating Effective Surveys w/Self Service Tools - October 2014

Don’t Monkey Around With Surveys

Creating Effective Surveys Using Self-Service Tools

October 16, 2014

Mike Pritchard

Steve Boespflug

Discussion topics

• How you use self-service tools

• Proper planning prevents p$%^ poor performance

• Things to look for in a self-service tool

• Your questions – what we can cover today

• Questionnaire

• Sampling

• Testing

• Reporting

• Improvements

10/16/2014 © 2014, 5 Circles Research, all rights reserved 2

Tool How you use it

Use of surveys and tools

• http://5circles.pdma-oct-2014.sgizmo.com/s3/


http://5circles.pdma-oct-2014.sgizmo.com/s3/






What do you do


Planning and doing


Choosing a self-service survey tool Class of tool High-end

Confirmit, Qualtrics

Mid-range SurveyWriter, SurveyGizmo, Survey Analytics QuestionPro LimeSurvey

Low-end Add-on Constant Contact Swiftpage Google Docs


SurveyMonkey?

What features are important to you?

• Question types

• Mobile support

• Logic

• Integration

• Add enhancements yourself

• Online reporting

• Analysis inside tool

• Export

• Pricing model

• Branding

• Support

• ??

Your questions

• 1. What kind of research is a good fit for self-administered surveys and what kind of research is not?

• 2. How can I avoid bias in survey questions?

• 3. How large a sample size is needed to be statistically valid?

• 4. Can I ask open-ended questions or should all questions be yes/no, scale and multiple choice?

• 5. How much time should the survey be limited to in order to keep people occupied?

• 6. For scale questions, when is it best to use 1 to 10 versus 1 to 5?

• 7. What is the best way to pre-test a survey?


Questionnaire

• Questionnaire design is Art, Science, and Experience

• DON’T design in the survey tool

• Use a questionnaire document approved by the team to program from – Prioritize questions based on objectives

– Question types

– Logic to improve data quality and survey taker experience

– Flow is important

– Think about accessibility, use of different devices

– Minimize bias through wording and randomizing options

• Numeric reporting values (coded options) rather than text


Sample Sizes

• Sample size calculations assume representative samples. They rarely are.

• Consumer studies. Sample of 400 is a good minimum standard, yielding reliability of ±5% at a confidence level of 95%. Actual minimum is 384, so 400 allows for a few rejects.

• Business studies. Sample of 200 is a good minimum standard, yielding reliability of ±7% at a confidence level of 95%. Actual minimum is 196, so 200 allows for a couple of rejects.

• Larger sample sizes allow: – Increased ability to discern differences among subgroups

– Advanced analysis. Sample of at least 800 is good for segmentation

• Smaller universes don’t reduce sample size requirements as much as we’d like. Just as bad, we need to reach a high percentage of the audience.


Universe Sample for ±5% / 95%

Percentage of universe

Sample for ±5% / 95%

Percentage of universe

100,000 or more 384 N/A 196 NA

4,000 350 8.8% 187 4.7%

Acquiring sample


Testing the survey

1. Test with the document – Question text clear – Question options exclusive, exhaustive, “other” and “none” as

appropriate – Put yourself in the mind of the survey taker, but check with people from

the real target audience – Go through the logic

• Test programming – all paths and boundary conditions • Use the tool to enhance testing

– URL queries to show hidden values, change paths, – Capture testing comments inside the real survey

• Client / stakeholders check • Soft launch

– Limited number of responses – Test all the way to the data. Pause to review, including survey taker

comments

• Full launch


Reporting

• Online reporting for monitoring

• Online reporting for real reporting – if you can’t avoid it – How do you clean and filter data?

– Clients reacting too soon

– Inadequate tools for statistical testing, etc.

• Export for more capable tools – SPSS format, even if you don’t use SPSS

– Excel

– R

– Tableau


Improvements

• Ask survey takers to rate the survey and provide comments

• Review other issues you didn’t think of earlier, possibilities that came up.


Special Offer from SurveyGizmo

• Sign-Up for a SurveyGizmo Account and Receive a 60-day Free Trial 1. Go to www.SurveyGizmo.com

2. Click on the pink button "Try Building a Survey"

3. In the pop-up window enter your email address and password. In the field titled "What's the title of your survey" enter "5Circles". This will act as a promo code giving you the 60-day trial rather than the standard 14-day free trial.


Questions?

Mike Pritchard

425-968-3883

[email protected]

www.5circles.com