Upload
pearl-stokes
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Questionnaires
At least two points in the development cycle:• Collecting demographic information when
conceptualising a site• Testing the usability of a live site.
• Try and pilot a survey before use – there will be ambiguities you hadn't noticed.
Questionnaires
• Questionnaires = Surveys are not as easy as they look. Any fool can string together a set of questions. It is very difficult to construct an unambiguous survey whose results can be used in a meaningful way.
• For fewer than 10 people, do an interview. It is a better and more fruitful use of resources.
• A printed questionnaire can't be changed. You've got to be certain what you're looking for.
Questionnaires
• There are ready made and pre-tested professional questionnaires for the usability of a working web site:– QUIS = Questionnaire for User Interface
Satisfaction (University of Maryland)– SUMI = Software Usability Measurement
Inventory (University of Cork)
• You can get a copy of SUMI free, but you need to buy a license to use both.
What to ask about
• Surveys are good for collecting basic, clear-cut information.
• Focus on questions that guide design decisions
• For each question:– Why do I need to know this?– How will I process the information?– What will I do with the information? If you don't
know the answer, delete the question
What to ask about
• Easy questions: Gender, age, profession, education, computer skill, type of computer, nationality
• These demographic questions check that you have sampled your target population.
• Next ask about skills, experience & lifestyle
What to ask about
• Needs and preferences:• What kinds of product would you like to buy
online?• What problems do people have with web
sites:– Which of these issues is the worst aspect of
browsing the web? Download speed – browser incompatibility – getting lost.
What to ask about
• Don't depend on stereotypes:– If you think men prefer black backgrounds, find
out by asking gender and preference of background colour.
Wake up!!
• You want to find out about what Napier students think of the student portal
• In threes, brainstorm ideas on what they might think about it and then convert these into topics.
• Eg: One might think that the layout is boring. So one of your topics might be the attractiveness of the page.
• (plenary)
Structuring Responses
• Responses come in three flavours: Checkbox, multiple choice and free.
• It's a good idea to vary the type of question so that responders don't get stuck on clicking the middle option all the way down the survey.
Free response
• Asks an open question and lets responders give any reply they want.
• Takes more effort from the responder, so these sections are often left blank.
• More difficult to collate answers because they are in different shapes and sizes.
• There is a restricted form that asks for e.g. quantities
Free response
• Good to put at the end of the survey to collect information the responder thought important, but which didn't fit the survey questions.
• Free responses are a cheap way to get new ideas. These ideas can be checked later in other data collecting methods.
Checkboxes, Checklists
• Keep writing to a minimum• Allow responder to answer many questions
quickly.• Can ask:– List products that user owns– List products the user would like– List problems in doing their jobs– List features they'd like on the site
Checkboxes, Checklists
• Problem: People will skip reading long lists or miss options
• Can replace a long checklist with yes/no radio buttons, but this is more work to answer.
• NB: have the default radio button position "nothing selected". If you have the "yes" button selected by default and the responder skips the question, you will get the wrong answer.
Multiple Choice
• Allows you to restrict the responses to easily understood categories.
• You may not have listed all possibilities, so you could have a box "other" and a space to write in what that is.
• Multiple choice and checkboxes are very easy to process
Multiple Choice
• User might miss the question by accident, so make sure the response defaults to "no response".
• Likert Scales collect feelings about a single parameter, eg:I very much prefer the new branding:strongly disagree 1 2 3 4 5 6 7 strongly agree
Interpreting Responses
• Document the most common response• Document the breadth of responses to a
question, not just the most popular answer• Count the total number of responses to a
checked item• A low response to a question may mean that it
is unclear.
Sampling
• A small number of responses is better than none at all
• Fewer than ten returns won't tell you much• Fifty or so returns can be relied on• If you're doing scientific research, you may
need as many as 500 if you want the results to be trustworthy.
Sampling
• Online surveys get a response rate of about 1 – 2%
• Email and snail mail surveys get about 5 – 10% returns. For 50 returns you'd have to send out 1000 mailshots.
• If you are working with small organisations through personal contact, you may get up to 100% returned.
Sampling
Improve your rate of return:• Offer a small gift to those who return the
survey• Include a gift with the survey• Use unusual paper and envelopes• Keep the survey short and say how long it
takes to fill in• Include a SAE
Sampling
• Include a letter of introduction from someone important
• Emphasise that responses will be kept confidential
• Emphasise that their responses will shape the web site
• Give them a deadline for responses• Query those who didn't respond, asking them
to return their forms please.
Sampling
Selecting survey recipients:• If there is a small number of specific users (as
in an intranet), everybody gets a copy and response rates are usually high.
• Trickier if you're targeting a mass market.– Advertise the survey on your web site– If there is a mailing list or newsgroup, use that– Hand out surveys on a street corner– Visit an industry convention
Sampling
• Snowballing effect: ask each respondent to suggest another potential recipient. In email surveys, ask respondent to forward the email to a friend.
• Try and send covering letters so it doesn't look like junk mail.
Sampling
Self Selection:• Some will choose to reply to your survey,
others will not.• People who are motivated to provide
feedback may have different user behaviour to those who aren't.
• Eg: Dissatisfied people are less motivated to help you. You won't find out why they are dissatisfied.
Sampling
• There will always be self-selection, biassing your results, but– It can be minimised– It isn't a reason not to survey.
• Simply state the problems in your report on the results.
Wake up!!!
1. Let us, as a class, decide on a web site that we all use.
2. Let's do a questionnaire on it.3. Then, as a class, add up the answers.
Avoiding Bias
• Respondents will try to read behind the wording of the question to give you the answer you want them to give. These answers may not be truthful
• The answer is to pre-test (pilot) the questionnaire. This will find questions that are misleading, ambiguous, insulting, or inviting bias. It will find questions that are always skipped or always given the same answer.
Avoiding Bias
Question Skipping:• People skip questions because they are:– Difficult to understand– Not viewed as relevant– Difficult to answer– Part of a long boring questionnaire
• Keep the questionnaire short• Ask respondents to answer every question
Avoiding Bias
Response order for checkboxes or radio buttons:• People often choose the first or the last
alternative in the list. One solution is to use a different order for every respondent.
• Put response options in alphabetical order or deliberately scramble them - otherwise respondents might try to second-guess the “right” answer.
Avoiding Bias
Rote answers:• Sometimes respondents fall into a pattern of
ticking the middle option all the way down the line. To avoid this, keep switching question type from multiple choice to free response to checklist
Avoiding Bias
Negative questions (eg, which of these are you LEAST likely to buy on the web):
• Respondents often miss negatives and respond as though they are positives.
• Try not to use negatives. If you must, emphasise the negating word (LEAST)
Avoiding Bias
Leading Questions:• Are the Iraqi home team terrorists, resistance,
insurgents or freedom fighters?• Sometimes the language of the question tells
the respondent your personal opinions.• Can you give an illustrative example?
Avoiding Bias
Range bias:• How often do you use the internet? 15 hrs a
day or more / 10 - 15 hours a day / 5-10 hours a day / less than 5 hours a day
• How often do you use the internet? At least once per day / 1 - 5 times per week / 1-5 times per month / less than once per month
• Q1 will give a more frequent use than Q2.• Consider logarithmic range increases.
Avoiding Bias
Range bias:• Pre-test to make sure you have an appropriate
range of responses• Consider a blank for writing in the answer.
When to Use Surveys
• Inexpensive • Large amounts of data• Good sample size • Reliable demographics• Best used: – Before a project starts, to identify demographics– When the site is live, to get user feedback.
Steps in implementing a survey
1. Create the survey on paper.2. Pre-test the survey questions.3. Turn the paper survey into an electronic
survey (email/web).4. Do usability testing on the electronic survey.5. Inform the target population about the
survey.
Bibliography
• Brinck, T., Gergle, D., and Wood, S. T. (2002) Usability for the Web: Designing Web Sites that Work, Morgan Kaufmann, San Francisco, USA
• Lazar, J., (2001) User Centred Web Development, Jones and Bartlett, Sudbury, USA
• Benyon, D., Turner, P., and Turner, S. (2005) Designing Interactive Systems, Addison Wesley, Harlow, UK.