QUESIONER DESIGN AND ANALYISIS

Why this topic is important

Questionnaire-based surveys are one of the most common tools used by market researchers to establish consumer preferences. Bad questionnaires are misleading and likely to yield meaningless data, so an awareness of the techniques of questionnaire design is essential to any student or researcher wanting to establish opinions on their subject specialism. In addition, you will find that a sound awareness of the principles of questionnaire design will allow you to look more critically at other people's research and to begin to question the methods and tools of analysis that they use.

The Stages of Survey Design

The following stages take a questionnaire-based survey approach as an example. This is not because we insist (or even recommend) that you should always be adopting this type of approach, but rather because we take you through the principles of questionnaire design at this point in the course. When using a different approach (e.g. a qualitative one), simply substitute section 6-9 with alternatives relevant to the methods you are using.

  1. SELECTING A TOPIC - Scale of topic should be manageable - not too wide 2. FORMULATING YOUR HYPOTHESIS - See later notes 3. LITERATURE SEARCH - What have other people written about this topic? 4. DISCUSSION WITH "INFORMANTS AND INTERESTED PARTIES" - Are there any

people that you will need to speak to clarify issues surrounding this topic? 5. SAMPLING - Selecting the people to be approached (See "Sampling Workbook") 6. QUESTIONNAIRE DESIGN - Translating the broad objectives of the study into questions

that will obtain the necessary information. 7. FIELDWORK - Collection of data through questionnaire or interview 8. DATA PROCESSING - Coding and inputting the responses 9. STATISTICAL ANALYSIS 10. ASSEMBLY OF RESULTS 11. WRITING UP THE RESULTS - Drawing conclusions / interpretations and relating the

findings to other research. You will have been given separate notes on report writing.

Hypotheses & Variables A hypothesis can be described as "a tentative answer to a research question" or

a "provisional prediction". The plural is hypotheses. Hypotheses should be: Stated clearly, using appropriate terminology Testable Statement of relationships between variables Limited in scope Examples of hypotheses: Health education programmes influence the number of people who smoke Newspapers affect people's voting patterns Attendance at lectures influences exam marks Diet influences intelligence In the above examples, "something" (e.g. diet, lecture attendance) affects

"something else" (e.g. intelligence, exam marks). These are variables. A variable is anything which is free to vary, and in order to describe them quantitatively, they have to be expressed in appropriate units (e.g. IQ scores, exam percentages)

The pairs of variables in the above examples have separate names. The variable we manipulate is called the independent variable (IV). The variable we are hypothesising will alter as a result of our manipulations is called the dependent variable (DV). The dependent variable alters as a consequence of the value of the independent variable - its value is dependent on this. The value of the independent variable is free to vary according to the whims of the experimenters.

Independent variable Dependent variable Health education programmes Number of people who smoke

Newspaper Voting patterns Attendance at lectures Exam marks Diet Intelligence

NB Many variables can be either dependent or independent, within the context of a particular study. For example, it could be argued that "Intelligence influences diet" or "Exam marks influence attendance at lectures".

Directional hypotheses

In the examples above, words like "influence" or "affects" are used without indicating direction. When a hypothesis states a predicted outcome (using words such as reduce, increase, lower, raise - it is called a directional or one-tailed hypothesis. Vaguer types of hypotheses (such as the ones given earlier) are known as non-directional or two-tailed hypotheses.

Question Types Two main categories of question:

Closed-Ended & Open-Ended Advantages and Disadvantages of Closed an

d Open-Ended Questions

Closed-ended questions

Name: Dichotomous Description: Question offering two choices Example: Did you watch television at all

yesterday? Yes / No

Name: Multiple Description: Question offering three or more

choices Example: Which of these shops do you

prefer? Next / River Island / Gap Top Shop/ Top Man

Name: Likert scale Description: Statement with which

respondent shows the amount of agreement / disagreement

Example: Assessment by course-work is easier than assessment by examination Strongly agreeAgreeNeither agree nor disagreeDisagree Strongly disagree

Name: Semantic differential Description: Scale is inscribed between two bipolar words and

respondent selects the point that most represents the direction and intensity of his / her feelings

Example: The degree I am taking is............. Interesting :_____:_____:_____:_____:_____:_____:_____:

Boring Useful :_____:_____:_____:_____:_____:_____:_____:

Useless Easy :_____:_____:_____:_____:_____:_____:_____: Difficult

Name: Rank order Description: Respondent is asked to rate or rank each option

that applies. This allows the researcher to obtain information on relative preferences, importance etc. Long lists should be avoided (respondents generally find it difficult to rank more than 5 items)

Example: Please indicate, in rank order, your preferred chocolate bar, putting 1 next to your favourite through to 5 for your least favourite.

Double Decker Crunchie Wispa Mars Bar Creme Egg

Name: Numeric Description: Respondent specifies a

particular value (can include decimal places) Example: How far (to the nearest kilometre)

did you travel today to reach this supermarket?

________km

Open-ended questions

Name: Unstructured

Description: Question that respondents can answer in an unlimited number of ways?

Example: Why did you enrol for this course at QMC / SCOT?

………………………………………………………………………………

………………………………………………………………………………

………………………………………………………………………………

Name: Word Association

Description: Words are presented one at a time and respondents give the first word that comes to mind

Example: What is the first thing that comes to mind when you hear the following?

Lecture Interesting Computer Exciting Exam Challenge Tutorial Rewarding

Name: Sentence completion Description: Incomplete sentences are presented, one at a time, and respondents are asked to complete the sentence Example: My worst shopping experience while visiting Tescburys happened when……………….

Name: Story completion Description: An incomplete story is presented and respondents asked to complete it Example: I sat down at the kitchen table, picked up a fork, then looked at the Chicken and Mushroom flavour Pot Noodles in front of me…….

NOW COMPLETE THE STORY

Advantages of Closed-Ended Questions Quick to answer Easy to code No difference between articulate and inarticulate

respondents

Disadvantages of Closed-Ended Questions Can draw misleading conclusions because of limited

range of options Researcher / interviewer cannot deal with qualifications

to responses e.g. "Yes, but….." or "It depends" where only Yes/No are given as options

Advantages of Open-Ended Questions Greater freedom of expression No bias due to limited response ranges Respondent can qualify their answers

Disadvantages of Open-Ended Questions Time consuming to code Researcher / interviewer may misinterpret

(and therefore misclassify) a response

Instructions in Questionnaires General instructions

Indicate that you are a student at Queen Margaret College / Scottish College of Textiles and that this work is being undertaken as part of your course. You should also have a letter from your tutor or supervisor to authenticate this.

You should address issues of confidentiality and/or anonymity. You could, for example, include a sentence at the beginning such as "All of the information you give me will be treated as completely confidential and it will not be possible for anyone to identify the information you give me when I write up the project report".

(If applicable) indicate how the person was selected to receive the questionnaire.

Indicate how it is to be answered. For example….. "Please answer all of the questions which apply, and leave the remainder blank"

Return questions (if not being delivered in person)

Question instructions Ensure that each question (or block of similar questions) has a clear instruction on how to respond.

Indicate the form of the answer (numeric, tick-box, rank etc.) and how many answers are expected, such as "most relevant", "one only" or "all which apply"

Routing instructions Where respondents' answers to an earlier question affects subsequent sets of questions, ensure that the route

which they should take is clearly specified. For example, "If YES, please go to Question 15"

Question Order Your respondents may refuse to co-operate if your survey begins with

awkward or embarrassing questions. People are more likely to give honest replies to personal questions if

some rapport has been developed with the interviewer. For the above reasons, it is generally best to keep all questions dealing

with demographic information (such as age) at the end of the questionnaire.

The Layout of the Questionnaire Print clearly (if computer printout is giving faint printouts, ask the IT staff

whether a new print cartridge can be installed) Allow adequate space between questions so that you can write down

any comments made (but don't waste too much paper!!) Write the questions themselves in lower case (i.e. like this writing) ,

INSTRUCTIONS IN UPPER CASE (i.e. capital letters)

Dealing with Problem Questions

Hypothetical questions

A hypothetical question is one in which you are asking respondents to indicate what they think they would do under particular imaginary circumstances. These can't always be avoided in some attitudinal research, but they are difficult to administer and often give rise to unreliable answers

Activity

Presuming / leading questions

These are often included in poor questionnaires because the researcher feels strongly about a topic and assumes that everyone will be of the same opinion.

Activity

Questions which rely on memory

Problems which tax the respondent's memory too much are likely to lead to non-response or inaccurate replies. For example "What did you have for lunch each day last week?"

Questions requiring prior knowledge

For example, "What is your National Insurance number?"

Sensitive questions

Personal details / health / age

Income

If you have to ask sensitive questions, the problem can be alleviated somewhat by the use of SHOW CARDS. Put all of the possible responses on a card, preferably mixed up, and ask the respondent to indicate which number relates to their own circumstances. For example,

Can you tell me the number on this card which corresponds to you income group?

SHOW CARD WITH……

£7,000 - 12,000

Over £60,000

£18,000 - £30,000

Under £7,000

£40,000 - £60,000

£12,000 - £18,000

£30,000 - £40,000

Mutually exclusive responses

In the show card above, you will note that somebody earning exactly £30,000 would perhaps wonder whether to give answer 3 or answer 7 on the show card. In practice, people are usually able to give their income as an approximation. You should, however, always watch out for questions where the multiple choice answers are not mutually exclusive and where a respondent will be uncertain about which category he/she falls under. It seems to be a particular problem with age brackets, and you can often see examples of mistakes here in even professionally produced surveys.

Long questions

If your questions are too long and detailed, the respondent may get lost and the responses will relate only to the beginning or the end of the question. Where definitions and qualifications are necessary, use show cards.

Checklist for Questionnaires

Before you "pilot" your questionnaire (see later section of this booklet), try going through the following checklist to spot whether any of these common mistakes apply to your own questionnaire:

1. Have you avoided all leading questions?

Make sure you haven't included phrases like "Wouldn't you say that….." or "Don't you agree that…….."

2. Is the question as specific as possible?

Avoid using words like "occasionally", "regularly", "often", "in this area". If you were to ask respondents how often they visit the cinema, for example, one person's idea of "regularly" may be every couple of months, which could be another respondent's idea of "occasionally" and another person's notion of "rarely". Far better to give explicit categories such as "More than once a week" "Every week" "More than once a month" etc.

3. Are the questions going to be understood by all respondents?

Avoid the two extremes of vocabulary (a) technical jargon; (b) slang or colloquialisms

4. Is each question applicable to all respondents?

If not, you will need a "filtering" question first

5. Are any of your questions double-barrelled?

For example:

"Are your lectures and tutorials enjoyable and easy to understand?" Yes / No

You may, of course, find the lectures in a subject easy but the tutorials more difficult, or you may find both easy to understand but do not enjoy them.

As a bit of light relief, here is a summary of how one author suggested we deal with sensitive questions: BARTON, J A (1958) Asking the embarrassing question. Public Opinion Quarterly 22 pp.67-8 He takes as his example the delicate question of whether a respondent has murdered his wife! 1. CASUAL APPROACH "Do you happen to have murdered your wife?" 2. NUMBERED CARD APPROACH "Will you please read off the number of this card which corresponds to what became of your wife?" 3. THE 'EVERYBODY' APPROACH "As you know, many people have been killing their wives these days. Do you happen to have killed yours?" 4. THE 'OTHER PEOPLE' APPROACH "Do you know any people who have murdered their wives?" PAUSE FOR REPLY ""How about yourself?"

PILOTING YOUR QUESTIONNAIRE

Before you deliver any questionnaire, you should "pilot" it (i.e. test it) to check that it is going to function effectively. There are a number of reasons why it is important to pilot a questionnaire:

To test how long it takes to complete

To check that the questions are not ambiguous

To check that the instructions are clear

To allow you to eliminate questions that do not yield usable data

Ideally it should be piloted on a group similar to the one that will form the population of your study. It is difficult to give an exact number for the pilot group, but as a rule of thumb, try to pilot on about 5-10% of your final sample number. The results from the pilot study, however, should not be included with your final results.

If respondents omit certain questions, you should be able to find out why.

Ask your "guinea pigs" the following questions:

How long did it take to complete?

Were the instructions clear?

Were any questions unclear or ambiguous?

Did you object to answering any questions?

Was the layout clear and attractive?

Any other comments?

You should then alter your questionnaire to take account of comments made in the pilot study.

Analysis of Survey Data

Influenced by: 1. The number of variables being examined

Univariate analysis = one variable (e.g. gender, age) Methods: charts (e.g. bar chart, pie chart) or frequency table

Bivariate = 2 variables (e.g. gender + purchase of computers) Methods: crosstabulations; scatterplots; regression; comparison of means (but we will only be covering the first two in this module)

Multivariate = 3 or more variables (e.g. income influenced by both education level and gender) - Note: you'll be relieved to hear that you won't be covering any of these as part of this module)

2. Levels of measurement Nominal

Ordinal

Interval

Ratio

3. Descriptive or inferential? Descriptive statistics: summarise patterns in responses (e.g. average age of respondents or number of

respondents who buy a product)

Inferential statistics: provide an idea about whether the patterns described in the sample are likely to apply in the population from which the sample is drawn ("tests of significance")

Frequency tables

Should have separate columns for:

(a) number of people who gave each answer (N) (b) percentage column

If (a) is excluded from the table, it is essential to include the total number of cases on whom percentages are calculated.

Example

Analysis of degree of agreement/disagreement with statement "Taking part in videoconferenced classes is a really exciting experience"

Answer N %

Strongly agree 254 34

Agree 201 27

Neutral 119 16

Disagree 97 13

Strongly Disagree 75 10

Total 746 100

Crosstabulation

Crosstabulation = tabular representation of relationship between two (or can be more than two) variables

Also called CONTINGENCY TABLE

Table divided into cells with each cell representing the coincidence of a specific value from each variable

The total for each row and each column is given at the end of the row and column. These totals are called the ROW MARGINALS and COLUMN MARGINALS.

It is normal (but not absolutely essential) for the dependent variable to appear in rows. This is because we may be crosstabulating one dependent variable by a series of possible explanatory variables.