Location, scope and amount of definitions and instructions defining the quality of survey response

Quest Workshop 24. - 26.4.2007 Ottawa

24.4.2007 2Petri Godenhjelm

How to communicate definitions and instructions

A lot of supporting communication is often needed in addition to a communication included in a question

Meaning of concepts is communicated not only in questions, but in definitions and instructions as well

How definitions and instructions are used and placed have important implications to the comparability of responses and survey quality

24.4.2007 3Petri Godenhjelm

Projects where obervations were done

Recreational fishing, household surveycognitive interviews in laboratory

International sourcingcognitive interviews in business place

Two agriculture surveyssurvey on the structure of agricultural holdingsfield crop surveycognitive interviews in farms

24.4.2007 4Petri Godenhjelm

Designer’s dilemmaC

lear

ness

of

a qu

esti

on t

o a

resp

onde

nt /

Pro

bab

ilit

y of

rea

din

g a

ques

tion

/P

rob

abil

ity

of a

nsw

erin

g

Amount of the words in aquestion and in a definition

24.4.2007 5Petri Godenhjelm

Counting fishing days

Question: Mark down on the table the amount of your and your household member’s combined fishing days in 2005 across fishing area and licence type?

Separate detailed defintion leaflet Answers in test interviews were mainly given on person who answered,

other members’ fishing days not included

Solution: main points from leaflet straight to the question Important definition ”If for example two persons of your household

fish, mark it as two fishing day” New placement and use of definitions produced more valid and

reliable data

24.4.2007 6Petri Godenhjelm

International sourcing

Understanding the concept international outsourcing was essential Definition was needed Testing found the complexity of the phenomenon Companies in enterprise groups were in uncertainty what the concept

meant; did the internal company activities belong to the scope of study

Solution a detailed compact definition was designed and placed properly focus to enterprise relocation (internal and external)

24.4.2007 7Petri Godenhjelm

Agricultural surveys - from EU-definitions to a definition in a questionnaire

Other gainful activities in EU document : ”information is collected on whether the holder and/or the spouse or

other family members or one or more partners on a group holding carry out any gainful activities that do not comprise any farm work as defined under L/1 to L6 but which are directly related to the holding and have an economic impact on the holding”

”several such activities can be carried out on the same holding. These should all be recorded”

”Non-separable gainful activities on the holding are excluded” ”Forestry activities are also excluded”

24.4.2007 8Petri Godenhjelm

Questionnaire definition and how to answer

Other gainful activities were quite easily answered by farmers But, to give an answer whether this activity has any connection to farm

work raised interesting points in test interviews Firstly lay-out problems were due to ignoring the whole question by

some farmers Secondly connection to farm work were too abstarct concept

Thirdly respondets talked about how the question is related to the tax system and tax forms

Decision to split the definition to separate questions It really took much more than words to write a question Iterative test process were essential

24.4.2007 9Petri Godenhjelm

Visual self-administration

Don Dillman’s new book and it’s 2007 update appendix were stimulating with relation to my recent testing projects

What kind of advise should be given to surveyors and clients who are going to use mixed-mode designs in future?

Unimode or mode-spesific design? Visual design and usability? What kind of implications visual design concepts bring to

questionnaire testing strategies? Fifth step, perceiving and attending, in question-answer

process: need for taking over new concepts