Upload
shiri
View
43
Download
0
Tags:
Embed Size (px)
DESCRIPTION
The Annual Structural Business Survey Developing and Testing an Electronic Form. Ger Snijkers Evrim Onat Rachel Visschers Statistics Netherlands Division of Business Statistics. 1.Dutch Annual Structural Business Survey. Annual survey of economic activity Mandatory - PowerPoint PPT Presentation
Citation preview
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 1
The Annual Structural Business SurveyDeveloping and Testing
an Electronic Form
Ger SnijkersEvrim Onat
Rachel Visschers
Statistics NetherlandsDivision of Business Statistics
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 2
1. Dutch Annual StructuralBusiness Survey
Annual survey of economic activity Mandatory 75.000 business each year:
• Sample of small firms, bigger firms each year Until now: paper questionnaire
• length may differ, 20 pages is typical Three parts:
• revenues and costs• summary of business accounts:
profits and losses• industry specific specifications
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 3
2. The paper questionnaireCharacteristics
A4 booklet• Right page: items• Left page: help texts - long and voluminous
Items are grouped in sections• Long sections
• Completion process: complicated and hard• Large amount of detailed information• Broad range of business information:
several departments, several respondents• No match with definitions used, but same label:
CBS – business definitions• Lay-out: misinterpratations and errors
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 4
Vernieuwde huisstijl:2. Onderzoek naar de PS
Q oud
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 5
3. The web questionnaireProject goal
Develop a web questionnaire• same contents• mixed-mode design: paper and web• support completion process:
- motivate respondents to use this mode• into the field: March 2006
Start of project: June 2004
In the field: Spring 2006
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 6
3. The web questionnaire Developing and testing
In five stages:
1. Testing the prototype (31-1-’05)• pre-tests to test usability: 3 waves
2. Revision of questionnaire (1-9-’05)• expert reviews
3. Testing of revised questionnaire (1-1-’06)• additional usability tests
4. Implementation of field pilot (1-3-’06)
5. Implementation of survey (1-3-’07)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 7
3.1. The web Q: The prototype
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 8
3.1. The prototype: 3 test waves
Test 1 Test 2 Test 3
Period Aug. 2004 Oct. 2004 Nov./Dec. 2004
Form Blaise IS Blaise -EDR
Blaise -CBSquest
On/off-line On-line Off-line Off-line
Mode Browser CD-rom Downloadablevia internet
Number of interviews
15 CBS employees
37 businesses
6 businesses
Tested by CBS employees,external designer
6 business interviewers
2 cogn. lab interviewer + 2 business interviewer
Results Experiences from CBS employees
Experiences from interviewers
Experiences with respondents
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 9
3.1. The prototypeResearch issues
1. How does the e-form work in practice?• Completing the questionnaire• Question-and-answer process
2. What features should be included to make it easy to use?• Respondent friendly: ‘Computer-assisted’ tools• User wishes
3. How should the web Q be designed in relation to the paper Q?• The same or a different design• ‘look-and-feel’ of paper and e-form
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 10
3.1. The prototypeResearch issues
1. How does the Q work?
Laborious and complex process• Long, complex questionnaire (≥ 25 items)• Complex completion process:
- several sessions, several informants- kick-and-rush behaviour
• Imagine ... a respondent sitting behind his/her computer ...
Respondent got lost in the questionnaire
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 11
2. Features to make it easy to use?• What am I supposed to do (next)?
• Easy to download, install, complete, send data back• It is one process: downloading – sending data back, • Clear instructions and explanations (but not read)
• How is the questionnaire built up? • Show how the questionnaire is structured: overview• Help to find the way in the questionnaire• No hidden rules, no unexpected functionalities
• Where am I? What did I do so far?• Provide overview of the completion process• Clear navigation, no scrolling• Printing function
3.1. The prototypeResearch issues
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 12
2. Features to make it easy to use? Con’d• Printing
• Passing on sections of Q to other departments• Checking the data before transmitting• Getting authorisation to release the data
• Calculations• Well accepted, … even expected
• Entry-search• ‘Google’-like search: “Where to put these data?”
• Navigation and overview• Choose a setup people are familiar with:
- setup of tax office, windows explorer
3.1. The prototypeResearch issues
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 13
3. Design of paper and web Q?• The computer is different than paper
• The web Q reacts to the respondent• Reading from the screen is different• Navigating and getting an overview
works differently• Kick-and-rush behaviour,
even stronger than on paper• The use of computer-assisted functionalities;
the respondent expects the computer to react
3.1. The prototypeResearch issues
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 14
3.1. The prototypeConclusions of pre-test waves
• Visual design Clear and logical Simple, transparent, consistent No hidden and unexpected functionalities
• Support the completion process Other mode, other features, other visual design
Different than paper form, same ‘look-and-feel’
• Tailor to kick-and-rush behaviour Small sections, small tasks Short and clear explanations
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 15
3.2. The revised questionnaire
Based on:• Pre-test results• Expert reviews
• Iterative process with- Professional designer- Questionnaire designers- Methodologists
• A user friendly design was put first, not the IT tool
• New prototypes designed in Power Point
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 16
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 17
3.3. The revised questionnaireConclusions of add. pre-tests
• 10 concurrent in-depth interviews• Usability and user friendliness has been
improved respondents enjoyed working with the
questionnaire they could handle the task even though … the task had not changed
Web questionnaire design is communication design
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 18
3.4. Field pilot
Set-up:
March-July 2006 7200 businesses, 5 industries Advance letter with
• internet address: www.cbs.nl/productiestatistiek• user name and password• leaflet to introduce web questionnaire
and explain why the survey is conducted• paper questionnaire not mentioned
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 19
Leaflet
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 20
3.4. Field pilot
Goals: Implementation of web questionnaire
• paper and web flows
Test the whole process• downloading – completing – sending-in data• investigate completion process in the field:
- debriefing interviews with respondents- audit trails
Response rates Data quality
• data editing, mode effects
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 21
3.5. Survey
Now running About 75.000 business received this
questionnaire
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 22
Web questionnaire design is communication design