I. What is a survey?

I. What is a survey?

What is a survey?

A tool to collect information from or about people to describe, compare, or explain their knowlege, feelings, values, and behaviour .

Source: Fink, A. (2009), How to conduct surveys, SAGE.

What is a survey?

Mostly a quantitative research tool ….

• Questionnaires sent by post

• Online questionnaires

• Face-to-face structured interviews

• Telephone interviews

Doing a survey – main stepsDesign survey

process

Develop questions

Test questions/tool and train

interviewers

Collect data

Analyze data

II. Survey design

Target group• Who do I want to interview? (who should be in and who out)

• Do I want to interview the whole target population or its sample?

Tracer studyTarget population: graduates

Sample: all graduates (of selected fields) or their sample

Timing • When is the best time to contact your target population?

• How much time is needed to collect responses?

Tracer study

Timing: at least 6-9 months after graduation

Urban Rural Urban Rural Urban Rural1 month 6 months 12 months

12 717 13

1913

18

13

26

17

28

22

26 49

24

39

2339

3525 23 21 20 14

Other (military/parental leave/studying/disabled)UnemployedTraining/probationWorking for familyEmigrationUnregistered employeeEntrepreneurRegistered employee

Source: The ETF Transition Study in the Kyrgyz Republic 2011/2012

Timing (Repeated) Cross-sectional design

Panel design

- population of graduates asked only once

- population of graduates asked repeatedly at different time points

Method of data collection

Self-administered interviews (e.g. web-based, paper-pencil )

Face-to-face interviews

Telephone interviews

Face-to-face interviews

• High response rate

• Possibility to clarify ambiguous answers/provide follow-up information

- More time needed

- More resources needed (most expensive)

Telephone interviews

• High response rate, but lower than in case of face-to-face

- Possibility of sample bias (not everybody has a phone)

• Less time consuming (in comparison with face-to-face interviews

• Less expensive (in comparison with face-to-face interviews)

Self-administered interviews

• Saves money and time

- Lower response rate

- Possible sample bias (web-based interviews)

• People tend to be more truthful while responding

Doing a survey – testing

Once a questionnaire is ready, it must be tested in the field!

- Do people understand to the questions?

- Do people complete the questionnaire?

- Do they often drop-out? - How long does it take to fill the questionnaire? - Can interviewers administer the questionnaire properly?

- Are all procedures standardised?

Doing a survey – testing- Try to anticipate the actual circumstances in which the survey will be conducted

- Choose respondents similar to the ones who will eventually complete the survey

- Enlist as many people in the trial as reasonable, considering also your resources - Focus on the clarity of the questions and the general format of the survey

Doing a survey – things to consider

• Reliability

• Costs

• Usefulness and credibility of results

• Comparability

• Validity

Doing a survey – things to consider

• Reliability

A reliable survey provides consistent information (i.e. repeated measurement yields the same results).

…… make sure that each question means the same thing to everyone.

Doing a survey – things to consider

• Validity

A valid survey provides accurate information.

Do our questions measure what they are supposed to measure?

How were the respondents selected?

Can we generalise our results? …………

III. How to increase a response rate?

What is a response rate?

The proportion of people who return the questionnaire out of the totalnumber of people to whom a questionnaire was distributed.

Example: If you distribute 100 questionnaires and you get back 80 of them,your response rate is 80%.

Response rate matters!

• If we get responses only from a small share of respondents →

- results can be biased and not representative

- results are not significant and useful

What to do?

1. Study preparation

Think about study feasibility!

Think how to reach your respondents/graduates!

- Do you know their addresses? Or their parents’ addresses?

- Do you know their telephone numbers?

- Can you contact them via internet?

- Do you need to use a snowball technique?

What to do?

2. Designing a questionnaire

- Add a pre-notification letter before contacting the respondents

- Add a cover letter while distributing the questionnaires

Personalised requests and communication are important!

- Make sure that respondents see the value of the survey

What to do?

2. Designing a questionnaire

- Limit the number of questions to those which are relevant

- Add clear instructions

- Use clear and simple language

- Add ‘thank you’ at the end of the questionnaire

What to do?

3. Fieldwork

Reminders!

e.g. postal survey – send reminders after 2 and 5 weeks

e.g. face-to-face/telephone interviews – contact respondents at least 3 times at different hour/day

- Contact people several times- Time all contacts- Use different strategies

Follow up of respondents is an important part of the survey fieldwork and can influence the quality of your results!

IV. How to ask a good question?

Doing a survey – preparing a questionnaire• What information is needed?

• Can such information be obtained?

• Format of questions: open vs. closed, scales etc.

• Length of the questionnaire

• The order of questions

• Piloting

Questionnaire – basic rules

• Introduction – provide a context of the survey

• Assure anonymity

• Simple format

• Clean and consistent lay-out

Questionnaire – questions order

• Begin with questions reflecting the subject of the study

• Ask people to recall things in a chronological order

• Group items that are of a similar topic

• Personal and demographic questions should be placed at the end

Questionnaire – length

• Too long questionnaire – negative effect on a response rate

• Too short questionnaire – limited possibilities to respond our objectives

Be efficient and ask questions that are relevant and needed!

Doing a survey – format of questions

Closed questionsExample: How would you rate your overall career perspectives over the next years?1. Very good2. Good3. Poor 4. Very poor

Open questionsExample: What are your overall career perspectives over the next years?------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Doing a survey – format of questions

Scales

1 2 3 4

Students always like to go to the school

Students are interested in the field of study

Students are satisfied with teaching and learning services

Students are present during the entire school day

Attendance to the school is regular during the school year

Thinking about students' motivation, please rate the following statements:(Likert) 1=strongly agree, 2=agree, 3=disagree, 4=strongly disagree

What is a good question?

• Avoid double negative questions

• Avoid asking two things in one question

• A question should have a meaning

• A question should be concrete

• A standard language should be used

• Avoid biased words and phrases

What is a good question?

• How bad is your teacher?

• How would you rate the quality of your teachers and teaching materials?

• Would you say that the students are not motivated to study?

Not concrete and biased

Two questions in one

Double negative question

What is a good question?

• Please, specify the expenditure on ICT:

• How teachers and students evaluate the quality of food provided at school?

• Do you think that math teachers are better than those teaching languages?

Abbrevition used – not clear

Two questions in one

Biased question

Doing a survey – scalesNumber of response categories depends on the survey purposes (usually between 3 to 5).

Example: Do you agree or disagree with the statement that students always like to go to school?

1=agree 2=neither agree nor disagree

3=disagree

1=strongly agree 2=agree 3=disagree 4=strongly disagree

1=strongly agree 2=agree 3=neither agree nor disagree

4=disagree 5=strongly disagree

Doing a survey – scales

Number of response categories must be balanced.

Example: Do you agree or disagree with the statement that students always like to go to school?

1=strongly agree, 2=agree, 3=disagree, 4=strongly disagree

1=strongly agree, 2=agree, 3=disagree

Doing a survey – closed questions/scales

Number of response categories for each question must be exhaustive.

Example: What is your current marital status?

1. Never married 2. Married / Living together

3. Divorced / Separated 3. Other

4. Widowed

Doing a survey – closed questions/scales

Always specify whether one answer or multiple answers are possible.

Example: What is your current marital status?

1. Never married 2. Married / Living together3. Other

Example: Why did you leave school before graduation?

1. Studies too difficult 2. I wanted to work 3. I needed to work4. I got married5. Other

References

A large part of the presentation was based on - Fink, A. (2009), How to conduct surveys, SAGE.