The Art and Science of Survey Research

What is a survey?

Surveys are a bit like porn – hard to define, but you know one when you see one!

But if you want to be all academic about it…

A survey is “a systematic method for collecting information from or about people to describe,

compare or explain – in a quantitative way -their knowledge, attitudes, experiences, and

behaviours”-adapted from Fink, 2003 & Groves et al., 2004

This presentation will focus on self-administered, cross-sectional surveys, analysed in SPSS

You should use a survey when you want to…

Ask about knowledge, attitudes, behaviours or experiences (“stuff”) on a larger scale

Ask about stuff in a standardized way

Quantify stuff

Focus on self-report

Allow participants to be anonymous

Once You’ve Decided on a Survey, You’ll Need to Consider…

The Science(what you just have to do)

&

The Art

(what you’ll do if you’re passionate about good research)

The Science

Step One

Identify the objectives of your study

Identify the rate of suicidal ideation in family carers of people with dementia

Identify the factors associated with suicide risk and resilience in this

group

Explore differences in suicidal ideation (& risk/resilience factors) between carers with different demographic

profiles

Here are the objectives from one of my surveys…

Step Two

Define the key concepts in your objectives

SUICIDAL IDEATION: having considered killing yourself or having attempted suicide

FAMILY CARERS: the primary carer for a family member with dementia who is at home, in care, or

has recently passed away

Here are the key concepts from my objectives…

RISK: factors identified in the literature e.g. depression, hopelessness,

RESILIENCE: factors identified in the literature e.g. physical health, optimism, social support

DEMOGRAPHIC PROFILES: age, gender, relationship to PWD, urban/rural, ethnicity, sexuality

Step Three: Select Questions

Demographic questionsBest adopted from large bodies (ABS, AIHW, CDC etc) or previous studies in your field

This allows you to compare the demographics of your sample with population level data or previous research

Some people put demographic questions at the end of a survey. Always put them at the beginning! That way, if people don’t finish you can at least compare the demographic data of completers vs

non-completers.

A Note on ScalesIn most surveys, the main questions will take the form of scales.

A scale is a group of questions (usually referred to as items) that address the same topic and are added together to give a single score that reflects the construct/concept of interest.

Scales are useful when multiple factors contribute to an overall concept & when you are unlikely to get a reliable response from a single question (e.g. Are you depressed?)

A Note on Scales Cont.In most surveys, you’ll have one scale for each concept/outcome of interest.

In my research I had one scale each for: depression, anxiety, suicidal ideation, physical health, social support etc

Your choice of scales should be informed by your literature review.

Scale creation is a science in itself, so unless there is good reason, steer clear of creating your own.

Step Three Cont.

For your key concepts, choose scales that:

Are informed by your literature review

Are reliable

Have been validated

Are appropriate for your population

Have clear scoring instructions

Are publicly available (or within your budget)

A Note on Reliability

Test-retestConsistency over time, or the extent to which a question or scale will produce the same result for the same person if repeated under the same conditions. Anything over 0.8 is good.

Internal consistencyHow well the items of a scale go together or can be considered to reflect the same construct/concept. Anything over 0.6 is acceptable and over 0.8 is good. But over 0.95 may indicate too much overlap or redundant items.

A Note on ValidityContent validityThe extent to which a scale represents all the facets of a given concept.

Face validityThe extent to which a scale looks like it’s measuring what you tell participants it’s measuring.

Convergent validityThe extent to which scores on this measure align with scores on measures of related concepts

Divergent (or discriminant) validityThe extent to which scores on this measure differ from scores on measures of unrelated or opposing concepts

Objective:

Key term:

Measure/Scale 1:Source/Reference:

Reliability:

Validity:

Scoring:

Previous Use:

Availability:

Measure/Scale 2:Source/Reference:

Reliability:

Validity:

Scoring:

Previous Use:

Availability:

Summarising Steps One to Three

Create a document that looks like this…

If you can’t fill in all the blanks, you either don’t have a clear idea of what you want to measure or the scales you have selected are not appropriate.

Step Four: Pilot testing

Gather together a small group of people & ask them to complete the survey as if they were a participant. They should pay attention to the following…

Are the instructions clear?Do the questions make sense?Are the response options sufficient/appropriate?Are skips easy to follow?How long did it take to complete?Do they understand how to complete/save/return the survey?

You should revise your survey accordingly, but do not change scales that have already been validated.

Step Five: Sampling

A good sample is a miniature version of the population you are researching.

Inclusion criteria: Who do you want?

Exclusion criteria: Who don’t you want?

Excluded:

• Paid carers

• Non-primary family carers

• Caring for someone with a condition other than dementia

• Person with dementia deceased more than 2 years ago

• Unable to read and write English

Here are the exclusion criteria from my survey…

Sampling Cont.

How many do you need?This should be informed by the existing literature and an understanding of the impact of power on your analyses

How many will you have to approach?This is always more than the number you need

How long is this likely to take?Take your answer to this question, double it, and add three months

Wanted: 1500

After 12 months of extensiverecruitment….

Got: 566

My recent experience…

Step Six: Ethics

How will participants provide informed consent?

Are there any risks to completing your survey? How will you manage these?

How will you ensure participant anonymity?

Step Seven: Administer Your Survey

This is the easy part!

Step Eight: Data Entry and Analysis

Set up your data set

A thing of beauty is a joy forever

-Keats

An ugly dataset is a pain in the arseforever

-O’Dwyer

Data Entry & Analysis Cont.

Set up your data set

Each item in your survey should have its own variable.

Each variable needs: Name, Type, Label, Values, Missing, Measure

Your dataset should be detailed enough that someone else could run your analyses without any

additional info about the variables.

This is the dataset from my survey…

If you create scale scores, you should always recode/transform into new/different variables. If your data set is too big, you can create a new dataset with just the scale

scores, but NEVER completely delete the item level data!

Data Entry & Analysis Cont.

Enter your dataDepending on your sample size, you should also double check or double enter 20% to check accuracy of data entry

Prepare your data setCheck for missing data

Check for extreme values

Check for implausible response patterns

Create scale scores (SAVE ALL SYNTAX)

Check internal consistency of scale scores

Good data preparation is the academic equivalent of ‘Measure Twice, Cut Once’

Don’t skimp. Be prepared for data prep to take longer than data analysis. And never delete or modify data without (a) a good reason & (b) keeping a copy.

Data Entry & Analysis Cont.

AnalysisShould be guided by your objectives and research questions

Common techniques in survey research are: descriptives, frequencies, chi-square, t-tests, and regressions

More advanced techniques include: factor analysis and structural equation modeling

Step Nine: Reporting

Methods Section– Brief description of each scale (including number of items, range

of scores and direction)– Reference original scale article– Internal consistency as reported previously– Internal consistency in your sample– Method of imputation (if applicable)

Analysis Section– Cronbach’s alpha– Descriptive analyses– Univariate analyses– Multivariate analyses

Reporting Cont.

Results Section

– Response rate

– Descriptive findings

– Univariate findings

– Multivariate findings

O’Dwyer, Moyle, Zimmer-Gembeck & De Leo.(2013). Suicidal ideation in family careres of people

with dementia: A pilot study.

International Journal of Geriatric Psychiatry, 11, 1182-1188.

How I reported my survey…

A Note on Online Surveys

Advantages• No postage• No data

entry• Increased

anonymity

Disadvantages• Sample bias • Missing data• Duplicates• Ineligible/imp

lausibleRecommendation: Make your survey available in both formats (paper/online) and give people the choice, but consider any differences in your analysis.

If you work through these steps, there’s a reasonable chance you’ll end up with

a good survey.

But who wants a good survey, when you can have a great survey?

And that’s where The Art comes in…

The Art

Recruitment

What would appeal to you?

An appealing headline, but a not so appealing study….

Recruitment Cont.

What information is NOT necessary?The goal is to get people to call or click on the link, everything else can come later

Make it personalIn my work I include a photo & brief bio of each of the team members on the information sheet

Be available and return calls promptlyIf you don’t respect a participant’s time, why should they respect yours?

FormatGroup like scales/questions together

Modify instructions to suit your objective/sample

Leave lots of white space

Make sure font size & colour are appropriate

In scales with more than 4 items, highlight every second one

Format Cont.

If questions break over a page or screen, provide response options again

If you ask an open question, leave enough space for the response

Online: Let participants know how much they have left

Online: Check layout on laptops and different browsers

Aesthetics / Design

DO NOT photocopy scales from journal articles and staple them together!!!!

Make your survey pleasing to the eyeIf it doesn’t look like you care, why should your participants?

Good design does not have to be expensive Try using Tables in Word, but making the lines invisible

Other Tips

Consider providing reminders at regular intervals after the survey is administered

Offer telephone support for people who may otherwise be unable to complete the survey

Practice calculated acts of kindness

Calculated Acts of Kindness

A calculated act of kindness is an act that benefits both parties. A bit like giving a lollipop to a child you are about to

vaccinate.

‘Take a break’ page in middle of survey, with a teabag & sugar sachet stapled in

Hand-written ‘Thank-You’ cards

Hand-written cover notes

Hand-addressed envelopes

List of resources

A few of the calculated acts of kindness from my survey…

A Note on Calculated Acts of Kindness

They are a way to acknowledge that you are the only person for whom this survey is a priority

They are about recognising that being a decent human being and being a researcher are not mutually exclusive

They improve response rates and reduce missing data

They support research more generally, as participants who have a positive experience with you are much more likely to volunteer again. Consider it good research karma.

If you don’t have time to practice them, you don’t have time to do the survey!!!

A Checklist for the Perfect Survey

Identify objectives Define key terms Select questions Prepare survey instrument (Don’t forget The Art) Pilot test and refine Prepare and submit relevant ethics documents (including

information sheet and consent form, if required) Administer survey (Don’t forget Calculated Acts of

Kindness!) Preparing data (Don’t skimp here!) Analyse data Report results