Survey Instruments and Questionnaire Design

SURVEY INSTRUMENTS & QUESTIONNAIRE DESIGN

Warren Seuradge Strategic Planning, Policy, and Partnerships

Region of Peel (2014)

Step 1

Problem Definition and Research Objectives

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Step 3

Data Analysis and

Measurement

!" S#rv"$ Pr%&"''

Step 1

Problem Definition and Research Objectives

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Step 3

Data Analysis and

Measurement

!" S#rv"$ Pr%&"''

Step 1

Problem Definition and Research Objectives

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(1) Identify Source of the Problem

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED

(1) Identify Source of the Problem

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

SITUATIONAL ANALYSIS

(1) Identify Source of the Problem

(2) Define the ProbLEM

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY

RESEARCH

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES

DETERMINE INFORMATION GAPS

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES

DETERMINE INFORMATION GAPS

(1) STAKEHOLDERS (2) PROBLEM SYMPTOMS (3) POTENTIAL CAUSES (4) COURSES OF ACTION? !" '-.-")"+- '/%#(0

,+&(#0":

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(1) Identify Source of the Problem

(2) Define the ProbLEM

(3) CREATE RESEARCH OBJECTIVES

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES

DETERMINE INFORMATION GAPS

Problem Statement

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(3) CREATE RESEARCH OBJECTIVES

(1) Identify Source of the Problem

(2) Define the ProbLEM

HYPOTHESIS TO CLOSE THE INFORMATION GAP IN THE PROBLEM STATEMENT

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES

DETERMINE INFORMATION GAPS

Problem Statement

Step 1

Problem Definition and Research Objectives

Pr%b(") R"&%*+,-,%+

(a) FAILURE TO MEET OBJECTIVES

WHAT HAPPENED VERSUS SHOULD HAVE HAPPENED (b) OPPORTUNITIES

WHAT HAPPENED VERSUS COULD HAPPEN

(3) CREATE RESEARCH OBJECTIVES

(1) Identify Source of the Problem

(2) Define the ProbLEM

HYPOTHESIS TO CLOSE THE INFORMATION GAP IN THE PROBLEM STATEMENT

IE. To gather information from a sample representative of the PEEL REGION POPULATION AMONG THOSE WHO ARE “VERY LIKELY” to use an emergency shelter within the next 6 months, as

measured on a scale of 1 = Very UNLIKELY and 5 = VERY LIKELY. This data would serve as an input into a forecasting model to predict service demand.

VALIDATE SYMPTOMS WITH PRIMARY AND SECONDARY RESEARCH

SITUATIONAL ANALYSIS

VALIDATE PROBABLE CAUSES

DETERMINE INFORMATION GAPS

Problem Statement

Step 1

Problem Definition and Research Objectives

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Step 3

Data Analysis and

Measurement

!" S#rv"$ Pr%&"''

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(a) EXPLORATORY BROADLY DEFINE THE PROBLEM

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(a) EXPLORATORY BROADLY DEFINE THE PROBLEM

EXPERIENCE SURVEYS CASE ANALYSIS

FOCUS GROUPS

SECOND-ARY

DATA

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(a) EXPLORATORY

(b) DESCRIPTIVE ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW

BROADLY DEFINE THE PROBLEM

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(a) EXPLORATORY

(b) DESCRIPTIVE ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW

CROSS-SECTIONAL STUDY

SAMPLE SURVEY

LONGITUDINAL STUDY PANELS

BROADLY DEFINE THE PROBLEM

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(b) DESCRIPTIVE

(c) CAUSAL IF X, THEN Y

(a) EXPLORATORY

ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW BROADLY DEFINE THE PROBLEM

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(b) DESCRIPTIVE

(c) CAUSAL IF X, THEN Y

(a) EXPLORATORY

BEFORE-AFTER WITH CONTROL GROUP

ONE-GROUP, BEFORE ONE GROUP, BEFORE-AFTER ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW BROADLY DEFINE THE PROBLEM

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

(b) DESCRIPTIVE

(c) CAUSAL

(a) EXPLORATORY

ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW BROADLY DEFINE THE PROBLEM IF X, THEN Y

A''"'':

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

R"'".r&/ D"',*+

A''"'':

(1) INTERNAL VALIDITY

(2) EXTERNAL VALIDITY

ANSWERS WHO, WHAT, WHERE, WHEN, WHY, & HOW BROADLY DEFINE THE PROBLEM IF X, THEN Y

(b) DESCRIPTIVE

(c) CAUSAL

(a) EXPLORATORY

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r"

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Screens Part 1 Have you visited a

shelter in the past month? Selection

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Warm-Ups

Screens Part 1

Part 2

Have you visited a shelter in the past month?

To What extent were you satisfied with the experience?

Selection

Ease

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Transitions

Warm-Ups

Screens Part 1

Part 2

Part 3

Have you visited a shelter in the past month?

To What extent were you satisfied with the experience?

The following part of this survey concerns your employment history.

Selection

Ease

Subject change

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Complicated Questions

Transitions

Warm-Ups

Screens Part 1

Part 2

Part 3

Part 4

Have you visited a shelter in the past month?

To What extent were you satisfied with the experience?

The following part of this survey concerns your employment history.

Rate each of the following factors on a scale of 1 = no need to 7 =

extremely need.

Selection

Ease

Subject change

Commitment

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

S#rv"$ S-r#&-#r" Question type Location Examples rationale

Complicated Questions

Classification and Demographics

Transitions

Warm-Ups

Screens Part 1

Part 2

Part 3

Part 4

Part 5

Have you visited a shelter in the past month?

To What extent were you satisfied with the experience?

The following part of this survey concerns your employment history.

Rate each of the following factors on a scale of 1 = no need to 7 =

extremely need.

What is the highest level of education you’ve attained.

Selection

Ease

Subject change

Commitment

Personal

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Thin

gs to

do

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Thin

gs to

do

•  Be Focused •  please rate your experience

with x on the following features.

•  Be Brief •  Did you use the emergency

shelter as a last-resort?

•  Use Simple Structure •  To what extent do you rely

on emergency shelters? (Quantify options)

•  Be Crystal Clear •  Under What Circumstances

would you find the emergency shelter useful? (LIST OPTIONS)

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Thin

gs to

do

•  Be Focused •  please rate your experience

with x on the following features.

•  Be Brief •  Did you use the emergency

shelter as a last-resort?

•  Use Simple Structure •  To what extent do you rely

on emergency shelters? (Quantify options)

•  Be Crystal Clear •  Under What Circumstances

would you find the emergency shelter useful? (LIST OPTIONS)

Thin

gs to

Avoi

d

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Q#"'-,%+ F%r)#(.-,%+

Thin

gs to

do

•  Be Focused •  please rate your experience

with x on the following features.

•  Be Brief •  Did you use the emergency

shelter as a last-resort?

•  Use Simple Structure •  To what extent do you rely

on emergency shelters? (Quantify options)

•  Be Crystal Clear •  Under What Circumstances

would you find the emergency shelter useful? (LIST OPTIONS)

•  Leading q’s •  Shouldn’t everybody have

the right to live free of poverty?

•  Loaded Q’s •  Have you quit smoking?

•  Double Barreled Q’s •  Do you think social service

agencies should offer more educational resources and outreach programs?

•  Overstated Q’s •  Do you think This program

will eliminate poverty in Peel?

Thin

gs to

Avoi

d

Step 1

Problem Definition and Research Objectives

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Step 3

Data Analysis and

Measurement

!" S#rv"$ Pr%&"''

Step 3

DATA ANALYSIS AND MEASUREMENT

S&.(" C/.r.&-"r,'-,&'

(a) NOMINAL SCALES LABELS OBJECTS (RACE, RELIGION, TYPE OF DWELLING, AGE, GENDER,

YES/NO Q’s) DESCRIPTION

Step 3

DATA ANALYSIS AND MEASUREMENT

S&.(" C/.r.&-"r,'-,&'

(a) NOMINAL SCALES LABELS OBJECTS (RACE, RELIGION, TYPE OF DWELLING, AGE, GENDER,

YES/NO Q’s) DESCRIPTION

(b) ORDINAL SCALES INDICATE RELATIVE SIZE DIFFERENCES

BETWEEN OBJECTS (SERVICE-FREQUENCY USAGE)

DESCRIPTION + ORDER

Step 3

DATA ANALYSIS AND MEASUREMENT

S&.(" C/.r.&-"r,'-,&'

(a) NOMINAL SCALES LABELS OBJECTS (RACE, RELIGION, TYPE OF DWELLING, AGE, GENDER,

YES/NO Q’s) DESCRIPTION

(b) ORDINAL SCALES INDICATE RELATIVE SIZE DIFFERENCES

BETWEEN OBJECTS (SERVICE-FREQUENCY USAGE)

DESCRIPTION + ORDER

(c) INTERVAL SCALES DESCRIPTORS ARE EQUAL DISTANCES

APART (RATING A SERVICE ON ITS OVERALL QUALITY FROM POOR = 1 and

EXCELLENT = 5) DESCRIPTION + ORDER + DISTANCE

Step 3

DATA ANALYSIS AND MEASUREMENT

S&.(" C/.r.&-"r,'-,&'

(a) NOMINAL SCALES LABELS OBJECTS (RACE, RELIGION, TYPE OF DWELLING, AGE, GENDER,

YES/NO Q’s) DESCRIPTION

(b) ORDINAL SCALES INDICATE RELATIVE SIZE DIFFERENCES

BETWEEN OBJECTS (SERVICE-FREQUENCY USAGE)

DESCRIPTION + ORDER

(c) INTERVAL SCALES DESCRIPTORS ARE EQUAL DISTANCES

APART (RATING A SERVICE ON ITS OVERALL QUALITY FROM POOR = 1 and

EXCELLENT = 5) DESCRIPTION + ORDER + DISTANCE

(d) RATIO SCALES TRUE ZERO ORIGIN EXISTS (# OF

$ SPENT ON X PER WEEK, DISTANCE TRAVELLED, # OF CHILDREN, # OF PEOPLE IN

HOUSEHOLD) DESCRIPTION + ORDER +

DISTANCE + ORIGIN

Step 3

DATA ANALYSIS AND MEASUREMENT

D.-. T1" .+0 S-.-,'-,&'

Step 3

DATA ANALYSIS AND MEASUREMENT

D.-. T1" .+0 S-.-,'-,&'

DATA Relationships Statistical Analysis

Step 3

DATA ANALYSIS AND MEASUREMENT

D.-. T1" .+0 S-.-,'-,&'

DATA Relationships Statistical Analysis

•  CROSS-TABULATIONS •  CHI-SQUARE TESTS (PRESENCE OF RELATIONSHIP) •  CRAMER’S V-STATISTIC (STRENGTH OF RELATIONSHIP)

(a) NOMINAL (CATEGORICAL) SCALES

Step 3

DATA ANALYSIS AND MEASUREMENT

D.-. T1" .+0 S-.-,'-,&'

DATA Relationships Statistical Analysis

•  CROSS-TABULATIONS •  CHI-SQUARE TESTS (PRESENCE OF RELATIONSHIP) •  CRAMER’S V-STATISTIC (STRENGTH OF RELATIONSHIP)

•  SPEARMAN RANK-ORDER CORRELATION •  KENDALL’S TAU

(a) NOMINAL (CATEGORICAL) SCALES

(b) ORDINAL (RANKING) SCALES

Step 3

DATA ANALYSIS AND MEASUREMENT

D.-. T1" .+0 S-.-,'-,&'

DATA Relationships Statistical Analysis

•  CROSS-TABULATIONS •  CHI-SQUARE TESTS (PRESENCE OF RELATIONSHIP) •  CRAMER’S V-STATISTIC (STRENGTH OF RELATIONSHIP)

•  SPEARMAN RANK-ORDER CORRELATION •  KENDALL’S TAU

•  SCATTER PLOT •  PEARSON PRODUCT MOMENT CORRELATION

(a) NOMINAL (CATEGORICAL) SCALES

(b) ORDINAL (RANKING) SCALES

(b) METRIC (INTERVAL/RATIO) SCALES

Step 1

Problem Definition and Research Objectives

Step 2

RESEARCH AND QUESTIONNAIRE DESIGN

Step 3

Data Analysis and

Measurement

!" S#rv"$ Pr%&"''

QUESTIONS?

Warren Seuradge Warren.seuradge@mail.mcgill.ca