Quantitative Methods

Survey Development

Prof. Paul Licker, Ph. D.

Agenda

• Survey Research• Instrumentation• Stages of Instrumentation• Construct-tion• Generation• Pretesting• Pilot Testing• Item Analysis• Reliability Assessment• Validity Assessment

Survey Research

• Gathering data about a phenomenon using a statistical approach by observing directly or indirectly some subset of the set of phenomena

• Often uses individuals’ recollections or perceptions of events or their evaluations of the events

Survey Data Gathering Methodologies

Personal Interviews Group Interviews Phone Interviews Mail-out Questionnaires Hand-out Questionnaires Clip-and-Mail Questionnaires Survey of Secondary Data

Survey Methodologies ReasoningThe goal of our research is to describe or relate the

behaviour, opinions, impressions, knowledge, etc. of people in situations.

If we cannot directly observe them, their behaviour, etc., we must rely on their own (“subjective”) observations of themselves.

Individuals and their actions and opinions are unique, however. No single self-observation can be generalised to everyone.

Survey Methodologies Reasoning-2

In order to describe the general situation, we aggregate a series of measurements, observations or judgments about individual (“subjective” impressions of) events.

The “objective” description is the average of all these “subjective” descriptions.

We assume all biases “average out” and compensate for individual differences.

Strengths...and…Weaknesses

+ Friendly, familiar, cheap

+ Relatively quick+ Useful if survey

questions are clear+ Ideal for Descriptive

research+ Uses common skills

- Hard to control- Samples are often very

large- Has hidden flaws that

can be easily ignored- Cannot easily be used

to draw causal inferences

- Retrospective

Example: QuestionnairesExample: Questionnaires? Questions are written out? Highly structured? Sent out or handed out to a large group? Response is on paper, to be mailed or

handed back? Anonymity, ease of reading and response,

and clarity are the key qualities? Can use the web or email for distribution

Questionnaire Procedure° Create Sample of Questions

° Pretest Questions

° Derive Sampling Frame, Draw Sample

° Distribute QuestionnairesMotivate Response

Enable Response

° Collect Responses

° Code and Record Responses

1. Mail-Out Questionnaires1. Mail-Out Questionnaires

Determine Sampling Frame Create Package, with response motivator Pretest all questions Expect 25% for public campaign, 50% for

in-company campaign response You must explain non-responses

Mail-out Questionnaire ExampleA sample of 1300 members of the ASM

was mailed a questionnaire concerning their careers in IS. The questionnaire was pretested on 20 ASM members and fellow academics. Only 256 replied on the first wave. A second wave of 1300 was sent out and an additional 162 replied. Of these 402 were actually usable. Eighteen responses came in after the research was written up.

2. Hand-Out 2. Hand-Out QuestionnairesQuestionnaires

Get company permission Use sampling frame Distribute through company, collect

through company Expect at least 50% response rate Watch interaction with company situation

in terms of date, time, season Confidentiality is a BIG concern

Hand-out Questionnaire Example

Using departmental secretaries, ALL 503 professional and managerial staff at a large financial company were given questionnaires on their perceptions of the role of IS in corporate success. A week later the secretaries attempted to collect the responses; only 478 were collected and returned; of these 353 were useful.

3. Mail-in Questionnaires3. Mail-in Questionnaires

Subjects “come across” the questionnaire and are self-motivated to complete it and return it (see also electronic variants)

Suffers from strong self-selection problems: only those who are motivated to return it do so…they may have an axe to grind.

Not considered valid research Similar to radio “talk” shows.

Mail-in Questionnaire Example

Readers of the Business Times were encouraged to fill out, clip, and return a questionnaire about their desires to leave South Africa. 74% of those who professional people who responded said they would leave if they could, which was widely reported and quoted. In fact, those who aren’t intending to leave would never clip the coupon, giving vastly inflated “positive” responses.

Examples of Question Format

1. Fill in the blank __________

2. Multiple Choice (one/more)

First Third Fifth Second Fourth Sixth

3. Rank Order

Item 1 ___ Item 2 ___ Item 3 ___ Item 4 ___

4. This one is ___Yes ___ No

5. N (4, 5, 6, 7-) Point Scale

6. Grid

7. Filter: If xxxx skip to 9

8. Distribution of 100%

A__ % B__% C__% D__%

A B C D E F

No Low Mod Hi V.Hi

Various choices for each item

Give clear di-

rections!

People will make errors!

Questionnaire Strengths…and…Weaknesses

+ Relatively inexpensive+ Can collect lots of data+ Easy to do+ Thought to be quick

and cheap+ Management is

relatively simple+ Data need little

interpretation+ No noverbals?????

- Questions must be ironclad

- Who is responding

- What about non-response?

- Followup is hard

- Can turn out to be expensive

- May require artistry, presentation important

Problems to Watch Out For

• Low response rates: must be explained

• Incomplete response: must be handled

• Lack of Pretesting

• Bad questions (ambiguous, meaningless)

• Open-ended questions: interpretation

• Self-selection

• Who is responding

Electronic Variants of Electronic Variants of Questionnaire SurveysQuestionnaire Surveys

• Web (requires programming, artistry)• Discussion Groups (may annoy lots of

people)• Electronic Mail (can destroy anonymity)

Can gather information quickly over a wide geographical area

Suffers from self-selection Has a halo effect (+ or -) on IS issues Sampling frame is a problem Easy to do very badly after a lot of effort

Questionnaire Diagnostic-1

A mail-out questionnaire is sent to 250 customers of a bank to find out expectations concerning E-commerce. The questionnaire contains 142 fill-in-the-blanks questions about finance, technology, banking, and customer expectations of possible E-commerce ventures. Can you expect some problems?

Questionnaire Diagnostic-2

Almost 1000 questionnaires are handed out to microcomputer users (obtained from Computer Services) through secretaries in a government department for research examining user-IT relations. The six-page questionnaires are to be picked up in one week by the secretaries. Only 151 of the questionnaires are returned. Will there be any problems in interpretation of the data?

Questionnaire Diagnostic-3

Six companies are simultaneously switching to SAP and you want to get impressions from IT staff concerning the switch, so you send out two questionnaires (during the switch and 12 mo. after) by mail to all IT staff. Mostly open-ended questions, the survey has a response rate of 4%. You determine that respondants don’t differ demographically from non-respondants. What’s the problem here?

Instrumentation• The development of valid and reliable

instruments to measure a phenomenon. • Instrument consists of items.• Items are stimuli to respondents• Items must correspond individually or in

sets to constructs• Respondents must be able to give “valid”

responses in a reliable way.• Instrument should be efficient.

Stages of Instrumentation

• Specify domain of constructs

• Generate sample of items

• Pretest and Pilot test for readability and content validity

• Purify instrument, remove extraneous items

• Assess validity (four types)

• Assess reliability or internal consistency

Constructs

An instrument measures constructs

The instrument suffers from the limitations of ALL instruments, namely:

Jiggle (accuracy)

Refinement (precision)

Failure (external reliability)

Consistency (internal reliability)

Ease of use, etc.

Instrumentation

Instrumentation is the task of building an instrument

The instrument must be appropriate for the domain of the construct

The instrument must be valid and reliable

The instrument must be efficient and productive

The instrument must be cost-effective

Instrument Use Process

Instrument

Phenomenon (Human Behaviour or Experience)

What the Instrument saysThe instrument

is a lens through which

to view the phenomenon Problem

areas

locus

use

quality

Instruments Measure Proxies

Theory Concept/Construct

ProxySurrogate

World of Ideas and Concepts

World of Reality and MeasurementActual

instantiation of the

theoretical concept

A variable that is highly

correlated (generally

causally) with the proxy

Measurement Challenges

ProxySurrogate

1. Conceptualising precisely to construct2. Instantiating (operationalising step 1) constructs3. Measuring proxies or determining available surrogates that can be measured reliably4. Demonstrating to others’ satisfaction that steps 1 to 3 can be performed validly

Valid, reliable measurements

Interpretation Challenges1. Determining that the reverse correspondence among proxies, surrogates and constructs is logically valid.2. Using that correspondence, interpret corresponding relationships among constructs (i.e., concepts) in the theory3. Reason whether or not the implied relationships among the constructs supports or denies support to your theory

Valid, reliable measurements

Theory Concept/Construct

Types of ValidityConstructIs the instrument actuallymeasuring the constructor events or is it its out-comes actually an artifact of the instrument

PredictiveCan the instrument dis-tinguish different cases(such as with controlvariables)?

ConvergentDo the items “converge” onthe constructs?

DiscriminantDo the items making up the variables correlate strongly within factors?

Content/FaceDoes the instrument adequately cover the content of the con-structs? Are the items repre-sentative of the content? Is theinstrument constructed “sen-sibly”?

Types of Reliability

InternalAll elements of a scale consistentlymeasure the desired constructwithin the instrument(within subject)

ExternalInstrument can be applied successful-ly in the physicaldomain desired

Instrument is stablein its usage from useto use (intersubject)

Construct-tion

• The goal is to construct an instrument consisting of variables that are internally reliable, valid reflections of the constructs and practical to administer (in the sense that valid responses can be collected at affordable cost)

• Primary tools are argument, debate, pretesting, pilot testing, correlational analysis

Generation of Items

• Literature survey

• Supervisor

• Your own intuitiion

• Create a large pool, don’t worry about initial size

• State items in comparable way, if building scales

PreTesting

• Use a panel of judges, experts in the field, people familiar with the culture of informants (=respondents)

• Can use other doctoral students, lecturers, industrial sponsors, etc.

Pilot Testing

• Test for readability• Test for respondability (reliability)• Purpose is to create a smaller set of items by

eliminating those that pose problems.• Such problems include jargon, form and format,

grammar, ambiguity, multidimensionality (two or more questions in one), cultural no-nos, language level, assumed intelligence, etc.)

Readability Procedures

1. Read the instrument yourself. Can it be read?

2. Have your supervisor and committee read the instrument

3. Pilot the instrument on a “captive” group first. Do a protocol analysis on this group. Speak with them as they hear each question and go over its possible meanings. Can they begin to answer the questions?

4. Pilot the instrument on a representative small group of respondents.

Item Analysis (Purification)

• Make sure that the factors apparent in the instrument are in fact “pure” and do not include extraneous factors that are merely artifacts of the questions themselves.

• Get rid of “garbage items”• Use item-scale correlation to calculate

Cronbach’s Alpha and eliminate those whose elimination will not lower the Alpha.

Assessing Reliability

• Reliability is of two types:– Item-scale (internal consistency)– Test-retest (external consistency)

• Overall alpha of 0.7 or 0.8, even 0.9 should be found.

Assessing Validity

• Factor analysis:– Do items load “purely” on one factor only?– Do any items fail to load on any factors?– Rule of thumb is to use factor loading of 0.5 as

criterion

The End Product

The Instrument

An exhaustive series of variables

Internally

consistent

Mutually Exclusive

The End Product

The Instument

That reflect the underlying constructs of your theory