MIS 650 Data Collection

MIS 650: Data Collection 1

MIS 650Data Collection

MIS 650: Data Collection 2

Chapter 3: Methodology Chapter Outline 3.1 Methodological Issues (Usually Validity

and Reliability, sometimes Ethics) 3.2 Sampling Methods 3.3 Data Collection Techniques 3.4 Data Integrity Issues 3.5 Analysis “Look-ahead”

MIS 650: Data Collection 3

Idea: Theory,Model Test Plan:

Methodology

Physical Testof Hypotheses

usingMethodology

Conclusions aboutIdeaWhat the data say

What the world says

What you sayWhat the theory says

data

Research methods

hypotheses

conclusions

MIS 650: Data Collection 4

3.1 Methdological Issues

• State of Theory in your area (well developed, speculative)

• Ability to generalize

• Role of data in your research; is it empirical?

• Formal or informal index of “goodness” of your methodology within a general critique

MIS 650: Data Collection 5

State of Theory

Theory vs. Experience: Is theory well developed or are we still experiencing rather than thinking about this area?

Role of Language: Are there well-defined terms and measures?

Proof vs. Communication: Role of paper Qualitative vs. Quantitative Research: Do

strong theories already exist?

MIS 650: Data Collection 6

Role of Data

• Data are Instances of Abstractions

• These instances have relationships which test relationships among abstractions

• The abstraction relationships are the theory

• We use DATA (measurements) to demonstrate the theoretical relationships among the abstractions

`

MIS 650: Data Collection 7

“Our theories are the scripts; the world, our stage; researchers, the stage managers; and data,

the film of the players’ performances. Our goal is to create excitement, sell tickets,

and satisfy the public.”

MIS 650: Data Collection 8

Classes of Problems

Sampling Problems (Cases, Companies, Individuals, Times, Tasks)

Observer Errors (Creating the wrong stimuli)

Subject Errors (Getting wrong responses)Recording Errors (Losing the data)Ethical Problems (Not deserving the data)

MIS 650: Data Collection 9

Where Students Often Fail

Lack of Theory to Guide MethodPoor Operationalization of ConceptsConvenience SamplesMeasurement ErrorsSloppy Data CollectionToo little data

MIS 650: Data Collection 10

3.2 Sampling

• Discuss how sample was obtained

• What was used as the sampling frame? Why?

• Were there any problems with representativeness?

• Were there any potential ethical problems?

MIS 650: Data Collection 11

Sampling Issues

• RepresentativenessUsually assured by “random” sampling

Not always an issue or an issue to the same degree

• ProcedureTopic/Hypotheses Universe Sampling

Frame Research Sample Actual Sample

MIS 650: Data Collection 12

Representativeness

Data points must be “unbiased”

This means that qualities of the source of the data should not (apparently) affect the content of the data

Generally this means that every potential data source has the same probability of being in the research sample

MIS 650: Data Collection 13

Representativeness, Cont’d

The question is then, “Do the sources of data in the research sample represent all those data points not present?”

If YES, then conclusions drawn from the data can be generalized to the whole universe.

If NO, then such conclusions will be deemed to apply only to the research sample.

MIS 650: Data Collection 14

Representativeness, Cont’d

Representativeness works in two ways:

1. Generalizability

Do the data represent the universe?

2. Confidence

How well do the data do that representation?

MIS 650: Data Collection 15

Representativeness, Cont’dConfidence

This becomes an issue because of random variation rather than bias. Random variation is only an accumulation of unknown biases.

SystematicBias

RandomVariation

Systematic bias pushes qualities of data source in particular directions thus increasing possibility of wrong conclusion.

Random variation pushes qualities of data source in many random directions, thus lowering confidence in conclusions

MIS 650: Data Collection 16

Procedure-1

Topic/Hypotheses UniverseTopic applies to particular part of the world and

your hypotheses can only be tested in a particular “world”

The universe is what your ideas are eventually going to “apply to”

MIS 650: Data Collection 17

Procedure-2

Universe Sampling FameSampling Frame is a systematic way to get to data

sources in your universe.

Examples include phone directories, databases, printed lists, physical “inventory”

All real sampling frames are inaccurate, out of date and incomplete. Problems must be addressed and discussed.

MIS 650: Data Collection 18

Procedure-3

Sampling Frame Research SampleResearch sample is the actual list of your data

sources. For generalization research sample should be “representative”

Research sample should be drawn “randomly” if possible or sometimes in a stratified manner.

Taking every nth item is common, or using random number table.

Not every item selected is “real”!

MIS 650: Data Collection 19

Procedure-4

Research Sample Actual SampleActual sample is smaller than research sample:

Sources may not be available

Scheduling is hard

Interruptions, lost data, accidents, etc.

Sampling frame may be inaccurate or out of date.

MIS 650: Data Collection 20

Sampling Issues Level of Aggregation Issues

Organization, Group, Individual, Task. Sampling Entity Issues

Site, Individual, Task, Time, Measurements

Sample Size IssuesParameterisation, Inference,

Description

MIS 650: Data Collection 21

Sample Structure

Universe (all possible things)

Sampling Frame (Systematic Division into Allowable/not Allowable)

Sample Situation

MIS 650: Data Collection 22

Ex-sample

Universe [Users]

Sampling Frame [Firm phone Directory]

Sample [Every 3rd] Situation

MIS 650: Data Collection 23

Problems in Sampling

Convenience Sampling -- unrepresentative Lack of a Sampling Frame -- can’t sample Too small a sample size -- low confidence Too large a sample size -- wasted effort Sampling the wrong thing -- useless Non-representative Sampling -- cannot

generalise

MIS 650: Data Collection 24

3.3 Data Collection Techniques

• What were the possible choices for data collection technique?

• Why did you choose method you did?• Describe the method in detail• Was there a role for observers, coders,

interpretation?• Show how you handled problems with the

technique you selected.

MIS 650: Data Collection 25

General Data Collection Methods

Dimensions: Real-time vs. retrospective

Observed now or subject recalls from past Projective vs. Subjective

Others’/subjects’ experience Researcher-driven vs. subject-driven

Researcher creates stimulus/subject does this Most common methods are case studies,

surveys and experiments Empirical vs. non-empirical

Survey Expt. Obsv’n Case

Ret RT RT RT/Ret

Pro/Sub Sub N/A Pro/Sub

Res Res Subj Subj/Pro

Emp Emp Emp Emp

MIS 650: Data Collection 26

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2. Stimulus formulation

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8. Response formulation

9. Response / Answer

11. Perceived Response

12. Recorded Response

MIS 650: Data Collection 27

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

1. (Actually H1): Prior experience with one application influences perception of innovation.

MIS 650: Data Collection 28

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

2. 3. “Which of the following applications have you used in the past 12 months?”

MIS 650: Data Collection 29

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

4. 5. “Do you know how to do your job?”

MIS 650: Data Collection 30

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

6. 7. <Hmmm, maybe I look like I don’t know what I’m doing here…better deny!>

MIS 650: Data Collection 31

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

8. 9. “Nope, haven’t used any of them”

MIS 650: Data Collection 32

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

Recorded Response

10. 11. <Hmm, he must be an idiot not to have used these appli-cations>

MIS 650: Data Collection 33

Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2.

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8.

9. Response / Answer

11. Perceived Response

12. Recorded Response

12. Don’t Know

MIS 650: Data Collection 34

Observer Errors

Mistakes that observers commit, usually not observing the right phenomenon or masking subjects’ behaviour

Subject Behaviour

Observer Behaviour

MIS 650: Data Collection 35

Observer Errors

• Intrusion, leading questions

Setting up the situation to give a predetermined answer, interfering with subjects’ ability to select an answer by supplying it, assuming an answer, not respecting silence

MIS 650: Data Collection 36

Observer Errors

Intrusion, leading questions

• Expectation management problems

Creating a situation in which subject tries to “guess” correct answer or tries to “please” the researcher by giving socially mandated or desirable responses

MIS 650: Data Collection 37

Observer Errors

Intrusion, leading questions

Expectation management problems

• Consultant effect

Interfering with “normal” behavior by changing the situation to favor socially-facilitated responses or by focusing attention on behavior under study

MIS 650: Data Collection 38

Observer Errors

Intrusion, leading questions

Expectation management problems

Consultant effect

• Hawthorne effect

A consultant-related effect in which behavior is enhanced because attention has been drawn to it.

MIS 650: Data Collection 39

Subject ErrorsMany things can influence the subject in his or her

responses. Here are some of the sources Memory effects Protocol Intrusion effects Subject Context and Limitation effects Researcher-Subject Interaction effects Subject Cognition effects Instrument-Subject Interactions

MIS 650: Data Collection 40

Subject Errors

Context / Protocol

“Subject” is the source ofvariance we desire

Sub-ject

Mem-ory

Instru-ment

E-vents

Researcher

Cog-nition

Re-sponse

Generally, these errors are most noticeable and problematic when subjects are used in a retrospective manner. However, any task requiring cognition or performance of any type is subject to most of these problems.

MIS 650: Data Collection 41

Memory EffectsMemory for events changes over time and under the

influence of other events Recency Primacy Von Restdorff “I don’t remember” “I used to know” Clustering

Time since event remembered

Rec

all/

reco

gnit

ion

MIS 650: Data Collection 42

Protocol Intrusion EffectsResponses are conditioned not only by what the

respondent might know, think or feel, but also by the presence of words or concepts in the stimulus or stimulus situation

Sequence Positive Halo Negative Halo “Mand” characteristics

A

B

C

D

MIS 650: Data Collection 43

Subject Context and Limitation EffectsHow the subject feels about you, your questions,

everything, determines the responses and how the responses are presented.

Stupidity Ignorance Ill Will towards you, the organization or “system”,

research, any group you are imagined to be part of or represent

Resistance

MIS 650: Data Collection 44

Researcher-Subject Interaction EffectsBecause you are present (or not), your being

around may affect what the respondent does and hence how the respondent replies.

Social facilitation

MIS 650: Data Collection 45

Subject Cognition EffectsThe subject is not just a machine that reacts. He or she

engages in games, strategizes, and tries to understand the situation while working as a response “machine”. Intrusion effects (halo (+/-), sequence)

Experimenter expectancy Evaluation apprehension Gamesmanship Face games, one-upmanship The problem of the in-group (technicians, mgrs)

MIS 650: Data Collection 46

Instrument-Subject Interactions

The instrument may prompt, provoke or prevent response because of its design

Poor scales for response Too many responses, fatigue Aesthetic reactions

MIS 650: Data Collection 47

Recording Errors

Failure to listenCategorization errorsGeneral carelessnessPrivacy problemsToo little room on mediumOver-reliance on tape or technologyPoor scales

MIS 650: Data Collection 48

Interpretation Errors

• Misunderstanding

• Poor conceptualization of constructs

• Poor scales

MIS 650: Data Collection 49

3.4 Data Integrity Issues

• How data will be recorded

• Potential problems with recording

• How data will be maintained

• Potential problems with maintenance

• How data will be stored, accessed

• Potential problems with storage, access

• Are data confidential?