MIS 650 Data Collection

MIS 650: Data Collection 1

MIS 650Data Collection


Chapter 3: Methodology Chapter Outline 3.1 Methodological Issues (Usually Validity

and Reliability, sometimes Ethics) 3.2 Sampling Methods 3.3 Data Collection Techniques 3.4 Data Integrity Issues 3.5 Analysis “Look-ahead”


Idea: Theory,Model Test Plan:

Methodology

Physical Testof Hypotheses

usingMethodology

Conclusions aboutIdeaWhat the data say

What the world says

What you sayWhat the theory says

data

Research methods

hypotheses

conclusions


3.1 Methdological Issues

• State of Theory in your area (well developed, speculative)

• Ability to generalize

• Role of data in your research; is it empirical?

• Formal or informal index of “goodness” of your methodology within a general critique


State of Theory

Theory vs. Experience: Is theory well developed or are we still experiencing rather than thinking about this area?

Role of Language: Are there well-defined terms and measures?

Proof vs. Communication: Role of paper Qualitative vs. Quantitative Research: Do

strong theories already exist?


Role of Data

• Data are Instances of Abstractions

• These instances have relationships which test relationships among abstractions

• The abstraction relationships are the theory

• We use DATA (measurements) to demonstrate the theoretical relationships among the abstractions

`


“Our theories are the scripts; the world, our stage; researchers, the stage managers; and data,

the film of the players’ performances. Our goal is to create excitement, sell tickets,

and satisfy the public.”


Classes of Problems

Sampling Problems (Cases, Companies, Individuals, Times, Tasks)

Observer Errors (Creating the wrong stimuli)

Subject Errors (Getting wrong responses)Recording Errors (Losing the data)Ethical Problems (Not deserving the data)


Where Students Often Fail

Lack of Theory to Guide MethodPoor Operationalization of ConceptsConvenience SamplesMeasurement ErrorsSloppy Data CollectionToo little data


3.2 Sampling

• Discuss how sample was obtained

• What was used as the sampling frame? Why?

• Were there any problems with representativeness?

• Were there any potential ethical problems?


Sampling Issues

• RepresentativenessUsually assured by “random” sampling

Not always an issue or an issue to the same degree

• ProcedureTopic/Hypotheses Universe Sampling

Frame Research Sample Actual Sample


Representativeness

Data points must be “unbiased”

This means that qualities of the source of the data should not (apparently) affect the content of the data

Generally this means that every potential data source has the same probability of being in the research sample


Representativeness, Cont’d

The question is then, “Do the sources of data in the research sample represent all those data points not present?”

If YES, then conclusions drawn from the data can be generalized to the whole universe.

If NO, then such conclusions will be deemed to apply only to the research sample.


Representativeness, Cont’d

Representativeness works in two ways:

1. Generalizability

Do the data represent the universe?

2. Confidence

How well do the data do that representation?


Representativeness, Cont’dConfidence

This becomes an issue because of random variation rather than bias. Random variation is only an accumulation of unknown biases.

SystematicBias

RandomVariation

Systematic bias pushes qualities of data source in particular directions thus increasing possibility of wrong conclusion.

Random variation pushes qualities of data source in many random directions, thus lowering confidence in conclusions


Procedure-1

Topic/Hypotheses UniverseTopic applies to particular part of the world and

your hypotheses can only be tested in a particular “world”

The universe is what your ideas are eventually going to “apply to”


Procedure-2

Universe Sampling FameSampling Frame is a systematic way to get to data

sources in your universe.

Examples include phone directories, databases, printed lists, physical “inventory”

All real sampling frames are inaccurate, out of date and incomplete. Problems must be addressed and discussed.


Procedure-3

Sampling Frame Research SampleResearch sample is the actual list of your data

sources. For generalization research sample should be “representative”

Research sample should be drawn “randomly” if possible or sometimes in a stratified manner.

Taking every nth item is common, or using random number table.

Not every item selected is “real”!


Procedure-4

Research Sample Actual SampleActual sample is smaller than research sample:

Sources may not be available

Scheduling is hard

Interruptions, lost data, accidents, etc.

Sampling frame may be inaccurate or out of date.


Sampling Issues Level of Aggregation Issues

Organization, Group, Individual, Task. Sampling Entity Issues

Site, Individual, Task, Time, Measurements

Sample Size IssuesParameterisation, Inference,

Description


Sample Structure

Universe (all possible things)

Sampling Frame (Systematic Division into Allowable/not Allowable)

Sample Situation


Ex-sample

Universe [Users]

Sampling Frame [Firm phone Directory]

Sample [Every 3rd] Situation


Problems in Sampling

Convenience Sampling -- unrepresentative Lack of a Sampling Frame -- can’t sample Too small a sample size -- low confidence Too large a sample size -- wasted effort Sampling the wrong thing -- useless Non-representative Sampling -- cannot

generalise


3.3 Data Collection Techniques

• What were the possible choices for data collection technique?

• Why did you choose method you did?• Describe the method in detail• Was there a role for observers, coders,

interpretation?• Show how you handled problems with the

technique you selected.


General Data Collection Methods

Dimensions: Real-time vs. retrospective

Observed now or subject recalls from past Projective vs. Subjective

Others’/subjects’ experience Researcher-driven vs. subject-driven

Researcher creates stimulus/subject does this Most common methods are case studies,

surveys and experiments Empirical vs. non-empirical

Survey Expt. Obsv’n Case

Ret RT RT RT/Ret

Pro/Sub Sub N/A Pro/Sub

Res Res Subj Subj/Pro

Emp Emp Emp Emp


Data Collection Model

ObserverSubjectInterpreter/

Coder

1. Theory

2. Stimulus formulation

3. Stimulus / Question

4.

10.

5. Perceived Stimulus

6. Knowledge

7. Ideas

8. Response formulation

9. Response / Answer

11. Perceived Response

12. Recorded Response




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




1. (Actually H1): Prior experience with one application influences perception of innovation.




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




2. 3. “Which of the following applications have you used in the past 12 months?”




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




4. 5. “Do you know how to do your job?”




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




6. 7. <Hmmm, maybe I look like I don’t know what I’m doing here…better deny!>




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




8. 9. “Nope, haven’t used any of them”




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.



Recorded Response

10. 11. <Hmm, he must be an idiot not to have used these appli-cations>




Coder

1. Theory

2.


4.

10.


6. Knowledge

7. Ideas

8.




12. Don’t Know


Observer Errors

Mistakes that observers commit, usually not observing the right phenomenon or masking subjects’ behaviour

Subject Behaviour

Observer Behaviour


Observer Errors

• Intrusion, leading questions

Setting up the situation to give a predetermined answer, interfering with subjects’ ability to select an answer by supplying it, assuming an answer, not respecting silence


Observer Errors

Intrusion, leading questions

• Expectation management problems

Creating a situation in which subject tries to “guess” correct answer or tries to “please” the researcher by giving socially mandated or desirable responses


Observer Errors


Expectation management problems

• Consultant effect

Interfering with “normal” behavior by changing the situation to favor socially-facilitated responses or by focusing attention on behavior under study


Observer Errors


Expectation management problems

Consultant effect

• Hawthorne effect

A consultant-related effect in which behavior is enhanced because attention has been drawn to it.


Subject ErrorsMany things can influence the subject in his or her

responses. Here are some of the sources Memory effects Protocol Intrusion effects Subject Context and Limitation effects Researcher-Subject Interaction effects Subject Cognition effects Instrument-Subject Interactions


Subject Errors

Context / Protocol

“Subject” is the source ofvariance we desire

Sub-ject

Mem-ory

Instru-ment

E-vents

Researcher

Cog-nition

Re-sponse

Generally, these errors are most noticeable and problematic when subjects are used in a retrospective manner. However, any task requiring cognition or performance of any type is subject to most of these problems.


Memory EffectsMemory for events changes over time and under the

influence of other events Recency Primacy Von Restdorff “I don’t remember” “I used to know” Clustering

Time since event remembered

Rec

all/

reco

gnit

ion


Protocol Intrusion EffectsResponses are conditioned not only by what the

respondent might know, think or feel, but also by the presence of words or concepts in the stimulus or stimulus situation

Sequence Positive Halo Negative Halo “Mand” characteristics

A

B

C

D


Subject Context and Limitation EffectsHow the subject feels about you, your questions,

everything, determines the responses and how the responses are presented.

Stupidity Ignorance Ill Will towards you, the organization or “system”,

research, any group you are imagined to be part of or represent

Resistance


Researcher-Subject Interaction EffectsBecause you are present (or not), your being

around may affect what the respondent does and hence how the respondent replies.

Social facilitation


Subject Cognition EffectsThe subject is not just a machine that reacts. He or she

engages in games, strategizes, and tries to understand the situation while working as a response “machine”. Intrusion effects (halo (+/-), sequence)

Experimenter expectancy Evaluation apprehension Gamesmanship Face games, one-upmanship The problem of the in-group (technicians, mgrs)


Instrument-Subject Interactions

The instrument may prompt, provoke or prevent response because of its design

Poor scales for response Too many responses, fatigue Aesthetic reactions


Recording Errors

Failure to listenCategorization errorsGeneral carelessnessPrivacy problemsToo little room on mediumOver-reliance on tape or technologyPoor scales


Interpretation Errors

• Misunderstanding

• Poor conceptualization of constructs

• Poor scales


3.4 Data Integrity Issues

• How data will be recorded

• Potential problems with recording

• How data will be maintained

• Potential problems with maintenance

• How data will be stored, accessed

• Potential problems with storage, access

• Are data confidential?

Documents

MIS 650 Data Collection