Web-Based Research:Issues, Problems and an

Example Technique

John H. Krantz

Hanover College

Outline

Brief History Issues in Internet Research An Example Method Problems and Solutions

History

Email for 2-3 Decades 1995

Krantz, Ballard, & Scher (1997) Within Subjects Reips (1997) Between Subjects First True Web Experiments

July 1995 List of Online Psychological Research – sponsored by APS for awhile http://psych.hanover.edu/research/exponnet.html Almost 200 studies and study sites listed

Why Do Internet Research

Musch & Reips (2000) Sample Size Statistical Power High Speed Ability to reach participants in other countries High external or ecological validity Low cost Ability to replicate a lab experiment with more

power Special populations

Issues

Is the Sample Representative Yes and No

Important to remember base of comparison Ideal vs. Lab

Krantz & Dalal (2000) Much more diverse that most lab samples Especially in age and education range Race is generally limited as is nationality

Sample Characteristics: Gender

Are Internet Samples Male Dominated? Krantz & Dalal (2000) % Female

GVU 1st (1994) 5% Reips (1996) English: 43%

German: 18% Krantz, et al. (1997) 44% Pasveer & Ellard (1998) 3rd Study 71%

More Recent Caddell & Utt (2004) 77% Meyerson & Tryon (2003) 45%

Sample Characteristics: Age

Are we still testing college sophomores? Krantz, et al. (1997) 43% > 30 Smith & Leigh (1997) 35% > 30 Pasveer & Ellard (1998) 45% > 25 GVU 1st 36% > 30 Caddell & Utt (2004) 60% > 30 Pattison & Rouse(2003) 16% > 30

Is the College Sophomore be Making a Comeback

Plot of activity of Psychological Research on the Net

Pattison & Rouse (2004) 76% 18-22

0

10000

20000

30000

40000

50000

Jan

FebM

ar AprM

ay Jun

July

AugSep O

ctNov

Dec

Monlth

# H

its

2003

Sample Characteristics: Race

How diverse are the samples? Unfortunately in general samples, diversity is still

limited Krantz et al. (1997) 89% White Smith & Leigh (1997) 86% White GVU 10th (1998b) 87% White O’Neil, Penrod, & Bornstein (2003) 82% White Meyerson & Tryon (2003) 93% White

However, as will be discussed later, web can make it possible to access special populations

Sample Characteristics: Nationality

Where to do the subjects come from? Largely North American, even US, even in

some European studies Krantz, et al. (1997) 86% N. Am. Senior, et al (1999) >80% N. Am

This study was conducted in England.

Is the Data Any Good

Emphatically Yes

Data Quality: Direct Comparisons

A number of studies, still, run both laboratory and internet samples.

Krantz, et al. (1997) Regression of web means on laboratory

samples (even though different types of samples):

lab mean = 1*(Web mean) + 0

r2 = .99

Compare to Established Data

Compare web results to previously published data sets

Myerson & Tryon (2003) Studied Sexual Boredom Scale of Watt &

Ewing (1996) Matched sample characteristics Found same internal consistency Form of administration was not a significant

factor

Direct Validity Comparison

Use same techniques to validate results Pasveer & Ellard (1998) Developed new scale Internal consistency Psychomectric properties

Data Validity: Arguments

Reips (2000) Statistical Power Limited Sample Population Limited External Validity Less than Optimal Voluntariness Motivational Confounding Experimenter Bias Nontransparency Limitations of what is feasible to research

Interesting Deviations

Stern & Faber (1997) Milgram’s lost letter technique Milgram, sent on Stern & Faber, returned to sender From doing favor to doing easy task

Often effects are smaller

Getting Subjects: General

http://psych.hanover.edu/research/exponnet.html

Other Pages: Social Psychology:

http://www.socialpsychology.org/expts.htm The Web Experiment List:

http://genpsylab-wexlist.unizh.ch/

Getting Subjects: Special Populations

Advertising Email Groups

Careful, get permission Can be thought spam Netiquette

Discussion Groups Same, permission

Sample Method for Doing Survey

Birnbaum, M. H. (2001). Introduction to Behavioral Research on the Internet. Upper Saddle River, NJ: Prentice Hall.

Lots of resources at http://ati.fullerton.edu/ NSF sponsored classes at Fullerton on Social

Psychology One more next January, look for the

announcement at this site

First Step: Where to put it

Setting up a server can be easy Apache web server http://www.apache.org Most popular server Freeware Any computer connected to web is now a

server May have issues of firewalls & dynamic IP

addresses

Second Step: SurveyWiz

In Birnbaum (2001) Also linked to from http://ati.fullerton.edu/

Under Birnbaum’s page Freely available We will build a very short survey to illustrate

Forms

SurveyWiz uses web forms to collect data What are forms:

Elements to allow web page viewers to input This data is then sent back to the server to be

processed

SurveyWiz What the page looks like My Take Can edit it with any web page editor

Examples I have done this year

Caddell & Utt Pattison & Reese

Third Step: Collecting the data

First way, let Birnbaum do it for you SurveyWiz is set up this way <FORM

action=http://psych.fullerton.edu/cgi-win/polyform.exe/generic method=post>

FTP://guest:guest99@psych.fullerton.edu You can type in the address box of your browser.

Note that this is FTP site. At the present time, password is guest99, as shown above, and permits download only.

Keeping the Data Local

Second, get this perl script (found at http://psych.fullerton.edu/mbirnbaum/programs/PERL_script2.htm)

The script and instructions are at this site Written by Billy Schmidt

Also change code in form command At my school the line reads: <FORM

action=http://psych.hanover.edu/cgi-bin/survey_wiz.pl method=post>

What is CGI

Common Gateway Interface Method for other programs to interact with

web servers In this case, this perl program takes data

from web forms and stores them in a file Stores data in Excel or SPSS readable files

Fourth Step: Example Data File

Data File Data coding scheme: all variable names

begin two numbers, see example The perl program sorts the variables by

numerical order

Another Method of Survey Generation

Schmit’s Survey Assistant He manages data More flexibility http://or.psychology.dal.ca/~wcs/hidden/

home.html

What Do I Need?

At NSF ATI site List of software resources

What Problems are important

Musch & Reips (2000) – 5 point scale No control over participant’s behavior 3.6 No control over motivation

3.4 Inability of participants to ask questions 3.3 Nonrepresentative sample

2.9 Manipulation and fraud 2.4 Ethical problems 1.5

Problems

Eliminating Multiple Entries Dropout Security Data Integrity

Multiple Entries

People submit too fast, while waiting for feedback

People will fake being two people Usually have same IP address SurveyWiz and most other methods, sends

the IP address of machine where survey is being taken

Can eliminate more than one from same IP, e.g. Schmidt (1997)

Security & Data Integrity

If on public server, others can access data and download

Others might fake pages to send data (unlikely)

Keep data in non-public directories so only researchers have access

Have CGI check for origin of survey to make sure it is yours (Schmidt, 1997)

Dropout

People will come but not finish Or data is incomplete Survey will sends a complete signal to help

you track Can use: (O’Neil, Penrod, & Bornstein, 2003)

Short sweet Financial incentives. If multiple pages to survey, several pages warm-

up so not drop-out during

Ethical Issues

Should not think of Ethical Issues of Web Research Alone

Compare to Ethics of doing Traditional Research

Not let our comfort with what we have done blind us to those problems

Conclusion

Many Benefits Easy Methods Easy Resources Not alternative to traditional methods A new tool still needs to be used thoughtfully