Preparing for Manuscript Submission

CARMA Internet Research ModuleJeff Stanton

May 18-20, 2006 Internet Data Collection Methods (Day 2-2)

Standard Preparation for MS Submission

1. Report analysis of a cross validation sample2. Assess non-response bias; analyze and report3. Do duplicate detection; report4. Do malicious data detection; report5. Read and cite Internet research method reviews

May 18-20, 2006 Internet Data Collection Methods (Day 2-3)

Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals

Lack of access control leads to junk dataYou used a password protected consent formYou filtered responses using timestamp and IP addressYou detected similarities in (or identical) response patterns:

Flip data, run correlations, look for high valuesStudyResponse studies ranged from 0% to 6.9% duplicates

using this screening methodSimulations showed that repeats would be needed on more

than 20% of cases to substantially disturb means/correlations

May 18-20, 2006 Internet Data Collection Methods (Day 2-4)

Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals II

Your sample is bogus because of coverage errorsYou argue that sample representativeness is a challenge in all

research and that purposive sampling is a better goal anyway

Make the sample fit the question: An Internet survey of migrant workers? Coal miners?

You show the consistency of results between web and cross-validation samples

You argue that a typical group of Internet respondents has to be an improvement over pure undergrad samples

You cite demographic studies of Internet: increasing normalization to the general population over time

May 18-20, 2006 Internet Data Collection Methods (Day 2-5)

Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals III

No one has ever demonstrated the equivalence of the measures you used when administered over the web

You cite research that factor structures replicate, substantive conclusions replicate, correlations generally replicate within the limits of sampling error, be wary of mean comparisons

You argue this is a higher standard than many other published studies in which:

Researchers routinely make up their own itemsModify items or response options of existing scalesTrim scale lengths and field abridged versions

May 18-20, 2006 Internet Data Collection Methods (Day 2-6)

Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals IV

All Internet research participants are volunteers by definition and therefore volunteer bias makes your sample unusableBelmont report and federal legislation require all research to

be conducted on volunteers, so volunteer bias is endemic to the whole social research enterprise

Volunteer bias can substantially limit projectability of means, but my study doesn’t care about means

Studies and simulations of the effect of volunteer bias generally show that correlations are reduced in magnitude because of restriction of range effects that volunteer bias causes

May 18-20, 2006 Internet Data Collection Methods (Day 2-7)

Strengthening Research Plans for Web Studies

A cross-validation sample using traditional RMs is never a bad thing

Use your web sample only to make tests of correlative structures and self-referential comparisons of means (e.g., within subjects)

Don’t compare means from web study to means from prior paper and pencil study without formal equating

Speeded and objective tests need careful testing and cross-validation

Assess correlations between substantive variables to demographics: If they don’t correlate, then the non-response bias may carry less weight

May 18-20, 2006 Internet Data Collection Methods (Day 2-8)

Useful References I

Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407.

Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.

Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and Word-Wide Web samples on determinants of female attractiveness. Behavior Research Methods, Instruments, and Computers, 29, 264-269.

Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313.

May 18-20, 2006 Internet Data Collection Methods (Day 2-9)

Useful References IISmith, M. A., & Leigh, B. (1997). Virtual subjects: Using the

Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, & Computers, 29, 496-505.

Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.

Yost, P.R. & Homer, L.E. (1998, April). Electronic versus Paper Surveys: Does the Medium Affect the Response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology. Dallas, TX.