Using the Internet to Collect Primary Research Data

Postgraduate Research Methods Day Conference,

The Open University,

Camden Regional Centre

Claire Hewson,

Department of Psychology,

The Open University

claire.hewson@open.ac.uk

A (very) Brief History of Internet-mediated Research (IMR)

Pioneers started implementing online data collection methods around the mid-1990s Bordia (1996); Dillman (1991); Gaiser (1997); Hewson (1994).

Reports (e.g. validation studies) have since supported the quality of data that can be obtained in IMR Corley & Scheepers (2002); Hewson & Charlton (2005).

A new era of IMR (Web 2.0) The web as organic, collaborative, interactive, ever-changing (e.g. Wikipedia); the rise of social media (e.g. Facebook, Twitter, YouTube).

IMR Data Collection Methods

Obtrusive Approaches Surveys, experiments, interviews, participant and / or disclosed observation approaches.

Unobtrusive Approaches Observation using traces and archives, e.g. discussion group archives, browsing behaviours, social networking links (SNA); lurking?

'Big Data' Data sets which become so large as to be difficult to manage; presently a 'hot topic'.

The Visualisation of Complex Networks

Nike+ involves the placement of a sensor underneath the footbed of your Nike running shoe in order to collect data about where you've run, how long it took and where you can improve over time.

http://www.visualcomplexity.com/vc/

Visualisation of a year's worth of runs uploaded to the Nike+ website. It plays back runs throughout three cities: New York, London and Tokyo. The runs showed tens of thousands of peoples' runs animating the city and bringing it to life. The software visualizes and follows individual runs, as well as showing the collective energy of all the runners, defining the city by the constantly changing paths of the people running in it.

Web-based Surveys and Experiments

There is now a large range of software for helping implement these, particularly survey packages (which can also support simple experimental designs), e.g. 'Surveymonkey', 'Qualtrics'.

Dedicated packages to support experimental web-based studies include: WEXTOR (Reips, wextor.org/wextor/en) and WEBEXP (Edinburgh webexp.info).

Web-based Surveys and Experiments

New design issues and principles emerge in this novel context. Useful resources and guidelines include:

Exploring Online Research Methods: www.restore.ac.uk/orm

WebSurveyMethodology: websm.org

Couper (2008). Designing Effective Web Surveys. CUP: New York.

Reips & Krantz (2010). Conducting True Experiments on the Web. In Gosling & Johnson (eds) Advanced Methods for Conducting Online Behavioral Research. APA.

Fielding, Lee & Blank (2008). The Sage Handbook of Online Research Methods. Sage: London.

Recruiting Participants in IMR

Sample Bias Issues?, e.g. Bordia (1997).

True Volunteers, place adverts in public spaces, e.g. newsgroups, webpages, IMR study clearing houses, Facebook.

List-based Sampling Frame, e.g. email lists, online panel.

Probability Sampling Methods, e.g. random selection from known list, online probability panel.

General Principle: methods which allow some estimation of the sampling frame are preferable where measures of response rate, non-response bias, etc. are important to the research design.

Online Sampling Resources

● Online Study Clearing Houses, e.g. Online Psychology

Research: http://www.onlinepsychresearch.co.uk

● Newsgroups, SNSs, e.g. Facebook: facebook.com, Newsgroups (e.g. http://uk.groups.yahoo.com; https://www.groups.google.com), Mailing Lists (see the searchable Catalist database: http://www.lsoft.com/catalist.html)

● Online Panels, e.g. Knowledge Networks: http://www.knowledgenetworks.com/knpanel

Online workforces, e.g. Mechanical Turk: www.mturk.com

OU Participant Panel, Jim will talk more about this

OU ‘Virtual Participant Panel’

URL: http://www.open.ac.uk/socialsciences/psychology/cogpsypanel/

People can participate in studies ad hoc and/or… … can sign up to be contacted for future studies. Sign-ups give certain information (e.g. sex, age,

location, handedness) which can be used to filter if particular characteristics are needed in a sample.

Currently has 1,245 signed-up members. Has a built-in ‘Questionnaire Manager’ to create

surveys.

OU ‘Virtual Participant Panel’

Instruction text

Question text

Response text

OU ‘Virtual Participant Panel’

Stimulus image (‘question’)

Response images (‘answers’)

Response buttons

SurveyMonkey Example

● Computer Attitudes Surveyhttps://www.surveymonkey.com/s/computer_attitudes_study

● Qualtrics – site license for all OU social science faculty students / staff. Visit 'openss.qualtrics.com' and sign up with a valid OU (open.ac.uk) email address.

Martin discussed using Qualtrics in a radiology expertise study this morning, and Jim will demonstrate an online experiment implementation using Qualtrics.

Importing studies from SurveyMonkey into Qualtrics?

Qualtrics

Qualtrics

Stimulus image

Response images

Ethics in Online Research

Some ethics issues require special consideration in an IMR context, the complexities of which are not always obvious to researchers or participants.

e.g. Data security, traceability (confidentiality, anonymity).

Key feature of IMR: involves acquisition of data from or about individuals in absence of face-to-face co-presence

COUPLED WITH greater scope for carrying out quite complex interactive procedures

The New BPS Guidelines (November 2013)

Document aims to outline some of the key ethics principles for researchers and ethics committees to keep in mind when designing and / or evaluating an IMR study

Emphasis on how to apply existing ethics principles in an IMR context

“Thinking is not optional”

The New BPS Guidelines (November 2013)

● Centred around the four main principles of the BPS 'Code of Human Research Ethics' (2011).

1. Respect for the autonomy and dignity of persons

2. Scientific value

3. Social responsibility

4. Maximising benefits and minimising harm

Main Ethics Issues to Consider in IMR

Public-private domain distinction Confidentiality* Copyright Consent; withdrawal; debrief*● Levels of control*● Disruption of social structures

Valid Consent

Verifying characteristics (e.g. age, sex)

Engagement with consent process (check boxes?)

Adequate information – not all risks online are obvious, e.g. potential data leakage (balance between information and overload)

Risks will be higher in some contexts than others – key principle of proportionality of ethics procedures to levels of risk of harm re-emerges

Withdrawal and Debrief

Robust withdrawal procedures may be more difficult to implement online (e.g. volunteer web survey)

Detecting withdrawal may be harder (use of partial data?)

Ensuring debrief difficult (e.g. exit by closing browser)

Enhanced risk in cases of deception (no debrief)?

Principle of proportionality of ethics procedures to levels of risk (of harm) re-emerges, e.g. effects of withdrawal without debrief (such as in very sensitive research)

Confidentiality, Anonymity

● Enhanced traceability of data, dissemination implications, use of quotes.

Other data leakage risks, e.g. email.

Traces on personal computing equipment.

Sensitivity of data is a factor associated with levels of risk of harm.

Researchers should carefully assess risks, and perhaps inform participants of these risks.

Data Protection

UK Data Protection Act (1998).

Issue of storing data on non-UK/EU servers?

Popular software packages (e.g. SurveyMonkey, Qualtrics) are US-based.

'Safe Harbour Scheme'.

Office of National Statistics SurveyMonkey, Safe Harbour Scheme

Levels of control

Over procedures, who takes part, participant behaviours...

Implications for scientific value (validity)

Implications for levels of harm (monitoring participant reactions, presenting debrief info., etc.)

Maximising Benefits, Minimising Harm

These considerations permeate the other issues

Careful assessment needed; trade-offs will be inevitable (e.g. anonymity versus traceability/verifiability; disclosure versus social disruption/harm)

“Thinking is not optional”

AoIR http://aoir.org/

Some Examples

● http://www.socialpsychology.org/expts.htm http://www.onlinepsychresearch.co.uk http://psych.hanover.edu/Research/exponnet.html http://www.wexlist.net