Debriefing in Online Experiment

Harryadin Mahardika, PhD

Harryadin Mahardika• Pop Economist

• FEUI & laporsuap.com

• Research objective:– “to liberate and empower consumer...”

• Current research:– Consumer empowerment

– Consumer intervention/engineering

– Mobile advertising

• Contact:– harryadin.mahardika@ui.ac.id / harryadin@gmail.com

– @HarrySastro

2

Discussion Agenda

• Informed consent• Debriefing• Randomization• Participant incentives

Risk of experiment on human subject

• Deception, manipulation, priming, scenario.– May harm participants’ mind.– Mental fatigue.

• Examples?

Informed consent• Information about the experiment and its risk.

– As a basis for participants to decide after they understand what the research involves (risks and benefits ).

• Written consent vs unwritten consent.– Written:

• Investigators must typically obtain and document voluntary informed consent from research subjects.

– Unwritten:• button on an online form to indicate they have read and

understood the consent form.

Informed consent in online experiment• Limited interaction with participants

– investigator often cannot tell whether a subject understood the informed consent statement.

• Online form:– Researchers can increase the likelihood that subjects are granting truly

informed consent by requiring feedback from subjects about their level of understanding,

• Example: – by requiring a “click to accept” for each element in an informed consent

statement or even administering short quizzes to establish that a subject understood.

• Reduce response rate:– Increase nonresponse to sensitive items (Singer, 1978)– Possibly produce biased data (Trice, 1987).

Risk in online experiment• It exposes subjects to innocuous questions and benign

or transient experiences with little lasting impact. • In general, online experiments is no more risky than any

of their offline counterparts. • In some respects, they may be less risky:

– The reduced social pressure (Sproull & Kiesler, 1991) in online surveys or experiments makes it easier for subjects to quit whenever they feel discomfort.

– This freedom to withdraw is no trivial benefit, given the strong pressures to continue in face-to-face studies (e.g., Milgram, 1963) and even telephone calls.

Risk in online experiment• Although risk in online settings is typically low,

the actual risk depends on the specifics of the study.

• For example:– Some questions in a survey or feedback from an

experiment may cause subjects to reflect on unpleasant experiences or to learn something unpleasant about themselves

– e.g., Nosek et al.’s, 2002b, research on automatic stereotyping.

Risk in online experiment

• Experiments that deliberately manipulate a:– subject’s sense of self-worth, – reveal a lack of cognitive ability, – challenge deeply held beliefs or attitudes, or – disclose some other real or perceived

characteristic

..... may result in mental or emotional harm to some subjects.

Debriefing

• American Psychological Association (2002) ethical guidelines call for debriefing subjects:– “Providing an explanation of the nature, results,

and conclusions of the research—as soon after their participation as practical.“

• If deception was involved:– Researcher needs to explain the value of the

research results and why deception was necessary.

Debriefing in online experiment

• When conducting research online:– Researchers can post debriefing materials at a

Web site, – Provide debriefing materials to those who leave

before completing the research (Nosek, Banaji, & Greenwald, 2002a).

– For example, researchers can deliver debriefing material through a link to a “leave the study” button or through a pop-up window, which executes when a subject leaves a defined Web.

Debriefing in online experiment

• Appropriate debriefing in online research may be difficult:– The absence of a researcher in the online setting

makes it difficult to assess a subject’s state.– Difficult to determine whether an individual has

been upset by an experimental procedure or understands feedback received.

Extraneous Variables cont.

Selection Bias Improper assignment of test units to

treatment conditions[sampling error]

Extraneous Variables cont.

MortalityLoss of test units while the experiment is in

progress[respondents selected no longer wish to participate]

Controlling Extraneous VariablesRandomisation Randomly assigning test units to experimental groups by using random

numbers

Matching Comparing test units on a set of key background variables before assigning

them to the treatment

Statistical Control Measuring the extraneous variables and adjusting for their effects through

statistical analysis

Design Control Use of experiments designed to control specific extraneous variables

Case: Beauty is in the eye of the beholder

Scientists have found a link between drinking alcohol and perceptions of beauty

80 students were shown colour photographs of 120 male and female students and were asked to rate the aesthetic properties on a 7-point scale from high unattractive to highly attractive

Half the students had drunk up to four units of alcohol, the other half had no alcohol.

The students who had consumed alcohol rated the people in the photographs as more attractive than the student who did not consume alcohol.

Source: http://www.theage.com.au/articles/2002/09/091031115991721.html