Turk data analysis

Hadas + mitcho

{hkotek, mitcho}@mit.edu

Hackl Lab Turkshop

April 2013

Today..

Goal: prepare data for analysis, get (basic) results.

The Manage Tab

Approve subjects

Get data

Decode your results

R!

R basics

Prepare data for analysis

Calculate accuracy

Discard subjects

Look at results

2

NOT Today..

Statistics

Graphics

…. those need to fit your design and data

There is no ‘one size fits all’

Requires more knowledge (take a stats course!)

3

Materials

These slides and materials on ESSL website:

http://web.mit.edu/hackl/www/lab/turkshop/

Download and unzip “week4-results” in examples.

And if you haven’t done so, install R 3.0 and RStudio 0.97:

http://cran.r-project.org/

http://www.rstudio.com/ide/download/desktop

4

The Manage tab

5

Monitor the progress of your experiment

Approve/reject subjects

Get your data!

The Manage tab

6

Approving/rejecting workers

7

Reject subjects who did not comply with instructions.

Completed more than one survey

(often a requirement for language studies!).

Did not complete survey.

Possibly also:

Failed on ‘catch’ items.

Exhibit guessing behavior.

But NOT: Non-native speakers!

May bias participants into saying they are native speakers.

Approving/rejecting workers

8

Open your .csv results file in Excel.

Custom sort

by WorkerId

Find

duplicates

Conditional formatting:

highlight unanswered

questions;

decide if anyone

missed too many.

What we wanted to test: blocking example

9

Effect of crossing animacy of causee and verb type on

acceptability of causative sentences.

# blocking 1 inanimate-v

That’s the ball that the coach bounced on the floor.

# blocking 1 animate-v

That’s the gymnast that the coach bounced on the floor.

# blocking 1 inanimate-make-v

That’s the ball that the coach made bounce on the floor.

# blocking 1 animate-make-v

That’s the gymnast that the coach made bounce on the floor.

What it looks like from the workers’ end

10

Our Turk survey

11

Forced choice: natural = 1, unnatural = 0

10 items

5 targets

Four conditions each

5 fillers

Two clearly unnatural, three clearly natural

8 lists

56 total participants

Understanding and decoding our results

12

Open the .csv results file in Excel.

What information does it give us?

What does it not have?

Open the .decode.csv file in Excel.

What information does it give us?

Combine the two files using decoder.py!

Now open the decoded file in Excel.

What information does it give us?

Fields in the decoded results

13

WorkerId: Amazon ID for the worker

AssignmentId: unique to that submission

AssignmentStatus: Submitted, Approved, Rejected

WorkTimeInSeconds

ListNumber, PresentationOrder

Section, Item, Condition

These came from the decode file

field_N: your actual stimuli

In our template skeletons:

Choice: forced choice value

Extras: english, foreignlang, numanswered, useragent

R

14

Open source statistics software

Free! So lots of people use it.

Designed specifically for statistical data analysis

Generally treats everything as “data made up of multiple observations”

as we will see

Really, a programming language of its own

It’s a little weird and annoying

RStudio: a better interface for R

Open RStudio. Let’s play with some R! [Tutorial]

Writing an analysis script

15

In an analysis script, we (might) want to:

1. Read in the results

2. Filter out parts of them

3. Check filler accuracy and disqualify participants

4. Recode some of the data

5. Look at the results

6. (Compute statistics)

7. (Make pretty charts)

Writing a script forces us to precisify what we are doing.

It also makes the procedure reproducible.

Eliminates Reduces human error

Writing our analysis script

16

In RStudio:

Start a new script and save it in the same folder as the results.

Set working directory to Source File Location.

[Tutorial]

Pre-baked version in blocking-analysis.R