Living Lab' for HCI - presentation made at HCI International 2009

EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter

2009HCIInternationalConference,SanDiego,CA

  Asafield,earlyfundamentalcontributionsfrom:–  Computerscientistsinterestedinchangesinwayswe

interactwithinformationsystems–  Psychologistsinterestedintheimplicationsofthese

changes

  Combustible,because:–  Computerscientistswanttocreategreattools,butdidn’t

knowhowtomeasureimpact–  Psychologistswanttogobeyondclassicalresearchofthe

brainandhumancognition

  TheneedtoestablishHCIasascience–  Adoptmethodsfrompsychology–  GoodExamples:Fitts’Law,ModelsofHumanMemory,

CognitiveandBehavioralModeling,InformationForaging–  Dualpurpose:understandnatureofhumanbehaviorand

buildupascienceofHCItechniques.

7/24/09 HCIC "Living Lab" 2

  Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore–  Beyondauserinfrontofcomputer;Yetevaluationmethodsmostly

stayedthesame–  Controlledlabstudyasthegoldstandardforacceptance

  ChangesandTrendsinSocialComputingandUbiComp


Old Assumptions New Considerations

Single display Multiple displays

Knowledge work Games, communication, social apps

Isolated worker Collaborative and social groups

Stationary location Mobile and stationary

Short task durations Short and long tasks, and tasks with no time boundries

Controllable experimental conditions Uncontrollable experimental conditions

Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations

  Hardtogeneralizetonewtaskcontexts(withinterruptions,othertasks,othergoals,unfocusedattention,moredisplays)

  Hardtogeneralizetoothertools,apps  Ecologicalconsiderations

  Adoptionofmobiletechnology  iPhonesinJapan,single‐handedinput[PARC]  BestsellingphonesinIndonesiacomeswithacompass[Bell]

  Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups

  AggregatebehaviorofWikipediaorDelicioususers

7/24/09 4 HCIC "Living Lab"

  Conductresearchonrealplatformsandservices–  Nottoreplacecontrolledlabstudies–  Expandourarsenaltocovernewsituations

  Someprinciples:–  Embeddedintherealworld–  Ecologicallyvalidsituations–  Embracethecomplexity–  Relyonbig‐data‐sciencetoextractpatterns

  Notfirsttosuggestthis:–  S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On

ecologicalvalidityandubiquitouscomputing.HCIJournal,2008–  EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital

FamilyPortrait[Rowan,Mynatt]


GroupLens / MovieLens [Riedl, Konstan, Univ. Minnesota]

Games with a Purpose [von Ahn et al]


World of Warcraft [Yee, Ducheneaut et al]

Wikipedia History Flow [Viégas et al]

Bucket Testing or A/B Testing [Kohavi et al]

A B

UbiFit [Consolvo et al]


  Masterdegreewasincomputationalmolecularbiology  Analogy:Justasbiologistsworkonmodelplantsand

genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.

  Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.


  Twodimensions–  1.Whetherthesystemisunderthecontroloftheresearcher–  2.Whetherthestudyisconductedinthelaborinthewild

System Control System Not in Control

Laboratory (1) Build a system, study in the Lab

(2) Adopt a system, study in the Lab

Wild (Real World)

(4) Build a system, release it, study in the Wild

(3) Adopt a system, study in the Wild


  TraditionalApproach;Numerousexamples  FavoredbyHCIfieldreviewers  Typicalsituationisthestudyofsomeinteractiontechnique

–  Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput

  Typicalmeasuresarequantitativeinnature–  performanceintime,performanceinaccuracy,eyetracking,learning

measures,userpreferences

  Issues:–  Notalwaysecologicallyvalid–  Hardtotakeallinteractionsintoaccount–  Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.


  Hardertofindintheliterature  Oftencomparingagainstanoldersystemasbaseline  Typicalcaseiscomparisonoftwosystems

–  (onewebsitewithanother,onewordprocessorvs.another)–  Whichhighlightingfeatureworksbetter–  Twotextinputtechniqueonacellphone

  Typicalmeasuresaresimilarto(1)  Issues:

–  Somesimilarissuesto(1)becauseit’sinlab–  Systemfeaturenotincontrol,sonotabletocomparefairly,or

isolatethefeature






Wild (Real World)




  Realapplicationsinecologicalvalidsituations  Realfindingscanbeappliedtoarunningsystem  Impactofresearchismoreimmediate,sincesystemisalready

running  Typicalcaseisloganalyticswithlargesubjectpools

–  logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.

  Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis

  Issues:–  Factorsnotincontrol,findingsnotcomparable–  Factorscannotbeisolated–  Reasonsforfailureisoftenjustguesswork


  Hypothesis:ConflictiswhatdrivesWikipediaforward.  Howtostudythis?

–  JohnTukeyparadigm–  Getalargepaper,andplotallofthedata!

–  DownloadedallofWikipediaandalloftherevisions–  Hadoop/MapReduce,MySQL,etc.


7/24/09 21

60%

65%

70%

75%

80%

85%

90%

95%

100%

2001 2002 2003 2004 2005 2006

Perc

enta

ge o

f tot

al e

dits

Article

User

Article Talk

User Talk

Other

Maintenance

HCIC "Living Lab"

Group A

Group B Group C

Group D

Number of users in user group A B C Total

Users with Korean point of view 10 6 0 16

Users with Japanese point of view 1 8 7 16

Neutral or Unidentified 7 3 6 17 7/24/09 22 HCIC "Living Lab"

Mediators

Sympathetic to parents

Sympathetic to husband

Anonymous (vandals/spammers)






  Hypothesis:SocialTaggingdoesn’tscaleovertime.  Howtostudythis?

–  Crawlasmuchtaggingdataaswecan.–  Studythenoiseinthesystem.

–  40machinesfor3months


Topics

Users Documents

Tags

T1…TnEncodingDecoding

Noise


Concepts


Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)

31 7/24/09 HCIC "Living Lab"

7/24/09 32

Guide

Web

Howto

Tips Help

Tools

Tip

Tricks

Tutorial

Tutorials

Reference

Semantic Similarity Graph

HCIC "Living Lab"






Wild (Real World)




  Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.–  Goodforcasesinwhicheconomicsmakessense[Google]–  Changestosystemispossible;Factorscanbecontrolled.

  TypicalcasemightbeA/Btesting,largesubjectpools  Typicalmeasuresarebeingdeveloped

–  Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?

–  Usabilitymeasuresvs.Usefulnessmeasures

  Issues:–  Effortandresourcerequirementisdroppingbutstillsignificant–  Hardforaresearchlabtotakeon


7/24/09 36

HCIC "Living Lab"

7/24/09 37

HCIC "Living

Lab"

7/24/09 38

HCIC "Living

Lab"

Personal Computing [Xerox PARC]

  Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.

  Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.


Characteriza7on Models

PrototypesEvalua7ons

  ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.

  LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.

http://asc‐parc.blogspot.comhttp://[email protected]

WikiDashboard MrTaggy SparTag.us

Technology

Living Lab' for HCI - presentation made at HCI International 2009