User-Centered Design

SWE 632Spring 2018

© Thomas LaToza

Looking ahead• Examined human cognition

• Have 2 ways to identify usability issues

• But... is HCI just identifying usability issues?

• What does design mean?

• How do we learn about user needs?

• How do we build designs?

• How do we evaluate designs?2

User-centered design

3

User-centered design

4

What problems may users

encounter w/ current ways of

doing things?

Who are the users?

What are the user’s needs?

What are the user’s tasks and

goals?

What extreme cases may

exist?

How does the product fit into the broader context of their lives?

Technology-centered design

5

What can this

technology do?

How might users use it?

What features

does it have?

Double diamond model of design• Question problem,

expand scope, discover fundamental issues

• Converge on problem

• Expand possible solutions

• Converge on solution

Fail fast• “Fail frequently, fail fast” David Kelley, founder of Ideo

• Failure is learning experience

• Crucial to understand correct problem to solve & ensure solution is appropriate

• Abstract requirements are invariably wrong

• Requirements produced by asking people what they want are wrong

7

Iterative model of design

8

(re) Define the problem

Needfinding

Brainstormideate

understand the users

Prototype

build

Test

Iteration• Repeated study and testing

• Use tests to determine what is working or not working

• Determine what the problem might be, redefining the problem

• Collect more data

• Generate new alternatives9

Flexibility-usability tradeoff

11

Flexibility-usability tradeoff• Jack of all trades, master of none

• Better understanding needs enables specialization and optimization for common cases

• System evolution over time:

• flexibility —> specialization12

Navigating a design space

13

• What are key decisions in interaction design

• What alternatives are possible

• What are tradeoffs between these alternatives

Hierarchy of design decisions• What are you (re)designing?

• the width of the text input

• the maximum length of a valid username

• when in the signup process users enter their username

• if the user must create a username when signing up

• whether users are anonymous or have a login

• if users can interact with other users in your application

14

Picking the right level of redesign• Where are the user’s pain points

• What are the underlying causes

• What would be the value to the user of addressing issue

• What do you have time to build (or change)15

Activities and tasks• Activity - set of tasks performed together for a common goal

• Go shopping

• Task - component of an activity, organized cohesive set of operations towards a single low-level goal

• Drive to market

• Find shopping basket

• Find item in store

• Pay for items16

Activities and tasks• Activities are hierarchical

• High-level activities spawn other activities, spawn tasks

• Software supports tasks and activities

• Important to design for activities, not just tasks

• Support whole activity seamlessly

• Ensure interactions between tasks do not interfere17

Example - iPod• Supports entire activity of listening to music

• discovering music

• purchasing music

• getting it into music player

• developing playlists

• sharing playlists

• listening to music

• ecosystem of external speakers and accessories

18

Example of a design process• How do you get from let's make listening to music better to designing

an iPod??

• Iterative design...

• But what does that actually look like more concretely?

• What insights into activity help inspire design?

• How does watching users help lead to these insights?

• How do insights translate into an actual real design?

• How do know the new design is actually better?19

Example

20

Domain: Debugging• Design goal: how do we better support activity of debugging in large,

complex codebases?

• Build a better debugging tool (?)

• What should it do? How would it help?

• Design a better watch window?? Support new types of breakpoints?

• What's really the key steps in debugging that lead users to struggle the most?

21

Observations of developers in the field

22

Par$cipants Tasks~90minutes

pickedoneoftheirowncodingtasksinvolvingunfamiliarcode17professionaldevelopers

Transcripts

(386pages)

InteresAng.Thislookslike,thislookslikethecodeisapproximatelythesamebutit’srefactored.Buttheothercodeis.

Changedwhatflagsit’s???

HeaddedanewflagthatIdon’tcareabout.Hejustrenamedacouplethings.

Well.

Sothechangeseemedtohavechangedsomeofthewaythesethingsareregistered,

butIdidn’tseeanythingthattalkedatallaboutwhethertheappisrunningorwhethertheappisbooted.Soitseemslike,thiswasuselesstome.

(annotatedwithobservernotesaboutgoalsandacAons)

Ac$vi$es

Coding activities working with unfamiliar code

23

Circlesize:%ofAme Edgethickness:%oftransiAonsobserved

ReproduceDebug

Investigate

Test6%

33%

28%

4%

11%

16% 5%Edit

Reuse

Compile50%

50%28%

40%

12%

20%

11%86%

3%

22%

67%11%

14%

22%

20%

18%

29%

11%86%3%

55%32%

5%6%

Longest activities related to control flow questions

24

4outofthe5longestinves$ga$onac$vi$es

5outofthe5longestdebuggingac$vi$es

Primary question Time (mins) Related control flow question

How is this data structure being mutated in this code? 83 Search downstream for writes to data

structure“Where [is] the code assuming that the tables are already there?” 53 Compare behaviors when tables are or are

not loadedHow [does] application state change when m is called denoting startup completion? 50 Find field writes caused by m

“Is [there] another reason why status could be non-zero?” 11

Find statements through which values flow into status

Where is method m generating an error? 66 Search downstream from m for error textWhat resources are being acquired to cause this deadlock? 51 Search downstream for acquire method

calls“When they have this attribute, they must use it somewhere to generate the content, so where is it?”

35 Search downstream for reads of attribute

“What [is] the test doing which is different from what my app is doing?” 30 Compare test traces to app traces

How are these thread pools interacting? 19 Search downstream for calls into thread pools

Longest debugging activity

25

RapidlyfoundmethodmimplemenAngcommand Unsurewhereitgeneratederror

StaAcallytraversedcallslookingforsomethingthatwouldgenerateerror

Trieddebugger

Didstringsearchforerror,foundit,butmanycallers

Steppedindebuggertofindsomethingrelevant

StaAcallytraversedcallstoexplore

WentbacktosteppingdebuggertoinspectvaluesFoundtheanswer

(66minutes)

Whereismethodmgenera$nganerror?

debugger

staAccalltraversal

grep

debugger

staAccalltraversal

debugger

3Sowe’llgothereandwe’lljustcrawlthroughthiscodeandwe’lltrytounderstandthat.Sothiscodehassomeotheroptionsinit.So,I’mjustscanningthroughtojustunderstandwhatthisisdoing.Typicallythesefunctionslookforsubcommandsforthemaincommand.Souhas[lookingcasestatementlookingatcharacterenteredbytheusertodispatchonwhatcommandtoexecute]onefunctionality.Andubhasanother,andufhasanother.Sothat’swhatthiscodeisactuallydoing,henceparse.4Andtheguythatwrotemostofthiscodewasprettyconsistentwithhiscodepatternsforhowhedoesstuff.So,againthefunctionnamesareidicativeofwhatisgoingon.AndhemakesheavyuseofElagspassingaround.Somoreprecisely,whatI’mlookingfor,I’mlookingforwhoisactuallyreturningthismemoryaccessstring.SoIdon’tseeanythingjustscrollingthroughthisfunction,clearlyit’snotthisfunction,butthisfunctioncallsabunchofotherfunctions,soIcouldwalkthroughallofthecallstotrytoisolatethat,butI’mgoingtoseeEirstifIcangetluckyandnarrowitdownfromtheotherendandlookatwheretheoutputiscomingfrom.5Searchingtheentireprojectandwe’rejustgoingtodoastringsearchforalloftheprojectandseeifthatcomesupwithany,withbasicallywherethatoutputcomesfrom.[doingsourceinsightsearch]So,luckilythisdoesn’tseemtobeapieceofoutputthatgetsspewedeverywherewhichisnice,butitseemsprettysparse.Andinfactsearchingtheprojectdidn’tactuallyEindthatatall.SoI’mreallynotgoingtobeabletoworkbackwardsfromtheerrorstring.6Sowe’llgobacktothesourceEileitself,sowe’llgobacktotheoriginalparseEile,sowehavenooptions.Nomodes,sowe’reinthedefaultmode,sowe’regoingintothisinstruction,thisisjustElags.I’mjusttryingtogetafeelfortheparametersthatwearedraggingalonghere.Idon’tknowifyouwanteditorialcommentoncodeorcodetoolswhileI’mwalkingthroughthis.“Ah,whatever,it’smostlyjustwhateveryou’rethinkingaboutwhileyourworkingonthetask,ifthat’swhatyou’rethinkingaboutthat’sEine.”7[laughing]Yeah,itwouldbeniceiflookingatthisfunctionontheparametersthemselveswereoverlaidwiththetype.Sothishasamouseoverthatissomething,butitdoesn’tactuallytellmewhatthetypeis.Soagainlookingatthefunctiondeclarationagain???SowhatIreallywanttoknowiswhichoneistheaddressthatI’mactuallygoingtodisassembleandbeonthat.SoSorightnowI’mmostlyjustreadingthecodeandtryingtounderstandstuff.ButafewthingsIdonoticieisalotofthelackoftheinitialization.SosomeofthatImightchangeisIultimatelydoeditthisfunction.[lookingatthemethodthatiscalledfromdispatchandthatdoestheactualworkforthesubcommandofinterest]8Justbecauseit’sannoying.Soscanningthorughhere,I’mjustlookingforthecallsareandwherewegonext,orwheretheoutputis,becauseagainI’minterestedinwhoisputtingthatoutputinthere.Sohereisthisfunctioncall,machinedisassemble.Here’saplacewheresourceinsightfallsshort,it’sshowingmethewrongpreviewforthedissasmeblefunction.[littlepreviewwindowinbottomofsourceinsightwindowforcallee]Iknowthatbecausethisisamemberfunctionandthisoneisnot.Thisisthewrongnumberofparameters,blah,blah,blah.Soagainwehavetogobacktobrowsetheprojectsymbolsforthedisassemblefunction.9Soagain,lotsofdifferentones,butIhavewhichobjectthisis.SoifIgobackhere,wearelookingatthemachineisourobjecthere,anditisamachineinfostruct,sowecangobackhereto[Eiguringouttypeofobjecttoreasonaboutdynamicdispatchformanuallygoingtocallee]gotothemachineinfoversionofdisassemble.Sothisisinterestingbecause,nowwe’reoutsideofwhatthedebuggeritselfisdoingandwe’renowinthedebuggerAPIs.Sothatmakestheriskofachangehigher.There’smoreofaregressionrisk,becauseit’snotjustthedebuggerthat’susingit,10it’sallthedebuggerincludingonesthatarenotours.SoI’mjustscanningthrough,typicallylookingatstufflikethisI’mjustinterestedinhowbigthefunctionis,howmanydifferentbranchesitcouldtake,howcomplexitisgoingtobetoEindoutwherewearegoing,justfromreadingthecode.OfcourseIcanattachthedebuggertothedebuggerandwalkthroughthatwhichisprobablywhatI’mgoingtodohereinaminute.11Yeah,sothiswillbealittleeasiertounderstandifIactuallywalkthroughthecode.SoI’lljustopenanotherdebuggersessionandattachittothisEirstone.[startsasecondwindbg]Sowestartedwithparseunassemble,whichisgoingtobeindbgeng.12Mighthavetolineupsymbolsforthis.Sowe’llwaitonthat,we’llgobackhere.[tosourceinsight]Decode.SojustscrollingthroughthefunctionandlookingattheEirstactualfunctioncallthatwewillmakegoestothisdecodefunction.AndagainI’mjustlookingforwherethatoutputcomesfromorifwewouldsetadifferentsetofbrnaches.Thecommentsare,thisisnicelycommentedcodewhichisraretosaytheleast,soit’sactuallyalittleeasiertotryandthrowoutpiecesofcodethatareprobablynotrelatedtowhatI’mlookingfor.BecauseIhavesomeinnateknowledgeastowhatI’mlookingfor,andthiserror[againthinkingaboutreadingsourcecodeasaEiltering/searchtask]isactuallymostlikelycomingfrom,we’rereadinganaddressthatisnotinthedumpEile.SoI’mlookingspeciEicallyforreadmemoryorreadpointerorstufflikethat.[he’sright–itdoesendupbeingfromoneofthesecalls]Butsincethisentirecodebasecalls,soIknowthatthatisgoingtobesomethinglikereadptrorreadvirtual,butIalsoknowthatthere’sabazillioncallstothatfunction,andit’snotveryeasytonarrowitdownthatway,soIcan’tgoaboutitthatway.[wantsdostringsearchofcalleetreeidentiEiers]14Sothedebuggeroverherecameback,sonowIcangogetsymbolsforthisversion.SoIcanpickthesymbolpathinthedebuggersothatIcanwalkthroughthecode,andagainwewaitalittlebitsowe’llgobackoverhere.[tosourceinsight]15Thispartofthecodeisactuallytakingaparttheinstructions,sobythispointwealreadyhavethedata,sothereaddatawouldhavealreadyoccurred,andwewouldhavefailedbythepointthatwegottothiscode.Sowe’llgobackhere,weneedtogobacktothewritedisassemble,Ibelievethisistherightone.[sourceinsightsymbolbrowserforit]16Sowe’llassumethatdecodefailed,butifwedothat,ifitfailstotallythenwewouldjustexit,whichdoesn’tseemtobewhat’shappening.Becauseotherwisethisfunctionwouldn’thavethistextoutputthatwe’reinterestedin.17SotheotherthingsthatInoticedwhenIwaslookingatthedeEintionsforunsassemblewhenwecalleddisassemble,there’sabunchofmachinespeciEicimplementationsofdisassemble.Soitcouldbethatwe’renotactuallycallingthemachineinfo,therecouldbeanx86onethatweareactuallycallingsincethisisdebuggingx86code.Somyia64version,whichapparentlyIdon’thavecodeforormaybeitwasremovedfromtheproject,samething,sothere’sclearly,sotheremightbesomethingwrongwithmyprojectwhichiswhythere’ssomanydeEinitionsfforthis.18Ok,sothedebuggeroverherecamebacksoIcanjustsetabreakpointonparseunassembleandthenwalkbackthroughthecode,ohweactuallydon’tthatonebececausethatoneisgoingtosucceed,wewantthefailurecasewhichisthisone.[demonstratedsomebehaviorandgotacallintoittwice]Ok,soweareatparseunassembled,sowe’llmakethedebuggerlookatthesamesourcecodethatwearelookingatinsourceinsightoverhere.Andthedebuggershould,ifitcanEindthecode,maybeitdoesn’tlikethiscodepath.ThatwilldeEinitelymakeithardertowalkthroughthecode.19Sowe’llgobackintodisassemblehere,sincethere’snotreallyabetterimplementationthatisabletodoit,we’llgobacktothemachineinfoone.Whatwouldreallybehelpfulhereistoknowwhatcodepathsaremostcommon,likethemetadatathatpreEixprovides,orsometracingtools.Ifthatwassomehowoverlaidwiththesourcecode,thenyoucouldseewhatcode20wasdeadeffectively,orwhatcodegetsrunincertainenvironments,we’lljustputthatinthepipedreampile.“Soyoujustwanttoseewhat’salwaysexecuted?”Soitwouldbenicetosee,solikepreEixonlydoesasetnumberofpaths,butlikeBenLiblithasaproject,you’refamiliarwithhim?“He’sfromwisonsin”Yeah,he’saresearcherfromWisconsin,hisstatisticaldebuggingishisthing,andhehasallthistracingstuffthatcomesupandbackandforth.Sothat,thethingaboutlookingatfailuredata,becausewehavefailuredatatoo,wecanseewhatcodepathexecuteswhenthingsfault,whatcodeexecutescommonlywhenstuffworks,soifwehadsomewaytosayinthesourcecode,becauseIcandoitfromthedebugger,21butIhadsomewaytosayinthesourcecode,ok,ifIgiveyouthesevalues,whatpathswillexecute.WhichIguessiseffectivelydebuggingthecode.“Soyou’dwanttospecifythosevaluesatfunctionentryratherthanjustrandomlyendupwiththevaluesfromplayingwiththeUI?”IthinkwhatI’msayingisthat,givenafunctiondeEintion,IEillinasetofvalues,sowhathappensifthisguyisnull,andthisguyisalsonull,[writingassertsonparams]“Makeabunchofassertsessentially”Yeah,itwouldbasicallyhighlightinthecodewhichpathsaregoingtoexecute,somethinglikethat.“Whatwouldyouusethatinformationfor,howwouldthatchangehowyouarelookingatthismethod,itwouldhelpyouruleoutpieces?”22Yeah,itwouldhelpmeruleoutwhichpathsweregoingtoexecute,socommonlywhenI’mlookingatcode,eithercodethatI’mfamiliarwithintheprojectthatIworkedoncommonlyorbecausemyjobispartiallytodebugeverybodyelse’scode,soalotofthetimeIhavecrashdumpsthatsaywhatthestateatthetimeofthefailurewas,andIhavethesourcecode,butIhavetodoalotofeitherqualiEicationofvaluesinthedebuggeritselforabunchofguessingwhateverinmyheadtotryandEigureoutwhichpaths,becausewe’relookingatastaticpointoftimeinthedebguggerandastaticpieceofcode.Andthe2won’t,youcanoverlythetwo,butyouwon’tnecessarilyknowwhichpathsexecuted,soyouhavetokindofwalkthroughbackwards.So,butIdohave,ingeneralIdohavetheparameters,thisisnull,thisisnotnull,thisisthisstaticvalue,thisisstaticvalue.23SoifIcouldoverlaywiththesource,sothatmight,forsomeofthesesigniEicantlylongerfunctions,itwouldhelpmeunderstandwhat’sgoingonthere.TheotherthingthatIdoalotwhenIlookatcodethatIown,I’mtypicallylookingforplacesthat,thisisforstuffthatImuchmorefamiliarwith,I’malwaysinterestedinwhatsortofthingscouldberefactored.WhereIcouldImakeafunctionsmaller,wherecouldIreducethenumberofparameters.Sohavingarefactoringmodeinthesourceeditorwouldbehelpful.Slickedithassomeinterestingthingswhereyoucanhighlightasectionofcodeandslickeditwill,ifyou’regoingtorefactorthis,thenyoualsoneedtodragalongtheselocalsandtheseparameters,andtheyhavetobepassed,anditmakesyourfunctiondeEinitionforyou.24Sothat’sveryinteresting.So,anyway,we’rebacktothis.Itdoesn’tlikemysourcepath,ohbecauseI’mgivingitthewrongone.[stilltryingtoloadsymbolsindebugger]25[waitingonittotrytoloadsymbolsagain,backtosourceinsight]Ok,againthecommentsarehelpful,becauseIcanbasicallyignorethisbranchbecauseI’mprettysurethatthedecoderdidn’tfailandIdon’tseethisoutput.[reasoningaboutwhatbranchesweretakenbasedonoutputbehavior]Butthisisinterestingtoseethisoutputinthecontextofthat,Iwaslookingforapieceofoutput,becausethisoutputissplitacross2sourcelinesasifsomeonehadasigniEicantlymorenarrowsourceeditorview.SothatmightmeanthatoneofthereasonsthatIcouldn’tEindthestringIwaslookingforbeforewasbecauseitwaswrapped.SomaybeifIgobacktomysearch,Iwassearchingfortheentirestring“memory26spaceaccessspaceerror”somaybeifIjustmakeitmemoryaccessandletitsearchalong,andthatEindssigniEicantlymoreentires,includingoneinutilsdotcpp,inatableoferrorstrings.[goestothatreference]Sowhereverthatguywas,thereyougo.Sothisislikeaninterestingsearchproblemingeneral.Actually,Idon’tthinkgoogleorlivesearchdothis,butifyougiveasetof4individualsearchterms,usuallyyougetallornothingfromasearchengine.Soyougetthesetofresultsthatgetall4terms,orinthiscaseall3terms,ornoterms.27Butyoudon’ttypicallygetatreedsetofterms,herearethesetofresultsthathaveallofyourterms,hereisthesetofresultsthathaveallminus1,allminus2,allthewaydownto0.Butinthatcase,thiswouldhavebeenveryhelpful,thiswouldhavepotentiallysavedmeagoodbitoftime.SoI’mlookingforacalltoerrorstringwiththeerrorvaluememory.[wantsthecallertothismethodwithaparticularparameter–theenumthatforcesthecasewhereitprintsthestring]Sowecanseehowmanycallerstherearehere,ok,sothere’saprettylargenumberofcallersofthis.Maybewecanlookatwherethosecallersareandnarrowthatdownbasedonwhatweknow.Sotherearealotintypeddata,alotinsystem.28SpeciEicallywe’relookingforcallingerrorstringwiththeEirstparameterofmemory,butthisisanothercasewheresearchgenerallyfailsingeneralbecauseofspacing.Sothisiserrorstringopenparen,andthenthewordinallcapsmemory[(MEMORY)].Butthere’sallsortsofpermutationsofhowthatcouldbespacedandstillbelegitamitecompilablecode,sowe’llstartwiththisoneandseeifwegetanything.Whichwedon’t.Sowe’llgobackhere29alotofspaces,errorspace,openparenspace,andthewordmemory,andwe’llsearchforthat.[stillnothing]SoI’mdonetryingtodothat.Solet’slookatcallersoferrorstring.[backtootherstrategyoflookingthroughcallers]Somaybeifwejustparsethroughhere,orstepthroughhere,wecanseewhichonesarecallingwiththeparameterofmemory.Butunfortunately,manyofthesearecallingwiththeEirstparameterasavariable.Sothatwouldmeanthatwhatwewerelookingatbeforeisnotasearchproblem,it’savariableinterpretationproblem.30So,I’mjustkindofsteppingthroughthesevalues,andinmyhead,I’mjusttryingtorememberwhichonesarelegitimateandwhichonesmightnotbe.SoitwouldbeniceifIcouldjusttakethiswholelistofresultvaluesandselectthemalloutofthiscombobox,andthenpastethemintonotepad,soIcouldthenremovethemfrommylist.SoIwouldn’thavetojustworryaboutrememberingthem.31Ithinkthat’ssomethingthatItendtodoalotwhendebuggingaswellasreadingcode,isthatIendupwithlotsofclipboarditems,butnotclipboardinthesenseofyou’resharingtextbetweenapplications,butclipboardinthesenseofthesearelittlehintsonwhichpathsIwentdownandwhichpathsIdidn’t.“Soyouwanttomakesureyou’renotrepeatingpaths,andthatyou’repursuingallthepathsthatyoumightwanttoreasonablypursue?”AndmorewhatIwasthinkingatthetimewhenIomittedapathorconsideredapath.SosometimeswhenIamactuallyeditingthecode,Iwillgothroughanoutputnotlikelytobethepathbecauseofthis,andthenalotofthosecommentsIwouldthenclipoutbeforethecodegetssubmittedbecausetheyaremostlyjustcodereviewercomments.Andtypically,that’ssomethingthatweseeincollaborativeworddocs.32It’sprettytypicalthatyou’llcollaborateonaworddoc,andpeoplewillputcommentsinlinewithstuff,butit’salittlelesstypicalforsourcecode,sourcecodecommentstendtobemissingintotalorthecommentbytheactualdeveloperorthemaintainer.There’snotreallyaplaceforcommentsforreaders.Thismayhavebeenperfectlyclearforthedeveloperwhowroteit,thesourcecodemaintainermightunderstandit,butthethousandsofotherpeoplewhoaregoingtoreaditfordebugging,forcustomers,forthedevelopersthemselves,there’sreallynoplaceforthemtoputcomments,andmaybethereshouldbe.“Whatstopspeoplelikefromjustcheckingthecommentintothesourcedepot?There’sjusttoomuchoverheadandyoudon’t’havetheauthoritytodothat,oryoudon’townthatcode?”33Ithinkit’snotnecessarilyauthority,it’srespectforone.Becausethisissomebodyelse’scode,sounlessyouaregoingtomakeanetpositivechange,Iwouldn’teffectapieceofcode.AndIwouldn’tconsidercommentstobeanetpositivechange,althoughmaybeIshould.Usuallyit’snotpermissions,itsusuallythischangedoesn’tneedtobepersisted.Orinmyopinion,itdoesn’tneedtobepersisted.Youcouldendupwithaveryinterestingsourcetreeifyoujustopenedituptoadhoccommentsbyanyreviewer.34Firstyou’dhavetoassumethatpeopledoit.Butthenyougetintoatoolsuchasacompilerthatwilldecideifyouarecorrectornot.Soifyouputacommentintherethatsaysthatthisisgoingtodothisinthiscaseandyou’rewrong,there’snocheckingforthat.“Youneedsomesortofauthorityaboutwhothispersoniswhoismakingthecomment”Maybethatwouldhelp.Ormaybe,justlikeyoudoforchanges,thatchangeiseffectivelyasuggestionthatyousendforreviewtothecodeowner.Maybewedothatwithcommentsto.Butthenthatputsanadditionalconstraintonthedeveloperswhichisalreadyanoverusedresource,orabusyresourceifwe’reexpectingdeveloperstodothis.So,35noneoftheseareclearlywhat’snotgoingonhere.Alotoftheseareclearlyunrelated,butverygeneric.Sothiserrorhereisbeingusedbyalotofverygenericfunctions.Which,Iguessisprettytypicalforc++code.Iguessanyobjectbasedcode,becauseyouseeitinC#andseeitinjaveaswell,soyouhavelotsofclassesandmembersthatoverriding???,soyouendupwithlotsofwrappers,aroundwrappers,aroundotherwrappers.Andit’sapuzzleinitselftoEigureoutwhichoneisactuallyofinterest.36Sowe’llgobacktothedebuggertoseeifIcanactuallywalkthroughsomeofthiscode.SoitlookslikeI’mnotactuallygoingtogetsourceinhere,whichisunfortunate.Sowe’lljusthaveto,we’llhavetokindofguesswhatisgoingonhere.[lookingthroughassemblytoseefunctioncallsamidstlotsofotherassembly]SosomeofthesefunctionsIdon’trememberseeingintheparsefunctionwherewestarted,soI’mgoingtogolookforthose.37Sothere’sgetrange.SotheotherthingthatIknowaboutthisisthatI’mrelativelycertainthattheoutputwillcomefromoneoftwofunctions.Sowe’llsetabreakpointonthose,andwe’llseewhothecallerofthoseis,butclearlyI’mwrong,becausewedidn’tcalleitherofthosefunctions,sowe’regoingthroughsomeotheroutputroutine.38Soagain,backtotheoriginalfunction,let’swalkthroughthisalittlemorecarefully.[backtosourceinsight]Sothat’sgoingtobetrue,thatisfalse,thatisfalse,soisthat,aswellasthat.[pickingguardsandpathstofollow]That’swherethatannoyingerrorcomesfrom.39Thatoneisfalseandsoisthisone.Sowe’rejustgoingtogoinhere,sotheEirstonethatwe’lldoiscallgetrange,andset???tofalse,sonow.Somaybewe’renotmakingitasfarastheunassembleditself,maybewe’restuckinthisgetrangefunction.Sohere’sanothercasewheretheeditorisprovidingmewhatitthinkcouldbetheimplementationthatI’mcalling,butI’msecondguessingthatbasedonexperience.Sowe’lllookforotherones.[browsinglistofsourceinsightsymbolsforthatmethod]Butthat’sprobablythecorrectone,solet’sgointogetrangeandwalk.40Ok,sothisonegoes,skipsthatnextcharacter,andthat’snotgoingtodoanythinginthere[reasoningaboutcall–won’thavethiseffectinit–basedonidentiferanddomainknowledge]andthenweregoingtocallget???addressmostlikely.No,we’regoingtocallevaluateaddressandherethesourcecodeeditordoesn’tgivemeapreviewofwhichoneitis.Sowe’llseeifweevenhavethisatall,sonowwehavetoguesswhichoftheseimplementationsweareactuallygoingtocall.41Thec++oneorthemasmone,ortheseareallprototypes.Butitmaybethatweonlywantthisone.Sowe’llgointothatoneandseeifwesee.Anotherfucnitoncallingafunctionwherewedon’tseethisatall.Nowthesourcecodeeditoristellingmethatthisisamacro,butitdoesn’tseemtobeabletoEindthemacrodeEinition.42So,we’lljustignorethatforthetimebeing.Here’sanotheronethatlookslikeamacro,butitdoesn’tseemtolikethat,sowe’llgooverhereandsearchforitinadifferentway,becausethatEinallyEinished.SonowI’minarazzlewindowtryingtosearchforthesamething,sincemysourcecodeeditorisfailing,I‘llgolook,I’llgolower.[Eixedatypoinsearch]43Wellthat’siterating,it’snotEindinganything.IalwaysEindmacrosverydifEiculttodealwithandI’munsureifthatisjustgeneralunfamiliarityorthelackofgoodtoolsorsomethingelseIdon’tknow.Butitalwaysseemthatwhenlookingatcodeinasourcecodeeditor,itwouldbehelpfultohaveaninstancemacroexpanderinline,soitwouldjustshowyouwhat44isactuallygoingtohappen.Becausemacrosarethereforthecodewriter,andmakesitdifEicultforeverythingelse.Sowhilethisissearchingandwe’respendingawholelotoftimewalkingthroughthiscodewithminimalresults.45Sowe’llgobackheretothedebugger.Andwhatarewelookingfor,we’relookingforthatfunction,whateverthatfunctionwas.Yeah,probablygetaddressexpression.So,we’llsetabreakpointthere,andsureenoughwecallthatguy.SoinsomewaysthisischeatingbecauseIcanbasicallyreprowhatI’mdoing.[canreproandstepthroughcoderatherthanjustgettingstaticsnapshot]46Butdebuggingpostmortemfailures,Idon’tgetachancetodothat,sothat’swhyIhavethisdichotomyofwhatthetoolsshoulddo,becauseontheonehandI’mreadingthecodeandmaybeeditingit,andontheotherhandIspendasmuchtimeifnotmorereadingcodeandtryingtounderstandwhatit’sdoingwithoutnecessarilyhavingtointeractwithitatall.Everytingveryslowtoday.[stillwaitingonEilesystemsearchanddebugger]SoI’llswitchoverhereanddoatotallyunrelatedtaskwhileI’mwaiting.47Ok,sothedebuggercamebacksoIcanwalkthorughhere.Well,it’sgottobethiscallhere.Yep.Sohere’swhatIwaslokingforinthesourcecodebefore,andIdidn’thaveanywaytotellotherthanwalkingthrough???,buthere,walkingthroughthedebugger,48Icanclearlyseethattheimplementation,thisparticularevalholderlocalvariable,whattypeitis.SoIcango,basicallygrabthisandgobacktothecodeandEindthemasmevalexpressionandlookforevaladdressinthere.But,again,myhandydandycodeeditorisnotveryhelpfultoday.Wait,doIknow?Ok.Thesymbolssaythisshouldbeineemasm,andthere’seemasm,butitsnotthere.Sothat’sodd,maybethere’ssomethingwrongwiththeproject,so.50[tryingtoaddEiletorpoject,butnotinEilesystem]So,noeemasm.So,clearlythatEileisnothere.Well,that’sbecauseit’snotthere.Myclientviewismessedup.51[editssdclientconEigEile]Oh,becausewhenIchangedthatbeforeIbrokeit.SoindoingsomethingelsetotallydifferenttodayIbasicallydeletedabunchofsourceEilesfrommymachine.So,we’llhavetoEixthat.[addingseveralextradirectorylinestohissdclientconEigEiletorestoreittonormalstate]52AndmagicallyalltheEileswillreappear,whichprobablyexplainswhymysourcedebuggingwasn’tworking.[doessdsynctogetalltheEileshedidn’thavebefore][wasn’tgettingsymbolsindebugger]53Ormaybenot,butalas,maybeIhavetheEileI’mlookingfornow,andlowandbeholdthereitis.[sourceinsightscansEilesystemandupdatesindex,andnowcannavigatetothatEile]Sowhatarewelookingfor,evaladdress.54SoifIwouldhavetoeditthisEile,IwouldprobablyEixsomeofthespacingissues,thesetabsnotreplacedwithspacesetc.ButI’malsomissingsomesourceEile,let’sseeifsearchingforeitherofthosethingsIcouldn’tEindbeforeactuallyyieldsanythingnow.[repeatssourceinsightsearch,stillnothing][illustratesreturningtoanoldpathwheninformationaboutpathschanges]Sowherearewe,we’reinevaladdress.55[skimmingthorughablockofseveralmethodsinsourceinsight]Sothisfunctioncallsthisfunction.Pushcontext,popcontext,compute???address,andwhatdoesitdo?56[goestocallee]Solet’sjusttestthis.Soweactuallymakeitbacktogetaddressexpression.Andwe’reingetaddressexpression.Ok,soitseemslikewearegoingtoreturnfromthisguywithoutdoinganything.Sonowwe’rebacktogetrange.57Andthat’swherewecallpeekchar,wedon’t.Andacasewheretheparameters,there’snocommentshere,buttheparametersarenamedwell,soIcanmakesomeassumptionsbasedonthem.Soifnothas???addressexpression,addressok,sowegodownthere.[goingbackandforthbetweendebuggerandsourceinsight–inspectingcodeinsourceinsight,lokingatvaluesindebugger]butwedocallgetaddressexpressionagain.Solet’s58we’llgo,butwedidn’tactuallycall,sowe’rebacktothatreset.Soweonlycallgetaddressexpressiononce,andthatcallisfromgetrange,solet’ssee.Sothere’s1,2.Sowe’renotgoingtodoanyofthisif.[stillreasoningaboutwhatpathstofollowinsourceinsight]59We’rejustgoingtoreturn?Soparseandassemble.Sowedoreturnfromgetlength,ohhaslength,sowearegoingtogointothatunassembledinstruction.We’llcheckthat,andtherewego,wegointotheunassembledinstruction.1:00[checkingindubggerwhichmethodgetssteppedinto,thenbacktosourceinsighttoreadit]Sohere’swherewecreatethat,???1:01Address,ok,calls,ok.Sowe’llassumethatit’srelatedtothiscall,yep.Sowecallmachinecodeassemble??ok1:02SoIwasrightintermsofreadingthecode.It’snicetohavethistoconEirm.Soitseemsthatweactuallymakeittothisdecodecall.“You’rebasicallyusingthedebuggertostep”Morelikejumparound.I’mnotreallysteppingthroughthecodebecausealotofthesefunctionsareprettylong.SoI’mbasicallyjustpickingapoint,readingcode,pickingapoint,andrunningtothatpointtomakesureweactuallygotthere,sojusttestingatheory.Solikethisone,1:03IamlookingforwherewecalldecodebecauseIthinkwe’llmakeitatleastthatfar.Sosincethesourcecameback[canEinallylinktosourceindebugger]Icanactuallyjustruntothispoint.Sincethesourcecodeofthedebuggerwasn’tworking,Iwasactuallyswitchingbackandforthbetweensourcecodedebugginandassemblydebugging.Therearedifferentthingsthatareeasiertodo.Ok,sothat’swherewecalldecode.[wantedtoknowcallerofdecode]1:04SonowI’mjustgoingtostepalittlebithere.Ah,Ibetthat’sit.Sowherearewe–we’reinmachinedotcpp.[sawacall–readaddressmemory=thathethinkstriggerserrorstring]Andwe’reindecode.[methodofmachinedotcpp]1:05AndIwentahabecauseIsawthefunctionreadaddressmemory.AndI’mpositingthatthatiswhereourerroriscomingfrom,andthisiswhat’scallingreadvirtual.SoIbetreadvirtualisfailing,sowe’llcontinuetostepthroughhere.Andthat’sactuallywhatIwanttochange.Iwanttochangethebehaviorwhenreadvirtual[Einallylocatedprogrampointwherehewantstomakeachange!–beinterestingtocomparetohowlongittakesdevelopersinothercases]failsso.SoIthinkIfoundwherethecodechangewouldneedtogo,butIneedtoconEirmthatthat’stheplacewherethecodechangeneedstogo,andthenIneedtoreadthroughthecodetoseewhatmightbethesafestwaytomakethischange.1:06Sowe’llputthecursorhereandwe’llruntothispoint,andwe’llstepintoreadinstructionmemory.AndIthinkallofthisislargely,wait,that’snotright.Here’swherewecallreadvirtual,sowe’llwalkthroughthisjusttomakesurewedon’t’,butIthinkwe’rejustgoingtohit,yeah,wehitthatbranch,andthenwegointothatbranch,yep.[veriEiedthatthatisthecallthatfailsbyjustseeinghowthereturnfromfailureiscausingittostepintootherbranchesofthemethodonthefailurepath]Readphysicalisgoingtobefalse,sowe’retocallreadvirtual,yepthat’souroffset,ouroffsetisourorigainlparameter,there’souroutparameter,let’ssee.Sowherearewe,we’reindump.cpp1:07Andwe’reatline8958.[stepping,inspectingsomeimmediates]Oh,that’snotright.Cpp.Sowe’lljustwalk,andthereisourreadvirtualfailure.Sowhenreadvirtualfailswithsomethingotherthansok,wegotodone.[stepping]Yep,andthen1:08Isthatinstructionmemory.Yeah,sothat’snull.Whatisthison,memorybites.Yeah,sowereadnothingandthere’snothinginourbuffer.Soclearthat.1:09What’sourstatusatthispoint,ourstatusisthehresult,ok.[checkinginimmediate]Ok,so,oksothere’swherewereturnthestatus.And,yepwefailed,sowereturnthatback,decodereturnstodisassemble,Sothat’sinteresting,wedon’tactuallycheckthestatushere,sowhatfunctionareweinhere,machinedotcpp,andwe’reindisassemble[functionname]

Why was this question so hard to answer?

26

HardtopickthecontrolflowpaththatleadsfromstarAngpointtotargetGuessandcheck:whichpathleadstothetarget?

m

error

Why are control flow questions frequent?

27

causality

ordering

choice

HelpsanswerquesAonsabout

Whatdoesthisdo?Whatcausesthistohappen?

DoesAhappenbeforeB?

Doesxalwaysoccur?InwhichsituaAonsdoesxoccur?

Whensca\eredacrossacodebase,findingstatementstoanswerthesequesAonscanbehard.

28

Defect-relatedfalseassumpAons&incorrectlyansweredquesAonsrelatedtocontrolflow

(commoncharacterisAcsofevidencesought)ReachabilityQues$ons

labobservaAons fieldobservaAons

28

PrimaryquesAonsfromlongestinvesAgaAon&debuggingacAviAesrelatedtocontrolflow

29

feasiblepaths

statementsmatchingsearchcriteria∩

Defect-relatedfalseassumpAons&incorrectlyansweredquesAonsrelatedtocontrolflow

(commoncharacterisAcsofevidencesought)

.

downstream upstream

searchcriteriaidenAfierstatementtype(fieldwrite/read,librarycall)

feasiblepathsfilter compare

ReachabilityQues$ons

Asearchalongfeasiblepathsdownstreamorupstreamfromastatementfortargetstatementsmatchingsearchcriteria

labobservaAons fieldobservaAons

29

PrimaryquesAonsfromlongestinvesAgaAon&debuggingacAviAesrelatedtocontrolflow

Reachability question: example

30

feasiblepaths

statementsmatchingsearchcriteria∩

Whereismethodmgenera$nganerror?

m

e

Asearchalongfeasiblepathsdownstreamorupstreamfromastatement(m)fortargetstatementsmatchingsearchcriteria(callsto methode)

Longest activities related to reachability questions

31

4outofthe5longestinves$ga$onac$vi$es

5outofthe5longestdebuggingac$vi$es

Primary question Time (mins) Related reachability question

How is this data structure being mutated in this code? 83 Search downstream for writes to data

structure“Where [is] the code assuming that the tables are already there?” 53 Compare behaviors when tables are or are

not loadedHow [does] application state change when m is called denoting startup completion? 50 Find field writes caused by m

“Is [there] another reason why status could be non-zero?” 11

Find statements through which values flow into status

Where is method m generating an error? 66 Search downstream from m for error textWhat resources are being acquired to cause this deadlock? 51 Search downstream for acquire method

calls“When they have this attribute, they must use it somewhere to generate the content, so where is it?”

35 Search downstream for reads of attribute

“What [is] the test doing which is different from what my app is doing?” 30 Compare test traces to app traces

How are these thread pools interacting? 19 Search downstream for calls into thread pools

Insights

32

‣Developerscanconstructincorrectmentalmodelsofcontrolflow,leadingthemtoinsertdefects

‣ThelongestinvesAgaAon&debuggingacAviAesinvolvedasingleprimaryquesAonaboutcontrolflow

‣FoundevidenceforanunderlyingcauseofthesedifficulAesChallengesansweringreachabilityques$ons

Paper prototype study• Built mockups of interface for task from lab study

• Asked 1 participant to complete lab study task with Eclipse & mockup of Reacher

• Paper overlay of Reacher commands on monitor

• Experimenter opened appropriate view

• Asked to think aloud, screen capture + audio recording34

Study results• Used Reacher to explore code, unable to complete task

• Barriers discovered

• Wanted to see methods before or after, not on path to origin or destination

• Switching between downstream and upstream confusing, particularly search cursor

• Found horizontal orientation confusing, as unlike debugger call stacks

• Wanted to know when a path might execute35

Examples of observed reachability questions Reacher supports Steps to use Reacher

What resources are being acquired to cause this deadlock? Search downstream for each method which might acquire a resource, pinning results to keep them visible

When they have this attribute, they must use it somewhere to generate the content, so where is it?

Search downstream for a field read of the attribute

How are these thread pools interacting? Search downstream for the thread pool classHow is data structure struct being mutated in this code (between o and d)?

Search downstream for struct class, scoping search to matching type names and searching for field writes.

How [does] application state change when m is called denoting startup completion?

Search downstream from m for all field writes

Step 2: Find statements matching search criteria

36

Step 3: Help developers understand paths and stay oriented

37

Goal:helpdevelopersreasonaboutcontrolflowbysummarizingstatementsalongpathsincompactvisualizaAon

Challenges:controlflowpathscanbe

complex

long

repeAAve

developersgetlostanddisorientednavigaAngcode

Approach:

visuallyencodeproperAesofpath

hidepathsbydefault

coalescesimilarpaths

usevisualizaAontosupportnavigaAon

Example

38

Evaluation

39

Method 12developers15minutestoanswerreachabilityquesAonx6Eclipseonlyon3tasksEclipsew/REACHERon3tasks

Tasks

BasedondeveloperquesAonsinlabstudy.

Example:

WhenanewviewiscreatedinjEdit.newView(View),whatmessages,inwhatorder,maybesentontheEditBus(EditBus.send())?

DoesREACHERenabledeveloperstoanswerreachabilityquesAonsfasterormoresuccessfully?

(ordercounterbalanced)

DeveloperswithREACHERwere5.6AmesmoresuccessfulthanthoseworkingwithEclipseonly.

Results

40

TaskAmeincludesonlyparAcipantsthatsucceeded.

(notenoughsuccessfultocompareAme)

REACHER helped developers stay oriented

41

WhennotusingREACHER,parAcipantsogenreportedbeinglostandconfused.

ParAcipantswithREACHERusedittojumpbetweenmethods.

“WhereamI?I’msolost.”“Thesecallstacksarehorrible.”“Therewasa call to ithere somewhere,butIdon’trememberthepath.”“I’mjusttoolost.”

“Itseemsprettycool ifyoucannavigateyourwayaroundacomplexgraph.”

“Ilikeitalot.Itseemslikeaneasywaytonavigatethecode.AndtheviewmapstomoreofhowIthinkofthecallhierarchy.”“Reacherwasmyhero.…It’salotmorefuntouseandlookat.”“Youdon’thavetothinkasmuch.”

ParAcipantsreportedthattheylikedworkingwithREACHER.

Reflection on design process• Started with a goal: make debugging in large, complex

codebases better

• Observed users to build insight into what key challenge was

• Rather than address usability challenges of existing debugging tools, designed new way to debug

• Gathered evidence that it worked better42

Needfinding

43

Needfinding (a.k.a. design research)• Goal: understand user’s needs

• Use of methods to gather qualitative data

• behaviors, attitudes, aptitudes of potential and existing users

• technical, business, and environmental contexts - domain

• vocabulary and social aspects of domain

• how existing products used

• Empowers team w/ credibility and authority, helping inform decisions44

Needfinding vs. market researchNeedfinding

• What users really need

• How they will really use product

• Qualitative methods to study in depth

• Small numbers of participants45

Market research

• Who might purchase item

• What factors influence purchasing

• Quantitative studies w/ focus groups, surveys

• Large numbers of participants

Example• Cooper conducted a user study for entry-level video editing

product

• Company built professional software, looking to move into consumer software

• Help connect those w/ computers and video cameras

• Found strongest desire for video editing was parents

• Found 1/12 had successfully connected camera, using work IT guy46

Solving the correct problem

• Practices may sometimes mask deeper problems

• Goal: uncover layers of practices to understand how problems emerge

47

Interviews• May include bother current users and potential users w/ related needs

• Questions

• context of how product fits into lives or work

• when, why, how is or will product be used

• what do users need to know to do jobs?

• current tasks and activities, including those not currently supported

• goals and motivations of using product

• problems and frustrations with current products or systems48

Observations• Most incapable of accurately assessing own behaviors

• May avoid talking about problems to avoid feeling dumb

• Observing yields more accurate data

• Capture behaviors: notes, pictures, video (if possible)

49

Contextual inquiry

• Method that includes both interviews and observations

• Next time

50

Ideation

51

Ideation• Process of generating, developing, communicating new ideas

• Guidelines and best practices

• Generate numerous ideas

• Number ideas

• Avoid premature dismissal of ideas

• Sharpen the focus - pose the right problem

• Build and jump - build to keep momentum on ideas, jump when theme tapers out

52