Talk to PS SIG CHI on Agile user research

The RITE+Krug Method

Getting crazy good stakeholder involvement in Agile user research

Jen McGinn & Ana ChangDecember 5, 2013

We’ll tell you our story

We’ll listen to your stories

We’ll give you summaries and links to further reading

Hopefully, you’ll leave with something helpful

And we’ll pick up some tricks from you

Format

Jen McGinn and Ana Chang 2PS SIG CHI Monthly Meeting, Dec 2013

Before we tell you our story, let’s start with you

Show of hands:– How many of you are working with Agile development

teams?– How many of you are conducting user research (feedback

sessions) with those Agile teams?– How many of you are feeling really AWESOME about how

the dev team is involved in that research and that they make changes as a result?


I can’t believe he clicked the blue button.

I assumed he would click the gray button.

We should fix that!

Late 2011: Traditional usability tests were conducted


The results are not statistically significant.

This is not a typical user.

That’s not a realistic task.

Later 2011: The stakeholders didn’t buy into the results

I don’t agree with your interpretation of what that means.


Current state of the application

Start of study

Current state of the application

Proposed new design

At the beginning of the study

When the study results are published

The feedback came too late for our Agile environment


Now it’s your turn

• First names only• Agile User Research only• In less than a minute, tell us:

– What user research you are conducting now (method, frequency, # of participants)

– One thing that is AWESOME about thatOR– One thing that’s not working, that you wish you could fix


Hey, what about RITE testing?

Ana had attended a conference


Great – so now what?

Rapid Iterative Test Environment (RITE)– Developed by Dennis Wixon & the folks at Microsoft games– Can change the (live) system between participants– Developers must attend the UT sessions– Keep testing until show-stoppers are fixed– Number of participants and time to conduct undefined


Early 2012: What’s been written on Agile + RITE?

• When I started looking for answers 18+ months ago:• Lots and lots on Agile UX + design• Not so much on Agile UX + user feedback

• RITE (described in general terms, but no specifics)• Co-design • Customer/user “proxies”

• However, there were some elements of Steve Krug’s approach that stuck with me


Steve Krug … why does that name sound familiar?

http://www.amazon.com/Steve-Krug/e/B001KHCFUU/ 11

The Krug Approach*

“A morning a sprint, that’s all we ask.”

• Test 3 users in a morning

• Invite everyone to attend: product owners, developers, designers, writers, marketing …

• Debrief over lunch: 1 hour

– Only people who observed can attend/vote (sort of)

– Focus ruthlessly on most serious problems

• Report takes less than 30 minutes to write

– Just an email

– 2 page max., mostly bullet lists

* Originally presented by Steve Krug at the Boston UXPA 2013 Conference, May 201312PS SIG CHI Monthly Meeting, Dec 2013

Our RITE + Krug Combination

• Elements from both RITE and KRUG– Require that a stakeholder from dev, design and PM attend each

session– Can change the prototype between participants– 3 or 4 participants– Findings discussed in 1-hour debrief– Minimal “report” (bullets on a wiki page)

• Sessions went viral: 12+ observers• Could test every as often as every 2 weeks

http://www.upassoc.org/upa_publications/jus/2013may/JUS_McGinn_May_2013.pdfJen McGinn and Ana Chang 13PS SIG CHI Monthly Meeting, Dec 2013

Artifacts: scaled down script


Artifacts: example wiki page with schedule and impact


RITE+Krug Tips & Tricks

• Before sessions: Hold a kickoff meeting with stakeholders to get buy-in into the process and time required

• During: Keep a running list of observations and email them to the stakeholders within 30 minutes of the last participant

• After sessions: Ask attendees of the stakeholder meeting to review your list of observations prior to the meeting

• During debrief: Walk through the findings to get agreement on what everyone saw (optional: agree to next steps to fix)

• After debrief: Immediately document the observations and list of changes agreed to and send out/post on wiki


Let’s talk about Diana

The conversation has changed from ‘what is it?’ to ‘how can we master it and make it our own?’

- Diana DeMarco Brown (over lunch one day)

http://www.amazon.com/Agile-User-Experience-Design-Practitioners/dp/0124159532/

17PS SIG CHI Monthly Meeting, Dec 2013

Diana’s Agile Research Recommendations*

• Conduct research before the development cycle if possible– But if you can’t, fit it in anyway

• Predictable and repeatable user feedback sessions during the cycle• Complement user feedback sessions with fast, low-cost data

collection methods:– Surveys– Customer phone calls– Web analytics – Card sorts– Un-moderated usability tests

* Originally presented by Diana Brown at the Boston UXPA 2013 Conference, May 201318PS SIG CHI Monthly Meeting, Dec 2013

Jeff Gothelf’s Approach

• Continuous & collaborative– Continuous: every sprint– Collaborative: conduct

research together (dev, design, PM, &research)

Jen McGinn and Ana Chang 19

http://www.amazon.com/Lean-UX-Applying-Principles-Experience/dp/1449311652

PS SIG CHI Monthly Meeting, Dec 2013

Test everything

“In order to maintain a regular cadence of user testing, your team must adopt a “test everything” policy. Whatever is ready on testing day is what goes in front of the users… you’ll find yourself taking advantage of your weekly test sessions to get insight at every stage of design and development.”

– Lean UX, Jeff Gothelf


What I do now: Continuous and Collaborative

• Collaborative– Dev, Design, Doc, and PM stakeholders participate or observe– User researcher will facilitate or advise– Your opportunity to get “out of the building”

• Continuous– Sessions take place at regular intervals, say once a sprint– User researcher will recruit 3 – 4 target participants per session +

debrief same day


What else have people written on User Research + Agile?

Agile + RITE testing, finally (!) described in detail (2013):• “The RITE Way to Prototype”, Shirey, Charng, & Nguyen

http://uxmag.com/articles/the-rite-way-to-prototype

Agile + Cognitive walkthroughs (2013):• “Informal cognitive walkthroughs (ICW): paring down and pairing

up for an agile world”, Grigoreanu & Mohannahttp://dx.doi.org/10.1145/2470654.2466421

Agile + Wizard of Oz testing (2006): • “Adding Usability Testing to an Agile Project”, Meszaros & Aston

http://dx.doi.org/10.1109/AGILE.2006.5


http://uxmag.com/articles/the-rite-way-to-prototype

http://dx.doi.org/10.1145/2470654.2466421

http://dx.doi.org/10.1109/AGILE.2006.5

Agile & Lean UX resources that include user research

• “Adapting Usability Investigations for Agile User-centered Design”, Sy (2007) http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf

• “Successful User Experience in an Agile Enterprise Environment”, Federoff & Courage (2009)http://link.springer.com/chapter/10.1007%2F978-3-642-02556-3_27

• Jeff Gothelf’s blog: http://www.jeffgothelf.com/blog/

• 5 tips for applying Lean UX to User Research, Jeff Sauro (2013) http://www.measuringusability.com/blog/lean-research.php


http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf

http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf

http://link.springer.com/chapter/10.1007/978-3-642-02556-3_27

http://www.jeffgothelf.com/blog/

http://www.measuringusability.com/blog/lean-research.php

Takeaways

• You and your team can incorporate user research into Agile development environments

• Align research with the Agile development & design cycles• Use lightweight research methods • Use a few participants• Require that key stakeholders attend (and/or conduct) user

research• Get feedback at regular intervals (once a sprint)• Get feedback on everything


Questions?

?

Jen McGinn and Ana Chang 25

jen dot mcginn at oracle dot com & ana dot ramirez dot chang at oracle dot com

PS SIG CHI Monthly Meeting, Dec 2013

The RITE+Krug Method

Getting crazy good stakeholder involvement in Agile user research

Jen McGinn & Ana ChangDecember 5, 2013

Different versions of the UI

Ease

of

Use

Large A/B study

Small, think-aloud study

Jen McGinn and Ana Chang

A/B testing

• A validation technique that you can use when you have:– Working code on the web– Large numbers of participants– A narrowly-defined problem

• “Which one of the designs results in fewer abandoned carts?”• “Which one of the designs results in a higher click-through rate?”

“Unless you are a huge property like – like Bing or Google … you’ll have more things to test than you have traffic

– Lean Analytics, Croll & Yaskovitz, Eric Reis, ed


“Formative” Feedback Sessions

• A discovery technique that you can use when you have:– drawings, sketches, wireframes– small numbers of participants– questions about features, mental models, terminology, iconography,

usability of your prototypes, current practices, pain points, values, design alternatives, or want to collect other qualitative data

“Initially, you’re looking for qualitative data … you’re speaking to people … You’re exploring. You’re getting out of the building.”

– Lean Analytics, Croll & Yaskovitz, Eric Reis, ed
