19
© 2008 Handmade Mobile. All Rights Reserved. 1 Mobile Design UK 3 Dec 2008 Jaeseung Chang ([email protected]) Usability Architect @ Handmade Mobile Entertainment Ltd. Cheap & Quick Internal User Testing - Flirtomatic’s Recent Challenge -

Cheap& Quick Internal user testing

Embed Size (px)

DESCRIPTION

Hwo soes one do user-testing on a shoe-string when incrementally improving a live service? Here's some pointers

Citation preview

Page 1: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 1Mobile Design UK

3 Dec 2008

Jaeseung Chang ([email protected])Usability Architect @ Handmade Mobile Entertainment Ltd.

Cheap & Quick Internal User Testing

- Flirtomatic’s Recent Challenge -

Page 2: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 2Mobile Design UK

Quick Introduction

- BA in Education from SNU

- MSc in Information Technology from QMUL

- PgDip in HCI from UCLIC

- MSc by Research in Computer Science from QMUL

- Open Education (Korea) / Usability Designer

- Samsung Electronics (Headquarter in Korea)

/ Mobile Usability Designer + Researcher

- Handmade Mobile (Flirtomatic) / Mobile Usability Architect

Page 3: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 3Mobile Design UK

Cheap & Quick Internal User Testing

– Flirtomatic’s Recent Challenge

1. Why Cheap & Quick?

2. Experiment Design

3. Good Lessons – Dos and Donts

Page 4: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 4Mobile Design UK

1. Why Cheap & Quick?1.1. Evaluation Needs

Requirement

Analysis

Fast

Prototyping

SpecificationImplementation

EvaluationEvaluation

- We all know the necessity of usability evaluation

- At least 2 times per 1 iteration cycle

- But the reality sometimes doesn’t help

Page 5: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 5Mobile Design UK

1. Why Cheap & Quick?1.2. Usability Firms

- Hiring professional research firms?

- Excellent facilities, professional experiment design,

reliable recruitment, & nice report (maybe?)

- But costs a fortune!

Page 6: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 6Mobile Design UK

1. Why Cheap & Quick?1.3. Self-Evaluation

- Then we should go DIY

- Venue: How about using our meeting room?

- Experiment design: We can do that maybe?

- Recruitment: Find some volunteers from our users?

- Observation and analysis?: 15 day trial version MORAE!

Page 7: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 7Mobile Design UK

Page 8: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 8Mobile Design UK

Cheap & Quick Internal User Testing

– Flirtomatic’s Recent Challenge

1. Why Cheap & Quick?

2. Experiment Design

3. Good Lessons – Dos and Donts

Page 9: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 9Mobile Design UK

2. Experiment Design2.1. Overview

- The 1st iteration of our ‘Redesign Project’

- Based on wireframes (workflows) produced so far

- Evaluation on our first redesign UI/IA works

- To discover good/bad points for more future works

- Included three parts: UT + GUI Preference + FGD

User TestingGUI

Preference

Focus Group

Discussion+ +

Page 10: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 10Mobile Design UK

2. Experiment Design2.2. User Testing

- Normal user testing techniques (think-aloud)

- Went through 6 sample tasks with 8 participants

- Used a hi-fi prototype to simulate our new UI

- Controlled experiment with video recording (MORAE)

- 1 moderator and multiple observers

CONFIDENTIALCONFIDENTIAL

Page 11: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 11Mobile Design UK

2. Experiment Design2.3. GUI Preference

- Asked the same participants to choose our new GUI designs

- A sort of multiple-choice survey questions

- Asked to choose the best 2 & the worst 1 design

CONFIDENTIAL

Page 12: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 12Mobile Design UK

2. Experiment Design2.4. Focus Group Discussion

- To collect real users’ preferences and behaviours

- In terms of using the current version of Flirtomatic

- Focused on gathering additional information

- To help redesign the details of our WAP service

- Went through 5 topics with 3 groups of users (m, f, & gay)

Page 13: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 13Mobile Design UK

Cheap & Quick Internal User Testing

– Flirtomatic’s Recent Challenge

1. Why Cheap & Quick?

2. Experiment Design

3. Good Lessons – Dos and Donts

Page 14: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 14Mobile Design UK

3. Good Lessons – Dos and Donts3.1. Outcomes

- Were able to conduct evaluation ourselves – DIY!

- Captured some important findings to validate our prototype

- Less than 2 wks (Experiment design + experiment + analysis)

- Less than 500.00 GBP

- Much quicker and cheaper than hiring research firms

- Were able to share the importance of usability evaluation

with the management level and all other departments

- Self-learn about the process of usability evaluation

Page 15: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 15Mobile Design UK

3. Good Lessons – Dos and Donts3.2. Difficulties

- Recruitment error: No show up, recruited friends, etc.

- MORAE error (due to the inexperienced operation):

No video recording, low quality audio recording, etc.

- Some mistakes of our colleagues:

intruding into the UT room to connect to wireless network,

interrupting UT sessions to ask questions to the moderator,

giving money to participants during the session, etc.

Page 16: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 16Mobile Design UK

3. Good Lessons – Dos and Donts3.3. Dos

- Good preparation in every way:

1. Documentation on task scenario, forms, analysis

2. Train yourself to master any SW tools (like MORAE)

3. Clean up the venue in advance

- Share the importance of usability evaluation with everyone

1. With the management level

2. Especially with colleagues nearby the UT room

(should politely ask silence during each session)

- Share the formal process of evaluation with team members

- Educate yourself how to run each session professionally

Page 17: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 17Mobile Design UK

3. Good Lessons – Dos and Donts3.4. Donts

- Do not rely on the quick list of recruited users

(importance of having some back-up participants)

- Do not expect everyone else in office will behave as you do

1. They don’t know how important it is

2. Actually they don’t care what you do

- Do not expect your team members understand everything

1. They don’t know how to formally run each session

2. Importance of pre-training for usability evaluation

Page 18: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 18Mobile Design UK

3. Good Lessons – Dos and Donts3.5. Resources

- MORAE: http://www.techsmith.com/morae.asp

- The Usability Methods Toolbox by James Hom:

http://jthom.best.vwh.net/usability

- Usability First: http://www.usabilityfirst.com/

- Jakob Nielsen: http://www.useit.com/

Page 19: Cheap& Quick Internal user testing

© 2008 Handmade Mobile. All Rights Reserved. 19Mobile Design UK

Thank you

Jaeseung Chang

([email protected])