Usability Test Report for Craigslist Mobile iPhone App€¦ · quickly became an advertising portal in 1996. Today, Craigslist is used in over 70 countries worldwide and supports

Usability Test Report for Craigslist Mobile

iPhone App

Table of Contents

2 Craigslist Mobile

Table of Contents

Table of Contents ............................................................................................................................ 2

Introduction .................................................................................................................................... 4

Methods .......................................................................................................................................... 6

Participants ............................................................................................................................. 6

Apparatus and setup ............................................................................................................... 7

Roles ...................................................................................................................................... 12

Measures ............................................................................................................................... 14

Procedure .............................................................................................................................. 16

Task-based Data Analysis .............................................................................................................. 23

Task 1: A complete search task ............................................................................................. 23

Task 2: A simple search task ................................................................................................. 40

Task 3: Post an ad ............................................................................................................... 445

Task 4: Change your post .................................................................................................... 500

Task 5: Repeat Task 1 on laptop ......................................................................................... 522

Post-test interview ................................................................................................................ 55

Table of Contents

3 Craigslist Mobile

Data analyses summary by task ............................................................................................ 59

Data analyses summary by participant ............................................................................. 6262

Data analyses summary by high level questions .............................................................. 6565

Appendixes ................................................................................................................................ 6666

Appendix A: Content form .................................................................................................... 66

Appendix B: Pretest questionnaire ....................................................................................... 67

Appendix C: Modified test protocol...................................................................................... 68

Appendix D: Post-test interview questions .......................................................................... 91

Appendix E: Checklist ............................................................................................................ 92

Appendix F: Task sheets ........................................................................................................ 93

Appendix G: Early version test protocol ......................................................................... 10101

Appendix H: Early version post-test questionnaire ........................................................ 12121

Introduction

4 Craigslist Mobile

Introduction

Product

Craigslist (http://craigslist.org), founded in 1995 by Craig Newmark, is a website devoted solely

to displaying classified advertisements based on specific categories. The service began as an

email distribution list that Mr. Newmark used to exchange information between friends, but

quickly became an advertising portal in 1996. Today, Craigslist is used in over 70 countries

worldwide and supports over 350,000 postings each day.

This report is an expert usability study of c-Mobile, the official Craigslist app for iOS, which

allows iOS users to search, navigate, and manage Craigslist postings. The c-Mobile app claims to

be designed so that a user has the ability to perform tasks on a mobile device they would

normally be able to do on the desktop Craigslist site. This includes tasks such as searching ads,

posting ads, uploading photos, sharing posts, and managing your Craigslist account directly from

the device. Given the explosive growth in the mobile market, the app allows an opportunity for a

more refined Craigslist experience on mobile. Furthermore, the fact that 64% of smartphone

owners now use their mobile device to shop online, it is crucial that a mobile-friendly Craigslist

app exist on the market. We have evaluated the application thoroughly and have asked a sample

of random participants to evaluate the application as well. The findings of the study will be

presented in the subsequent pages of the report.

Objectives

The objective of this study was to identify usability flaws and the ease of use of the application

by performing a heuristic evaluation of the c-Mobile app; and then design and run a series of

tasks that a user may take when using the application on a mobile device. The analysis was

targeted at identifying user goals, overall site usability, and to be able

to provide recommendations on the usability flaws captured through user evaluation.

Introduction

5 Craigslist Mobile

Usability Measures

Our study consisted of eight participants, all of which were iPhone users or had been iPhone

users sometime in the past. Usability was measured on effectiveness, efficiency, and satisfaction

of the application. These may break down further into measures of repeated use, learnability,

performance, success, context of use, behavioral indications, sharing, and emotional reactions to

the product. Descriptions of the usability measures can be seen in the table below.

Effectiveness Efficiency Satisfaction

Was the product simple

and easy to use?

Would the user use the

product again in the

future?

How learnable was the

product and its interface?

Was there positive or

negative contextual

mapping of the product

in the mobile

environment?

Were users able to complete

the tasks quickly and

without mistakes?

Were they successful in

completing their tasks?

How did they feel about

using the product?

Were they frustrated?

Were they happy?

What sort of verbal or facial

gestures did the users make

when interacting with the

product?

What other reactions did

they have to it?

Did they feel that they

needed to share it with

friends or family?

Did they freely share within

the app?

Methods

6 Craigslist Mobile

Methods

Participants

Eight participants (all iPhone users), with the following profile characteristics, have evaluated

the Craigslist Mobile app.

Gender

Female 3

Male 5

TOTAL (participants) 8

Age

< 21 0

21-25 4

26-30 1

31-35 3

>35 0


Methods

7 Craigslist Mobile

Craigslist web usage

Only once or twice till now 2

Not frequently, only when there is a need 1

Frequently, whenever it can be useful 5


Did use the app before

Yes 4

No 4


All participants were mainly divided into 2 subgroups based on their early experiences with the

app:

● Novice users (4): people who never used any mobile apps of Craigslist before the test

● Experienced users (4): people who had used the Craigslist Mobile app before the test.

Apparatus and setup

● Craigslist Mobile (Version 4.1): An iPhone Application for Craigslist developed by

Escargot Studios.

Methods

8 Craigslist Mobile

Info about Craigslist Mobile from

https://itunes.apple.com/

iPhone Screenshot of

Craigslist Mobile homepage

● Webcam: A Webcam was used to capture facial gestures and expressions of the

participant.

Webcam Facial gesture captured by the

webcam

Methods

9 Craigslist Mobile

● Camera and tripod: A video camera mounted on a tripod was placed behind the

participant. It was used to capture hand and finger movement.

Camera and tripod Hand and finger movement captured by

the camera

Methods

10 Craigslist Mobile

● Reflector App (Version 1.2.3): Computer software developed by Squirrels LLC was

used to mirror and record the screen of iPhone, wirelessly.

Info about Reflector App from

https://www.reflectorapp.com/

iPhone screen captured by Reflector App

● Laptops: A laptop was used for data logging, another one was connected with

webcam for recording facial gesture and expression, and a third one installed with the

Reflector app was used to recording iPhone screen.

● iPhone: Participants performed tasks on an early prepared iPhone 4S. History was

cleaned after each test.

Methods


● Study setup:

A noise free environment was used for only 2 out of 8 participants. The other data collection

sessions were in natural environments like in the living room of a home with the TV on.

Administrator

User Data

logger

observer

Methods


Roles

At least three testers were involved in each usability test, and their responsibilities are described

below.

● Moderator

- Responsible for interaction with participants

- Provided overview of study to participants

- Read instruction to participants

- Conducted the test

- Responded to participant's requests for assistance

● Data Logger

- Set up equipment/apparatus

- Monitored and recorded iPhone screen

- Monitored and recorded participants' facial expressions

Methods


● Test Observer

- Observed silently

- Data logged participants' behavior and comments with respect

to specific tasks

- Timing

Methods


Measures

● Performance measures

Task completion rate: Each task requires the participant to obtain or input specific

data. The task is completed when the participant indicates that the task goal has been

obtained successfully.

Task time: The time to complete a task. It is measured from the time the participant

begins the task to the time he or she signals completion.

Error count: Errors are committed when user actions deviate from normal procedure;

including slips, mistakes, and omissions. The evaluator observes when and where the

participant makes an error and evaluates the severity of the problem.

● Behavioral measures

Facial expression: Facial expressions convey emotional state. From facial expression,

evaluators can detect if participants are delighted or bored, calmed or surprised,

relaxed or frustrated, confident or confused.

Body movement: Some gestures reflect cognition and perception process. For

example, scrolling means participants are probably scanning and searching. Eye

gazing means they are probably processing. Sighing means tiredness or frustration.

Methods


● Subjective measures

Quick interview: After each task, the participant will be asked a few questions about

the task that she/he has just performed, during which some subjective data will be

collected.

Post-test questionnaire: After all tasks, an overall subjective evaluation of the product

will be collected via a questionnaire, including the difficulty of tasks, joyfulness, and

overall satisfaction.

Methods


Procedure

● Workflow

(Protocol varied for participants based on performance)

Test protocol

Pilot test

Recruiting

Screening

Usability test

Consent form

Pretest questionnaire

Task 1: Mimic the whole search

process Task 2: A simple search task

Task 3: Post an ad

Task 4: Change a post

Task 5: Repeat Task 1 on laptop

User Choice/Free play

Post-test interview

Methods


● Recruitment, Screening & Reminders

Potential participants were sent out e-mails (see appendix) inviting them to help out in a

study to understand how users interact with a product. An effort was made to include

wide variety of background among participants. Upon receiving participant confirmation,

they were interviewed on phone and screened to be suited for the Usability test. After

passing the screening criteria, participants were scheduled on a date feasible for both the

usability test team and also the participant. E-mail reminders were sent out 24 hours

before the actual test day to all participants.

● Pilot Study

One day before the actual study took place, a pilot study was conducted make sure no

difficulties arise during the main study. Based on the pilot study, required changes were

made to the actual study.

● Introduction and consent form

When the participant came in, the moderator greeted and made the participant feel

comfortable. Participant was told that there is no right or wrong answer and was asked to

feel at home if they can. Participants who asked if the set up was required were explained

the purpose of cameras and data capture. Once that was accomplished, the moderator

briefly described the purpose of this study, and let the participant read and then sign the

content form (Appendix A).

● Pretest questionnaire

After the consent form, the participant was asked to complete a pretest demographic and

background information questionnaire (Appendix B).

Methods


Usability testing

The moderator made the participant feel comfortable in the test area and introduced the

iPhone on which the test was to be conducted, and briefly talked about the tasks and the

duration of the test procedure.

At the start, the moderator explained the instructions from the printed copy and inquired

if the participants had any questions regarding what they were supposed to do (the

participant had a copy of the instructions always available), see Task Sheets in Appendix

F. If the participant had no further questions, the task began.

During the test, the participant's performance on the phone (or the laptop for Task 5)

was monitored and recorded at the same time. The participant’s behaviors were observed

and recorded by camera and by taking notes.

After the task, the participant was asked a few questions about the task that she/he had

just performed.

The detailed test protocol is attached in Appendix C. The early version of test protocol is

also attached, see Appendix G

Participants were asked to perform part or all of the following 5 tasks.

Methods


Task 1 was to mimic the whole search process, which included 5 subtasks: searching,

changing filter, replying to the post by email, sharing a post, and finding direction (see

screenshots below). According to his/her responses to the pretest questionnaires, the

participant was asked to search apartments or cars.

Search for apartments Search for cars Change filters

Reply a post Find direction

Methods


Task 2 was a simplified searching task, asking participants to look for jobs or pets. The

moderator varied this task based on the participant

Search for Pets Search for Jobs

Task 3 asked participants to add a post about a thing they would like to sell at home (e.g.,

bicycle, books, etc.)

Post an ad Contact Upload photos

Methods


Task 4 asked participants to make changes and finally delete the post in Task 3.

Change/Delete Post

Task 5 required the participant to repeat the searching part of Task 1 on the Craigslist website

using an early prepared laptop.

Laptop task

Methods


User Choice/Free Play- Participants were given a choice to use craigslist mobile app for any of

their personal needs or just use the app for another 5 minutes for free play.

● Post-test interview

At the end of the test, the moderator had a brief discussion with the participant, asking

his/her overall satisfaction, preference, and suggestion about this product. Questions are

attached in Appendix D. The early version of post-test interview is also attached, see

Appendix H.

Task-based Data Analysis



Data collection was done through observation and watching videos recorded on the camcorder

and the iPhone screen capture software.

Task 1: A complete search task

Task 1 - subtask 1: Search for apartments or cars

A: Search for apartments.

Now, suppose you plan to move out of the apartment that you currently live in and start

looking for candidate apartments through Craigslist using this App. Please only search

for the apartments that you want to have a look. Once you find what you are looking for,

share the post with your friends of family, reply to the post and get directions.

B: Search for cars

Now, suppose you plan to buy a used car and start looking for candidate cars through

Craigslist using this App. Please only search for the cars that you want to have a look.

Once you find what you are looking for, share the post with your friends of family, reply

to the post and get directions.



Performance data

Measures Novice users Experienced users

Completion 2 of 4 passed 3 of 5* passed

Task time Range: 5 min ~ 20 min

2 of 4 took longer than 10

min

Range: 1 min 40 sec ~ 9 min 45 second

2 of 5 took longer than 5 min

Error count 3 of 4 had more than 10

errors

1 of 5 had more than 10 errors

* One experienced user was tested both A: search for apartments and B: search for cars.

Behavior data


Select right location 4 of 4 passed;

2 took really long time

5 of 5 passed;

2 complained

Select right category 2 of 4 passed;

1 selected a wrong category;

1 did not specify category at all.

5 of 5 passed

Know how to use

filter

1 of 4 passed

3 users did not use filter. They

typed all search criteria in the

search box and got zero results.

They came back to the homepage

and pressed "Browse all," then they

3 of 5 passed

1 user did not use filter at

all. He typed all search

criteria in the search box.



got results but the results were

irrelevant to their search criteria.

Easily identify

targets of interest

2 of 4 passed 2 of 5 passed

Know how to

bookmark


Know how to save

search results


Successful

Navigation

0 of 4 passed

3 users had to go back to home

screen many times.

1 of 5 passed.

1 had to go back to

homepage many times;

2 users sometimes forgot

how to get back to the

previous page.



Passing rate based on behavioral observation

Difficulty level based on behavioral observation

Navigation

Identify targets Difficult for both novice and experienced users

Use filter

Save results Difficult for novice users and some experienced

users

Bookmark

Select category

Only difficult for novice users

Select location Not difficult



Subjective data

Questions Novice users Experienced users

Do you think the search results

are well organized and clearly

displayed on the screen?

2 of 3 no

- "They are randomly

represented"

3 of 5 no

- "There is no Sort function."

Use number 1 to 7 to indicate

difficulty level (1 = very easy, 7

= very difficult)

3 responses collected: 3, 3,

and 6

Range: 3 ~ 6

Mean: 4 (relatively hard)

5 responses collected: 4, 1,

3, 4, and 2

Range: 1 ~ 4

Mean: 2.8 (relatively easy)

Comments

Participant #2 (Novice)

- "Nothing is clear."

- "I found 2 apartments, but they are not exactly what I was looking for. Because I cannot see

the clear locations through a handy map."

- "No direct map."

- "I will shift back to laptop eventually."

Participant #4 (Experienced)



- "Filter options are not detailed enough, for example , no specify make, year, etc."

- "Why do I have to specify location first before I select category? If I select category first, the

app directs me to location. I thought I clicked the wrong button."


- "Icons make no sense and are sideways."

- "Page elements / look and feel of the app is not concise."

- "I could not figure out how to filter."

- "Posts are not listed by title. They just seem random."

- "Proper pricing is not available in post list (for example, some just say $10)."


- "Filter is not helpful. It does not narrow the results efficiently. No mileage, no make, no

year."

- "Hard to extract useful info from a long post."

- "Results are not sorted in order."




- "Couldn't find what I wanted"

- "Wish I can change filter without going back"

- "Sometimes it's so hard to find it on this app. I just use the computer instead. Can I do that

now?"

- “What the hell? Does it really want me to swim to middle of the Atlantic Ocean”(finding

location)



Problems revealed in subtask 1: Search for apartments or cars

1. Search Function is confusing: The main problem almost all users encountered was

that after setting up the filter and typing in the search field, the search yielded zero

results in the case of apartments. Users did not understand why. The filter and the

search operations are two distinct functions and for optimal results only one of the

two functions should be used at a time. This operation was poorly designed.

What users did:

What users should do:

Zero Results

Results



2. Navigation is hard: Many times users were lost with no choice and had to go back

to homepage and start over again.

3. Results are not well organized: Targets of interest are hard to locate. There is no

sort function. Results are randomly presented on the screen.

Home

screen

Home

screen

aaing ti

cgange

Results

screen

Random results, no

way to optimize

search



4. Homepage is ill designed: Items (i.e., search box, category, filter) are placed on the

screen with no clear organization. The order is against logic and users' intuition. The

functions of "Search" button and "Browse all" button are confusing.

5. Icons and button labels are confusing and hard to remember: The only way to

figure them out is trial and error, even for experienced users.

Confusing icons



Task 1 - subtask 2: Change filter

A: Search for apartments.

Now assume you want to rent an apartment in San Jose, and it has the following features:

2 bedroom

has at least one parking spot

price under $2,500

dogs allowed

Please search for an apartment/house having the desired qualifications

B: Search for cars

Now assume you want to buy a used car that has the following features:

four-door sedan

no earlier than 2005

price under $8,000

sold by owner (not dealer)

mileage less than 80k

Please search for a car having the desired qualifications



Performance data


Completion 3 of 3 passed 3 of 4 passed

Task time Range: 3 min 27 sec ~ 7 min Range: 4 min 10 sec ~ 6 min

Error count 3 of 3 less than 10 error 4 of 4 had less than 10 errors

Behavior data

Measures Novice and experienced users

Know how to use

filter

1. Some criteria can be specified through the filter. Most users

complained that there should be more relevant filter options.

2. For the criteria that cannot be specified by the filter, only 1

user typed them in the search box, and others used those criteria

to examine potential targets one by one.



Subjective data

Questions Novice and experienced users

Do you think the search results are clearly

related to your search criteria?

2 of 3 said No.

Use number 1 to 7 to indicate difficulty level (1

= very easy, 7 = very difficult)

4 responses were collected:

1, 2, 2, and 4

Range: 1 ~ 4

Mean: 2.3 (Relatively easy)

Comments


- "I want more criteria, like if they have dishwasher, wash machine, and number of bathroom."

- "Single family house, townhouse, and apartments are not clearly separate."


- The "clear" button and the "done" button look so similar. I meant to press "done" but pressed

"click" instead. It is so annoying. I have to start all over again.




- This app sucks. I give up.


- "This was easier than what I wanted to find"

- "Damn. It jumps to the pictures."

- "Wish the filter had better options for cars."

- “Again?? I am sorry but I refuse to do this task again, it feels like the app is making a

mockery of me” (location task)

- “I use this app from time to time but you made me see the white light, I’ve never used it this

deeply, I used to search and if I find something I like I use the website instead, I am deleting

this app right now and downloading another one and see if that’s any better than this one.”



Problems revealed in subtask 2: Change filter

1. Filter is ill designed. Many key features are missing, such as mileage, year for cars

etc.

2. The function of search box is not clear to most users. There is a search box on the

top of results (see below), but users barely noticed or used it as a secondary filter.



Task 1 - subtask 3, 4, 5: Reply a post, Share a post, and Find direction

Please reply to the post by email, share this post to any of your friends. And find

directions through the app for this location.

Performance data

Completion Experienced users

Reply a post 2 of 4 passed

Share a post 2 of 4 passed

Find direction 0 of 4 passed

Subjective data

Subtask Comments from experienced users

Reply a post - "I am completely confused by those nonsense icons."

- "Confused by icons and layout."

Share a post - "Is there a way of doing it?"

Find direction - "I cannot do it. It says 'Missing Location'."

- "It is so slow. I don't want to wait any more."

- "What? Is it in the middle of the ocean?"

- "This again. I am not going to search the direction!"



Problems revealed in subtask 3, 4, 5

1. Icons are confusing.

2. Part of the process is performed outside of the app. It caused the most confusion

and users did not know how to get back and had to press the iPhone home button.

3. Bad user experience results in reliability concerns. Some users did not want to

contact or share a post using this app. One user said, "I would call them directly. I

already know this app sucks. I am not going to contact them using this app."



Task 2: A simple search task

Task 2: Look for jobs or pets

A: Look for jobs

Based on your current qualification, look for an appropriate job through craigslist. Since

this is a job, feel free to change the region of preference based on your need. If you plan

to move out of this state then go ahead and search accordingly. If you think that this job

would help another friend of yours then feel free to email this posting to him/her.

B: Look for pets

Assume you decide to get a new pet, look through craigslist and find a pet that you would

really want. Feel free to share this post if you wish to. No compulsion to do so. If you do

not have any particular pet in mind, then please search for a golden retriever pup.

Performance data


Completion 1 of 1 passed 4 of 5* passed

Task time 3 min 22 sec Range: 2 min 13 sec ~ 8 min 45 sec

Error count 4 successful users had less than 5 errors

1 unsuccessful user had more than 20

* one experienced user was tested both A: Look for jobs and B: Look for pets.



Behavior data


Select right location 1 of 1 passed 4 of 5 passed

Select right category 1 of 1 passed 5 of 5 passed

Know how to use

filter


Easily identify

targets of interest


Know how to

bookmark


Know how to save

search results


Easy navigation 0 of 1 passed 3 of 5 passed

Subjective data


Do you think the search results are clearly

related to your search criteria?

2 of 6 said No.

Use number 1 to 7 to indicate difficulty level (1

= very easy, 7 = very difficult)

6 responses were collected

either 1 or 2

Mean: 1.7 (Easy)



Comments


- "There is no 'full-time' option."

- "This is easier than searching."


- This is terrible!


- "No 'full-time' option in filter."

- "Filter is limited. No way to select experience level, salary range"


- “Where do I find pets? Hmmm”

- "I think the flaw was in the position I was looking for rather than the app."

- "Found what I was looking for and genuinely contacted the person posting the ad."



Problems revealed in Task 2: A simple search task

1. Navigation is still hard. Task 2 was the second search task. At this point users

became familiar with the app, and they generally rated this task easier than the first

one. However, navigation was still a problem. Users had problems finding the pet’s

category in the app.

Spent lot of time

trying to figure out

where the pets

section was and

made many errors.



2. Location must be specified before any search. This design is so awkward for the

users who wanted to search nationwide jobs. Cleaning up later is also cumbersome.



Task 3: Post an ad

Task 3: Post an ad

Now, think of a product at home you would like to sell (e.g. used book, bicycle,

computer, etc.). Then use this pre-created account to post an ad on Craigslist.

Account: [email protected]

Password: xxxxxx

Performance data



Task time Range: 7 min ~ 11 min 50 sec Range: 3 min 26 sec ~ 7 min

Error count < 5 errors < 5 errors

Behavior data







Subjective data


Do you think the posting

process confusing? If yes,

which part confused you?

- "There is no clear feedback. I actually don't know if I

successfully posted it or not. "

- "An error message with red font was shown when I

posted the ad. At first I thought it was something that I

needed to fix. After reading further I saw that it was just

verification for the post in email."

- "I don't know the difference between 'Hide' and

'Anonymize'"

- "why are there options visible when I'm not even able to

use them?"

- "The 'New Post' button was highlighted. I thought it is

the button that needs to be pressed. So I pressed it several

times but no responding. It turns out it is already pressed

by default."



- "There is only one account. Why I have to select the

account to continue? Otherwise, I cannot go anywhere."

Use number 1 to 7 to

indicate difficulty level (1

= very easy, 7 = very

difficult)

4 experienced users responded with 1, 2, 2, 2;

Mean: 1.8 (Easy)

2 novice users responded with 5 and 6

Mean: 5.5 (Hard)



Problems revealed in Task 3: Post an ad

1. Poor Feedback. When the message is presented, the screen shows a web view,

which is awkward and inconsistent. Using red font for a regular message causes

panic.

2. Items are not ordered in the conventional way. To post an ad, the user needs to (1)

select "Post new" first, (2) specify location, and (3) select an account to proceed. But

the three items are presented from bottom to top, which is so against web convention.

1

2

3



3. Bad Interface Dialog. Terminologies are vague and not intuitive.

Poor card sorting

and also the

subcategories are

confusing.



Task 4: Change your post


Please change the location to LA for the ad that you just posted. Please let me know if

you have problems.

…

Please delete this post, and please let me know when you finish.

Performance data



Subjective data


Do you think the posting

process confusing? If yes,

which part confused you?

- "I did not notice 'My Postings'"

- "I could not find a way to change the post to another city

without actually creating a brand new post"

- "Everything!"



Use number 1 to 7 to

indicate difficulty level (1 =

very easy, 7 = very

difficult)

3 responses collected: 2, 2, 1

Mean: 1.7 (Easy)

Problems revealed in Task 4: Change your post

1. There is no way to change the location to LA. Users can only change locations

within neighborhood. Only two experienced users reached that point.

2. Bad categorization. Many users did not think "My postings" should be in the "Post"

category. To users, "Post" is a verb and it is connected to a posting action, just like

"Search." But my posts should be in a history storage or organizer, similar to "My

favorites."





A: Look for apartments

Now let’s forget about the iPhone for a few minutes. Please use this laptop to do the same

task you did earlier, i.e. assume you want to rent an apartment in San Jose, and it has the

following features:

● 2 bedroom, 1 bathroom

● has at least one parking spot

● price under $2,500

● dogs allowed

Please use this laptop provided to you to do the same task on the craigslist website. The

url for the website in case you need it is www.craigslist.org

B: Look for cars

Now let’s forget about the iPhone for a few minutes. Please use this laptop to do the same

task you did earlier, i.e. assume you want to buy a used car, and it has the following

features:

● four-door sedan

● no earlier than 2005


● sold by owner (not dealer)

● mileage less than 80k



Performance data



Task time Range: 4 min ~ 5 min 30 sec Range: 3 min ~ 6 min 40 sec

Error count No obvious error No obvious error

Behavior data







Subjective data


Which experience did you

find better, the mobile

version or the web

version?

All users think the web is better

Do you think there was a

successful mapping or

similarity between the

website and the app?

All users think the app is very different from the web and

there is no mapping between the two versions.

Comments:

-- "They are completely different and the mapping is not

consistent. I would rather use mobile safari to browse

the web version than this terrible app. There is no value

in this app. Extremely frustrating."

-- "The website is more flexible when narrowing down the

results, and more easy when navigating."

-- " I found what I was looking for initially that I couldn't

find on the app….do you think that’s coz it’s a bigger

screen or something?"



Post-test interview

After usability test, 5 users (4 experienced and 1 novice, Participant #4 - #8) were

interviewed with the followed questions.

Do you think the labels clearly communicate their functions?

-- "No. Button captions are misleading. Cannot tell what the real functions are."

-- "No. Why are some of them sideways?"

-- "No. They are fuzzy. And the position is kind of weird."

-- "No. I don't use them at all."

Did you feel that you were guided step by step?

-- "Not at all. Pretty much by trial and error."

-- "No. I fumbled around a lot and gave up."

-- "Sort of…only when selecting location and category."

Do you think the items (such as buttons, boxes, links, etc.) are well organized on the screen?

-- "Not really. There is no 'OK' button to confirm my action. I have to click 'back.' It is confusing

because going back may cancel my early actions."

-- "No. everything seems scattered and inconsistent."

-- "Not at all. Why some buttons are at the bottom of the screen? Sometimes cannot see them."

Which part or process you feel is the most annoying, confusing, or frustrating?

-- "They are all annoying. The most annoying part is to set up the filter."

-- "All of it. Filtering is impossible, and when I was taken to the web craigslist site when I hit the



settings icon, I just about lost it!"

-- "I don't know if the item is what I need when scanning the result page. Font does not look

good. Page is kind of busy."

-- "The transition between screens… why does it take me here? And sometimes you can see crazy

big fonts and images… "

Do you have any suggestion for this app?

-- "First, clarify buttons labels and icons. And they should follow the web convention."

-- "Get rid of it!"

-- "Clean up the screen."

-- "Change colors."



3 novice users (Participant #1 - #3) were interviewed with a post-test questionnaire

(Appendix H). The statements and their corresponding answers are shown below.

Strongly

disagree

Disagree Neutral Agree Strongly

agree

It is easy to navigate through this app.

The buttons/labels/icons clearly

communicate their functions.

Clicking on links takes me to what I expect.

It is easy to find what I want through this

app.

The items (e.g., icons, buttons, pictures,

boxes, etc.) are well organized on the

screen.

I enjoy using this app.

I will use it again.

I will recommend this app to my friends

Participant #1

Participant #2

Participant #3



Problems revealed in Post-test interview

1. Overall very bad experiences. All users had negative attitudes to the product (They

did not want to use it again). Some users felt annoyed and frustrated.

2. Functionalities are not clearly communicated to users. Terminologies are vague

and confusing. Users were left with no choice but trial and error.

3. Navigation is hard. Users did not feel that they were guided at all.

4. Screen is cluttered. Items on the screen are not clearly organized. The order is

against convention and intuition.

5. Visual presentation is not appealing.



Data analyses summary by task

Passing rate (Performance data)

Difficulty level based on passing rate




Task 3: Post an ad

Task 5: A search task on web

No users successfully completed Task 1: A complete search task

More than half of the users failed on Task 4: Change your post

100%

29%

100%

83%

0%

0% 50% 100%



Task 3: Post an ad





Percentage of navigating smoothly (Behavior data)

Difficulty level based on navigation performance



Task 3: Post an ad



The navigation on the mobile app was much worse than the navigation on the web

There was a learning effect

100%

71%

60%

29%

11%

0% 50% 100%



Task 3: Post an ad





Self evaluation of difficulty (Subjective data)

(1 = very easy, 7 = very difficult)

Difficulty level based on self evaluation


Task 3: Post an ad




Task 1 was perceived the hardest and Task 5 was perceived the easiest, which was

consistent with performance data and behavior data.

1

1.7

3

1.7

3.3

0 1 2 3 4



Task 3: Post an ad





Data analyses summary by participant

Novice users (2 males and 2 females)

Longer Task Time

Compared with experienced users, novice users took much longer time. Therefore, within

one hour, novice users were tested with fewer tasks. For example,

Task time Novice users Experienced users

Task 1-subtask 1 Range: 5 min ~ 20 min


Range: 1 min 40 sec ~ 9 min

45 second


Task 3 Range: 7 min ~ 11 min 50 sec Range: 3 min 26 sec ~ 7 min

More errors

Compared with experienced users, novice users made more errors. They had hard time to

figure out some "accessory" functions, such as how to use filter, how to bookmark results,

and how to save data. Therefore, their navigation became very bumpy and error-prone.

For example,

Error count Novice users Experienced users

Task 1-subtask 1 3 of 4 had more than 10 errors 1 of 5 had more than 10 errors



More difficult

Compared with experienced users, novice users reported higher difficulty level. For

example,

Difficulty level Novice users Experienced users

Task 1-subtask 1 Mean = 4 Mean = 2.8

Task 3 Mean = 5.5 Mean = 1.8

(1 = very easy, 7 = very difficult)



Experienced users (3 males and 1 females)

Completion rate was not high

The four experienced users had used this app before the test. But they were not frequent

users. Some advanced functions, such as contacting and sharing, were actually new to

them. Therefore, their completion rate for those tasks was not higher compared to the

novice users.

Navigation was not smooth

Icons and labels are confusing and hard to remember. Organization of screen items is not

consistent and against the convention. Even the experienced users had difficulty to figure

out the meaning of an icon, what to do next, or how to get back to the early page.

Making errors

Buttons are small and hard to touch. They have different functions but look kind similar.

Those factors caused some errors. Generally, the experienced users realized the error and

fixed it right away.



Data analyses summary by high level questions

Does it communicate its purpose?

No. Users understood the overall purpose of this app. But they had hard time to figure out

the meanings of specific button labels, icons, and some functionalities.

Is it easy to navigate between functions?

No. For example, all users had difficulty to navigate from "Search" to "Reply" to "Share" to

"Find direction." And sometimes it was hard to figure out how to go back to the early page.

Does the screen design support efficient interaction?

No. Users reported that the organization of the screen items was against web convention and

people's intuition, which caused the interaction awkward and error-prone.

Am I guided through the interface or left on my own?

Users felt they were left alone.

Appendices


Appendices

Appendix A: Content form

Content Form

I agree to participate in this usability study conducted by XXXX.

I understand that the information is collected for research purposes only and that my name and

data will not be used for any other purpose. I understand that there is no minimum risk in the test

and no additional personal information will be collected.

I understand that I can leave at any time without any penalty.

I agree to immediately raise any concerns or areas of discomfort with the study moderator.

Your signature:____________________________________________________

Date:______________________________________________________________

Thank you!

We appreciate your participation.

Appendices


Appendix B: Pretest questionnaire

Please check the most appropriate selection: 1. Age range:

younger than 21

21-25

26-30

31-35

36- 40

older than 40

2. Gender:

Male

Female

3. Circle the one that fits you the best

I rent an apartment or house

I own an apartment or house

Others .

4. Circle the one that fits you the best

I barely drive cars and so far I don’t need a car.

I may want to get a car or replace the current one

I would never buy a used car

Others .

5. How frequently do you visit www.craigslist.org?

Never visit it --> skip Question 6 and go to Question 7

Only a few time and a while ago

Not frequently, only when there is a need

Frequently, whenever it can be useful

Very frequently, almost every day

6. What function(s) in Craigslist do you use or want to use?

Basically browsing posts

Contact people through Craigslist email

Post ads

Report scams

7. Have you ever use any Craigslist phone app?

Yes

No

If the answer is "Yes", have you ever use the app that we will test today?

Yes

No

Appendices


Appendix C: Modified test protocol

Tasks/Questions Observations/Comments

Task 1: Search (A: apartments for rent; B: cars for sale)

1A: Search apartments for rent

Instruction: Now, suppose you plan to move out of

the apartment that you currently live in and start

looking for candidate apartments through Craigslist

using this App. Please only search for the apartments

that you want to have a look.

Use Task sheet 1 -- search apartments

Observations

Select right location

Select right category

Know how to use filter

Easily identify targets of interest

Know how to save search data or

how to add to favorite

Appendices


Easy navigation

Measures

Completion: ___Yes ___No

Task time:

Error count:

Comments:

Appendices


Q: How many candidate apartments have you

found?

Q: Do you think the searching results are well

organized and clearly displayed on the screen?

___Yes ___No

Explain

Q: How was the experience for you? Please rate it

on this scale.

1= Very easy 7= Very difficult

Instruction: Now assume you want to rent an

apartment in San Jose, and it has the following

features:

● 2 bedroom



● dogs allowed

If you don’t have any question, please get started.

Observations

Measures

Appendices


And let me know when you finish searching.

Use Test sheet 1 -- search apartments change

filter


Task time:

Error count:

Comments:


found?


on this scale.


Instruction: Please reply to one of the posts via

email. Find directions through the app for this

location.

Observations

Appendices


Measures


Task time:

Error count:

Comments:

How difficult:


Instruction: Please share this post via email to any

of your friends.

Observations

Appendices


Measures


Task time:

Error count:

Comments:

How difficult:


Instruction: Please find directions through the app

for this location.

Observations

Appendices


Measures


Task time:

Error count:

Comments:

How difficult:


1B: Search cars for sale

Instruction: Now, suppose you plan to buy a used

car and start looking for candidate cars through

Craigslist using this App. Please only search for the

cars that you want to have a look.


Observations


Appendices



Use Task sheet 1 -- search cars






Easy navigation

Measures

Appendices



Task time:

Error count:

Comments:

Q: How many candidate cars have you found?

Q: Do you think the searching results are well


___Yes ___No

Explain


on this scale.


Instruction: Now assume you want to buy a used

car that has the following features:

● four-door sedan

Observations

Appendices






Please search for a car having the desired

qualifications



Use Test sheet 1 -- search cars change filter

Measures


Task time:

Error count:

Comments:


found?

Appendices



on this scale.


Instruction: Please reply to one of the posts via

email. Find directions through the app for this

location.

Observations

Measures


Task time:

Error count:

Comments:

Appendices


How difficult:


Instruction: Please share this post via email to any

of your friends.

Observations

Measures


Task time:

Error count:

Comments:

Appendices


How difficult:


Instruction: Please find directions through the app

for this location.

Observations

Measures


Task time:

Error count:

Comments:

How difficult:

Appendices



Task 2: More search (A: jobs; B: pets)

2A: For people who look for jobs

Instruction: Based on your current qualification

look for an appropriate full time job through

craigslist. Since this is a job, feel free to change the

region of preference based on your need. If you plan

to move out of this state then go ahead and search

accordingly. If you think that this job would help

another friend of yours then feel free to email this

posting to him/her.

Observations







Easy navigation

Appendices


Measures


Task time:

Error count:

Comments:

Q: How many jobs did you find which closely

matched your requirement?

Q: Do you think the search results are closely related

to your search criteria?

Yes No

Explain

Appendices



on this scale.


2B: For people who look for pets

Instruction: Assume you decide to get a new pet,

look through craigslist and find a pet that you would

really want.

Try searching with the given criteria.

Look for a Labrador pup in any of the nearby

locations, can change location if needed. Feel free to

share this post if you wish to. No compulsion to do

so

Observations







Easy navigation

Appendices


Measures


Task time:

Error count:

Comments:

Q: How many candidate pets have you found?



Yes No

Explain

Appendices



on this scale.


Task 3: Post an ad

Instruction: Now, think of a product at home you

would like to sell (e.g. used book, bicycle, computer,

etc.). Then use this pre-created account to post an ad

on Craigslist.


Please let me know when you finish posting.

Account: [email protected]

Pw: craigsphone

Observations



Easy navigation

Appendices


Measures


Task time:

Error count:

Comments:

How difficult:

1 = very easy; 7 = very difficult

Q: Was the posting process confusing to you? If yes,

which part?

Task 4: Make changes on your post

Instruction: Please change the location to LA for

the ad that you just posted. Please let me know when

you finish.

…

Observations


Appendices


Please delete this post, and please let me know when

you finish.


Easy navigation

Measures


Task time:

Error count:

Comments:

How difficult:

1 = very easy; 7 = very difficult

Appendices


Q: Was the edit/delete process confusing to you? If

yes, which part?

Task 5: Repeat Task 1 on the laptop

5A: For people who look for apartments

Instruction: Now let’s forget about the iPhone for a

few minutes. Please use this laptop to do the same

task you did earlier, i.e. assume you want to rent an


features:




● dogs allowed

Please use this laptop provided to you to do the

same task on the craigslist website. The url for

the website in case you need it is

www.craigslist.org

Observations:

Measures


Task time:

Error count:

Appendices


Use Task sheet -- search apartments change filter Comments:

Q: Which experience did you find better, the mobile

version or the web version?

Do you think there was a successful mapping or

similarity between the website and the app?

5B: For people who look for cars



task you did earlier, i.e. assume you want to buy a

used car, and it has the following features:

● four-door sedan








www.craigslist.org

Observations:

Measures


Appendices


Use the same Test sheet as Task 2 (2B)

Error count:

Task time:

Comments:





Appendices


Appendix D: Post-test interview questions

Post-test interview

Do you think the labels clearly communicate their functions? And why?

Did you feel that you were guided step by step? And why?

Do you think the items (such as buttons, boxes, links, etc.) are well organized on the screen? And

why?

Which part or process you feel is the most annoying, confusing, or frustrating?

Do you have any suggestion for this app?

Appendices


Appendix E: Checklist

Participant # ;

Time ; Date ;

Moderator ;

logger ;

Observer ;

Location .

Study Docs: Room Setup:

Consent Form Furniture Setup

Session Introduction Camera for recording figure movement

Task sheets Tripod

Pretest questionnaire Webcam for recording facial expression

Test protocol iPhone, battery, charger, clear history

Laptop for testing, clear history

Laptop for data logging

Laptop for recording

Gifts for participants

Pencils

Appendices


Appendix F: Task sheets

Task 1: Search apartments

Now, suppose you plan to move out of the

apartment/home that you currently live in and start



that you want to have a look at tomorrow.

If you don’t have any questions, please get started.

Please let me know when you finish searching.

Appendices


Task 1: Search cars

Now, suppose you plan to buy a used car and start

looking for candidate cars through Craigslist using

this App. Please only search for the cars that you

want to have a look at tomorrow.



Appendices


Task 1: Search apartments change filter

Now assume you want to rent an apartment in San

Jose, and it has the following features:

2 bedroom, 1 bathroom

has at least one parking spot

price under $2,500

dogs allowed

Please only search for the apartments that you want

to have a look at tomorrow.



Appendices


Task 1: Search cars change filter

Now assume you want to buy a used car and it has

the following features:

four-door sedan

no earlier than 2005

price under $8,000

sold by owner (not dealer)

mileage less than 80k

Please only search for the cars that you want to have

a look at tomorrow.



Appendices


Task 2: Look for jobs

Based on your current qualification look for an

appropriate full time job through craigslist. Since

this is a job, feel free to change the region of

preference based on your need. If you plan to move

out of this state then go ahead and search

accordingly. If you think that this job would help

another friend of yours then feel free to email this

posting to him/her.

Appendices


Task 2: Look for pets

Assume you decide to get a new pet, look through

craigslist and find a pet that you would really want.

Try searching with the given criteria.

Look for a Labrador pup in any of the nearby

locations, can change location if needed. Feel free to

share this post if you wish to. No compulsion to do

so

Appendices


Task 3: Post an ad

Now, suppose you want to sell this book through

Craigslist. Please use this pre-created account to post

an ad. Feel free to take pictures using this phone.



Appendices



Please change the location to LA for the ad that you

just posted. Please let me know when you finish.

…


you finish.

Appendices


Appendix G: Early version test protocol

Tasks/Questions Observations/Comments

Task 1: Search (A: apartments for rent; B: cars for sale)

1A: Search apartments for rent

Q: Please describe the ideal apartment that you want

to rent. How many bedrooms, bathrooms, parking

spots, price range, pet allowed or not, which floor,

location, etc.?

What are the key features for you?

# Bedrooms

# Bathrooms

Price Range

pets

Sq. Footage

Location

Key Features

Instruction: Now, suppose you plan to move out of

the apartment that you currently live in and start



Observations

__ complete task

Appendices


that you want to have a look.



__ select right location

__ select right category

__ know how to use filter

__ easily identify targets of interest

from all candidates

__ know how to bookmark and check

detailed information

__ easy navigation

Measures


Task time: __________

Error count: __________

Appendices



found?

Q: How was the experience for you to set up the

search criteria? Please rate it on this scale.




Yes No

Please explain

Q: Do you think the search results are well


Yes No

Please explain

Q: Did you save the information of those candidate

apartments that are desirable for you?

Yes No

If yes, please show how to retrieve the

information.

Observations:

Appendices


__ identify the icon

__ select the post

Measures:


Any behavioral and subjective data

If no, explain

Q: How are you going to contact them? If use the Craigslist app (circle one)

● Craigslist account

● other email

● text message

and please ask him/her to show how

Appendices


Observations:

__ complete task


__ select the right method of contact

Measures:



If not by using this app, explain

1B: Search cars for sale

Q: Please describe the ideal car that you want to buy.

Type (sedan, minivan, SUV, crossover, convertible,

Type

Color

Appendices


etc.), color, year, history (e.g. first owner, accident

history), price range, etc.?

What are the key features for you (e.g. high MPG,

tough, easy maintain)?

Price Range

No earlier than year

Mileage

Key Features

Instruction: Now, suppose you plan to buy a used

car and start looking for candidate cars through

Craigslist using this App. Please only search for the

cars that you want to have a look.



Observations

__ complete task





from all candidates



__ easy navigation

Appendices


Measures


Task time __________

Error count __________



Q: How was the experience for you to set up the

search criteria? Please rate it on this scale.




Yes No

Please explain

Q: Do you think the search results are well


Yes No

Appendices


Please explain

Q: Did you save the information of those candidate

cars that are desirable for you?

Yes No

If yes, please show how to retrieve the

information.

Observations:


__ select the post

Measures


Appendices


Q: How are you going to contact them? If use the Craigslist app (circle one)

● Craigslist account

● other email

● text message

and please ask him/her to show how

Observations:


__ select the right method of contact

Measures


Appendices


If not use this app, explain

Task 2: Change the filter for Task 1


Instruction: Now assume you want to rent an


features:




● dogs allowed

Observations:

__ complete task




Appendices


Please search for an apartment having the desired

qualifications.


from all candidates



__ easy navigation

Measures






found?



Yes No

Please explain


Instruction: Now assume you want to buy a used Observations:

Appendices


car that has the following features:

● four-door sedan





Please search for a car having the desired

qualifications.

__ complete task





from all candidates



__ easy navigation

Measures





Appendices




Yes No

Please explain

Task 3: Repeat Task 2 on the laptop




task you did earlier, i.e. assume you want to rent an


features:




● dogs allowed




www.craigslist.org

Observations:

__ complete task





from all candidates



__ easy navigation

Appendices


Measures







Please explain your reason for saying

so




so




Observations:

Appendices


task you did earlier, i.e. assume you want to buy a

used car, and it has the following features:

● four-door sedan








www.craigslist.org

__ complete task





from all candidates



__ easy navigation

Measures




Appendices






so




so

Task 4: Post an ad

Instruction: Now, think of a product at home you

would like to sell (e.g. used book, bicycle, computer,

etc.). Then use this pre-created account to post an ad

on Craigslist.



Observations:


__ successfully log in

__ select account to proceed

__ know how to select area

__ select a right category

__ select a city/neighborhood

__ input title/price range/description

Appendices


__ know how to randomize/show/hide

email

__ know how to insert images

__ understand confirmation message

Measures




Q: Do you think the labels clearly communicate their

functions?

Yes No

If no, explain

Q: How easy or difficult to navigate between

functions?


Appendices


Q: Was the posting process confusing to you? If yes,

which part?

Yes No

If yes, explain

Q: Did you feel that you were guided step by step? Yes No

If no, explain

Q: Do you think the items (such as buttons, boxes,

links, etc.) are well organized on the screen?

Yes No

If no, explain

Task 5: Make changes on your post

Instruction: Please change the location to San

Francisco for the ad that you just posted. Please let

me know when you finish.

…

Observations:

__ select account to proceed

__ successfully change the location

__ understand confirmation

Appendices



you finish.

Measures:




Q: Do you think the labels clearly communicate their

functions?

Yes No

If no, explain

Q: How easy or difficult to navigate between

functions?


Q: Was the edit/delete process confusing to you? If

yes, which part?

Yes No

Appendices


If yes, explain

Q: Did you feel that you were guided step by step? Yes No

If no, explain

Q: Do you think the items (such as buttons, boxes,

links, etc.) are well organized on the screen?

Yes No

If no, explain

Appendices


Appendix H: Early version post-test questionnaire

Please circle the statement that is most appropriate to you

It is easy to navigate through this app. Strongly

disagree


agree

The buttons/labels/icons clearly communicate

their functions.

Strongly

disagree


agree

Clicking on links takes me to what I expect. Strongly

disagree


agree

It is easy to find what I want through this app. Strongly

disagree


agree

The items (e.g., icons, buttons, pictures, boxes,

etc.) are well organized on the screen.

Strongly

disagree


agree

I enjoy using this app. Strongly

disagree


agree

I will use it again. Strongly

disagree


agree

I will recommend this app to my friends Strongly

disagree


agree

Documents

Usability Test Report for Craigslist Mobile iPhone App€¦ · quickly became an advertising portal in 1996. Today, Craigslist is used in over 70 countries worldwide and supports