32
NEXTRAVEL Usability Testing Findings and Recommendations February 2012

Usability Testing Report

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: Usability Testing Report

NEXTRAVELUsability Testing Findings and Recommendations

February 2012

Page 2: Usability Testing Report

2

Usability research goals

• Primary usability research goals• Observe Tumbler user experience to determine:

whether evaluators understand the tumbler functionality how evaluators perceive and interact with the tumblers Interest of using tumblers to leverage their searches

• Observe Detail View user experience to determine: whether evaluators can easily find out the detail view how well the detailed information can serve their needs

• Observe Filter/NLA user experience to determine: whether evaluators can easily locate the filter/ NLA button how well the filters/NLAs can help narrow results

• Observe Map user experience to determine: whether evaluators can easily locate the map functionality whether evaluators can accurately interpret all the information in the map Interest and engagement with the map

• Observe Ambiguous Search user experience to determine: whether evaluators understand the ambiguous search functionality how evaluators perceive and interact with the recommendations in the tumblers Interest of using Ambiguous Search

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 3: Usability Testing Report

3

Key insights: summary

• Key insights:• Tumbler Search: clearly understood, but not always useful

• Detail View: considered useful and comprehensive

• Filter/NLA: considered useful but not easy to find out

• Map: not easy to find or understand

• Ambiguous Search: consistent with evaluators’ expectations, but the quality was not good enough

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 4: Usability Testing Report

4

Key insights: Tumbler Search

• Tumbler service clearly understood• All evaluators could use tumblers to narrow their search results• 7 out of 9 evaluators thought it was cool to search hotels by using

tumblers “I think the spinners are a good idea. It does actually help you find a hotel

quicker that close to particular areas that you would like to get to.”----P2 “I definitely like using the spinners. I've never used anything like this before. The

spinners were so convenient and easy to use, even for a first time user of the website, like myself. ” ----P8

• Even though P2 said the tumblers were cool to use, she also mentioned to get rid of the tumblers: “It's frustrating me that I cannot see this(the hotel list) in a bigger way. I don't

see a way to get rid of this(the tumblers) either. I would really get this(tumblers) a little smaller… maybe just focus on one at a time. ”----P2

• Only P7 expressed strong dislike of using the tumblers because she couldn’t stand too much movement “I wish to get rid of the screen(tumblers) and only look at the hotel list below the

spinners. I would rather have only one thing move at a time. I wish the movement on the hotel list could be toned down.”----P7

• One out of 9 evaluators didn’t express explicitly his attitude for using the tumblers to narrow searches “I don’t know how much I like the spinners.”

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 5: Usability Testing Report

5

Key insights: Tumbler Search

• Thoughts In the usability tests, evaluators were dictated to use the tumblers to

accomplish certain tasks

The tumblers took up from 1/3 to 1/2 of an evaluator’s screen vertically

Even though most of them felt “cool” about using the tumblers, we don’t know how much they would like to keep using the tumblers for all hotel searches

• Recommendations Need to further explore whether should make the tumblers hidable

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 6: Usability Testing Report

6

Key insights: Tumbler Search

• Neighborhood feature not always useful• 3 out of 9 evaluators thought the neighborhood feature was only

useful for people who had been to the place before “Show neighborhood function only to people who have been to the place before.

The neighborhood function looks annoying and stupid to me.”----P4

“Have a prompt saying 'have you been to this place before?', and use that to control the comp up of neighborhoods and landmarks. ”----P4

“I can only recognize Loop, Museum Campus and Airport of the neighborhoods because I’m not living there. The neighborhood tumbler can get confusing if I’m just casually browsing. The least useful feature for me is the neighborhoods.” ----P6

“I’ll use the neighborhood and landmark function if I know the city.”----P10

• One evaluator(P7) skipped task 2, and she strongly disliked using tumblers

• Even thought one evaluator(P9) claimed he thought the neighborhood feature was useful, he didn’t use it to find a hotel before asked explicitly to

• Recommendations Experiment using landmarks as the default content for the 2nd tumbler

Shorten the neighborhood list©2012, Proprietary and Confidential property of Nextag, Inc.

Page 7: Usability Testing Report

7

Key insights: Tumbler Search

• Landmark list too long• One evaluator said the landmark list in the 2nd tumbler was too

long and needed too many clicks

• Most evaluators thought it was helpful to have the landmark information in a city

• Recommendations Shorten the landmark list to only give most famous landmarks

Rank landmarks by its popularity

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 8: Usability Testing Report

8

Key insights: Tumbler Search

• Tumbler names/links not clear and easy to see• P4 scrolled the “Destination” tumbler to look for Loop neighborhood,

and she said she didn’t notice the “Destination” tumbler was the “City” tumbler [4:30]

• P6 said it’s hard to see the links above the tumblers “Oh…I see! You can change this(the 2nd tumbler) to landmarks! That’s

interesting. I didn’t know that. “ “If they(filter options above the tumblers) were a little easier to see, I would

have started using them sooner.”

• Recommendations Change the name destinations to cities

Make links above the tumblers more prominent

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 9: Usability Testing Report

9

Key insights: Tumbler Search

• Scrolling not fast enough• One evaluator thought the tumblers were a little sticky, and 3

other evaluators tried to type to accelerate the scroll of the tumblers

“They got a little bit stuck, but they are not so bad.”----P2 “I would like to type and see the spinners scroll to the destination.” ----P5 “It’ll be cooler if I can just type to let the spinners spin.”----P6 “I want the tumblers to go faster because it took a long time going from A to

Z.”----P10

• One evaluator wished that the tumblers could keep scrolling when he pressed the arrows

• Other evaluators didn’t see any problem of scrolling the tumblers, and some of them thought the scrolling was very convenient and seamless.

• Recommendations Enable type to accelerate the scroll

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 10: Usability Testing Report

10

Key insights: Tumbler Search

• Scrolling surprises• One evaluator dragged the tumblers to scroll, but she

unconsciously clicked the empty space somewhere in the tumbler and surprisingly found that the tumbler spinned in the reverse direction(P2 [10:30])

“Sometimes the spinners are a little funny. I think the spinners is fine…Maybe a little bit tweaky” ----P2

• One evaluator used the scroll arrows in the 1st tumbler and found it was a little frozen (P3[00:15])

• Recommendations Clearly inform users all the ways they can scroll or possibly move the

tumblers

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 11: Usability Testing Report

11

Key insights: Tumbler Search

• Defaulted sort order not clear • 2 evaluators(P2 & P9) changed the sort order from “Best

Match” to “Price low-high”• 2 evaluators questioned the sort order of hotels

“It gives you hotels recommended. I’m not sure what they based that on. ” ----P5

“to be helpful to know why you recommend it. Ur…probably values, right?”----P9

• One evaluator said she might want to see the hotel list ordered by price

“They(the hotel list) are sorted by best match. I guess that's OK. I may want to look by price though. ” ----P8

• Recommendations Further explore what’s the best default order

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 12: Usability Testing Report

12

Key insights: Tumbler Search

• Poor readability • Some evaluators reported problems of reading the contents in

the tumblers One evaluator said his first impression was that the website was too text-

heavy(P3)

One suggested to use another color, like yellow or light orange, for the highlighted search result in the tumblers to differentiate it from other un-highlighted text

• Recommendations ????I don’t know…it’s the problem of using tumblers

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 13: Usability Testing Report

13

Key insights: Detail View

• Detail View is useful and necessary• almost all evaluators liked the information in detail view

8 evaluators thought the detail view gave them all the information that they could possibly need. Only one evaluator said he disliked the detail view and it was not clean, but he didn’t give any reasons for this opinion.

©2012, Proprietary and Confidential property of Nextag, Inc.

All evaluators could easily find how to open the detail view, get hotel rates and book a hotel. 2 evaluators mentioned it would be more desirable if they could book a hotel directly from the website.

Page 14: Usability Testing Report

14

Key insights: Detail View

• Detail View problems• 2 evaluators expected to see long and detailed descriptions

2 evaluators found there were only one or two sentences for the hotel description, so they complained it was too short

• One evaluator wanted to know breakfast information• 2 evaluators suggested other ways of displaying amenities

One would like to have both amenity icons and an amenity list One suggested not to show the icons of unavailable amenities---” I don't see

the points of actually displaying them, if something is not available. It created confusion. If something is not available, either put a line through it or throw it. ”(P3)

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 15: Usability Testing Report

15

Key insights: Detail View

• Detail View problems• One evaluator clicked the highlighted hotel name in the 3rd

tumbler to open the detail view “That would maybe open up a pop-up window, but it's not.”---P8

• Thoughts Once the tumblers become the main focus, user will expect to get all

the information from the tumblers

• Recommendations Need to further explore all the advantages and defects of using 2

exactly same hotel lists

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 16: Usability Testing Report

16

Key insights: Detail View

• Detail View problems• most evaluators thought the star ratings were guest/user

ratings One evaluator expected to see user reviews in detail view by clicking the hotel

stars, and he was surprised that he couldn’t find any recommendations for giving 3 stars to the hotel

• Recommendations Include user rating contents from TripAdvisor

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 17: Usability Testing Report

17

Key insights: Filters/NLAs

• Filters/ NLAs are clear & easy to use• all evaluators liked the filters/NLAs

One said the filter/NLA options covered all the thing he wanted One said the filters/NLAs were perfect and extremely clear. She rated

it ten out of ten.

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 18: Usability Testing Report

18

Key insights: Filters/NLAs

• Filters/ NLAs problems• 2 evaluator said the filter/NLA button might be hard to see• Some evaluators said the amenity icons were hard to recognize

when not highlighted “At first it (the pool icon) looked like jacuzzi, but if you hove over, it would tell

me.”----P2 “That’s a pool?! Could be a little clearer about that!”----P9 Some evaluators didn’t notice there were tool tips until they 2nd time hovered

over amenity icons Evaluators were guessing the meanings of each amenity icon before they

checked through all the tool tips

• Some evaluators thought star ratings were guest/user ratings• Recommendations

Further explore ways to place the filter/NLA button

Re-design amenity icons to make them easier to recognize

Include user ratings as another filter/NLA option

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 19: Usability Testing Report

19

Key insights: Filters/NLAs

• Narrow it?• Immediately, P3 clicked on “Narrow your results” to find a

hotel around Navy Pier, even though he had already noticed & opened the landmark list

• Immediately, P4 clicked on “Narrow your results” to find a hotel closest to the beach

• Thoughts Some well-known travel websites allow users to select hotels close to

a certain neighborhood or landmark under the filters/NLAs

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 20: Usability Testing Report

20

Key insights: Filters/NLAs

• Narrow it?• Immediately, P3 clicked on “Narrow your results” to find a

hotel around Navy Pier, even though he had already noticed & opened the landmark list

• Immediately, P4 clicked on “Narrow your results” to find a hotel closest to the beach

• Thoughts Some well-known travel websites allow users to select hotels close to

a certain neighborhood or landmark under the filters/NLAs

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 21: Usability Testing Report

21

Key insights: Ambiguous Search

• Ambiguous Search approximated user expectations• 7 out of 9 evaluators thought the tumblers should show places

that actually had nice ocean beaches when they searched “beach” the 7 evaluators seemed very excited about the ambiguous search feature.

One said the function seemed unique. the 7 evaluators checked the entire region list to prove their initial guess,

most of them questioned the recommendation quality since they saw Arizona, Indiana and Kentucky were also in the list

• Recommendations Tell users explicitly what will happen after searching ambiguous

keywords

Improve the search quality

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 22: Usability Testing Report

22

Key insights: Ambiguous Search

• Ambiguous Search approximated user expectations• 2 out of 9 evaluators couldn’t understand the defaulted

recommendations in the tumblers by searching “beach” One couldn’t understand why the 1st tumbler changed from a city list to a

region list after searching “beach”. He said it was very confusing and not user-friendly. “It's putting me in Florida now, which I don't understand. It(the 1st tumbler) should be a search filter by actual city and I don't know where it is. It's not user friendly. ”---P3

One said the search was not clear.” It seemed to go to the destination with the word ”beach” in it, not the beach I want to go to.”----P10

• Recommendations Inform novice users the website is capable of ambiguous searches

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 23: Usability Testing Report

23

Key insights: Ambiguous Search

• Ambiguous Search approximated user expectations• One evaluator disliked the default recommendation by

searching beach “I can see why pick Miami Beach because that was… I don't know…maybe it's

the most popular search under the word beach. I don't know Miami Beach is the destination I want to go. Maybe I want to go to West Palm Beach or somewhere that just simply has a beach, like St. Augustine”----P5

• Thoughts No users knows how the search engine works in our website (our

website hasn’t established its reputation yet)

The recommendation in the tumblers looks too strong and pushy, but it doesn’t give any explicit reasons

Personalized recommendations are more desirable

• Recommendations Need further explore ways to give “desirable recommendations”

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 24: Usability Testing Report

24

Key insights: Map

• Map feature is desirable• All evaluators thought the map feature was great

Some evaluators liked clean maps and they thought no more information was needed in the hotel map

Almost all evaluators could properly interpret the information in the maps

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 25: Usability Testing Report

25

Key insights: Map

• Map problems• 6 evaluators wished the map button could be more prominent

One said she didn’t notice the map until she was asked to find it out explicitly by the task

One wished the map could be more obvious because he almost didn’t see it

• one evaluator(P10) didn’t find the, and he supposed the map in detail view was the one mentioned in the final task

• 3 evaluators had problems of closing the map The map close button went down below the bottom of one evaluator’s

browser, so he couldn’t find how to close the map(P6[21:35]) One evaluator wished the map close button could be bigger

• Recommendations Further explore ways to place and design the map button/icon

Make the map shorter or place the close button in the upper right corner of the hotel map

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 26: Usability Testing Report

26

Key insights: Map

• Map problems• All evaluators didn’t notice the city map before they were

asked explicitly to tell the intention of 2 maps. 7 evaluators precisely got the function of each map.

• Two evaluators didn’t answer all the map questions• Recommendations

Provide description about each map’s functionality

Experiment using different markers in the city map and the hotel map

Give city name pop-ups when users mouse over the markers in the city map

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 27: Usability Testing Report

27

Key insights: Map

• Map problems• One evaluator(P9) mentioned he would like to see more hotels

instead of one or three, suddenly he noticed the map was connected to the tumbler above it

• One evaluator(P5) complained that the information in the pop-ups was the same with the 3rd tumbler list, and he would rather to have no pop-ups in the map

©2012, Proprietary and Confidential property of Nextag, Inc.

RecommendationsFurther explore the advantages and defects of using tumblers to navigate the map

Page 28: Usability Testing Report

28

Key insights: Map

• Map problems• Some evaluators said they would also like to see major

attractions, promotions or restaurants in the hotel map• One evaluator said he would like to see public/private

beaches in the map when he searched “beach”• Recommendations

Include more information to make the map more useful

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 29: Usability Testing Report

29

Appendix

APPENDIX

Page 30: Usability Testing Report

30

Participants and test procedure

• Web site participant attributes

• Online usability sessions:o 9 participants: 4(female) + 5 (male)o Ages: 4(18 – 34) + 5(35 – 55)o Income: $40K – $100Ko Averaged Web Experience

• Usability test procedure• Online web site research conducted via usertesting.com

o Evaluators completed several directed searches for hotels in Chicago, then reviewed the findability and usability of Detail View and Filters/NLAs. Evaluators also completed a directed ambiguous search for hotels in places that had beaches, then reviewed the findability and usability of Map.

©2012, Proprietary and Confidential property of Nextag, Inc.

Page 31: Usability Testing Report

31

Test plan

Scenario: Imagine you are looking for hotels for an upcoming vacation. You are considering Chicago because there are rich things to see and do. You are also thinking about going to beaches to enjoy the sunshine and water. You will be using a travel website to research and book quality hotels with good price. Please use Safari or Chrome for all the tasks.

Task 1. Go to [http://travel.nextagqa.com/] and search for a hotel in Chicago. When you’ve found a hotel you like, briefly describe what you found and why you like it. You do not need to make a booking.1.What is your initial impression of this site?2.Were you able to find a hotel you like?3.What site features, if any, were most useful?4.What site features, if any, were least useful?

Task 2. Making sure you are still on the search results page, use the 3 Spinners below the search bar to find a hotel in the Loop neighborhood of Chicago and another hotel close to an attraction called Navy Pier in Chicago, briefly describe what you found.1.What information does each spinner provide? 2.What, if anything, did you like about using the spinners to find the hotels?3.What, if anything, did you dislike?4.How can the process of searching using the spinners be improved?

Task 3. You’d like to do more research about the hotel you found near Navy Pier:1.Is there a way to look at detailed information about the hotel?

Task 4. If you haven’t done so already, please click on the hotel name in the yellow frame with a green border:1.Is this the information you expected? Please explain. 2.Is there a way to look at the rates for this hotel? 3.Is there a way to book this hotel?

Task 5. Go back to the home page:1.Is there a way to only look at 3 star hotels with pools?2.If you found a way to only see 3 star hotels with pools, were the filter options clear?3.What would you do to improve the process of filtering hotels?

Task 6. Imagine you’d like to find a hotel close to a beach. Type ”beach” into the search box, then use the spinners to find a hotel you find interesting. Briefly explain what you found.1.Is this what you expected before you started to search? Please explain.2.What, if anything, did you like about using this search method and spinners to find the hotel?3.What, if anything, that bothered you about finding an interesting hotel?4.How could the process of searching using the spinners be improved?

Task 7. You’d like to stay at the hotel closest to the beach.1.Is there a way to see the location of the hotels?2.What do you think the two maps are for?3.What do you think the green markers on the map mean? 4.What do you think the red markers on the map mean?5.What other information would you like to see on the map?

©2012, Proprietary and Confidential property of Nextag, Inc.