Memo

To: Karen Bennett, Manager, User Experience, IHGcc: Dr. Carol Barnum

From: CBR China: Yufei Duan, Yina Li, Ying Li, Qianying Liu, Niven Sellars, Michael Somer

Subject: Usability Testing Findings Report for Chinese Holiday Inn website

Date: April 25, 2008

Included with this memo is team CBR China’s final report for our research project on the Holiday Inn’s Chinese website. The following are included within this report:

Project purpose Goals set by Team CBR China Processes followed during the project

o Heuristic reviewo Test plano Recruitment and screeningo Real-time loggingo Analysis

Categorized results and findings Recommendations

In addition to the aforementioned items included in this report, team CBR China also includes appendices of our test logs, participant consent forms, screeners, questionnaires, scenarios, and other materials in support of testing. Other research components include the team’s heuristic evaluation, personas, and test plan.

Team CBR China would like to extend its gratitude to our sponsor, Karen Bennett, as well as our instructor Dr. Carol Barnum, who has assisted us throughout this project. Without both of you we would not have had the chance to engage in a real world usability experience.

Thank you again,The members of team CBR China

1

TABLE OF CONTENTS

EXECUTIVE SUMMARY............................................................................................................................3

TEST OBJECTIVE...........................................................................................................................................3OUR PROCESS...............................................................................................................................................4OUR RESULTS...............................................................................................................................................4BACKGROUND..............................................................................................................................................5REPORT OVERVIEW......................................................................................................................................6

METHODOLOGY.........................................................................................................................................7

OVERVIEW....................................................................................................................................................7USER PROFILES............................................................................................................................................7USABILITY CRITERIA....................................................................................................................................8TEST OBJECTIVES.........................................................................................................................................9

FINDINGS AND RECOMMENDATIONS...............................................................................................17

POSITIVE FINDINGS....................................................................................................................................17Positive Impressions.............................................................................................................................17

SEVERITY RATINGS....................................................................................................................................19COLLATED FINDINGS, RATED BY SEVERITY...............................................................................................19USER IMPRESSIONS....................................................................................................................................21

General comments................................................................................................................................21Negative Impressions............................................................................................................................21Suggestions...........................................................................................................................................21Findings and Recommendations...........................................................................................................22

OTHER FINDINGS........................................................................................................................................27Miscellaneous Findings........................................................................................................................28Localization Findings...........................................................................................................................29

POST-TASK ANALYSIS............................................................................................................................31

FUTURE DEVELOPMENT AND TESTING......................................................................................................36

CONCLUSION.............................................................................................................................................38

FINDINGS....................................................................................................................................................38COMMENTS.................................................................................................................................................38

2

Executive Summary

Test Objective

Karen Bennett, Manager of User Experience at IHG, presented her concerns for the usability study of the China Holiday Inn website as follows:

Do users connect better with the Holiday Inn or home page or a competitor site, such as www.elong.com?

What features do the users connect with and like (IHG vs competitors)?

Does the booking process work for the Chinese user and if not, why not?

Does a competitor site have a better booking process, and if so, why?

Our study focused on gathering qualitative and quantitative data to address IHG’s concerns about the usability of the site.

Based on IHG’s concerns and the heuristic evaluation we conducted, we assessed the ease and difficulty of the following hotel booking tasks on www.HolidayInn.com.cn:

The general feeling/layout of the site: Is it easy to find the information users want to know the first time they visit?

The procedure of booking a hotel online: Basic searching: is it easy to use? Entering personal information: Does the website require

reasonable and suitable information for Chinese users? Do the users understand all information requirements?

Language: Can the users understand the language used on the Holiday inn website? Do users understand all the phrasings in the Holiday Inn website? Are there any translation mistakes or misunderstandings?

Satisfaction with the site: Which aspects do users like with the site and which aspects do they dislike?

3

Perceived reliability of the site: Do users trust the website? Do they fill out the personal information readily?

Navigation: Can users find clear and efficient navigation when booking a hotel or browsing the website?

Compare Holiday Inn website to competitor site: How do users feel about the elong.com website in comparison to the Holiday Inn site?

Compare Holiday Inn China website to the Holiday Inn English website: How do users feel about the English version of the same site?

Note: Since we concluded our usability testing on the Holiday Inn China website, the following changes have been made to the website:

There is now a map feature available when search results are displayed.

The input fields have now been changed from the upper right-hand corner to the center of the page.

Our Process

The purpose of the Chinese HolidayInn.com.cn usability test was to collect feedback about how users use the Holiday Inn online hotel booking site in China, what problems they may encounter using the site, what improvements they would like to have, and what additional information they desire. Following industry standard methods of usability testing, our team undertook an evaluation of the site, developed user profiles, and developed a six-scenario test plan.

Our Results

The team found many consistent issues among users that were areas of concern. Our findings have been divided into seven categories:

Aesthetics: The look and feel of the website, and whether or not it is also functional

Navigation: How well users can find their way around the website

Layout: The arrangement of page elements and their effectiveness at guiding the user

4

Feature Requests: Options that users want and expect but do not find

Brand Identity: How well users recognize Holiday Inn and other IHG hotels as a brand

Mental Model: The process flow and design users expect to find and how well the website matches their expectation

Based on the feedback we received from our participants as well as our analysis of the quantitative data from testing, we identified 14 significant usability issues with the Holiday Inn China website. The following table lists these issues and indicates their impact on the usability of the site, along with the number of participants who experienced each problem.

Usability Problem # Affected SeverityNavigation: The ‘start over’ and ‘finish booking’ options on the confirm booking page confused the user

1/6 low

Layout: Users found the Priority Club Login options confusing. 4/6 high

Layout: Users think advertisements are overwhelming 3/6 medium

Layout: Users want the total price for their stay to be listed, instead of a per-night rate only

3/6 high

Layout: Users have trouble finding the cancellation link to cancel their reservation

3/6 high

Layout: Error messages were not seen by the user or they were misinterpreted

5/6 high

Aesthetics: Search results of hotels are displayed in a two-color format, confusing users’ perception of how many results are actually being displayed

2/6 low

Feature Requests: Users complain about not seeing promotional rate information

2/6 low

5

Feature Requests: Users want to see a drop-down list of cities instead of having to type city names in

3/6 high

Feature Requests: Users want to see hotel rankings by different categories (customer experience and price)

4/6 medium

Feature Requests: Users want more information on hotel amenities

4/6 high

Mental Model: Users are confused as to whether or not they are required to input full Chinese name, or just family name

5/6 high

Feedback: Loading screen has no progress bar. This confuses users as to whether or not their information is being processed

6/6 high

6

Background

We are the Chinese Holiday Inn Website usability group, a team of four Chinese students and two American students in SPSU’s Bachelor of Science in Technical Communication degree program. Dr. Carol Barnum, the professor of our usability testing and research course, suggested the idea of this usability test of the Chinese Holiday Inn website.

Karen Bennett was interested in having the Chinese Holiday Inn website evaluated for usability questions and issues. In the initial meeting, she discussed the scope of the site and its typical users, as well as competitor sites. She also discussed what concerns she already had about the website going into testing; for example, she wanted to make sure that there were not any translation issues.

From the information provided by the sponsor, the project team learned that the audience for the website is mostly domestic (mainland) Chinese. They book hotel rooms online for business and/or leisure purposes. Their ages span from the mid-twenties to the fifties. Most of them are not comfortable using computers. The majority of them do not speak or read English.

Purpose

Our team worked with Ms. Bennett’s information and questions to more fully define:

The scope—How much of the site would be included in the evaluation

The user base—What type of users work with the site Use cases—Typical tasks users are expected to perform

We experienced some difficulty in finding participants within the identified age bracket; therefore we expanded our user’s age range to include the early twenties as well.

We found our users based on these criteria; however, all of our users (because they live in the US) speak and read English.

The outcome of this analysis resulted in the creation of two personas, a heuristic evaluation, and a detailed test plan with a list of suggested tasks and questions for the test sessions.

7

Overall, our purpose was to collect information about how users interact with the website and to identify areas in which the site could be improved to better benefit its users.

Report Overview

The following major sections are included in our report:

Methodology Post-Task Analysis Findings and Recommendations Conclusion

The appendices of this report include all materials that are relevant to the usability test so that, if needed, subsequent usability tests can be easily conducted. The appendices include the following:

Appendix A: Heuristic Evaluation—Describes the criteria we used in our initial evaluation of the website, and identifies our areas of concern that were later used to create the test plan

Appendix B: Personas Report—Describes the characteristics, goals, needs, and preferences of our anticipated users.

Appendix C: Test Plan—Describes user types, team roles, test objectives, and methodology for recruiting participants.

Appendix D: Participant Recruiting Materials—Pre/Post task/test questionnaires, screener questionnaire

Appendix E: Testing materials—facilitator scripts, consent form, scenarios

Appendix F: Session Logs—information and notes on each individual usability test conducted.

Methodology

Overview

This section of the report provides information on the following: Typical user profiles for the Holiday Inn China website Criteria by which we judged the usability of the site Goals for the test Profiles of the actual test participants Test location and lab equipment Tasks presented to participants during testing

8

Other materials used throughout our testing process

User Profiles

We based our search for users on the two personas we created. These personas were created based on the information we were given about the site’s actual users.

The situation of testing a Chinese website within the United States is somewhat difficult to realistically accomplish. Our users speak and read English, although many of the Holiday Inn China’s users do not. Because of this, the test results only represent one of the user groups. The characteristics of this user group, along with other questions about prior experience, behaviors, and other criteria, made up a screening questionnaire (Appendix A) for identifying suitable test participants.

Here is a description of an ideal user: someone who is 45 years old and has an eight years old son. The family has been in the United States for about 13 years. She works part-time in her husband’s computer shop. The family income is about $90,000 a year. She is not very comfortable using computers and internet. She use internet for email and information. She likes travel and has booked hotel rooms online before. She wants to go back to China this summer for Olympic Games and visiting her relatives.

To recruit participants similar to the primary persona the team created, we developed a general description of the user profile characteristics, as follows:

Must be able to speak and read ChineseTravel purpose: either for business or for pleasureAge: 20-55Gender: female and maleExperience booking a hotel room online Use internet at least 5 hours per weekNo experience with the Chinese Holiday Inn website

A screening questionnaire was created for selecting participants from the target user group and it is included in Appendix D.

Usability Criteria

When we conducted the heuristic evaluation of the website, we applied Jakob Nielsen’s 10 heuristics adapted for the Web (as cited in Dr. Carol Barnum’s text, Usability Testing and Research).

Detailed information about our heuristic evaluation can be found in Appendix A: Heuristic Evaluation.

9

From the heuristic evaluation, we determined the issues we felt would be problems for users. These became the basis for our scenarios.

Test ObjectivesWe combined our goals and our sponsor’s concerns to create our test objectives. We created post-task and post-test questionnaires to assess the ease or difficulty of the following hotel booking tasks on www.HolidayInn.com.cn:

The general feeling/layout of the site: Is it easy to find the information users want to know the first time they visit?

The procedure of booking a hotel online: Basic searching: Is it easy to use? Advanced searching: Can users accomplish their goals on the

advanced search screen? Entering personal information: Does the website require

reasonable and suitable information for Chinese users? (e.g. U.S. uses State, China uses Province; Americans have SSN for social security number, Chinese have IDNO for Identity Card Number ). Do the users understand all information requirements?

Language: Can users understand the language used on the Holiday Inn website? Are there any translation mistakes or misunderstandings? (Such as: “king size bed” translated in Chinese to a very big size bed.

Logging in to Priority Club: Is it easy for users to find the log-in page? Does the website offer the benefit information of being a member of Priority Club? Is the information clear?

Information in confirmation email: How confident are users in receiving a confirmation email? Does the confirmation email contain the information users need?

Satisfaction with the site: Which aspects do users like and which aspects do they dislike?

Perceived reliability of the site: Do users trust the website? Do they fill out the personal information readily?

Navigation: Can users find the most efficient navigation when booking a hotel or browsing the website?

The test was designed to allow us to extract mostly qualitative data.Participants were given a pre-test questionnaire to get additional information about their experiences booking a hotel reservation online, then given scenarios that directed them to perform specific tasks, and lastly a post-task

10

questionnaire designed to extract detailed and specific feedback about www.HolidayInn.com.cn features.

The test was conducted in a full-scale usability lab. Users were given 40 minutes to complete five scenarios. We also designed an extra scenario in case there was some time left.

Data CollectionThe evaluation measures are a combination of quantitative measurements and subjective observations. The quantitative and qualitative data measures are listed below.

Quantitative DataThe following data were recorded when we carried out the Holiday Inn web site usability test:

Time to find an available hotel Number of participants completing each task Number of problems encountered during each task Number of participants that failed a task The participant’s rating of level of difficulty in booking a hotel room on

the Holiday Inn site The participant’s rating of level of confidence in booking a hotel onthe

Holiday Inn site The participant’s level of reaction to the look and appeal of the website A comparison of satisfaction ratings for using the competitor site vs.the

Holiday Inn website

Qualitative Data Facial expressions Verbal comments

o Positive o Negativeo Questions askedo Nature of calls for help

Screening Profiles

Including the pilot test, we collected the following data from our six participants.

Pilot User

1st User

2nd User

3rd User

4th User

5thUser

Age Range 25-35 25-35 46-55 20-25 20-25 36-45

Gender F F M M F FOccupatio Computer Account Real Electroni Architect Mana

11

n Science Student ant Estate

c Engineeri

ng student

ure student

gement

1. In the past year, how often have you booked a

hotel online?

Never

1-3 times X X X

More than 3 times X X X

2. How often do you use

the internet?

Less than two hours

a day2-5 hours

a day X XMore than 5 hours a

dayX X X X

Less than 5 hours a week

3. What do you mainly

use the internet

for? Circle all that apply

Entertainment X X

Information / Research

X X X X X

E-Mail X X X X XSocial Networking

X X X X

OtherBusiness

(Stocks)X X X X

4. How many nights

have you spent at a hotel over the last six

months?

7 3 2 more than 5

More than 20 nights

3

5. How many of

those Business

_____ 0 0 0 0 1

12

nights over the last six

months were for:

Pleasure_____ 7 3 2 5

More than 20 nights

2

6. What websites have you used the most to

book hotel reservation

s?

hotel.com

Hotels.com;

expedia.com;

Marriott and

Hilton

Chaoicehotel.

com

www.bestridehote

l.comwww.hotel.com

Travelocity

or hotel's own websi

te

7. Do you mostly

book hotel reservation

s for yourself, or

does someone else do it for you?

I booked most hotel nights

online by myself. I

travel with my friends,

sometimes I did it,

and sometime

s they did.

Yes, I book

hotel for myself

NA/A by myself

I do it myself, and I

oftentimes

booking online for my aunt.

For myself and for

others

8. Finish the

following sentence: When it

comes to providing my credit

card number online:

a. I do not mind X X X X

b. I will do it, if

absolutely necessary

X

c. I don’t give out

credit card info online

X

Test Environment and EquipmentThe test was conducted at the Usability Center at Southern Polytechnic Usability Center. The usability center includes three rooms:

13

Evaluation RoomThe evaluation room is where participants complete tasks. The room has the following equipment:

SPSU-standard PC configuration (Windows XP, campus network connection)Keyboard and mouse19” MonitorCamerasMicrophoneTelephone/intercom between control room and evaluation room

Control RoomThe control room is where the team observed and recorded the test. There is a one way mirror between the evaluation room and the control room. The control room has the following equipment:

A standard PC with Morae logging softwareMonitors which display what the cameras are capturing in the evaluation roomHeadphones and microphones for communication among the team membersVideo recording and editing equipmentA picture-in-picture digital mixing boardA phone/intercom for the participant to call the help deskLighting controls

Executive Viewing RoomThe Executive viewing room provided opportunities for additional observers watching the test without interrupting the team. The executive viewing room has the following equipment:

SPSU-standard PC configurationMonitorsLCD Projector

Test Scenarios/TasksOverview/Briefing (10 minutes)

The facilitator welcomed the participant and asked him/her sign the video consent form. Next, the facilitator explained the facilities, introduced the team, and asked the participant to think out loud. Also, the facilitator worked with the participant to complete the pre-test questionnaire. All test materials are included in Appendix D.

14

Scenarios are included here, as well as in Appendix D, to provide the context for analyzing the findings, which follow.

Scenario 1: Look and Feel (5 min)

You and your family (if you are not married, it can be your friend) are thinking of going back to China for the summer Olympics in Beijing. You heard that the Holiday Inn has several locations in Beijing. You will visit the Chinese Holiday Inn website (www.holidayinn.com.cn ).Take a moment to look at the homepage without clicking on anything. While looking at the homepage, please express what you think of the website and what you might do to book a hotel room.

Scenario 2: Book A Hotel Room (15 min)

You decide to book a hotel room for yourself and another person coming with you on the trip to China. Your booking dates are between July 21st and July 25th. Here is the information you may need: • First Name: Jing• Last Name: Li• Location: anywhere in Beijing• Check in date: July21st, 2008 • Check out date: July 25th, 2008 • Credit Card Information: Visa, card number: 4111 1111 1111 1111,

expiration date: 08/2009, card verification number: 635• E-mail Address: CBRholiday@hotmail.com Please write down the confirmation number when you completed the reservation. 

Scenario 3: Cancel a Hotel Reservation (5 min)

You just received a phone call from your relative who lives in Beijing. She wants you and the person traveling with you to stay at her house during the visit.

You accept the invitation and decide to cancel your Holiday Inn reservation. Go ahead and cancel your reservation.

Scenario 4: Find a Holiday Inn in Changchun (5 min)

You have some relatives in Changchun, Jilin province. You want to go visit them during your vacation.

15

See if there is a Holiday Inn available in Changchun from July 26th to July 30th.

Scenario 5: Book a Hotel Room at another Site (10 min)

Go to the website www.elong.com.

See if a hotel room is available for you and the person going with you between July 21st and July 25th in Beijing.

Note: You are not going to actually book a hotel room on this website. Stop when you get to the last step in completing the booking process.

Scenario 6: Look and Feel of Holiday Inn English Site (Optional)

(5 min)

Go to the Holiday Inn English-language website (www.holidayinn.com).

Take a moment to look at the homepage without clicking on anything.

While looking at the homepage, please express what you think of the website compared to the Holiday Inn Chinese site and eLong site.

Closing (5-10 minutes)

Participant completes the post-test questionnaire.

Findings/Recommendations

Positive FindingsBelow are the positive results that the users wanted to make sure were known.

Positive Impressions: “I like the way they set up the website, it is very clear.” (U4) “This website is easy to understand, it should not take long for me to

work on it.” (U2) “All the options I need are right here in the center of the page” (U2)

initial reaction to home page

16

Homepage hotel search: Four out of six participants felt that the homepage’s hotel search box was very useful. They felt that it was convenient that it was located on the homepage and were glad that they did not have to search around the site to find this section. Most users said it was straightforward to use, easy to find, and easy to understand. To quote one of our users (U4), “I like the design, it is very clear.”

17

Homepage is easy to navigate and understand. Information is centered which makes the page easy to use

Calendar feature: Three out of six participants mentioned that they were glad a calendar feature was integrated into the hotel search forms. Most users found it convenient because rather than having to type in a date and figuring out what format to put it in, they were able to look at the calendar and just pick the days they wished to stay and the website would take care of all the formatting.

Reputation lives up to its name: Two of six participants felt strongly that because the site was Holiday Inn, they knew they could trust the website and also knew what kind of standards Holiday Inn has.

Brand variety: Two of six participants were happy to see that other well known hotels were displayed in addition to just Holiday Inn hotels. One user chose Crown Plaza while another chose Holiday Inn Express. It allowed the users to feel that there was a healthy variety of hotel chains to pick and choose from.

Severity Ratings

18

Drop-down menu is well-received by the users

Other well-known hotels that are part of the IHG brand are displayed in addition to Holiday Inn

After our team compiled all of our findings, we assigned a severity rating to problems that our users encountered. We divided and defined our severity ratings into the following categories:

High: Findings where the majority of our users experienced the same problem. Also, high severity ratings were labeled when users encountered an obstacle or difficulty preventing them from finishing their task.

Medium: Findings when two to three of our users encountered the same problem. The medium rating was also applied when elements of the site frustrated the users but did not keep them from completing their task.

Low: Findings that either could be problematic or were confusing at first. The findings were not considered difficult enough to be a problem, but frustrating enough to deserve attention.

Collated findings, rated by severity

Based on the feedback we received from our participants as well as our analysis of the quantitative data from testing, we identified 14 significant usability issues with the Holiday Inn China website. The following table lists these issues and indicates their impact on the usability of the site, along with the number of participants who experienced each problem.

Usability Problem # Affected SeverityNavigation: The ‘start over’ and ‘finish booking’ options on the confirm booking page confused the user

1/6 low

Layout: Users found the Priority Club Login options confusing. 4/6 high

Layout: Users think advertisements are overwhelming 3/6 medium

Layout: Users want the total price for their stay to be listed, instead of a per-night rate only

3/6 high

Layout: Users have trouble finding the cancellation link to cancel their reservation

3/6 high

19

Layout: Error messages were not seen by the user or they were misinterpreted

5/6 high

Aesthetics: Search results of hotels are displayed in a two-color format, confusing users’ perception of how many results are actually being displayed

2/6 low

Feature Requests: Users complain about not seeing promotional rate information

2/6 low

Feature Requests: Users want to see a drop-down list of cities instead of having to type city names in

3/6 high

Feature Requests: Users want to see hotel rankings by different categories (customer experience and price)

4/6 medium

Feature Requests: Users want more information on hotel amenities

4/6 high

Mental Model: Users are confused as to whether or not they are required to input full Chinese name, or just family name

5/6 high

Feedback: Loading screen has no progress bar. This confuses users as to whether or not their information is being processed

6/6 high

User ImpressionsQualitative feedback is indicated by a sample of the general comments we received from users about their experiences and preferences. The complete set of comments can be found in the test logs, Appendix F.

20

General comments: “Are they sister hotels?”—User 2, reaction to the different hotel logos

at the bottom of the page “Why is the English name needed?—Pilot user (P), reaction to required

field on reservations page “What is Priority Club number?”—User 1 “Where is the cancellation button?”—User 1

Negative Impressions: “I would go to another hotel reservation site at the point.”—User P,

reaction to confusing error messages “It is not as easy as booking.”—User 4, reaction to cancelling

reservation “So many advertisements.”—User P, reaction to home page “Loading page is too slow, too boring.” –User 4, reaction to blank

loading page “I am waiting again, most websites have a progress bar.”—User 4,

reaction to blank loading page “So many ads on this website makes me uncomfortable.”—User P “I don’t want to have to type in the city name, there should be a

dropdown menu.”—User 1 “I am not satisfied at all.”—User 1

Suggestions: “I think they should use Ming (first name) instead of Mingzi (first and

last name). It’s so confusing.”—User 5, reaction to Chinese name input field

“I want to know the total price instead of the per night price.”—User 5 “They should show pictures of the actual hotel—both inside and out.”—

User 2 “I want to know what kind of services they provide, such as breakfast,

prices, user ratings…”—User 2

Recommendations

Based on the findings of our usability test, we make the following recommendations to improve the Holiday Inn China website. Recommendations are based on the severity ratings for the problems users encountered.

“Sort by” option (feature request)

Four out of six participants showed interest in the competitor website’s option to arrange search results by price range and/or by user comments. On the Holiday Inn China website, users do not have a “sort by” option. User’s verbalized their desire to have an option that allowed them to sort their search results by different categories.

21

Recommendation: Create a “sort by” option that includes at least price range, and user ratings

Idle Loading screen (aesthetics)

Five out of six participants complained that the page displayed while results are being compiled has no progress bar. They are confused as to whether or not their request is being processed.

Recommendation: Add a progress bar to the loading screen or display a message that informs the user that his/her request is being processed.

Error Message (layout)

Error messages were not seen or they were misinterpreted by the user. When trying to find a hotel in Changchun, Jilin Province the error message reads “No hotel matches your requirement,” when in fact there is no Holiday Inn in Changchun.

On the cancellation page the user is required to enter his/her last name and first name separately. However, most users entered their first and last names together. The error message on the cancellation page says “The name you entered does not match the name you registered with.” This does not help the user figure out what he/she did wrong.

Recommendation: First, highlight and display the error message next to the incorrect field. Secondly, provide clear instructions as to how to fix error.

22

The competitor site, elong.com, offers a sort-by option that allows users to customize their search options

Hotel Amenities (feature request)

On the search results page, users complained that there were no pictures of the interior of the hotels. Five out of six participants were not aware that they could click on the hotel pictures to find additional pictures of the hotel. Nor did they see the link to additional pictures. Users also complained about not having enough information about the hotel’s services offered.

Recommendation: Make the link that leads to hotel pictures more visible. List more amenities of the hotel, such as pool and food service provided.

Input Name Field (mental model)

On the registration page, users are asked to input their English name and their Chinese name. Most users just put in their English name because it’s a required field. On the cancellation page most users input their full name into the field that only requires a last name. The Chinese audience is not used to entering their first and last names separately. The phrase used for “family name” on the website looks similar to the phrase “first and last name.” This also adds to the user’s confusion.

Recommendation: Allow the user to enter his/her first and last name into one field, instead of requiring the user to enter his/her first name separately from his/her last name.

23

The error message does not specify exactly what the problem is

Advertisements (layout)

Three out of six participants found the advertisements throughout the website to be distracting and irrelevant to the purpose of the website. Many of them said the advertisements were overwhelming. When they saw the English Holiday Inn website, they said the advertisements were appropriate.

Recommendation: Reduce the number of advertisements on the website and make the pictures relevant.

24

The Chinese name input-field is not clearly translated. Users are confused as to what they should do at the point.

Drop-down city menu (feature request)

Several users expressed their desire to have a drop-down menu that provides the names of cities in the region they are searching in. They communicated that having to type the city name made them feel that they were wasting time. They were also concerned that they may have misspelled city names.

Recommendation: Create a drop-down menu that lists city names so that users have less to type in the input fields.

Cancellation Link (layout)

Users had trouble finding the cancellation link to cancel their reservations. Most of the users communicated that they would, under normal circumstances, leave the website and call the hotel directly.

Recommendation: Put a cancellation button on the main page.

Price Listing (layout)

Users want the total price for their stay to be listed, instead of a per-night rate only. Users had trouble finding a total price for their reservation.

Recommendation: List the total price that guests will pay, and not just a per-night price.

25

Advertisements are reported to be overwhelming

Priority Club Login (layout)

Four out of six users were confused as to whether or not they had to log n as a Priority Club member to even use the website. They were not aware of what Priority Club is. On the cancellation page the Priority Club login option appears first and confuses users.

Recommendation: Reduce the size of the Priority Club login.

Other Findings

In addition to the findings that were based on our severity rating system, we had two additional categories:

26

The per-night listing stands out, while users must click another link to view the total price for their stay

Miscellaneous Findings: Findings that were labeled as miscellaneous were for problems that users encountered that we did not expect.

Localization Findings: All of our users have been in the United States for at least two years and upwards of twenty years. While being here, they have picked up on certain aspects of how American sites and business operates. Also, they have English speaking skills and can translate Chinese to English when using the site. In contrast, a native Chinese person may not have any familiarity with English. Localization findings have been grouped for potential problems that a native Chinese person or audience may encounter.

Miscellaneous Findings

Confirmation checks before finalizing hotel reservation: Our second participant missed the area where a confirmation check was required to finalize the hotel reservation. The area being mentioned is shown below:

Confusion from “nights” section of the hotel search box: Our second user initially entered in his stay range with the built-in calendar feature. After entering in the dates, he then continued to the “nights” dropdown box and picked the number he felt corresponded with the duration he would be staying. The choice the user made did not match with the dates he selected and it changed his date range without him noticing it. After he did notice this, he thought that the addition of the “nights” stayed drop-box seemed redundant and confusing. This user was the only one to actually try selecting the “nights” they would be staying. The rest of our users only used the built-in calendar feature. The following screenshot shows the area where the problem occurred.

Interim option: Our first participant made a mistake during the confirmation page and chose “start over” rather than “confirm reservation.” While this was accidental, we think it might be good to have an additional option in this area for modifying one’s hotel reservation details. This could potentially be useful

27

for allowing users to go back and make a change to something rather than having to start over from the beginning of the hotel search option.

A user can use the back button on the browser to make changes, but depending on the user’s local settings, the information might not be saved and the user would have to re-fill out all the information again. The recommended change is shown below.

Inclusion of a search feature: Our first user felt that it would be nice if a search feature was added. This could be a benefit to users seeking Holiday Inn hotel locations in different cities.

Localization Findings

Returning Home: Many Americans are used to clicking on a logo or banner towards the top of a website to return to the homepage. There was confusion among several users about how to return to the homepage, but most tried clicking the logo to return to the homepage. However, one of our users was uncertain as to how to return to the homepage; rather than clicking the Holiday Inn logo, the user clicked the “back” button on the browser until reaching the homepage again.

It might be helpful to include a “Home” or “Return to Homepage” button in the navigation area for those who are not accustomed to clicking logos to return to the homepage.

Submit button did not appear clickable: One of our participants was uncertain as to how to initiate the hotel search from the homepage. The user eventually found the button (see below) to initiate the search, but it took significantly longer than the rest of our participants.

We do not have a clear recommendation for this issue but think it might either be worth looking into or modifying the area to make it stand out more.

28

Trust symbol confusion: One of our users was confused about the “Trust” image found on the homepage (shown below).. Most native Chinese people prefer to use cash when making a transaction, so graphics that Americans might recognize as being “security checks” will not be as easily recognized by native Chinese residents.

We do not have a recommendation for this but feel that it might be worth looking into further.

CNY currency confusion: One of our participants was uncertain about what “CNY” stood for when placed before the numerical value. CNY designation is known internationally for Chinese yen; however, native residents of China would be able to identify better with the Chinese symbol for yen.

Our recommendation would be to include the character and/or symbol for yen in addition to “CNY” so that native users would not be confused about what currency type is required.

Character Symbol

Post Task Questionnaire Analysis

Four out of our six participants were asked follow-up questions based on the experience they had with the scenarios we presented to them. Below are

29

questions we presented to them, in addition to tables that show the answer sekected by them. The first row of the tables below are the answer selections presented to the users and the second row identifies which user chose that answer choice. (e.g., When “#4” is shown under an answer choice, it means our fourth user selected this answer choice.)

1. Reaction to the look and feel of the website

One of the questions we asked our users was to rate their reaction to the look and feel of the website. This was asked after first introducing the users to the Chinese site’s homepage and letting them look around the page. The responses were gauged based on how visually appealing they found the website as well as how they thought the “feel” of the site was.

Very Appealing

Somewhat Appealing

Neutral (No opinion)

Somewhat unappealing

Very unappealin

g

#4, #5 #2, #3

2. Interest in continuing the booking process

After we introduced the site to the users, we wanted to find out their interest in continuing to look for a hotel based on their first impression of the site. Since first impressions usually will determine if a user wishes to continue using a website, we felt it would be appropriate to find out if users would continue trying to book a hotel based on what they have seen so far.

3. Features users enjoyed

After our users completed the task of booking a hotel room, we asked them to list any features that they enjoyed during the booking process.

Our second user liked how the website provided information about if the hotels served breakfast or not.

30

Very Interested

Somewhat Interested

Neutral (No opinion)

Not Very Interested

No Interest at all

#2, #3, #4, #5

Our third user said it was easy to tell which hotels had rooms available and which hotels did not.

Our fourth user liked the way the website was set up. It was very clear to her.

Our fifth user liked the map that was shown after performing a hotel search. (This feature was added in later during our testing time and only the fifth user was able to see this feature.)

4. Features users disliked

In addition to listing the features users liked, we asked them to list any features they did not like during the booking process.

Our second user felt that the site’s performance was a little slow. The third user did not like that a picture of the room they booked

was not available for them to see. Our fourth and fifth users did not see the area saying what the total

price would be for their stay on the confirmation page. User 4 added up the total because it was not clearly visible.

Our fifth user felt misinformed when viewing one of the hotel choices displayed from her search results. The hotel initially said that rooms were available that met her criteria, but after continuing to the booking process it said that no rooms were available.

5. How easy/difficult was cancelling

After users went through our scenario to try cancelling their hotel reservation, we asked them how easy or difficult the process was for them.

Very Easy Somewhat Easy

Not Easy or Difficult

Somewhat Difficult

Very Difficult

#3, #5 #2 #4

6. Level of confidence in being able to cancel

After asking the users about how easy or difficult they found cancelling a reservation, we continued with a follow-up question to rate their confidence level of being able to cancel a reservation on their own without assistance. The rating was based on a scale of 1 to 10, 10 being the most confident. The top row in the table found below is the numerical “confidence” value. The row underneath is how many users identified with number choice shown above.

31

1 2 3 4 5 6 7 8 9 10

#5 #4 #3 #2

7. Explain confidence rating

After asking our users to choose a numerical value representing their confidence level, we asked them to explain why they chose that value.

a. The fourth user chose ‘8’ because although she failed to perform the action online, she was confident that she would just be able to call in to cancel her reservation.

b. The fifth user said that she would feel very confident about cancelling a hotel as long an e-mail is sent confirming her cancellation.

8. How easy/difficult is it to find pertinent information

Next we asked users to rate how easy or difficult it was for them to find information that was pertinent to them on the website. (This question was only addressed to two users.)

Very Easy Somewhat Easy

Not Easy or Difficult

Somewhat Difficult

Very Difficult

#4 #5

9. What information would you like to see on the website?

For users that felt it was difficult to find information on the site, we asked a follow-up question asking what information they would have liked to see on the website.

a. Our second user would have liked the search results page to show cities that have hotels available when a hotel cannot be found in a city (e.g., In Changchun there are no hotels available). Our user would have liked the results to say something along the lines that no hotels are currently available in Changchun;

32

however (nearest city) has hotels available. Would you like to view the hotel listings for (nearest city)?

b. Our fifth user encountered an error message that others did not. Although other users encountered error messages. the fifth user’s error message appeared in the form of a technical error. Instead of an error message saying what the user did wrong, the error message displayed a coding error with no explanatory information. (The error message was this: U_2008_AN)

10. Reaction to the look and feel of the elong website

We also asked our users to view the www.elong.com website. Just like the first question where we wanted the users to rate their reaction to the “look and feel” of the Holiday Inn China site, we asked them to rate their reaction to the “look and feel” of elong’s website.

Very Appealing

Somewhat Appealing

Neutral (No opinion)

Somewhat unappealing

Very unappealin

g

#5 #2, #4 #3

11. Reaction to the look and feel of the English Holiday Inn website

When time permitted, we asked users to view Holiday Inn’s English-language website. We asked them to rate their initial impression to the “look and feel” of the English site. This question was addressed by only two users.

Very Appealing

Somewhat Appealing

Neutral (No opinion)

Somewhat unappealing

Very unappealin

g

#4 #3

12. Comments concerning all three websites

For those users who visited the Chinese Holiday Inn site, the English Holiday Inn site, and elong’s website, we asked them to share any comments they had concerning the three sites.

Our second user felt that Holiday Inn’s English website would be the easiest site for him to book a hotel after we had exposed him to the Chinese site and elong’s website.

Our second user liked that elong gave users a city selection box rather then having to type in a city name. Another reason the user liked this is because they did not have to worry about entering in a city name that may be misspelled, thus not showing the user any results because

33

of their spelling error. The listing of cities prevents a spelling error from occurring.

Our third user said that most native Chinese residents are not familiar with booking a hotel room through an online service.

Our third user did not like how many options were presented to them on the elong site.

Our fourth user did not like that membership logins were shown on each page for the Holiday Inn sites.

Our fourth user thought that all the advertisements across the three websites she saw were distractive.

34

Conclusion

The usability course that our team has taken while conducting this project has been an extremely informative experience in portraying how usability testing is performed. While conducting this project, we have not only gone by guidelines and principles taught during the course, but we have also received hands-on professional input from our instructor Dr. Carol Barnum.

The team for this project consisted of six undergraduate students pursuing degrees in the Technical Communication major at Southern Polytechnic State University. Four of the team members were Chinese exchange students and the other two members of the team were local American students.

The team conducted six testing sessions in SPSU’s state-of-the-art Usability Center. Following the completion of testing, the team collectively gathered data and organized it based on methods taught during the usability course, as well as methods described in Dr. Carol Barnum’s book, Usability Testing and Research.

Findings

After analyzing logs and recordings made during our research sessions, we divided our findings into seven different categories:

Layout Navigation Aesthetics Feature Request Brand Identity Mental Model Feedback

RecommendationThe six users that we received feedback from all have lived in the United States for at least two years. In the findings section of our report, we identified issues that could be problems for Chinese language speakers using the website.

We recommend that further testing be conducted to:

Confirm that recommended changes for English-speaking, U.S. based Chinese users improve their experience

35

Test the changes with a study of Chinese-speaking users to confirm that these changes are effective for native Chinese users of the website and to uncover other areas that need to be address for these users.

We are confident that the research performed in our usability study will be helpful for making improvements to the Holiday Inn Chinese website. In addition, we strongly encourage additional testing for any future enhancements to the site.

36