FOREVER 21 Usability Evaluation Report HCI 460 July 27, 2015 Group 6 Gelayol Moradzadeh Vicky Moreira Mauli Shukla Frank Sweis

 

Table of Contents

I. Executive Summary……………………………………………………………… 2-3

II. Introduction………………………………………………………………………… 3

III. Methodology………………………………………………………………………… 4

A. Participants………………………………………………………………… 4 B. Procedure…………………………………………………………………… 4

IV. Findings and Recommendations..…………………………………………… 4-27

A. Positive Findings………………………………………………………… 5-6 B. Negative Findings……………………………………………………… 7-27

V. Team Contributions……………………………………………………………… 28

VI. Appendix………………………………………………………………………… 29-31

1 

I. Executive Summary This report describes the usability issues that users experience while using the Forever 21 Android application to perform common shopping tasks such as browsing and purchasing. Forever 21 is a popular brand with stores located around the world due to its affordably priced and trendy clothing. Their mobile application is designed to help customers purchase their prefered apparel and accessories online, as well as locate stores around them instantly on their mobile devices. To explore the usability issues our team prepared a test plan that defined our objectives and also framed our research questions in a way that would help us identify the strengths and weaknesses of the application. We also defined our target participant, evaluation measures, the tasks for users to perform, and possible questions that could be asked before and after the usability test session. In order to conduct the test we first created a moderator’s guide, which included a consent form, introduction, and a series of tasks to keep all team members on track during the test sessions. We conducted a total of 6 tests with both male and female participants that were between the ages of 25 and 45. These participants assisted us in forming both positive and negative findings. We found three major positive findings in the application and they are regarding the following:

● As soon as the user gets to the shopping page, adding to cart is an easy process ● Icons used in the application are very intuitive ● Important elements are not hidden behind multiple screens

Some issues were also found during the usability evaluations. The major issues are:

● Application has a slow speed. ● Application does not have proper scaling in some pages of the app, which makes it difficult for the

participant to obtain information with ease. ● Users do not notice the “shop” button easily which causes to miss some of the products available

in the application. ● There is no quick way on home page to access shopping and searching features, which slows down

the user in reaching the wanted page in the application. ● While scanning a barcode, there is no feature to help focus barcodes properly. ● No feedback or message is provided to the user to inform the users when they have successfully

shared and posted items on their social media successfully. ● Two map buttons are available on the application for locating other Forever 21 stores. ● Location results are in random order causing users to scroll in order to find the closest store.

By conducting the usability evaluation sessions, we found out that all these issues triggers frustration and dissatisfaction with using the application in the user. The most common problems that were observed were unresponsiveness and slowness in the application the difficulty of finding the closest Forever 21 store. In addition to methodology and procedure,

2 

this report describes the most common problems with the application including the recommendation for resolving the issues.

II. Introduction

The focus of our study was the Forever 21 Android application. With more than 1,000,000 downloads, the brand has shown to be a popular and trendy store worldwide. This store primarily targets a younger demographic with affordable clothing prices, which now includes male clothing and accessories. Due to the store’s popularity, Forever 21 is a recognizable brand to the everyday consumer. In recent years, the company has adopted new marketing tactics to increase their client base. Such tactics include introducing a men’s line to their collection and releasing a mobile application in 2012 for quick shopping experiences and become part of today’s mobile-driven world. We evaluated Forever 21’s application based on real world scenarios that it’s users face. The scope of our research was the user’s experience browsing, searching, sharing an item of clothing, and finding the closest Forever 21 store near by. In particular, we wanted to answer the following questions:

1. Can users navigate the application easily to search and buy (add to bag) clothing? 2. How many taps and/or screens does it take for users to perform the outlined tasks

(such as buying clothing)? 3. What are the major usability flaws that:

a. Impede users from purchasing an item? b. Impede users from finding a Forever 21 store? c. Impede users from sharing an item on Pinterest?

4. Are users satisfied with their interactions with the application when searching and buying clothing, exploring styles, finding stores, and sharing on social media? (data will be gathered through the wrap-up questions)

We created specific tasks for our participants to address these questions. Our main goal for these tasks was also to identify the application’s strengths and assess how the application could be further improved to better meet the needs and expectations of its users. As a group, we identified some areas areas of improvement for the application, prioritized these areas based on their impact on each task, and identified actionable recommendations which were all incorporated into this report.

3 

III. Methodology

Participants Each team member conducted at least one sit-by evaluation with participants located around Chicago and identified their own method for recruiting and recording their interview; there were six interviews in total. All participants were over 18 years old and familiar with online shopping and mobile applications. We selected participants that either shopped for themselves online or were shopping for their kids. Our team obtained consent from participants by using a consent form.

Procedure

Each member of our group individually conducted sit-by evaluations that consisted of predefined tasks that we had identified in a Moderator’s Guide and below:

● Browse for an item on the app ● Prepare item for purchase ● Scan barcode using scan feature ● Share an item on social media ● Find information of store closest to user

Within each task, we decided to note quantitative measures such as the number of taps or screens taken to accomplish a given task. We also noted the number of errors or challenges users encountered while completing the tasks. We predefined errors to be when users navigated to the wrong screen, mistapped an element on the application, asked for help, or were unable to accomplish a task. We utilized questions on a Likert scale to understand the user’s level of satisfaction and the level of difficulty they encountered. Our group also evaluated the tasks based on qualitative measures such as user comments, feedback, and through wrap up questions that specifically asked how they felt at certain points through the tasks and overall what they liked and disliked.

IV. Findings and Recommendations We found a total of 3 positive findings and 11 negative findings. The negative findings are grouped by tasks and then each finding is ordered based on their appropriate severity rating (from highest to lowest) within each task. Our severity levels are based on how much each finding impacts the user’s interaction with the website, if it impedes the user from reaching their goal, and the feasibility of improving the finding. We defined each severity level as:

● High: This usability problem will slow down some users when completing a common task leading them to either find another way of completing it or possibly making

4 

them unwilling or unable to complete the task. High severity level denotes that the specified usability problem should be first priority to fix.

● Medium: This usability problem will make some users feel frustrated or irritated, but will not affect task completion. Medium severity denotes the specified usability problem should be second priority to fix.

● Low: This usability problem does not affect the user’s performance as much. This is a quality/aesthetic issue. Even though this is a minor issue, too many low level problems can create a larger usability issue. Low severity level denotes that the specified usability problem should be last priority to fix.

All specific tasks include an explanation of each task, a summary with quotes of the participants’ experience, the results of the likert scale, and an explanation of the results/data gathered. Each finding include images as visual aids to help with the implementations of our recommendations.

Positive Findings

1. Shopping experience (Adding to bag)

Forever 21 uses its mobile website for its application’s shopping experience. Their website uses responsive web design, which means that it is optimized to visit over a range of devices from desktop to smartphone. The screenshots below showcase how the website responds to screen size (smartphone vs. desktop) and input method (touch vs. cursor). The responsive design allows for the “Shopping cart” button to populate in the same area a customer would expect to see it on the website. The icon sizing and placement are intuitive to the user thereby making it a easy and quick task to complete on the mobile application.

5 

2. Icons are intuitive

Forever 21 uses a mixture of real world and common interface metaphors for it’s navigating toolkit. This allows users to recognize the functionality of each icon, and quickly become familiar with the application.

3. Elements are not hidden beneath multiple screens. Forever 21 is designed so that all features of the application are accessible by one or two screens. Along with the real world metaphors, this allows the user to quickly scan the application and get to what they want. However, the application falters when the user begins to use the app. This is due to poor performance and lack of optimization.

6 

Negative Findings

1. General Findings consistent throughout all tasks

❖ Tasks Explanation: Some issues were recurring and consistent throughout multiple tasks. Therefore, below are the issues that impacted all our tasks with the negative feedback and comments from our participants. As such, most of these findings should be treated as highly severe. We found that the type of device (e.g. older device) increases these problems tremendously.

❖ Participant’s Experience

The overall user experience for this application caused moments of confusion and frustration for our participants. Throughout the entire process of going through all the tasks, our participants encountered issues where more than one participant demonstrated frustration with the speed of the application and unresponsiveness of elements on the app. One participant exclaimed, “Ahhh, I can’t go back… it’s so slow!” Another recurring issue was the sizing of elements throughout the application. A participant succinctly put it, “ The application didn’t look slick. Font sizes were weird and it looked like a rough draft.”

7 

❖ Task Findings

Finding 1 (Objective: Major usability flaws, number of taps required)

There are overall issues with the application responsiveness and speed.

Justification The slow speed of the application causes high frustration and dissatisfaction to the users. The application suppose to provide the users a quicker and simpler experience to the user than actually going to the store or website so users expect the application to run faster and make purchases quickly. They do not expect or want to have to wait for a single command to take more than 3 seconds.

Recommendation

Make the speed of the application faster by optimizing it for all hardware.

Severity Level High

Visuals

Currently: Takes about 5 to 10 seconds for pages and features to load.

8 

(https://dribbble.com/shots/1689922-Side-Menu-Animation?1408428320)

Recommendation: This is an example of how fast we recommend the responsiveness of the Forever 21 Android mobile application should be at all times.

Finding 2 (Objective: Major usability flaws, number of taps required)

User inputs are unresponsive.

Justification Once a button is clicked it does not immediately respond and complete the expected action. For example, once the scan button is clicked in the main menu, the application freezes. Sometimes the button successfully links to the scan tool while other times it doesn’t and once the application freezes, the user may click other buttons to get out of the frozen area of the app. Upon hitting other buttons, the application still not directed to the expected page.

Recommendation

It is critical to identify the cause of the button irresponsiveness. If the application is slow because the buttons are not created/linked properly, it would be important to either properly link those buttons. If the placement of hot spot on the tabs or the size of the tabs is an issue that should also be further studied and rectified.

Severity Level High

9 

Visuals

Currently: This is the blank page that sometimes appears when the app

is processing a request to go to another page.

Recommendation: This expected pages should appear instantly after choosing (tapping)

the appropriate buttons and tabs.

10 

Finding 3 (Objective: Navigating to find Forever 21 store)

Elements within the application do not scale properly or make best use of high resolution displays.

Justification The lack of proper scaling makes elements such as the bottom and upper toolkit harder to navigate. This creates small text which is hard to read and creates lots blank space that can be used for improved usability, most notably when searching for a store. This is also aesthetically unpleasant. New smartphones can offset some of the usability performance problems of the application, but with their higher resolution displays, this create a new usability problem.

Recommendation

The application must detect the resolution of the smartphone and elements (toolkits, font size and font type) accordingly.

Severity Level Medium

Visuals

Currently: 1440p Display 720p Display 640p Display

11 

Recommendation : 1440p Display 720p Display 640p Display

2. Task: Preparing an item for purchasing

❖ Task Explanation: For this task, we asked participants to prepare the item that they’d already found for purchasing. In this task users we expected to add the item to their cart successfully.

❖ Participant experience: Participants completed this task without difficulties. However,

most of them didn’t notice the shop button at the bottom of the page. Most common issue that was observed was that male users had hard time finding clothing for men. And it’s because they didn’t notice the “shop” button so that they can search by gender. "How do I shop for men in this app?!" "For some reason you have to scroll all the way down to get to shop to men which is stupid. There should be a really easy way" " just clicking around because the website moves slow and there is no options for men here" "this is annoying" "Why there is a shop page? oh... this makes more sense than home page! I should have came here at the first step." “There should be a hamburger dropdown menu on the homepage where you can select whether you want to shop for women or men’s clothing. The hamburger menu that opens up once you go to the shop section would be better placed on the homepage.”

❖ Table: Level of difficulty for Purchasing (1 being Very Easy and 5 being Very Difficult)

Participant

Rating (1-5)

12 

P1 3

P2 1

P3 1

P4 1

P5 1

P6 1

❖ Data Description:

● 3 out of 6 participants completed the task with 5 taps. One participant completed the task with 3 taps and two other participants completed the task with 7 and 9 taps. On average, participants completed their tasks with 3 screens.

● All participants but one rated the purchasing task as being very easy. one participant rated moderate for this task. The participant thought the “Continue shopping” tab took a long time to work which made the user frustrated.

❖ Findings:

Finding 4 (Objective: Navigating to shop and buy, number of screens required)

The placement of the menu is not clear or intuitive to the user and the “Shop” button in the menu is not very accessible to the user through the application.

Justification Participants expected the application to automatically open up to the shop section of the application since that is the primary function of the mobile application. The unintuitive placement of the menu causes users to click on the wrong buttons until they finally locate the “shopping” button. The amount of time spent on attempting to find what they are looking for wastes the user’s time and requires them to to spend more time inspecting the app for what they’re looking for by either having to scroll or tap through the application more than needed.

Recommendation

Make the “Home” page, “Shop” page and remove the bottom menu bar. Put a searching space at the top of the “home” page.

Severity Level Medium

13 

Visuals

Currently Recommendation

Finding 5 (Objective: Navigating to shop and buy)

Home page of application offers limited space for shopping and searching

Justification Forever 21 is a clothing store, and most users will access the application to shop. However, the user must go through various screens to access the main feature of the application to like search for products and shop.

Recommendation

Our recommendation supported by user feedback is to introduce a hamburger menu that links directly to different areas of the shop section on the homepage.

Severity Level Medium

14 

Visuals

Currently

Recommendation

15 

3. Task: Scan Barcode Findings ❖ Task Explanation: For this task, we asked our participants to scan a Forever 21 price

tag with the built-in scan feature of the application. Scanning the price tag should allow the user to find it on the online store, read reviews, and share via social media.

❖ Participant experience: While conducting the tasks, our participants were not sure if

the application was scanning, if the tag was framed properly, or if the application was working at all. Participant said, “Come on! Focus!” when the application was slow to respond. Participant 6 refused to complete the task and said, “I would never use this. Why would I scan an item I found in the store already to check to check the online inventory?”

❖ Table: Level of difficulty for scan barcode (1 being Very Easy and 5 being Very Difficult)

Participant

Rating (1-5)

P1 5

P2 4

P3 4

P4 4

P5 5

P6 1

❖ Data Description:

● 4 out of 6 participants took less than 3 taps and 2 screens during this task. ● Half of the participants were able to complete the task successfully. ● Two of the participants that were unsuccessfully in completing the task took 5

and 9 taps respectively. ● The remaining unsuccessful participant refused to complete the task, because

she did not believe it was something she would ever use. ● Aside from the user that refused to complete the task, all of the participants

rated the difficult level for this task as difficult or very difficult.

❖ Task Findings

Finding 6 (Objective: Impedes users from purchasing/searching for an item)

16 

There is no system feedback when scanning a barcode.

Justification While scanning a barcode, the user has no feedback on whether the position of the barcode is correct or that the application is scanning. The user wastes time holding and fiddling around with the camera to see if they are doing it correctly. The user can also manually enter the barcode, and may decide to do that rather than attempt to scan the barcode.

Recommendation

Have the borders of the camera frame light up green when the barcode is properly positioned and scanned.

Severity Level High

Visuals

Currently Recommendation

4. Task: Sharing an item on social media

❖ Task Explanation: We asked our participants to share the item that they like in the Forever 21 app on their social media account.

❖ Participant experience: During this task some of the participants were struggling to

remember their passwords on social media. The application does not link to social media websites successfully. It took lots of taps for most of the participants to share the item because they had to sign in to their account and get out of the app completely. Participants were not sure if they posted on their social media

17 

successfully because there was no feedback from the app. They had to go forth and back between their social media account and the app several times. One participant said, "There is no visible way to know on the page that you've posted." Another one exclaimed, "It's freaky twitter opens inside this app... ahh I can't go back...it's so slow!"

❖ Table: Level of difficulty for Social media (1 being Very Easy and 5 being Very Difficult)

Participant Rating (1-5)

P1 5

P2 3

P3 3

P4 3

P5 3

P6 1

❖ Data Description None of the participants completed the task successfully. The task was considered successful if the users could see the shared item on their social media. While some of the participants believed they have completed the task successfully (there is no feedback if the item has been shared or not), they all realized that the item had not been shared after checking their social media page. Therefore, the task was considered as incomplete for all the participants. Although participants were not able to complete the task, all but one thought this task was moderate. One participant mentioned it was a very easy task, because the process of getting to the share screen and logging in was easy; it was a system error that caused the post to not register on her social media site.

❖ Findings

Finding 7 (Objective: Usability flaws impede users from sharing item, number of screens and taps required)

There is no feedback when user taps on sharing icon and there are lots of steps to connect to social media websites.

Justification Sharing on social media should not take the user a lot of time or steps to complete this task. The application’s lack of system status creates

18 

uncertainty for the user when they attempt to share. As a result, the user may turn to alternative methods to share.

Recommendation Fix the functional problems related to connecting to social media websites. Provide feedback when user has posted on their page successfully.

Severity Level Low

Visuals

Currently Recommendation

5. Task: Locating a Forever 21 Store

❖ Task Explanation: We asked our participants to use the Forever 21 application to find a store location near them.

❖ Participant experience: Participants were able to easily locate where to search for the

nearest Forever 21 location. However, it was confusing for them to see two different “locator” buttons on the application. The application’s top most locator button takes the participant to a page that automatically populates nearby stores based on your current location if location services are on. Participants were confused by the ordering of store locations and said things like, “Why doesn’t it put it [the locations] in order?” and "Why isn't Michigan Ave listed? It's the second closest... it's not listed by distance... hmm I wonder how it’s sorted?".

19 

❖ Table: Level of difficulty for locating a Forever 21 Store (1 being Very Easy and 5 being Very Difficult)

Participant Rating (1-5)

P1 5

P2 4

P3 4

P4 4

P5 3

P6 1

❖ Data Description: ● Two of the six participants took over 20 screen taps and over 6 separate

screens to finally find the closest location of a Forever 21 to them. ● Only one participant was able to complete the task successfully and as

expected with just one tap and one screen with the application displaying the top result as the store in closest proximity to her.

● All but one participant encountered an error while trying to complete this task.

❖ Findings:

Finding 8 (Objective: Application Navigation)

There are two “map” or “location services” buttons on the application.

Justification Having two buttons for location services on the application creates confusion and difficulty in learnability since there are two paths to do the same kind of task. Also, the two separate location services buttons have different features from each other as they do not link to the same page.

Recommendation For the time being, since the Forever 21 app simply opens up the Forever 21 website inside the application, which causes there to be two location services options on the app, make the application’s location

20 

services button link to the same page as the location services on the website.

Severity Level Medium

Visuals

Currently

Recommendation

Finding 9 (Objective: Usability flaws impede users from finding Forever 21 store)

21 

The location services button to find the nearest Forever 21 store does not generate results in order of proximity. There is no logic to the order of results.

Justification Since the location services button automatically populates results based on your current location, users expect to see a list of store locations by distance (closest to furthest) to their current location. By not having the store locations list out in nearest distance order, it could potentially make users believe that there aren’t Forever 21 locations in their vicinity. This could in turn make locations lose in person traffic because shoppers are not aware of the locations nearest to them.

Recommendation Our recommendation would be to populate the results in order of proximity regardless if the application bases it off current GPS location or zip code.

Severity Level Medium

Visuals

Currently: The locations are listed at random .

22 

Recommendation: The Forever 21 locations include the distance

from the user’s location and are ordered from the closest to furthest distances.

Finding 10 (Objective: Usability flaws impede users from finding Forever 21 store)

Store Locator: The store icons under the map view do not contain useful contact information.

Justification The app currently only shows the store name when viewing the map. This causes the user to have to move to a separate screen for store information (hours, address, phone number). If a user is comparing the hours of a few stores, they must switch back and forth between the two screens, and leads to wasted time and effort. The listed view should be an alternative, rather than a requirement to find store information.

Recommendation Populate the store pins on the map with store information.

Severity Level Medium

23 

Visuals

Currently: Icons only displays only the store name.

24 

Recommendation: Add Forever 21 location and

options to call or get step by step directions.

Finding 11 (Objective: Usability flaws impede users from finding Forever 21 store)

The “Store Locator” tool does not open to the map view.

Justification Since the list of stores the application populates is not sorted by distance, the map view is more helpful for the user to determine the closest store. Other mapping applications and competitors show the map view first. This gives the user a visual and spatial reference to find their store

Recommendation Have the store locator default to the map view

Severity Level Low

25 

Visuals

Currently: Defaults to this page.

Recommendation: Default to this page automatically when map icon is tapped.

26 

6. Overall Experience & Satisfaction ❖ Explanation: We asked our participants rank their satisfaction with the Forever 21

application on a Likert scale from Very Unsatisfied to Very Satisfied.

❖ Table: My level of satisfaction navigating the Forever 21 application was:

Participant

Very satisfied- very unsatisfied

P1 Unsatisfied

P2 Unsatisfied

P3 Unsatisfied

P4 Unsatisfied

P5 Satisfied

P6 Neither satisfied or unsatisfied

❖ Data Description: Out of six participants only one participant was satisfied with their overall experience navigating the Forever 21 application. The participant was happy with the shopping experience on the application, and was able to quickly add the dress to her bag. It should be noted that the same participant also struggled the most trying to find the shop page of the application.

27 

V. Team Member Contributions

Team Member Email Address Specific Contributions

Mauli Shukla mauli.isha@gmail.com Attended all meetings, contributed in brainstorming and writing the User Test Plan, Moderator’s Guide, and writing/ finalizing negative findings. Conducted 1 usability evaluation. Wrote the procedure and participant sections on Evaluation Report.

Vicky Moreira vmorei1@hotmail.com Attended all meetings, contributed in brainstorming and writing the User Test Plan, Moderator’s Guide, and putting together a long list of negative findings. Conducted 2 usability evaluations. Created the spreadsheet to compile all team member’s participant's data. Wrote the introduction, made visuals for findings, and set up the layout and format of the Evaluation Report.

Frank Sweis frsweis@gmail.com Attended all meetings, contributed in brainstorming and writing the User Test Plan and Moderator’s Guide. Conducted 2 usability evaluations. Wrote and finalized positive and negative findings as well as created visuals for finding.

Gelayol Moradzadeh gelayol.moradzadeh@gmail.com Attended all meetings, contributed in brainstorming, writing the User Test Plan and Moderator’s Guide, wrote and finalized negative findings, positive findings and Executive Summary on the Usability Report. Conducted 1 usability evaluation.

28 

VI. Appendix Below are tables containing the data gathered from our usability evaluation wrap-up session. See attachments for Test Plan, Mod Guide, template of signed Consent Forms, and a spreadsheet of all the data gathered from the participants (includes two sheets- 1 with participants demographic information and 1 with observations notes and their feedback).

Finding a piece of clothing:

Participant Number of taps Number of screens

Number of errors

Complete the task?

P1 8 3 2 Y

P2 15 4 0 Y

P3 7 5 4 Y

P4 8 6 0 Y

P5 6 5 1 Y

P6 7 2 0 Y

Purchasing a clothing:

Participant Number of taps Number of screens

Number of errors

Complete the task?

P1 5 2 2 Y

P2 7 2 0 Y

P3 2 2 0 Y

P4 9 5 1 Y

P5 5 4 0 Y

29 

P6 3 1 0 Y

Sharing on Social media

Participant Number of taps Number of screens

Number of errors

Complete the task?

P1 10 3 3 N

P2 25 10 3 N

P3 14 4 2 N

P4 3 3 1 N

P5 11 6 0 N

P6 6 1 2 N

Finding Store:

Participant Number of taps Number of screens

Number of errors Complete the task?

P1 3 3 1 N

P2 10 3 1 N

P3 3 1 1 Y

P4 22 6 1 Y

P5 23 9 1 Y

P6 1 1 0 Y

30 

Scan Barcode:

Participant

Number of taps

Number of screens

Number Challenges Complete the task?

P1 2 2 1 Y

P2 2 2 1 Y

P3 3 1 2 Y

P4 9 3 2 N

P5 5 2 1 N

P6 1 1 0 N

31