Heuristics for developing and evaluating smartphone mobile websites

Vasileios Xanthopoulos, MSc

York University ABSTRACT

Websites for smartphone use require different design and development approaches to desktop websites, taking into account the different physical designs, functionalities, and contexts of use, as well as the mental load of working with each platform. This study investigated usability problems with 7 smartphone websites via both iPhone and Android smartphones. 24 participants undertook tasks using a retrospective think aloud protocol. The usability problems identified were analyzed using a grounded theory approach where they were iteratively categorized with similar

problems and factoring for frequency of occurrences and

mean severity resulting in a final list of 4 categories and 16

problem subcategories. This categorization of problems was transformed into a set of 16 heuristics for the development and evaluation of websites for smartphone use. Comparing

the mobile heuristics with well-established web heuristics

showed high overlap but with a specialized view concerning

the mobile web. The use of the new heuristics increase the usability and user experience of smartphone websites and help create a more trustworthy, profitable and hospitable mobile web.

Author Keywords

Heuristics; mobile usability; mobile phone; mobile web

heuristics; smartphone;

INTRODUCTION

The mobile web has been an everyday commodity of the past

few years with 25% and rising, of global web access being

mobile [1]. Usability has surprisingly fallen behind causing

user frustration and confusion. The purpose of this study is

to develop heuristics for the design and development of

websites viewed on smartphones.

Smartphones and mobile web access have transformed every

aspect of human life. It was not always like this, I am

fortunate enough to remember the phones with the round dial

and the feedback sound they made when the dial returned to

its initial position. I hated having to call my father from that

phone because, back then, mobile phone numbers in Greece

started with 0. 0 made the round dial go all the way around

which took more time because I had to wait for the dial to

return back to its initial position to dial the next number

which made you think ‘how badly do I want to talk to this

person to have to endure this?’. Another issue with land line

phones was the fact that they were stationary. If you

happened to live in a big house you had to run to answer that

phone before the caller got bored and just hung up rendering

your sprint effort moot.

Nowadays mobile phones have reached 7billion subscribers

worldwide with 30% of those being smartphones [1]. 30% of

7 billion people have the ability to access the web on their

smartphones. Viewing websites on our mobile phones has

been a much different experience than accessing the web on

desktops/laptops, with this experience often being a tedious

one because designers and developers, without taking into

account the differences of mobile phones such as the smaller

screen size, input and output functions and context of use,

they adopted the same guidelines for designing and

developing websites meant for conventional web to the

mobile web. Why they do that? One reason could be that

there are no mobile web heuristics to help them achieve in

building a usable website for mobile use. The results of

adopting heuristics for conventional web access can be found

in everyday interaction with the mobile web. Websites look

different, they are difficult to use and navigate resulting in

poor user experience and task performance.

The approach followed to resolving these problems was,

firstly, understanding the need for developing new heuristics

by examining the differences between conventional and

mobile web access that would provide the foundation for

conducting a usability study to discover usability problems

that would eventually contribute to the development of

heuristics that would provide ‘rules of thumb’ that would

help mobile web access a usable and seamless experience.

Sections to follow are review of the relevant literature,

method section where the process and the equipment and

materials used to conduct this study, results section,

discussion section where the results will be discussed as well

as comparisons with well-established heuristics, limitations

and further work in the field and conclusions will be drawn

in the conclusion section.

REVIEW OF RELEVANT LITERATURE

Mobile web access being in its infancy has a lot of obstacles

to overcome and research has focused on the differences in

accessing the web on a mobile phone rather than from a

desktop pc or laptop (conventional web access). The

differences can be identified by just looking at the two

devices. Their physical design is one important factor to be

considered, the contexts of use are, theoretically, countless

for the mobile phone so accessing the web while on the move

is an important feature of mobile phones if not the most

important but how good are humans in multitasking i.e

walking and browsing, listening to the announcements on a

train to avoid missing your stop while browsing those are just

some of the questions that someone could ask even if he does

not have any special kind of knowledge about mobile web

access. These are reasonable questions that need to be

answered and that is why this literature review is structured

in a sequence to answer these questions as they were asked.

Small screen size

Screen size is the most obvious difference between accessing

the web from a mobile phone and accessing the web using

conventional methods.

[2] investigated the effect of small screen size and results

showed that users interacted in a much higher level when

reading from small screen displays rather than those of larger

displays because they had to page forward and backwards

much more in order to have a view of the text than those on

conventional displays on desktop pcs. Based on that fact, he

predicted that users of mobile web will make use of scrolling

and paging much more than conventional display users.

[3] conducted an experiment to measure the effects of screen

size on usability and perceived usability using 3 devices with

representative screen sizes on which users would interact

with an application. Results showed that perceived usability

and effectiveness were not affected by the screen size but

efficiency was significantly affected. [4] conducted a study

to examine the impact of screen size on users while they try

to achieve certain goals on mobile devices. Results showed

that the group with the large display answered twice as many

questions than the one with the small size display, also screen

size affects task performance, small screen users used the

search facilities twice as many time then large screen users

and that users of small screen performed a lot more

navigation actions such as scrolling or paging than their

conventional counterparts, interestingly most of the scrolling

was down and right. [2-4] showed not only the severe effects

of screen size and but also, [4] promoted the importance of

search functions on a webpage especially when accessing the

web through a mobile phone.

Search functions on the mobile web

Realizing the importance of search facilities on the mobile

web, [5] conducted a study on how to improve search on

mobile devices and results showed that when users succeed

in their search they do so quickly, in 2-3 minutes, and with a

small number of interactions while failures took

considerably more time. On average, users took twice as long

to successfully complete a search and were 60% less

successful than when using the conventional large screen

interface.

Context of use of mobile web

With accessing the web from our smartphone being so

common nowadays, the need for deeper understanding of

this phenomenon was required for this study to move

forward. More specifically where, when and how users

access the Web using their phones. Usage and its context are

very important pieces of understanding the needs users have

since the mobile phone was built to be used in a wide variety

of contexts.

[6] conducted an important study to identify the contexts

under which mobile internet is used most frequently and

what is the impact of context on the ease of use. Results

showed that participants used the internet 61 minutes on

average, the most frequent context of use was identified

when participants had a hedonic goal, their emotional state

was joyful, they were stationary, visual and auditory

distractions were low, very few people were around them and

the interactions were reported as low. 85.9% of 256 contexts

identified participants were stationary when interacting with

their mobile phone and 69.5% of total use was for hedonic

purposes rather than utilitarian and one hand interaction was

used 76.6% of the sessions. [6],[7] agreed on the fact that

users access the web from a stationary position. Additionally,

[7] shows very short sessions of usage. Why is this the case?

Cognitive aspects of mobile use in different contexts

[8] provided an explanation on why mobile usage while in

different contexts is done in short sessions conduct in a field

study were participants were asked to visit certain websites

and the researchers would record their actions while a page

was loading. Results showed that participant’s attention

shifted away 46% for mobile situations such as the railway

station, 70% for Metro platform while waiting for a train and

80% in a long quite street. The effect of context had a

significant effect on the duration of continuous attention to

the mobile device with bursts of attention of 8-16 seconds

while in the lab and the café while, in the escalator or a busy

street the bursts of attention were much shorter, below 6

seconds. [7],[8] showed consistency in their results that

usage sessions are very short because the second showed that

attentional resources are limited

Visual information density and navigation

How information is presented in small screen devices is of

grave importance because displaying information suitable

for a conventional screen would be inappropriate and

unusable because it is widely known that the legibility and

also the readability is hampered by increased density of text

on the screen [9],[10].

How do users navigate through webpage after webpage of,

admittedly, visually dense pages, what kind of problems they

face and what causes these problems?

[11] investigated whether the influence from cognitive

preview or visual density, affects the usability of small

screen devices and observe the effects on navigational

performance by manipulating text size, information density

and cognitive preview in older users. Results shows that font

size did not significantly affect performance but there was a

meaningful interaction between font size and size of preview

showing that the combination of the two contribute to

performance with a stronger impact on the preview size.

Also, font size did not affect navigation performance either

but the size of the preview affected disorientation measures.

Best performance was observed in the 4th condition of large

text/large preview and poorest performance was observed in

large font/small preview.

Conventional vs mobile web access

After examining the effects of context of use and cognition

in mobile web, we need to get to the chase by examining the

differences between conventional and mobile methods of

accessing the web. The questions to be answered is how easy

it is for users to browse full websites on their mobile phone

compared to the conventional web?

[12] evaluated mobile web browsing compared to desktop

web browsing. Results showed that user’s performance was

poor on mobile phones, the average completion time on

mobile was 5.7 minutes while on desktop it was 1.41minutes

and total average task completion time for all tasks on

desktop browsers for all participants was under 6 minutes

while the same average was 23 minutes for mobile phones.

[13] reached the same conclusion with results again showing

that on mobile optimized versions, participants were 30-40%

faster but they were annoyed by the limited features of the

optimized version. Setting aside the limited features issue,

mobile optimized pages, although far from perfect, are

clearly more usable and efficient so why not every company

designs a mobile optimized version of its full website? A few

companies decided to create mobile tailored websites

suitable for mobile viewing. These tailored websites were

designed to have fewer functions than the full website and

their design was fit for viewing on small screen displays but

far from perfect, with usability problems persisting. A

staggering report came from Google Inc. reporting that in

2011, only 21% of its largest advertisers have mobile

friendly websites [14].

This study was conducted to help close the gap presented in

the literature that mobile web access and interaction is

different from the conventional web access and there are no

heuristics to be used to help design and develop mobile

usable websites or evaluate the existing ones. This study

presents a unique and easy to follow and understand

heuristics and help designers and developers to finally, build

and evaluate websites for mobile phones.

METHOD

Design

The development of heuristics for websites viewed and

interacted with on a smartphone required extensive user

testing. Websites and participants to perform tasks on those

websites were identified. Usability problems identified

during retrospective think aloud were categorized iteratively

following a grounded theory approach. The mobile web has

been around for a few years now and its rise and usage ratio

does not justify the lack of design and development

heuristics for the creation of websites for mobile use with

significant usability and user experience issue raised by

users. To solve this problem, a usability study was conducted

to help designers and developers by producing heuristics for

smartphone websites. 7 interactive websites were identified,

a mixture of mobile optimized and non-optimized ones, or

full website as they are called in relevant literature. It was

decided that a homogenous sample was needed in terms of

age and web experience. 24 participants were identified who

fit certain criteria such as age, experience with mobile web

and experience with mobiles in general, to establish a

coherent group of users of the same general attitude and

attribute to mobile web because age and different experience

levels would be confound variables and they would affect

our results. These users would participate in a between

participant design usability study with 12 participants being

Android users and 12 being iPhone users both user types

would complete all tasks in all 7 websites and rate the

usability problem on a scale (0-4)[18].

Participants

24 participants took part in this study. 7 women and 17 men

between the ages of 20 and 30 years old with a mean average

of 26.58 years (standard deviation= 3.175). On average these

participants have owned a smartphone for 2.79 years (SD =

.977). Their weekly web access via their smartphone

estimated at 4.75 (SD = .532) on a 5 point Likert scale from

‘never’ to ‘everyday’ with 79.17% (19 of 24 participants) of

them reporting everyday access to the web via their

smartphones during the previous week, with a mean average

of 2.667 hours (SD = 1.5156) of daily web access. Results

showed a mean average of 3.33 (SD=.816) on that Likert

scale with 50% of them, not surprisingly, choosing the

middle choice/ground and 37.5% leaning toward the ‘expert’

side of the scale. A 5 point Likert scale from ‘Not Important’

to ‘Very important’ was used and a mean average of 3.75

(SD= 1.327) showed that participants considered accessing

the web through their phones as ‘important’ but not ‘very

important’. At the end of the session participants were

offered coffee and cookies as a reward for their participation.

Equipment

A laptop was used, throughout user testing, which carried the

software needed for recording video and audio during user

testing. Each participant would use his own mobile phone.

The video recording equipment for mobile phones was self-

made. Borrowing Steve Krug’s idea, a Creative 720p

resolution USB 2.0 webcam and a lightweight LED reading

light were purchased and with the help of a lot of duct tape,

the LED reading light’s flexible neck attached firmly on the

mobile phone and the webcam was taped on it to focus on

the mobile phone’s display firmly throughout the user

testing. The webcam’s native software was used to record the

first stage of the user testing where users perform tasks.

Camtasia 8.0, a screen capturing software was used to

capture the retrospective think aloud stage of the session and

rendering the finalized file for each participant. Finally, IBM

SPSS Statistics 20 was used for data analysis.

Materials

A pre-screening questionnaire divided in two sections was

given to the participant. The first section was for

demographic information and the second section consisted

of Likert scales, open-ended questions, and closed check box

questions. The questionnaire was given to the participant

prior to the main testing session after he/she had read and

signed 2 consent forms devised for both audio voice

recording and mobile display recording. During the test

session, pieces of paper with the website URL and tasks to

be performed on each website and a sheet with the severity

rating definitions, were given to the participant.

Procedure

Each session lasted around 60 minutes depending on how

much the user had to say during the retrospective think aloud

portion of the session. The users were greeted with coffee

and biscuits and were given the consent forms for video and

audio recording to read and sign them. After signing the

consent forms, the researcher explained to them the

procedure that would follow.

The mobile testing webcam was equipped on the

participant’s mobile phone and a brief test on the audio and

video recording quality followed. The participant was

handed a piece of paper with the website he needed to visit

and the tasks to be completed on that website so he would

not have to ask the facilitator again and again if he had

forgotten the task or he did not know how to type the URL

of the website, which might make him feel uncomfortable.

After the completion of the two tasks of that particular

website, the second website task paper was handed to them

and so on until the 3rd website-task paper was handed to

them. At that point the task session was paused and the

retrospective think aloud portion followed for the 3 first

websites. For each participant who took part in this study the

order of websites was reversed to accommodate for the

participants becoming tired and bored close to the end of the

process. During the retrospective think aloud portion,

participants would go through the replay of their interaction,

fast-forwarding in IDLE periods for example when pages

were loading, with the first 3 websites and talk about

problems they encountered as well as any good features they

encountered for each website separately.

If participants proved reluctant to talk they were kindly

prompted by the researcher on particular parts of the replay

video where the researcher detected uncertainty in their

(inter)actions, such as repeated scrolling left and right on the

same section of the website indicating that the user is looking

for something, or any prolonged pauses during the task that

could mean that the user is lost or cannot find something

important to continue with the task. Also, few participants

were reluctant to talk because they were shy and/or because

of their character. Those participants were prompted on the

homepage of each website to answer questions such as ‘what

do you see here’, ‘do you detect any problems or something

good you would like to mention’ and ‘what are your thoughts

of what you see on your display?’ If the user identified a

problem, the process was paused and the participant was

asked to rate the problem for its severity on a 4 point scale

where 1 = cosmetic, 2=minor, 3= major and 4 = catastrophic.

After the retrospective portion of testing was completed, the

user resumed the task portion with the 4 remaining websites.

At the end, the researcher thanked the users for their

participation in the experiment.

Data Analysis

A grounded theory approach was followed by the researchers

in the sense that the categories emerged from the data itself.

We proceeded with identifying patterns and recurring

themes. The first iteration of this process was the grouping

of usability problems of the same subject/theme and a title

was given to each group accordingly. The next iteration

included the creation of subcategories within these

categories and the merging of categories into more abstract

categories if necessary. Subcategories were identified and

each subcategory was then further analyzed for further

placement into one of the categories or as a higher level

category in itself. The third and last iteration of this data

analysis process included finalizing the abstract high level

categories, merging stand-alone categories into higher level

categories based on how and where the user identified the

problem. The completion of the third iteration resulted in the

first list of categorized problems. Those problems were then

further analyzed for frequency of occurrences and mean

severity ratings (1 - 4) to decide which of those would be

included in the final subcategory list. Categories with lower

than 3 frequency of occurrences were omitted or merged into

other subcategories. A second coder took a random sample

of approximately 10% of the problems identified by the first

coder and coded them independently into the initial set of

categories. The inter-coder reliability between the two sets

of coding was 82%. This inter-coder reliability was

considered adequate, so the first coder’s categorizations were

used.

RESULTS

138 distinct problems were identified by the participants

during user testing. Emerging categorization of those

problems after the iterative categorization resulted in a total

of 4 categories and 32 subcategories. Categories identified

were ‘Presentation’, ‘Content’, ‘Information Architecture’

and ‘Interaction’. Frequency of occurrences and mean

severity ratings were calculated. It became apparent that

further categorization and merging of categories were to

follow. Since [20], with 30 participants, omitted categories

with lower than 5 frequency of occurrences, we decided that

categories with less than 3 frequency of occurrences would

be omitted since this study had 24 participants because 32 is

a large number of problem subcategories.

Merging those subcategories with less than 3 occurrences

into other similar categories if appropriate. If it was deemed

inappropriate to merge into other subcategories, they would

be omitted from the final set of problem categories. This

process resulted in 16 problems being omitted along with

their subcategories, resulting in a new total of 122 problems,

4 categories and 16 subcategories reduced from 32

subcategories.

‘Presentation’ category had 3 subcategories, ‘Content’ had 3,

‘Information architecture’ had 3 and ‘Interaction’ had 7

subcategories. Interaction category was the category with the

highest frequency of occurrences with 45 occurrences,

followed by ‘Content’ with 34 occurrences, ‘Presentation’

with 26 occurrences and ‘Information Architecture’ with 17

occurrences (figure 1).

Results from examining each category individually showed

that the most frequently occurring subcategory for

Interaction was ‘Broken interaction consistency/conventions

not followed’ with 12 occurrences, the most frequently

occurring subcategory for ‘Content’ was ‘Too much

content/pictures/featurism’ with 22 occurrences which was

the most frequently occurring problem overall,

‘Text/interactive elements not large/clear/distinct enough’

for ‘Presentation’ category with 15 occurrences and last but

not least, ‘Content is not properly categorized/grouped’ was

the most frequently occurring subcategory for ‘Information

Architecture’ category with 10 occurrences.

‘Too much content/pictures/featurism’ was the most frequent

problem identified in this study but which one of the 4

categories was rated as the most severe one, thus, identifying

which category proved to be the most problematic for users.

Results showed that ‘information architecture’ was the most

problematic with a mean average of 3.3 followed by

‘Interaction’ with 2.99, ‘Presentation’ with 2.94 and

‘Content’ with 2.54 mean severity.

Finally, negative problem subcategory names were

transformed into positive heuristics (table 1).

Figure 1: Graph depicting frequency of occurrences per category

The analysis led to the identification of the most severe as

well as the most frequent categories and subcategories.

Problems with high severity should be addressed but also

problems appearing frequently cannot be ignored because

the cumulative difficulty and frustration they cause could

still severely hinder user performance and experience. One

example could be the problem subcategory ‘Too much

content/pictures/featurism’ of the ‘Content’ category which

has the highest frequency of occurrences of all the

subcategories. In this spirit, subcategories with severity

mean of over 3 in the 0-4 scale. 10 problem subcategories

were identified as very severe and should be prioritized when

addressing usability problems but, again, the need to address

problems with high frequency of occurrence cannot be

overstated. [15] reported user frustration has a time factor

embedded in so if the user faced a problem once but he

overcame it fairly quickly and the same problem persisted

requiring workarounds, even short ones, would be a problem

of increased severity according to [16]. The ‘8 or more’

frequency criterion was decided considering [20] criterion

for the same frequency measure. They had identified 907

problems and they set the criterion for high frequency at 10

occurrences thus, the decision for setting the criterion at 8 or

more. 7 subcategories were identified as occurring frequently

based on 8 occurrences or more criterion. Problem

subcategories were identified as being both of high severity

Table 1: VX heuristics

and high frequency. They can be identified as the severest

usability problems that must be fixed as soon as possible on

existing websites and must be avoided at all costs when

building a website for mobile use.

DISCUSSION

Overview and rationale

The mobile web, even today, offers a mediocre user

experience with the majority of websites having low

usability, making users prefer the conventional way of

accessing the web for what they deem as ‘serious’ tasks.

Using the same design guidelines for the design and

development of mobile websites proves unsuitable for

mobile web access because they do not consider the purpose,

physical design and context of use of mobile phones as seen

in the literature review. Mobile phones’ screen size, context

of use and cognitive requirements are very different from

those of a desktop or a laptop computer. Although, users may

be expecting the interaction to be as easy and straight

forward as the interaction with conventional desktop/laptop

web, the interaction is different and users prefer the

conventional ways than the mobile web.

The smaller screen size affects efficiency, task completion,

the cognitive workload required for interaction in different

contexts and the amount of interactions needed by the user.

Mobile phones are most often used indoors, for hedonic

purposes, when the user is stationary and there are not a lot

of people around. When mobile phones are used on the

move, the interaction is done in short bursts of less than 6

seconds because attentional resources are limited and

interaction with the mobile and sampling the environment

challenge the brain’s attentional capacity.

138 usability problems, for both full and mobile-optimized

websites, were identified by this usability study which

focused on producing usability heuristics for the mobile web.

First, the identified usability problems went through an

iterative grounded categorization process with 3 iterations to

be categorized into problem categories and subcategories

resulting in 4 major categories namely, Presentation,

Content, Information Architecture and Interaction and each

category had its own problem subcategories labeled

appropriately to represent the emerged problem. These

categories went through another iteration of categorization

where frequency and severity were measured and the

subcategories with lower than 3 frequency of occurrence

were omitted from the final problem table if they could not

be merged with other subcategories to form a new

subcategory with more than 3 occurrences while others were

merged into one category.

The results of this last iteration produced a finalized list of

16 evidence-based problem subcategories grouped into 4

major categories. Presentation had 3 heuristics, Content had

3, Information architecture had 3 and finally, Interaction had

7. One explanation for the majority of usability problems

being grouped in the Interaction category is that websites

being interactive is a given or at least they try to make them

interactive, leading to increased interaction problems

identified. These 16 problem subcategories were turned into

heuristics by transforming the negative problem subcategory

titles to 16 positive heuristic titles.

Interpretation and Analysis

Results showed that the most frequent usability problem was

identified as being ‘Too much content/pictures and

featurism’ which was also researched by [11], presented

during the literature review and it is not surprising. The

advances in e-marketing requiring an ever rising portion of a

page and the ever increasing features and functions fighting

for their own portion of the website can be compared to a

high value real estate where everyone wants a piece of. If that

was true for the conventional web, it is especially true and

important for the mobile web where that real estate is a hut

in terms of size. Also, Information Architecture is the

category with the highest mean severity of the four

categories with all of its 3 subcategories being rated as of

high severity. Structure, placement and grouping of

information are very important for the user to find his way

towards the completion of a task.

If information is not grouped or placed appropriately, user

has to search more than he wants to and should have to,

prolonging the task duration and increasing the interactions

he has to perform on that device. That device being a mobile

phone which, as seen in literature, inherently requires a lot

more interactions than the conventional web, leading to an

increase in effort needed, workload, cumulative frustration

and time. That is why it is not surprising that users rated

problems related to ‘Information Architecture’ so highly.

The highest severity subcategory from ‘Interaction’ category

is none other than ‘Broken consistency and convention not

followed.’ Anyone who has performed usability evaluations

knows that this problem comes up a lot and there is very good

reason why. Conventions are practices concerning structure,

placement, design and behavior of elements of the website

that have been in place for so long, they became norms. The

majority of websites try to keep conventions in the design

because users expect those conventions to be in place.

Inability to follow conventions leads to a phenomenon that

can be compared to ‘change blindness’, the inability of

human beings to identify changes in their visual periphery,

in the sense that if the user expects something to be placed

on the right side of the website and with a particular label, it

will take a lot of time for him/her to identify if he/ she ever

does, the same element if it is on the left side no matter how

big it is. This phenomenon happened numerous times during

user testing providing this study with a subcategory of high

frequency of occurrence and severity.

An interesting and unexpected problem came up during user

testing which led to a problem subcategory, based on its

frequency and eventually made it to the final list of

heuristics. The ‘Choose language type based on the context

and website’s target users’ heuristic and how it came to be a

problem subcategory is worth discussing. During user testing

users were asked to find and enable Facebook’s option to

‘review tags before they are posted on their timeline’. Most

users had a big problem with finding that option because

Facebook’s website was in Greek and the majority of users

did not know what the Greek translation of ‘tag’ was. This

problem led to the realization that even if they were Greeks

and they preferred the website in Greek, they had never used

the Greek word for it because ‘tag’ is a universal word when

it comes to Facebook. The interesting thing about this

particular usability problem they identified is that, firstly,

that usability problem would not have come up in an expert

evaluation if it was performed by the researcher of this study

because it had never occurred that something like that would

happen. Secondly, this problem illustrates how context

relevant language use supersedes the need of merely using

native language.

Comparison of study’s heuristics to conventional web heuristics

An important point in the discussion of this study is how the

heuristics proposed by this study fit in with the heuristics for

conventional web. Molich and Nielsen’s heuristics

[17,18,19] are the most popular heuristics, used for design

and evaluation of websites for years and Petrie and Power’s

heuristics [20], published in 2012 provide the most modern

and empirically sound heuristics for interactive websites.

Molich-Nielsen’s heuristics were compared to our new

heuristics (VX heuristics). This comparison proved

problematic because the labels of those heuristics are too

abstract and the discrepancy between the label and its

description in the type of language used makes them very

hard to remember.

Only 4 out of 10 of Nielsen’s heuristics are represented in

VX heuristics. 5 out of 7 of Interaction heuristics from VX

heuristics are not covered by any of the Nielsen’s heuristics

and this could be because Nielsen’s developed these

heuristics in 1990 and revised them in 1995. Back then,

websites lacked one important ingredient, interactivity. For

the same reason, only 1 out of 3 ‘Presentation’ heuristics of

VX heuristics were covered by Nielsen’s heuristics and that

heuristic was navigation’s design leaving out presentation

aspects of interactive elements, again highlighting the lack

of interactivity on Nielsen’s heuristics.

Comparison continues with the VX heuristics compared to

Petrie and Power’s web heuristics published in 2012. Petrie

and Powers’ heuristics [20], cover 87.5% of the problems

identified by this study with overlap of 14 out of 16 heuristics

of VX heuristics, again, VX heuristics lack error related

heuristics because incidentally, users made mistakes or slips

that did not result in errors. It was expected to have

overlapping heuristics with Petrie and Power’s heuristics

because both VX heuristics and Petrie and Power’s heuristics

investigate website usability problems. A central claim of

this study is that using web heuristics for the conventional

web is a mistake and leads to usability and user experience

issues. This overlap might make this claim seem rejected.

This overlap consists of two categories of overlap. The first

category is for general heuristics where the overlap is 100%

for each pair in terms of principle, in other words the first

category addresses heuristics that actually mean the same

thing and they are about the same problem such as: These

heuristics are the same and they describe the same problems

and the same general principle. The fact that VX heuristics

have such a high overlap with well-known heuristics is very

important and adds to its external validity. The second

category though addresses heuristics where the label is

similar and the general principle is the same but VX identify

heuristics specialized for the mobile web. The same general

principals apply stemming from testing websites, there are

major differences though, and those can only be seen when

reading the descriptions and examples provided by our newly

proposed heuristics. Examples follow:

# 3 VX heuristic and #1 Petrie and Power’s heuristic

were presented to overlap but the description of the first

paints a specialized picture about the mobile web based

on user data. It describes the fact that text and interactive

elements are expected to be small on a full website

viewed on a mobile phone but users need to be able to

recognize the text before they zoom in because they,

first, look for content and then they zoom in to

read/select.

# 4 VX heuristic and #6 Petrie and Power’s heuristic

overlapped based on the general principle of avoiding

having too much content on a page. That is true for both

mobile and conventional web but the issue for the

mobile web is much more intense because of the screen

size and the inability users have anyway to not be able

to perceive the whole page that makes them using

scrolling, zooming and paging functions a lot more, as

seen in the review of literature. The same goes for #5

VX heuristic

Another overlapping heuristic is #7 VX heuristic which

overlaps with #8 Petrie and Power’s heuristic where

content on mobile phones must be categorized and

grouped properly because large amounts of content

make for a lot of interactions required by the users to go

through it all. Content must be organized in a way that

users can find what they want without having to read

irrelevant to them information or having to scroll large

amounts of content to get to where they want. Especially

if they know what they are after.

An important overlapping heuristic is #9 VX heuristic

with #4, #13 Petrie and Power’s heuristics. Interaction

indicators must be proper and salient enough, in others

words the user needs to be informed whenever

something changes.

When users of this study used search filters on the left

side bar, they were automatically zoomed in close to the

filter they selected. The problem was that the rest of the

page was not visible so they did not know if selecting

this filter actually changed the results on the screen

because that part of the page was out of sight on the

mobile screen. Another reason why this usability

problem came up frequently is that the loading indicator

was out of sight too because the designers had placed it

in the middle of the results page but when designing on

a desktop or a laptop pc.

The above stated, demonstrate the fact that heuristics are

similar in their general principles but the differences between

the mobile web and the conventional web makes them

specialized to the mobile web’s restrictions and that is what

the heuristic descriptions are explaining including examples

from user data collected during user testing.

Limitations

The comparison between our new heuristics and Nielsen’s

heuristics helped identify weaknesses of our heuristics. 3 of

Nielsen’s heuristics are concerned with errors and

documentation which none of VX heuristics covers due to

the lack of errors appearing during user testing. That can be

attributed to the fact that the tasks to be performed by the

users were error free but not mistake/slip free. In other words

users made mistakes but those mistakes did not result in any

kind of error. The forms, an error causing feature of the web,

had auto-complete embedded, calendars and lists for date

input and radio buttons. In fact, calendars and the auto-

complete function of input fields were identified as some of

the good features that made their interaction much easier and

efficient. Another limitation some could identify about this

study is the fact that the usability study took place in a

laboratory environment, isolated by any kind of visual or

auditory distractions which is the opposite of how the mobile

phones are supposed to be used. Instead a field study would

increase result validity. Those comments are considered

perfectly reasonable and might be correct but the review of

relevant literature and especially literature on context of use

discovered that mobile phones are primarily used indoors

and when not too many people are around and more

specifically, mobile web use is most common when sitting

on the couch of one’s own home because the couch is a

comfortable place for users to access the mobile web and

there is no computer in that room. The above stated facts

could suggest that conducting studies for the mobile web in

a lab might not be invalid. Other limitations could be the

amount and quality of tasks to be performed by the user.

Tasks were simple and short, albeit very common for users

visiting these kinds of websites which were identified by the

users themselves. Or the fact the homogeneity of participant

sample and especially all users being of Greek nationality

and the role culture plays in usability evaluation.

Future work

Further work is needed to focus on examining whether these

heuristics are more effective in designing and developing of

mobile websites. Also, future work should focus on the

limitations mentioned in the previous section and examine

how effective and efficient field usability studies are

compared to laboratory studies when testing for mobile web

usability. Evaluations must be conducted using these new

heuristics and the results must be compared to results from

other heuristics. The limitation of those heuristics to identify

and evaluate error resulting interactions and feature is

something that needs to be addressed.

Benefits and implication

This study will greatly improve the mobile web because this

study and its products are based on solid research

foundations, deep understanding of the literature

surrounding mobile phones and the web as individual entities

and together, forming the mobile web.

The improvements directly resulting from adopting these

heuristics will not only be usable websites but also seamless

interaction with the web leading to improved user

experience. This improved user experience could lead to a

greater mobile web market penetration and the percentage of

web access through mobile could increase because users

would enjoy going online and browsing for goods,

information and services. The percentage of utilitarian

mobile web access could be increased, thus allowing more

users to trust the mobile web for their utilitarian tasks and

they would be less dependent on stationary means such as

desktop/laptop computers thus, making mobile web access,

really mobile. Speaking about trust, some of the participants

of this study mentioned that they would perform account

setting changed and pay online from their PC rather than

their mobile phone. Improved usability and user experience

could lead to user trusting their mobile phone to perform

tasks they would not perform otherwise because they did not

trust the mobile web mainly because of its design and the

increased number of usability problems they identified and

had to deal with.

Also these heuristics, being reasonable in number makes

them fairly easy to remember. Being able to remember the

heuristics when evaluating a website greatly increases

efficiency and performance because evaluators and designers

would not have to go back and forth reading revisiting the

table of heuristics and their descriptions. Additionally,

following these heuristics could solve another issue

discussed in the literature review, information density.

Information density has proven to be a major problem,

especially for the mobile web. Those heuristics cover this

problem by mean of a ‘Content’ category heuristic ‘Provide

the user with sufficient content but not excessive’ which was

derived by the most frequent usability problem occurring

during this study’s evaluation sessions. Avoiding high

information density allows for clarity, making the important,

for the user and the client, functions clearer and in

conjunction with heuristics on structure (Information

architecture) and design (presentation) make these functions

more visible and readily distinguishable.

Adoption of these heuristics for the design and development

of mobile websites could lead to websites of higher usability

and user experience, finally, making the mobile web a place

where the user would be able to perform most of the tasks he

used to perform on the web via conventional means.

CONCLUSIONS

This study showed mobile phone’s differences to the

conventional web access must be taken into consideration

when designing, developing and evaluating websites on a

mobile phone. Screen size, mental load, I/O functions and

context of use make for a mobile web in need of specialized

heuristics that adhere to the attributes and restrictions of the

mobile web. While heuristics for conventional web access

and mobile web adhere to the same basic design and

interaction principles, the mobile web heuristics proposed by

this study suggest a specialized approach. These heuristics

take consider the same basic design and interaction

principles of the conventional web and factoring the

attributes of mobile phones and their differences from

conventional web access, would help design and develop

mobile website with increased usability and user experience

that could turn mobile web into a more hospitable ‘place’.

Users would be able to perform both utilitarian and hedonic

tasks and decrease dependency to the conventional web to

those who previously used it for what they deem as ‘serious’

tasks due to mobile web’s usability problems and low

trustworthiness.

REFERENCES

[1] M. Meeker.(2014, May). "Internet Trends 2014"

[Online]. Available: http://www.kpcb.com/internet-trends.

[2] A. Dillon et al. “The effect of display size and text

splitting on reading lengthy text from the screen”. Behaviour

and Information Technology 9 (3) 215–227, 1990.

[3] D. Raptis, et al. "Does size matter? Investigating the

impact of mobile phone screen size on users’ perceived

usability, effectiveness and efficiency.," Proc. 15th Int. Conf.

Human-Comput. Interaction With Mobile Devices Services.

2003.

[4] M. Jones et al. "Improving Web interaction on small

displays," Comput. Networks: Int. J. Computer

Telecommunications Networking, vol. 31, pp. 1129–1137,

1999.

[5] M. Jones et al. "Improving web search on small screen

devices," Interacting With Computers, vol. 15, p. 479–

495,2013.

[6] H. Kim et al. "An Empirical Study of the Use Contexts

and Usability Problems in Mobile Internet," Proc. 35th

Hawaii Int. Conf. Syst. Sciences, 2002.

[7] Y. Cui & V. Roto, "How People Use the Web on Mobile

Devices," Www 2008 / Alternate Track: Industrial Practice

and Experience, 2008.

[8] A. Oulasvirta. et al. “Interaction in 4-second bursts: The

fragmented nature of attentional resources in mobile HCI”.

In Proceedings of SIGCHI Conference on Human Factors in

Computing Systems, ACM Press, New York, pp. 919-928,

2005.

[9] Norman, K.L., 1991. “The Psychology of Menu

Selection”. Ablex, Norwood, NJ.

[10] M. Ziefle. “Instruction format and navigation aids in

mobile devices”. In: Holzinger, A. (Ed.), Usability and

Human Computer Interaction for Education and Work.

LNCS 5298. Springer, , Berlin, Heidelberg, pp. 339-358,

2008.

[11] M. Ziefle, "Information presentation in small screen

devices: The trade-off between visual density and menu

foresight," Applied Ergonomics, vol. 41, pp. 719–730, 2010.

[12] S. Shrestha, "Mobile Web Browsing: Usability Study,"

Mobility '07 Proc. 4th Int. Conf. Mobile Technology,

Applications, Syst., pp. 187–194, 2007.

[13] G. Schmiedl et al. "Mobile Phone Web Browsing – A

Study on Usage and Usability Of The Mobile Web," Proc.

11th Int. Conf. Human-Comput. Interaction With Mobile

Devices Services, no. 70, 2009.

[14] A. Ha: Google Pushing Advertisers to Build for Mobile

Search. Adweek (2011)

[15] J. Lazar. "Severity and Impact of Computer User

Frustration: A Comparison of Student and Workplace

Users," Interacting With Computers, vol. 18, pp. 187–207,

2006.

[16] J. Nielsen. “Severity Ratings” [Online]. Available:

http://people.cs.uct.ac.za/~gaz/teach/hons/papers/Severity%

20Ratings%20for%20Usability%20Problems.html.

[17] R. Molich and J. Nielsen. Improving a human computer

dialogue. Communications of the ACM, 33(3), 338 – 348,

1990.

[18] J. Nielsen. “Usability engineering”. San Diego, CA:

Morgan Kaufmann, 1993.

[19] J. Nielsen. and R. Molich. Heuristic evaluation of user

interfaces. Proc. CHI 1990, ACM Press (1990), 249-256,

1990.

[20] H. Petrie and C. Power, "What do users really care

about?: a comparison of usability problems found by users

and experts on highly interactive websites," CHI '12

Proceedings of the SIGCHI Conference on Human Factors

in Computing Systems, pp. 2107-2116, 2012.