Seminar Thesis in E-Government Systems - unifr.ch€¦ · Seminar Thesis in E-Government Systems . University of Fribourg, Department of Informatics . eGovernment . Evaluation of

Seminar Thesis in E-Government Systems

University of Fribourg, Department of Informatics

eGovernment

Evaluation of SmartParticipation Platform.

STUDENT NAMES: José Alberto Mancera Andrade STUDENT NUMBERS: 10-801-207 COURSE NAME: Electronic Business and Recommender Systems

DEPARTMENT: Department of Informatics

SUPERVISOR: ASSISTANT:

Prof. Dr. Andreas Meier Luis Terán

DATE OF SUBMISSION:

29-02-2016

II

Table of contents

List of Figures ............................................................................................................. V

1. Introduction ............................................................................................................. 1

1.1 Problem statement ............................................................................................ 1

1.2 Research Objectives and Methodology ............................................................. 2

1.2.1 Research Questions ................................................................................... 2

1.2.2 Objectives and Output of the thesis ............................................................ 2

1.2.3 Research Methodology ............................................................................... 2

1.3 Timetable .......................................................................................................... 3

1.4 Addressees ....................................................................................................... 3

2. User Centered Design (UCD) ................................................................................. 4

2.1 UCD Design Process ........................................................................................ 5

2.2 Elements involved in Human Computer Interaction (HCI) ................................. 6

3. SmartParticipation Evaluation Design .................................................................... 7

3.1 Main Evaluation Methodologies ........................................................................ 7

3.1.1 Usability ...................................................................................................... 7

3.1.1.1 User Segmentation .................................................................................. 7

3.1.1.2 User Tasks Assignment ........................................................................... 8

3.1.1.3 Questionnaire .......................................................................................... 8

3.1.2 Heuristics .................................................................................................... 9

3.1.2.1 Design Principles Definitions ................................................................. 10

3.1.2.2 Severity Errors ....................................................................................... 10

4. Smart Participation Evaluation .............................................................................. 11

4.1 Heuristic Evaluation ........................................................................................ 12

4.1.1. - Main Page ............................................................................................. 12

4.1.2. – Participate or Article List ....................................................................... 13

4.1.3. – Show Article Top .................................................................................. 14

4.1.4. – Show Article Down ............................................................................... 15

III

4.1.5. – Article Statistics .................................................................................... 16

4.1.6. – Login ..................................................................................................... 17

4.1.7. – Forgot Password .................................................................................. 18

4.1.8. – Register ................................................................................................ 19

4.1.9. – Search .................................................................................................. 20

4.1.10. – User Profile ......................................................................................... 22

4.1.11. – User Profile Update ............................................................................ 22

4.1.12. – User Settings ...................................................................................... 23

4.1.13. – Feedback Page .................................................................................. 24

4.1.14. – Recommendations .............................................................................. 25

4.1.15. – New Article ......................................................................................... 26

4.2 Usability Evaluation ......................................................................................... 30

4.2.1. – Usability Evaluation Matrices Results ................................................... 30

4.2.1. – Qualitative Analysis .............................................................................. 33

5. Evaluation Analysis and Recommendations Resume .......................................... 34

5.1 Main Problems Detected ................................................................................. 34

5.2 Recommendations .......................................................................................... 36

6. Low Level Prototype ............................................................................................. 38

6.1 SmartParticipation Web Pages ....................................................................... 38

6.1.1. - Main Page ............................................................................................. 38

6.1.2. – About .................................................................................................... 40

6.1.3. – Participate or Article list ........................................................................ 41

6.1.4. – Show Article .......................................................................................... 42

6.1.5. – Login ..................................................................................................... 44

6.1.6. – Register ................................................................................................ 45

6.1.7. – Search .................................................................................................. 45

6.1.8. – Feedback Page .................................................................................... 46

6.1.9. – User Profile ........................................................................................... 47

IV

6.1.10. – New Article ......................................................................................... 48

6.1.11. – Recommendations .............................................................................. 48

7. Conclusions .......................................................................................................... 50

8. Future Work .......................................................................................................... 51

9. References ........................................................................................................... 52

V

List of Figures

Figure 1.- Timetable Table .......................................................................................................... 3

Figure 2- User Centered Design Cycle ......................................................................................... 4

Figure 3.- User Centered Design Process .................................................................................... 5

Figure 4.- Human Computer Interaction Aspects ....................................................................... 6

Figure 5.- User's Evaluation Tasks .............................................................................................. 8

Figure 6.- Question 1 .................................................................................................................. 9

Figure 7.- Question 2 .................................................................................................................. 9

Figure 8.-Common Design Principles .......................................................................................... 9

Figure 9.- Severity Error Scale ................................................................................................... 11

Figure 10.- SmartParticipation Evaluation Process .................................................................. 11

Figure 11.- Main Page Heuristic Evaluation ............................................................................. 12

Figure 12.- Article List Heuristic Evaluation .............................................................................. 13

Figure 13.- Article Top Heuristic Evaluation ............................................................................. 14

Figure 14.- Article Down Heuristic Evaluation .......................................................................... 15

Figure 15.- Article Statistics Heuristic Evaluation ..................................................................... 16

Figure 16.- Login Heuristic Evaluation ...................................................................................... 17

Figure 17.- Forgot Password Heuristic Evaluation ................................................................... 18

Figure 18.- Register Heuristic Evaluation ................................................................................. 20

Figure 19.- Search Heuristic Evaluation .................................................................................... 21

Figure 20.- User Profile and User Profile Update Heuristic Evaluation .................................... 23

Figure 21.- User Settings Heuristic Evaluation ......................................................................... 24

Figure 22.- Feedback Page Heuristic Evaluation ...................................................................... 24

Figure 23.- Recommendations Heuristic Evaluation ................................................................ 25

Figure 24.- New Article Heuristic Evaluation ............................................................................ 29

Figure 25.- Easy Task Analysis .................................................................................................. 31

Figure 26.- Medium Task Analysis ............................................................................................ 32

Figure 27.- Hard Tasks analysis ................................................................................................ 32

Figure 28.- Comfort and Experience Level ................................................................................ 33

Figure 29. - Main Problems Detected ....................................................................................... 36

Figure 30.- Recommendations.................................................................................................. 37

VI

Figure 31. - Main Page ............................................................................................................. 38

Figure 32.- Main Page with Video ............................................................................................ 39

Figure 33.- About Page ............................................................................................................. 40

Figure 34.- Participate Page ..................................................................................................... 41

Figure 35.- Review Article ......................................................................................................... 42

Figure 36.- Article with Negative Comment ............................................................................. 43

Figure 37.- Sing In Window ...................................................................................................... 44

Figure 38.- Forgot Password Mini Window .............................................................................. 44

Figure 39.- Sing Up ................................................................................................................... 45

Figure 40.- Search Mini Window .............................................................................................. 45

Figure 41.- Search Results ........................................................................................................ 46

Figure 42.- Feedback Window .................................................................................................. 46

Figure 43.- User Profile ............................................................................................................. 47

Figure 44.- New Article Creation .............................................................................................. 48

Figure 45.- Recommendations Page......................................................................................... 49

1

1. Introduction

1.1 Problem statement

Overview

The relationship between politicians and citizens has been always in constant

interaction since the creation of governments. Most of the time, citizens find difficult

to feel identified with a certain political party or candidate and in certain cases

politicians lose the visibility of the topics or problems that society demand to be

attended.

SmartParticipation Web Platform emerges as a potential solution to cope with these

problems. It supports open discussions and political community building through

dynamic profiling.

The goal of this seminar thesis is to evaluate the platform from the user centered

design perspective and present a deep analysis that will help to provide not only a

better diagnostics from the usability and performance but also specific

recommendations to improve the system in a future version.

2

1.2 Research Objectives and Methodology

1.2.1 Research Questions

The next group of questions is the guideline of our study, each of them is answered

in sequence during the evolution of the document.

1. What is the design impact from the user perspective using the platform?

2. What are the required parameters, elements or models to consider in order to

evaluate the platform?

3. What are the User interaction issues in the platform?

4. What are the potential recommendations or improvements needed to create a

better version of the platform from the user centered design perspective?

1.2.2 Objectives and Output of the thesis

The goal of this seminar thesis is to evaluate the SmartParticipation platform from the

user centered design perspective and present a deep analysis which includes a

customized selection of parameters, elements or models that play a main role in the

evaluation of the platform. In addition the study includes an analysis of the user

interface and interaction issues which eventually will help to provide specific

recommendations and improvements for future versions of the platform.

1.2.3 Research Methodology

In order to answer the questions of this case of study, our analysis will be based on

the SmartParticipation platform, user data, previous studies in the use of

eGovernment systems and recommender systems. Furthermore, selected textbooks

will be taken into account, research papers, similar study cases or examples provided

by seminar supervisor.

3

1.3 Timetable

10-07-2015 Acceptance of working title

10-11-2015 Submission of the proposal

October 2015 Continue literature Research and reading.

Writing Chapter 1, 2 and 3

November 2015 Writing Chapter 4,5,6 Draft of the paper

Finishing the report

Revision and Correction

22.12.2015 Midterm Appointment

27.02.2016 Submission of the thesis report

29.02.2016 Presentation of the thesis report Figure 1.- Timetable Table

1.4 Addressees

The target audience of this paper is focused mainly on students in the fields of

information management, marketing, political sciences and professionals who are

involved in the field of eGovernment. The results of this seminar should provide the

parties mentioned above not only valuable knowledge in order to better understand,

analyze and improve the quality and design of Voting Advice Applications, but also a

better understanding of interface influence and the impacts that User Centered

Design plays in the design of such platforms as SmartParticipation.

4

2. User Centered Design (UCD)

In order to get a better understanding and interpretation of the findings of this

seminar study, it is necessary to understand the basic notions of User Centered

Design (UCD) and some terminology that is used later in the result interpretation.

UCD emerges as field, which answers some questions related to a specific platform

or web service such as: What can I easily build on this platform? What can I create

from the available tools? Or what do I as a programmer find interesting?

UCD is a discipline concerned with the Design, Implementation and Evaluation of a

platform or web application that humans use or interact. The figure 2 shows these

three main interactions that are inside of a main cycle, which is in constant movement

with the aim to improve the platform.

Figure 2- User Centered Design Cycle

UCD is vastly used nowadays to understand user experience and interaction with the

system in the way to identify important elements that help designers to improve the

platform in a new version.

5

2.1 UCD Design Process

The UCD design process is shown in the figure 3 and it contains the phases that a

conceptual web product has to walkthrough in order to become a complete

application that fulfil the final user expectations.

Articulate

User and task descriptions

• Questionnaires• Interviews• Focus groups

• Questionnaires• Diary studies• Ethnography• Interviews• Observation• Focus groups

Brainstorm

Throw-away prototypes

• Participatoryinteraction• Walk-through• Paper prototypes• Sketching• Storyboards• observation

• User involvement• Focus groups• Representations and metaphors

Refine

Testable prototypes

• Heuristics• Usability• WOz

• GUI design• Interface guidelines• Style guides

Complete

Complete systems

• Field Testing

Goa

lM

etho

ds a

nd to

ols

Prod

uct /

Res

ult

Figure 3.- User Centered Design Process

There are four main goals in the UCD design process:

• Articulate: The idea is to find the problem to be solved and list the desires

and characteristics that the final users have by many methods in order to get

the types of users and general task descriptions that in a way our future

system should fulfill.

• Brainstorm: Here the goal is to collect ideas from users, programmers and

start to generate the first approach of the platform. It is not necessary to start

to build the application, here paper prototypes are an option for fast

implementation and practical to receive some feedback from the final users or

even for the designers.

• Refine: In this part, the designers proceed to generate a first attempt for a

prototype in low, moderate or high level implementation. In this stage the

prototype should be tested by users or other techniques/methods and receive

some feedback that may affect again the prototype design.

6

• Complete: Here is the phase of launching the product of platform as a formal

service and measure the success or go back to a previous stage to continue

its refinement.

Based on these definitions, this seminar paper is focused mainly on the Brainstorm

and Refine phase for the SmartParticipation Platform. The seminar does not attempt

to achieve a full functional prototype.

2.2 Elements involved in Human Computer Interaction (HCI)

In addition to the UCD design process, defining the important aspects involved in HCI

are the key to support the seminar work analysis, the figure 4 shows the main

elements related to the Human Computer Interaction.

Figure 4.- Human Computer Interaction Aspects

Cognitive Process: Focus to understand the way that we user thinks and the role of

his attention, perception, memory, reasoning and problem solving.

Physiology: Understand the consequences and implications to see how the body

interacts with the system in terms of vision, hearing, touch/feeling and movement.

Emotions: Refers to moods, sentiments and emotions involved in the use of the

system or application and its effects.

Society: Focus mainly in how the environment of a person or people who surround

him affect or influence his decisions.

These elements will be referred later in our analysis when the evaluation of the

system takes place. For the moment is important just to keep in mind their meanings.

HCI

Cognitive Process

Emotions

Society

Physology

7

3. SmartParticipation Evaluation Design

As it was mentioned previously, SmartParticipation is already a full functioning

platform and categorized as a complete system, nevertheless it has not been

analyzed from the UCD design process. In order to perform an evaluation, it is

needed to bring the current system (SmartParticipation) back to the refine and

brainstorm phases of our model and perform two main evaluation techniques:

Heuristics and Usability. Based on the results of these techniques, a potential

brainstorm stage will be integrated in the analysis in order to get additional features

that may be integrated in a new low prototype fidelity version.

3.1 Main Evaluation Methodologies

Considering the purpose or the platform and the time to perform the evaluation,

Usability and Heuristics methodologies are used with the aim to provide a first

diagnostic of the system. The combination of both methodologies bring a powerful

result in the sense that one is focused in the users and the other in the functionality

design.

3.1.1 Usability

This method is conducted with real users by three specific techniques at the same

time: observations, walk-through and think a loud method. These techniques will

show user’s abilities, problems performing tasks in the system and level of

satisfaction or frustration.

3.1.1.1 User Segmentation

The Usability evaluation requires to segment users in groups in order to understand

the level of involvement with the platform. In this case the study expects to bring

generous feedback because the platform is working as a complete system and the

user can test the real functionality. For the purposes of our study we will select a

minimum of 8 participants and prepare an evaluation where all the subjects are

assigned to all tasks (within subjects) in different order, based on the characteristics

of system, we will choose users who are not familiar with e-Government platforms.

8

3.1.1.2 User Tasks Assignment

In order to conduct the usability evaluation mentioned above there is needed to

design a series of tasks that all the types of users have to perform. The proposed

tasks for the SmartParticipation system are shown in figure 5.

Figure 5.- User's Evaluation Tasks

The performing of these tasks by the users will provide important feedback

information with respect the usability of the system

3.1.1.3 Questionnaire

The usability evaluation also considers a questionnaire, which consists of two

questions, one at the beginning of the test (Figure 6) and a second one (Figure 7) at

the end of the usability evaluation. The scale used is from 1 to 5 where 1 represents

beginner or not at all comfortable and 5 experienced and very comfortable.

These questions have the aim to help to know better the user experience with the

platform in order to obtain a qualitative analysis that can be useful to get some

insiders about how particular users perceive the usability of the platform.

Easy

Explore main page and

understand the main purpose of

the platform

MediumSee articles

Create an account

Review and rate an article

Fullfill feedback Formulary

Follow a tutorial for recommendations

Hard

Modify User Profile

Create a new article with description and images

Lost password, retrieve a new one

9

Figure 6.- Question 1

Figure 7.- Question 2

3.1.2 Heuristics

This evaluation method has the advantage that there are not end users required and

it involves a systematic analytical inspection to see if the SmartParticipation interface

complies to design guidelines. This methodology applies common design principles

from D.Norman and from ergonomic principles Bastien/Scapin in references [2] and

[3]. The principles are shown in figure 8.

Figure 8.-Common Design Principles

Visibility

Feedback

Visible Constraints

Mapping

Consistency

Affordance

Idioms and Propulation Steriotypes

10

3.1.2.1 Design Principles Definitions

1. - Visibility: It is focused to see if the functions of the system are visible in a way

that the user can observe that these functions are relevant, otherwise invisible

automatic controls can make it more difficult to use.

2. - Feedback: It verifies if the system has simple interactions with the user such as

sound, highlight, animation, tactile or combination of these. Using it in the right way

can also provide the necessary visibility for user interaction and inciting the user for

the next step. (i.e. After pressing a button, the user receives a screen notification that

something has happened).

3. - Visible Constraints: It detects the action restrictions that the user can do in a

system and analyses if the design prevent the user from selecting incorrect options or

it helps the user to reduce the input error. (i.e. Entering a date in a system).

4. - Mapping: It verifies the natural relationship between controls and displays, it

ensures that the user knows for instance, what is the button that send an email or

delete an element.

5. - Consistency: It is related to see if the interface have similar operations and use

similar elements for similar tasks, for example that the publish article button is always

on the right and it does not change position depending of the different webpage. The

main benefit is that consistency helps to make interfaces easy to learn and use.

6. - Affordance: Refers to an attribute or an object that allows people to know how to

use it (suggest how one could use it, “to give a clue”). In graphical interfaces it

measures the intuitively level of the interface.

7.- Idioms and Population Stereotypes: It deals with interface idioms, which

represents arbitrary behaviors (i.e. red means danger, green means safe) and

population stereotypes varies in different cultures like presenting temperatures in

Fahrenheit or Celsius depending of the region where the website is consulted.

3.1.2.2 Severity Errors

Based on the heuristic design properties, each of them will be weighted during the

evaluation and the values that correspond to the severity of the error are defined from

11

1 to 5, where 1 is acceptable and 5 very critical and it has to be fixed as soon as

possible. (See figure 9)

Severity Level Description

1 I do not agree that this is a usability problem at all

2 Cosmetic problem only: no need to be fixed unless extra time is available

on the project.

3 Minor usability problem: fixing this should be given low priority.

4 Major usability problem: important to fix, should be given high priority.

5 Usability catastrophe: imperative to fix this before product can be released.

Figure 9.- Severity Error Scale

In the next section, these principles will provide useful recommendations during the

Heuristic analysis.

4. Smart Participation Evaluation In this section the two evaluation methodologies are applied to the SmartParticipation

platform, the chapter is composed by the application of both methodologies with their

respective comments and later there will be a general analysis section.

The complete evaluation process is shown in the figure 10, where we can see the

order of each evaluation.

HEURISTICS USABILITY Results

Figure 10.- SmartParticipation Evaluation Process

In the figure 10, heuristics is used in the first instance to recognize immediately errors

that can affect directly human computer interaction with the system and later be able

to create a low level prototype (graphical interface) that can be tested with real users

in the usability test.

12

4.1 Heuristic Evaluation

The evaluation will be performed in every page of the SmartParticipation platform

with an evaluation matrix that contains the design principles and their respective

weights from figure 9.

4.1.1. - Main Page

The main webpage provides a clear explanation of the purpose of the system,

nevertheless it is necessary to re-arrange the information in a new version, where it

looks like more as a commercial platform.

Principle Weight Analysis Result

Visibility 2 It is just a cosmetic problem and just need to re arrange the

information

Feedback 1 With the interaction with links or buttons there is always a progress

icon

Mapping 1 There is no problem knowing the function of the buttons

Consistency 1 The arrange of menus is kept

Idioms 3 The Interface has too much information that confuses the user and

it does not provide a clear purpose of the platform.

Affordance 1 The buttons with names make clear their function Figure 11.- Main Page Heuristic Evaluation

13

4.1.2. – Participate or Article List


Visibility 2 It is just a cosmetic problem and just need to re arrange the table


icon


Consistency 1 The arrange of menus is kept

Idioms 2 The Interface speaks the user language but it does not explains the

units used for the popularity section.

Affordance 1 The buttons with names make clear their function Figure 12.- Article List Heuristic Evaluation

14

4.1.3. – Show Article Top


Visibility 2 It is just a cosmetic problem and the presentation of the picture

could be more centered


icon


Consistency 2 The arrange of menus is kept but the colors of the buttons change

Idioms 1 The Interface speaks the user language

Affordance 1 The buttons with names make clear their function Figure 13.- Article Top Heuristic Evaluation

15

4.1.4. – Show Article Down


Visibility 2 It is just a cosmetic problem to change with other sections


icon




Affordance 1 The buttons with names make clear their function Figure 14.- Article Down Heuristic Evaluation

16

4.1.5. – Article Statistics


Visibility 2 It is just a cosmetic problem to change with other sections and

modification of the green section


icon



Idioms 1 The Interface speaks the user language but there is no explanation

of details of the statistics.

Affordance 1 The buttons with names make clear their function Figure 15.- Article Statistics Heuristic Evaluation

17

4.1.6. – Login


Visibility 2 It is just a cosmetic problem to change the size of the login

textfields.


icon and also a message for errors.


Consistency 1 The arrange of menus is kept.


Affordance 1 The buttons with names make clear their function. Figure 16.- Login Heuristic Evaluation

18

4.1.7. – Forgot Password


Visibility 2 It is just a cosmetic problem to change the size of the textfields.






Affordance 1 The buttons with names make clear their function Figure 17.- Forgot Password Heuristic Evaluation

19

4.1.8. – Register

20


Visibility 2 It is just a cosmetic problem to change the size of the textfields.






Affordance 1 The buttons with names make clear their function Figure 18.- Register Heuristic Evaluation

4.1.9. – Search

21


Visibility 2 It is just a cosmetic problem to change the colors of the text fields.

Feedback 4 With the interaction with links or buttons there is always a progress icon, a message for errors and no action by pushing search button for articles and when the text field is empty


Consistency 1 The arrange of menus is kept but no consistency in the articles

search with error messages

Idioms 2 The Interface speaks the user language but statistics need to

clarify units

Affordance 1 The buttons with names make clear their function Figure 19.- Search Heuristic Evaluation

22

4.1.10. – User Profile

4.1.11. – User Profile Update

23


Visibility 2 It is just a cosmetic problem to change the size of the text fields.


icon.


Consistency 1 The arrange of menus is maintained

Idioms 1 The Interface speaks the user language but too much text

sometimes that confuse the user.

Affordance 1 The buttons with names make clear their function

Constraints 1 User does not make mistake by the scale between discussion and

opinion. Figure 20.- User Profile and User Profile Update Heuristic Evaluation

4.1.12. – User Settings

24


Visibility 2 It is just a cosmetic problem to change the size of the text fields.


and provide feedback for bad input information icon.





Constraints 1 There is a verification of the input data in the textfield Figure 21.- User Settings Heuristic Evaluation

4.1.13. – Feedback Page


Visibility 1 No cosmetic problem


icon.


Consistency 1 The arrange of menus is maintained.



Constraints 1 User does not make mistake by the scales between awful and

awesome Figure 22.- Feedback Page Heuristic Evaluation

25

4.1.14. – Recommendations


Visibility 2 Cosmetic problem for the colors and size of submenus


icon and tutorial well explained.





Constraints 1 User does not make mistake by the scales between awful and

awesome Figure 23.- Recommendations Heuristic Evaluation

26

4.1.15. – New Article

27

28

29


Visibility 2 Cosmetic problem for the colors and size of submenus


icon and tutorial well explained.


Consistency 1 The arrange of menus is consistent.



Constraints 1 User does not make mistake by the scales or rating articles. Figure 24.- New Article Heuristic Evaluation

Based on all these properties among all the different menus, and options in the

SmartParticipation Platform. It was possible to find several minor improvements, the

platform originally has a good functioning and purpose to test some recommender

algorithms to the users and it achieves properly its purpose. Nevertheless, the

heuristic evaluation gives some recommendations through its properties in order to

improve the design in a new prototype from the perspective of a web designer.

It is important to remark that the heuristic evaluation was performed by one person,

so there may be still some unidentified improvements to the platform, which will be

complemented by a usability evaluation in the next section.

30

4.2 Usability Evaluation

As it was explained in section 3.1.1, a series of tasks will be performed from easy to

hard difficulty level and the evaluator must observe and not interfere with the logic of

the user while the user is performing the task. As previously mentioned the usability

evaluation was performed with 8 users in a within modality (all the users perform all

the tasks), using the think aloud method, in order that the users speak their thoughts

while doing the assigned task.

The users come from different educational backgrounds no related to computer

science, they are in an age range between 27-30 years old and their anonymity in

this study is preserved. In addition, this study does not make any distinction between

male and females.

Participants will be given instructions on what they are to do and they will be

encouraged to complete task as quickly as possible. They must not have a mobile

device that may distract them during gameplay, and will be monitored to ensure

proper task order. At the beginning and end the user will answer a short two

questions questionnaire in order to measure qualitative analysis.

The usability evaluation in this seminar is considered as a complement of the

heuristic analysis, which will provide us a deeper idea of the user decision process,

understanding or attitude, during the interaction of the user with the platform.

The next subsections resume the final findings derived from the transcript and

observation analysis of the 8 users.

4.2.1. – Usability Evaluation Matrices Results

The next tables’ matrices show the task, the time to complete the task in average by

the users and the analysis results.

Easy Task with Time Analysis Results

Explore main page and understand the main purpose of the platform.

The user got confused with the main interface, it was hard to

find the small box that describes the main purpose of the

system.

The main webpage does not talk a homogeneous language for

the users who are not familiar with computer science terms.

It is hard to guess the order of start to read the information, the

31

Average time to

complete the task

among users: 1.04 min

size of fonts do not express a clear hierarchy order where to

start to read.

Some users had to go to other secondary webpages to guess

what the system was about

The main interface does not reflect an intuitive way to tell the

users what is the platform about.

Users expressed that they spend already a lot of time and they

are not willing to continue to explore the webpage. Figure 25.- Easy Task Analysis

Medium Tasks with Time Analysis Results

Review Articles

Average time to complete

the task among users:

2.14 min

The user does not understand the meaning of the

recommendation numbers and it confuses him/her.

The statistics are not clear and units are not specified, in a

certain way the user would prefer a nomenclature box

The images should be bigger

The appearance of the green colors in the background give

the sensation to the user of being navigating in an old

fashion interface

Some users do not find disturbing the green color in the

statistics background but they would like to know the

meaning of the values.

Create an account



2.07 min

The interface does not speak the language of the user, it is

not clear where is the personal data, instead there is a big

explanation of algorithms that user may not understand.

The amount of information presented at the first time of an

account creation is very extensive and confusing for the

user.

It is not specified the symbols allowed in the new password

or the minimum length of the password.

Review and rate an article



The user is not sure about the functionality of the rate scale

purpose.

User took time to understand that in order to rate an article,

it should select the value and confirm with a button.

The article can be rated several times, the user it is not sure

if he can only rate one.

32

2.40 min The platform does not have a constraint to indicate if article

rate can be done only one time or several times.

Section to provide positive or negative commentaries is

tedious and difficult for the user to split his thoughts, he

would like to write ideally only one big commentary,

therefore it discourages him to provide comments.

Follow a tutorial for recommendations Average time to complete


2.59 min

It was very hard to find the tutorial, the button is confused as

a red title with no action.

The first description of the tutorial confuses the user when it

mention “columns”, the user does not understand to which

column the tutorial is referring.

The different colors in the tutorial section are helpful and

were a good insight for the users to realize that these were

the columns that the tutorial was talking about. Figure 26.- Medium Task Analysis

Hard Tasks with Time Analysis Results

Modify user profile

Average time to

complete the task


The slider does not show the values, so the user does not

know what is the value selected.

User gets confuse sometimes for the order or where to find the

personal details section.

It is not trivial to find all the options.

Once the values in the profile are changed, after the change,

the webpage redirect to another section, which causes a

consistency problem in the user.

Create a new article with description and images Average time to

complete the task


The user could not find how to add pictures.

Too many options with the article editor that make it confusing

The editor interface is not intuitive

In order to add the picture most of the users had to create an

article with only text and later re-edit it and attempt again to

add the pictures.

Figure 27.- Hard Tasks analysis

33

4.2.1. – Qualitative Analysis

The qualitative analysis was measured based upon two questions that were asked

before and after the experiment. (Figure 6 and Figure 7). The chart below (Figure 28)

shows the results of the user’s expertise in forum participation level and the

perceived comfort level of the experience of the platform by the end of the

experiment. As we can see, the expertise level of the participants that performed this

experiment was not high (an average of 1.5 out of 5), which shows they rarely use

forum participation platforms. The results also show that users expressed a low level

of comfort (an average of 2.375 out of 5) during experiment.

Figure 28.- Comfort and Experience Level

0

0.5

1

1.5

2

2.5

3

3.5

1 2 3 4 5 6 7 8

COM

FORT

LEV

EL A

ND

EXPE

RIEN

CE (1

-5)

PARTICIPANTS

User Experience Level and Comfort

Expertise Comfort

34

5. Evaluation Analysis and Recommendations Resume As it was shown in the previous section, the results of both evaluation methods show

that in general the main purpose of the system works as expected but there is a good

opportunity for improvements in the user interaction aspect of the platform. As it was

analyzed, users found difficult to navigate in all the menus of the system. In the next

subsection a series of main detected problems are shown and later a series of

recommendations are proposed in order to improve the human computer interaction

(HCI) of the SmartParticipation platform.

5.1 Main Problems Detected

The figure 29 provides a list of the main problems detected based on the heuristics

and usability methods. It is important to mention that the next problems detected are

presented in an abstract way based on the results of two evaluation methods in order

to present the information in a common terminology that the web designers and final

users understand. However in order to create a new version of the platform, it is

necessary that the web designer take in consideration the results of the sections 4.1

and 4.2.

Property Description of the problem by evaluation method

Visibility Heuristics: Cosmetic problems in the menus, contrast of colors, size of

fonts, submenu arrangement and information presentation

that cause navigation confusion for users.

Usability: The buttons and their arrangement causes confusion in the

article creation, revision, tutorial and profile modifications.

Feedback Heuristics: There is no feedback of the button search article in the search

section when it is empty and if we type any kind of input.

Usability: The user could not understand that the article search was not

returning a value because the user was typing incorrect

inputs.

Constraints Heuristics: There is no feedback of the button search article in the search

35

section when it is empty and if we type any kind of input.

Usability: The user could not understand that the article search was not

returning a value because the user was typing incorrect

inputs.

The user made several errors creating an account in the

password section, it was not specified the password syntax

requirements.

Modifying profile the slider does not show the values, so the

user does not know what is the value selected.

Mapping Heuristics: Although each button has a clear specification of the

functionality, sometimes their position creates confusion.

Usability: Understood the functionality of certain buttons such as login or

username or password but sometimes it got confused if he

has to push a button to update his profile for instance.

Consistency Heuristics: Although most of the buttons are clear in their function, there

are some that have some different color that could create

confusion with the user.

Usability: The user got confused with the colors and size of some

buttons, especially the tutorial button which was so big that

was confused with a title.

Affordance Heuristics: There is in principle no problem with the affordance of the site,

except for the tutorial part of the recommendation and the

main page that offers too much information in a unstructured

way.

Usability: Users experience problems for the tutorial section, it was not

intuitive as expected in some sections, such as the tutorial or

update the user’s profile

Idioms and user language

Heuristics: The main page, profile and tutorial speak sometimes a

technical language or provide some unnecessary information

36

to the user such as the technologies used to develop the web

service or the calculation of the recommender algorithm.

Usability: The user got confused by reading numerous times the

instructions of the tutorial and also to discover the main

purpose of the platform at the main page.

The statistics did not offered to the users a valued since they

do not have an explanation of the units or their purpose. Figure 29. - Main Problems Detected

5.2 Recommendations

The table in the figure 30 considers the problems detected during the evaluation and

provides a general skeleton recommendation for each section. It is important to

consider that a web developer who is willing to program these recommendations,

should follow the figure 30. Nevertheless, the details in the design improvements of

each section should be consulted in the results of the usability and heuristic

evaluation from sections 4.1 and 4.2.

Section Recommendation

Main Page Make it simpler and create an about page to add all the credits of

the authors and extra information

Participate or Article list

The presentation of the articles should be based on their image

preview and in principle no statistics preview but if the popularity

score must be displayed, then a small legend should explain the

units or the meaning of the number.

Article page In this part statistics should have a legend or a figure that the user

can map directly the meaning of the numbers and all the article

should be displayed in one page. In addition positive or negative

comments should be added at the end.

Login This should be instead of a new page, a discrete sub window in

the current page in order that the user can authenticate without

changing the current web page.

37

Forgot Password It should open a submenu or redirect to a new page that shows

several options to remember your password.

Register This should be a submenu or a new webpage where the user can

introduce the minimum amount of information, in order to avoid

that he will get desperate and lose the interest.

Search The option should be map with a universal symbol and display a

submenu that allows the user search by user or article.

User Profile This section is critical, processes as update or user settings

should be visible in one single page without menus in order to

avoid confusion and the personal information should be shown

first and later advanced options.

Feedback Page This option should be available with or without login in order to

encourage user’s participation and should be visible from the

main page.

New Article It is crucial to simplify the way that the photos and posts can be

created and most of the users prefer a simple editor, therefore

adding photos or text to the article should be easier without

embedded script editors.

Recommendations The tutorial button should be mapped in an icon and the

information should be presented in a graphical way, rather than

text. Figure 30.- Recommendations

Based on the evaluation and analysis results of the platform, it is possible to take a

further step and consider all the changes and generate a low level prototype that

could take into account the most important recommendations.

38

6. Low Level Prototype The aim of the low level prototype is to present a further idea, skeleton version of the

SmartParticipation layout or interface, this will help to solve the most critical usability

problems and provide a simple graphical idea to a web designer who would like to

implement a more functional and complete prototype.

6.1 SmartParticipation Web Pages

Based on the previous evaluations, the new prototype considers less sections and

present the information in a more compact way, with the aim that the user feels more

comfortable with the interface interaction.

6.1.1. - Main Page

The new redesigned main page has the aim to present the minimum of information

that the user cannot get confuse by a bomb of information as we can observe in the

figure 31.

Figure 31. - Main Page

39

In case that the user would like to know more about how SmartParticipation works, it

is necessary to click in the button “How it Works” which display a CSS script that

open a video mini window in the main page as we can observe in the figure 32.

Figure 32.- Main Page with Video

40

6.1.2. – About

In order to eliminate the confusion of the different information of the web main page,

it is important to add the objective, authors and technologies used to create the site

in a different web page with the less text possible and more graphics. A potential

proposal of these characteristics is presented in the figure 33.

Figure 33.- About Page

41

6.1.3. – Participate or Article list

The participate webpage allows to visualize the articles by their representative image,

with a small title at the bottom of each picture with a little description of the meaning

of the numbers that are located below each image. A button “create article” is also

available but it will require to sign in. (Figure 34)

Figure 34.- Participate Page

42

6.1.4. – Show Article

Simplicity should be a must, users during the usability test showed to spend more

time navigating through articles and see statistics. The new proposal design of this

section involves a design all in one with sequential order of the information. (Figure

35). When the user rate the article over 50 then the page suggest to write a positive

comment and when it does in the opposite, it suggest a negative comment. (Figure

36).

Figure 35.- Review Article

43

Figure 36.- Article with Negative Comment

44

6.1.5. – Login

In the previous version sign in was performed in an additional page, based on the

user’s behavior, most of them does not like to change web page while they want to

sing in, because they may be doing something already in the current page. That is

why a CSS in HTML5 can solve the problem with a micro window as it is presented in

figure 37.

Figure 37.- Sing In Window

The micro window also count with the options of “Forgot Password”, which in case to

be pressed a new micro window appears (Figure 38).

Figure 38.- Forgot Password Mini Window

45

6.1.6. – Register

Following the figure 36, if we are new users, we can Sign Up and then a new mini

window appears to start the registration process. (Figure 39)

Figure 39.- Sing Up

6.1.7. – Search

The search mini window (Figure 40) allows the user to select the type of search and

keep the user in the same web page.

Figure 40.- Search Mini Window

The user is redirected to a new search page in case that he clicks on the button

“Search” and have a display menu of information.(Figure 41)

46

Figure 41.- Search Results

6.1.8. – Feedback Page

The feedback can be accesses with or without login and it is also a mini window

(Figure 42), this window disappears automatically after the “submit” button is pushed.

Figure 42.- Feedback Window

47

6.1.9. – User Profile

This template for user profile solves the problem of having two different pages

describing the user, here the user can access first to their personal data and later to

its recommender options. The button “Update” is always visible in case that the user

would like to make changes. (Figure 43)

Figure 43.- User Profile

48

6.1.10. – New Article

The new version or the article creation is to make easier to the user to update the

information quickly without an embedded text editor that confuses the users in order

to add or remove media elements.(Figure 44)

Figure 44.- New Article Creation

6.1.11. – Recommendations

The new recommendations page considers to organize the information in main

categories:

Popular Articles

Recommended Articles

Recommended Users

Finally, if the user gets lost, it can click on the “Icon question mark” in order to start a

tutorial that can explain the meaning of each of the sections.(Figure 45)

49

Figure 45.- Recommendations Page

Before finishing this section, it is important to mention that the low prototype

presented in this seminar is a general guideline skeleton. The web designer who

would like to improve this prototype can move forward from this template model and

add more details or consider more characteristics such as text field verifications or

simple notification errors from the usability and heuristic evaluations.

50

7. Conclusions

After an extensive analysis and evaluation of the SmartParticipation platform, it was

possible to identify different opportunity design elements in the platform that needed

to be changed. The heuristics and usability evaluation provided important points of

improvements in order to inspire a low level prototype redesign.

Although the heuristic evaluation provides important feedback, it is always crucial to

consider also evaluations that consider users. Final user’s utilization and feedback

are vital to improve a system and it remains as a main advice for future evaluations of

the SmartParticipation platform.

The users who participating during this study were convinced that the idea of having

a platform of discussing topics and articles, represents an interesting service for

them, nevertheless the presentation of the information and the facility navigate in the

platform play an important role to keep users interested and willing to participate.

Finally, the user centered design cycle can always be used to find potential

improvements in current web platforms and avoid that they become obsolete, during

this seminar the challenge was to apply new concepts from User Centered Design

into a system, find the correct users and finally translate the improvements creating a

low level prototype that can provide examples of a desired new version of the

SmartParticipation Platform.

51

8. Future Work

After the creation of the second prototype based on the results of the evaluations,

there are some further steps that are necessary to bring the second low level

prototype to production.

Development of a SmartParticipation App for iOS or Android

Second heuristic evaluation considering more evaluators, rather than one.

Creation of a high or medium level prototype of the proposed new system, which

implies add all the functionalities of the original version.

Make a new usability evaluation based on the second high level prototype version

of SmartParticipation and eventually compare the same tasks between the

original platform and new prototype in order to see the differences y make

quantitative analysis.

Make usability evaluation considering behavior between male and female users or

by different age segments.

Integrate social networks and security guidelines to keep the integrity of the site.

Such as verification of data input in the text fields and SSL encryption for the data

transmission between the local computer and the server.

52

9. References

[1]. - Teran, Luís. Smart Participation: A Fuzzy-Based Recommender System for

Political Community-Building. Ph.D. Thesis of the University of Fribourg, Department

of Informatics, 2014.

[2].- Scapin, DL, Bastien JMC, Ergonomic Criteria for evaluating the ergonomic

quality of interactive systems, Behaviour & Information technology, 16, 1997, pp.220

– 231

[3].- Bastien, JMC, Scapin, DL, Leulier C., The ergonomic criteria and the ISO 9241-

10 Dialogue Principles: A comparison in an evaluation task, Interacting with

computers, 11(3), 1999, pp.299 – 322.

[4].- Human Computer Interaction 3nd ed – Dix, A., Finlay, J., Abowd, G.D., and

Beale. R. Prentice Hall, 2004.

[5].-Interaction Design, 2nd ed – Sharp, H., Preece, J and Y. Rogers. John Wiley and

Sons Ltd. West Sussex, UK. 2007.

[6].-Handbook of Human Computer Interaction – Jacko, J.A. and Sears, A. Lawrence

Erlbaum Associates, New Jersey, USA. 2003.

Documents

Seminar Thesis in E-Government Systems - unifr.ch€¦ · Seminar Thesis in E-Government Systems . University of Fribourg, Department of Informatics . eGovernment . Evaluation of