This site ﬁts YOU mom! - UvA...This site ﬁts YOU mom! The design and implementation of a model underlying an adaptive website to support mothers during decision-making about whether

This site fits YOU mom!The design and implementation of a model underlying an adaptive website tosupport mothers during decision-making about whether or not to have their

daughter vaccinated against HPV

Eva A. van Weel10244743

Bachelor thesisCredits: 18 EC

Bachelor Opleiding Kunstmatige Intelligentie

University of AmsterdamFaculty of ScienceScience Park 904

1098 XH Amsterdam

SupervisorsDr. F.M. Nack

Informatics InstituteFaculty of ScienceUniversity of AmsterdamScience Park 9041098 XH Amsterdam

Dr. A. Heuvelink

Perceptual and Cognitive SystemsTNO

Kampweg 53769 DE Soesterberg

June 27th, 2014

Acknowledgements

I wish to thank Annerieke Heuvelink for her guidance, support and inspiration during the process of con-ducting research and writing my thesis at TNO. Furthermore, I would like to express my gratitude to FrankNack for providing me with clear explanations, instructive drawings and advice during my project. More-over, I would like to express my appreciation to both for calming down my worries, having confidence inme and having a good laugh from time to time. Also, I would like to thank Wilma Otten and Hilde vanKeulen for taking the time to participate in my expert evaluation and Mark Neerincx for his help with theinteraction patterns described in this thesis. Mirjam Pot I would like to thank for providing me with insightsinto knowledge representations by showing me the domain from a different perspective. Finally, I want tooffer my special thanks to my parents and friends who have been supportive and loving throughout my entiregraduation project.

i

Abstract

In order to support mothers during decision-making about whether or not to have their daughtervaccinated against HPV, an interactive and adaptive website with two virtual assistants has been devel-opment. To achieve this adaptivity, the adaptive hypermedia approach was followed and four submodelswere defined by means of the sCE method. Overall, the implemented adaptations of the web pagesand the interactions of the virtual assistants were performed properly according to the expert evaluation.However, some adaptations were not convenient or noticed at all. This thesis thus provides insights intothe models necessary to construct an adaptive website and how those models can be implemented.

ii

Contents

1 Introduction 1

2 Related work 2

3 Models 33.1 Domain model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.2 User model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2.1 User’s degree of knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2.2 User’s background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2.3 A user template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.3 Adaptation model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.3.1 Adaptive presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3.2 Adaptive navigation support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.4 Embodied conversational agent model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.4.1 Relevance presented material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.4.2 Situated feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.4.3 Website functioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Implementation of the models 174.1 TailorBuilder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 WebSpeaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.3 A simplified implementation of the models . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5 Evaluation 20

6 Results 21

7 Discussion 23

8 Conclusion and future work 24

Appendices 27

A Code examples 27

B Expert evaluation guidelines 28

C Questionnaire expert evaluation (in Dutch) 29

iii

1 Introduction

Worldwide, cervical cancer is the third most common cancer among women (Ferlay et al., 2010). In theNetherlands, statistics reveal that in 2012 there were 735 cases diagnosed and 215 deaths from cervicalcancer (Integraal kankercentrum Nederland, 2014). Virtually all of these cervical cancers are caused by aninfection with the human papillomavirus (HPV) of which types 16 and 18 cause almost 71% of all cases(Munoz et al., 2004). The available vaccination against these types of the virus is believed to be capable ofdecreasing the number of women who contract cervical cancer by half (Gezondheidsraad, 2008). Therefore,the Dutch government started a national campaign to vaccinate 12 year old girls against these types of HPVin 2009. While the anticipated participation rate for this campaign was 70%, only 52% of the girls whowere invited to participate, actually completed all three steps in the vaccination procedure (Ensing et al.,2011). This participation rate was especially low compared to the participation rate of over 90% for theother vaccinations offered by the National Immunization Program (van Lier et al., 2009).

Due to this unexpectedly low participation rate, the Organisatie voor Toegepast NatuurwetenschappelijkOnderzoek (TNO) researched the willingness of mothers and daughters to be vaccinated against HPV. Thisresearch was commissioned by the Rijksinstituut voor Volksgezondheid en Milieu and ZorgOnderzoek Ned-erland - Medische Wetenschappen. According to this research, which was conducted by Van Keulen et al.(2010a), mothers fulfilled the most influential role in the decision-making with regard to the HPV vaccina-tion. Further research from Van Keulen et al. (2010b) indicated that mothers wanted more interactive andpersonalised information about the HPV vaccination. Moreover, 50% of the mothers had neither acquirednor processed detailed information about the vaccination and three months after the decision, 25% of themothers still felt ambivalent about their choice.

As a result of these studies, TNO is developing a website on which interactive and personalised informationabout the HPV vaccination will be presented to mothers in order to help them make an informed decisionabout whether or not their daughters should be vaccinated. Besides receiving interactive and personalisedinformation on the web pages, the mothers will be guided through the website and provided with feedbackby two virtual assistants: a peer mother and a doctor. The peer will provide feedback to the mother abouther progress on the website, guide her by suggesting what information she should read next and explain whythat information would be relevant to her. Furthermore, the peer will show empathy and understanding. Thedoctor will provide the mother with medical information and personalised feedback based on her answersto knowledge related questions.

In this thesis, we will present our research on the models underlying this interactive and personalised web-site. We will elaborate on those types of information that can be (automatically) acquired from the motherduring her interaction with the website, so that the website can be adapted to her information needs. Forthe website to be able to adapt its content, it needs to decide at what time during interaction certain contentneeds to be shown and how that content should be presented. We will describe on what information thesedecisions will be based and how these decisions affect the content of the web pages and the actions of thevirtual assistants. Therefore, a model will be constructed in which these decisions and the information tomake these decisions are stored. After introducing the relevant submodels, a simplified implementation ofthe proposed model will be presented. Furthermore, the evaluation of the implementation by two expertsfrom TNO will be described and the results presented. We will finish with conclusions and suggestions forfuture work.

1

2 Related work

The use of web-based tailored, i.e. personalised, interactive programs to support health related decision-making has been researched before in various domains, for instance breast cancer (Banegas et al., 2013) andlifestyle (Schulz et al., 2013; Ezendam et al., 2012). In these studies positive effects of the tailored programscompared to control conditions were found, respectively: reduced decisional conflict towards the decisionto take drugs to reduce the risk to contract breast cancer, decreased alcohol consumption and positive short-term effects on diet. Furthermore, research has been conducted to evaluate the effectiveness of tailored web-based interactive programs with virtual assistants. The results from Hudlicka (2013) suggest that virtualcoach-based training of mindfulness is potentially more effective than a self-administered program andFriederichs et al. (2014) showed that the use of an avatar in a web-based physical activity interventionincreased self-reported physical activity compared to the control condition without the avatar.

The web-based tailored interactive programs deployed in the studies mentioned contain various function-alities, such as acquiring user specific information by means of questions or other forms of user input andproviding feedback or information to the user based on the answers given. These functionalities were eitherprovided to the user by means of text on the website, a virtual assistant or a combination of both. To providethese functionalities a model of the user has to be formed based on which the feedback and information onthe website can be adapted to the user as "the challenge in an information-rich world is not only to makeinformation available to people at any time, at any place, and in any form, but specifically to say the “right”thing at the “right” time in the “right” way." (Fischer, 2001, p. 65). A possible way to implement thisadaptivity and to dispose of the “one-size-fits-all” approach is called Adaptive Hypermedia (AH). “Adaptivehypermedia systems build a model of the goals, preferences and knowledge of each individual user, and usethis model throughout the interaction with the user, in order to adapt to the needs of that user” (Brusilovsky,2001, p. 87). AH systems contain at least three types of models: a user model with user specific information,a domain model with information about the concepts in the domain and the relations between these conceptsand an adaptation model which provides information about possible adaptations given the user and the do-main (Knutov et al., 2009). Within an AH system different models and techniques can be used to constructthese three models. Two possible approaches to model the user his knowledge are called the overlay modeland the stereotype model. An overlay model estimates the user his knowledge of each concept in the do-main model and the stereotype model classifies the user his knowledge by determining to which stereotypethe user belongs (Brusilovsky, 1996). Furthermore, different approaches can be followed when modellingthe adaptations within an adaptive system (Brusilovsky, 1996), i.e. the presentation of information and thenavigation support can be adaptive. Information can for example be emphasised, shown or not be shown orpresented differently by means of images or graphs. Adaptive navigation support can consist of for instancehiding, disabling or removing links, providing link sorting or direct guidance.

In order for the virtual assistant to interact with the user, it has to possess several characteristics. It shouldhave the ability to recognise and respond to verbal and nonverbal input, to generate verbal and nonverbaloutput, deal with conversational functions such as turn taking and provide signals that indicate the state ofthe conversation. A virtual assistant that has these properties is called an embodied conversational agent(ECA) (Cassell et al., 2000). An ECA engages a user in a dialogue by means of using speech, gesturesand other verbal and nonverbal cues which give the experience of human face-to-face interaction and thepossibility to build a relationship (Bickmore and Cassell, 2001). Using an ECA has multiple advantagesover using only text as it improves recall of the presented information (Beun et al., 2003) and transfer oflearning (Atkinson, 2002). Furthermore, the visual presence of the ECA is critical, as a voice alone is less

2

effective in conveying a message (Rosenberg-Kima et al., 2007).

Even though web-based tailored interactive programs with ECAs are not new, promoting vaccine uptakeby means of such a program is. Therefore, this work will elaborate on the models necessary to provide aweb-based tailored interactive program with an ECA to support health related decision-making with a focuson the decision about the HPV vaccination. The corresponding research question is the following: What isthe required model underlying an interactive web-based adaptive feedback program with a virtual assistantto support health related decision-making?

During the development of this interactive and adaptive website, the situated cognitive engineering (sCE)method will be used, which describes the process of iterative development cycles (Neerincx and Lindenberg,2008). The specifications of the website will be refined in each of these cycles, where the initial specifica-tions will be described in this thesis. The specifications of the website consist of requirements (what), claims(why) and use cases (where, when, who), see Figure 1. These specifications are initially based on analysisof the domain and can change during the process of development. The model underlying the adaptivity ofthe website, will be described in terms of these specifications and after evaluation be refined by adjustingthe specifications.

Figure 1: The relations between the defined requirement, its claims and use cases

3 Models

In order to provide the requested interactivity and personalisation on the website, the introduced adaptivehypermedia (AH) approach is followed. This approach allows to dispose of the idea of “one-size-fits-all”and offers the ability to adapt the content of the website to the user. To achieve this, the website requiresinformation about the domain, i.e. what information can be adapted, the user and the actual adaptations thatcan be made. Therefore, three submodels are defined which represent this information: a domain model(DM), a user model (UM) and an adaptation model (AM) respectively. Besides these three submodels, asdescribed in the AH approach, a fourth model is defined which represents the interactions of the virtualassistants with the user, i.e. the embodied conversational agent model (ECAM).

Figure 2 shows the relations between the models proposed. The UM and the DM provide input to the AM,based on which the AM can reason about which adaptation to perform given the user and the domain. TheECAM receives its input from the UM and DM, so the ECA can appropriately interact with the user. Also,ECAM receives input from the AM, so the ECA can provide the user with meta-communication about theperformed adaptations.

3

Figure 2: Relations proposed models

With the sCE approach in mind, the first four requirements (Req) of the model are defined based on thesemodels:

DM-Req 1: the DM shall define the concepts in the domain and the relations between these concepts.

UM-Req 1: the UM shall store information about the user that is a relevant source for adaptation.

AM-Req 1: the AM shall adapt the content of the web pages to the user based on the input from the DMand the UM.

ECAM-Req 1: the ECAM shall adapt the interactions of the ECAs based on the input from the DM andthe UM or the input from the AM.

For each of these requirements claims (reasons why they should hold) are defined, which are presented witheither a + or a - to indicate a possible positive or negative effect when incorporating the requirement in thesystem.

Claims DM-Req 1:+ the DM enables the order and form of the information on the website to be adapted to the user.

Claims UM-Req 1:+ The UM provides the information about the user based on which the information on the website can be

adapted to the user’s level of knowledge, needs and interests.

Claims AM-Req 1:+ The user is likely to understand the provided information better. + The user is likely to enjoy the visit

to the website, as the information on the website is adapted to the user’s level of knowledge, needs andinterests.

Claims ECAM-Req 1:+ The user will belief the statements of the ECA.+ The user will enjoy the interactions of the ECA.+ The user will find the interaction with the ECA relevant.

In the following subsections the individual models will be described by means of providing sub requirementsand corresponding claims, use cases and rules explaining how the sub requirements can be fulfilled.

4

3.1 Domain model

As described in DM-Req 1, the DM contains information about the concepts in the domain and the re-lations between these concepts. It provides a structure of the information in the domain and shows whatinformation can be adapted. The information in the DM is usually be structured hierarchically, where someof the concepts in the hierarchy have child- or sub-concepts. This hierarchical structuring enables a logicalnavigation of the user through the website. Besides hierarchical relations, it is also possible for relations toexist between concepts in different branches of the hierarchy. These relations can be used to offer alternativeroutes for navigation to the user (direct links between subparts of the website) or to adapt the content of asubpart of the website given the user’s visit to a related subpart. Two sub requirements and correspondingclaims can thus be defined:

DM-Req 1.1: the DM shall define the hierarchical relations between the concepts in the domain.

DM-Req 1.2: the DM shall define the relations between concepts in different branches of the hierarchy.

Claims DM-Req 1.1:+ Based on the DM logical navigational routes between the concepts in the domain can be defined.- The DM limits the number of ways in which the content of the website can be approached.

Claims DM-Req 1.2:+ The DM allows logical skip/quick routes between different parts of the website (subconcepts in the

domain) to be defined.+ The DM allows the website content (subconcepts in the domain) to be adapted based on whether related

content is yet visited.

In order to fulfil these requirements, a hierarchy needs to be constructed in which these different types of re-lations between the concepts are defined. For the website to support mothers during decision-making aboutwhether or not to have their daughter vaccinated against HPV, a DM is constructed in which these relationsare defined. On the highest level in the DM, various knowledge categories are presented, such as basic in-formation, side effects and benefits early age. Each of these knowledge categories contain basic informationabout the subject and in most cases additional or more detailed information. Besides these hierarchical rela-tions, there also exist relations between the information in different branches of the hierarchy. As describedabove, these relations indicate either that the same information is present in two or more branches or thatinformation is somehow linked to information in another branch, e.g. that information is also relevant toread given the topic currently being read.

3.2 User model

As described in UM-Req 1, the UM stores those types of information about the user that provide a relevantsource for adaption. In order to obtain this information, two methods can be utilised to acquire user input,that is 1) query the user or 2) monitor the behaviour of the user during interaction. The obtained informationbe either dynamic, i.e. the values change during interaction with the system, or static, i.e. the values do notchange during the course of interaction. Furthermore, the information can be domain independent or domaindependent. The former involves information about the user that can be acquired from the user independentof the domain in which the model is applied, for instance age, gender, religion or level of education. The

5

latter on the other hand, involves information that is directly related to the domain of the application, suchas the user’s level of knowledge about the subject or the information that has already been visited.

For the website about the HPV vaccination, the following requirements are established:

UM-Req 1.1: the UM shall keep track of the user’s degree of knowledge of the concepts in the domain

UM-Req 1.2: the UM shall acquire information about the user’s background

In the following subsections each requirement will be elaborated on by presenting the corresponding claims,use cases and rules on how to fulfil the requirements.

3.2.1 User’s degree of knowledge

The claims why to assess the user’s degree of knowledge about the concepts in the domain are presentedbelow:

Claims UM-Req 1.1:+ The presented information can be adapted to the user’s degree of knowledge.- The user could experience irritations or impatience by providing the system with information based on

which the user’s degree of knowledge can be determined.

For this requirement two more sub requirements are established in order to specify the requirement evenfurther. For each requirement the corresponding claims are presented as well.

UM-Req 1.1.1: the UM shall store the user’s answers to knowledge related questions

UM-Req 1.1.2: the UM shall store which concepts in the domain are visited by the user

Claims UM-Req 1.1.1:+ The user’s degree of knowledge about the concepts in the domain can be determined.- The user could experience irritations or impatience by answering the knowledge related questions as the

website is most likely visited to obtain information instead of to provide information.

Claims UM-Req 1.1.2:+ The website knows which information has been presented to the user and can from that point on be

considered as knowledge that the user possesses.+ The system can keep track of the progress made by the user.

A use case will now be described which illustrates the above mentioned requirements. Furthermore, rulesare provided indicating how the requirements are fulfilled.

UM-Use case: acquire the user’s degree of knowledge about facts and fables of the HPV vaccination

1. The mother visits the facts and fables knowledge category on the website

2. Statements are shown to the mother about various facts and fables and she is asked to indicated whichstatements she thinks are facts and which are fables

3. The mother indicates which statements she thinks are facts and which are fables by clicking either“Fact” or “Fable” for each statement

6

4. The website stores the answers

5. The website stores that this specific part of the knowledge category has been visited by setting theconcept’s visited boolean to true

6. The website determines the degree of knowledge of the user about the current concept, i.e. facts andfables, in the domain

The final step in this use case describes the website being able to determine the user’s degree of knowledgeabout facts and fables. As the degree of knowledge does not immediately follow from the mother’s answers,a mapping has to be made from the answers to the corresponding degree of knowledge. Each answer isgiven a score which represents the user’s knowledge about that specific part of the domain, e.g. 0 for anincorrect answer and 1 for a correct answer. For each concept in the domain, the degree of knowledge issubsequently calculated by means of these scores. By calculating the user’s weighted average knowledgefor each concept in the domain, different weights can be assigned in the case that not all knowledge isequally important within the concept in the domain. The following calculation is used, where wi representsthe weight of each question i, si corresponds to the obtained score, n equals the total number of questionsanswered so far within that specific concept and t represents the total possible score within that specificconcept, i.e. the score that corresponds to the user having answered all the questions correctly.

Kw =

n∑i=1

wi ×sit

In order to provide adaptations based on the degree of knowledge of the user, this score can be mapped to aboolean, i.e. the user has either knowledge or not, discrete values, i.e. low knowledge, average knowledgeand high knowledge, or continuous values. Given the adaptations and the interactions of the ECAs whichfit this application, as will be described in Sections 3.3 and 3.4, the weighted average knowledge will bemapped to a discrete set of values. The following algorithm describes this mapping:

degreeOfKnowledgeConcept = "";if Kw < 0.3 then

degreeOfKnowledgeConcept← "low";endelse if Kw < 0.7 then

degreeOfKnowledgeConcept← "average";endelse

degreeOfKnowledgeConcept← "high";endAlgorithm 1: Mapping of weighted average knowledge to a discrete value to be used by the AM or ECAM

The values 0.3 and 0.7 in Algorithm 2 are chosen by estimation. During the development cycles of thewebsite, these values can be adjusted to fit the degree of knowlege of the user better.

3.2.2 User’s background

Background information about the user can be a relevant source to base adaptations on. For UM-Req 1.2,the following claims are established:

7

Claims UM-Req 1.2:+ The information can be adapted to the user’s background information.- The user could experience irritations or impatience by providing the system with information about its

background.

For this specific website, static domain independent variable, such as gender and age, are not relevant as thewebsite is targeted to any mother that has to decide whether or not to have her daughter vaccinated againstHPV. The following background information, however, is relevant and therefore needs to be written downin terms of requirements for the website. Why this information is relevant will be described by the claimsfor each of these sub requirements.

UM-Req 1.2.1: the UM shall store how much information the user has already gathered about the topicbefore interaction with the system

UM-Req 1.2.2: the UM shall store the user’s level of education

The claims corresponding with these requirements are the following:

Claims UM-Req 1.2.1:+ The provided information can be adapted to the user’s initial degree of knowledge.- The user could experience irritations or impatience by providing the system with information about the

amount of information already gathered.

Claims UM-Req 1.2.2:+ The presented information can be adapted to the user’s level of education.- The user could experience irritations or impatience by providing the system with information about its

level of education.

In order to fulfil these requirements, two questions are posed to the user upon entrance of the website.Therefore, the following rules are used:

amountOfKnowledgeGathered == null;levelOfEducation == null;if the user enters the web page about background info then

if amountOfKnowledgeGathered == null thenpose question about amount of info gathered;amountOfKnowledgeGathered← "low|average|high";

endif levelOfEducation == null then

pose question about amount level of education;levelOfEducation← "low|average|high";

endend

Algorithm 2: Retreiving the user’s background information

8

3.2.3 A user template

By fulfilling the requirements introduced above, a profile of the user is constructed which contains the userspecific values that are gathered either by posing questions or by monitoring the behaviour of the user on thewebsite, e.g. what links are clicked on. Such a profile is called a user template, which for each user containsthe same attributes with corresponding possible values. In the end, different users have obtained differentvalues for these attributes based on their own interaction with the website. A part of the user template forthe HPV website is shown below:

Level of education: "low|average|high"Amount of info gathered: "low|average|high"Score of knowledge question 1 concept "Facts & Fables": "0|1"Degree of knowledge concept "Facts & Fables": "low|average|high"Facts & Fables visited: "true|false"

A user template enables the website to create a profile that is descriptive for each individual user. Reasoningabout possible adaptations based on the values of the attributes in the user template occurs within the AM.This reasoning can be complex, as the adaptations can be made based on various (combinations) of valueswithin the template, as will be described in the next section.

3.3 Adaptation model

The AM coordinates the possible adaptations the system can perform, as defined in AM-Req 1. Theseadaptations should ensure that the system fits the user’s information needs and level of knowledge. The AMlinks the DM and UM by specifying how the domain needs to be adapted to the user at what time duringthe interaction. Within adaptive hypermedia systems a distinction is made between two types of possibleadaptations, that is adaptive presentation and adaptive navigation support. Adaptive presentation includesthose adaptations that change the presentation of the information offered by the system, whereas adaptivenavigation support adapts the navigation support to the user, e.g. the colour of the links. These types ofadaptations form the two main requirements for the AM of the website.

AM-Req 1.1: the AM shall provide the user with adaptive presentation

AM-Req 1.2: the AM shall provide the user with adaptive navigation support

The corresponding claims are the following:

Claims AM-Req 1.1:+ The user will enjoy the presented material.+ The user will better understand the presented material.

Claims AM-Req 1.2:+ The user will appreciate the presented navigation support.+ The user will obtain more information, as the navigation support helps to navigate to the most relevant

information.+ The user does not have to search and think what information would be relevant to read next.- The user might experience irritations, as the user would rather decide where to navigate without

support.

9

3.3.1 Adaptive presentation

The types of adaptive presentation, that are relevant for this website, are: multimedia adaptation, naturallanguage adaptation and canned text adaptation. Multimedia adaptations consist of adapting the presentingmultimedia to the user, such as images, graphs or videos. Adaptation possibilities involve the showing orhiding of certain types of multimedia to specific users, such as showing graphs with various numbers andstatistics to users with a higher level of education and more simplified images to users with a lower level ofeducation. Natural language adaptations involve the adjustment of the formulation of the texts presented bythe website to the user’s level of education or current degree of knowledge. Lastly, canned text adaptationsinclude the adaption of certain fragments of the text that are presented to the user. Fragments can be insertedor removed, altered or dimmed or highlighted based on the user interacting with the system. The followingthree sub requirements are established:

AM-Req 1.1.1: the AM shall provide multimedia adaptations

AM-Req 1.1.2: the AM shall provide natural language adaptations

AM-Req 1.1.3: the AM shall provide canned text adaptations


Claims AM-Req 1.1.1:+ The presented multimedia is understandable for the user.+ The presented multimedia is relevant for the user to view.

Claims AM-Req 1.1.2:+ The presented texts are understandable for the user.

Claims AM-Req 1.1.3:+ The presented information is relevant for the user to view.+ The user does not experience irritations caused by information that is presented more than once within

various knowledge categories on the website.- The user could experience irritations or inconvenience caused by the alterations made to the text.

Two use cases will now be described in which all three of the above defined requirements are encounteredfor mothers with different levels of education.

AM-Use case 1: mother with high level of education visits the knowledge category Probabilities

1. The mother visits the Probabilities knowledge category on the website

2. A graph is shown to the mother indicating the probabilities of contracting HPV

3. The corresponding text on the web page contains information about the statistics described in thegraph and references to articles from which the statistics are obtained

4. Some fragments of text are grey of colour, which indicates that the mother has already seen thisinformation

10

5. A text fragment is inserted on top of the page explaining in which knowledge category the grey outinformation has been read before

AM-Use case 2: mother with low level of education visits the knowledge category Probabilities

1. The mother visits the Probabilities knowledge category on the website

2. A descriptive image is shown to the mother indicating the probabilities of contracting HPV

3. The corresponding text on the web page contains information about the image, where the informationis provided in a descriptive way, i.e. without the use of statistics

4. Some fragments of text are grey of colour, which indicates that the mother has already seen thisinformation

5. A text fragment is inserted on top of the page explaining in which knowledge category the grey outinformation has been read before

The following rules show how the website determines which presentation to use:

if levelOfEducation == "low" thenshow descriptive image probabilities HPV;use descriptive text probabilites HPV;

endelse if levelOfEducation == "high" then

show graph probabilities HPV;use numerical text probabilies HPV;

endif Facts & Fables visited == "true" then

change colour of fragment about Facts & Fables to grey;Insert fragment on top of page “info read in Facts & Fables”;

endAlgorithm 3: Adaptive presentation of the information in knowledge category “Probabilities”

3.3.2 Adaptive navigation support

For the adaptive navigation support, adaptions include direct guidance and adaptive link annotation. Directguidance includes the presence of for example a next button on a web page. This button guides the userdirectly to the next (relevant) page. Adaptive link hiding involves the hiding or showing of links to the userbased on the interaction with the system. The adaptive annotation of links comprises the changing of theappearance of the links, e.g. changing the colour when the user visited the link or enlarging the font sizewhen the user hoovers over the link. The following sub requirements are established:

AM-Req 1.2.1: the AM shall provide direct guidance

AM-Req 1.2.2: the AM shall provide adaptive link hiding

AM-Req 1.2.3: the AM shall provide adaptive link annotation

The corresponding claims are the following (given that the adaptations are performed correctly):

11

Claims AM-Req 1.2.1:+ The user is directed through the information in a logical and relevant order and does not have to decide

on that order herself.- The user could experience irritations as she wants to be able to click on every link that is available on

the website.

Claims AM-Req 1.2.2:+ The user does not encounter information that is not relevant at that point during the interaction, as only

relevant links are presented.- The user could experience irritations as she wants to be able to click on every link that is available on

the website.

Claims AM-Req 1.2.3:+ The user can easily see which information has been visited before, as the appearance of the links is

changed.

Two use cases will be described in which the above mentioned requirements are met for mothers with a lowand a high level of knowledge about HPV:

AM-Use case 3: mother with a low level of knowledge about HPV visits the knowledge category Basic information

1. The mother visits the Basic information knowledge category on the website

2. An easy readable text is shown about the basic information

3. A link is shown via which the mother can read more detailed information

4. The next button returns the mother to the main menu

5. Back in the main menu, the colour of the link has changed, indicating that the category is visited

AM-Use case 4: mother with a high level of knowledge about HPV visits the knowledge category Basic information

1. The mother visits the Basic information knowledge category on the website

2. A detailed text is shown about the basic information

3. The link to the detailed information is hidden, as the detailed variant of the text has been shown to themother

4. The next button returns the mother to the main menu

5. Back in the main menu, the colour of the link has changed, indicating that the category is visited

The following rules show how the website determines which navigation support to use (in this example for

12

the basic information based on the level of knowledge of the mother about HPV):

if degreeOfKnowledge(basicInfo) == "low" thenpresent easy variant basic info about HPV;show link to more detailed info about HPV;

endelse if degreeOfKnowledge(basicInfo) == "high" then

show detailed variant basic info about HPV;hide link to more detailed info about HPV;

endshow next button to main menu;basic info visited == "true";if nextButton == "clicked" then

navigate main menu;change colour link basic info in main menu;

endAlgorithm 4: Adaptive navigation support in knowledge category “Basic information”and main menu

Besides adapting the content of the web pages, the interactions of the available virtual assistant can alsobe adapted. The following section will elaborate on the aspects of the ECAM that allow these adapta-tions.

3.4 Embodied conversational agent model

The ECAM describes under which conditions the ECA performs which actions, i.e. how to adapt its be-haviour to the user (see ECAM-Req 1). The ECA can perform any of the following actions: inform aboutrelevance of the presented material, provide situated feedback about for instance the user’s answers to knowl-edge related questions and explain about the website’s functioning. For each of these actions requirementsare established:

ECAM-Req 1.1: the ECA shall inform the user about the relevance of the presented material

ECAM-Req 1.2: the ECA shall provide situated feedback

ECAM-Req 1.3: the ECA shall explain the user about the website’s functioning


Claims ECAM-Req 1.1:+ The user understands why certain information is shown.- The user could experience irritations because the ECA explains about the relevance, while the user would

like to read the information.

Claims ECAM-Req 1.2:+ The user is will listen to the feedback of the ECA, as it is targeted to that user.+ By providing situated feedback, the user is more likely to understand to the feedback of the ECA, as it is

targeted to that user.

13

- The user could experience irritations because she wants to read the information on the web pages insteadof listening to the spoken feedback.

Claims ECAM-Req 1.3:+ The user understands how the website functions without having to figure that out herself- The user could experience irritations because she wants to figure out how the website works

herself.

3.4.1 Relevance presented material

The ECA can inform the user about the relevance of various types of presented material, i.e. the relevanceof posed questions and provided adaptations. Therefore, three sub requirements are defined:

ECAM-Req 1.1.1: the ECA shall inform the user about the relevance of answering the knowledge re-lated questions

ECAM-Req 1.1.2: the ECA shall inform the user about the relevance of answering the backgroundrelated questions

ECAM-Req 1.1.3: the ECA shall inform the user about the relevance of the adaptations made

Below the corresponding claims are presented:

Claims ECAM-Req 1.1.1:+ The user understands why the questions are posed.- The user could experience irritations because she does not want to listen to the explanation given.

Claims ECAM-Req 1.1.2:+ The user understands why the questions are posed.- The user could experience irritations because she does not want to listen to the explanation given.

Claims ECAM-Req 1.1.3:+ The user understands the adaptations that are presented.+ The user will appreciate the adaptations that are presented.- The user could experience irritations because she does not want to listen to the explanation.

The relevance of the presented material can be indicated to the user in various ways, e.g. by bodily move-ments such as pointing or by speech. At the end of this section a large use case will be described in whichthese requirements are met (in combination with the other ECA requirements).

3.4.2 Situated feedback

The provision of situated feedback by the ECA occurs only in specific situations, for instance after incor-rectly answering a knowledge related question or after reading all available information the system has tooffer. The ECA can then inform the user about the progress made on the website. Furthermore, the ECAcan suggest to the user what information to read next based on the answers provided by the user to questionsposed on the website. To achieve this, the following situated feedback sub requirements are defined:

14

ECAM-Req 1.2.1: the ECA shall provide the user with feedback about the user’s answers to the knowl-edge related questions

ECAM-Req 1.2.2: the ECA shall provide the user with feedback about the user’s answers to the back-ground related questions

ECAM-Req 1.2.3: the ECA shall provide the user with feedback about the progress made on the website

Below the corresponding claims are presented:

Claims ECAM-Req 1.2.1:+ The user understands the presented material.+ The user is more likely to read the presented material, as it is adapted to her.- The user could experience irritations because she does not want to listen to the feedback.

Claims ECAM-Req 1.2.2:+ The user is understands the presented material.+ The user reads the presented material, as it is adapted to her.- The user could experience irritations because she does not want to listen to the feedback.

Claims ECAM-Req 1.2.3:+ The user knows how much of the information is already visited.+ The user knows how much of the information is left to visit.- The user could experience irritations because she does not want to listen to the feedback.

3.4.3 Website functioning

The last interaction that is defined in this model consists of the ECA’s ability to inform the user about thefunctioning of the website. The following sub requirements are defined:

ECAM-Req 1.3.1: the ECA shall explain how the appearance of the knowledge category links changesafter a category has been visited

ECAM-Req 1.3.2: the ECA shall inform the user how the speech of the ECA can be repeated

The corresponding claims will now be presented, after which a larger use case will be described in whichall of the above ECA requirements are met:

Claims ECAM-Req 1.3.1:+ The user understand why the appearance is changed.- The user could experience irritations because she does not want to listen to the explanation.

Claims ECAM-Req 1.3.2:+ The user knows how to repeat the speech of the ECA in the case that it is not heard or understood.

ECAM-Use case 1: from first entrance of website to first knowledge category

1. The mother enters the website

15

2. The web page shows the user two questions. The first about the user’s level of education and thesecond about the amount of information already gathered by the user about the subject.

3. The ECA introduces herself and explains that the mother can repeat the virtual assistants speech bydouble clicking the assistant (ECAM-Req 1.3.2). The ECA asks the user subsequently to answer thequestions and informs the user that the answers to the questions enable the ECA to better guide theuser on the website (ECAM-Req 1.1.2).

4. The mother answers that her level of education is low and that the amount of information gathered islow as well.

5. The mother enters the main menu and the ECA suggests to read the basic information given that shehas not gathered much information before (ECAM-Req 1.2.2).

6. The mother visits the basic information and returns to the main menu

7. The ECA explains that the links that the mother has visited will from this point on look differentlyto indicate what information she has already read. This information is accompanied by a pointingmovement of the hand towards the visited link (ECAM-Req 1.3.1).

8. The mother clicks on the knowledge category “Benefits young age”

9. The web page poses the mother the knowledge related question in order to ascertain the user’s level ofknowledge about why the daughters are vaccinated at this age. The ECA explains that the informationwill be adapted to the mother based on the answers that she provides to the knowledge questions onthe website (ECAM-Req 1.1.1). Furthermore, the website shows that some information has alreadybeen read before and where on the website. In this specific case a fragment about the benefits of theyoung age when receiving the vaccination, which was also presented in the basic information. TheECA explains why the fragment of information is coloured differently (ECAM-Req 1.1.3).

10. The mother thinks that the young age has to do with a lower chance of already having contracted thevirus. As this is correct, the ECA confirms the mother’s answers (ECAM-Req 1.2.1).

11. When the mother returns to the main menu, the ECA informs the mother about her progress byindicating that she has already seen two categories (ECAM-Req 1.2.3).

In the following section the implementation of the proposed submodels on the website will be discussed.

16

4 Implementation of the models

All the aspects of the four different models, i.e. the DM, UM, AD and ECAM, described in Section 3could be used to create the ultimate adaptive system. In this system the information, the navigation andthe interaction of the ECA(s) are all adapted to the user’s needs and current location within the domain.However, during the implementation of such a system, different elements can influence the feasibility of thesystem. Examples of such elements are memory capacity, system speed and the possibilities within the cho-sen programming environment. The following subsections will elaborate on the programming environmentused to create the website to support mothers during decision making about the HPV vaccination and theimplementation of the proposed models.

4.1 TailorBuilder

To create the adaptive website, a programming environment called TailorBuilder (TB)1 is used. TB isan environment which provides an interface to easily create questionnaires by constructing questions andadvice items. These advice items contain information that is to be presented to the user only under certainconditions. Based on the answer of the user to the posed question, an advice item is selected and presented tothe user. By pressing a ”next” or ”previous” button the user can subsequently navigate through the questionsand the information.

4.2 WebSpeaking

The ECAs used within the adaptive system are created by a company called WebSpeaking (WS)2 that pro-vides the movements and speech of the ECAs. In Figure 3 the ECAs are shown which are used for theimplementation: a doctor (left) and a peer mother (right).

Figure 3: The ECAs used within the system: a doctor (left) and a peer mother (right)

1For more information about TB visit: https://www.tailorbuilder.com2For more information about WS visit: http://www.webspeaking.nl/

17

https://www.tailorbuilder.com

http://www.webspeaking.nl/

4.3 A simplified implementation of the models

In order to implement (aspects) of the models proposed in Section 3, a more sophisticated structure thanquestion/advice items would be necessary. This structure would for example allow to store informationabout the user, other than its answers to questions within the questionnaire, such as the clicking behaviourof the user. Based on such information about the user, concepts in the domain can subsequently be adapted,after which the interaction of the ECA can be adapted as well. Unfortunately, it became clear during myresearch that the TB environment (chosen by TNO at the beginning of the project) is not very suitable forprogramming an adaptive website. Because of the limitations offered and the necessity to work around thepredefined TB structures, a simplified version of the proposed models was programmed. Besides limitationsprovided by the programming environment, other requirements could not be implemented yet, because thecontent was not available. No natural language adaptations are performed, as the texts were not availablein different variants of reading difficulty. Also, there were no multimedia adaptations implemented, asthere were no variants of the images which could be used for mothers with different levels of education orknowledge.

The following sub requirements of the AM and the ECA are implemented, as described in Sections 3.3 and3.4:

• AM-Req 1.1.3: the AM shall provide canned text adaptations

Three types of canned text adaptations are implemented within the website, namely the insertion,alteration and highlighting of text fragments. For each of these adaptations, implementation exampleswill be provided below.

1. Fragment insertion - occurs when a user visits a web page on which information is presentedwhich he or she already viewed on another web page. At the top of the page, a small text appearsexplaining that the page contains information that the user has already seen and is therefore pre-sented in a different colour. This way, the user can decide whether or not to read the informationagain or to skip it. For each reoccurring fragment of text on the website, variables are storedthat indicate whether the fragment has already been presented to the user. During loading of theweb page, the system decides to insert the specific fragment or not based on the values of thevariables connected to that web page.

2. Fragment alteration - occurs based on the answers the user provides to the knowledge relatedquestions. The information presented on the web page then consists of feedback that is adaptedto the answer of the user. Given the predefined relations between the possible answers and thecorresponding text fragments, the systems can decide which fragment to show when a certainanswer is provided. The content of the web page is thus altered based on these predefinedrelations and the answer(s) given.

3. Fragment highlighting - occurs based on the answers of the user as well. Fragments are high-lighted in two different ways in the implementation, that is either by changing the colour or byunderlining the text fragment. An example is provided in Figure 4, where the colour of the textfragments changes to green when the question is answered correctly by the user and changed tored when the answer is incorrect.

18

Figure 4: Changing the color of a text fragment based on the answer provided by the user

• AM-Req 1.2.1: the AM shall provide direct guidance, i.e. the mother is guided from one page to thenext by means of “next” and “previous” buttons.

• AM-Req 1.2.3: the AM shall provide adaptive link annotation

Within the website adaptive navigation support is included as well, namely adaptive link annotation.Figure 5 shows the difference in the appearance of the links when unvisited (left) and visited (right).

Figure 5: Adaptive link annotation: appearance visited link (left) and unvisited link (right)

All but one of the ECAM requirements defined in Section 3.4 (ECAM-Req 1.1.1-1.1.3, 1.2.1-1.2.3and 1.3.1) are implemented by calling predefined segments, which contain the speech and movementsof the ECA. These segments are development by WS and are approached by means of Javascript calls.Given values from the UM, such as the amount of information gathered and the location of the useron the website, different segments are called. In ECAM-use case 1, an ECA fragment correspondingto the mother having gathered few information is for example called, where in the case of a large ofamount of information gathered a different segment would be called.

ECAM-Req 1.3.2: the ECA shall explain how the appearance of the knowledge category linkschanges after a category has been visited has not been implemented in the website as such a speechfragment was not yet recorded for the ECA.

As the TB environment does not allow to export code, a few examples of code are include in Ap-pendix A. The adaptive website itself can be found here: https://www.tailorbuilder.com/cgi-bin/runtime_login.pl?grid=49&prid=11&taal=NL. The following section will elaborateon the method of evaluation of the implementation.

19

https://www.tailorbuilder.com/cgi-bin/runtime_login.pl?grid=49&prid=11&taal=NL

https://www.tailorbuilder.com/cgi-bin/runtime_login.pl?grid=49&prid=11&taal=NL

5 Evaluation

In order to evaluate the implementation, two experts at TNO were approached. Dr. W. Otten, who hasextensive experience in the field of informed decision making and the development of decision-aids,and Dr. H.M. van Keulen, who conducted the prior research about the HPV vaccination uptake andhas experience in the field of changing health behaviours among adults.

The experts were asked to visit the website and evaluate the adaptions both on the web pages andthe interactions of the ECAs, i.e. are the adaptions logical and correct from an expert point of view(no user experience evaluation). For this evaluation there were no time restrictions set. To be able tocompare both evaluations, the experts received guidelines that had to be followed when interactingwith the website, see Appendix B. This way, it was ensured that both experts received the same kindof adaptations. Besides receiving the same kind of adaptations, the guidelines were structured suchthat the experts received various kinds of adaptions during the visit on the website.

By following these guidelines, the experts encountered the following specific adaptations:

(a) The suggestion of the ECA to view the basic information after entering the main menu for thefirst time (ECAM-Req 1.2.2: the ECA shall provide the user with feedback about the user’s tothe background related questions→ based on the information gathered from the user by meansof a question that little information was gathered beforehand (UM-Req 1.1.2)).

(b) Feedback about the effectiveness of possible other methods to protect your daughter against HPVand cervical cancer (ECAM-Req 1.2.1: the ECA shall provide the user with feedback about theuser’s answers to the knowledge related questions→ based on the answers to the questions onhow effective the user thinks other methods of protection than the vaccination are to preventagainst cervical cancer (UM-Req 1.1.1)).

(c) A highlighted sentence in the feedback about the effectiveness of having safe sex (AM-Req1.1.3: the AM shall provide canned text adaptations→ based on the answer that safe sex is be-lieved to be more effectiveness to prevent cervical cancer than the vaccination (UM-Req 1.1.1)).

Furthermore, the experts received other adaptations which were not based on the information providedby following the guidelines, but which were based on the actions they choose themselves. Theseadaptations correspond to the requirements described in Section 4.3, such as adaptive link annotationand feedback on the user’s progress.

Before interacting with the website, the experts received a questionnaire, see Appendix C, that con-tained one question about each possible adaptation that could be encountered during the interactionwith the website. This questionnaire had to be filled in during or after the visit on the website. More-over, the experts were asked to write down the order in which they visited the topics on the website,such that this information could be used when analysing the provided answers. Lastly, the expertswere asked to write down an explanation for each answers if they felt that the adaptation was notperformed correctly or optimally. In the next section the results of this evaluation will be presented.

20

6 Results

Each question of the questionnaire consisted of a 5-point Likert scale, where a score of 1 indicates anegative score, 3 neutral and 5 positive. In Table 1 the results are presented for both experts, showingthe Likert scale scores provided for each question, the mean and the difference. Green indicates apositive mean, i.e. a score above 3, white a neutral mean, i.e. a score of 3, and red a negative score,i.e. a mean below 3. Blue indicates a difference of 2 between the scores of the experts. The results ofquestion 3 can not be used, as expert 1 did not give a score, because the question was not understood.

Actor QuestionScore

expert 1Score

expert 2 Mean DifferenceECA 1: Feedback KQ 5 3 4 2

2: Suggestions KQ 4 3 3.5 13: Suggestions BQ - 4 - -4: Suggestions info visited 3 4 3.5 15: Inform about progress 2 3 2.5 1

Web pages 6: Feedback KQ 4 5 4.5 17: Repeated info 3 5 4 28: Back-references 4 2 3 29: Inform about progress 5 4 4.5 1

Table 1: Results questionnaire expert evaluation implementation

For each question, the experts also provided notes to elaborate on the scores they provided on theLikert scales. Below, these notes will be presented (translated from Dutch), where the number of theenumeration corresponds to the number of the question of which the notes are presented:

1. Expert 1 indicats that there is a nice connection between the feedback provided by the ECA andthe answers given to the knowledge related questions. However, the ECA’s speech was too fastand there was no possibility to repeat what had been said. The ECA’s text is too long, thereforethe tendency arose to read the text on the web page, which subsequently led to missing what theECA had to say. The movements of the ECA on the other hand were pleasant and supportedwell what was said. Expert 2 noticed that the speech of the doctor ECA was rather controlling.Furthermore, the doctor indicated that she would check the experts’s answers to the knowledgequestion within the “Andere manieren om te beschermen” section, which was experienced astaking an exam. According to expert 2, the ECA should help the mother, not act controlling.

2. The notes indicate that expert 1 only encountered such suggestions twice, which was experiencedas nice. Expert 2, on the other hand, indicates that such suggestions were not encountered at all.

3. Expert 1 indicated that the question was not understood. Expert 2 noted that the provided sug-gestions of the ECA correspond quite well with the answers given to the background relatedquestions, i.e. about the information gathered so far, level of education and the intention whetheror not to get the daughter vaccinated. A suggestion is done to view the practical information orthe general information in a pleasant manner. When entering the main menu, a suggestion isdone based on the information that few information was gathered about the topic beforehand.Expert 2 wonders what happens with the information about the level of education.

21

4. Expert 1 indicated that only 2 suggestions were encountered and that therefore the answer is notreally applicable. Expert 2 reported that it was not clear why the suggestion to view “Protectionand how does it work” after reading the basic information was made. Furthermore, the sugges-tion is made to consider the advantages and disadvantages of the vaccination after viewing halfof the available components on the website. Expert 2 indicates that the wording for this sugges-tion is a little strange. It was pleasant that the ECA encouraged at that moment. According toexpert 2, the ECA can appear more often within the main menu.

5. Expert 1 reported that no information about the progress was received. Expert 2 indicated thatthe ECA explained how the progress is indicated the first time the main menu is visited. Thatwas good. After that the ECA was observed once more, as soon as half of the components ofthe main menu have been visited. Expert 2 points out that the ECA is providing few informationabout the progress made, that can be done more.

6. Expert 1 reported that the layout of the information could be improved and the amount of textreduced. Scrolling should be avoided as much as possible. Expert 2 indicates that this is donewell within the following components: “Facts & Fables”, “Other methods of prevention” and“Probabilities”, however too patronising (especially with the “Probabilities” component).

7. Expert 1 indicated that the grey text is not convenient. It does not matter if text is repeated. Thatcan be nice sometimes, because it can be placed within different contexts. Expert 1 recommendsto leave out the grey colour change, but to keep the text. Expert 2 experienced difficulties toassess this adaptation. “Was that the reason I saw grey text within the “Probabilities” compo-nent?”. Expert 2 indicates no text had been presented more than once and therefore assumedthat the adaptation was successful.

8. Expert 1 reported that a back-reference had been encountered once, which was fine, but person-ally this did not have to be done by means of grey text. Expert 2 indicated that this was notencounter at all and posed the question whether this involved the grey text presented within the“Kans” component.

9. Expert 1 indicated that this was nicely done, only different colours would be preferred (samecolours as logo). Expert 2 reported that if this question considers the main menu that it is allclear then. However, if the question considers the progress bar, then it is less clear. That baris confusing as it seems that it indicates the progress per component, while the layout of thebar remains the same. A suggestion is made to either use different colours or explain how theprogress bars work. Furthermore, expert 2 reported that it is unclear which component is beingviewed, as there is no heading indicating which component is currently viewed. Maybe the textabout the website being a decision aid (banner on top of web page) could change per componentindicating which component is currently viewed. The progress bar might also be more clearthen.

In the next section the notable results will be discussed, i.e. the questions were the difference is 2apart or the mean is 3 or below 3.

22

7 Discussion

Overall the adaptations appear to be performed correctly, as most of the means are 3.5 or higherand the experts for most questions do not differ more than 1 point in score. Below the results fromquestion 1, 5, 7 and 8 will be discussed, as these results had a mean of 3 or lower, a difference of 2 ora combination of both.

Question 1: a difference of 2 between the expert’s scoresThe explanations provided by the experts about the first type of adaptation, i.e. the connection betweenthe feedback and the answers given to the knowledge related questions, indicate that expert 2 foundthe texts of the ECA too controlling and directive. No remarks were made about a wrong connectionand therefore it can be concluded that the score is based on remark about the content of the speech,not the adaptation itself.

Question 4: a mean of 2.5From the explanations provided by the experts it can be concluded that the progress adaptation of theECA was not or barely observed by both experts. The progress is thus provided to the user in a toosubtle way and should be made more explicit or applied more often.

Question 7: a difference of 2 between the expert’s scoresFrom the comments of the experts, it appears that expert 1 noticed that the repeated text’s font changedcolour, but that no added value was experienced. Expert 2 appears not to have noticed the adaptations,as she indicates that no information has been observed twice. As the guidelines were followed by theexperts, this adaptation was present and thus not noticed by expert 2. Therefore it can be concludedthat the use of grey text to indicate that information has been seen before, is not a convenient way toindicate this. Instead of notifying the user about this, the text could either be removed or not changedof colour. As expert 1 indicates “it does not matter if text is repeated”.

Question 8: a difference of 2 between the expert’s scores and a mean of 3The explanations provided by the experts show that the same observation holds as for the results ofquestion 7. The canned text adaptation of inserting an extra fragment to indicate where the grey texthas been read before, i.e. a back-reference, is not experienced as convenient. Expert 1 noticed theadaptation, but did not see the added value and expert 2 did not notice the adaptation. This adaptationshould either be made more explicit or not performed at all.

In the next section this thesis will be concluded and recommendations are made for future work.

23

8 Conclusion and future work

For the design and implementation of a model underlying an adaptive website to support mothersduring decision-making about whether or not the have their daughter vaccinated against HPV, thefollowing submodels were defined: DM, UM, AM and ECAM. Each of these models were defined interms of requirements and claims as to why these requirements should hold.

From the expert evaluation it can be concluded that the implemented requirements of the AM andthe ECAM overall performed properly. It can indirectly also be concluded that the requirementsof the DM and UM have been implemented appropriately, as the AM and the ECAM are based onthis information. Some results, however, indicate a need for further research. The progress shouldbe made more explicit by the ECA in a next implementation and the use of grey text and back-references should be investigated further. Is the use of this canned text adaption necessary or shouldthe same information be presented twice in different contexts? Besides improvements of the currentimplementation, extra adaptations, such as natural language adaptations or multimedia adaptations,could be added to a next implementation. To enable this, different versions the existing texts andimages need to be created. Most importantly, a user evaluation will have to be conducted once theimplementation has been refined in order to evaluate the user experience of the website. Duringthis evaluation the provided claims for the user experience, e.g. the user understands the presentedinformation, can be verified. In the case that some claims do not hold for certain requirements,then these requirements need to be implemented differently in a next development cycle. After theserefinements, the new implementation should be evaluated again to see if the stated claims are fulfilled.These cycles should be repeated until all the claims hold for the proposed requirements.

24

References

Robert K Atkinson. Optimizing learning from examples using animated pedagogical agents. Journalof Educational Psychology, 94(2):416, 2002.

Matthew P Banegas, Jennifer B McClure, William E Barlow, Peter A Ubel, Dylan M Smith, Brian JZikmund-Fisher, Sarah M Greene, and Angela Fagerlin. Results from a randomized trial of aweb-based, tailored decision aid for women at high risk for breast cancer. Patient education andcounseling, 91(3):364–371, 2013.

Robbert-Jan Beun, Eveliene De Vos, and Cilia Witteman. Embodied conversational agents: effectson memory performance and anthropomorphisation. In Intelligent virtual agents, pages 315–319.Springer, 2003.

Timothy Bickmore and Justine Cassell. Relational agents: a model and implementation of buildinguser trust. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages396–403. ACM, 2001.

Peter Brusilovsky. Methods and techniques of adaptive hypermedia. User modeling and user-adaptedinteraction, 6(2-3):87–129, 1996.

Peter Brusilovsky. Adaptive hypermedia. User modeling and user-adapted interaction, 11(1-2):87–110, 2001.

Justine Cassell, Tim Bickmore, Lee Campbell, and Hannes Vilhjálmsson. Designing embodied con-versational agents. Embodied conversational agents, page 29, 2000.

E Ensing, M Steen, R Laferte, I Zonnenberg, and P De Hoogh. Opkomst hpv-vaccinaties per 31december 2010. deel ii van de rapportage: de cijfers, 2011.

Nicole PM Ezendam, Johannes Brug, and Anke Oenema. Evaluation of the web-based computer-tailored fataintphat intervention to promote energy balance among adolescents: results from aschool cluster randomized trial. Archives of pediatrics & adolescent medicine, 166(3):248–255,2012.

Jacques Ferlay, Hai-Rim Shin, Freddie Bray, David Forman, Colin Mathers, and Donald MaxwellParkin. Estimates of worldwide burden of cancer in 2008: Globocan 2008. International journalof cancer, 127(12):2893–2917, 2010.

Gerhard Fischer. User modeling in human–computer interaction. User modeling and user-adaptedinteraction, 11(1-2):65–86, 2001.

Stijn Friederichs, Catherine Bolman, Anke Oenema, Janneke Guyaux, and Lilian Lechner. Moti-vational interviewing in a web-based physical activity intervention with an avatar: Randomizedcontrolled trial. Journal of medical Internet research, 16(2), 2014.

Gezondheidsraad. Vaccinatie tegen baarmoederhalskanker. Den Haag: Gezondheidsraad, 2008.

Eva Hudlicka. Virtual training and coaching of health behavior: Example from mindfulness medita-tion training. Patient education and counseling, 92(2):160–166, 2013.

25

Integraal kankercentrum Nederland. Nederlandse kankerregistratie, June 2014. URL http://www.cijfersoverkanker.nl/.

Evgeny Knutov, Paul De Bra, and Mykola Pechenizkiy. Ah 12 years later: a comprehensive survey ofadaptive hypermedia methods and techniques. New Review of Hypermedia and Multimedia, 15(1):5–38, 2009.

Nubia Munoz, F Xavier Bosch, Xavier Castellsagué, Mireia Díaz, Silvia de Sanjose, Doudja Ham-mouda, Keerti V Shah, and Chris JLM Meijer. Against which human papillomavirus types shallwe vaccinate and screen? the international perspective. International Journal of Cancer, 111(2):278–285, 2004.

Mark A Neerincx and Jasper Lindenberg. Situated cognitive engineering for complex task environ-ments. Naturalistic decision making and macrocognition, pages 373–390, 2008.

Rinat B Rosenberg-Kima, Amy L Baylor, E Ashby Plant, and Celeste E Doerr. The importanceof interface agent visual presence: Voice alone is less effective in impacting young womenâAZsattitudes toward engineering. In Persuasive Technology, pages 214–222. Springer, 2007.

Daniela N Schulz, Math JJM Candel, Stef PJ Kremers, Dominique A Reinwand, Astrid Jander, andHein de Vries. Effects of a web-based tailored intervention to reduce alcohol consumption in adults:Randomized controlled trial. Journal of medical Internet research, 15(9), 2013.

HM Van Keulen, M Fekkes, W Otten, S Van der Pal, P Kocken, R Ruiter, and TGWM Paulussen.Onderzoek naar de hpv-vaccinatiebereidheid bij moeders en dochters naar aanleiding van de in-haalcampagne in nederland. A study into willingness of mothers and daughters to be vaccinatedagainst HPV within the framework of the catch-up campaign in the Netherlands. Report KvL/GB,2010a.

HM Van Keulen, M Fekkes, and TGWM Paulussen. Factsheet tevredenheidsonderzoek hpv-vaccinatiecampagne 2010. Leiden: TNO Kwaliteit van Leven, 2010b.

EA van Lier, PJ Oomen, MWM Oostenbrug, SLN Zwakhals, IH Drijfhout, PAAM de Hoogh, andHE de Melker. Vaccinatiegraad rijksvaccinatieprogramma nederland: verslagjaar 2009.[immunisa-tion coverage national immunisation programme in the netherlands; year of report 2009]. Bilthoven,the Netherlands: RIVM, 210021010, 2009.

26

http://www.cijfersoverkanker.nl/

http://www.cijfersoverkanker.nl/

Appendices

A Code examples

The javascript call to ECA fragment

<script type="text/javascript" src="//d1v5fcs94ctw3p.cloudfront.net/accounts/248/cdm-director.js?CodeBaby=page:Start-Adviseur;"></script><script>CodeBaby.start = function() {

CodeBaby.conversation.trigger(’Cn_Adv_1’);};

</script>

Show advice items given a conditionIn this example, show the appropriate feedback, call the corresponding ECA fragment (above men-tioned Javascript is situated in [AV_RE_VEILIG]) and show the appropriate graph given that the userindicated that the practising safe sex is more effective to prevent cervical cancer than the vaccination.

IF (VR_RE_VEILIG GE VR_RE_INENTING){

[AV_RE_VEILIG];[ADI_RE_VEILIG];[GR_RE_VEILIG];

}

Change the colour of a fragment of text by means of HTML and JavascriptWhen the text fragment below "leeftijdHPV" has been visited before, the colour of the text is changed.

<span id="leeftijdHPV">Het is het beste om de HPV-inenting op jonge leeftijd te doen, als een meisjenog niet seksueel actief is. Vandaar de leeftijd van 12 jaar. Het heeft geen zin om een HPV-inentingte halen als je eenmaal besmet bent met HPV. Door de inenting op 12-jarige leeftijd te doen, is eenmeisje al beschermd als zij wÃl’l seksueel actief wordt.</span>

if(localStorage && localStorage.getItem(’leeftijdHPV’) == ’waar’){document.getElementById("leeftijdHPV").style.color="#A4A4A4";

}

27

B Expert evaluation guidelines

1. Visit the website as if you are a mother who is about to make the decision whether or not to lether daughter get vaccinated against HPV

2. When asked how much information you have already gathered about HPV and the vaccination,indicate that you have gathered little information so far. After providing this answer, the vir-tual assistant will offer the possibility to view general or practical information, select practicalinformation

3. Within the Other methods of protection component indicate that you think that every method toprotect your daughter against HPV and cervical cancer is more efficient than the HPV vaccina-tion

4. Follow the suggestions made by the virtual assistant when interacting with the website

5. Aside from the guidelines mentioned above, you are free to navigate through the website in amanner that fits you best

28

Expert evaluatie van de website

U gaat zo de keuzehulp inenten tegen baarmoederhalskanker doorlopen. Graag wil ik u vragen om de

website te doorlopen alsof u een moeder bent die weinig informatie heeft opzocht over HPV en de

inenting. Dit betekent dat u de vraag over hoeveel informatie u al heeft opgezocht kunt beantwoorden met

weinig. Daarna kunt u klikken op Algemene informatie, indien u de vraag over de intentie tot inenten met

ja beantwoordt. Binnen het onderdeel “manieren om te beschermen” bent u van mening dat alle andere

manieren om te beschermen effectiever zijn dan de inenting. Ook wil ik u vragen de adviezen van de

virtuele assistent op te volgen. Verder bent u vrij om door de website te navigeren zoals u zelf wilt.

Na elke onderstaande vraag is er ruimte om notities te maken. Ik wil u verzoeken om van deze ruimte

gebruik te maken en zoveel mogelijk het proces van het doorlopen van de website te noteren. Helemaal

achterop de vragenlijst is meer open ruimte te vinden. De vragenlijst mag tijdens of na het doorlopen van

de website ingevuld worden. Maak hierbij gebruik van de notities die u gemaakt heeft tijdens het

doorlopen van de website. Ook wil ik u vragen de volgorde van de onderdelen op te schrijven, zoals u

door de website gelopen bent. Als u op extra informatie klikt, dan dit ook graag noteren.

Het doel van deze evaluatie is het evalueren van de adaptaties op de website. Indien verkeerde informatie

wordt getoond, zoals bijvoorbeeld verkeerde feedback op een antwoord, dan is het van belang dat u dit

opschrijft. Deze evaluatie is echter niet bedoeld om op de inhoud van de teksten in te gaan.

Virtuele assistent

1. De feedback van de virtuele assistent sluit goed aan op mijn antwoorden op de kennisvragen binnen de

websiteonderdelen …

helemaal mee oneens ☐ ☐ ☐ ☐ ☐ helemaal mee eens

Ruimte voor notities. Indien u de feedback van de virtuele assistent niet goed vond aansluiten, kunt u dan

aangeven op welke plekken op de website daar sprake van was?

Click here to enter text.

2. De suggesties van de virtuele assistent om bepaalde informatie te bekijken op de website sluiten goed

aan op mijn antwoorden op de kennisvragen binnen de websiteonderdelen ……


Ruimte voor notities. Indien u de suggesties van de virtuele assistent niet goed vond aansluiten, kunt u

dan aangeven op welke plekken op de website daar sprake van was?



aan op mijn antwoorden op de vragen over mijn achtergrond informatie…





C Questionnaire expert evaluation (in Dutch)

29


aan op de informatie die ik eerder al heb bekeken…





5. De virtuele assistent geeft mijn voortgang op de website goed aan…


Ruimte voor notities. Indien u vond dat de virtuele assistent uw voortgang niet goed aangaf, kunt u dan



Informatievoorziening

6. De feedback op de websitepagina’s sluit goed aan op mijn antwoorden op de kennisvragen binnen de

websiteonderdelen …


Ruimte voor notities. Indien u de feedback op de websitepagina’s niet goed vond aansluiten, kunt u dan



7. De website past de informatie op de pagina’s zodanig aan dat ik informatie niet dubbel hoef te lezen…


Ruimte voor notities. Indien u vond dat de website de informatie niet zodanig aanpaste dat u informatie

niet dubbel hoeft te lezen, kunt u dan aangeven op welke plekken op de website daar sprake van was?


8. Op de pagina’s op de website wordt duidelijk terugverwezen naar eerder gelezen informatie…


Ruimte voor notities. Indien u vond dat de website u niet duidelijk terugverwees naar eerder gelezen

informatie, kunt u dan aangeven op welke plekken op de website daar sprake van was?


9. De website laat mij duidelijk zien welke onderdelen in het menu ik reeds heb bekeken…

helemaal mee oneens ☐ ☐ ☐ ☐ ☐ helemaal eens

30

Documents

This site ﬁts YOU mom! - UvA...This site ﬁts YOU mom! The design and implementation of a model underlying an adaptive website to support mothers during decision-making about whether