An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to

Organizational Websites

Cécile Paris

Anne-Marie Vercoustre Stephen Wan PeterWilkins

Ross Wilkinson Mingfang Wu

Outline

• Problem statement• System Description• Experiment• Discussion

Finding Information about a Corporation

• Often done through the web, searching and browsing

• Differences between results from such search-and-browse and printed brochures:– Printed brochures are coherent documents – and include

more than just information about a specific topic

– Printed brochures are typically tailored to an audience

Challenges in accessing an organization’s website

• Browse– know the organization’s structure

– know the web data structure

– otherwise -> users feel lost

• Search– It works for the “right” query or someone with the “right” mental

model how querying/searching works

– Otherwise, it may cause zero results or too many results

– Result lists is disorganized - each result is presented independently, although each result may be a piece of the overall picture

Our research question and approach

Can a tailored hypermedia system offering tailored and coherent information provide a good alternative to search-and-browse?

• Approach: Given a modest user model and a query:– Present information that not only matches the query but also is tailored

to the user.

– Present other information that is relevant to the context of the query (here information about an organization), like printed brochures do (e.g., introduction, contact details, etc.)

– Deliver the result as a concise and readable summary.

– Note that the summary may not cover all information a user needs, but is an informative guide that will lead the user to the needed information.

Our Tailor Hypermedia System: Percy

Dat

a In

terf

ace

User Model

Stereotype

Individual

Virtual Document Planner

DiscourseRules

Content Planner

DataSources

‘Documents’

VD

P D

ata

Inte

rfac

e

Inte

rfac

e S

peci

ficat

ion

Grammar(s)Lexicon(s)TemplatesStylesheets

SurfaceRealiser

PresentationRules

PresentationPlanner

Percy – user model

• Based on a requirements analysis• Contains:

– Stereotypical information• Current job position (e.g., scientist or CEO)

• Industry sector (e.g. science, engineering, finance)

– Individual information• Interests (as represented by a query term, e.g., language

technology)

• Preferred delivery channel (e.g web page, or hand-held device)

Percy – discourse planner

• Discourse rules or Schemas– They indicate which information to

present and how to organize it– They are used by the discourse

planner to build a discourse tree– Each rule extends a branch in the discourse tree– Coherence relations amongst

siblings indicate how the information fits together

Percy – presentation planner

• Discourse tree (which indicates content and organization) is extended with presentation markup

• Uses presentation plans

PPalm

C

PWeb

C

Percy – surface generator

• Traversal of final discourse tree to generate document

• Conversion from tree markup of layout to device dependent markup– Device dependent markup may

include use of further style sheetsSurface Realisation

XSLT templates

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

PWeb

C

PERCY – an example delivery to a scientistIntroduction to CSIRO

Short introduction to the divisions that have the projects

related to the query

Show the expanded query

Short description of the

matched project

PERCY – an example delivery to a scientist Link to the detailed description

Link to the onlinedemonstration

Publications

Contact person

PERCY – an example delivery to a CEO

Short introduction of how to do business with CSIRO

PERCY – an example delivery to a CEO

The project description and business relevance

Preliminary Evaluation

• Question Is this coherent and tailored delivery is useful interface to

organizational websites ?

Possible evaluation criteria

• User’s interactive experience– format

– content

• User’s attitudes towards the received message– intention

• The influence of the delivery message– The further interaction with the corporate

Experimental setting

• Test systems: PERCY vs. search-and-browse mechanism: a website search engine (Panoptic)

• Subjects: 20 university students• Search topics:

– T1: Human computer interaction

– T2: Mathematical modeling

– T3: Image analysis

– T4: Language technology

Evaluation instruments

• Post-system questionnaire (PSQ)– To get users’ opinion on the evaluation criteria

independently on each test system

• Exit questionnaire (EQ)– To get users’ opinion of the second system compared with

the first system

• All questions are evaluated on a 5-point Likert scale

Experiment design

• Latin square design to reduce the sequence effect of systems

Subjects System, topic

No. 1-5 Percy(t1,t2) PSQ Search (t3,t4) PSQ EQ

No. 6-10 Percy (t3,t4) PSQ Search (t1,t2) PSQ EQ

No. 11-15 Search(t1,t2) PSQ Percy (t3,t4) PSQ EQ

No. 16-20 Search (t3,t4) PSQ Percy (t1,t2) PSQ EQ

Post-system questionnaire – Measuring the content of the delivered information

• Q1: The system provides me accurate information

• Q2: The system provides sufficient information

• Q3: The information provided by the system meets my need.

• Q4: The system provides me comprehensive information

Q1 Q2 Q3 Q4

Search 3.50 3.60 3.35 3.70

Percy 3.65 3.30 3.65 3.55

Post-system questionnaire – Measuring the format of the delivered information

• Q5: The structure of the presented information is clear to me.

• Q6: I think the presented information is organized in a useful format.

• Q7: I think the presented information serves well as a useful online brochure.

Q5 Q6 Q7

Search 3.65 3.50 2.75

Percy 4.00 3.80 3.50

Post-system questionnaire – Understanding the intention of the user

• Q8: I would like to get more information on some specific projects presented.

• Q9: I would like to get more information because I have not got the needed information.

Q8 Q9

Search 3.95 3.15

Percy 3.90 2.90

Exit-questionnaire – Evaluating users’ preferences

Q1 Q2 Q3 Q4 Q5 Q6 Q7

Search 2.70 3.00 3.10 2.80 3.30 3.50 2.70

Percy 3.70 3.60 3.50 4.20 4.20 3.40 4.00

P < 0.01 0.03 0.04

focus o

n the s

earch

topic bett

er

make s

earch

task ea

sier

deliver

higher quali

ty info

attrac

t atte

ntion bett

er

provide bett

er ex

planati

on

provide more

compreh

ensiv

e info

prefere

nce

Summary of experimental results

• With Percy, subjects can better focus on their tasks.• With Percy, subjects understand better why a piece

of information is presented.• Subjects preferred Percy.

Discussion

• About tailored hypermedia(anything to say about the delivery method itself or limitations?)

– Delivery – who is in charge, user/author? or it is situation dependent? – How to apply AH to a dynamic resource (or even a different domain)

with minimum effort?– Is user’s expectation of AH system different from IR?

• About the evaluation– What to compare – Is it fair to compare the adaptive system with the

non-adaptive system? Or should we compare two adaptive systems? – Will the preference brings the enhancement of user performance? – (How to measure long term benefit?)