User-informed design for search and discovery in digital ... · • Categorising users of digital cultural heritage • Understanding what users search for and why • Considering

User-informed design for search and discovery in digital cultural heritage

Paul Clough

Information School, University of Sheffield

UNESCO-NDL workshop, New Delhi, 25-27 October 2017 Page 1

Outline

• Brief introduction• Categorising users of digital cultural

heritage• Understanding what users search for

and why• Considering how knowledge about users

can inform system design and evaluation activities


“So what use are the digital libraries, if all they do is put digitally unusable information on the web?”Christine Borgman

Source: http://www.digitalhumanities.org/dhq/vol/3/4/000077/000077.html


“We are proposing that it is necessary to look at the users of digital libraries in a more analytic way than is typical in studies of digital library usability. Digital library users, like those categorized by Marchionini, Plaisant and Komlodi, are not a homogenous group with a limited set of interests and opinions. If we are to design digital libraries that meet the needs of diverse user communities, we need to understand what those needs and values are and to incorporate the consequences of those user differences in our collections, interfaces, and access tools”Lorraine Normore


“Cultural heritage involves rich and highly heterogeneous collections that are challenging to archive and convey to the general public”

How do we get users in?

How do we keep users in?


Search and discovery in context


User differences• Age• Gender• Technical

expertise• Domain

knowledge• Cognitive styles• …

Plus wider context• Task and need• Goal• Device• Location• Time• …

Will impact on• Information

seeking behaviour

• User preferences

• Notions of success

• …

User-informed approaches to design

• Clearly understanding users’ information needs, their behaviours and the tasks they engage in is critical in developing systems that support good information access and interaction – “Research into user needs, tasks and resources is required

before the design can begin” (Allen, 1996:291). – “Know thy user” (most UX/HCI people)

• User-centred design techniques aid– Design and development– Adaptation and personalisation– Evaluation

• Information systems and services typically designed for user categories (vs. individuals) and usage situations


Understanding Users and their

Environment

Development and

Implementation

Monitoring and Improvement

Build an interactive version

Identify needs /

Establishrequirement

s

(Re) Design

Evaluate

FINAL PRODUC

T


Agile

Data collection:• Use of

behavioural data (e.g. logs)

• Ask them directly (surveys, interviews, diary studies, focus groups, etc.)

• Review of past literature

• Combinations

On-going

Understanding users: Characterising and classifying users


Walsh, D., Clough, P., and Foster, J. (2016) User Categories for Digital Cultural Heritage, In Proceedings of the First International Workshop on Accessing Cultural Heritage at Scale. Available online: http://ceur-ws.org/Vol-1611/paper6.pdf

Walsh D., Hall M., Clough P., Foster J. (2017) The Ghost in the Museum Website: Investigating the General Public's Interactions with Museum Websites. In: Kamps J., Tsakonas G., Manolopoulos Y., Iliadis L., Karydis I. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2017. Lecture Notes in Computer Science, vol 10450. Springer. Available online: https://repository.edgehill.ac.uk/9473/

General categories of user

• Russell-Rose & Tate (2012) describe general categories of user - novices and experts– Categories based on the user’s domain and technical

expertise– Shown to affect how users seek information and the

kinds of support they may require

• Frequent query reformulation

• Often click back from result to search results

• Spend longer on search tasks

• More pages examined

• Often follow more links and go deeper

• Spend shorter time on search tasks

Experience over time can turn novices into experts UNESCO-NDL workshop, New Delhi, 25-27 October 2017

Page 10

User studies in digital cultural heritage

• Many articles within (digital) cultural heritage discuss users in varying levels of detail– Often categorised into generic groups

(stereotypes) and described with personas– Groups often based on dimensions such as

• Expertise (e.g. expert vs. novice), • Profession/role (e.g. student, historian) • Motivation/goal (e.g. explorer, casual user)

– Often particular tasks or situations are also described as scenarios or use cases


Example persona – UK National Archives


Characteristics or dimensions to help distinguish types of user

Case study: review of user groups

• Conducted systematic review of previous studies in (digital) cultural heritage– Focused on studies where groups

identified by names and described with characteristics

• Broad categories of user– Professionals and experts– Semi-experts and hobbyists– Lay users, novices and non-experts

• In total 58 distinct names– Re-grouped based on broad levels of

technical and domain expertise (low and high) and motivation

NoviceExpertProfessionalMuseum Information ProfessionalLay userNon-expertHobbyistNon-professionalCasual Leisure UserGeneral VisistorsEducational VisitorsSpecialist VisitorsStaffHobbyistsScholarsProfessional ResearchersRummagersObject seekersSurfersTeachers K-16Students K-16Apprentice InvestigatorsInformed UsersGeneral Public....


http://ceur-ws.org/Vol-1611/paper6.pdf

Case study: users of museum website

Search collections

Information about physical museums (e.g. opening times)

Purchase items from museum shop


Study outline

• Goals of study included determining the frequency of user groups, their defining characteristics and differences between general public and non-professional

• Conducted online (pop-up) survey available for 4 weeks to users of NML website– Asked users to identify with one or more pre-defined groups– Users rated against 8 dimensions (e.g. domain knowledge,

purpose of visit, motivation and frequency of visit)– Also gathered information about search tasks and support

• After filtering dataset left with 564 responses– General public and non-professional groups make up ~70% of

the responses (the ‘bounce’ groups)

• Performed statistical analysis to identify similar groups and their distinguishing characteristics


Group Pre-merge

Merged

General Public 253 253

Non-professional 89 137

Non-professional/General Public

48 -

Student 33 33

Other 26 26

Teacher 18 25

Academic 16 25

Museum Staff 10 10

Academic/Teacher 9 -

Non-professional/Teacher/ General Public

7 -

Gender61% female, 37% male

Age20% 18-34, 36% 35-54, 26% 55-64, 15% 65-74

Education34% degree-level, 24% further education, 21% masters-level, 11% secondary school

Employment37% full-time, 22% retired, 19% part-time, 10% students

Location35% Liverpool/Merseyside, 23% north-west England, 18% rest of England, 18% world-wide

Low 2 3 4 High0

0.10.20.30.40.50.60.7

General Public Non-professional StudentAcademic Teacher Museum Staff


Domain expertise

Defining the user groups

General Public

• Personal / Pass Time

• Pre-Visit / Collections

• Novice / Intermediate

• Local

Non-professionals

• Personal• Collections• Intermediate• Local / Distant

Students

• Study• Novice /

Intermediate

Academics

• Study / Work• Collections• Intermediate /

Expert• Distant

Teachers

• Pre-Visit• Intermediate• Local

Museum Staff

• Work• Intermediate /

Expert• Local


Understanding users:What do people search for, how and why?


Clough P., Hill T., Paramita M.L., Goodale P. (2017) Europeana: What Users Search for and Why. In: Kamps J., Tsakonas G., Manolopoulos Y., Iliadis L., Karydis I. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2017. Lecture Notes in Computer Science, vol 10450. Springer. Jesper, S., Clough, P. and Hall, M. (2013) Regional Effects on Query Reformulation Patterns, In Proceedings of The International Conference on Theory and Practice of Digital Libraries (TPDL 2013), pp. 382-385.

Transaction log data• Common for many online systems to record interactions

between people using the system and responses from the system

• Offers potentially valuable behavioural and usage information for a wide range of applications – e.g. web usage, marketing, search analytics etc.

• Presents lots of issues and challenges– Can be used to study patterns of user behaviour – tells you what people

do (and how), but not why

– Potential issues around ethics and privacy (e.g. consent)

– Needs to be complemented with more in-depth analysis and understanding of context

– Identifying non-human traffic and ‘real’ queries can be hard

– Distinguish between site search vs. referrals


Behavioural data - Google Analytics


Case study: UK National Archives


Study outline

• Analysis of 10 years of web server logs including Discovery, bookshop, website, databases etc.– Usual web and search analytics analyses, e.g. session

length, popular queries, referrals, – Further analysis included identification of named

entities and prominent people, analysis of questions, analysis of queries to specific databases

• Analysis of logs from UK Government Web Archive– Included analysis of downloads and URLs

• Further manual studies carried out– Analysis of subject content of queries– Strategies for query reformulation across countries





Jesper, S., Clough, P. and Hall, M. (2013) Regional Effects on Query Reformulation Patterns, In Proceedings of The International Conference on Theory and Practice of Digital Libraries (TPDL 2013), pp. 382-385.

Case study: Europeana


Study outline

• Goals of study: To better understand what information people were searching for and why for their current visit– Use of pop-up survey to gather information about (real) user’s

current search activities (e.g. goals, tasks, uses, etc.) – Information about their profile (including category of user)

and feedback about experiences with Europeana

• Ran survey for 2 weeks with 240 complete responses from sample of users that generally represent Europeana users

• Performed various analyses of the responses– Classify type of search task (e.g. known-item, topical)– Categorised subject content of search tasks– Categorised types of motivation and information use


Download the data: http://bit.ly/europeanaSearchTasks


Academic (e.g., lecturer, professor, post doc researcher, academic support)Cultural heritage enthusiast (e.g., hobbyist, genealogist, amateur historian)Cultural heritage professional (e.g., curator, historian, archivist)Student (e.g., college, university, further education)Teacher (e.g., primary and secondary teaching)

What information are you looking for right now?

Why are you looking for this information?

Type Search Task

"Boletín Oficial de Instrucción Pública" To prepare teaching material. Known-item searchI'm studying series of prints regarding the life of Saint Francis of Assisi

PHD Specific-subject search

siblings Ancestor General topical searchHistory of British India. To write an article Specific-subject searchI am looking for photographs of The Trachian tomb near to village of Mezek, Bulgaria.

To write a article. Specific-subject search

Historical images General interest General topical searchPictures of famous biographers for an academic journal General topical searchI am looking for Great War photographs that taken on exactly 100 years ago

To share the photos on social media Specific-subject search

Documents on the medieval family "de Germiny" who was living in the département of Vosges (France).

Because the ancestors of Louis XIV king of France are described differently according to different authors and I want to make up my own opinion.

Specific-subject search

Nothing specific. Just exploring. Doing a little how-to, to teach others how to use Europeana easily (a library worker just trying to make everything even more easier to readers).

Browse/Explore

I want to find information about old routes/path in the SW of Spain

To write an article General topical search

I want working women images and videos from the XX century.

To create a video about women and Europe during the XX century

General topical search

Catalan history Personal interest learning about catalan history


an old journal (Archiv für slavische Philologie)

to write an article Known-item search

I am trying to explore images of objects and monuments from ancient Italy and the Roman Empire.

I am a librarian teaching a session for students in an Art & Archaeology of Ancient Italy class at a university. They need to find an object or monument that has not been covered in class to write a research paper.



Search Tasks

Definitions Examples

Specific-item search

Search for specific item (i.e., known-item) typically expressed precisely

“I am looking for the 1919 film ‘Les fetes de la victorie’.”

11.3%

By named author

Search for information by a specific named author or provider

“to look for paintings by Henriette Ronner”, “I am searching for … artifacts from the Regional Archaeological Museum Plovdiv”

7.1%


Find information for specified (or named) subject (i.e., person, place, location, etc.) forming the main subject of the request

“I am looking for pictures of Stuttgart”

24.6%

General topical search

Find information for general subject

“Italian medieval illuminations”, “Looking at examples of art made by women”

47.1%

Browsing or exploring

Used to identify searches where the user has no specific goal

“I am just browsing your collections”

7.1%

Ambiguous or unclear

The search request is unclear or difficult to determine category

“I am an Opera lover”, “book”

3%UNESCO-NDL workshop, New Delhi, 25-27 October 2017 Page 30

Motivation and use• Task-closure:

– 83.9% were involved in “open-ended” tasks (e.g. scholarly research”)

• Modification: – 36.8% represent

“unmediated” cases, i.e. the information found will be used without modification (e.g., to illustrate a presentation)

• Type of output: – 64.4% would be textual in

form (e.g., academic article), 6.9% in a visual form, 3.4% in audiovisual form


I want working women images and videos from the XX century.

To create a video about women and Europe during the XX century

Create new work

Open-ended

Unmediated

Video


Understanding users:How knowledge about users can inform system design and evaluation activities

Understanding users

• When trying to understand users’ search and discovery activities we may be trying to answer the following:– Characterising and classifying (who are the

users?)– Needs and tasks (what do they want to do?)– Goals (why are they trying to do it?)– Behaviours (how are they doing it?)– Preferences (how would they like to do it?)– Performance (how well have they done it?)– ….


• Information Behaviour

• Information Seeking

• Interactive Search

• …

User-informed design and evaluation

• Understanding users and their information use environment will help inform design (and evaluation) activities– Assessing the effectiveness of or user satisfaction with

current systems or features (e.g. usage of features)– Designing new features/functionality to support users (or

categories of users) with carrying out immediate (e.g. search task) and longer term (e.g. work task) goals

– Design for individuals, groups, tasks etc. an open question– Need multiple forms of data to build up accurate and

holistic picture of users and usage– Key challenge is how you integrate the consequences of

user studies into collections and systems to provide effective support


Supporting the retrieval process

Sutcliffe, A.G. and Ennis, M. (1998). Towards a cognitive theory of information retrieval. Interacting with Computers, 10:321–351.


Supporting wider goals

Darby, P. and Clough, P. (2013) Investigating the Information Seeking Behaviour of Genealogists and Family Historians, Journal of Information Science, Volume 39(1), pp. 75-86.


Useful sources for thinking about users and developing search and discovery


“Decades of digitisation have made a wealth of digital cultural material available online. Yet search — the dominant interface to these collections — is incapable of representing this abundance. Search is ungenerous: it withholds information, and demands a query. This paper argues for a more generous alternative: rich, browsable interfaces that reveal the scale and complexity of digital heritage collections.”Mitchell WhitelawUNESCO-NDL workshop, New Delhi, 25-27 October 2017

Page 38

Thank you

Any questions?
