PATHS Evaluation of the 1st paths prototype

Embed Size (px)

DESCRIPTION

 

Text of PATHS Evaluation of the 1st paths prototype

  • 1. PATHS Project EU-ICT-270082Grant Agreement No.ICT-2009-270082Project AcronymPATHSProject full title Personalised Access To cultural Heritage Spaces D5.1 Evaluation of the first PATHS prototypeAuthors:Jillian Griffiths (MDR)Paula Goodale (USFD)Contributors: Sam Minelli (Alinari)Andrea de Pollo (Alinari)Rodrigo Agerri (UPV/EHU)Aitor Soroa (UPV/EHU)Mark Hall (USFD)Stein Runar Bergheim (AVINET)Konstantinos Chandrinos (i-Seive)George Chryssochoidis (i-Seive)Kate Fernie (MDR)Tom Usher (MDR)Project funded under FP7-ICT-2009-6 Challenge 4 Digital Libraries and ContentStatusFinalDistribution levelPublicDate of deliveryM21TypeReportProject website http://www.paths-project.euProject Coordinator Dr. Mark StevensonUniversity of SheffieldD5.1 Evalutation of the first PATHS Prototype 1

2. PATHS Project EU-ICT-270082Change LogVersionDate Amended byChanges0.117-07-2012 Jillian Griffiths Document created with draft content0.1.117-07-2012 Sam Minelli Results from Expert Review added0.1.224-08-2012 Jillian Griffiths Initial results from UK and Spanishdemos added0.1.327-08-2012 Aitor Soroa Data from WP2 added0.1.429-08-2012 Mark Hall Review against the FunctionalSpecification added0.1.512-09-2012 Jillian Griffiths Editing of document0.1.614-09-2012 Jillian Griffiths Data from WP3 added0.228-09-2012 Jillian Griffiths Editing of documentResults from lab evaluations addedResults from demonstrationevaluations addedResults from WP3 addedDraft v1.0 circulated0.2.101-10-2012 Jillian Griffiths Additional data from lab evaluationsaddedEditing of document0.305-10-2012 Jillian Griffiths Editing of documentDraft recommendations addedWP3 Cognitive Walkthrough dataaddedLab evaluation data addedDraft v2.0 circulated0.3.108-10-2012 Jillian Griffiths Editing of document1.012-10-2012 Jillian Griffiths Final versionD5.1 Evalutation of the first PATHS Prototype2 3. PATHS Project EU-ICT-270082Table of Contents1. Executive Summary8 2. Introduction 9 2.1 PATHS Evaluation Methodology9 2.1.1 Objectives of the evaluation9 2.1.2 Development of the evaluation methodology 10 2.1.3 Strategy10 2.2 Field-based evaluation, Demonstration sessions 10 2.3 Laboratory-based evaluation11 2.4 Project-wide evaluation11 2.5 Review against the Functional Specification11 3. Implementation 12 3.1 Participants12 3.2 Evaluation protocol 13 3.3 Piloting13 3.4 Field-based evaluation, Demonstration sessions14 3.4.1 Demonstration participants and sampling approach14 3.4.2 Demonstration Protocol15 3.4.3 Measurement of Demonstration-based evaluation 15 3.4.4 Data collection 16 3.4.5 Ethics17 3.5 Laboratory-based evaluation 17 3.5.1 Goals 17 3.5.2 Sample17 3.5.2.1 UK sample 18 3.5.2.2 Ireland sample18 3.5.3 Laboratory Evaluation Protocol19 3.5.4 Data collection 20 3.5.5 Ethics22 3.6 Project-wide evaluation 22 4. Results23 4.1 Results of the Demonstration and Laboratory evaluations 23 4.1.1 Participant profile 23 4.1.2 User type 30 4.1.3 Use of cultural heritage information32 4.1.4 Initial impressions to the brief introduction to PATHS and usability semanticdifferentials: Demonstration responses41 4.1.5 Ease of use, Usefulness and Inventiveness of PATHS44 4.1.6 Suggestions for improvement of PATHS50 4.1.7 Preference for exploring content60 4.1.8 Path flexibility, interaction and related content 61 4.1.9 Preferred devices for PATHS activities69 4.1.10 PATHS and social media 70 4.1.11 Final view of PATHS: responses to usability semantic differentials 71 4.2 Additional laboratory-based evaluation activities 76 4.2.1 Cognitive Style Analysis test 76 4.2.2 Tasks 78 4.2.2.1 Structured tasks78 D5.1 Evalutation of the first PATHS Prototype 3 4. PATHS Project EU-ICT-270082 4.2.2.2 Path creation tasks81 4.2.2.3 Task observation data84 4.2.2.4 Path data87 4.2.2.5 Interview data 92 4.2.2.6 Observations difficulties and errors 95 4.2.3 PATHS support and use for tasks97 4.3 Results of the project-wide activities 98 4.3.1 Content analysis and enrichment98 4.3.1.1 Content analysis: evaluation of scalability99 4.3.1.2 Ontology extension 100 4.3.1.3 Intracollection links100 4.3.1.4 Quality, evaluation of how the automatic links compare to manual annotations100 4.3.1.5 Background links 101 4.3.1.6 Quality, evaluation of a sample of the links between equivalent items and Wikipedia articles 101 4.3.2 System architecture102 4.3.2.1 Test harness results 103 4.3.3 User Interaction and Interface Design Cognitive Walkthrough109 4.4 Review against the Functional Specification115 5. Conclusions120 6. Recommendations for development of PATHS 122 6.1 Registration/User Accounts122 6.2 Explore 122 6.3 Search123 6.4 Search results123 6.5 Item Record Content and Layout123 6.6 Finding a path124 6.7 Following a path124 6.8 Workspace (search)124 6.9 Workspace (path)124 6.10 Creating a path125 6.11 Publish/share path 125 6.12 Overall125 6.13 Content126 7. References 127 AppendicesAppendix 1 Demonstration evaluation: Participant Pack including Information Sheet,Consent Form and Questionnaire ........................................................................................129 Appendix 2 Demonstration evaluation: Moderator Pack, including focus group schedule ..144 Appendix 3 Laboratory Profile Questionnaire ......................................................................158 Appendix 4 Laboratory evaluation Task Feedback Questionnaire ......................................162 Appendix 5 Laboratory evaluation Feedback Questionnaire ...............................................165 Appendix 6 Laboratory evaluation Interview Schedule ........................................................176 Appendix 7 Laboratory evaluation tasks ..............................................................................177 Appendix 8 Laboratory evaluation Participant Information Sheet ........................................181 Appendix 9 Laboratory evaluation Participant Consent Form .............................................184 D5.1 Evalutation of the first PATHS Prototype4 5. PATHS Project EU-ICT-270082List of figuresFigure 1 Overarching Evaluation Protocol .............................................................................13 Figure 2 Demonstration Evaluation Protocol .........................................................................15 Figure 3 Demonstration evaluation measures .......................................................................16 Figure 4 Laboratory-based Evaluation Protocol ....................................................................19 Figure 5 Task rotation order...................................................................................................20 Figure 6 Cognitive Style Analysis matrix................................................................................21 Figure 7 Gender: Demonstration responses ..........................................................................23 Figure 8 Gender: Laboratory responses ................................................................................23 Figure 9 Age group: Demonstration responses .....................................................................24 Figure 10 Age group: Laboratory responses .........................................................................24 Figure 11 Country of residence: Demonstration responses ..................................................25 Figure 12 Country of residence: Laboratory responses .........................................................25 Figure 13 Occupation/status: Demonstration responses .......................................................26 Figure 14 Occupation/status: Laboratory responses .............................................................26 Figure 15 Occupation/status: visual representation of all participants ...................................27 Figure 16 Internet experience: Demonstration responses .....................................................28 Figure 17 Internet experience: Laboratory responses ...........................................................28 Figure 18 Web search experience: Laboratory responses ....................................................29 Figure 19 Searching online: Demonstration responses .........................................................29 Figure 20 Use of cultural heritage information: Laboratory responses ..................................30 Figure 21 User type: Demonstration responses ....................................................................30 Figure 22 End User participant types Figure 23 Expert User participant types.................................................................................31 Figure 24 User type: Laboratory responses ..........................................................................31 Figure 25 Types of cultural heritage information used: Demonstration responses ................32 Figure 26 Types of cultural heritage information used: Laboratory responses ......................32 Figure 27 Web sites used for cultural heritage information: Demonstration responses ........34 Figure 28 Web sites used for cultural heritage information: Laboratory responses ...............34 Figure 29 Information behaviour preferences: Demonstration responses .............................35 Figure 30 Information behaviour preferences: Laboratory responses ...................................36 Figure 31 Frequency of use of cultural heritage information by task: Demonstrationresponses ..............................................................................................................................37 Figure 32 Frequency of use of cultural heritage information by task: Laboratory responses 38 Figure 33 Top five uses of cultural heritage information: Demonstration responses .............38 Figure 34 Top five uses of cultural heritage information: Laboratory responses ...................39 Figure 35 Use of cultural heritage