View
0
Download
0
Category
Preview:
Citation preview
Evaluating the next generation of user experience in
information retrieval through the visualisation power
of Microsoft PivotViewer
A study submitted in partial fulfilment of the requirements for the degree of Master of Science in
Information Systems Management
at
Souvik Dey
Registration No: 100151932
by
Date: September 2011 Supervisor: Dr. Paul Clough
i
Background: Today, the Web, through its various search engines, has emerged as the
primary source, where people tend to access and gather information from. Searching
for information on the Web has become a key part of our life. However, the interfaces
provided by the major search engines and the techniques used to search content on the
Web, have hardly changed since their inception. There is hardly any support provided
by the pre-existing systems to interact with the required criteria of search, and assist in
formulating and refining queries. After years of development, Microsoft Corporation
has launched a Silverlight control called PivotViewer that allows an interactive
interface, which allows the users to navigate through massive data collections in a way
different from traditional interfaces. This piece of research aims to study this interface
and decide on the possibility of it being a next generation of user interface in
information searching.
Aims: Firstly, this dissertation aims to carry out an investigation of information
retrieval and visualisation systems, along with an exploration of user behaviour in
information searching. Secondly it is aimed to design and implement a Pivot collection
with a large dataset and finally carry out a comparative evaluation and analysis of Pivot
interface with respect to traditional Web interfaces for information retrieval and
visualisation.
Methods: Based on the guidelines set by Microsoft Silverlight Team, a PivotViewer
interface was set up, using 10,000 historic images of St. Andrews University Library. To
evaluate the above system and PivotViewer interface in general, 10 participants from
Information School at the University of Sheffield, was invited to experience this new
interface and carry out tasks in three vivid scenarios. Performance criteria for
evaluations were set and views from the participants were gathered and evaluated.
Results: From the observations and tests carried out, PivotViewer emerged as a clear
winner in terms of speed, as well as the performance criteria set. It provided a rich
multifaceted multimedia visual experience to the users and proved to be more efficient
and acceptable to the users as compared to the traditional search interfaces.
Conclusions: This dissertation illustrates the benefits of incorporating interactive
techniques of information visualisation, through Microsoft PivotViewer, for information
retrieval, keeping in line with the user behaviours in modern day searching. From the
study taken, the Pivot interface can be said to epitomise instances of next-generation
Web search interfaces that has the potential to provide the users to have more active
participation in the information exploration process, from the vastness of the
information space on the Web.
ABSTRACT
ii
This study could not have been accomplished successfully without the assistance of
several people whose contributions I heartily acknowledge. Firstly, I would like to
express my deepest and sincerest appreciation to my supervisor, Dr. Paul Clough,
whose encouragement, support and guidance from the inception of the project, through
to the conclusion, helped me to develop a deeper understanding of the subject. Without
his guidance, this study would not have been possible. Secondly, I would like to
sincerely acknowledge the help of Mark Hall, for his time and expertise, in helping me
out to generate the metadata for the St. Andrews collection. At the same time, I equally
appreciate the help of Dr. Robert Villa, who helped me and guided me on several
occasions, apart from agreeing to carry out my pilot study before the experimentations.
In addition, I would like to express my gratitude to Andy Stones and Paul Fenn for
helping me in resolving technical and system issues, whenever there was a need and I
am also grateful for every support and help received from the Information School at the
University of Sheffield.
It is also a pleasure to thank two of my dear colleagues and friends for their all round
support, in particular, Farhad Sadri and Athira Shridhar. Special thanks are also
extended to Arnab Dutta for his timely advices and guidance. I am also very heartily
thankful to my dearest friends Debanjali Bhattacharjee and Prema Dutta, for their help
in reading and correcting my written work throughout my studies. I would also like to
thank my friends back home Aniruddha Roy, Kaustuv Chatterjee and Subrata Sikdar, for
giving me the moral support and comfort in every step.
Finally, I would like to convey my deepest regards to my family for their endless love
that formed the most important part of my life and for always being there when I
needed them, helping me to face difficulties. Without their undying support and
encouragement, this journey would not have been possible. They gave me an
opportunity to study in one of the best universities in UK, which I shall treasure all my
life.
I would like to dedicate this project to my parents, Papri Dey and Prasanta Dey, in the
hope that I make worthwhile use of my study in UK and live up to their dreams, and
also to my dearest late sister Sohini Hazra, who would be proud of me to hear of all my
achievements till now, if she was here today.
ACKNOWLEDGEMENTS
iii
Abstract................................................................................................................................................. i Acknowledgement.......................................................................................................................... ii Contents................................................................................................................................................ iii List of figures...................................................................................................................................... v List of tables........................................................................................................................................ vii Chapter 1: Introduction............................................................................................................... 1
1.1 Present Scenario................................................................................................................... 2 1.2 Information and image retrieval systems................................................................. 2 1.3 Efficient Interface Design.................................................................................................. 4 1.4 Structure of Dissertation................................................................................................... 5
Chapter 2: Research Aims........................................................................................................... 6
2.1 Motivation............................................................................................................................... 6 2.2 Research Objectives........................................................................................................... 8 2.3 Research Questions............................................................................................................. 9 2.4 Scope of research................................................................................................................. 9
Chapter 3: Literature Review................................................................................................... 10
3.1 Introduction........................................................................................................................... 10 3.2 Information Retrieval........................................................................................................ 11
3.2.1 Traditional Information Retrieval................................................................. 11 3.2.2 Web-based Information Searching................................................................ 13 3.2.3 Information retrieval of large faceted datasets....................................... 14 3.2.4 Web User Interface............................................................................................... 17
3.3 Information Visualisation................................................................................................ 18 3.3.1 Web-based Information Visualisation.......................................................... 20 3.3.2 Visualisation of faceted datasets..................................................................... 22
3.4 User Behaviour: Web Searching.................................................................................... 23 3.5 Summary................................................................................................................................. 26
Chapter 4: Microsoft PivotViewer.......................................................................................... 27
4.1 Concept of collection.......................................................................................................... 27 4.2 PivotViewer Collection...................................................................................................... 28
4.2.1 Collection XML Schema (CXML)...................................................................... 29 4.2.1.1 Structure of schema................................................................................... 30 4.2.2 Collection Image Content.................................................................................... 31
4.2.2.1 Deep Zoom Image Collections............................................................... 31 4.2.2.2 Deploying Deep Zoom Collections...................................................... 35
4.3 Collection Design................................................................................................................. 37 4.3.1 Kinds of Collections.............................................................................................. 38 4.3.2 Facets and Facet Categories.............................................................................. 39 4.3.3 Organizing Facet Categories in PivotViewer............................................. 40
4.3.4 Facet Naming................................................................................................…………... 42 4.3.5 Facet Values and Formatting................................................................................... 43 4.3.6 Imagery............................................................................................................................. 44
4.4. Summary.................................................................................................................................... 45
CONTENTS
iv
Chapter 5: Research Methodology........................................................................................ 46
5.1 Literature Review............................................................................................................... 46 5.2 Dataset (St Andrews University Library Photographs)..................................... 48 5.3 Overview of metadata...................................................................................................... 49 5.4 Building the Pivot Collection......................................................................................... 50 5.5 Creating Pivot Silverlight Application....................................................................... 54 5.6 Design Checklist.................................................................................................................. 59 5.7 User Experiments............................................................................................................... 60
5.7.1 Ethical Considerations........................................................................................ 61 5.7.2 Selection of participants.................................................................................... 61 5.7.3 Rationale.................................................................................................................. 61 5.7.4 Preliminary Questionnaire............................................................................... 62 5.7.5 Evaluation of user-centred expectations.................................................... 63 5.7.6 Pivot awareness..................................................................................................... 63 5.7.7. A general explanation of using PivotViewer............................................ 64 5.7.8. User-centred interface evaluation................................................................ 65 5.7.9. Interface evaluation parameters................................................................... 67 5.7.10. Post evaluation questionnaire..................................................................... 68
5.8. Pilot Study.............................................................................................................................. 69 5.9. Timetable............................................................................................................................... 70 5.10. Summary............................................................................................................................. 71
Chapter 6: Results and analysis.............................................................................................. 72
6.1 General User Overview..................................................................................................... 72 6.2 Search Behaviour Patterns.............................................................................................. 74 6.3 Evaluation of user-centred expectations.................................................................. 80 6.4 Microsoft PivotViewer awareness................................................................................ 81 6.5 User-centred interface evaluations............................................................................. 82
6.5.1 Scenario: A................................................................................................................ 82 6.5.2 Scenario: B................................................................................................................ 84 6.5.2 Scenario: C................................................................................................................ 87
6.6 Summary of interface evaluations............................................................................... 89 6.7. Post Experimentation Results: 1................................................................................. 91
6.7.1 Evaluation of Tasks.............................................................................................. 91 6.7.2 Results on Pivot Interface.................................................................................. 91 6.7.3 Technique and Quality of Results Retrieved by PivotViewer............ 92
6.8. Post Experimentation Results - 2................................................................................ 94 6.9. Summary................................................................................................................................ 97
Chapter 7: Discussions................................................................................................................. 98 7.1 PivotViewer as an Information Retrieval System.................................................. 99 7.2 PivotViewer as an Information Visualisation System.......................................... 101 7.3. Summary................................................................................................................................. 102
Chapter 8: Conclusion................................................................................................................... 103
8.1 Research Summary............................................................................................................. 103 8.2 List of Objectives.................................................................................................................. 104 8.3 Results....................................................................................................................................... 106 8.4 Limitations.............................................................................................................................. 106 8.5 Future Research.....................................................................................……………….…… 107 8.6 Summary…………………………………................................................................................. 109
Appendices.......................................................................................................................................... 111 References........................................................................................................................................... 126
v
Fig 1: Images returned by Google Image when queried for “Endangered
Species” [July 20, 2011]..................................................................................................... 4 Fig 2: Scope of this dissertation.................................................................................................... 9 Fig 3.1: Yahoo Directory................................................................................................................. 15
Fig 3.2: The Phlat interface with a query of a single keyword and two filters………………………………...................................................................................................... 16
Fig 3.3: FacetMap Interface........................................................................................................ 16
Fig 3.4: Primary colours as described in Opponent Process Theory……...... 19 Fig 3.5: FacetLens interface........................................................................................................ 20 Fig 3.6: FacetMap interface......................................................................................................... 20
Fig 3.7: Grokker - Information visualisation......................................................................... 23 Fig 4.1: Example of a Pivot interface.......................................................................................... 27 Fig 4.2: Pivot Architecture.............................................................................................................. 28 Fig 4.3: Simple example of a CXML file...................................................................................... 29
Fig 4.4: Simple Pivot Interface based on the previous sample CXML code. 30 Fig 4.5: Structure of schema.......................................................................................................... 31
Fig 4.6: DZI – DZC based on Morton Layout.................................................................... 32 Fig 4.7: Iterative process in designing..................................................................................... 37 Fig 4.8: Facets and Facet Categories.......................................................................................... 39 Fig 4.9: Information Panel- I......................................................................................................... 40 Fig 4.10: Information Panel: II..................................................................................................... 41 Fig 4.11: Guidelines for facet values.......................................................................................... 43 Fig 4.12: Example for enhanced collection imagery........................................................... 44 Fig 5.1: Example of metadata associated with images in St. Andrews University
Library historic photographic collection............................................................... 49 Fig 5.2: First step to start building the collection................................................................. 51 Fig 5.3: Columns representing facet categories for St. Andrews Collection ........... 52 Fig 5.4: Image showing Pivot ‘Publish Button’...................................................................... 54 Fig 5.5: Creating Pivot Silverlight Application...................................................................... 54 Fig 5.6: Mainpage.xaml..................................................................................................................... 55 Fig 5.7: MainPage.xaml.cs................................................................................................................ 56 Fig 5.8: HTML coding to bind CXML............................................................................................ 57 Fig 5.9: Pivot Collection of St. Andrews University photographs.................................. 58 Fig 5.10: Facet categories in Pivot collection of St. Andrews University
photographs..................................................................................................................... 58 Fig 5.11: Various features of Pivot Interface ........................................................................ 64 Fig 5.12: St. Andrews Interface and Pivot interface with exactly same datasets.... 65 Fig 5.13: Endangered species Pivot interface and Google Images.............................. 66 Fig 5.14: IMDb and Netflix PivotViewer .................................................................................. 66 Fig 6.1: Participant age group....................................................................................................... 72 Fig 6.2: Professional background of participants................................................................ 72 Fig 6.3: Educational background of participants................................................................. 73 Fig 6.4: User PC familiarity............................................................................................................. 73 Fig 6.5: User photo search experience....................................................................................... 73 Fig 6.6: User Internet usage........................................................................................................... 74 Fig 6.7: Confidence levels in Web searching........................................................................... 74 Fig 6.8: Online search habits ......................................................................................................... 75
LIST OF FIGURES
vi
Fig 6.9: User satisfaction levels on text-based searching ................................................. 76 Fig 6.10: User satisfaction levels in image searching.......................................................... 76 Fig 6.11: Overall traditional search interface satisfaction levels.................................. 76 Fig 6.12: Number of pages viewed by users, before they usually stop
searching for information........................................................................................... 77 Fig 6.13: Graph indicating the favourable image search techniques (for each
questions a to e) and their corresponding frequency/percentage of support................................................................................................................................. 78
Fig 6.14: Graph indicating the array of experiences with image searching techniques (for each questions a to d) and their corresponding frequency/percentage of support............................................................................. 78
Fig 6.15: User-centred expectations........................................................................................... 80 Fig 6.16: PivotViewer awareness................................................................................................. 81 Fig 6.17: Familiarity of using Microsoft PivotViewer......................................................... 81 Fig 6.18: Graphs comparing the time taken and number of clicks for completing
scenario-A- Task1 and 2................................................................................................. 82 Fig 6.19: Interface evaluation summary- scenario-A.......................................................... 84 Fig 6.20: Graphs comparing the time taken and number of clicks for completing
scenario-B- Task1 and 2................................................................................................. 85 Fig 6.21: Interface evaluation summary- scenario-B.......................................................... 86 Fig 6.22: Graphs comparing the time taken and of number clicks for completing
scenario-C- Task1 and 2................................................................................................. 87 Fig 6.23: Mean rating summary for scenario-C..................................................................... 89 Fig 6.24: Summary of time taken and number of clicks for all tasks............................ 90 Fig 6.25: Overall interface evaluation summary................................................................... 90 Fig 6.26: Analysis of tasks undertaken...................................................................................... 91 Fig 6.27: Analysis of Pivot interface........................................................................................... 92 Fig 6.28: Analysis of technique/quality of results retrieved by PivotViewer.......... 93
vii
Table 3.1: Basic section to encompass a system.............................................................................. 11 Table 3.2: Collection of studies on Information Visualisation................................................... 19 Table 3.3: Categorised visualisation dimensions and related studies................................... 21 Table 3.4: User behavioural pattern studies..................................................................................... 23 Table 4.1: Organisation of DZI and DZC............................................................................................... 33 Table 4.2: Summary of procedures to build Pivot Collection.................................................... 33 Table 4.3: Comparative chart between JPEG and PNG formats................................................ 34 Table 4.4: Kinds of Collections…………………………………………….................................................. 38 Table 5.1: Evaluation methodologies carried out in previous studies................................... 47 Table 5.2: System requirements for executing and building PivotViewer applications 50 Table 5.3: Settings of St Andrews collection..................................................................................... 53 Table 5.4: Miscellaneous Property Adjustments............................................................................. 53 Table 5.5: Design Checklist....................................................................................................................... 60 Table 5.6: Parameters for interface evaluation................................................................................ 67 Table 5.7: Post evaluation questionnaire categories Part- 1...................................................... 68 Table 5.8: Approximate time needed for total experiment......................................................... 70 Table 6.1: Participant age groups........................................................................................................... 73 Table 6.2: User professions....................................................................................................................... 73 Table 6.3: Educational background of participants....................................................................... 73 Table 6.4: Familiarity with PC................................................................................................................. 73 Table 6.5: User photo search experience........................................................................................... 73 Table 6.6: User Internet usage................................................................................................................. 74 Table 6.7: User confidence levels when searching the web....................................................... 74 Table 6.8: Online search habits............................................................................................................... 75 Table 6.9: Responses to favourable search topics ......................................................................... 75 Table 6.10: User satisfaction levels for text-based searching ................................................... 76 Table 6.11: User satisfaction levels for image searching............................................................. 76 Table 6.12: Overall traditional search interface satisfaction levels........................................ 76 Table 6.13: Number of pages viewed by users, before they usually stop searching for
information.............................................................................................................................. 77 Table 6.14: Table indicating the favourable image search techniques and their corresponding frequency/percentage of support by users................................77 Table 6.15: Table indicating the array of experiences with image searching
techniques and their corresponding frequency/percentage of support by users.................................................................................................................................... 78 Table 6.16: Table indicating the users’ expectation levels on certain features in user interfaces for information visualisation and retrieval.............................. 80 Table 6.17: Familiarity with using Microsoft PivotViewer........................................................ 81 Table 6.18: Time taken for carrying out each of the tasks and the number of clicks- Scenario-A Interfaces.......................................................................................... 82 Table 6.19: Performance results based on five performance criteria for St. Andrews
interface.................................................................................................................................. 83 Table 6.20: Performance results based on five performance criteria for PivotViewer
interface................................................................................................................................... 83 Table 6.21: Mean rating summary for scenario-A ......................................................................... 84 Table 6.22: Time taken to carry out each of the tasks and the number of clicks-
Scenario B................................................................................................................................. 84 Table 6.23: Results based on five performance criteria for Google Images....................... 86 Table 6.24: Results based on five performance criteria for PivotViewer........................... 86 Table 6.25: Mean rating summary for scenario-B ......................................................................... 86 Table 6.26: Time taken for carrying out each of the tasks and the number of clicks-
Scenario-C ............................................................................................................................... 87 Table 6.27: Results based on five performance criteria for IMDB ........................................ 88
LIST OF TABLES
viii
Table 6.28: Results based on five performance criteria for Netflix Pivot.............……….. 88 Table 6.29: Interface evaluation summary- scenario-C.............................................................. 89 Table 6.30: Summary of individual task results between PivotViewer and other
(traditional) interfaces....................................................................................................... 89 Table 6.31: Overall performance summary ..................................................................................... 90 Table 6.32: Results of tasks undertaken............................................................................................. 91 Table 6.33: Results of Pivot interface ................................................................................................. 92 Table 6.34: Technique and Quality of results retrieved.............................................................. 92 Table 6.35: User views on PivotViewer advantages..................................................................... 94 Table 6.36: User views on PivotViewer’s disadvantages............................................................ 95 Table 6.37: Extra functionalities and recommendations for interfaces............................... 96 Table 6.38: Additional comments......................................................................................................... 96
Word Count
Sections included: 1 (Introduction) to 8 (Conclusion)
Pages included: 1 to 110 (excluding tables and figure names)
Package used: Microsoft Word 2007
Word Count: 25,986
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 1 sdey1@sheffield.ac.uk - 100151932
Chapter 1
With the exponential increase in the plethora of information generated in the world
today, the design of visually interactive representation of information is undergoing a
substantial paradigm of transition through technological advancement, with the
persistent maturation of our information age. A rapid transformation in the cyber age
with the centralisation of data and ubiquity of information digitization, coupled with the
technological advancements in networking architectures, has resulted into a radical
growth in the bandwidth of information consumption and use. The rate at which this
information is growing, it becomes increasingly significant to find an efficient solution to
manage and explore the required information from this abundance for analysis. Often, it
turns out to be quite time-consuming when a particular information or data needs to be
manually fetched from this abundance. Besides, there is also a need for proper
scalability and accessibility in the method of such a representation because, although the
required data may be fetched, it is more difficult to establish an inter-relationship with
the other data collections in other places or across the website, which further
complicates the process. However, with the advancements in Computer Graphics
technology, we are now stepping into an era where large scale analysis of quantitative as
well as qualitative data is now at reality. But then, even with such a steadfast progress,
there is a need for addressing the satisfaction level in the paradigm of human cognitive
process, when it comes to using the traditional – text based and often page based
information or image retrieval systems.
INTRODUCTION
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 2 sdey1@sheffield.ac.uk - 100151932
1.1 Present Scenario
Information retrieval, storage technologies, manipulation and analysis of data in the
form of pictures, texts and numbers have been successfully dealt with over the years,
with the help of hybrid technologies and efficient computer computing systems. Today,
we have several useful applications in the form of database technologies, spreadsheets
and word-processors that go miles in simplifying several of our projects, businesses and
tasks that were initially complicated, strenuous, and quite time-consuming. Massive
developments in computer science and technology, precisely in improving the speed of
processing and allowing machines to manipulate huge amounts of information, that
include not just texts and images, but also applications that are capable of handling data
in the form of user generated video and audio resources are in abundance. However,
Nguyen and Huang (2004) point out,
These problems with the current user interfaces have led many designers of
information systems to realize the need for the development of new generation
user interfaces that can reduce the human cognitive cost for their next generation
information systems.
With every passing second, there is an increasingly high amount of usage of such
applications that generate additional magnitudes of multimedia and as such, a huge
amount of data and information are being added to the global information network. Such
an exponential growth also inevitably allows these collections to be globally available on
the World Wide Web and get heaped amongst the abundance of their ubiquity.
1.2 Information and image retrieval systems
The principle work of a proper image retrieval system is to help users retrieve a certain
category of image or images effectively in the least amount of time from the repository.
Years of research and study in the area of information retrieval have significantly
improved the technology behind such systems by introducing automated techniques that
indexed documents based on the words which could be easily retrieved by users through
queries. However, image retrieval is more challenging, given the fact that indexing
images is complicated since unlike textual information, they are devoid of units that have
a proper meaning which would enable a quick extract. As such, images are often
associated with some meta-data or a set of descriptive text, comprising of categories,
”
“
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 3 sdey1@sheffield.ac.uk - 100151932
titles and keywords which enable images to be extracted and filtered using traditional
information retrieval methodologies. For example, if a tourist wants to locate the picture
of a wonderful sea beach in a particular country, in order to make a decision for a
vacation, he can simply enter “sea beach” as a contextual query or search the related
category, assuming the image collection has the suitable metadata associated with it.
In text based information retrieval systems, the result extracted is generally a file or a
group of files which have an associated filename and a preview of its contents or
highlighted text that mark the query. Users can overview these results and make a
decision on its relevance to their requirements, however, with images, only a miniature
version called thumbnail is visible, seeing which the user can narrow down and decide
on his relevant requirements by judgements. However, there may be many other images
in abundance on the Web that might be hidden beneath the huge repository of pictures
that he may not come across initially.
Short Scenario
If we consider pictures of endangered species, for example: there may be various species,
with different originating locations and which may have different class and family. Taking
into consideration a popular search engine, such as Google Images (or Flickr, Picasa etc.), if
we make a query on “Endangered Species”, it will return a collection of thousands of
images with related web pages that may often be irrelevant or may require us longer time
to realize exactly what we want (fig-1). Besides, it may not be exactly what we were
looking for and our actual requirement may be hidden beneath the millions of results that
were processed. The large numbers of thumbnails that are retrieved are however limited
to per page views and the user will need to browse further or refine the search query in
order to narrow down to the actual requirements which, if complicated, may take ages.
Besides, documenting annotations are very laborious and the results are most often quite
subjective. As such, research and development studies in image retrieval are mostly
focussed on generating objective methods for spontaneously indexing images.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 4 sdey1@sheffield.ac.uk - 100151932
1.3 Efficient Interface Design
Interface design for information retrieval and human computer interaction needs to be
future-proof by taking technology to new heights and obscuring the perimeters between
metadata and content, and the boundaries that segregate explicit from the implicit and
public from private. From a collection of navigable pages to a juxtaposition of vibrant
knowledge pattern, the perspective on Web and technology is constantly repositioning
itself into our society by feeding a vivid accumulation of information sources in a
bionomic array of services. The core of this process lies in the exponential increase of
social web-applications, coupled with the structural modifications lead by the
pervasiveness of web-network applications and advanced technological devices, which
has initiated a redefinition of our understanding and knowledge of information retrieval,
architecture, visualisation, communication and storage.
In order to create an effective user experience in this area, a profound perception of
upcoming format of contents, along with the knowledge and perception of technological
changes and understanding of the character of metadata is essential. If carefully
observed, there is an increasing trend in the rise of broadly structured, subjective and
transient data available which emanates into creating a hybrid inter-subjective, multi-
faceted array of contents associated with people around the globe and metadata. As such,
novel advancements are required to manage rising paradigms such as visualisation of
information snippets and semantic collaborative tagging.
Fig 1 – Images returned by Google Image when queried for “Endangered Species” [Accessed: July 20, 2011]
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 5 sdey1@sheffield.ac.uk - 100151932
1.4 Structure of Dissertation
Chapter 2 provides the research motivations for carrying out this project, along with the
research aims and related objectives. Based on these, a few research questions are asked
and the scope of the project has been laid down.
Chapter 3 furnishes an exploration on the background to this research, contextualising it
in the area of information retrieval, information visualisation and user behaviour in
information searching. Relevant works by authors have been discussed in these areas, in
the chapter, to formulate the ideas percolating on these areas and enable an insight to
the project taken.
Chapter 4 introduces the PivotViewer interface and discusses the concept of
PivotViewer collections and architecture. Schema and image content formats are briefly
discussed, along with the procedures for collection design, with explanations based on
facets and facet categories. Summaries on facet naming, organising, formatting and
quality of imagery have been discussed
Chapter 5 describes the research methodology undertaken to carry out the dissertation,
by discussing the relevant literature. It also focuses on the practical implementation
details in building the PivotViewer application in Silverlight, with focus on the dataset
adopted and the steps undertaken. Finally, the methods of user experiments are
discussed in details with the views on ethical considerations undertaken and the
rationale.
Chapter 6 presents the findings of the experiments undertaken in to four categories:
general user overview and search behaviour patterns, leading to the analysis of the user-
centred expectations, evaluations between PivotViewer and traditional interface and
post experimentation evaluations.
Chapter 7 discusses the results obtained from the user experiments and relates them to
the literature discussed in Chapter 3, with the view to answering the research questions
posed in Chapter 2.
Chapter 8 summarises the main findings of this research, stating the aims and objectives
achieved in the research, followed by the limitations encountered and the possibilities
for further work.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 6 sdey1@sheffield.ac.uk - 100151932
Chapter 2
2.1 Motivation
Ever since the digital space in the Web flagged off with the dominance of the hypertext, it
has transformed into an age of advanced improvements and break-through technologies,
especially in visualisation and graphics with the sophistication of human interactivities.
It simultaneously weaved diverse resources that comprised of semantic, spatial, dynamic
and social relationships with rich links. As such, technological advancements have
rapidly increased the information and data resource domains on the Web and it is
dynamically expanding with every passing second. While, it is evident that there has
been a significant contribution and development in Web based search engines that are
capable of indexing a magnitude of documents and information, and returning the results
in near fraction of seconds (Brin and Page, 1988; Ghemawat et al., 2003), the user
experiences in information seeking behaviour and the level of interactivities have not
seen much progress and there has been no drastic change. The process of crafting
queries and the methods to deliver the queried results have fundamentally remained
unchanged. According to Broder et al. (2010), pages and applications with a clean query
bar has always been the traditional form interfaces where users feed textual queries, and
are presented with a format that is traditionally an array of list-based textual hyperlinks
or “10 blue links”, each essentially with a title, followed by short snippets comprising of
the queried texts and an associated URL to allow the users to be redirected to the
particular document or website so that they can scavenge their required information.
Throughout the years, these generic interfaces have been canonically used for
exploratory searching either in simple or complex chained queries, resulting in the
generated catalogued result sets. Though such an approach may direct the users to
retrieve relevant information, it could however still be difficult to accumulate an overall
inter-relation of the whole set of collection of resources that are available on that
particular topic or query from the information space and gather a sense of orientation.
RESEARCH AIMS
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 7 sdey1@sheffield.ac.uk - 100151932
Research and analysis on Web search behaviour of users have demonstrated that a
significant number of Web based search queries comprise of hardly one to three terms
(Jansen and Pooch, 2001; Spink et. al. 2001). These indicate that users seeking
information from search engines frequently face complexities in formulating or
constructing queries that find their most precise information requirements. Arguably,
search engines today provide hardly any support for users in reconstructing or
formulating their queries and it is often the onus of the users to adjudicate on the words
to use during the subsequent manual querying. Consequently, Web users rarely persist
in crafting further alterations to their queries (Silverstein et al., 1993; Spink et. al. 2001).
The traditional static forms of display of Web search results might have been a driving
factor to such user behaviour. The users are hardly given the choice to explore or play
around with the fetched results and as such, they have to resort to considering each
result individually and also in a given sequence.
Attributing to this present scenario, the theme of this piece of research work is to
demonstrate these underlying issues and instituting and evaluating a novel paradigm of
visually interactive next-generation of information retrieval interface, as provided by a
Microsoft product called PivotViewer. For the purpose of the research, an interface is
developed using the above technology on a set of image collection from a pre-existing
search database. At the same time, a couple of other ready-made pivot models or
prototypes designed by organisations have been taken into consideration for the
purpose of evaluation, with search based interfaces on similar collections present today.
These would empower the users to participate in a rather active role in working with the
information retrieval process provided by these systems, which will then be monitored
and evaluated through a set of feedback based on their experiences. The capability of the
users to explore their search results and at the same time, interacting and co-relating the
retrieved results illustrates and signifies a leap towards the vision Yao had for Web
information retrieval process systems (Yao, 2002).
This dissertation recognizes the importance of fundamental human decision making
while formulating queries from the information space, as they attempt to identify and
retrieve useful and relevant information according to their needs. Through the
development of such a next-generation interface using Microsoft PivotViewer, there will
be a substantial impact and knowledge on how users retrieve a needle of information
from an ever growing hay-stack of information space in the future.
“The public has a low tolerance of going in depth through what is retrieved”
- Spink et al. (2001),
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 8 sdey1@sheffield.ac.uk - 100151932
2.2 Research Objectives
The central theme of the research is to carry out a qualitative analysis by gathering a
significant understanding of information visualisation and establish a relation by
investigating how the interactive visualisation technique of Microsoft Pivot Viewer can
sustain information explorers to seek available information, through the massive amount
of data available on the Web.
As such, with my research I aim to fulfil the following aims and objectives:
Aim -1
To carry out a study-oriented investigation of information retrieval and visualisation
systems, along with the exploration of associated user behaviour, in information searching.
Objective 1: Investigate related literature in the areas of:
o Information Retrieval
o Information Visualisation
o User behaviour in information searching
Aim -2
Design and implement a Pivot collection with a large dataset.
Objective 1:
To study the techniques and features, to design and build pivot collections.
Objective 2:
To transform the historic photo collection available in St. Andrews University Library,
into a prototype Pivot collection.
Aim -3
To carry out a comparative evaluation and analysis of Pivot interface with respect to
traditional Web interfaces for information retrieval and visualisation.
Objective 1: To perform a comparative performance evaluation between the Pivot
interface developed on St. Andrews University photographic collection and the original
St. Andrews University Library interface and explore the results.
Objective 2: To perform a comparative analysis and performance evaluation, on
image retrieval on a particular subject, between an open-ended interface (Google
Images), with that of a Pivot interface on the particular subject.
Objective 3: To perform a comparative analysis and evaluation of two interfaces
dedicated to the same domain of information retrieval service - IMDb and Netflix
PivotViewer interface.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 9 sdey1@sheffield.ac.uk - 100151932
2.3 Research Questions
The fundamental hypothesis related to this research is that information retrieval through
visualisation by Microsoft PivotViewer is a novel mechanism of information seeing that
can substitute established query based or web-based interaction methodologies, in
terms of a higher multi-level way of navigating magnanimous data collections. To test
this hypothesis, I shall pose the following research questions:
Q1: How can information visualisation through Microsoft PivotViewer help to explore
information?
Q2: How can interactive exploration system through Microsoft PivotViewer be
architected and developed according to resources available on St. Andrews University
photo collection?
Q3: What are the ways of improvements and effects of information retrieval through
visualisations using PivotViewer on resource seeking?
2.4 Scope of research
There are three major research communities that could be adequately associated with
my research interests that encompass the domain of my research questions:
Q1: Web Research
Q2: Information Visualisation
Q3: Information Retrieval
This piece of research shall however focus at the cross-section of Information
Visualisation and Information Seeking that is Visualisation for Information Retrieval
(Fig-1). However it is worth mentioning that the research will also touch the
fundamental aspects of Human Computer Interaction.
Information
Visualisation
Information
Retrieval
Web Research
Visualisation for
Information
Retrieval
Visualisation for Web-
based Search and
Exploration
Information
Visualisati
on on the
web
Web based Information
Retrieval
Fig- 2: Scope of this
dissertation
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 10 sdey1@sheffield.ac.uk - 100151932
Chapter 3
3.1 Introduction
Searching for information is an inherent part of our lives and the rapid progress of
technology is playing a significant role in the area of information visualisation and
retrieval to help it in its evolution. Earlier, studies and articles related to the area of
visualisation and information retrieval was mostly authored and limited within the
domain of computer scientists (Chen, 2003). Today, there are several books that are
focussing on the algorithms and methods of information retrieval systems (Hearst, 2009;
Andrews, 2006). Over the years, due to technological developments, there has been an
exponential growth of interest and activities in the area, which has lead to the
publication of several articles in recent years that depicts novel visualisation systems
(Andrews, 2006; Chau, 2011, Chen, 2003; Dix et al, 2010, Hearst, 2009). While, some
present new algorithmic approaches (Nguyen et al, 2005), and others have carried out
detailed studies with human interactions for information retrieval (Whittaker et al,
2009), research on user interfaces are still being considered rare (Hearst, 2009). There is
a rising number in the taxonomies of information visualisation techniques (Chen, 2003)
and research in this domain is now disseminated across several platforms.
This chapter aims to discuss the three important areas of information and computer
science that this dissertation relates:
Information Retrieval
Information Visualisation
User behaviour in Web searching
For the purpose of this literature review, a ‘T-Shaped’ path is chosen, whereby mostly
studies related to a broader overview of certain areas are discussed (symbolising the
head of ‘T’), and in other cases, an in depth and ‘longer’ analysis of an area is taken into
consideration (symbolising a bottom half of ‘T’). Careful selection of literature is
gathered for the purpose of both, the practical engineering of an interface as well as to
contribute to the academic world an essential and synthesised set of ideas.
LITERATURE REVIEW
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 11 sdey1@sheffield.ac.uk - 100151932
3.2 Information Retrieval
Information Retrieval is defined as the process that deals with the representation,
storage, organisation of, and access to information items (Baeza-Yates and Ribeiro-Neto,
1999). This is the science, that essentially involves an automated process, which is
carried out by custom a developed system, that index digital document collections
automatically (Hoeber, 2007; Rodden, 2002).
3.2.1 Traditional Information Retrieval
Traditionally, an information retrieval system involves extracting words individually
from a collection of digital documents and associating them uniquely with a pointer that
links them to the set relevant documents, according to the information fed by the user,
while at the same time, minimising as many non-relevant documents as possible
(Rijsbergen, 1979) . Users can query a text to search for a required topic or fetch a
document for retrieving particular information. The system in concern processes the
query and retrieves the documents that contain the particular text in query. This may be
presented in a format that is sorted either by an estimated relevance or ordered
according to the frequency of the term in the document(s). Though, the significance of
information retrieval has been broadly on text and documents (Voorhees, 2004), similar
technique have also been applied successfully on retrieving images, audio and videos
(Baeza-Yates and Ribeiro-Neto, 1999).
Traditionally, there are two basic sections that encompass such a system:
Input User query
Digital document collection
Output Set of relevant documents matching the query
It is however said to be a good challenge to optimise the data structures to accommodate
documents as well as the queries that saves considerable amount of memory, but at the
same time, facilitates an efficient and precise coordination of matching queries to the
“But do you know that, although I have kept the diary [on a phonograph] for months
past, it never once struck me how I was going to find any particular part of it in case I
wanted to look it up?”
Dr Seward, Bram Stoker's Dracula, 1897
Table 3.1: Information Retrieval System
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 12 sdey1@sheffield.ac.uk - 100151932
digital documents. Matching may be carried out in either of the following ways (Baeza-
Yates and Ribeiro-Neto, 1999, Rijsbergen, 1979):
Boolean Matching
Vector-based Matching
Fuzzy Matching
Probabilistic Matching
There is a considerable amount of benefit in indexing and saving the internal
representation of this collection, for efficient querying with time. Appropriate precision
metric and recall metrics are used to order the fetched results in accordance to the
information need and relevance of documents (Hoeber, 2007). Studies carried out
earlier, acknowledged feedback as the measure of improvising effective retrieval
(Rijsbergen, 1979), however gradually it transcended to user’s crafting of better queries,
and slowly further extending to interactive and automated interfaces, such as query
expansion (Efthimiadis, 1996).
Contrary to traditional systems involving queries and ways to improve precision (Baeza-
Yates and Ribeiro-Neto, 1999), several researches have slowly been carried out in this
area where focus was given towards researcher’s motivations, such as Bates (1989), who
characterised information sources and search tools as ‘evolving berrypicking’, where
digital information search comprised of multiple successive queries in the information
space. Quite similarly, Kuhlthatu (1991) exclaims in his paper that information seeking is
not just about the queries, but rather a dynamic process, that involves the user’s
‘thoughts, feelings and actions’. In another theory suggested by Pirolli et al. (2001), called
information foraging theory, it is said that while searching, “humans are drawn
towards groups of relevant items in an analogous way to animals attempting to locate
dense patches of food in the wild”. Besides, researchers have also carried out studies to
automate digital document collection into clusters or groups, according to some
common words that correlate them (Rodden, 2002), which was thought to be more
efficient than matching documents individually using queries (Maarek et al, 1991;
Jardine and Rijsbergen, 1971); however this has not been proved (Willet, 1988). The
cluster hypothesis articulates that:
“Closely associated documents tend to be relevant to the same requests”
(Rijsbergen, 1979), or in other words,
“Relevant documents tend to be more similar to each other than non-relevant
documents” (Hearst and Pedersen, 1996).
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 13 sdey1@sheffield.ac.uk - 100151932
As user requirements related to finding an information may often also be vague (Rodden,
2002), Pirolli and his associates demonstrated that clustering presented the users with
an overview of related retrieved information(Pirolli et al, 1996), which was further
confirmed in theory by Hearst and Pedersen (1996) using Scatter/Gather system
(Cutting el al, 1992). Kural et. al (2001) states that the effectiveness of query clustering is
however dependant on the understanding by the users and the compactness in the
representation of the clusters. Besides, according to information foraging theory, the
measure of information scent, defined as - “the strength of local cues ... in providing an
indication of the utility of relevance of navigational path leading to some digital
information source” (Pirolli et al. 2001), is also determined by the information clustering
quality. However, according to Furnas (1997), traditional querying is always a good
point of starting as it is often hard to ascertain a quality information scent.
3.2.2 Web-based Information Searching
Retrieving information from the universal hypertext space has amalgamated deeply with
our society today (Lawrence and Giles, 1999) with a reported statistic of around 88%
users using the Web search engines to carry out certain tasks (Nielsen, 2004).
Additionally supported by previous studies, a reported 85% new Web pages are
reported to be found using search engines (Graphics, Visualization, & Usability Center,
1998). According to Nielsen, Web search engines have slowly transformed over the years
into ‘answering engines’, with people generally harvesting specific information(Nielsen,
2004), without deeper exploring into individual web sites, guided by their natural and
instinctive ‘low tolerance’ level (Spink et al., 2001). This is evidently due to the
magnanimity of the ever growing collection being added every second and ‘potential
generality of individual documents’ (Hoeber, 2007).
Web-based information retrieval is said to have always been considered hand in hand
between searching and browsing (Bates, 1989). This kind of information gathering may
often involve crafting several queries which might tend to be complicated with
expanding information sources, considering innumerable individual and unknown items
that are on the rise. Although there have been considerable improvements in the
performance and efficiencies of search engines over the past few years, instances of
insignificant and irrelevant documents, surprisingly often in the top 10 to 5 list, are still
on the rise (Chau, 2011). We might consider the Web as a single distributed repository of
information collection, however traditional information retrieval approaches are said to
be practically inefficient to handle it (Yang, 2005). According to Yang, retrieving
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 14 sdey1@sheffield.ac.uk - 100151932
information from the Web “must deal with mostly short and unfocused queries posed
against a massive collection of heterogeneous, hyper-linked documents that change
dynamically” (Yang, 2005), which is a well defined challenge compared to traditional
retrieval systems.
Web search-engines traditionally assist the users with a brief summary of the pages, in
the presented list of retrieved results, which allow the users to perceive the contents, so
that they could click to browse it and obtain the potential information that they seek.
Apart from highlighted keywords (McDonald and Chen 2006), further data inform of
page popularity, relevant score, frequency of keywords etc. provide additional assistance
to the users to search their required information (Brin and Page 1998). One good
example in this respect is Google search engine’s PageRank algorithm (Brin and Page,
1998). These attributes, comprising of textual or numerical values, help organise the list
of retrieved results and ensure directing the users’ perception towards the quality and
relevance of the material, so that they can make a better judgement in utilising their time
and effort in exploring them further (Chau, 2011). Once again, as demonstrated by
Spink, Wolfram, Jansen, and Saracevic that users generally have a “low tolerance of going
in depth through what is retrieved” (Spink et al. 2001), and hence, this may sometimes
lead to be an exhaustive and often futile if they rely only on a subset of such attributes in
making a judgement and go through links and documents irrelevant to their queries
(Chaur, 2011).
Given the scope of this dissertation, since this piece of research is not directed deeper in
this field, for further interest reader may follow further papers, journals and books in
this area (Baeza-Yates and Ribeiro-Neto, 1999; Kobayashi and Takeda, 2000; Kosala and
Blockeel, 2000; Rasmussen, 2003; Yang, 2005).
3.2.3 Information retrieval of large faceted datasets
In order to retrieve information from large datasets, Smith et al (2006) has suggested the
use of interactive exploration through ‘categorisation’ or ‘browseable hierarchy’. Such
a technique is said to leverage the power of log (n) to quickly retrieve the required item
from a large aggregated set of items. One such example is the Yahoo search portal
(dir.yahoo.com) shown in Fig 3.1.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 15 sdey1@sheffield.ac.uk - 100151932
However, one drawback of such a system is that it is difficult to come to a conclusion
regarding the best set of categorisation, as the facets may be evolving. Several other
studies have been carried out (Zeng et al., 2004; Sanderson and Croft, 1999; Cutting et
al., 1992; Pirolli et al, 1996) to handle large datasets using automatic clustering, that can
be said to have some relation with categorisation (Smith et al, 2006). According to Smith
and his colleagues, such techniques are also apparently target-oriented instead of
matching textual similarities. But, some still provide additional facilities such as dual-
mode interfaces, by clustering the search information only when initial keyword
retrieval is completed (Sheng et al., 2004).
To refine categorisations in the representation of data collections, the use of ‘faceted
metadata’ is seeing an increasing importance recently (Smith et al., 2006). ‘Facets’ are a
set of orthogonal categories that represent the attributes or metadata of the data
collection. For example, a historic collection of photographs may have thousands of
images with metadata, such as ‘location’, ‘year’, ‘originator’ etc. These can be categorised
into facets and represented visually. In a system by Yee et al. (2003) called Flamenco
system, an efficient and interactive user experience has also been demonstrated through
faceted metadata integration into dynamic-query interface. Besides, another system
called Phlat (Fig 3.2) (Curtell et al, 2006), as emphasised information retrieval using
faceted interface through user- tag integration.
Fig 3.1: Yahoo Directory
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 16 sdey1@sheffield.ac.uk - 100151932
Several organisations have also capitalised such a technology and have commercialised
them:
i411 (now IntelligenX - www.intelligenx.com)
Endeca (www.endeca.com)
Inxight (now owned by SAP)
FacetMap (Smith et al., 2006) also demonstrated several features in the use of facets
including query previewing and top-level organisational interface system (Fig. 3.3). But
unlike most other interfaces, it also has dynamically adjustable screen space allocation
based on the retrieved items.
Fig 3.2: The Phlat interface with a query of a single keyword and two filters
Fig 3.3: FacetMap Interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 17 sdey1@sheffield.ac.uk - 100151932
3.2.4 Web User Interface
A certain area worth mentioning, however, which apparently has a comparatively lacking
attention in this area of information retrieval on the Web, is the research on interface.
(Hoeber, 2007). Although the user interface is the main access point of the system, there
is a dearth of recognition on the significance of the functionality and features of the user
interface, in terms of providing the users a proper means to carry out their queries or
filter their retrieved results. Apparently, a simple text box where users can feed their
queries followed by a section below that displays list-based search results have been
mostly prevalent amongst all major engines such as Google, Yahoo etc. There are not
many studies, however, if such a feature is actually adequate enough to assist the users
(Hoeber, 2007).
According to Marchionini (1992),
Much research and development is required to produce interfaces that allow end
users to be productive amid such complexity and that protect them from
information overload.
His remarks also render further impetus in this study on interactive information
retrieval and information visualisation, which shall be carried out through Microsoft
PivotViewer particularly in this study.
” “
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 18 sdey1@sheffield.ac.uk - 100151932
3.3 Information Visualisation
Information visualisation is process of representing either abstract data, static or
temporal information and related concepts in the form of a graphical and visually
attractive form (Ware, 2004; Spence 2001). The prime aspect of information
visualisation is, to accelerate human cognitive understanding, by capitalising on their
visual information processing capabilities (Spence, 2001; Shneiderman, 1996). In other
words, information visualisation provides a bridge between the computer interface
systems and the human vision (Zhu and Chen, 2005) and catalyses them with tools that
amplify their cognitive intelligence (Chen 2004, Spence, 2001; Card et al, 1999;
Shneiderman, 1996). Information visualisation facilitates in representing a large or
complex set of abstract data in a pellucid and consistent manner, by giving the user the
possibility to explore and compare the inter relations between the various data visually
(Tuft, 1990).
Dimensions and attributes that may be associated with the data such as – spatial
coordinates, shape, colour etc. are required to be carefully considered, while designing
the interface (Hoeber, 2007) so that they can be perceived easily by the user. Ware has
suggested some more excellent information on this topic in his book, especially
regarding spatial location and colour of objects as compared to that of the shape and size
of glyph (Ware, 2004). Apart from that, care must also be taken so that the users are not
burdened with non-essential visual complexities in terms of perception and
interpretation. According to Tufte (1997), the design strategy of the interface should be
of the smallest effective difference; i.e. recommends making “all visual distinctions as
subtle as possible, but still clear and effective”. (Tufte, 1997).
Besides, the selection of proper colour scheme, which is quite an important criterion for
encoding valuable information as well as appeal to the interface, poses a subtle challenge
as it may lead to an increase in noise if overused in the interface or the retrieved
information space. As Tufte rightly points out, it might lead to a “visual war with the
heavily encoded information” (Tufte, 2001), if not carefully used. Tuft has suggested
many more graphical and related principles in relation to designing (Tufte, 2001, 1997,
1990), apart from many other authors emphasising the use of proper understanding of
colour as a theory (Ware, 2004; Stone, 2003).
D
Definition: “The use of computer-supported, interactive, visual representations of
abstract data to amplify cognition”
- Card, Mackinley and Shneiderman (1999)
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 19 sdey1@sheffield.ac.uk - 100151932
Opponent process theory (Ware, 2004), states that, there is a direct inference of usage
of colour in information systems visualisation. To delve further, according to this theory,
six primary colours are organised (Fig. 3.4)
perceptually as pairs along three axes or
channels: red-green, yellow-blue, black-white
(luminance), which on varying monotonically
across one or more chromatic channels, is said
to produce an ordered colour scale (Ware,
2004, Tuft, 2001). Apart from that, varying the
luminance also helps in creating the
perceptions of depth, motion and form in the
system display.
However, to additionally help in the cognitive assistance through an effective visual
presentation, the ability to have an interactive manipulation with the retrieved visual
data is an intrinsic component of a good information visualisation system (Card et al.,
1999). But to enable the viewers to maintain their control on the system at the same time
ensuing that the system generates essential feedback associated with the user’s actions is
always a challenge. (Benderson and Shneiderman, 2003).
Over the years, there have been several researches carried out on visualisation in the
area of information seeking. A collection of major studies have been grouped as follows:
Author(s) Area of study
Mackinlay et al (1991) 3D interfaces for focussed context views
Robertson et al. (1991) Hierarchy visualisations
Card et al. (1996) Windows layouts
Ahlberg. and Shneiderman (1994)
Visual information seeking – filters with visualisations
Jones (1999) and
Spoerri (1993) Interactive visualisation of query terms
Hearst (1995) Per-result search visualisation – integrating document-query similarities into result lists
Clarkson et al. (2009) ResultMaps – visualised context overview for retrieved results
Wise et al. (1995) Spatialization in representing large document collections
Lee et al. (2009) and Stefaner et al. (2008)
Amalgamating visualisation with faceted dataset navigation.
Fig 3.4: Primary colours as described in
Opponent Process Theory
Table 3.2: Collection of studies on Information Visualisation
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 20 sdey1@sheffield.ac.uk - 100151932
3.3.1 Web-based Information Visualisation
Today, there is a growing tendency in the hypertext environment, to cross beyond the
traditional ’10 blue links’ (Broder et al, 2010) and embrace a rather exploratory and
interactive search methodology (White, 2006). But only a few interesting studies have
been carried out in this domain to date. Faceted search navigation (English et al., 2002;
Hearst, 2009) have been introduced to provide an exploratory navigation facility to users
for searching without explicit queries. FacetLens (Lee, 2009) and FacetMap (Smith,
2006), shown in Fig. 3.4 and Fig. 3.6 respectively, are notable researches in this area that
can be said to have lead to the evolution of PivotViewer tool that this dissertation shall
be evaluating.
There may however be a tendency to reach an aspect of cognitive overload due to
complex faceted techniques (Wilson and Schraefel, 2008), but at the same time, there
have been much effort in directing to enhance them with effective visualisations
(Stefaner et al, 2008).
Several studies based on information visualisation have demonstrated that users are
inclined to perceive information and carry out tasks more efficiently if proper
visualisation techniques are incorporated in the system (Carter 1947; Pinker 1990;
Hearst 1995; Xiang et al. 2005). According to Buja and his colleagues, research in the
area of visualisation encompasses two main areas (Buja et al., 1996):
Rendering: involved with construction of visual material (graphs) based on
information, and
Manipulation: iterative process utilised by the users to manipulate the data and
re-render the visualisations (Kumar and Benbasat 2004).
It can be said that these also form some of the core concept of PivotViewer where users
can filter datasets visually.
Fig 3.5: FacetLens interface Fig 3.6: FacetMap interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 21 sdey1@sheffield.ac.uk - 100151932
Shneiderman also proposed a similar structure that classified graph construction based
on the data for visualisation, which had seven categories depending on the data: one-
dimensional, two-dimensional, three-dimensional, multidimensional, temporal, tree, and
network (Shneiderman, 1996). Besides, Turetken and Sharda also suggested a structure,
especially for the Web, that classified the visualisation depending on dimensions
(Turetken and Sharda, 2007) such as:
Area Author(s) Related work
Semantic content Metadata
Brin and Page (1998); McDonald and Chen (2006)
URL snippets
Salton (1989) Tokenization and indexing algorithms for web documents
Hasan et al. (1995) MultiSurf system – relating search results to search queries
Sebrechts et al. (1999) NIRVE system – 2D and 3D search result representation system
Roussinov and Ramsey (1998); Chen et al. (1998, 2003)
Self-organising map
Turetken and Sharda, (2005, 2004) Fisheye view
Connectivity structure
Munzner (1998);
Mak et al. (2002); Pirolli et al. (2003)
Application of web document connectivity structures
Ben-Shaul et al. (1999) Fetuccino system (IBM) – visualising connectivity of search results
Chen et al. (2002) CI Spider Systems that have directly connected Web documents
Chau et al. (2007) Redips
Mukherjea and Hara (1999) Card-Vis
Metadata
Chernoff (1973) Introduced gylphs – multidimentional visualisations based on metadata.
Chernoff (1973); Scott (1992) Chernoff Faces – representing facial features
Kleiner and Hartigan
(1981); Chuah and Eick (1998); Ebert et al. (1997); Xiong and Donath (1999); Roberts et al.
(2002); Fanea et al. (2005); Forsell et al. (2006); Wiza et al. (2003); Cellary et al. (2004)
Various researches on gylphs
Scott (1992); Rohrer et al. (1998); Sangole and Knopf
(2002)
Visualisations on economic data, weather information,
and text documents
Table 3.3: Categorised visualisation dimensions and related studies
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 22 sdey1@sheffield.ac.uk - 100151932
3.3.2 Visualisation of faceted datasets
Over the years, through various studies and research, information visualisation has
slowly emerged into two directions (Smith et al. 2006):
1. Static visualisation – comprises of a set of pre-generated visualisation for a large
data collection.
2. Dynamic visualisation – generation of a visualised result based on user query,
generally on the actual large data collection.
Clustering technique, as discussed earlier, promises to enable effective visualisation
along with an essence of interactivity and manipulation, along with an amalgamation of
introducing a possible 2D or 3D interactivity, such as, the Galaxy of News by Rennison
(1994). Apart from the drawbacks of clustering previously mentioned, its
unpredictability and originality in interactivity, is also an added issue. The dynamically-
clustered, force-directed visualisation was developed by Zhang et al. (2002), which was
however rendered unsuitable for end-user study. ThemeScape (Hetzler et al., 1998),
designed to spatially visualise large document collections by similarity and an analogous
system by Fabrikant (2001), showed that users comprehend an in-depth hierarchy when
they zoom into the spatial information space. Smith et al. (2006) also demonstrated a
similar interface called FacetMap that tried to arrange large collection hierarchical
metadata spatially instead of a clustered document similarity.
Accumulation of an unsorted data collection necessarily abstracts essential details if a
particular axis is chosen, along which the individually grouped items could be collapsed
(Smith et al., 2006). However, studies have suggested methods to scale visualisations to
content heavy databases with lossless compression techniques in information
visualisations (Fekete & Plaisant, 2002; Jerding & Stasko, 1998). But in order to allow
dynamic transformation for user interactions of these techniques, which is limited by
static pre-processing of the individual dataset elements from the system memory, a
native GPU (Graphics Processing Unit) could be exploited (Fekete & Plaisant, 2002).
However, a downside to this as pointed out by Smith et al. (2006) is that, the
visualisation tends to be rather less dynamic with the transferring of visualisation to the
GPU and reducing per-item information from the system.
Besides, there have also been studies on visually representing search queries of large
datasets using the concepts of dynamic queries (Ahlberg et al, 1992), such as Grokker, as
shown in the Fig. 3.7
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 23 sdey1@sheffield.ac.uk - 100151932
In most of the above literature, visualisation is very domain-specific, meaning that the
attributes or metadata has to be created in advance and it is difficult to import a new
dataset or facet item (Smith et al, 2006), one exception being Relation Browser++
(Zhang, J., and Marchionini, G., 2005), which presented facets along with their metadata
values and integrated graphics to visualise data distribution, and it could also be
interactively manipulated with mouse-over effects. Amongst the domain-agnostic
visualisation tools, Sportfire (Ahlberg, 1996) and Polaris (Stolte et al, 2002) can be
considered as sophisticated database exploration interfaces, that allow high-end
manipulation and usability through the facets, with the assumption that the users are
expert in their domain.
3.4 User Behaviour – Web Searching
There have been several studies over the years to evaluate user behaviour patterns in
Web search. Most notable studies in this area are:
Author Related Research
Silverstein et al. (1999) Query log analysis of Web search
Spink et al. (2001) Analysis of public and their queries
Jansen and Pooch (2001) Web searching review
Fig 3.7: Grokker - Information visualisation
Table 3.4: User behavioural pattern studies
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 24 sdey1@sheffield.ac.uk - 100151932
Such studies have demonstrated numerous behavioural patterns of the users’
capabilities in formulating search queries. Results from such studies have indicated that
users generally tend to carry out queries in one or two short terms and very rarely
formulate queries that contain over three terms (Jansen and Pooch, 2001; Spink et al.
2001). Besides, it should be pointed out that only a rare use advanced searching, yet with
incorrect procedures (Spink et al. 2001). At the same time, users hardly tend to modify
or refine queries (Spink et al. 2001; Silverstein et al., 1999).
Marked by a dearth of inclination to evaluate further search results on the Web, it was
also identified that users rarely considered browsing more than three pages for
obtaining their results. (Spink et al. 2001; Silverstein et al., 1999). Besides, it has also
been found that due to their low tolerance level (Spink et al. 2001), users avoided much
navigation and scrolling and had a higher tendency to look for documents from the top
results in the retrieved lists (Kobayashi and Takeda, 2000). In addition, Yang (2005)
concluded through his studies, that users generally tend to expect immediate answers
using minimum effort. However, he did not expand on why they demonstrated such
behaviour. Much of it could be attributed to the lack of interactivity of retrieval process
and visualisation of the search results.
Besides, current search interfaces also lack the ability to help users in vague situations,
with no proper assistance in articulating their needs and directing towards a solution
(Marchionini, 1992). Ideally, there should be a higher cognitive search burden on the
system rather than the user (yang, 2005) and researchers have predicted the possibility
of developing even better interfaces over the coming years, in visually interactive
information search interfaces (Kobayashi and Takeda, 2000).
According to Ferrara (2008), a Web and information architect with over a decade of
experience, the user behaviour in information searching could be attributed to the
following six categories:
1. Domain expertise – Users having the familiarity in a subject are said to have a
better ‘verbiage’ or expertise to search for information in that particular domain.
Hšlscher & Strube (2000) has studied this characteristic through a Web search
behaviour and Bhavanani (2002), through his domain specific studies on users’
have thrown a further light on this.
2. Search experience - Search experiences and technicalities of using search
refinements (e.g. - using Boolean operators), have also indicated to play a vital
role, which is however not essential. Ferrara points out that, those with lower
technical knowledge of searching but greater domain knowledge can also obtain
results easily, as compared to technophiles having lower domain knowledge
(Ferrara, 2008).
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 25 sdey1@sheffield.ac.uk - 100151932
3. Cognitive style- The trends in gathering new information, also has an influence
in the way users explore information. Researchers such as Nigel et al. (2002)
have proposed several cognitive styles and associated them with schemas.
However, Ferrara (2008) has suggested the existence of the spectrum comprising
of:
a. Global thinkers – who tend to have a broader overview and
b. Analytical thinkers – those who have a narrowed down search approach
on a particular criterion.
To further support this, White and Drucker (2007) also indicate that users
generally tend to leverage themselves somewhere within the extremes of the
above spectrums, but mostly with a closer inclination to either of them.
4. Global type –Broader (2002) has suggested that search patterns can also be
categorised according to their goals:
a. Navigational Search- which seeks location oriented data.
b. Informational Search – which seeks documents on particular domain.
c. Transactional Search – which fulfils the user to accomplish an online
activity.
5. Mode of seeking – When users use an information retrieval interface, situations
can be primarily be of the following categories (Rosenfeld & Morville, 2002;
Spencer, 2006):
a. Searching for a known item – when users are aware of exactly what to
extract or look for from the information space;
b. Exploratory searches – where users have a broad awareness of what they
want and not any precise set of criterion;
c. Vague or when users are unaware and have no idea of what to look for.
(expressed in further details by Bates, 1989 on her ’berrypicking’ article).
6. Situational Idiosyncrasies - Ferrara (2002) adds a penultimate layer of
unpredictability in user search behaviour, by suggesting that a similar search by
a particular can have contrasting results depending on two or more different
occasions depending on the idiosyncrasies of the user. According to him, users
tend to respond with a different search approach depending on the situations for
retrieving information from a particular search interface, e.g.- a user on a
leisurely mood will tend to have a more casual approach and end result or
information through their search on a particular topic, as compared to another
given time if the situation was quite pressing such as an approaching deadline.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 26 sdey1@sheffield.ac.uk - 100151932
3.5 Summary
In this chapter information retrieval techniques have been thoroughly discussed with
relation to major works done in this area. Particular focus was highlighted upon
explaining the studies carried out on traditional information retrieval systems and Web-
based information searching. Special attention was also given, to carefully study the
findings in the area of information retrieval of large faceted datasets due to its close
association with the technology employed in PivotViewer. Apart from this, a brief
discussion of Web user interface has also been carried out. Besides, studies related to
information visualisation was also thoroughly evaluated with special focus and
discussions made in the area of Web-based information visualisation and visualisation of
faceted-datasets. Finally, user behaviour in Web searching has been briefly indicated,
along with an overall mention of related studies by researchers, denoted in tabular form
for certain topics.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 27 sdey1@sheffield.ac.uk - 100151932
Chapter 4
4.1 Introduction
PivotViewer (or simply Pivot) is a product of Microsoft that works in the Silverlight 4
framework as a control, when hosted inside a Web script. It is said to allow the users, to
visualise and explore large collections of data, in an interactive way. It uses a Web app
framework or a plug-in, which also has an API that can pass data between the control
and a hosting website.
According to a demonstration by Microsoft Silverlight Team (2010), this control is a
“multifaceted multimedia experience of a data set that provides an opportunity to look at
trends within the data, and kind of observe larger forces at work and drill down into
specifics, and then formulate hypothesis, so that users can move from the data to
something closer to knowledge”.
Pivot uses Microsoft Deep Zoom technology in a Silverlight interface, to browse through
collections of data, by allowing the users to zoom in and out of the collection, as well as,
change the facets of the metadata, that the user is interested in, filter out sets and bring
them back in and analyse the individual items, while all the time having a broader
picture. It allows users to sort, organise and categorise data dynamically. Developers can
either implement the full application experience, that takes advantage of the Pivot or
they can embed just a frame of the
pivot control into an existing
website, with a customisable
control, that can obtain all the
technology required to consume and
interact with pivot collections, using
Silverlight. In this Chapter, the
technology of PivotViewer is
discussed in details.
MICROSOFT PIVOTVIEWER
Fig: 4.1: Example of a Pivot Interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 28 sdey1@sheffield.ac.uk - 100151932
4.2. PivotViewer Collection
As defined by the Microsoft Silverlight Team (2011), the collections in Pivot is said to be
composed of two parts:
CXML – Collection XML Schema – which is an XML file (with a “.cxml” extension)
that contains the detailed descriptions of the items in the collection, and
Collection Image Content – A set of arranged images in a Deep Zoom format
There are three discrete steps to create a Pivot collection:
1. Data collection – A set of data needs to be collected, that is to be turned into a
dataset and represented. [More details in Section 4.3-Collection Design]
2. Creating CXML and Image rendering – After the data sources have been decided,
it needs to be formulated into CXML (or Collection XML). The images will also need
to be transformed into a Deep Zoom Format. There are a variety of tools to achieve
this, from MS Excel Pivot plug-in to open source libraries. [More details in Section
4.2.1 for CXML schema, Section 4.2.2 for Collection Image content ]
3. Collection Hosting – After creating or building the collection, it needs to be hosted
into a server according to accessibility requirements.
Pivot Architecture
A collection in Pivot is similar to any particular Web content. The set of relevant files on
the server is accessed by the local client, which understands the format it needs to be
displayed. In traditional interfaces, these files are generally HTML and images (apart
from any other scripts). However, in Pivot, these files are CXML and Deep Zoom
Collection (DZC) images. Users can access this collection from the hosted web page,
where the PivotViewer displays the content, using Silverlight Control.
Fig 4.2: Pivot Architecture
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 29 sdey1@sheffield.ac.uk - 100151932
4.2.1 Collection XML Schema (CXML)
CXML or Collection XML is the main data structure or in higher level, a collection of
property or value pairings, of the data/image collections that attributes to the experience
provided within the PivotViewer. Use of categories as filters in the search interface of
PivotViewer is perhaps the most important and underlying characteristics of
PivotViewer. This is achieved through ‘facets’ that can be grouped into ‘facet
categories’.
A simple example of a CXML file is provided below which results in the visualisation of a
single data:
The above code results in the visualisation of a single data in the interface as shown
below. (Note: This need to be run either through the PivotViewer Application Tool
provided by Microsoft Live Labs or via an ASP.NET server – which shall be discussed in
details shortly in the Chapter 5 - Methodology). Each section of the above code
mentioned above has been dissected and presented describing their functionalities:
<?xmlversion="1.0"?>
<CollectionName="Hello World Collection"
SchemaVersion="1.0"
xmlns="http://schemas.microsoft.com/collection/metadata/2009"
xmlns:p="http://schemas.microsoft.com/livelabs/pivot/collection/2009"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<FacetCategories>
<FacetCategory Name="Hello World Facet Category One"
Type="String"/>
</FacetCategories>
<Items ImgBase="helloworld.dzc">
<Item Img="#0" Id="0"
Href="http://www.getpivot.com"Name="Hello World!">
<Description>This is the only item in the
collection.</Description>
<Facets>
<Facet Name="Hello World Facet Category One">
<String Value="Hello World Facet
Value"/>
</Facet>
</Facets>
</Item>
</Items>
</Collection>
Source: Microsoft Silverlight Team (2011)
Fig 4.3: Simple example of a CXML file
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 30 sdey1@sheffield.ac.uk - 100151932
4.2.1.1 Structure of schema
The collection of PivotViewer is a combination of a set of image collection associated
with a CXML file. The beginning of the CXML file comprises of a top-level node called
Collection, which defines the overall properties of all the elements in the set of
collection.
The Collection node has two main children which has further sub-children (Microsoft
Silverlight Team, 2011):
1. FacetCategories - This is one of the immediate children of the Collection
container. Every facet linked with an item should have an associated face
category. It consists of the facet name, string format, visibility options and type of
the facet, which may either be Number, String, LongString, DateTime or Link.
Fig 4.4: Simple Pivot Interface based on the previous sample CXML code
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 31 sdey1@sheffield.ac.uk - 100151932
2. Items – it contains the definition of every individual item in the collection and
thus contains most of the information related to the collection in the file. Each
item constitutes of the following properties:
a. Href – a link that redirects from a double-click
b. Name – a name given to the item
c. Description – description associated with the items
d. Id – an identifier associated with the image or information
e. Img – the image asset
f. Any other custom facets or properties that might be necessary to include.
However, in this connection it must be noted that all the above properties are
always not necessary and can be omitted if necessary, e.g. - we may not want the
users to double-click on the picture and get redirected to a link and hence we
may completely remove the Href property.
Fig 4.5: Structure of schema
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 32 sdey1@sheffield.ac.uk - 100151932
4.2.2 Collection Image Content
One of the most significant characteristics of PivotViewer is the use of images through a
technique called Deep Zoom format. In the initial section, an overview of Deep Zoom
Images and Collections are discussed, followed by a quick guide on how to build such a
collection for PivotViewer. The later sections will cover information regarding the tools
and technologies that can be used in order to author the images, the encoding
considerations for the images, and finally on how to deploy the deep zoom images on
production servers.
4.2.2.1 Deep Zoom Image Collections
Overview
The primary form of display of images in PivotViewer consists of a Deep Zoom Collection
format that comprises of individual Deep Zoom images. In order to achieve this, the large
dataset set represented by the collection of random sized images are pre-rendered and
tiled to allow a progressive access. There are various tools and techniques that can
render the images to this format.
The individual images are encoded into a set of Deep Zoom Image (DZI) tiled pyramid.
These sets of images are then encoded into a collection called Deep Zoom Collection
(DZC) and are represented in a layout as described by MSDN called ‘Morton Layout’.
More information about Deep Zoom file format and Morton Layout can be
obtained at: http://msdn.microsoft.com/en-us/library/cc645077(VS.95).aspx
Fig 4.6: DZI – DZC based on Morton Layout
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 33 sdey1@sheffield.ac.uk - 100151932
The overview of file organisation of DZI and DZC are briefly discussed below:
Format Overview of file organisation
Deep Zoom Image (DZI)
1. XML file containing details of the collection.
2. Subdirectory containing multiple subdirectories having
the images pre-rendered in various resolutions. It
contains the same name as the XML file followed by
“_files”, e.g. – if the XML file name was abc.xml, the name
of the subdirectory will be abc_files.
Deep Zoom Collection (DZC)
1. XML file containing details of individual DZI images in
the collection.
2. A subdirectory containing the DZI images with the
naming structure similar as above.
Sparse Images
PivotViewer also has additional support for sparse images,
i.e. the images that has varied resolutions in various areas,
which can be said to have been generated by forming
multiple images at varied scales of resolution.
DZI and DZC content creation
There are several methods to create the Deep Zoom images for PivotViewer collection.
However, the suitability of each method is dependent on whether the collection or the
production is either a onetime build or part of a continuous or dynamic work flow. A
quick summary of the procedure is provided below:
Procedure Description
Pivot Collection Tool – MS Excel Plug-in
This is perhaps the simplest procedure to generate pivot
collections from scratch. The Excel spreadsheet is used to
store the facet information in the form of separate
columns. A column contains all the information of the
source of all the images that need to be generated
corresponding to the other associated fields. DZIs and
DZC can be automatically published and created using
this plug-in.
Manual Deep Zoom creation
CXML file can be manually created from the data
available in any particular dataset by using tools to
export it in XML. This could be achieved by using either
Deep Zoom Composer or Deep Zoom command line
utility, which can be downloaded freely. Both the above
tools can also be automated using shell scripting.
Custom Applications
In order to accommodate a particular domain of
workflow, custom applications can also be used. In order
to achieve this, the Deep Zoom Tools Library can be used
to create DZI and DZC as Windows application.
Table 4.1: Organisation of DZI and DZC
Table 4.2: Summary of procedures to build Pivot Collection
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 34 sdey1@sheffield.ac.uk - 100151932
Image Compression Guidelines
The formats for the images that can be used to create the DZI and DZC collections can be
of JPEG or PNG formats. However, each of these formats have their associated
advantages and disadvantages and hence it should be appropriately chosen, based on the
system and the nature of work flow, such that an optimised experience is achieved.
Hence a proper balance must be maintained between in the compression or the quality
of the images based on the processing speed of the server. It should be noted while
adjusting the image settings, that an increase in the compression can improve the
performance of the interface but at the same time it can also degrade their quality.
Based on the formats, here is a comparative chart between JPEG and PNG formats:
JPG PNG
Compression Lossy Lossless
Continuous
tone images
Provides efficient
compression
Provides inefficient
compression
Suitability Best for web based images
Appropriate for simple
diagrams, texts, single
colour images
Alpha channel Has no support
Supports alpha channel
meaning it can support
images with transparency
Quality Lower compared to source Comparable to source
Image size Smaller Larger (alpha channel
increases it even further)
Experience Faster due to lower
bandwidth usage
Slower due to higher
bandwidth usage.
While both the above formats have their respective advantages and disadvantages, it is
always advisable to experiment with a set of collection and carefully evaluating with the
system and nature of the content available. Compression parameters can be adjusted to
accommodate suitability with the system and keeping in mind with the quality of the
images obtained. There are no rules in this regard and the optimal configuration can only
be achieved through experimentation based on the efficiencies of the system.
However, it must be mentioned here, that for maximum compatibility and efficiency,
consistency must be maintained in the format of the images used for the particular
collection.
Table 4.3: Comparative chart between JPEG and PNG formats
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 35 sdey1@sheffield.ac.uk - 100151932
4.2.2.2 Deploying Deep Zoom Collections
Overview
There are no special Web services which are separately required for deploying Deep
Zoom Collections. The XML file along with the associated image files in the sub-
directories can be accessed using traditional HTTP protocols.
Working with several images based on Deep Zoom Collections can lead to high volumes
of files and sub-directories that are necessary to generate the tile pyramids. This can
attribute to the total space necessary to accommodate all the files at a much higher size
than the actual source. As such, working with thousands of files can prove to be quite
challenging and hence there are some suggested techniques by Microsoft Silverlight
Team to help deploy Deep Zoom Collection to a web server:
1. In place generation: This technique suggests in rendering the Deep Zoom
Images(DZI) and Deep Zoom Collection(DZC) directly into the server that
contains the set of image contents, thus avoiding the hassle in transferring them
to another hosted location. Hence it is advantageous as a lot of time can be saved
in preventing the duplication of large set of files and sub-directories. However, it
may not be feasible in scenarios where the production server provides no
support for the execution of rendering process, for proper verification before a
final deployment.
2. Content Reduction: The content at the top position of the DZI pyramid, which is
also included in the DZC pyramid, is often duplicated. As such, for every DZI
pyramid, the files and sub-directories that make the tip can be removed. For each
content present in the collection, a maximum of nine files and directories can be
deleted. This significantly reduces the total size as well as number of images. At
the same time, several tools used to create Deep Zoom contents also allow in
configuring image dimensions, which can be used to set up proper image sizes
and effectively reduce some memory.
3. Creating ZIP: Custom generators that are used to build contents can be used to
render DZI and DZC directly into ZIP files instead of individual images and
directories. This significantly reduces the time required to deploy a single or a
few ZIP files, as compared to hundreds and thousands of individual directories
and files. However, it should also be noted that, it is also required to un-ZIP the
files into the appropriate directory formation on the production server to
facilitate the interface to deploy the collection.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 36 sdey1@sheffield.ac.uk - 100151932
Updating Collections
Users are able to view the Pivot Collections in the browser through the embedded CXML
within the body of the website, which is cached by the application in the server. As a
result, any updates directly on the collection present the server on collection with which
the CXML is linked, will only invalidate the collection and throw exceptions, creating
inconsistencies in the results obtained.
Hence, to properly update a Deep Zoom Collection, a new collection has to be generated.
This collection should then be deployed at a different URL, but in this connection, it must
be also mentioned that the URL should contain some identifier that can distinguish it
uniquely, such as version number or timestamp. This new URL can then be hosted into
the webpage and updated. The URL indicating the CXML file may remain the same, only
the URL inside the CXML file that points to the collection needs to be updated. This
procedure ensures that the users who are currently viewing the Pivot collection page are
not disrupted, but at the same time, new visitors are able to interact with the updated
collection.
If there are new Deep Zoom image files deployed and no other changes made for any of
the other individual Deep Zoom images, updating only the Deep Zoom collection will
make the necessary changes instead of re-rendering the whole collection and deploying
it. This also enables the PivotViewer to reuse other images in the local cache of the
system, which were rendered in Deep Zoom format previously.
Caching Collections
Though the web server treats the Deep Zoom collection as a set of static items, an
optimal efficiency can be obtained by active caching. This can be achieved by allowing
the DZC an unlimited cache, which will ensure significant reduction of multiple content
access permissions along with arbitrary proxy servers, by serving the requests via the
caches, also enabling load reduction in server at minimal bandwidth costs.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 37 sdey1@sheffield.ac.uk - 100151932
4.3 Collection Design
The challenge in designing a large collection is in deciding the technique to present the
entire information collection at once. They prime factors in designing a proper set can be
segregated into three steps:
Collecting data or subject matter with quality details and/or images
Deciding on the categories to create filters on
Linking the relevant contents
A proper design comes with several iterations starting from a simple set to gradually
increasing in complexity.
The steps in this process are discussed below:
1. Define the Subject Matter: As an initial step, the theme or topic of the collection
needs to be decided. It should be kept in mind to include such a subject, so that
users would be keen to explore more, instead of just looking for a particular data.
This is necessary to lead the users into discovering trends or patterns from the
vivid views, across all the contents.
2. Define the User: Secondly, users should be defined of the collection requirements
according to the needs. For example, a photograph enthusiast would look for
different contents in a camera collection than someone who is just shopping for a
camera.
3. Define the Key Tasks: Thirdly, the key tasks, whether it is goal oriented or general
browsing, that shall be carried out by the users in the interface, should be defined
and specified. The important views and information that should be accessible to the
users for viewing should also be decided.
4. Choose Content and Data: Fourthly, the facet categories, details to be included in
the info panel, general view etc. which will allow the users to carry out those tasks
need to be decided.
5. Evaluate and Iterate: Lastly, the collection should be experimented on users for a
proper evaluation. Previews can be used to explore and experiment with the
collections, and the tasks defined in step three should be carried out with the
contents decided in step four. Further iterations should be carried out from step
four, in case the tasks are difficult to achieve.
Fig 4.7: Iterative process in designing
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 38 sdey1@sheffield.ac.uk - 100151932
4.3.1 Kinds of Collections
Collections can be categorised into three primary categories according to their size and
behaviour to customer queries. They can be either composed of a static (or pre
generated) XML or a dynamic XML, based on the user query. The following table aims to
distinguish the collections categorically.
Simple Collections Linked Collections Just in Time Collections
Implementation: Easy
Size: Up to 3,000 items
Implementation: Medium
Size: Limited by storage complexity
Implementation: Difficult
Size: Unbounded
Details:
A very common
collection
Comprises of static
data collection which
is loaded at one time.
Contains static visuals
contained in one Deep
Zoom Collection
Details:
Frequently used for
collections with
thousands of contents.
Comprises of inter-
linked simple collections
Consists of static data,
which is loaded per
simple collection at one
instance of time.
Contains static visuals
contained in multiple
Deep Zoom Collection
Details:
Used generally for very
large datasets
comprising of sets of
nearly and over a
hundred thousand
contents.
Comprises of dynamic
data, which is retrieved
dynamically on queries.
Contains partially
dynamic visuals
contained in dynamic
Deep Zoom Collection
Table 4.4: Kinds of Collections
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 39 sdey1@sheffield.ac.uk - 100151932
4.3.2 Facets and Facet Categories
Having already chosen the theme or subject matter for the collection, the next significant
step in the collection design is selecting the facets and the facet categories to implement,
and deciding where to put them on display. Facets signify the data or value of a content
or item, and the facet categories are a collection of facets under a particular category
name. For example: If a collection consists of facet categories called “Country”, then
“India”, “UK” etc. might be some of the facets in that category. There can be several forms
in which facets might be present in PivotViewer. Generally, they are presented as filters,
part of information in the information panel, and/or links associated to an item. [Details
have already been covered in Section 4.2.1 on CXML, about facet categories and facets]
The following diagram depicts how facets and facet categories are displayed in Pivot:
Some of the important points to note while designing facet categories (as devised by MS
Silverlight Team, 2010) are:
1. Naming and formatting – Facet categories should be appropriately named,
formatted and ordered to optimise readability and comprehension. (For
details, Facet Naming and Formatting sections in Sections 3.4.5 and 3.4.4 can be
followed).
2. Presence as a filter – It can be chosen, as to which category to include in
filter panel. All categories in filter panels can also sort their individual facet
entities.
3. Presence in the info panel – It can be chosen, as to which category to
include in the information panel as a part of all the metadata associated with
the content.
4. Keyword Filtering – By allowing keyword filtering, users can enter
keywords to be used as filters while querying. It should however be noted
that only facet categories that assist in identifying the contents are included
in keyword filtering, instead of rather broad or tangential properties.
Fig 4.8: Facets and Facet Categories
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 40 sdey1@sheffield.ac.uk - 100151932
4.3.3 PivotViewer Interface Features
There are a few important features in the PivotViewer interface that should be briefly
introduced and discussed to help arrange in organising facet categories properly:
Filter Panel
This serves as the primary area which allows the users to explore the facets and facet
categories to slice and dice the collections to narrow down their search criteria, by
forming combinations (refer Fig. 4.8).
Info Panel
This panel consists of textual information related to the image content and is visible
when the user either selects or zooms in on the picture. It plays two important roles:
Furnishing Details – It acts as a container for the detailed information associated
with the contents, and allows the users to explore the details about the facet they are
presently viewing.
Encouraging Exploration – It also contains related links that assists the users to
discover similar contents or navigate away to another content or set of contents.
Title: It displays the name of the content or item.
Description: This field contains additional details about the item and is visible above the
properties section of the info panel.
Properties: This section consists of the facets associated with the related item, or the
facet categories that it is a part of. There are three distinct types of texts that can be
displayed here:
Fig 4.9: Information Panel- I
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 41 sdey1@sheffield.ac.uk - 100151932
Filters – These are the texts, when clicked, filter the collection to show all the
items that belong to the same facets.
Static Text – this is used to generally provide further information or text about
the items that are not necessarily required to be a part of the filter panel.
Links - These comprise of general hyperlinks, which may be used for constructing
linked collections.
The figure gives a further reference (Fig. 4.10).
For certain cases, it may be required to visualise the facet categories discreetly in the info
panel as compared the filter panel. For example, a situation may arise where a different
format of data may be used in each of them, though the actual data may be the same. In
such cases, two facet categories may be created: one that can appear in the info panel
and the other that can appear in the filter panel.
Other Links
The bottom most section of the info panel contains an array of links associated with the
content. These are:
Links Related to Collections – These consist of the links that are related to the
collection items in some way. These are fetched directly from the web based on
the facets of the content selected.
Notices and Copyright Information Links – This consists of the copyright
attribution text, which can be provided as a link that may be specified in the
CXML. This attribution text appears at the very bottom of the item info panel.
Fig 4.10: Information Panel - II
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 42 sdey1@sheffield.ac.uk - 100151932
4.3.4 Facet Naming Facets and facet category names should be concise and clear, so that they don’t get
truncated in common usage (Microsoft Silverlight Team, 2010). Hence, below are some
of the guidelines that can be followed to achieve this:
Capitalization –Headline or title style conventions for capitalization may be
followed. However, facet values are case-sensitive, and hence the values should
also use consistent casing.
Plurality - singular form of the category names may be used. For example: Rather
than naming "Locations," for a category name, "Location" could be used.
Brevity and Redundancy – To name the categories, a few words should be used
as possible to name the categories. Unnecessary long phrases should be avoided,
keeping in mind that the users are already navigating within the context of the
collection that may be inherently implied in the context.
Acronyms – Use of acronyms may add confusion to certain sets of users and
hence may be avoided to prevent ambiguity and confusion.
Numbers – Direct use of Arabic numerals ("1, 2, 3..” etc) should be used, rather
than writing them out (e.g. "one, two, three"), for quick visibility and proper
legibility.
String vs. LongString – By using LongString, multiple textual lines can be
wrapped onto multiple lines. This should however, be used only for non-filtered
textual information, and which are longer than a number of words. String format
should be used for all other purposes.
Sorting Behaviour – Sorting of facets in categories can be carried out, based on
alphabetical order or according to the quantity of available items in each facet.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 43 sdey1@sheffield.ac.uk - 100151932
4.3.5 Facet Values and Formatting
In order to have a proper exploration experience, the following should be kept while
creating and formatting facet values:
Single Item Values – categories that may have the probability of having a very
large number of facets or those that may have only a few items in each facet
should be avoided to provide a better exploratory experience. The pictures below
provide an illustration (Fig. 4.11):
Compound Category – Situations may arise when it is necessary to create a
‘compound category’, in order to display multiple values together in the info
panel, as well as making useful filters. In order to do this, separate filterable
facets may be created individually, and then add another separate facet (non-
filterable) corresponding to those values.
Distribution for Interest – the facet categories should be investigated as to how
they are displayed in the interface. If certain facets have no or very less values and
there are gaps in the view, they may be omitted to provide a better experience.
Creating compound categories as mentioned above may also be considered to
optimise the results in the view.
Consistent Weights and Measures – a level of consistency should be maintained
across the collections while naming units.
Currency – Facet values representing currency values should use appropriate
currency formats (e.g. Formatting the column to represent currency).
Number Formatting – Numbers should be formatted according to the needs with
the appropriate decimal places (e.g. Year should have no decimals).
Units – Units should be included in the appropriate number formats instead of
facet names.
Fig 4.11: Guidelines for facet values
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 44 sdey1@sheffield.ac.uk - 100151932
4.3.6 Imagery
One of the prime features of PivotViewer is the visual experience through the large set of
imagery. Below are some of the suggestions that may be incorporated to produce an
optimal visual experience, as provided by Microsoft Silverlight Team (2010):
High Image Quality –recommends using images of at least 500px, however it
may be suggested to use the best quality images as possible, since the images are
converted into Deep Zoom format and there are no extra delays caused after all
the images have been rendered.
Consistent Aspect Ratio - In order to have a uniform, dense and regular display
of the images, a level of consistency should be maintained in terms of the aspect
ratios chosen, by ensuring that all the images are pruned to a similar ratio.
Source Image Format – Though not essential, but it is recommended that the
images are in JPG format, with a low compression level (maintaining a quality
level of 85%-95%), before converting to Deep Zoom format.
Edge Treatments – Image sets having very light colours may be given darker
edges to ensure a higher contrast with the PivotViewer interface background.
Enhanced Collection Imagery – In order to enhance the visual power of the
images, additional graphics and colour schemes may be added to deliver a better
visual experience of the contents. Graphical additions may help in visualising the
statistics and/or add useful labels providing in-context labels. This information
may help the users to overview multiple contents faster, instead of viewing the
info panel for the details each time. Hence, adding colour schemes and graphics
may enable the users to scan for particular type of contents easily, at the same
time, also provide additional information when zoomed in.
Fig 4.12: Example for enhanced collection imagery
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 45 sdey1@sheffield.ac.uk - 100151932
4.4. Summary
In this chapter a brief introduction about Microsoft PivotViewer is provided initially. This
was followed by several concepts on the technicalities associated with the architecture
and creating collections with this interface has been elaborated. To achieve this, the
concept of PivotViewer Collection was explained with an introduction to Collection XML
Schema and the structure of the schema. Along with this, the collection image content
was explained highlighting two key associated technologies – Deep Zoom Image
Collection and deploying the Deep Zoom collections. Next, designing the Pivot collection
was explained through insights on the kinds of collections, the concept of facets and facet
categories and steps how to organise the facet categories in PivotViewer. Apart from this,
guidelines on facet naming and key factors in providing suitable facet values and
formatting have been explained. Finally, it has been concluded with a brief discussion
and guidelines on the imagery that should be used in PivotViewer
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 46 sdey1@sheffield.ac.uk - 100151932
Chapter 5
This chapter discusses the procedures undertaken, to carry out the practical
implementation of a PivotViewer interface, based on a historic data collection. Detailed
descriptions of the evaluation mythologies undertaken in this research, highlighting the
related literature are discussed. This is followed by brief explanations, on the types of
user experiments undertaken to evaluate the Pivot interface designed, as well as other
pre-existing systems in such a way so that a complete overview of the present systems
are obtained along with what the users feel about PivotViewer. A careful approach has
been taken to obtain a holistic yet detailed data from the experiments, such that it can be
carefully analysed and evaluated as to what the users actually think about the Web based
search interfaces and information retrieval systems that currently exist in the market
and how they compare them with the Pivot interface, released by Microsoft.
5.1 Literature Review
According to Plaisant (2004), carrying out evaluations on information visualisation
always has a challenge. These challenges are carefully noted down while crafting out the
experimentations and laying out the questions for evaluations. Besides, a study of over
fifty users on information visualisation systems (Komlodi et al., 2004), showed that there
are four thematic areas that could be employed for optimal evaluation:
1. Controlled experiments that compared design elements – such as studies on
widgets (Ahlberg and Shneiderman, 1994b), studies to relate information to
geographical display (Irani and Ware, 2003), etc.
2. Usability evaluation – for example studies on the problems users encountered
with interface tools and how designers refined them (Sutcliffe et al., 2000; Byrd,
1999).
RESEARCH METHODOLOGY
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 47 sdey1@sheffield.ac.uk - 100151932
3. Controlled experiments that compared tools – often a common study, mostly
comparing novel technology with pre-existing or state of art technique (Plaisant
et al., 2002).
4. Case studies – though time consuming, these enable users to carry out real tasks
in a natural environment, thus reflecting on the feasibility and usefulness in
context of the scenarios (Trafton et al., 2000).
The above themes are carefully considered and incorporated in the experimentation that
has been set up for evaluating the systems. Apart from this, several other techniques of
evaluation have also been studied to provide an insight into the best evaluation
methodologies that can be adopted in this research to bring out an effective and robust
result. Some of them include:
Author(s) Related study
Munzner et al. (2003) TreeJuxtaposer – task conduction and interpretation of results
Hong et al. (2003) Zoomology – custom design for a single set of data leading to a useful evaluation tool
Spenke and Beilken (2000) InfoZoom – tool to manipulate tables
Auber et al. (2003) EVAT – another powerful analytical tool
Morse et al. (2003) Taxnote – emphasising evaluating the importance of labelling of the retrieved displayed information.
Sheth (2003) Demonstrated the benefits of combining several tools
Apart from this, the ImageCLEF studies (Clough et al., 2004; Clough and Sanderson,
2003) have also been carefully studied and followed to obtain a deeper understanding of
suitable evaluation techniques that might be useful to incorporate in this project. Finally,
a few piece of previous dissertation and doctoral thesis works on image retrieval,
information retrieval and visualisation have also been closely studied (Hoeber, 2007;
Read, 2007; Yang, 2002) to get an insight into the difficulties faced as well the
effectiveness of their methodologies in evaluation in this domain of study and a careful
attention to detail has been considered in creating the user experiments as well the
questionnaire. A formative technique has been adopted for the purpose of the study.
Table 5.1: Evaluation Methodologies carried out in previous studies
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 48 sdey1@sheffield.ac.uk - 100151932
5.2 Dataset (St Andrews University Library Photographs)
In order to carry out a proper evaluation of the PivotViewer, not only a large collection of
images was needed, but also a collection that had quality captions to facilitate proper
textual retrieval and faceted filtering. Besides, to accommodate such a study based on
huge number of pictures, copyright issues were also to be considered as the digitisation
of images today meant that they have high market values and it would restrict proper
distribution in case further research work is carried out in this domain in future. Hence,
taking all these accountabilities into consideration, a historic collection of St. Andrews
University Library was used. The Information School in the University of Sheffield,
already has research links with the library at St. Andrews University, which currently has
an archive of one of the largest historic collection of images, of over 300,000 pictures,
taken by some of the famous Scottish photographers (Reid, 1999). A cross-section of
around 28,133 images has already been used as a part of a digitisation project in the
CLEF studies carried out at the University of Sheffield.
Given the scope of this research, the local resources available and considering the time
constraints for the overall evaluation, out of this collection, a set of 10,000 images (with a
resolution of approximately 368x234 pixels each) was used for the purpose of
PivotViewer in this project. Due to the historic nature of the collection, a high number of
images (around 82%) from the original collection were in greyscale, and belonged to the
year range between 1832 and 1992.
It should be mentioned here, that this historic collection was considered for the project
as it represented a realistic archive of images with high quality captions. Besides, the
metadata associated with the images also contain a varying set of information, on a range
of topics, which would prove to be challenging for both image as well as textual retrieval
through the visualisation power of PivotViewer.
It should be noted that full permission to use the image collection was granted by Dr.
Paul Clough, who was a lead researcher of CLEF on the St. Andrews collection, in the
University of Sheffield, and hence there are no associated copyright issues in this piece of
dissertation.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 49 sdey1@sheffield.ac.uk - 100151932
5.3 Overview of metadata
The vital metadata of the images was distributed into text documents with the same
filename as that of the images and had 8 different sections (shown in Fig. 5.1 below):
Title – It contains the title of the picture explaining the picture in a short
sentence.
Short title – It’s a shorter version of the above title [often similar or exactly same
as the title].
Location – contains the information regarding the location of where the
photograph relates to, in terms of its city and country [often either cities or only
country name present].
Description – contains the description of the photograph in further details.
Date – the date the photograph was taken on.
Photographer (or Originator) – the name of the photographer(s) or the
photography company from where the photograph has been obtained.
Categories – the related subject or tags that the photograph can be said to be
related to.
Notes – any other details or notes about the photographs.
Fig 5.1: Example of metadata associated with images in St. Andrews
University Library historic photographic collection
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 50 sdey1@sheffield.ac.uk - 100151932
5.4 Building the Pivot Collection
This section will briefly discuss the procedures taken to implement the photographic
collection using the PivotViewer tool for the dataset chosen and discussed above.
System requirements for executing and building PivotViewer applications:
System Microsoft Windows Vista/7 (Aero enabled),
2-GHz 32-bit(x86)/64-bit(x64) processor,
2GB RAM.
Graphics Memory 256 MB of video memory (a dedicated GPU is
recommended) [Note: Pivot does not support most
Intel integrated video chipsets].
For application
execution
.NET Framework 3.5 SP1 (or higher)
Internet Explorer 8 (or above)/Firefox/Chrome
For Application
building and testing
Visual Studio 2010
Silverlight 4 Tools for Visual Studio 2010
Silverlight 4 Toolkit (released April 2010)
PivotViewer Control
Techniques for creating Deep Zoom content for Pivot was discussed in the previous
chapter. For the purpose of building the CXML file for the current St. Andrews data set
and generate its Deep Zoom image collections for purpose of this dissertation, Microsoft
Pivot Collection Excel add-in is used. In short, this tool allows developers to create the
Pivot collection by creating the individual facet categories for the overall collection and
then filling in the spreadsheet with the corresponding metadata for each of the
individual items, which can then be published and exported to required Deep Zoom
format understood by the Pivot application.
Table 5.2: System requirements for executing and building PivotViewer
applications
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 51 sdey1@sheffield.ac.uk - 100151932
After the Excel add-in has been installed, a new ribbon called the Pivot Collections will
be added. There are three special components in this tool as briefly described below:
Excel Ribbon Tab or “Pivot Collections” Tab - the main control area of the
collection, that is used to associate images with their related metadata, create
collection-wide properties, test and publish or render the entire collection.
“Collection Items” and “Collection Properties” Worksheets – These
worksheets (visible as tabs at the bottom) harbour the data of the collections and
properties associated with the whole collection, respectively.
Excel Table
Rows– correspond to a particular item in the collection.
Columns– correspond to a particular facet category
Cells– contain the item’s value (or facet)
As a first step to start building the collection, the option to create a New Collection
should be clicked (Shown in Fig. 5.2 below)
This will create a new workbook with the following default columns:
Image Location- Contains the path of source image file.
Name- Contains the name of the particular item, which is displayed at the top of
the info panel in PivotViewer.
Href- A URL that can be associated with this item which will redirect the user to
the link provided if the item is clicked in the PivotViewer interface.
Description- Contains the description of the related item.
Preview – provides an automatic thumbnail of the image provided at the ‘image
location’ column.
Fig 5.2: First step to start building the collection
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 52 sdey1@sheffield.ac.uk - 100151932
For the purpose of the St. Andrews collection, the Href column has been omitted since
there was no further requirement for navigating to a web URL through the images. The
preview column was also deleted as it had no affect on the front-end of the system and
besides, it was taking extra resources as well as time, to generate the pictures, thus
delaying the process. In addition, six new columns representing the facet categories of
the St. Andrews University Library were added to accommodate the metadata:
Record ID Title Year Month Originator Notes Category
The corresponding details of the collection set were then filled in.
[Note: given the magnitude of data, scripting was used to gather the data from the
individual text files that contained the metadata for the individual images into a separate
excel sheet which was then transferred in the excel Pivot sheet]
Adjusting Visibility of Categories
By default, when new fields are added, the facet categories automatically appear in both
the filter panel and the info panel of the PivotViewer. Hence, the “Category Properties”
panel in the tab is used to modify visibility for each of the facet categories.
Fig 5.3: Columns representing facet categories for St. Andrews collection
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 53 sdey1@sheffield.ac.uk - 100151932
The fields can have the following modification options:
Filter Panel – This enables the images to be filtered in the PivotViewer according
to the elements in the field of the chosen column as facet categories.
Info Panel – Gives an option for the current category to appear in the info panel.
Keyword filtering – Allows the elements in the field for keyword filtering.
Keyword filtering enables the viewer to provide keywords to apply as filters in
the collection.
In the St. Andrews collection, the following setting have been incorporated
Categories Filter Panel Info Panel Keyword filtering
Image Location Name
Record – ID ✓
Title ✓ ✓
Year ✓ ✓ ✓
Month ✓ ✓ ✓
Originator ✓ ✓ ✓
Location ✓ ✓ ✓
Notes ✓
Category ✓ ✓ ✓
Description
Miscellaneous Property Adjustments
Categories Requirement Adjustment of property
Year Numeric facet The entire column should be formatted to change the ‘Format Cells’ property to number with decimal places to 0
Category Multiple Values To accommodate multiple values as filter options within the same facet category, the values should be separated individually by ‘||’
Notes LongStrings
Since the notes generally contain long set of textual data, these are generally needed to be set to equivalent LongString format. To achieve this, the column should be formatted to select ‘Text’ tab and then check ‘Wrap Text’ in the Alignment section
Table 5.3: Settings of St Andrews collection
Table 5.4: Miscellaneous Property Adjustments
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 54 sdey1@sheffield.ac.uk - 100151932
After all these adjustments are done and the data set is filled, this entire set needs to be
published into the CXML file along with the set of Deep Zoom images. This is achieved by
using the “Publish” button (shown in Fig. 5.4).
5.5 Creating Pivot Silverlight Application
Now since we already have the CXML file and the associated Deep Zoom collection, this
has to be deployed using an ASP.NET server to create a web interface. To begin, a new
Silverlight Application needs to be created in Visual Studio 2010. After that, the
“System.Windows.Pivot.dll” needs to be referenced. This file can be located in the
directory where the PivotViewer SDK has been installed.
Fig 5.4: Image showing Pivot ‘Publish Button’
Fig 5.5: Creating Pivot Silverlight Application
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 55 sdey1@sheffield.ac.uk - 100151932
Next, the file MainPage.xaml needs to be opened and updated with XMLNS declarations
to allow PivotViewer control namespace:
After this, we need to code in C#.NET to load the CXML collection into the ASP.NET
server. For this the file MainPage.xaml.cs (the code behind of MainPage.xaml, accessed
by double-clicking it) is to be modified so that it can match the CXML with a local port
and deploy. The code is given below:
<UserControl x:Class="SilverlightApplication1.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:pivot="clr-namespace:System.Windows.Pivot;assembly=System.Windows.Pivot" mc:Ignorable="d" d:DesignWidth="972" d:DesignHeight="298" Loaded="UserControl_Loaded"> <Grid x:Name="MainCanvas"> <!-- The PivotViewer control --> <pivot:PivotViewer Name="PivotViewerControl" Grid.Column="0" ItemDoubleClicked="PivotViewerControl_ItemDoubleClicked"> </pivot:PivotViewer> </Grid> </UserControl>
using System; using System.Collections.Generic; using System.Linq; using System.Net; using System.Windows; using System.Windows.Controls; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Animation; using System.Windows.Shapes; using System.Windows.Pivot; using System.Windows.Browser; namespace SilverlightApplication1 { public partial class MainPage : UserControl { private static readonly string CXMLCollectionUri = "http://localhost:42912/ClientBin/PivotCollection.cxml"; public MainPage() { InitializeComponent(); PivotViewerControl.LoadCollection(CXMLCollectionUri, ""); }
Fig 5.6: MainPage.xaml
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 56 sdey1@sheffield.ac.uk - 100151932
Visual studio automatically creates a Web application that hosts the PivotViewer
Silverlight application when it is created – one is a dynamic ‘aspx’ file and the other is a
static ‘html’ file but both with similar codes.
Finally the html file has to be coded to allow binding of the CXML as follows:
private void OpenLink(string linkUri) { HtmlPage.Window.Navigate(new Uri(linkUri, UriKind.RelativeOrAbsolute), "PivotViewerSampleTargetFrame"); } private void PivotViewerControl_ItemDoubleClicked(object sender, ItemEventArgs e) { PivotItem piv_item = PivotViewerControl.GetItem(e.ItemId); if (!string.IsNullOrWhiteSpace(piv_item.Href)) { PivotViewerControl.CurrentItemId = e.ItemId; OpenLink(piv_item.Href); } else { MessageBox.Show("No Web Page..."); } } private void UserControl_Loaded(object sender, RoutedEventArgs e) { } } }
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head> <title>SilverlightApplication1</title> <style type="text/css"> html, body { height: 100%; overflow: auto; } body { padding: 0; margin: 0; } #silverlightControlHost { height: 100%; text-align:center; }
Fig 5.6: MainPage.xaml.cs
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 57 sdey1@sheffield.ac.uk - 100151932
</style> <script type="text/javascript" src="Silverlight.js"></script> <script type="text/javascript"> function onSilverlightError(sender, args) { var appSource = ""; if (sender != null && sender != 0) { appSource = sender.getHost().Source; } var errorType = args.ErrorType; var iErrorCode = args.ErrorCode; if (errorType == "ImageError" || errorType == "MediaError") { return; } var errMsg = "Unhandled Error in Silverlight Application " + appSource + "\n" ; errMsg += "Code: "+ iErrorCode + " \n"; errMsg += "Category: " + errorType + " \n"; errMsg += "Message: " + args.ErrorMessage + " \n"; if (errorType == "ParserError") { errMsg += "File: " + args.xamlFile + " \n"; errMsg += "Line: " + args.lineNumber + " \n"; errMsg += "Position: " + args.charPosition + " \n"; } else if (errorType == "RuntimeError") { if (args.lineNumber != 0) { errMsg += "Line: " + args.lineNumber + " \n"; errMsg += "Position: " + args.charPosition + " \n"; } errMsg += "MethodName: " + args.methodName + " \n"; } throw new Error(errMsg); } </script> </head> <body> <form id="form1" runat="server" style="height:100%"> <div id="silverlightControlHost"> <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="100%" height="100%"> <param name="source" value="ClientBin/SilverlightApplication1.xap"/> <param name="onError" value="onSilverlightError" /> <param name="background" value="white" /> <param name="minRuntimeVersion" value="4.0.50401.0" /> <param name="autoUpgrade" value="true" /> <a href="http://go.microsoft.com/fwlink/?LinkID=149156&v=4.0.50401.0" style="text-decoration:none"> <img src="http://go.microsoft.com/fwlink/?LinkId=161376" alt="Get Microsoft Silverlight" style="border-style:none"/> </a> </object><iframe id="_sl_historyFrame" style="visibility:hidden;height:0px;width:0px;border:0px"></iframe></div> </form> </body> </html>
Fig 5.8: HTML coding to bind CXML
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 58 sdey1@sheffield.ac.uk - 100151932
Finally, the Deep Zoom folder and the CXML file that has been published using Excel
Pivot tool should be added to the ClientBin folder located in the Silverlight Web
application directory of the application that has been created, as it contains an XAP file
which needs to be accessed by the program while building the solution. The solution is
then built and run, and we have the Pivot Collection of St. Andrews University historic
photographs.
Fig 5.9: Pivot collection of St. Andrews University photographs.
Fig 5.9: Facet categories in
Pivot collection of St.
Andrews University
photographs.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 59 sdey1@sheffield.ac.uk - 100151932
5.6 Design Checklist
Microsoft Silverlight team (2011) has laid down a design checklist for implementation of
an efficient PivotViewer interface. For the purpose of this experiment, the guidelines
provided by them have been closely observed while implementing the system:
[✓= Checked; X= Not Applicable; ? = is investigated]
Basics
Is the collection named well? Would the target user
understand what it is about? ✓
The collection is named St. Andrews University Photographic collection
which is easy to understand by users.
Good quality iconography, branding and attribution? x
Since this is not a commercial product, these attributes are not
considered given the scope of this piece of study.
Is the default view interesting given the subject matter and
target user? ?
The default view shows the entire collection of the photograph, which
is later evaluated from the user experiments.
Images
Images should be of good quality ✓
Only the images with higher resolution (i.e. those having an extension
“_big”) from the supplied images is chosen.
Possibility of adding text or graphics to the items? x
Given the variety of topics and categories, extra graphics would mean
too much noise in the collection. Hence this is not included. Besides,
the metadata already had sufficient texts and hence no extra text or
graphics has been included.
Facet
Categories
and
Facets
Careful attention to the categories in the filter pane?
Number of such categories? (5-8 is ideal) ✓
The categories are given according to the metadata provided, and there
are 6 filterable categories.
Should any be added? x
Given the current metadata associated, no extra needs to be added.
Are there any categories a user might need not in the
collection currently? ✓
Careful attention is taken to include only those as filterable
categories that the users might actually need.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 60 sdey1@sheffield.ac.uk - 100151932
Are there any categories with a long list of facets? If so, can
the values be collapsed? ✓
Care is taken to keep these categories in the info panel only.
Are the categories in the right order? ?/ ✓
The categories are ordered according to the metadata provided and
hence it is subjective and mainly depends on the overall view of what
the user actually thinks after using the interface.
Are the categories named properly? Do they use the
correct conventions and will your target user be familiar
with the terms used?
✓
Care is taken to include simple and effective conventions that user
would be familiar with.
Are the facet values formatted properly? ✓
The facets are formatted according to the content, e.g.- the year is
numerical with 0 decimal places etc.
Should this collection be linked to other collections? ?
This collection could be expanded to include the overall 300,000
images archived by St. Andrews University Library, assuming they
have similar faceted data associated, however this could be further
investigated in future studies.
Are you happy with the search results? ?
Since this is based on user experimentations it has been discussed
further in user evaluations
5.7 User Experiments
To carry out a holistic yet significantly deep evaluation of the PivotViewer as compared
to traditional information retrieval and visualisation systems, user-centred lab based
experimentation was set up.
Table 5.5: Design Checklist
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 61 sdey1@sheffield.ac.uk - 100151932
5.7.1 Ethical Considerations
The user-evaluation was conducted according to the ethics guidelines laid down by the
University of Sheffield for post graduate students:
http://www.shef.ac.uk/is/research/ethics/pgt.html
The users were contacted via e-mail and social media sites along with an attached
Participant Information Sheet. Before the experiments were conducted, they were asked
to sign the Participant Consent Form [Appendix]. Every user was allocated a unique
identifier (S1, S2, ... , S11) which enabled in keeping the participant data collection
anonymous.
5.7.2 Selection of participants
Since the use of Pivot interface is aimed at general users with only basic PC experience,
there was no requirement for setting up a special user group with special qualities.
However, due to the time constraints and the nature of qualitative research required,
instead of a random set of people, a small number of volunteers from various age groups
and with varied education and information searching experience was chosen. These
volunteers comprised of a multi-cultural group of ten postgraduate taught students from
the Information School at the University of Sheffield.
5.7.3 Rationale
In order to properly reach a consensus about the visualisation power of PivotViewer as
compared to the traditional systems, a series of observations in various categories and
scenarios are necessary to identify a set of common patterns. This would lead to a
tentative hypothesis that can ultimately help the study to culminate in a probable theory
of suitability about this technology. Hence, in order to achieve the aims of this study
(Chapter 2 – Section 2.2), it is required to carry out an inductive research. The overall
theme of the experiments was based on the exploratory interviews (Oppenheim, 1992),
to gather an in-depth data of the users’ ideas about the traditional visualisation systems
as compared to PivotViewer in its various aspects. Besides, according to Fowler (1995),
questions having multiple choices should encompass a wide range of choices to enable
sufficient detail for differential analysis. Hence, while building the questionnaire, care
has been taken to keep sufficiently wide ranges depending on the nature of the
questions.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 62 sdey1@sheffield.ac.uk - 100151932
Incorporating the above considerations, the experiment was categorised into six
sections:
Preliminary Questionnaire
Evaluation of user-centred expectations
Pivot awareness
A general explanation of using PivotViewer
User-centred interface evaluation
Post evaluation questionnaire
5.7.4 Preliminary Questionnaire
The primary aim of this questionnaire is to gather if there exist any chief dependencies
amongst the participants in relation to the experiments that were undertaken. This
questionnaire was separated into two sections:
Basic Details
This section addressed the following areas:
Age/Profession/Education background
Familiarity with PC/Browser and prior experience levels in carrying out photo
search
Frequency of Internet usage
Confidence levels in Web search
General Familiarity
Frequency and popular topics/categories while using Web based search
Most favourable text-based search interface and the corresponding level of
satisfaction
Most favourable image-based search interface and the corresponding level of
satisfaction
Number of pages generally browsed while before stopping
Nature of image search techniques employed
Familiarity of image search techniques
Analysis on the levels of difficulty for the search techniques encountered
Vagueness encountered in search habit
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 63 sdey1@sheffield.ac.uk - 100151932
5.7.5 Evaluation of user-centred expectations
This section tries to evaluate the users perceptions about the current information
retrieval systems based on various criteria as mentioned below. It tries to build an
overall notion of how the users currently feel the traditional interfaces are and what kind
of changes they would possibly want the systems to be like, in the future. The purpose of
this section was to identify if the users are satisfied with the systems they are currently
using and map their requirements to see if it fits in well with what PivotViewer has to
offer.
The questionnaire in this section was designed to find if the users were:
Satisfied with the current image and information retrieval systems,
Satisfied with browsing individual pages for information searching
Also whether they would want all images and information to:
Be accessible in a single browser
Be available on the web to be ordered and colour coded according to particular
categories within a single browser or interface.
Show their associated information, details and various miscellaneous properties
at the time of viewing.
Link to other images based on similar associated information/categories
instantly
5.7.6 Pivot awareness
Given the fact that this product was a recent launch of Microsoft Corporation in the
Silverlight domain, it would be interesting to find how many users actually head about
PivotViewer before. At the same time, it would be interesting to know, out of those who
have heard of it, how many have actually used it and have been familiar about with it.
This is necessary because it would be worth a study to actually familiarise the
participants with the interface and then try and relate that experience with the
traditional systems that they have been using for a long time.
This section had two short questions:
If the users were familiar with the Pivot interface
Their level of familiarity with the interface (if they are familiar).
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 64 sdey1@sheffield.ac.uk - 100151932
5.7.7. A general explanation of using PivotViewer
Every participant (regardless of their experiences with Pivot and Web search), was
familiarised with the interface of PivotViewer. They were demonstrated to perform the
following:
Zooming in and out of the interface using mouse wheel as well as the controls
provided for the same
How to use the text-based search facility to bring out the required image
How to view the info panel and all the different facets in it
Familiarised the users with the different kinds of information in the info panel
and enables them to distinguish between filterable keywords and static
information.
How to use facet categories to perform multiple filtering
How to use the sort option under every facet category according to the necessary
requirements to order the filter lists
How to use the mouse to filter elements that had numerical data
Such a demonstration is provided to allow the users to have a direct interaction instead
of the PivotViewer Interface so that they are confident enough to carry out the
experiments.
Fig 5.12: Various features in
Pivot interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 65 sdey1@sheffield.ac.uk - 100151932
Fig- 2 My Dissertation Interest lies
in the area of Visualisation for
Information Retrieval
5.7.8. User-centred interface evaluation
This is the central assessment section of the entire project. This section is divided into
three scenarios, where in each scenario, two interfaces were evaluated, one of them
being a Pivot interface. Users were asked to carry out three tasks for each of the
scenarios. The tasks were carefully crafted in a way such that they were based on
Shneiderman’s conceptual framework for image retrieval (Rosenfeld & Morville, 2002;
Spencer, 2006).
1. General searching familiarity with the systems
2. Searching for a known item (and/or related information)
3. Exploratory searching ( searching based on some defined multiple
categories)
After carrying out tasks in each of the given scenarios, the users were asked to fill out a
questionnaire that had several parameters of comparative evaluation between the two
systems. (More details in Appendix – User-centred Interface Evaluation)
Scenario A – Interfaces with exactly similar datasets
The St. Andrews University Library already has an interface where it has archived its
historic photographic collection. [Which can be accessed at: http://special.st-
andrews.ac.uk/saspecial/index.php]. Borrowing as a subset of 10,000 images from the
collection and using the same metadata that the engine uses, a Pivot interface was
designed as explained earlier. Now, in this first scenario, these two systems were
evaluated based on a set of parameters explained in the next section. This would enable a
perfect evaluation of between two interfaces that had exactly the similar images and
associated data and hence provide with an interesting find on what the users feel about
the original system and the new (prototype) system. (Questionnaire: Scenario A)
Fig 5.13: St. Andrews Interface (left) and Pivot interface (right) with exactly same datasets
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 66 sdey1@sheffield.ac.uk - 100151932
Scenario B – Open ended interface and Pivot interface on a particular subject
The next scenario was based on evaluating between widely used open ended image
search interface i.e. Google Images and a sample Pivot interface that had a repository of
endangered species. Tasks were set for the users to carry out in both the interfaces based
on the criteria mentioned in the previous section. After the tasks were done, the users
were asked to fill out a set of questionnaire that compared the two systems based on a
few parameters. [The pivot interface was last accessed on 15th August, 2011 at:
http://www.352media.com/silverlight/pivotviewer/demos/endangered/endangered-
species.aspx]. (More details in Appendix – User-centred Interface Evaluation
Questionnaire: Scenario B)
Scenario C – Interfaces dedicated to a similar domain
This last scenario of tasks involved the user to carry out search in some of the largest
movie databases in the market currently. IMDb has one of the largest and most popular
movie databases in the world and has an interface that is capable of searching movie
details along with all its related information ranging from cast to genre. Netflix also has a
similar repository but based on Pivot viewer. Similar to the above two scenarios, the
users were asked to perform a same set of tasks in both the interfaces and fill out the
comparative questionnaire to evaluate both the systems. (Questionnaire: Scenario C)
Fig 5.14: Endangered species Pivot interface (left) and Google Images (right)
Fig 5.15: IMDb (left) and Netflix PivotViewer (right)
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 67 sdey1@sheffield.ac.uk - 100151932
5.7.9. Interface evaluation parameters
As mentioned in all the above three scenarios, the users were asked to take a
comparative evaluation questionnaire for each of the systems as soon as they finished
the tasks assigned for each of the above scenarios. For this a set of parameters were set
to obtain a comparative evaluation of the systems. The criteria/parameters for
evaluation were common for all the systems evaluated above. They are given below:
Evaluation Parameters
Procedure
Time taken (sec) The total time taken for the individual tasks was
monitored for each of the above interfaces, in every
individual scenario, using a stopwatch.
Number of clicks
The number of clicks to carry out each of the tasks was
monitored using an application that was set to register the
number of mouse clicks.
Satisfaction
The users were asked specifically give their ratings for the
individual interfaces how satisfied they were and
interactive they felt while during carrying out the tasks.
Usability
The users were asked to provide their views on how
usable they felt about each of the interfaces and how easy
they found the interfaces.
Visual Appeal The users were asked to rate the overall visual appeal of
the interfaces in relation with the image retrieval process.
Confidence Having used the interfaces to carry out the tasks, the users
were asked to rate their confidence in using the interfaces.
Results
Finally, the users were asked to rate how they felt about
the results retrieved by the interfaces for the tasks that
were carried out and also how they felt they have done in
the tasks.
(*More details in Appendix: – User-centred Interface Evaluation Questionnaire)
Table 5.6: Parameters for interface evaluation*
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 68 sdey1@sheffield.ac.uk - 100151932
5.7.10. Post evaluation questionnaire
This was the final section of questionnaire and was divided into two parts. This set of
questionnaire was mainly aimed at getting a proper structured feedback based on the
overall task-based experimentations carried out. It was divided into two sections, part
one being a point based questionnaire to gather views of the users on a scale of 1 to 5 to
indicate the agreement or disagreement of the user in connection to the questions asked.
The next section was more open and encouraged open answers. The two sections are
briefly discussed below:
Part 1
This section consisted of three categories on which the users’ views were gathered:
Tasks taken
The views of the users were taken to indicate if they felt that the
tasks undertaken were realistic and whether they had a clear idea
on what they were supposed to do. It was also asked if it was a
new experience for them in working with picture collections in
terms of the visualisation provided by PivotViewer.
(more details in Appendix – Post Evaluation Questionnaire)
Pivot interface
The following areas of the pivot were addressed in this category to
get the users views:
The clarity of the facets and descriptions in Pivot interface.
Whether the facets and categories helped them to have a
mental picture of what they were searching for from the
whole collection.
If the Pivot was helpful due to its categorical arrangement
and whether it increased their interest in using it further.
If the pivot interface was difficult to use.
(more details in Appendix – Post Evaluation Questionnaire)
Results
retrieved by
Pivot
Here a detailed analysis and evaluation on the retrieved results
from the Pivot was obtained through a detailed questionnaire.
Users were asked if the images were hard to comprehend and
whether the details provided by pivot was enough. The
arrangement and overall functionality of the Pivot interface
compared to traditional page-wise browsing was also attempted
to be covered in this section. (Appendix: Post Evaluation Questionnaire)
Table 5.7: Post evaluation questionnaire categories Part- 1
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 69 sdey1@sheffield.ac.uk - 100151932
Part 2
This is the penultimate part of the experimentation where the users are finally asked to
indicate their preferences between Microsoft PivotViewer and traditional retrieval
systems. This was followed by three open ended questions that required the participants
to put their views regarding:
The advantages of the interface they preferred that made searching easier for
them
The disadvantages (if any) that might have made searching difficult for them
Any extra features or functionalities that the users wish to be included in future
for their preferred choice of interface
5.8 Pilot Study
A pilot study was undertaken with the help of a lecturer in Information Systems at the
University of Sheffield as a participant, prior to taking the actual experiments with the
users. The fundamental reasons for carrying out this study were to:
Develop and test the adequacy of the research/experimentation
Assess the feasibility of the research
Analyse if the research is realistic and workable
Identify issues which might occur using the proposed methods
Collect a preliminary data
Assess data analysis to uncover potential issues
Determining the approximate time required to complete the experimentation
Having taken the pilot studies, a few changes were introduced in the methodology
adopted to carry out the experiments.
Firstly, it was noted that the interfaces took significantly long time to load from the local
system every time during the experimentation. As such, it was recommended to run the
applications before the beginning of the experiments and make full use of the local cache.
This would reduce the time taken for users to wait during the experiments and avoid
unnecessary delay.
Secondly, it was identified that the users also needed time to actually familiarise
themselves with the PivotViewer interface. As such, an extra session was allocated to
allow the users to play around with the interface in general. At the same time, sessions
were also included for the participants to carry out general searches on each interface,
before the beginning of all the scenarios during the experiment, so that a state of
saturation is reached on their level of confidence in using the interfaces so that a proper
and fair result is reached.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 70 sdey1@sheffield.ac.uk - 100151932
Given the time constraint, the post evaluation questionnaire was condensed so that users
can provide a much deeper insight. At the same time, certain ambiguities in the
questionnaire were eradicated to make the whole experiment more user-friendly,
interesting and enjoyable.
5.9 Timetable
From the pilot studies and self evaluations, the total approximate time for the whole
experiment has been tabulated below:
Stages of Experimentations
Expected Time (in minutes)
Minimum Maximum
General Tutorial/Awareness of Pivot 5 7
Allowing the user to familiarise with the Pivot interface 5 7
I - Preliminary Questionnaire – Basic Details 1 2
II - Preliminary Questionnaire – General Familiarity 3 5
III - Evaluation of User-centred expectations 1 2
IV – Pivot Awareness
V – User-centred Interface Evaluation
Scenario – A Tasks + Evaluation 10 12
Scenario – B Tasks + Evaluation 12 15
Scenario – C Tasks + Evaluation 12 15
VI - Post evaluation questionnaire - Part 1 2 3
Post evaluation questionnaire - Part 2 3 5
Total expected time for evaluation completion 54 71
Table 5.8: Approximate time needed for total experiment
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 71 sdey1@sheffield.ac.uk - 100151932
5.10 Summary
In this chapter, the research methodology taken to carry out this study has been
explained with a brief overview highlighting the literature adopted for this purpose. The
dataset used from St. Andrews University Library photographic collection has been
explained with an overview on the metadata. This was followed by explaining the actual
steps in building the Pivot collection and finally creating the application in Visual Studio
has been explained. For the purpose of an effective system, a design checklist has also
been followed which was recommended by Microsoft. Next, the user experiments taken
were explained stepwise, having briefly illustration the ethical considerations, selection
of participants and the rationale. Next, preliminary questionnaires were discussed
followed by the evaluations of user-centred expectations that were carried out. A short
study taken to find the Pivot awareness was also indicated before explaining the user
centred interface evaluations that were carried out. A brief introduction about the
PivotViewer interface that was enlightened to the users has also been highlighted. The
parameters set for the evaluation of the Pivot were also discussed along with the reasons
behind post evaluation questionnaire. Finally, it was concluded with the discussions
about the pilot study undertaken, along with a short timetable indicating the overall time
required for the whole study.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 72 sdey1@sheffield.ac.uk - 100151932
Chapter 6
The methodologies and ideas previously discussed in Chapter 4 were implemented and
experiments were successfully carried out with the help of 11 participants that included
the single pilot study. Ideally, the data gathered from a pilot study should be ignored,
however, in this case it is included since it had no major effect on the data, but only
identification of issues related to time adjustments in the experimental setup, the
ambiguity of a few questionnaire and the manner of tests conducted, which was rectified,
as mentioned in the previous chapter. The interfaces was designed and chosen carefully.
The tasks were set in such a way so that users could carry out the tasks effectively and at
the same time exploit and stretch the limitations of the systems as much as possible.
This chapter will present the results for the study undertaken through the detailed tasks
and user evaluations. A set of detailed data and graphical results are presented for each
of the experiment undertaken on the three different interfaces as well as the users’ views
and recommendations. At the end of each sub stage of experiments taken, a summary of
results are also discussed.
6.1 General User Overview
The participants who agreed to volunteer for the experimentation were all currently post
graduate taught students in the Information School at the University of Sheffield. While
most of them were only studying (64%), some were also working (36%) along with their
studies. Around 54.55% of the volunteers were in the age group between 26 to 33, with
the next highest being in the range 18 to 25 (36.36%).
RESULTS AND ANALYSIS
Fig 6.2 – Professional background of participants Fig 6.1 – Participant age group
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 73 sdey1@sheffield.ac.uk - 100151932
A very high number of participants indicated that they had a proper IT educational
background (91.91%) along with proper PC skills (81.81%) with mean experience level
of 5.81 out of 6. As such, this experiment can be said to have negligible limitations
imposed by users being weak in
PC/IT skills, which would
otherwise have introduce some
constraints in the data
gathered. On the other hand,
this promises to make it more
challenging and interesting to
note, how they would perform
with the system, in an
unfamiliar and how well each
individual critique the system,
which we shall soon find out.
Around 54.54% users indicated that they had very good prior experience in photo
searching with an overall mean experience of 5.81 denoting that most users were
confident and aware of photo searching. The other users were not so sure, however, the
trend showed that they were quite aware and familiar with the concept.
Table 6.2 –User professions
Fig 6.3 – Educational background of participants
Fig 6.4 – User PC familiarity
Fig 6.5 – User photo search experience
Table 6.5 – User photo search
experience
Table 6.4 –Familiarity
with PC
Table 6.1 – Participant age
groups Table 6.3 – Educational background
of participants
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 74 sdey1@sheffield.ac.uk - 100151932
There was no surprise in finding that today
Internet has become an essential part of our lives
with over 81.81% users indicating of using the
Internet several times during the day and the rest
using it at least once every day. However, when it
came to the level of confidence in searching the
views were not comparatively as high. Though
none of the users believed that they had a low
confidence level, around 34.37% of them still felt
that they were not very confident in using the
Web for searching.
6.2 Search Behaviour Patterns
While it was evident that users believed that they were quite familiar with searching and
search techniques, it was now necessary to take a step further and find out about their
search habits in a more detailed manner and establish some patterns for further analysis.
This piece of result is essential as it would throw light on the current trend in search
behaviour, the users likes and dislikes about the current interfaces, their tolerance and
satisfaction level with the present systems, certain insights on their searching strategies
and the difficulties that they might find in using these present systems.
Fig 6.6 – User Internet usage
Fig 6.7 – Confidence levels in Web searching
Table 6.7 – User confidence levels in
Web searching
Table 6.6 – User Internet usage
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 75 sdey1@sheffield.ac.uk - 100151932
To begin with, the participants were asked to indicate their habit of using the Web for
searching. It was learnt that a little more than half of the participants (54.55%) search
for information on the Web several times during the day, while others (36.36%)
searched at least once every day and rarely (9.09%) certain times over a week.
Next it was essential to know the most popular search topic or category they generally
seek information on. This was essential since organisations looking for adopting or
implementing a suitable next generation experience in search interface in the future, can
benefit from the user trends and interests to begin with, and tap the scenario of demand.
The following results were obtained from the users, which showed a wide variety:
User Favourable search topic S1 ----No particular topic---- S2 News, Shopping, Events S3 Technology, Music S4 Academic literature S5 Movie, Jobs S6 News S7 IT jobs in UK S8 ----No particular topic---- S9 Cars
S10 Technology, Education S11 Research Papers
When it came to deciding about the actual interfaces, Google emerged as the clear winner
as the users’ choice for the most favourable and popular search interface for text-based
searching in the Web, with all the participants favouring it. But given the total support
for Google as the most favourable interface for searching information, not all users were
Table 6.8 – Online search habits
Fig 6.8 – Online search habits
Table 6.9 – Responses to
favourable search topics
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 76 sdey1@sheffield.ac.uk - 100151932
highly satisfied with it. Around 45.46% of the users felt highly satisfied, with the overall
mean comparatively high in 4.45 out of 5.
When it came to opinion about a
favourable image retrieval system,
Google images again emerged as the
clear winner, with only a few users
indicating also about Flickr and Bing.
However, the level of satisfaction
with image searching proved to be
lower than that of textual
information searching with mean
only a little above moderate at 3.63
The overall satisfaction with the
current search interfaces has seen a
similar response at a mean of 3.63.
Table 6.10 – User satisfaction levels
for text based searching
Fig 6.9 – User satisfaction levels on
text-based searching
Table 6.11 – User satisfaction levels
for image searching
Table 6.12 – Overall traditional
search interface satisfaction levels
hing
Fig 6.11 – Overall traditional search interface satisfaction levels
Fig 6.10 – User satisfaction levels in image searching
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 77 sdey1@sheffield.ac.uk - 100151932
Perhaps one of the most important findings was to get an idea of what the tolerance
levels of the users were in finding suitable information while searching page. For this
they were asked to indicate the number of pages viewed before they stopped while
searching for information. The results indicated a trend that users rarely searched more
than 3-4 pages, with the mean at 3.273. This is however subjective depending on the
search experience of individual people, but at the same time, it indicates a level of
persistence that users would likely agree to undertake in case there was a requirement.
The users were also asked to indicate their favourable image search techniques and
related familiarity to gather an idea about the strategies employed while carrying out
such tasks, so that it can be related to this study later on. The following responses were
received:
Favourable image search techniques f %
a. Enter the desired query in text-based format in search engines and obtain the images from first page usually with no further browsing.
9 81.82
b. Follow hyperlinks and navigate page to page until desired image is retrieved 5 45.45
c. Use specific image retrieval web-site(s) (e.g. – Flickr/Picasa) 4 36.36
d. Use content based search engine(s) 1 9.091
e. Other procedures 0 0
Fig 6.12 – Number of pages viewed by users, before they
usually stop searching for information
Table 6.13 – Number of pages
viewed by users, before they
usually stop searching for
information
Table 6.14 – Table indicating the favourable image search techniques and their
corresponding frequency/percentage of support by users
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 78 sdey1@sheffield.ac.uk - 100151932
Experiences with image search techniques f %
a. Can search for a particular and unique image according to
need (e.g. – a well known architecture, famous painting etc.) 9 81.82
b. Can search for image(s) based on a broader category or
subject (e.g. - a family of endangered species, flowers etc.) 9 81.82
c. Can search for images on a specific subject (e.g. – a latest
game wall paper, a movie etc.) 9 81.82
d. Can search images with a combined multiple criteria, (e.g. –
movie wallpaper with a particular actor, within a genre and
the year of its release etc.) and have been successful in doing
so in the past.
6 54.55
Table 6.15 – Table indicating the array of experiences with image searching techniques
and their corresponding frequency/percentage of support by users
Fig 6.13 – Graph indicating the favourable
image search techniques (for each questions a
to e) and their corresponding
frequency/percentage of support.
Fig 6.14 – Graph indicating the array of
experiences with image searching techniques
(for each questions a to d) and their
corresponding frequency/percentage of support.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 79 sdey1@sheffield.ac.uk - 100151932
Users were also asked which of the procedure(s) they usually found difficult, from their
searching experiences in the past.
User Users’ views on difficult experiences in searching
S1 “Combined multiple criteria. Difficult to start”
S2 “Searching images on a specific subject”
S3 “Based on a broader category or subject needs to search several times”
S4 “Combined multiple criteria”
S5 --- NA---
S6 --- NA---
S7 --- NA---
S8 “Combined multiple criteria, because I cannot realise the information of the particular image”
S9 “The first procedure because it involves searching for unique things which may be unavailable”
S10 “Search for combined multiple criteria, hard to get the exact result, consistency”
S11 “The more general(2) and (4), where I’m exploring/learning what I am looking for”
From the above results a few patterns of emerge with respect to the difficulties faced by
users in their search methods. Users mostly have a lack of confidence in carrying out
searches that have multiple criteria or which may require crafting complex queries.
Besides, they are also issues related to consistency in the results obtained, depending on
the queries formulated in traditional systems to obtain such search results.
Finally, to address the issue of vagueness (Rodden, 2002), often encountered by users
while searching, it was asked if the users encountered situations where they accidentally
came across an information or image that was quite useful to them and whether they
would find it more favourable if the system that they usually search information from,
can address their cognitive sense to guide them to their desired requirements. Most of
the study participants agreed that they did come across situations, where they did not
know where to start from while searching for particular information. Almost all of them
however agreed that it would be useful if the system could assist in their level of
cognizance while searching for information in situations where the search criteria is not
clear.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 80 sdey1@sheffield.ac.uk - 100151932
6.3 Evaluation of user-centred expectations
As discussed in the previous chapter, a set of ratings were gathered from the users,
regarding how they felt about the key aspects and features of the information retrieval
systems that are popular today, and how they would perceive if a few changes were
introduced or added. The following table shows the results with the mean levels of
agreement on a scale of 1 to 5:
Scenarios Mean 1. Satisfaction level with current image and information retrieval interfaces most
familiar with.
3.7272727
2. Satisfaction level with the current system of clicking and browsing to retrieve
images and required information from individual pages at a time.
3.0909091
3. Perception levels if all images and information were accessible in a single
browser without browsing further.
3.4545455
4. Perception levels if all images and information available on the web or the
company could be categorised and colour coded according to particular categories within a single browser or interface.
3.6363636
5. Perception levels if all images and information had embedded visual
information (e.g.- short title or data on the image itself as a header).
3.7272727
6. Perception levels if all images and information could show their associated
further information, details and various miscellaneous properties when users are viewing them.
3.6363636
7. Perception levels if all images and information could link to other images based
on similar associated information.
3.7272727
Table 6.16 – Table indicating the users’ expectation levels on certain features in user
interfaces for information visualisation and retrieval
Fig 6.15 -
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 81 sdey1@sheffield.ac.uk - 100151932
6.4 Microsoft PivotViewer awareness
Since Microsoft PivotViewer is a
comparatively new technology with no
notable or wide implementations yet, a short
section of the experiment was also taken in
order to know, if the users were aware of this
new technology, and how familiar they were
with using it. The results indicated that around
63.6% of the users were aware of this product,
or at least, heard or read about it. However,
there was no surprise to learn that around
81.82% of them have never used it before.
This probably made this project even more
interesting, since the users were introduced to
a new technology and given complete control
of it. This also ensured that the users could
provide their ideas without any pre-conceived
notions about the interface.
Table 6.17 –Familiarity with using
Microsoft PivotViewer
Fig 6.16
Fig 6.17 –Familiarity of using Microsoft PivotViewer
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 82 sdey1@sheffield.ac.uk - 100151932
6.5 User-centred interface evaluations
6.5.1 Scenario – A
In this scenario, a similar set of tasks were carried out on St. Andrews University Library
Photographic collection and the PivotViewer interface designed for this study and the
time taken for carrying out each of the tasks and the number of clicks were recorded, as
shown in the table below.
Based on the above data collected, comparative charts were constructed to generate the
overall trend as shown below.
Table 6.18: Time taken for carrying out each of the tasks and the number of clicks- Scenario-A
Fig 6.18: Graphs comparing the time taken and number of clicks for completing scenario-
A- Task1 and 2
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 83 sdey1@sheffield.ac.uk - 100151932
From the above charts it is clear that the time taken by users to carry out the tasks in
PivotViewer was much lesser than that in the original St. Andrews University Library.
The total time taken to complete the known task was 1.54 times faster and the complex
search was around 2.46 times faster in PivotViewer, as compared to the original
interface.
The number of clicks, on the other hand, for finding the known item was 1.27 times
higher in PivotViewer. But on the other hand, for carrying out a complex search, the
number of clicks was comparatively similar (around 1.05 times less) in Pivot interface.
Overall, the time taken by users to carry out the tasks in PivotViewer was 35.1279
seconds lesser or 1.925 times (almost twice) faster than the original interface. However,
on average, the total number of clicks on both the interfaces was similar with Pivot being
0.54 clicks more.
The interfaces were also evaluated on the basis of five performance criteria as shown:
Table 6.19: Performance results based on five performance criteria for St. Andrews interface
Table 6.20: Performance results based on five performance criteria for PivotViewer interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 84 sdey1@sheffield.ac.uk - 100151932
The results indicated that PivotViewer also scored high in the performance criteria, with
overall mean of 4.25 as compared to that of St. Andrews Library interface at 2.65, i.e.
nearly 1.6 times higher than the original or existing interface. The highest margin of
difference was in the PivotViewer’s visual appeal which scored comparatively high than
the St. Andrews Interface. However, the only criteria where the two interfaces had the
smallest difference were in the confidence level of the users.
6.5.2 Scenario – B
In this scenario, a similar set of tasks were carried out on Google Images and a sample
online Pivot interface on endangered species. The time taken for carrying out each of the
tasks and the number of clicks were recorded, as shown in the table below.
Table 6.22: Time taken to carry out each of the tasks and the number of clicks-
Scenario-B
Fig 6.19: Interface evaluation summary- scenario-A
Table 6.21: Mean rating summary for
scenario-A
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 85 sdey1@sheffield.ac.uk - 100151932
Based on the above data collected, comparative charts were constructed to obtain the
overall trend as shown below:
From the above charts it is clear that, the time taken by users to carry out the tasks in
PivotViewer was comparatively lesser than that in Google Images. The total time taken to
complete the known task was twice as fast and the complex search was around 2.13
times faster in PivotViewer as compared to Google Images.
The number of clicks, for finding the information related to the known item in
PivotViewer was lesser (1.42 times) than that in Google Images. However, for carrying
out the complex search, the number of clicks was quite higher (2.1 times) in Pivot
interface as compared to Google Images.
On an average, it was identified that the time taken by Pivot interface was much lesser
than that of Google Images. Overall, the time taken by users to carry out the tasks in
PivotViewer was around 41.6 seconds lesser and around 1.938 times (almost twice)
faster than the time they found similar required information using Google Images.
However, the on average, the total number of clicks on the Pivot interface was slightly
higher (1.81) than that of the traditional search interface provided by Google.
Fig 6.20: Graphs comparing the time taken and number of clicks for completing scenario-
B- Task1 and 2
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 86 sdey1@sheffield.ac.uk - 100151932
The interfaces were also evaluated on the basis of five performance criteria as shown:
According to the five performance criteria evaluated, the overall performance of Google
Images was moderate with the lowest being in the area of satisfaction with the interface.
But the users were quite confident, on the other hand, to use both the interfaces. Overall,
the Pivot interface performed better in all the criteria than the Google Images, in carrying
out the tasks given, with the overall satisfaction levels above 4.09 in all the sections, as
compared to Google Images, which performed comparatively less. This can also be
clearly noted from the chart above.
Table 6.23: Results based on five performance criteria for Google Images
Table 6.24: Results based on five
performance criteria for PivotViewer
Table 6.25: Mean rating summary
for scenario-B
Fig 6.21: Interface evaluation summary- scenario-B
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 87 sdey1@sheffield.ac.uk - 100151932
6.5.3 Scenario – C
In this final scenario, a similar to the above to scenarios, a set of tasks were carried out
on two interfaces, the IMDb database and a Pivot interface created by Netflix. The time
taken for carrying out each of the tasks and the number of clicks were recorded, as
shown in the table below:
Based on the above data collected, comparative charts were constructed to obtain the
overall trend as shown below:
Table 6.26: Time taken for carrying out each of the tasks and the number of clicks- Scenario-C
Fig 6.22: Graphs comparing the time taken and number and number clicks
for completing scenario-C- Task1 and 2
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 88 sdey1@sheffield.ac.uk - 100151932
From the above data it can be noted that the average time taken to search a known item
in IMDb was around 16.32 seconds slower than Netflix Pivot, which in other words was
2.29 times faster to retrieve. However, the average was slightly narrower during a
complex search, with the time difference of around 11.29 seconds by which Pivot
performed (1.52 times) better than IMDb. As for the number of clicks, both for the
known item as well as the complex search, Pivot had extra clicks of 1 and 1.54 clicks on
average respectively.
Overall, the Netflix PivotViewer interface enabled the users to carry out the tasks 27.61
seconds earlier (or 1.8 times faster), than the system provided by IMDb. However in
terms of the number of clicks, Pivot interface required relatively more clicks (an overall
average of 2.54) to accomplish the same tasks.
The interfaces were also evaluated on the basis of five performance criteria as shown:
The Pivot interface continued to show overall high performance criteria, of average
rating above 4.54 in all the five sections. On the other hand, IMDb only had a maximum
performance level in the area of users’ confidence in using the interface and the
satisfaction levels provided in the results retrieved at an average of 3.45 and 3.40
respectively. Pivot interface, on the other hand, performed highly in all the areas,
especially in results retrieved, visual appeal and level of satisfaction.
Table 6.27: Results based on five performance criteria for IMDb
Table 6.28: Results based on five performance criteria for Netflix Pivot
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 89 sdey1@sheffield.ac.uk - 100151932
6.6 Summary of interface evaluations
From all the above experiments, an overall summary was computed. The net average
time taken to complete all the tasks in traditional query based interfaces were found to
have taken 104.35 seconds more than that of Pivot, i.e. in Pivot, users carried out the
tasks 1.93 times quicker than traditional interfaces. At the same time, it was also noted
that the difference in time while performing complex search was much more, between
the Pivot and traditional technologies. However, the number of clicks required to
perform the tasks were found to be quite higher (average of 4.909 clicks more) than the
other interfaces.
These findings have been tabulated in the table below for each individual task
undertaken by the participants:
In order to have a further visual insight, the summary has also been depicted in graphical
manner, for each section of task scenarios.
Table 6.29: Interface evaluation
summary- scenario-C
Fig 6.23: Mean rating summary for Scenario-C
Table 6.30: Summary of individual task results between PivotViewer and other
(traditional) interfaces
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 90 sdey1@sheffield.ac.uk - 100151932
In terms of performance criteria, PivotViewer has scored highly in all the scenario tasks
set for the users as compared to the other interfaces. While other interfaces had the
highest rating in the area of confidence, PivotViewer scored the maximum in the section
of ‘results retrieved’. This was followed by visual appeal, satisfaction, usability and finally
level of confidence. If we consider only the other interfaces, we can notice that the level
of confidence was the only section they rated highly compared to all other features. This
indicates that they mostly tend to use the interfaces since they may be quite popular
interfaces and they are confident in using them. However, in terms of Pivot interface,
confidence levels, though higher compared to other interfaces, apparently scored lower
than all other aspects indicating, that
users identified themselves more with
an experience that they liked, in a
visually appealing interface in Pivot
that is satisfying, useable and
retrieved relevant information
quickly.
Fig 6.24: Summary of time taken (left) and no of clicks (right) for all the tasks
Fig 6.25: Overall interface evaluation summary Table 6.31: Overall performance summary
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 91 sdey1@sheffield.ac.uk - 100151932
6.7. Post Experimentation Results - 1
6.7.1 Evaluation of Tasks
Three key areas were focussed in relation to the experiments taken by the users, and
their views were gathered in these aspects. The participants were asked if the
experiments were realistic, to which the
mean satisfaction level was 4 out of 5,
which meant that users did feel that the
experiments undertaken were realistic. At
the same time, they also seemed to have a
clear idea (4.36) of what they were
supposed to do. One important aspect was
in finding if the users had a new
experience through this experiment, to
which they did respond well, with a high
rating level of 4.36, to which the users felt
that this was a novel experience in
information retrieval through
visualisation.
6.7.2 Results on Pivot Interface
A few general questions regarding Pivot interface was asked and views were obtained
from the participants in terms of how strongly they felt about this new interface, based
on four criteria –
Providing a clarity of description and facets
Helping in forming a ‘mental picture’ of the information to look for
Helping in retaining interest
Easy/difficulty in using
The highest agreeable result in the above categories was that, the Pivot interface helped
in retaining the interests of the users (4.63). The second highest category was that the
PivotViewer helped the users in forming a ‘mental picture’ (4.18) of the information they
were looking for by sublimating the area of vagueness that users often tend to be in
when searching for information (Rodden, 2002; Marchioninni 1992). The users also felt
Fig 6.26: Analysis on tasks undertaken
Table 6.32: Results of
tasks undertaken
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 92 sdey1@sheffield.ac.uk - 100151932
that the descriptions in the interface and the facets were clear and helped them in finding
the required information. But it was also interesting to know, that though this was a new
interface to many (81% having never used it before), they did not find it difficult to use.
The reports are shown in table and graphical form below:
6.7.3 Technique and Quality of Results Retrieved by PivotViewer
To study the quality of the results retrieved and the technique incorporated in
PivotViewer, the analysis was based on a set of four areas, the mean agreement results
for each of which is provided in the table shown below:
Table 6.33: Results of Pivot interface
Table 6.34: Technique and Quality of results retrieved
Fig 6.27: Analysis of Pivot interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 93 sdey1@sheffield.ac.uk - 100151932
From the above readings, it can be said that though there were thousands of images
accommodated in a single interface, unlike traditional Web interfaces, the images
retrieved by PivotViewer was not hard to comprehend. Also in addition, it can be said
that the overall collection did not overwhelm the users. The details associated with the
images, though included only as a panel on the right, the users felt that it was quite
enough in terms of its content and the information they were looking. However, they did
agree mostly to the proposition, that traditional browsing interfaces could accommodate
more details than the Pivot interfaces. In terms of the technicalities in Pivot, the users
majorly found the interface very easy to zoom in and out. They also strongly felt, that the
arrangement of the pictures and/or information, according to the various categories, was
quite useful and that they were able to find out the information they were looking for
relatively quickly.
Fig 6.28: Analysis of technique/quality of results retrieved by PivotViewer
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 94 sdey1@sheffield.ac.uk - 100151932
6.8. Post Experimentation Results - 2
From the final section of the experimentation, views were gathered regarding the
advantages and disadvantages of the PivotViewer as a search interface for retrieving
visual information from a large data collection. An open-ended methodology was
adopted as explained earlier, for which the users indicated the following views regarding
the advantages of the PivotViewer as compared to the traditional interfaces:
User Advantages of Pivot interface that made searching easier as compared to
traditional interfaces
S1 “• Easy to use • Simple • Clear & objective • Efficient”
S2 “• Easier to search • Less time-consuming • Specific and clear”
S3 “It is very clear and easy to find the information that I am looking for, also I am very used to using it”
S4 “•Option for distinctive search •Easy to use”
S5 “More user friendly and targeted”
S6 “The interface of Microsoft Pivot is so user friendly”
S7 “• The visual display of search results • The interface”
S8 “The filtering functions and window in the middle”
S9 “The categories and interface of the system”
S10 “Everything is there, easy, availability, a combination of images and info”
S11 “•Consistent category of information • Easy to slice and dice the collection”
Clearly from the above data and user’s opinions, the following key advantages of using
PivotViewer can be proposed:
Easy to use: This is perhaps one of the prime advantages of PivotViewer. Almost every
participant indicated that the Pivot was easier to use than the traditional interfaces.
Searching was easier and the users felt more comfortable and interested. The interface
was simple and user friendly.
Visually appealing: Powered by Windows aero technology, Pivot was visually appealing
to the users, in terms of retrieving the images or information, grouping them into
categories, slicing and dicing them according to the filter criteria, and displaying the
results along with its associated metadata. Besides, the layout was also quite appealing.
Quick: The overall time taken by the users practically was faster than the traditional
interfaces. This can also be attributed to the friendly and easy to use interface design.
Consistent search results: It provided consistent and clear results for the information
required.
Table 6.35 – User views on PivotViewer advantages
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 95 sdey1@sheffield.ac.uk - 100151932
The users were also asked to indicate some of the disadvantages of Pivot if any, and the
following few responses were received:
User Disadvantages of Pivot interface as compared to traditional
interfaces Relevance
S1 ---- none----
S2 ---- none----
S3 “There always come out same pictures which is not useful” ✓
S4 “Fixed options seems to prohibit flexibility of search” ✓
S5 “No”
S6 “No”
S7 ---- none----
S8 “No”
S9 “The overall presentation of the view not attractive and a bit blurry” ✓
S10 “Maybe when first time use, I feel unconfident, but later, I really like it when I get familiar with the interface”
✓
S11 “Switching between it and web pages may be awkward” ✓
From the above data, user’s opinions during the experimentations, and self evaluations,
the following disadvantages can be proposed:
Rigidity: The PivotViewer is quite rigid in terms of what it presents to the users. It
generally consists of a set of categories, based on a particular domain of implementation.
As such it has little flexibility in terms of incorporating additional categories without re-
generating the entire collection. Also sometimes users may want to see all the available
pictures or information, rather than just the information provided in the data collection.
Application load time: The initial load time of the application is another disadvantage of
the interface. Though being an application, since it is Web based, this cannot be ignored
since users tend to switch between websites or may navigate away from the page by
following an URL associated with the image or information. For such open ended-
navigation, every time the user comes back to the URL with the Pivot interface, the CXML
will take time to be loaded into to the Silverlight platform. As such, overall searching
flexibility could be narrowed and slowed down. The larger the collection of information
or images to be published, the slower is the application load time.
Build Time: The initial building time for the Pivot collection is quite delaying and slow
and the time required to build is directly proportional to the size of dataset.
Table 6.36 – User views on PivotViewer’s disadvantages
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 96 sdey1@sheffield.ac.uk - 100151932
Non-sharable content: The information from a PivotViewer can only be retrieved or
noted down, but has no features to allow downloading or sharing of the contents.
Non-portable: The facets and facet categories are decided in advance, meaning that it
cannot be ported easily for different categories of items and has to be re-designed.
The users were finally asked to provide recommendations for additional functionalities
that they would like the interface to have in future and the following recommendations
were received:
User Extra functionalities and features recommended to be included Relevance
S1 “Notes, share, recommendations; Users can supply images” ✓
S2 “Voice over search” ✓
S3 “The page can change by itself then I do not need to click ‘next’”
S4 “Recommendations on search related items” ✓
S5 “I think it is a new concept for me and it was quite useful”
S6 “No”
S7 “Edit or add metadata” ✓
S8 ---- none----
S9 “Cool design and layout”
S10 ---- none----
S11 “Ad-hoc categories” ✓
User Other Comments Relevance
S1 “You can search if you do not know what you are looking for” ✓
S2 “Great job”
S3 ---- none----
S4 “Overall much better way of retrieving images and information than other applications already available and in use”
✓
S5 ---- none----
S6 ---- none----
S7 “Good experience” ✓
S8 ---- none----
S9 “The Pivot is a nice technology and will prove to be good in the years to come”
✓
S10 ---- none----
S11 ---- none----
Table 6.37 – Extra functionalities and recommendations for interfaces
Table 6.38 – Additional comments
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 97 sdey1@sheffield.ac.uk - 100151932
6.9. Summary
This chapter began with a report on the general overview of the users which participated
in the study. A majority of the study participants were IT literature which indirectly
helped in the proper evaluation of the interface. Though many of the users indicated of
having high experience in photo and information searching, only 63% indicated of
having a high confidence in searching, the major factor in it being in the difficulties
encountered in complex searching. There was a high level of satisfaction in text-based
satisfaction in Google, however in the end of the study, PivotViewer scored higher in
terms of satisfaction. Apart from this, several other factors were also evaluated in
relation to the PivotViewer as compared to the other interfaces, where in each criteria
the PivotViewer managed to score higher than the traditional interfaces. However, the
number of clicks used in PivotViewer was more as compared to the traditional interfaces
evaluated. This was followed by the report on post evaluation results that indicated a
clear, fair and realistic experimentation. The technique used by Pivot and the quality of
the results retrieved by pivot was also strongly appreciated by the users. Finally, based
on the overall results and the post evaluation questionnaires, a few advantages and
disadvantages of the Pivot system were highlighted. Results were reported for each
individual tests undertaken and was appropriately illustrated in graphical
representations throughout the results.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 98 sdey1@sheffield.ac.uk - 100151932
Chapter 7
This chapter aims to discuss the results that have been obtained from the
experimentation. It should be noted that though the sample size of the participating
users were only a few, the study process was crafted in such a manner that the users
were intrigued to think about every area of the experience they might have encountered,
and hence provide with quality response. The overall data from the obtained results have
indicated some clear patterns which are discussed here.
Microsoft PivotViewer is a technology that incorporates and deals with the
representation, storage, organisation or and access to information items, or in other
words, it matches the definition of a proper information retrieval system, as suggested
by Baeza-Yates and Ribeiro-Neto (1999). At the same, PivotViewer is an interactive
system that helps in visually representing data to amplify cognition, something that Card
et al. (1999) considers a proper information visualisation system. As a result, from the
results obtained and according to the views of the researchers, PivotViewer may be
considered as a visually interactive information retrieval system. It can also be
considered to be a custom developed system to digitise collections, as studied earlier by
Hoeber (2007) and Rodden (2002), considering its main property to digitise collections.
From the results obtained, as well as user comments received, Pivot has been identified
to provide the data quickly (Table 6.30) and consistently across data collections. It
minimises the possibility of retrieving non-relevant documents, something that
Rijsbergen (1979) indicated, should be a significant feature of any proper information
retrieval system. However this entirely depends on the semantics of the metadata
included in the data collection. While some authors have mostly concentrated on images
(Baeza-Yates and Ribeiro-Neto, 1999) many have researched on text and documents
(Voorhees, 2004). PivotViewer provides the means to bring these two categories
together. Though it was not concentrated in this project, textual data can be represented
in the same way as image data by using appropriate faceted links to documents.
[NB: Tables and figure names are often included in the discussions in relation to Chapter 6 - Results]
DISCUSSIONS
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 99 sdey1@sheffield.ac.uk - 100151932
Today, users have a high familiarity with computers (Table 6.4) and are known to
develop an overview of relations and patterns from the information they seek, through
the abstract data (Card et al, 1999). Given the level of interactivity offered by
PivotViewer, in allowing the users, to quickly and efficiently ‘slice and dice’ across
selections from an ever growing information space, it can be said that it helps in
increasing their cognitive vision and intelligence, as indicated by researchers in various
studies (Chen 2004, Nguyen and Huang, 2004, Spence, 2001; Card et al, 1999,
Shneiderman, 1996). This can be attributed to the fact, that users rated the PivotViewer
quite highly in several criteria (Table 6.33 and 6.44) in terms of the interface, the
visualisation techniques used and the quality of the results received. Addressing the area
of ‘cognitive ability’, users were particularly asked in the post-evaluation questionnaire,
if the interface helped in forming a ‘mental-picture’ to which users highly agreed with a
mean rating of around 4.18 (Table 6.28). The aspects of PivotViewer, as an Information
retrieval system and an information visualisation system is further assessed, based on
the previous research works, as well as the evaluations carried out in this study.
7.1 PivotViewer as an Information Retrieval System
Pivot thoroughly exploits the technique of ‘categorisation’ in information search, as
initiated by Smith et al. (2006), by introducing interactive exploration through facets. It
adds the richness in its interface, by enabling the users with the possibility to obtain
some really granular information that is both easy to comprehend (Table 6.34) and clear
in its descriptions, which helps in retaining the interests (Table 6.33). At the same time, it
should be mentioned here that Pivot furnishes the technology that facilitates the users to
see the larger trends in the data and gather a holistic view of the required information,
by combining it with the granulated detail.
In some collections, the value can also be in exploring and finding out something we have
never known before (Rodden, 2002). In other collections we can find some very specific
set of information, or if the information is seen as an aggregate, some insights may pop
out which one may haven’t seen otherwise. Perhaps one of the most intriguing features
of PivotViewer is its efficacy in arranging data collections, according to the desired
criteria chosen by the user and also allowing to selectively filter those collections and
gather subset of that information, smoothly and quickly. As discussed in Section 6.7.3
and shown in Fig. 6.29, users not only had a positive response in the Pivot’s ability to
quickly retrieve the information and also present a proper arrangement of the
collections, the overall time as noted down during the experimentations from Table 6.30
and Fig 6.25, also provides a further indication to this phenomenon.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 100 sdey1@sheffield.ac.uk - 100151932
Perhaps appropriate precision and recall metrics to order fetched results, as indicated by
Hoeber (2007), is used to order the results according to the information needs, keeping
in mind that the section in the info panel of Pivot also fetches related links from the Web,
which may be associated with the item being viewed. Pivot is not just about queries but
is also quite a dynamic process, which according to Kuhlthatu (1991), should be an
inherent property in information seeking, that involves the users ‘thoughts, feelings and
actions’. The results from the performance criteria, for each of the scenarios set up for
experimentations, indicated a positive result favouring PivotViewer in this aspect as
compared to the traditional systems. At the same time, by allowing users to follow their
own curiosity, Pivot effectively outlines the aspects of information foraging theory, as
described by Pirolli et al. (2001). However, Pivot has also maintained the traditional
querying technique in its interface, which according to Frunas (1997), is a good starting
point to ascertain a quality information scent.
It has often being indicated that users generally have a ‘low tolerance’ level (Spink et al.
2001), when searching for information. Also in this study, on an average, users are seen
to generally view approximately 3 pages, before they stop searching for any information
(Table 6.13, Fig 6.12). This is because users are found to mostly carry out searching only
on text or query based retrieval systems (Table 6.14 - Part a). Pivot on the other hand
introduces a multifaceted visual environment to search information, which is a
completely new experience (Table 6.32), something that most users are not familiar of
(Fig. 6.17). Provided the metadata associated with the collection are relevant, Pivot
isolates the possibility of generating instances of insignificant contents, which according
to Chau (2011), is on the rise today and often visible in the top 10 to 5 list. As traditional
information retrieval systems are said to be inefficient to handle it (Yang, 2005), Pivot,
on the other hand is a novel technology, and has a whole new dimension to unfold
through further commercial expansion into key businesses and research. Pivot also
successfully addresses the issue of dealing with short unfocussed queries, which is said
to be a challenge in traditional retrieval systems as expressed by Yang (2005).
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 101 sdey1@sheffield.ac.uk - 100151932
7.2 PivotViewer as an Information Visualisation System
Users have increasingly shown a tendency to perceive information better and carry out
tasks more efficiently in PivotViewer as compared to the other traditional systems (Table
6.30, Table 6.31). This was also shown through various information visualisation studies
previously taken (Carter 1947; Pinker 1990; Hearst 1995; Xiang et al. 2005).
As discussed in Chgapter 2, Hoeber (2007) indicated that spatial coordinates, shape,
colour etc. are required to be carefully considered as an intrinsic requirement, while
designing a proper visually interactive interface, which can perhaps also influence the
human cognizance (Card et al., 1999), something which PivotViewer has incorporated
through its simplicity and structure. In fact, though it was an interface which most of the
users were not familiar with, their confidence levels in the tasks taken compared to the
other traditional interfaces, were higher (Fig 6.26, Table 6.31). Also from the
experimentations (particularly Scenario B) carried out in this study, it was seen that the
PivotViewer interface could make use of different colour representations, to allow the
users to easily scan across datasets and look for trends and interesting patterns. In other
words, it has been designed in such a way, so that it allows the users a more natural way,
to comprehend large sets of information (Table 3.64-Qh) without getting losing their
direction. The visual distinctions in Pivot are subtle, clear and effective (Table 3.63-Qd),
something which Tufte (1997) mentioned should be the design strategy, such that it
makes the ‘smallest effective difference’.
One of the most notable features of Pivot is the that it completely eradicates the
traditional ’10 blue links’ (Broder et al, 2010) hypertext environment, with a rather
exploratory search interface, a concept which has been always supported by many
authors (Broder et al., 2010, White, 2006, Hearst, 2009). Much of Pivot’s influence can be
said to be based on the FacetLens (Lee, 2009) and FacetMap (Smith, 2006) technology
that introduced multiple categories across which data could be visualised. To add to that,
Pivot used an improved and additional feature of Deep Zoom technology that allowed
users to smoothly and continuously navigate across data. In traditional Web based
interfaces, zooming or navigating to particular information required clicking between
individual HTML pages. However, studies showed that this often instigates the users to
lose sight of the context (Broder et al., 2010). However, with Pivot, the metadata could be
obtained in the same interface, without navigating away to a different Web page. But it
should be mentioned in this regard, that due to the nature of navigational and interactive
search, on an average, Pivot required more clicks to carry out the similar tasks as
compared to the traditional interfaces (Fig. 6.25, Table 6.30).
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 102 sdey1@sheffield.ac.uk - 100151932
In addition, the overall experimentation results from the users indicated that the
smoothness and continuity provided by Pivot in navigating across the data, helped the
users to get a clear picture and understand what they were looking for and the also
realise how they reached there. Gary Flake, a Microsoft technical and director of Live
Labs, indicated that if there is a sudden transition, users lose their way; on the other
hand, if it is smooth and continuous, users tend to have a mental model of how to they
reached the particular location (Microsoft News Center, 2010).
According to the words of the director of reporting and digital analytics within
Microsoft’s global marketing organization, Michael Moore, “Pivot is a beautiful merge of
quantitative and qualitative analysis” which “facilitates a subjective analysis of the
underlying objective data”. However, one of the greatest drawbacks of PivotViewer is
that, the essential categories for the datasets are picked in advance and hence the system
is difficult to port to other data collections with different items, something that Smith et
al. (2006) found in a similar study for information visualisation. Also the facets may be
evolving from time to time, meaning continuous changes in the schema is required from
time to time. This may indicate the use of higher number of categories, which on the
downside may also overwhelm the users, due a cognitive overload as indicated by
Wilson and Schraefel (2008). But overall, Pivot offers an interface that makes sure that
the trends and patterns of information appear immediately to the user through an
experience that is not possible when data is stuck in hyperlinked pages or constituted in
the form of rows and columns.
7.3 Summary
In this Chapter the results obtained in the previous chapter has been thoroughly
discussed with respect to the background literature discussed in Chapter 2. Special care
was taken to the visualisation and retrieval aspect of the PivotViewer. The results clearly
indicated a shift in the appreciation of the users from traditional search interfaces to a
more interactive system, something that has been discussed, prototyped and improved
by many authors over the years. This piece of analysis gives a closer look at the potential
of PivotViewer as an interface which may open doors to novel and far efficient ways of
information searching on the Web in the future.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 103 sdey1@sheffield.ac.uk - 100151932
Chapter 8
8.1 Research Summary
The primary theme of this piece of research has been to study and explore the
techniques in Microsoft PivotViewer, to support information visualisation and
interaction on a set of data collections, and understand the potential advantages and
disadvantages that this interface may impose, as compared to the traditional interfaces,
through user-centred evaluations. To achieve this, a new prototype interface was
designed in PivotViewer, based on the St. Andrews University collection, and the system
was compared with the pre-existing and official interface available online. In addition,
two similar sets of experiments were carried out. One of the experiments was to evaluate
an open ended-search interface in Google Images and compare it with a Pivot interface
based on a particular search domain. The other was based on evaluating between the
traditional interfaces of IMDb and Netflix PivotViewer, both of which were operating in
the same domain. The experiments ensured a challenging and holistic evaluation of the
Pivot interface based on an array of defined performance criteria. The results gathered
from the study indicated a positive graph, illustrating the advantages of granting the
users a visually enriching experience in information retrieval, through interactive
manipulations and explorations.
In this chapter, the primary lists of objectives achieved are briefly discussed, and a
summary of results are illustrated. Finally, the limitations of this research are highlighted
and finally concluded with an outline of possible future research.
CONCLUSION
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 104 sdey1@sheffield.ac.uk - 100151932
8.2 List of Objectives
The list of objectives described in Chapter 2 was closely followed in this research and
met successfully. This section summarises these achievements:
Aim -1
To carry out a study-oriented investigation of information retrieval and visualisation
systems, along with the exploration of associated user behaviour, in information searching.
Fulfilment of objectives:
In Chapter 3, detailed literature reviews were carried out in the areas of:
Information Retrieval
A study of traditional forms of information retrieval systems by various authors
were discussed in Section 3.1 (Chapter 3), followed by insights on Web-based
information searching. Information retrieval techniques, in large faceted datasets,
were also discussed, keeping in relation to the present study. Literature on web user
interfaces was also briefly discussed.
Information Visualisation
A brief introduction on information visualisation was provided and related studies
were discussed in Section 3.2, with certain emphasis on key theories. This was
followed by discussions and reviews on Web-based information visualisation by
eminent authors. Finally literature on visualisation of large faceted datasets was
reviewed and discussed.
User behaviour in information searching
Behaviour of users in information searching was discussed in section 3.3, along with
some references to key literature in this area.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 105 sdey1@sheffield.ac.uk - 100151932
Aim -2
Design and implement a Pivot collection with a large dataset.
Fulfilment of objectives:
Objective 1:
To study the techniques and features, to design and build pivot collections.
In Chapter 4, the key features and techniques used in designing, developing and
publishing a Pivot collection has been identified and discussed.
Objective 2:
To transform the historic photo collection available in St. Andrews University Library,
into a prototype Pivot collection.
In Chapter 5, the technique to implement a PivotViewer collection based on 10,000
images of St. Andrews University Library historic collection was successfully
implemented and tested with users.
Aim -3
Evaluation and analysis of Pivot interface compared to traditional Web interfaces for
information retrieval and visualisation.
Fulfilment of objectives:
Objective 1: To perform a comparative performance evaluation between the Pivot
interface developed on St. Andrews University photographic collection and the original
interface and explore the results.
The evaluation was successfully carried out as explained in Section 5.7.8, with a
detailed result analysed in Section 6.5.1.
Objective 2: To perform a comparative analysis and performance evaluation, on
image retrieval on a particular domain, between an open-ended interface (Google
Images), with that of a Pivot interface on a particular dataset.
The evaluation was successfully carried out as explained in Section 5.7.8, with a
detailed result analysed in Section 6.5.2.
Objective 3: To perform a comparative analysis and evaluation of two interfaces
dedicated to the similar domain of information retrieval service - IMDb and Netflix
PivotViewer interface.
The evaluation was successfully carried out as explained in Section 5.7.8, with a
detailed result analysed in Section 6.5.3.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 106 sdey1@sheffield.ac.uk - 100151932
8.3 Results
From the observations and tests carried out, PivotViewer emerged as a clear winner in
terms of speed, as well as the performance criteria set. The time taken by Pivot Viewer
was comparatively less (almost half) the time taken to do the same task in other
interfaces. However, the number of clicks in Pivot was slightly higher. In terms of
performance, PivotViewer scored higher than traditional interfaces in all the criteria:
satisfaction, usability, visual appeal, confidence, and results retrieved. Marginal
differences favouring Pivot more, compared to others, were in the area of satisfaction
and visual appeal; and the differences were closest in the area of confidence. Users
agreed highly to the fact that it was a novel experience, which helped them form a clear
mental picture of the collections. The interface also proved to be quick and quite easy to
use. Overall Pivot provided a rich multifaceted multimedia visual experience to the users
and proved to be more efficient and acceptable to the users as compared to the
traditional search interfaces.
8.4 Limitations
Given the scope of the project, several limitations have been identified:
The evaluation of the system has been based on the handful of user experiments
that has been carried out. It may not fully reflect the overall sentiment of a vast
majority of users. But care has been taken to include a varied set of users, and the
overall positive results indicate a success. However, it could have been better to
include a wide array of participants, to further affirm the results by achieving
saturation in result quality.
This project does not include the evaluation of the stand alone system and
measurement of internal technicalities, in terms of processing speed and
resource consumption levels, with that of the servers evaluated against. This is
due to lack of accessibility of the systems as well as the limited time.
The St. Andrews University Library Photographic collections had more number of
photo collections (around 30 thousand) as compared to the Pivot prototype
collection which had 10,000 pictures analysed. This is due to the lack of
resources available in publishing the magnitude of data on the local server
provided. However, though this may not indicate an ideal comparison situation,
the focus was more on the performance criteria of the interface, in terms of
delivering overall user experience.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 107 sdey1@sheffield.ac.uk - 100151932
The experiments were based on static data collections only. It would be
interesting to implement and evaluate dynamic features.
Colour schemes have not been incorporated in building the St. Andrews
Collection to visually distinguish data, due to lack of resources, which could
otherwise have been better. However, we have evaluated a complete interface
with proper imagery, through the other Pivot interfaces included in the test.
The quality of the images obtained from the St. Andrews Collection was not of a
very high resolution and hence the experience may not have been optimal.
8.5 Future Research
In this dissertation, an image retrieval interface has been successfully designed using
PivotViewer which proves to be a promising tool for the next-generation of user
interfaces. It allows users to navigate and manipulate the search criteria, to retrieve
information from large data collections, using faceted metadata flexibly. Despite the fact
that the interface was new to the users and they were not familiar in using it before, it
was strongly preferred by most of the participants of the experiments. This also
highlights the fact that, categorisation in image and information retrieval, along with
proper visualisation, is a successful approach, in the area of search and retrieval systems.
Studies in future need to be carried out to compare the effects and possibilities of
including custom personalisation of the interface, history, sharing and researching the
efficacy of the technology, on the probability of text collections. Besides, there are also
several areas of improvements that can be added and experimented in further research,
by introducing further clarity in establishing facet relationships by using colour schemes
and running the application on a higher configuration system with dedicated GPU.
Besides, further thoughts needs to be introduced in incorporating tools, specifically for
evaluating such systems, by keeping in mind, the challenges and problems in carrying
out proper evaluation for information visualisation systems (Plaisant, 2004; Rhyne,
2004), by ensuring qualitative and quantitative effectiveness. At the same time, it is also
required to carry out further iterations on the designing the system and enable in the
deployment of the system online to a wider array of users for further assessments, to
ensure a qualitative saturation of results.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 108 sdey1@sheffield.ac.uk - 100151932
Through this research, several additional features of improvements have been identified
that that could enrich the user experiences in information retrieval systems. For
example, currently there are no easy methods to gather information on all the research
work written by authors in the area of information visualisation in the UK (or around the
world), since publication works have mostly been distributed across several established
journal portals. It would be interesting to study and investigate ways, to bring together
all the papers, by scaling to a larger dataset, comparable to the combined repository of
databases, such as Emerald, ACM, IEEE and use URLs to directly link them to these sites
through PivotViewer after the users are able to slice and dice to obtain the required
paper. In this way, there could be a central repository of information resources that can
be accessible via the pre-existing systems without any infringement on the copyright
laws, but rather a joint effort. Besides, studies can also be carried out on organisations to
tap useful information and investigate the means to enable proper channels, to gather
business intelligence and evaluate it. This research has dealt with static datasets, in
future dynamic datasets needs to be implemented and assessed.
Several challenges are also expected in this research, one of which being the inclusion of
many facets. Apart from finding ways to handle many facets, an optimal technique to
chalk out meaningful and important facets needs to be also incorporated. Currently in
PivotViewer, there are no means for the users to save their search points and retrieve a
pre-existing filtered search state. It would be interesting to investigate the possibilities of
tweaking the Pivot SDK to include this additional feature, apart from allowing other
features such as history and sharing.
Due to the constraints and time limits for the project, internal system based evaluations
were not carried out for the project to assess the performance as compared to the pre-
existing system. In future study, a thorough internal evaluation is also proposed to be
carried out, that should properly analyse the technical specifications along with testing
the processing speeds while filtering large datasets. In order to achieve this, several
prototypes need to be designed, with variable and dynamic datasets, on several domains.
Proper overall evaluation on all the systems would help in achieving saturated results,
which can then be properly analysed and assessed to identify the risks and benefits, by
carrying out feasibility studies. This could also be successful set up interest groups in this
area, leading major players to carry out transformations in the way users access data
today, by bringing in the possibility of a next generation of experience in information
visualisation and manipulation of data collections.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 109 sdey1@sheffield.ac.uk - 100151932
Finally, it should be noted that technology is always shifting and moving from
proprietary architecture. There is a possibility of shift from plug-in architecture to a
more open kind of architecture with the introduction of HTML 5. Being a platform
specific product, there might be several challenges in future. It will be interesting to
study how Microsoft Corporation responds to this situation, and how overall we find a
further solution in this spectrum, for the generations to come.
8.6 Summary
In this dissertation, PivotViewer has been described as an interesting visualisation
interface that supports interactive exploration, enabling cognitive sense-making through
faceted data collections by highlighting trends and relationships within the contents. It
introduces a new level of interactivity and features that enable users to easily retrieve
the required information, identify and navigate across related patterns, and compare the
trends.
To highlight the utility of the interface, several insights gained were reported and
graphically analysed from participating users, who explored across three implemented
Pivot interfaces (one designed for the purpose of the project, two pre-existing in the Web
– Endangered Species sample PivotViewer from Microsoft and Netflix PivotViewer),
while each time comparing with a traditional interfaces (St. Andrews University Library
Photographic Collection, Google Images, IMDb). The results were also presented from a
formative user study taken to identify the performance and usability of the interface as
compared to the traditional interfaces. PivotViewer was quite innovative, enjoyable and
interactive to use, according to the users in exploring the tasks given and overall they
liked it and rated it highly. Finally, potential future work using the PivotViewer has been
presented.
From the study it was learnt that it is essential today, to allow users to follow their own
curiosity in knowledge searching. Allowing the users to swim through contents,
undertaking specific tasks or gathering data, based on some pre-defined criteria in mind
in a fast, easy and effective way in today’s fast-paced environment, is no longer an added
luxury but actually a necessity
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 110 sdey1@sheffield.ac.uk - 100151932
Manual information gathering is all about finding a few items from the various
collections, making a choice and then acting on that choice based on the data available. If
the user is able to browse through huge amount of aggregated data, it is much easier and
interesting to browse through that data. At the same time, if it could be something visual
and easy to relate to, like PivotViewer, it could make information retrieval to a whole
new level. Besides, information is no longer the domain of just a few individuals but has
now spread to a completely new genre of business and market competition with the
growth of industries. Information retrieval and visualisation systems, such as
PivotViewer, are increasingly essential for business intelligence and to explore
interactive trends. The impact of information, through developers, designers and
business decision makers are effectively changing the way our world functions today.
With a tool such as PivotViewer, it is estimated that large volumes of data could be
interacted with, in such a way that it will bring in new insights, gathering more wisdom
from what was earlier obscure and mysterious, and enable an insight that will help users
to make better decisions as well as formulate improved policies. Since Pivot has the
technology to work with almost any data format. As such with an increase in the number
of potential users, the domain of its expansion could be as vast and varied as the
information available today, in other words, practically limitless.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 111 sdey1@sheffield.ac.uk - 100151932
APPENDIX
List of interesting Pivot applications
http://memorabilia.hardrock.com/
http://www.molbiol.ox.ac.uk/data.shtml
BioPivot – University of Oxford, UK http://www.molbiol.ox.ac.uk/data.shtml
Buxton Collection – Bill Buxton’s UX devices http://research.microsoft.com/en-
us/um/people/bibuxton/buxtoncollection/pivot.htm
Classic Cardboard – ProjectVerona http://www.classiccardboard.com/
Cricket Video Vault – Blackcaps http://www.blackcaps.co.nz/vault/
Flickr Demo – LobsterPot http://pivot.lobsterpot.com.au/flickr
percollate – social media dashboard http://www.percollate.com/Demo
PhotoPivot – organize your photo collection
http://www.photopivot.com/photopivot/
tweetpivot – pivot the twitterverse http://www.tweetpivot.com/
Microsoft BI Customers – BI > Solutions > Customers
http://www.microsoft.com/bi/en-us/Solutions/Pages/Customers.aspx
MVPChat – MVP Twitter Threads http://mvpchat.championds.com/
Netflix instant watch movies+ – Windows Azure Team
http://netflixpivot.cloudapp.net/
PivotViewer Interactive Demos – Official Microsoft Silverlight
http://www.microsoft.com/silverlight/pivotviewer/
The TEAM Project – UCSD and Conservation International
http://gis.team.sdsc.edu/teamimages/
Top SharePoint Internet Sites – WSSDEM: http://www.wssdemo.com/livepivot/
Windows Phone 7 Marketplace - WP7 App Hub: http://www.wp7apphub.com/
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 112 sdey1@sheffield.ac.uk - 100151932
Participant Consent Form
This form is based on a pro forma provided by The University of Sheffield. It was used to obtain
consent of volunteer participation for the experimentations undertaken.
Title of Project: Name of Researcher: Participant Identification Number for this project: Please initial box 1. I confirm that I have read and understand the information sheet/letter
(delete as applicable) dated [insert date] for the above project and have had the opportunity to ask questions.
2. I understand that my participation is voluntary and that I am free to withdraw
at any time without giving any reason. Insert contact number here of lead researcher/member of research team (as appropriate).
3. I understand that my responses will be anonymised before analysis. I give permission for members of the research team to have access to my anonymised responses.
4. I agree to take part in the above research project. ________________________ ________________ ____________________ Name of Participant Date Signature (or legal representative) _________________________ ________________ ____________________ Name of person taking consent Date Signature (if different from lead researcher) To be signed and dated in presence of the participant _________________________ ________________ ____________________ Lead Researcher Date Signature To be signed and dated in presence of the participant Copies: Once this has been signed by all parties the participant should receive a copy of the signed and dated participant consent form, the letter/pre-written script/information sheet and any other written information provided to the participants. A copy for the signed and dated consent form should be placed in the project’s main record (e.g. a site file), which must be kept in a secure location.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 113 sdey1@sheffield.ac.uk - 100151932
Participant Information Sheet
The aim of this research is to carry out an investigation of information retrieval and
visualisation systems, along with an exploration of user behaviour in information
searching. Secondly it is aimed to design and implement a Pivot collection with a large
dataset and finally carry out a comparative evaluation and analysis of Pivot interface
with respect to traditional Web interfaces for information retrieval and visualisation.
The aim of the lab session you have been invited to participate in today is to assess the
functionality and ease of use of three PivotViewer interfaces as compared to three
traditional image and information search interfaces that allow users to browse and
navigate for photos and information. The feedback you give will be used in the
assessment of PivotViewer interface as a visually interactive information retrieval
system.
The evaluation will consist of an opening questionnaire, to record some personal
characteristics (such as age, education background etc.) and your previous experience
using the internet and search engines.
Following brief instructions from the researcher, you will then be invited to familiarise
yourself with the software being evaluated, before undertaking some suggested search
tasks. Most of these tasks have no right or wrong solution, their aim is to assess how
you make use of the software.
Finally, you will be asked to complete a post evaluation questionnaire, giving your
opinion of the software.
Participation in this evaluation is entirely optional. You may choose not to answer any
question or participate in any task. You may also choose to withdraw from the
evaluation at any time.
All data collected in the evaluation will be completely anonymised (i.e. you will not be
personally identifiable from the data). To take note of the time and number of clicks in
certain areas of the evaluation, special software shall be used. All associated data
collected in this evaluation, will be deleted within one year of the project’s completion.
If you have any concerns during or after the evaluation, you may talk to the researcher
Souvik Dey (sdey1@shef.ac.uk) or to the project supervisor, Dr Paul Clough
(p.d.clough@sheffield.ac.uk, Tel 0114 222 2664).
Many thanks for your time and co-operation, which is greatly appreciated.
- Souvik Dey
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 114 sdey1@sheffield.ac.uk - 100151932
1. Preliminary Questionnaire I - Basic Details
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 115 sdey1@sheffield.ac.uk - 100151932
II – General Familiarity
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 116 sdey1@sheffield.ac.uk - 100151932
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 117 sdey1@sheffield.ac.uk - 100151932
III - Evaluation of user-centred expectations
IV - Pivot Awareness
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 118 sdey1@sheffield.ac.uk - 100151932
2. User–centred Interface Evaluation
I. Scenario – A: St. Andrews University Library Photo Collection
i. Task list for Implemented Interface
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 119 sdey1@sheffield.ac.uk - 100151932
ii. Evaluation of Scenario – A:
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 120 sdey1@sheffield.ac.uk - 100151932
II - Scenario – B:
i. Task list for Scenario B (Pre-existing interface-1) Open ended browsing/retrieval with Google Images vs. Pivot Collection on endangered species.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 121 sdey1@sheffield.ac.uk - 100151932
ii. Evaluation of Scenario – B:
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 122 sdey1@sheffield.ac.uk - 100151932
III - Scenario – C:
i. Task list for Scenario C (Pre-existing interface-2)
Close ended visual information retrieval on movies using IMDB vs. Netflix Pivot Collection:
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 123 sdey1@sheffield.ac.uk - 100151932
ii. Evaluation of Scenario – C:
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 124 sdey1@sheffield.ac.uk - 100151932
3. Post-experimentation questionnaire
i. Part 1
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 125 sdey1@sheffield.ac.uk - 100151932
ii. Part 2
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 126 sdey1@sheffield.ac.uk - 100151932
Ahlberg, C., (1996). “Spotfire: an information exploration environment”. ACM SIGMOD
Record, 25(4), pp. 25-29.
Ahlberg, C. and Shneiderman, B. (1994a). “Visual information seeking: Tight coupling of
dynamic query filters with starfield displays”. In CHI ’94: Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems. Pp. 313–317. ACM.
Ahlberg, C., Shneiderman, B. (1994b) “The alphaslider: a compact and rapid selector”, In
Proc. of the SIGCHI conference on Human factors in computing systems. Pp. 365 – 371.
Ahlberg, C., Williamson, C., and Shneiderman, B., (1992). “Dynamic queries for
information exploration: An implementation and evaluation”, In: Proc. CHI ’92, pp. 619-
626.
Andrews, K. (2006). “Evaluating information visualisations”. In Proceedings of the 2006
AVI workshop on Beyond time and errors: novel evaluation methods for information
visualization (BELIV '06). ACM, New York, NY, USA.
Auber, D., Delest, M., Domenger, J-P., Ferraro, P., Strandh, R. (2003). “EVAT - Environment
for Visualization and Analysis of Trees”. In Poster Compendium of IEEE Information
Visualization.
Baeza-Yates, R and Ribeiro-Neto, B. (1999). Modern Information Retrieval. Addison
Wesley
Bates, M. J. (1989) “The design of browsing and berrypicking techniques for the online
search interface”. Online Information Review, 13(5) pp. - 407– 424.
Belkin, N. J., Marchetti, P. G. and Cool, C. (1993). “BRAQUE: Design of an interface to
support user interaction in information retrieval”. Information Processing and
Management, 29(3), pp. 325-344.
REFERENCES
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 127 sdey1@sheffield.ac.uk - 100151932
Benderson, B. B. and Shneiderman, B. (2003). The Craft of Information Visualization.
Morgan Kaufmann.
Ben-Shaul,I., Herscovici,M., Jacovi,M., Maarek, Y. S., Pelleg, D., Shtalhaimb, M., Soroka, V.
and Ur, S. (1999). “Adding support for dynamic and focused search with Fetuccino”. In
Proceedings of the 8th International. Word Wide Web Conference (WWW8). Pp.-575–587.
Bhavanani, S. K. (2002). “Domain-Specific Search Strategies for the Effective Retrieval of
Healthcare and Shopping Information,” CHI 2002, pp. 610-611.
Brin, S. and Page, L. (1988). “The anatomy of a large-scale hyper-textual web search
engine”. In: Proceedings of the Seventh International World Wide Web Conference.
Broder, A. (2002). “A taxonomy of web search”. SIGIR Forum. 36(2), pp. 3-10.
Broder, B., Churchill, E. F., Hearst, M., Pell, B., Raghavan, P. and Tomkins, A. (2010)
“Search is dead!: long live search (panel)”. In WWW 2010: World Wide Web Conference.
pp- 1337–1338. ACM.
Burel, G and Cano, A. E. (2010). “Understanding web documents using semantic overlays.
In Proceedings of the 15th international conference on Intelligent user interfaces (IUI '10)”.
pp. 405-406. ACM, New York, NY, USA
Buja, A., Cook, D., and Swayne, D. (1996). “Interactive high-dimensional data
visualization”. J. Comput. Graph. Statist. 5(1), pp. 78–99.
Byrd, D. (1999). “A scrollbar-based visualization for document navigation”. Proc. of the
fourth ACM Conference on Digital Libraries. Pp. 122-129.
Campbell, I. (2000). “Interactive evaluation of the Ostensive Model using a new test
collection of images with multiple relevance assessments”. Information Retrieval. 2(1),
pp. 85-112.
Card, S. K., Mackinlay, J. D. and Shneiderman. B. (1999). Using vision to think, chapter 1:
Information Visualization, pp. 1–34.
Card, S. K., Robertson, G. G. and York, W. (1996). “The WebBook and the Web Forager: an
information workspace for the world-wide web”. In CHI ’96: Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems. ACM.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 128 sdey1@sheffield.ac.uk - 100151932
Carter, L. F. (1947). “An experiment on the design of tables and graphs used for
presenting numerical data”. J. Appl. Psychol. 31, pp. 640–650.
Cellary, W., Wiza, W., and Walczak, K. (2004). “Visualizing Web search results in 3D”.
IEEE Comput. 37(5), pp. 87–89.
Chau, M. (2011). “Visualizing web search results using glyphs: Design and evaluation of a
flower metaphor”. ACM Trans. Manage. Inf. Syst. 2, 1(2). pp. 2-27.
Chau, M. and Chen, H. (2008). “A machine learning approach to Web page filtering using
content and structure analysis”. Decis. Supp. Syst. 44(2), pp. 482–494.
Chau, M., Shiu, B., Chan, I., and Chen, H. (2007). “Redips: Backlink search and analysis on
the Web for business intelligence analysis”. J.Amer.Soc.Inf.Sci.Technol. 58(3), pp. 351–
365.
Chen, C. (2003). “Review of the Journal Information Visualization”. Journal Information
Visualization. 9(9). http://www.dlib.org/dlib/september03/09journalreview.html
[Accessed 19 May, 2011].
Chen, H., Chau, M., and Zeng, D. (2002). “CI spider: A tool for competitive intelligence on
the Web”. Decis. Supp. Syst. 34(1), pp. 1–17.
Chen, H., Houston, A. L., Sewell, R. R., and Schatz, B. R. (1998). “Internet browsing and
searching: User evaluation of category map and concept space techniques”.
J.Amer.Soc.Inf.Sci.49(7), pp. 582–603.
Chen, H., Lally, A. M., Zhu, B., and Chau, M. (2003). “HelpfulMed: Intelligent searching for
medical information over the Internet”. J. Amer. Soc. Inf. Sci. Technol. 54(7), pp. 683–694.
Chernoff, H. (1973). “The use of faces to represent points in k-dimensional space
graphically”. J.Amer.Statist. Assoc. 68, pp. 361–368.
Chuah, M. C. and Eick, S. G. (1998). “Information rich glyphs for software management
data”. IEEE Comput. Graph. Appl. 18(4), pp. 24–29.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 129 sdey1@sheffield.ac.uk - 100151932
Church, K., Neumann, J., Cherubini, M. and Oliver, N. (2010). “SocialSearchBrowser: a
novel mobile search and information discovery tool”. In Proceedings of the 15th
international conference on Intelligent user interfaces (IUI '10). pp. - 101-110. ACM, New
York, NY, USA
Clarkson, E. C. Desai, K. and Foley, J. D. (2009) “ResultMaps: Visualization for search
interfaces.” TVCG: Transactions on Visualization and Computer Graphics, 15(6), pp. 1057–
1064.
Clough, P, Sanderson, M. (2003). “The CLEF 2003 Cross Language Image Retrieval Track”.
Lecture Notes in Computer Science. Springer: Berlin, Heidelberg. 3115
Clough, P, Muller, H, Deselaers, T., Grubinger, M. Lehmann, T. M., Jensen, J. and Hersh, W. (2003). “The CLEF 2005 Cross Language Image Retrieval Track”. C. Peters et al. (Eds.): CLEF 2005, LNCS 4022, pp. 535–557, 2006. Springer-Verlag: Berlin Heidelberg
Cutrell, E., Robbins, C., Dumais, S., and Sarin, R., (2006). “Fast, flexible filtering with Phlat
– Personal search and organization made easy”, In Proc. CHI ’06.
Cutting, D. R., Karger, D. R., Pedersen, J. O. and Tukey, J. W. (1992). “Scatter/Gather: A
cluster-based approach to browsing large document collections.” In Proceedings of
SIGIR'92, pp. 318-329. ACM.
Dix, A., Quigley, A., Subramanian, S. and Terrenghi, L. (2010). “Workshop on coupled
display visual interfaces”. In: Giuseppe Santucci (Ed.) Proceedings of the International
Conference on Advanced Visual Interfaces (AVI '10). pp. 408-410 ACM, New York, NY, USA
Ebert, D. S., Kukla, J. M., Shaw, C. D., Zwa, A., Soboroff, I., and Roberts, D. A. (1997).
“Automatic shape interpolation for glyph-based information visualization”. In
Proceedings of the IEEE Visualization Conference.
Efthimiadis, E. N. (1996). “Query expansion. Annual Review of Information Systems and
Technology” (ARIST), 31, pp. 121–187
Ellis,G. and Dix, A. (2006). “An explorative analysis of user evaluation studies in
information visualisation”. In Proceedings of the 2006 AVI workshop on Beyond time and
errors: novel evaluation methods for information visualization (BELIV '06). pp. 1-7 ACM,
New York, NY, USA
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 130 sdey1@sheffield.ac.uk - 100151932
English, J., Hearst,M., Sinha,R. Swearingen, K. and Yee, K.P. (2002). “Flexible search and
navigation using faceted metadata”. In SIGIR ’02: conference on Information Retrieval, pp.
11–15
Enser, P. (2000) “Visual image retrieval: seeking the alliance of concept-based and
content-based paradigms”. Journal of Information Science. 26(4). pp. 199-210
Fabrikant, S., (2001). “Evaluating the usability of the scale metaphor for querying
semantic spaces”, COSIT 2001, pp. 156-172.
Faisal, S., Cairns, P., Blandford, A. (2007). “Challenges of Evaluating the Information
Visualisation Experience” In: Ramduny-Ellis,D. & Rachovides,D. (Eds) Proceedings of the
21st BCS HCI Group Conference HCI 2007. 3-7 September 2007. Lancaster University. Pp.
167-170. British Computer Society, UK.
Fanea, E., Carpendale, S., and Isenberg, T. (2005). “An interactive 3D integration of
parallel coordinates and star glyphs”. In Proceedings of the IEEE Symposium on
Information Visualization (INFOVIS). Pp. 149–156.
Fekete, J., Plaisant, C., (2002). “Interactive information visualization of a million items”,
In: Proc. InfoVis '02, pp. 117.
Ferrara, J. (2008). Search Behaviour Patterns [online]. US: Boxes and Arrows.
www.boxesandarrows.com/view/search-behavior[Accessed 15 August 2011].
Forsell, C., Seipel, S., and Lind, M. (2006). “Surface glyphs for efficient visualization of
spatial multivariate data”. Inf. Visualiz. 5, pp. 112–124.
Ford, N. J., Wilson, T.D., Foster, A.E., Ellis, D., Spink, A. (2002) “Cognitive styles in
information seeking analysis”. Journal of the American Society for Information Science and
Technology, 53(9), pp. 728-735.
Fowler, F.J. (1995). Improving survey questions : design and evaluation. London: Sage.
Ghemawat, S., Gobioff, H. and Leung, S. T. (2003). “The google file system”. In:
Proceedings of the ACM Symposium on Operating System Principles. ACM, NY, USA.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 131 sdey1@sheffield.ac.uk - 100151932
Goodburn, D. P. J., Vernik, R. J. (1999). Integrated Visualisation and Description of Complex
Systems. Salisbury, DSTO.
Graphics, Visualization, & Usability Center (1998). GVU’s 10th WWW user survey [online]
http://www.gvu.gatech.edu/user_surveys/survey-1998-10/ [accessed Aug 02, 2011]
Hasan,M., Golovchinsky, G., Noik, E., Charoenkitkarn, N., Chignell, M., Mendelzon, A., and
Modjeska, D. (1995). “Browsing local and global information”. In Proceedings of the
Conference of the Centre for Advanced Studies on Collaborative Research. Pp. 228–240.
Hering, E (1964). Outlines of a Theory of Light Sense (Grundzge der Lehr von Lichtsinn,
1920). Harvard University Press.
Hearst, M. (2009). Search User Interfaces. 1st Edition. USA: Cambridge University Press
Hearst, M. (1999). “User interfaces and visualization”. In Ricardo Baeza-Yates and
Berthier Ribeiro-Neto, (Eds), Modern Information Retrieval. Addison-Wesley, 1999.
Hearst, M. A. (1995) “TileBars: Visualization of term distribution information in full text
information access”. In CHI ’95: Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems, pp. 59–66. ACM.
Hetzler, B., Whitney, P., Martucci, L., and Thomas, J., (1998) “Multi-faceted insight
through interoperable visual information analysis paradigms”, In: Proc. InfoVis ’98, pp.
137-144.
Hoeber, O (2007). “A study on interactive visualisation for web information retrieval”.
University of Regina, US.
Hong, J. Y., D'Andries, J., Richman, M., Westfall, M. (2003). “Zoomology: Comparing Two
Large Hierarchical Trees”. In Poster Compendium of IEEE Information Visualization.
Hšlscher, C & Strube, G. (2000). Web Search Behavior of Internet Experts and Newbies
[online]. Center for Cognitive Science, Institute for Computer Science & Social Research,
University of Freiburg: Germany. www9.org/w9cdrom/81/81.html[Accessed 16 Aug
2011].
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 132 sdey1@sheffield.ac.uk - 100151932
Janjusevic,T., Zhang, Q., Chandramouli, K.and Izquierdo, E. (2010). “Concept based
interactive retrieval for social environment”. In Proceedings of the 2010 ACM workshop on
Social, adaptive and personalized multimedia interaction and access (SAPMIA '10). ACM,
New York, NY, USA.
Irani, P., Ware, C., (2003). “Diagramming information structures using 3D perceptual
primitives”. ACM Transactions on Computer-Human Interaction. 10(1), pp. 1-19.
Jansen, B. J. and Pooch, U. (2001). “A review of Web searching studies and a framework
for future research”. Journal of the American Society for Information Science and
Technology, 52(3), pp-235–246.
Jerding, D. and Stasko, J. (1998). “The Information Mural: A Technique for Displaying and
Navigating Large Information Spaces”. IEEE Transactions on Visualization and Computer
Graphics, pp. 257–271.
Jardine, N and Rijsbergen, C. J. van (1971). “The use of hierarchic clustering in
information retrieval”. Information Storage and Retrieval. 7(5), pp. 217-240.
Jones, S. (1999). “VQuery: A graphical user interface for boolean query specification and
dynamic result preview”. In UIST 1999: Symposium on User Interface Software and
Technology, pp. 143–151. ACM.
Kobayashi, M. and Takeda, K. (2000). “Information retrieval on the Web”. ACM Computing
Surveys, 32(2), pp. 114–173
Kleiner, B. and Hartigan, J. A. (1981). “Representing points in many dimensions by trees
and castles”. J. Amer. Statist. Assoc. 76, pp. 260–269. Cambridge, USA
Kobayashi, M. and Takeda, K. (2000). “Information retrieval on the Web”. ACM Computing
Surveys, 32(2), pp. 114–173.
Komlodi, A., Sears, A., Stanziola, E., (2004). “Information Visualization Evaluation
Review”. ISRC Tech. Report, Dept. of Information Systems, UMBC. UMBC-ISRC-2004-1.
Kosala, R and Blockeel, H. (2000). “Web mining research: A survey.” SIGKDD Explorations,
2(1), pp. 1–15.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 133 sdey1@sheffield.ac.uk - 100151932
Kot, B., Wuensche, B., Grundy, J. and Hosking, J. (2005). “Information visualisation
utilising 3D computer game engines case study: a source code comprehension tool”. In:
Proceedings of the 6th ACM SIGCHI New Zealand chapter's international conference on
Computer-human interaction: making CHI natural (CHINZ '05). pp. 53-60. ACM, New
York, NY, USA.
Kumar, N. and Benbasat, I. (2004). “The effect of relationship encoding, task, type, and
complexity on information representation: An empirical evaluation of 2D and 3D line
graphs”. MIS Quart. 28(2), pp. 55–281.
Kural, Y., Robertson, S. and Jones. S. (2001). “Deciphering cluster representations.”
Information Processing and Management, 37(4), pp. 593-601.
Lawrence, S. and Giles, C. L. (1999). “Accessibility of information on the Web”. Nature,
400, pp. 107–109.
Lee, B., Smith, G., Robertson, G. G., Czerwinski, M. and Tan D. S. (2009) “FacetLens:
exposing trends and relationships to support sense making within faceted datasets”. In
CHI ’09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems,
pp. 1293–1302. ACM.
Lieberman, M. D., Taheri, S., Guo, W., Mirrashed, F., Yahav, I., Aris, A. and Shneiderman, B.
(2011). “Visual Exploration across Biomedical Databases”. IEEE/ACM Trans. Comput. Biol.
Bioinformatics. 8 (2) pp. 536-550.
Maarek, Y. S., Berry, D. M., Kaiser, G. E. (1991). “An Information Retrieval Approach for
Automatically Constructing Software Libraries”. IEE Transactions on Software
Engineering. 17(8), pp. 800-813.
Marchionini, G. (1992). “Interfaces for end-user information seeking.” Journal of the
American Society for Information Science, 43(2), pp. 156–163.
Mak, J. Y. S., Leong, H. V., and Chan, A. T. S. (2002). “Dynamic structuring of web
information for access visualization”. In Proceedings of the ACM Symposium on Applied
Computing. Pp. 778–784.
Mcdonald,D.M. and Chen, H. (2006). “Summary in context: Searching versus browsing”.
ACM Trans. Inf. Syst. 24(1), pp. 111–141.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 134 sdey1@sheffield.ac.uk - 100151932
Mackinlay, J. D., Robertson, G. G. and Card, S. K.(1991). “The perspective wall: detail and
context smoothly integrated”. In CHI ’91: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, 27, pp. 173–176.
Maurer, D. (2006). Four Modes of Seeking Information and How to Design for Them.
[online]. US: Boxes and Arrows.
www.boxesandarrows.com/view/four_modes_of_seeking_information_and_how_to_desi
gn_for_them[Accessed 15 August 2011].
Microsoft Silverlight Team (2011). Microsoft PivotViewer [Online]. US: Microsoft
Corporation. www.microsoft.com/silverlight/pivotviewer/ [Accessed Aug 16 2011].
Microsoft News Center (2010). New Microsoft Live Labs Pivot Technology Brings
Information to Life [online] URL
http://www.microsoft.com/presspass/features/2010/feb10/02-11pivot.mspx
[Accessed 10 July, 2011]Microsoft Corporation: Long Beach, US
Morse, D. R., Ytow, N., Roberts, D. McL., Sato, A. (2003). “Comparison of Multiple
Taxonomic Hierarchies Using TaxoNote”. In Poster Compendium of IEEE Information
Visualization.
Mukherjea, S. and Hara, Y. (1999). “Visualizing World-Wide Web search engine results”.
In Proceedings of the International Conference on Information Visualisation. Pp. 400–405.
Munzer, T. (1998). “Exploring large graphs in 3D hyperbolic space”. IEEE Comput. Graph.
Appl. 18(4), pp. 18–23.
Munzner, T., Guimbretière, F., Tasiran, S., Zhang, L. and Zhou, Y. (2003). “TreeJuxtaposer:
Scalable tree comparison using Focus+Context with guaranteed visibility”. ACM
Transactions on Graphics, SIGGRAPH 03. Pp. 453-462.
Nielsen, J. (2004). When search engines become answer engines [Online]. Alertbox URL:
www.useit.com/alertbox/20040816.html [accessed Aug 01, 2011]
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 135 sdey1@sheffield.ac.uk - 100151932
Nguyen, Q.V. and Huang, M. L. (2005). “EncCon: An Approach to Constructing Interactive
Visualization of Large Hierarchical Data”. Journal of Information Systems. Sage
Publications.
Oppenhiem, A.N. (1992). Questionnaire design, interviewing and attitude measurement.
London: Pinter.
Pattinson, T. R., Vernik, R. J. (2001). Rapid Assembly and Deployment of Domain
Visualisation Solutions. Salisbury, DSTO
Pattison, T., Vernik,R. and Phillips,M. (2001). “Information visualisation using
composable layouts and visual sets”. In: Proceedings of the 2001 Asia-Pacific symposium
on Information visualisation - Volume 9 (APVis '01), Vol. 9. pp. 1-10. Australian Computer
Society, Inc., Darlinghurst, Australia
Pattison, T. & Phillips,M. (2001). ”View coordination architecture for information
visualisation. In Proceedings of the 2001 Asia-Pacific symposium on Information
visualisation - Volume 9 (APVis '01)” Vol. 9. Australian Computer Society, Inc., pp. 165-
169 Darlinghurst, Australia.
Pflughoeft, K., Zahedi, M., and Soofi, E. (2005a). “Data visualization using figural
animation”. In Proceedings of the 11th Americas Conference on Information Systems.
Pflughoeft, K., Zahedi, M., and Soofi, E. (2005b). “Figural animation visualization: The
system and its application”. In Proceedings of the AIS SIGDSS Pre-ICIS Workshop.
Pinker, S. (1990). “A theory of graph comprehension”. In Ed. R. Freedle. Artificial
Intelligence and the Future of Testing. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 73–
126.
Pirolli,P., Card, S. K., and Van Der Wege, M. M. (2003). “The effects of information scent on
visual search in the hyperbolic tree browser”. ACM Trans. Comput. Hum. Interact. 10(1),
pp. 20–52.
Pirolli,P., Card, S. K., and Van Der Wege, M. M. (2001). “Visual information foraging in a
focus + context visualization”. In Proceedings of CHI 2001, pp. 506-513. ACM.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 136 sdey1@sheffield.ac.uk - 100151932
Pirolli, P, Schank, P., Hearst, M. and Diehl, C. (1996). “Scatter/Gather browsing
communicates the topic structure of a very large text collection.” In Proceedings of
CHI'96, pp. 213-220. ACM.
Plaisant, C. (2004). “The Challenge of Information Visualization Evaluation”.
In Proceedings of the working conference on Advanced visual interfaces (AVI '04). ACM,
New York, NY, USA, pp. 109-116.
Plaisant, C., Grosjean, J., and Bederson, B. B. (2002). “SpaceTree: Supporting exploration
in large node-link tree: design evolution and empirical evaluation”. IEEE Symposium on
Information Visualization. Pp. 57-64.
Reid, N. H. (1999). “The photographic collection in St. Andrews University Library”.
Scottish Archives. 5, pp. 83-90
Rasmussen, E. M. (2003). “Indexing and retrieval for the Web”. Annual Review of
Information Science and Technology, 37(1), pp. 91–124.
Rennison, E., (1994) “Galaxy of news: an approach to visualizing and understanding
expansive news landscapes”. In Proc. UIST ’94.
Rhyne, TM. (2004). “Top Scientific Visualisation Research Problems”. IEE Computer
Graphics and Applications. Pp. 13-17
Rijsbergen., C. J. (1979). Information Retrieval. Butterworths.
Roberts, J., Boukhelifa, N., and Rodgers, P. (2002). “Multiform glyph based web search
result visualization”. In Proceedings of the 6th International Conference on Information
Visualisation.
Robertson, G. G., Mackinlay, J. D. and Card. S. K. (1991). “Cone trees: Animated 3d
visualizations of hierarchical information.” In CHI ’91: Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, pp.189–194. ACM.
Rodden, K (2002). “Evaluating Similarity-based visualisations as interfaces for image
browsing”. University of Cambridge, UK
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 137 sdey1@sheffield.ac.uk - 100151932
Rohrer,R.,Ebert,D., and Sibert, J. (1998). The shape of Shakespeare: Visualizing text using
implicit surfaces. In Proceedings of the IEEE Symposium on Information Visualization. Pp.
121–129.
Rosenfeld & Morville, (2002). Information Architecture for the World Wide Web:
Designing Large-Scale Web Sites. 2nd Edition. USA: O’Reilly.
Roussinov,D. and Ramsey, M. (1998). “Information forage through adaptive
visualization”. In Proceedings of the 3rd ACM Conference on Digital Libraries. Pp. 303–
304.
Salton, G. (1989). “Automatic Text Processing”. Addison-Wesley, Reading, MA
Sanderson, M., Croft, B. (1999). “Deriving concept hierarchies from text”, In Proc. SIGIR
’99, pp. 206-213.
Sangole,A. And Knopf, G. K. (2002). “Representing high-dimensional data sets as closed
surfaces”. Inf. Visualiz. 1, pp.- 111–119.
Sebrechts, M. M., Cugini, J. V., Laskowski, S. J., Vasilakis, J., and Miller, M. S. (1999).
“Visualization of search results: A comparative evaluation of text, 2D, and 3D interfaces”.
In Proceedings of the 22nd Annual ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR’99). Pp. 3–10.
Scott, D. W. (1992). “Multivariate Density Estimation: Theory, Practice, and
Visualization”. John Wiley and Sons, New York.
Sheth, N., Börner, K., Baumgartner, J., Mane, K., Wernert, E. (2003). “Treemap, Radial
Tree, and 3D Tree Visualizations”. In Poster Compendium of IEEE Information
Visualization.
Shneiderman, B. (1996). “The eyes have it: A task by data type taxonomy for information
visualizations”. In Proceedings of the IEEE Symposium on Visual Languages. Pp. 336–343
Silverstein,C., Henzinger,, M., Marais, H. and Moricz. M. (1999). “Analysis of a very large
web search engine query log”. SIGIR Forum, 33(1), pp-6–12.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 138 sdey1@sheffield.ac.uk - 100151932
Smith, G., Czerwinski, M., Meyers, B., Robbins, D., Robertson, G., Tan, D. S. (2006).
“FacetMap: A Scalable Search and Browse Visualization”. In: IEEE Transactions on
Visualization and Computer Graphics. 12(5), pp. 797 – 804. IEEE Computer Society.
Spence, R. (2001). Information Visualization. ACM Press.
Spencer, D. (2006). Four Modes of Seeking Information and How to Design for Them
[online]. US: Boxes and Arrows.
www.boxesandarrows.com/view/four_modes_of_seeking_information_and_how_to_desi
gn_for_them [Accessed 16 Aug 2011].
Spenke, M., Beilken, C. (2000). “InfoZoom - Analysing Formula One racing results with an
interactive data mining and visualization tool”. Ebecken, N. Data mining II, pp. 455–464.
Spink, A., Wolfram, D., Jansen, B. J. and Saracevic. T.(2001). “Searching the web: the public
and their queries”. Journal of the American Society for Information Science and
Technology, 52(3).
Spoerri, A. (1993) “InfoCrystal: A visual tool for information retrieval”. In VIS ’93:
Conference on Visualization, pp. 150–157. IEEE Computer Society.
Stefaner,M., Urban,T., and Seefelder, M.(2008). “Elastic lists for facet browsing and
resource analysis in the enterprise”. In DEXA 2008: conference on Database and Expert
Systems Application, pp 397–401. IEEE Computer Society
Stolte, C., Tang, D., and Hanrahan, P., (2002). “Polaris: a system for query, analysis, and
visualization of multidimensional relational databases”, Transactions on Visualization
and Computer Graphics, 8(1), pp. 52-65.
Stone, M. C. (2003) A Field Guide to Digital Color. A. K. Peters.
Sutcliffe, A. G., Ennis, M., Hu, J. (2000). “Evaluating the effectiveness of visual user
interfaces for information retrieval”. International Journal of Human Computer Studies.
53(5), pp. 741-763.
Trafton, J., Tsui, T., Miyamoto, R.; Ballas, J., Raymond, P. (2000). “Turning pictures into
numbers: extracting and generating information from complex visualizations.”
International Journal of Human Computer Studies, 53(5), pp. 827-850.
Tufte, E. (1990). Envisioning Information. Graphics Press.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 139 sdey1@sheffield.ac.uk - 100151932
Tufte, E. (1997). Visual Explanations. Graphics Press.
Tufte, E. (2001). The Visual Display of Quantitative Information. Graphics Press.
Turetken, O. and Sharda, R. (2007). “Visualization of web spaces: state of the art and
future directions”. SIGMIS Database. 38 (3). pp. 51-81.
Turetken, O. and Sharda, R. (2005). “Clustering-Based visual interfaces for presentation
of web search results: An empirical investigation”. Inf. Syst. Front. 7(3), pp. 273–297.
Turetken, O. and Sharda, R. (2004). “Development of a fisheye-based information search
processing aid (FISPA) for managing information overload in the web environment”.
Decis. Supp. Syst. 37(3), pp. 415–434.
Voorhees., E. M. (2004) “Overview of TREC 2004”. In Proceedings of the Thirteenth Text
Retrieval Conference (TREC 2004).
Ward, M. O. (2002). “A taxonomy of glyph placement strategies for multidimensional
data visualization”. Inf. Visualiz. 1(3-4), 194–210.
Ware, C. (2004) Information Visualization: Perception for Design. Morgan Kaufmann.
White, R. W. & Drucker, S. M. (2007). “Investigating Behavioral Variability in Web
Search,” International World Wide Web Conference 2007, pp. 21-30.
White, R. W., Kules, B., Drucker, S.M., and Schraefel, M.C. (2006). “Supporting exploratory
search: Introduction”. Communications of the ACM, 49(4) pp. 36–39
Whittaker, S., Bergman, O., Clough, P. (2009). “Easy on that trigger dad: a study of long
term family photo retrieval”. 14(1). pp. 31-43 London: Springer Verlag.
Willett, P. (1988). “Recent trends in hierarchic document clustering: A critical review.”
Information Processing and Management, 24(5), pp. 577-597.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 140 sdey1@sheffield.ac.uk - 100151932
Wilson, M.L., and Schraefel, M.C. (2008). “Improving exploratory search interfaces:
Adding value or information overload?“ In HCIR 08: Workshop on Human-Computer
Interaction and Information Retrieval, pp. 81–84
Wise, J. A, Thomas, J. J., Pennock, K., Lantrip, D. Pottier, M. Schur, A. and Crow, V. (1995).
“Visualizing the non-visual: Spatial analysis and interaction with information from text
documents”. In InfoVis 1995: Symposium on Information Visualization, pp. 51–58. IEEE
Computer Society.
Wiza ,W., Walczak, K., and Cellary, W. (2003). “AVE - Method for 3D visualization of
search results”. In Proceedings of the International Conference on Web Engineering. pp.
04–207.
Wunsche, B. (2004). “A survey, classification and analysis of perceptual concepts and
their application for the effective visualisation of complex information”. In: Neville
Churcher and Clare Churcher (Eds.), Proceedings of the 2004 Australasian symposium on
Information Visualisation - Volume 35 (APVis '04) Vol. 35. Australian Computer Society,
Inc., pp. 17-24 Darlinghurst, Australia.
Xiong, R. and Donath, J. (1999). “PeopleGarden: Creating data portraits for users”. In
Proceedings of the 12th Annual ACM Symposium on User Interface Software and
Technology. Pp. 37–44.
Xiang, Y., Chau, M., Atabakhsh, H., and Chen, H. (2005). “Visualizing criminal
relationships: Comparison of a hyperbolic tree and a hierarchical list”. Decis. Supp. Syst.
41, pp. 69–83.
Yang, K. (2005). “Information retrieval on the Web”. Annual Review of Information
Science and Technology, 39(1), pp. 33 – 80
Yao, Y. (2002). “Information retrieval support systems”. In Proceedings of the 2002 IEEE
World Congress on Computational Intelligence.
Yee, P., Swearingen, K., Li, K., and Hearst, M., (2003) “Faceted metadata for image search
and browsing,” In Proc. CHI ’03, pp. 401-408.
INF 6000 – Dissertation – Microsoft PivotViewer 2011
Year – 2011 Page 141 sdey1@sheffield.ac.uk - 100151932
Zeng, H., He, Q., Chen, Z., Ma, W., Ma, J. (2004), “Learning to cluster web search results”
In: Proc. SIGIR ‘04’.
Zhang, J., and Marchionini, G., (2005). “Evaluation and evolution of a browse and search
interface: Relation Browser++”. The National Conference on Digital Government Research.
Zhang, J., Mostafa, J., and Tripath, H. (2002). “Information retrieval by semantic analysis
and visualization of concept space for the D-lib magazine.” D-lib online magazine.
Zhu, B. and Chen, H. (2005). “Information visualization”. Annual Review of Information
Science and Technology, 39(1), pp. 139 – 177
Zudilova-Seinstra, E., Martens,J-B. and Adriaansen, T. (2010).”Interactive data
exploration and knowledge discovery”. In: Santucci, G. (Ed.). Proceedings of the
International Conference on Advanced Visual Interfaces (AVI '10).pp. 421-422 ACM, New
York, NY, USA.
---
Recommended