Upload
thomas-davies
View
430
Download
5
Embed Size (px)
DESCRIPTION
The tablet device market is growing substantially and is expected to sell 140 million devices in 2014 and therefore there is a need for interaction guidelines that spans both applications and platforms. This report presents a set of guidelines for gestural tablet interaction based on experienced and inexperienced participants who were asked to create gestures for common computer actions. 18 participants created 324 gestures for 18 actions; gestures that were analysed and categorized before being rated on their appropriateness for the intended action. This data provided the foundation of the set of guidelines for gestural tablet interaction that I created and were then validated by another set of users.
Citation preview
C O V E N T R Y
U N I V E R S I T Y
Faculty of Engineering and Computing
Department of Computing and the Digital Environment
Creative Computing
303CDE - Individual Project
Guidelines for Gestural Tablet Interaction
Author: Thomas Davies
SID: 2179955
Supervisor: John Halloran
Submitted in partial fulfillment of the requirements for the Degree of Bachelor of Creative
Computing
Academic Year: 2010/11
Declaration of IndividualityThis project is all my own work and has not been copied in part or in whole from any other source
except where duly acknowledged. As such, all use of previously published work (from books,
journals, magazines, internet, etc) has been acknowledged within the main report to an entry in the
References list.
I agree that an electronic copy of this report may be stored and used for the purposes of plagiarism
prevention and detection.
I understand that cheating and plagiarism constitute a breach of University Regulations and will be
dealt with accordingly.
Signed: Date:
Office Stamp
Copyright
The copyright of this project and report belongs to Coventry University.
Declaration of Individuality i
AbstractThe tablet devices market is growing substantially and it expected to sell 140 million devices in 2014
and therefore there is a need for interaction guidelines that span both applications and platforms.
This report presents a set of guidelines for gestural tablet interaction based on experienced and
inexperienced participants who were asked to create gestures for common computer actions. 18
participants created 324 gestures for 18 actions; gestures that were analysed and categorized before
being rated on their appropriateness for the intended action. This data provided the foundation of the
set of guidelines for gestural tablet interaction that I created and were then validated by another set
of users.
Abstract iii
Table of ContentsAdditional Materials on the CD viii
Acknowledgements ix
Introduction & Background 1
Background to the Project 1
Recent History 1
Market Potential 2
Method of Study 2
Findings 3
The Need for a Gesture Set 4
Conclusion 5
Section Summary 6
Methodology 7
Introduction 7
Researching Ethics 7
Guessability Testing 7
Inter-Rater Agreement 11
Gesture-Meaning Association Testing 12
Section Summary 14
Formative Investigation 15
Introduction 15
Guessability Study 15
Inter-Rater Agreement 15
Section Summary 17
Findings 19
Introduction 19
Guessability Study 19
Rating Study 22
Table of Contents v
Section Summary 25
The Gesture Set 27
Introduction 27
Guidelines for Gestural Tablet Interaction 27
Section Summary 32
Summative Investigation 33
Introduction 33
Gesture-Meaning Association Testing 33
Alterations to Original Methodology 34
Results 34
Findings 35
Section Summary 37
Discussion & Conclusion 39
Conclusion 39
Value & Benefit 39
Implications 40
Challenges 40
Future Work 40
Project Management 43
Supervisor Meetings & Progress 43
References 45
Bibliography 47
Appendix A 49
Guessability Study Documentation & Forms 49
Appendix B 85
Gesture Screenshots 85
Appendix C 123
Rating Study Documentation & Forms 123
Appendix D 127
vi Guidelines for Gestural Tablet Interaction
Gesture Data Charts 127
Appendix E 137
Gesture-Meaning Association Results 137
Appendix F 139
Supervisor Correspondence 139
Table of Contents vii
Additional Materials on the CD
viii
Acknowledgements‘Technology alone is not enough’
Steve Jobs
This thesis would not have been possible without the support and guidance of Dr John Halloran who
has assisted this project from the very beginning, through his enthusiasm and immense knowledge.
I would also like to express my gratitude to all the Creative Computing staff who have played an
instrumental role in my development over the past three years.
Finally, I wish to thank my parents who have always supported me and ensured that I had every
chance of fulfilling my dreams.
Acknowledgements ix
1. Introduction & Background1.1. Background to the Project
The launch of the iPad in early 2010 ushered in a new class of tablet devices that are radically
different than the earlier tablet PCs of the 1990s and early 2000s. Styli and desktop-based operating
systems have been replaced by multi-touch displays and mobile operating systems. After 40 years of
using the same human-computer interaction paradigms created by the infamous Xerox PARC team of
the 1960s and 1970s, change has come in the form of light, compact, and efficient modern tablet
devices (Saffer 2008: 3). This shift in paradigm is different however to that of the desktop era where
only two major platforms led the way - Windows and Mac OS - as multiple companies have ventured
into the tablet sector. Users will be introduced into this new method of interaction via a fragmented
sector in both hardware and software and therefore it is important to define a set of multi-platform
guidelines and standards as the methods we create today will define the landscape for decades to
come (Saffer 2008: 4). 2011 will be a defining year for tablet devices as many manufacturers will be
launching their products, aiming to eradicate Apple's huge market share.
1.2. Recent History
RIM
HP WebOS
Android
AppleiPad releasedApr 2010
January 2010
April 2010
July 2010
October2010
January 2011
April 2011
July 2011
October 2011
HP TouchPad announcedFeb 2011
Playbook announcedSep 2010
Honeycomb OS releasedFeb 2011
Galaxy Tab releasedNov 2011
iPad 2 releasedMar 2011
HP TouchPad releasedSummer 2011
Motorola Xoom releasedFeb 2011
Galaxy Tab 10 releasedSummer 2011
Playbook releasedApr 2011
Figure 1.1 Timeline of recent history in the tablet sector
In September 2010, Research in Motion (RIM) introduced their rival to Apple's iPad; the PlayBook.
The OS will not be based on RIM's smartphone operating system but on QNX Software Systems' OS
which 'is designed specifically for the tablet size computer' (BBC News 2010). QNX was acquired by
RIM in April 2010, 5 months before the PlayBook announcement (RIM 2010). The PlayBook -
Chapter 1: Introduction & Background 1
designed for the enterprise sector - is expected to be 'commercially available early in 2011' (BBC
2010).
While Samsung released their Galaxy Tab in November 2010, it was using an Android build that was
designed for smartphones and not tablets and therefore cannot be considered as a legitimate tablet
device. Therefore, throughout 2010, Apple dominated the sector with a 'market share in tablet
computers at 95 percent in the third quarter' (Bloomberg 2010).
The next major tablet announcement took place on February 2nd, 2011 when Google announced the
Honeycomb update to their Android OS that was targeted specifically for the tablet form factor
(Google 2011). This Google update then resulted in several manufacturers; Samsung, Motorola and
Asus announcing that their Honeycomb-based tablets will launch in the first half of 2011. One week
after the Google announcement, HP revealed plans for their recently acquired Palm webOS mobile
operating system. Along with updates to their smartphones, HP introduced the TouchPad; a webOS-
based tablet device that will launch in Summer 2011 (Palm 2011).
On March 2nd, 2011 Apple announced the second iteration of the iPad and that it would be available
in the North American market on March 11th and on March 25th for the international market. Apple
announced their second tablet device significantly later than all competitors, yet were able to launch
the product significantly before all other manufacturers - except Motorola with their Xoom tablet.
Therefore, 2011 will see four mobile operating systems vying for market share; Apple's iOS, webOS,
Android and RIM's Tablet OS. While three of those OSs are not licensed to run on third-party
hardware, Android is open-source and is currently used by several hardware manufacturers;
fragmentation will occur on both the hardware and software aspects of tablet devices.
1.3. Market Potential
The iPad became commercially available on April 3rd, 2010 and sold its millionth device 28 days after
launch. Comparatively, it took 73 days for the original iPhone to reach the 1 million mark. (Yahoo
News 2010). By the end of 2010 - 9 months since launch - Apple had sold over 14 million devices
(Apple 2011). Meanwhile, research firms 'forecast 57 million tablets will be sold' in 2011 and expects
the sector to reach 171 million devices in 2014 (Fox Business 2011) amounting to one in five adults
owning a tablet device (Palm: 2011).
1.4. Method of Study
The fundamental basics of both the International Usability Partners and the University of
Washington’s study consists of this ‘guessability study methodology’ (Wobbrock et al 2009: 1). As the
University of Washington used live hardware, users were shown ‘the effect of gestures’ via an on-
screen animation and then asked to perform a gesture to invoke that effect. However, the
International Usability Partners study utilised a ‘lo-fi prototype simulating a touchscreen’ with before
and after screenshots, asking users how they would produce what was on the after screenshot
(Mauney et al 2010: 2) as to ‘focus on the user’s intuitive behaviour’ (Mauney et al 2010: 13). Dan
Saffer also agrees that it is important to ‘employ the knowledge and intuition of those who will use it’
2 Guidelines for Gestural Tablet Interaction
and similarly suggests using this method, or reversing it by demonstrating ‘a gesture and see what
feature users would expect that gesture to trigger.’ (Saffer 2008: 28).
For the specifics, both studies chose to remove all bias by having ‘no elements specific to Windows
or the Macintosh’ while ensuring that ‘the commands were application-agnostic’ (Wobbrock et al
2009: 2). This is important because the studies wanted to limit the amount of gestures that were
learnt from previous experiences rather than being purely intuitive.
The University of Washington were concerned about the lack of research and that ‘gestures have
been defined by system designers’ and not users, therefore they believe that it is right to ‘consider
the types of gestures people make without regard for recognition or technical concerns.’ (Wobbrock
et al 2009: 1). While the International Usability Partners study selected a range of participants, both
experienced and inexperienced the University of Washington study only ‘recruited non-technical
people without prior experience using touch screens’ as they felt this was the only subset of people
that ‘would behave with and reason about interactive tabletops differently than designers and system
builders.’ (Wobbrock et al 2009: 2). Therefore, it was interesting to note that the International Usability
Partners research found it difficult - even with inexperienced users - to understand when a participant
performed a gesture that was naturally intuitive due to ‘local and media exposure of
touchscreens.’ (Mauney et al 2010: 12).
1.5. Findings
The International Usability Partners study found that on tablet-sized devices there was ‘frequent
usage of the ‘tap’ gesture’ suggesting ‘participants imagined a button or context menu to trigger the
required action with a tap.’ (Mauney et al 2010: 10). Their study produced different results to the
University of Washington’s research into surface computers where ‘users rarely care about the
number of fingers they employ’ (Wobbrock et al 2009: 1). While tablet devices and surface computers
differ in size, tablet devices are large enough to take several inputs at once similar to surface
computers.
An issue with the International Usability Partners report is that is doesn’t specifically state how the
proposed gesture set was developed. Questions remain as to whether it was solely based on the
analysis of their user studies, or a combination of that and a designer’s perspective. Yet, the
University of Washington clearly states that the final gesture set ‘was developed in light of the
agreement participants exhibited in choosing gestures for each command.’ So, if ‘more participants
used the same gesture for a given command, the more likely that gesture would be assigned to that
command’ and the study actually found that the ‘user-defined gesture set emerged as a surprisingly
consistent collection founded on actual user behavior’ (Wobbrock et al 2009: 2).
While the International Usability Partners study was purely quantitative; producing only data from the
gesture testing, the University of Washington used ‘Likert scales concerning gesture goodness and
ease’ (Wobbrock et al 2009: 3) and utilised the think aloud method to gauge how users were feeling
when creating the gestures.
Chapter 1: Introduction & Background 3
1.6. The Need for a Gesture Set
While the likes of Apple and Google have interface guidelines that they advise third-party developers
to follow, the majority of the guidelines focus on the aesthetics of the application and not the
interaction techniques. In 2000, Jef Raskin stated that modern computers were 'a feast for the eyes,
with glorious colorful backgrounds and three-dimensional tromp l'oeil', a fact still true for modern
tablet devices. Yet while aesthetics play a role in the overall experience, 'the quality of any interface is
ultimately determined by the quality of the interaction between one human and one system' (Raskin
2000: xix).
1.6.1. Habituation
After prolonged and persistent practice, people will habitually complete a task without consciously
thinking about it. Therefore the gestures that are developed for tablet devices will become habit for
users, whether they are good or bad.
The issue with developing habit for bad practices is that 'you cannot break a habit by an act of
violation' but simply by 'a time-consuming training process' (Raskin 2000: 20). It is therefore pivotal
that the correct gestures are learnt by users otherwise they will struggle to habituate to another set of
gestures.
1.6.2. Inconsistent Gesture Usage
Many current iPad apps are creating different methods to complete a similar action, yet this does not
benefit the end user as 'having multiple options can shift your locus of attention from the task to the
choice of method' which drastically hinders the user's experience of the product (Raskin 2000: 20).
Altering a method for an action can act as an unique selling point and is actively promoted in
advertising as shown by Apple's original iPhone adverts that highlighted key interaction styles and
techniques in Figure 1.2.
Figure 1.2 Apple’s ‘How to’ iPhone advert (Apple 2007)
While this differentiates manufacturers from each other, the practice is detrimental to users as
'inconsistent gesture usage can lead to frustration, confusion and usability problems' (Ginsburg 2010:
22).
4 Guidelines for Gestural Tablet Interaction
1.7. Conclusion
When discussing the need for gestural standards, Dan Saffer imagines a scenario where physical
movements would be required to turn on the light in a darkened room. With no indication about what
gesture is required, you attempt several random movements in an attempt to turn on the light. While
this might be an 'imaginable future scenario' for free-form gestural interfaces, these sorts of
scenarios are already appearing for touch-based systems.
Dan Saffer suggests setting up 'some sort of standards board, similar to what the Web has with the
W3C', a board that has helped define the future of the web across a multitude of devices. But, if we
do not work with the individual companies - such as Apple, Google and HP - they will simply
continue to create gestures and try and force them onto end users as standards.
The concern on individual companies leading the charge for gestural standards is the possibility of
them patenting their creations. Saffer states that the actual gesture probably can't be patented as it
fails two US patent requirements; it's natural and too obvious. However, Apple is currently attempting
to patent 'a gesture tied to a specific action' - specifically the Pinch to Shrink gesture - and if this is
accepted, then they will be able to license the patent or force infringing parties to stop using the
gesture. This would then force other manufacturers to create their own gestures so that they won't
infringe on others' patents and will then look to protect these gestures in retaliation. While Saffer
attempts to reassure us by stating that 'patent holders do not have to enforce their patents', they still
have the option of suing infringing parties in their arsenal.
Ctrl + C is the standard command to copy an item in Windows and it is shared by all third-party
Windows developers; the command is even similar on Mac OS X with Command + C (Where
'Command' replaces 'Ctrl' for all commands on Mac OS X). With further patent claims by tablet
manufacturers and operating system developers, users will have to learn 'new commands for actions
such cut-and-paste for every program' and platform (Saffer 2008: 174-176).
If we don't act soon, users will be unable to go from platform to platform and utilise their habitual
knowledge as they will have to 'remember a crazy number of gestures for common actions' (Saffer
2008: 174). We are on the brink of a new computing paradigm which could radically change how
people interact with technology and therefore we should create a set of standard gestures that can
work across different platforms and applications.
Chapter 1: Introduction & Background 5
1.8. Section Summary
1.8.1. The Market
• The iPad was launched in 2010 and this spurred many other manufacturers to produce their own
competing tablet devices.
• Major companies launching tablet devices in 2011 include Apple, HP, Motorola, RIM, Samsung and
Acer.
• The tablet market is very different to the early desktop computing landscape. Multiple companies
have entered the tablet market, and the dominance of a couple of platforms seems unlikely.
• Industry experts expect the fledgling market to grow substantially over the next 5 years, with the
sector selling over 150 million tablet devices in 2014.
1.8.2. Prior Research
• Tablet devices received little exposure in academia.
• Previous academic research is limited to touch-based phones and surface computers.
• Research involved real users and potential users throughout the entire study.
• They noted the lack of user involvement in current gestural creation.
1.8.3. The Need for Guidelines
• Gestures used by the end users will become habit, whether they are good or bad practice.
• Breaking a habit is challenging and can only be achieved by a time-consuming process.
• Inconsistent gestures have already appeared across both applications and platforms.
• Manufacturers and software developers are keen to protect their intellectual property and have
begun patenting their creations and suing infringing parties. Action/gesture relationships are a key
issue in these lawsuits.
6 Guidelines for Gestural Tablet Interaction
2. Methodology2.1. Introduction
This section discusses the methodology for the entire study, detailing the processes behind the three
stages of creating a set of guidelines for gestural tablet interaction. As shown in Figure 2.1, the
individual stages lead on and inform the proceeding stage.
Inter-Rater Agreement
Guessability StudyGesture-Meaning Association Test
Figure 2.1 The 3 stages of the research
2.2. Researching Ethics
Collecting information from the public can be challenging as they are often concerned about how
their data will be used. Therefore, it is pivotal that I handle their data with care; informing them that it
will be used anonymously. Moreover, making participants more comfortable benefits me as the
researcher because ‘it will yield better results, since the subjects will likely open up more if they know
that they (and their data) are being treated well.’ (Saffer 2007: 74).
2.3. Guessability Testing
2.3.1. Process
Recruit Particpants
Planning
Create Discussion Guide
Guessability Test
Think Aloud Protocol
Observation & Questioning
Data AnalysisPresentation of
Data
Figure 2.2 Guessability testing process
2.3.2. Guessability Method
Prior to undertaking the test, the participants completed a consent form that confirmed they were
willing to participate in the study and that the results from their tests will be used in the report
anonymously.
Following this, they completed a short questionnaire that documented their demographics in terms of
age and gender along with information about their experience using tablet devices. While there was
no specific age range requirements, it will be interesting to see how different subsections of people
could possibly create different gestures. The results from questioning the users’ experience would
ensure I have a 50/50 split; half experienced and half inexperienced.
Chapter 2: Methodology 7
When conducting the actual tests, subjects were shown a short animation - as shown in Figure 2.3 -
that displayed the outcome of the gesture for that action. To ensure that the participants were not
guided to perform a specific gesture, the animations were single slides and lacked any animation. For
example, for the Zoom-in action, the video didn’t slowly transition the smaller squares into larger
squares as this could result in the users performing a gesture based on the video and not based on
intuition.
Slide 1 - Before Screenshot Slide 2 - After Screenshot
Figure 2.3 Zoom-in action slides for video
This approach is similar to the University of Washington’s study where after the animation, the
participants were shown ‘the objects as they were before’ the action was performed and then they
were expected to perform the gesture they felt would invoke that desired action (Wobbrock et al
2009: 4). The before screenshot was placed as the background of a drawing app - called Doodle
Buddy - on the iPad that allowed for multi-finger input and following the drawing of the gesture, I
would take a screenshot of the canvas. After each gesture, the participants were asked to rate their
created gesture based on the statement: ‘The gesture I picked is a good match for its intended
purpose’ and rated it on a 5-point likert scale from Strongly Disagree to Strongly Agree. The rating -
along with the recorded video of the participants performing the gestures and screenshots - will be
used in analysing the results from this test.
2.3.3. Participants
For most usability studies, Jakob Nielsen recommends 5 users as they find 90% of issues. This study
however, isn’t a typical usability study and therefore there isn’t enough data available to set a
precedent.
The International Usability Partners study used 40 participants per country, with 20 experienced and
20 inexperienced users. They gave no reason as to why they selected that number and I also feel that
40 participants are too many for my project due to time and resource constraints.
I attempted to find similarities between this method and other types of usability studies. Card sorting
seems similar as it looks for similarities in user’s decisions with the aim of producing a taxonomy and
gaining understanding of their mental models. For card sorting, Nielsen suggests 15 users. As I also
have two types of users; experienced and inexperienced I would need around 30.
8 Guidelines for Gestural Tablet Interaction
When testing multiple categories, you don’t need to test 15 users per group as there will be overlaps
in the results. As Nielsen discussed the 5 person rule for most usability studies, he stated that you
only need 3 users per category when testing 2 groups. That amounts to 60% of the original 5 person
rule. Therefore, the recommendation would be to use 18 participants; 9 experienced and 9
inexperienced.
Usually, the selection ‘of participants whose background and abilities are representative of your
product’s intended end user is a crucial element of the testing process,’ yet one main aim of this
study is to discover how outcomes from experienced and inexperienced users differ (Rubin 1994).
Moreover, the modern tablet market is still maturing and there is no knowledge of what specific types
of people currently use tablet devices and who will in the future.
2.3.4. Equipment & Materials
Device Usage
iPad For showing the short animations and for completing all the guessability testing
Panasonic Lumix TZ7 For recording the participants performing the gestures
Table 2.1 Set of actions to be used in the Guessability Testing
For the study, the participants will be using an Apple iPad. Unlike the International Usability Partners
study which utilised webcam rig that recorded the piece of paper that was placed on an immovable
object, I feel it is important to allow the user to fully embrace a tablet by being able to grasp and hold
the physical object in a way that is natural to them. By making participants feel more relaxed by
allowing them to use the tablet, the results extracted from the study will be more reliable. I will use an
iPad application that allows me to place an image - for this study; a screenshot - in the background
and then ask the participant to enact a gesture that they feel will accomplish the given task.
2.3.5. Environment
To further allow participants to relax I will use a guerrilla usability testing style. One method is known
as coffee shop testing where ‘researchers go to their local coffee haunt in search of study
participants.’ (Ginsburg 2010: 181) This method is also beneficial to me as I lack the big budget
required to have a usability lab and entering such labs can often be a daunting task for any
participant. The consumer electronics companies are advertising these products with relaxing
imagery; lounges and living rooms with the users resting their feet on a table. I aim to mimic this calm
environment to ensure my results are reliable.
2.3.6. Think Aloud Protocol
The guessability method yields a lot of quantitative data, however to fully understand users’ mental
models, qualitative data is necessary. During the studies, participants were asked to think aloud as
this will allow us to ‘obtain rich qualitative data that illuminates user’s mental models’ (Wobbrock et
al. 2009: 1). Utilising this method allowed us to have participants ‘express their confusion, frustration
and even their delight’ as well as discuss their thought process and reasoning behind the
gestures .they were performing (Rubin 1994: 218). Moreover, it also allowed me to ask more informed
questions after each gesture as I had a basic understand as to why they felt that gesture should
invoke a specific action.
Chapter 2: Methodology 9
2.3.7. Questioning
Working in collaboration with the think aloud protocol, questioning participants allowed us to ‘fully
understand the “why” behind performance and preference’ (Rubin 1994: 219). I also ensured that I
was asking open-ended questions to ‘elicit more and better information’ than closed-ended
questions as they can suggest to the participants that you are only looking for a limited answer that
rarely yields enough qualitative information. Moreover, the open-ended questions ‘help prevent
“leading the witness”’ which is especially critical when creating a set of guidelines for interaction
(Goodwin 2009: 60).
2.3.8. Set of Actions
Based on the International Usability Partners and University of Washington studies, along with Dan
Saffer’s Designing Gestural Interfaces, I selected 18 actions that are required for every day
computational tasks. The 18 actions are shown in Table 2.2.
No. Action No. Action
1 Select single object 10 Back
2 Select multiple objects 11 Forward
3 Zoom-in 12 Rotate
4 Zoom-out 13 Open menu
5 Help 14 Close menu
6 Print 15 Move object
7 Undo 16 Cut
8 Redo 17 Copy
9 Go home 18 Paste
Table 2.2 Set of actions to be used in the Guessability Testing
10 Guidelines for Gestural Tablet Interaction
2.4. Inter-Rater Agreement
2.4.1. Process
Recruit Particpants
Planning
Create Discussion Guide
Inter-Rater Agreement Test
Data AnalysisPresentation of
Data
Figure 2.4 Inter-Rater agreement process
2.4.2. Testing Method
The inter-rater agreement method is similar to the categorisation tests that are utilised in icon testing,
in which participants are asked to ‘arrange icons into meaningful groups’ to measure the ‘users’
recognition of the images and also their understanding of the concepts represented by the
images’ (Horton 1994: 301).
Following the guessability study, I had 18 gestures per action and I had to decipher a single gesture
from all of those options that would be the sole representation for a specific action. For certain
actions, many gestures were very similar, yet had individual traits that also made them slightly
different. Therefore, I had to find a consensus, where multiple people could categorise the gestures,
allowing me to recognise the similarities in participants’ groupings. These similarities would allow me
to understand different participants’ mental models which will provide a solid foundation when
crafting the set of guidelines for gestural tablet interaction.
However, participants would have to categorise all of the gestures in one attempt, and with 18
gestures per action, with a total of 18 actions that would require the participants to view 324
gestures. Moreover, there was concern about participants forgetting why certain gestures were
placed into certain categories as they progressed through all 324 gestures. Users would also
probably review already-categorised gestures which would further prolong the test.
Therefore, I decided to initially run a pilot test for this categorisation method so this method’s
effectiveness could be evaluated. The pilot would involve 3 participants and individually, they were
asked to group 90 gestures (based on 5 actions, each with 18 gestures), placing gestures that they
felt were similar together.
2.4.3. Data Analysis
I will be utilising the Fleiss’s Kappa method as it allows me to measure category agreement between
several raters. The result from the Fleiss’s Kappa measure will be a number between 0 and 1, where a
higher number would suggest more of an agreement between the raters, therefore suggesting a
better agreed category.
Based on this category score, I will be able to select the action/gesture pairings that are agreed upon
by the participants of the inter-rater agreement study; resulting in a more reliable set of guidelines for
gestural tablet interaction.
Chapter 2: Methodology 11
2.5. Gesture-Meaning Association Testing
2.5.1. Introduction
The legitimacy of gestures hasn't been tested until recently. While there are many precedents in other
aspects of usability study design, there is no common way to test a set of gestures to see whether
the relation with the desired action is intuitive.
Icons and gestures are, by definition, similar as they are signs or symbols that refer to a specific item
or action. However, icons benefit from being a visual symbol that is placed on the visual user
interface while gestures are invisible until the gesture is performed.
2.5.2. Process
Recruit Particpants
PlanningGesture-Meaning Association Test
Data AnalysisPresentation of
Data
Figure 2.5 Gesture-Meaning Association process
2.5.3. Participants
When Jakob Nielsen conducted icon intuitiveness tests, a small set of users were invited to
participate; around 3 - 5 users. While this is a small number, it was important to select the correct
users as ‘no testing is effective without adequate test subjects.’ (Horton 1994: 293) and ensure that
the test subjects reflect the tablets’ user base. Moreover, William Horton suggests that the
participants shouldn't have ‘inappropriate knowledge of the product’ (1994: 293), however we can
not ignore the fact that Apple - along with other manufacturers - are teaching people how to use the
products through their multi-million dollar advertising campaigns and have to accept that this could
affect some users’ decisions.
2.5.4. Testing Method
There are two main methods to test the effectiveness of icons and they can be adapted to suit
gesture evaluations too. William Horton suggests usability testing and image-meaning association
analysis.
When conducting usability testing, icons are placed 'in a prototype of the user interface and we test
whether users can do meaningful work with the system.' This allows us to analyse how well the icons
integrate 'with other aspects of user-interface design' (Horton 1994: 301). Even though this can also
be done with lo-fidelity prototypes, it won't validate the set of gestures I create. To inform the design
of the gestures I utilised a tablet device that participants used to create gestures based on a desired
action. If this method was selected, we would merely be recreating the exact same test from the
previous stage. Therefore, to validate the gesture set we need to reverse this concept and ask a
different set of participants what they expect the gesture to do. For icon design, this is known as
image-meaning association which measures 'the degree to which the graphic chosen for the icon
represented the intended concept' (Nielsen 1995). For the purpose of gesture testing this method will
be known as gesture-meaning association.
As shown in the following chart, there are several options for gesture-meaning association. This
project used the gesture-meaning matching method as it provided us with more data; allowing us to
12 Guidelines for Gestural Tablet Interaction
'observe which are matched first and which later' and 'which are matched quickly and which are
matched only by a process of elimination' (Horton 1994: 300). This knowledge was fundamental in
ranking the gestures in terms of user acceptance.
Method Description
Open-ended meaning-for-
gesture
Shows the participant the gesture and asks “What does this gesture mean?”
This tests the association a user already has for a gesture.
Multiple-choice meaning-
for-gesture
Presents the user with a gesture and a list of possible meanings for the user to pick
from. This requires four or five plausible choices for the testing to be valid.
Gesture-for-meaning Asks the user to identify which of several gestures best represents a concept. Can use
it to test for completely different gestures or test different gestures for a single action.
Gesture-meaning matching Provides participants with gestures and verbal labels for the same concepts but in a
different order. Allows us to observe how quickly and accurately users match the image
with the appropriate label.
Table 2.3 Gesture-meaning association options (Horton 1994: 297-300)
Therefore, I had 32 pieces of paper; 18 slips that listed the actions and another 18 that showcased
the gesture. The action slips were placed on one side while the gesture slips were placed on the
other; random in order. I then, individually, asked the three participants to match the action/gesture
pairs.
2.5.5. Data Analysis
For the final aspect of this report, I analysed two performance measures as documented by William
Horton (1994: 302):
Method Description
Accuracy of selection Following the completion of the exercises, I checked the action/gesture relationships
that were successfully paired and asked the participants about the pairings that were
incorrect to discover why they felt their was a relationship between that specific action
and gesture
Speed with which
selections are made
I also noted the length of time it took for each pair to be chosen; paying special
attention to the pairings that seemed to be created through a process of elimination
Table 2.4 Data analysis on the results from the Gesture-meaning association tests
Chapter 2: Methodology 13
2.6. Section Summary
2.6.1. General
• There are three stages to this research; guessability study, inter-rater agreement testing and then
gesture-meaning association testing.
• These individual stages led on from each other and informed the proceeding stage.
2.6.2. Guessability Study
• 18 participants individually created gestures for 18 commonly used computational actions.
• 50% of the participants had prior tablet experience, while the other half hadn’t used a tablet device
before.
• The participants used an iPad to draw the gestures on, and a screenshot was taken for each
gesture created while the entire test was video recorded.
2.6.3. Inter-Rater Agreement
• It is a categorisation test that is used in icon usability testing.
• Gestures for specific actions from the guessability study were similar, yet had unique traits that
might differentiate them from one another to a variety of users.
• Participants are asked to arrange the gestures created from the guessability study into categories.
• Will allow me to understand users’ mental models.
• The test would require each participant to watch all 324 gestures and categorise them. I was
concerned over the possibility of participant fatigue.
• Decided to run a pilot with 3 participants who would group 90 gestures based on 5 actions that
each had 18 individual gestures.
• If successful, the whole test will be initiated.
2.6.4. Gesture-Meaning Association Testing
• Currently, there is no precedent in the process of validating gestures.
• This test is also based on icon usability testing.
• 32 slips of paper will be placed in front of a participant; 18 actions and 18 gestures.
• The 3 participants will be asked to match a gesture to a specific action.
• Will record accuracy of selection and speed with which selections are made.
14 Guidelines for Gestural Tablet Interaction
3. Formative Investigation3.1. IntroductionIn the previous section, I introduced the methodologies that would be utilised throughout this
research; guessability study, inter-rater agreement and gesture-meaning association testing. Section
3 is an account on the first two stages; the guessability study and the inter-rater agreement.
It will detail the issues discovered with the inter-rater agreement pilot that took place with three
participants and how I revised that stage of the research to ensure that I would still produce similar
ranking results to inform the final part of the project.
3.2. Guessability Study
Figure 4.1 Subject B participating in the Guessability Test
As shown in Appendix A, I documented all 18 tests through consent and feedback forms, along with
a monitoring sheet where I detailed my observations while the participants were creating their
gestures. Appendix B showcases all 324 gestures that were created by the 18 participants.
To ensure participants were relaxed and in an environment where they felt they would use a tablet
device, I allowed the individual participant to decide where their test would take place. By using 18
separate locations, I had to continually setup the camera to record the iPad screen, ensuring the
environment was well lit and that the device was clearly in shot. Yet, if I had utilised a single location
for the tests, I would have saved a considerable amount of time as the test layout would have already
been setup. While I believe the inconsistent locations provided more accurate results, the additional
time did lead to increased participant stress due to the build-up process of setting up the test.
3.3. Inter-Rater Agreement
During the methodology section, I discussed the need to have a consensus on the type of gestures
being performed - a tap, swipe, curve etc - and therefore decided to use the inter-rater agreement
method. Yet there were concerns about the complexity and time-consuming nature of the method
and as a result, I chose to run a pilot.
For the pilot, I selected 3 participants to group 18 number of gestures from a set of 5 actions, with a
total of 90 gestures overall. Individually, each participant was asked to watch a gesture video and
Chapter 3: Formative Investigation 15
group the gestures on how they saw fit. This began successfully, however participants started to
struggle as they reached the thirties and were constantly reviewing previous gesture videos to decide
what group the current gesture belongs to. This was concerning because the study had amassed 324
gestures among 18 participants and 18 actions, yet the participants were struggling with only 10% of
all gestures in the pilot. Therefore, it was decided that I should terminate the pilot and consider other
options.
3.3.1. 2nd Stage Revision
Instead, for each action I grouped the 18 gestures based on the type of gesture. Following this, I
asked 3 people to rate the effectiveness of the gesture for that action; the same question that I asked
the original guessability study participants. By averaging these ratings, I was able to have a
consensus on whether specific gestures were a good match for the intended action. To stem the
possibility of fatigue, participants were asked to rate only one action at any one sitting, amounting to
only 18 gestures at a time. This process therefore, required a significant amount of time and lasted
two weeks.
Figure 4.2 A rater participating in the replacement rating study
By combining my grouping data along with their ratings I was able to extrapolate which type of
gestures were accepted among participants. This data then allowed me to create the set of gestures.
16 Guidelines for Gestural Tablet Interaction
3.4. Section Summary
3.4.1. Guessability Study
• This stage took place without any major issue.
• Allowing the participant to decide the test location resulted in a time-consuming setup process for
every test. This led to slightly increased participant stress.
• 324 gestures were created by 18 participants.
• Appendix A documents the consent, feedback and expert forms for all 18 participants.
• Appendix B shows all 324 gestures that were created.
3.4.2. Inter-Rater Agreement
• During the early stages, this stage was progressing well.
• Raters struggled with the test when they reached around the 30th gesture of the test.
• They began re-watching previous gesture videos to ensure they would put the current gesture in
the correct group.
• There are over 300 gestures, yet the raters were struggling with only 10% of the overall gesture
amount.
• Decided to terminate the pilot and select another method.
3.4.3. 2nd Stage Revision
• I grouped the gestures based on the type of gesture - for example: tap, swipe, pinch.
• 3 raters would rate each individual gesture on it’s relevancy to the specific action.
• Asked the same question as the guessability study participants - ‘The gesture they picked is a
good match for its intended purpose’
• Coupling the gesture groupings with the rating scores allows me to identify good matches for
action/gesture relationships.
• This stage will inform the design of the set of guidelines for gestural tablet interaction.
Chapter 3: Formative Investigation 17
4. Findings4.1. Introduction
Chapter 3 discussed how the guessability study and inter-rater agreement pilot progressed. I also
examined the issues concerning the inter-rater agreement pilot and why there was a need for another
approach as the raters were struggling to complete the entire test.
In this chapter, I will discuss the results from the first two stages; guessability study and the inter-
rater agreement replacement test. The second stage was replaced with a basic rating routine that
didn’t require the raters to go through the entire study in one sitting.
4.2. Guessability Study
Appendix B documents the type of gestures created by the participants for each action.
Figure 4.1 A selection of gestures for the Help action (top row) and the Close menu action
4.2.1. Desktop Metaphors
Like other studies, I discovered that many participants were relying on the desktop-based metaphors
that have been prominent for over 30 years. Even though I reiterated the multi-touch capability of the
device - even demoing it to them prior to the user study - many users still only utilised a tap; akin to a
single click or double click on a desktop mouse. This finding was also noted in the University of
Washington study where ‘participants still often thought of the desktop paradigm’ (2009: 8), while my
study found that 23% of all gestures were either single tap or double tap. This statistic is quite
concerning as it could result in a reliance of desktop metaphors that were designed over 30 years
ago and fail to take advantage of the unique multi-touch display that blankets the front of modern
tablets.
4.2.2. The use of Letters and Symbols
With several actions, many participants utilised letters or symbols as their choice of gestures; with in
many cases being the most popular choice. This was a very interesting discovering as it was a huge
shift from the desktop-based gestures that were more common for other actions in this study.
Chapter 4: Findings 19
It is also very interesting to note how participants were drawing letters or words to describe the
action they desired during the studies; letter or words associated with desktop-based shortcuts.
Tap and Hold
ʻPʼ Letter
Multi-Tap
Tap
Swipe
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
For example, when asked what gesture they would choose to print a document, 6 out of 18
participants decided to draw the letter ‘P’ on the screen. Even though the letter ‘P’ is a rather
common shortcut on the desktop - and would probably be used by inexperienced desktop users;
one participant even utilised the more obscure shortcut letter ‘Y’ for redo.
However, once participants thought of using a letter for an action, it seemed to be the default gesture
for subsequent actions that were deemed challenging by the participants. This is in sharp contrast to
the University of Washington study where they found the single tap gesture was utilised for difficult
actions instead (Mauney et al 2010: 10).
Moreover, I feel apprehensive in using symbols or letters as gestures due to their ‘potential for
showing differences across cultures’ and therefore gestures will need to be adapted across cultures
and countries (Mauney et al 2010: 6). Yet, I can’t dismiss the overwhelming support for symbolic
gestures that were created by the participants of this study while developing my own gesture set.
4.2.3. Interaction with other objects
When I asked Subject M what gesture they felt was suitable for printing, they performed a swipe
gesture in the direction of the printer in their bedroom. Even though only one participant executed
this gesture, I found it to be rather significant. When I asked him to explain their choice, Subject M
stated that it felt natural to gesture in the direction of his printer, yet would not feel comfortable
performing a similar move with a mouse on a desktop computer. Unfortunately, he was unable to
further explain his choice of gesture, however, while observing the process, it seemed like he had
forgotten about the constraints of desktop computing - one of the points I hoped all participants
followed - and imagined how he would wish to interact with future technology.
Figure 4.2 Print action gestures
20 Guidelines for Gestural Tablet Interaction
Aza Raskin believes that the job of an interface designer is to remove as much of the interface as
possible so ‘you don’t even realize you are using’ an interface, because ‘the best interface is no
interface' (Raskin 2008). It is therefore, these types of gestures that make ‘invisible interfaces’ a viable
option in tablet devices - more so than on the desktop - due to the range of possible input methods.
This finding is both a positive and negative result from my insistence of participants deciding on
where the tests should take place - based on where they felt comfortable - as this result wouldn’t
have been discovered in a closed lab environment, yet the lack of printers in other participant’s
locations mean’t that this outcome was not possible for them.
4.2.4. Smartphone experience and advertising influence
Many of the participants for this study were under 30 and were therefore aware of touchscreen
devices; many having used a touch-based smartphone within the past couple of years. Therefore,
even without having experience on a tablet device, many participants were aware of gestures utilised
on the iPhone and Android-based smartphones; gestures that are still relevant for tablets.
Moreover, many participants stated that the gestures they performed for certain actions, were based
on the methods seen in television advertising for smartphones; especially the iPhone. While 50% of
participants were required to have no tablet-based experience, the ubiquitous nature of touch-based
smartphones influenced the results of this study.
4.2.5. Struggling to perform gestures
Occasionally, some participants were unable to perform a gesture for a specific action. Before the
guessability study, I presumed that some users would be unable to execute gestures for cut, copy
and paste as I deemed them to be the most difficult actions in the set. However, it was interesting to
discover that Participant C - an inexperienced, over 50s participant - was unable to perform both the
zoom-in and zoom-out actions. The overwhelming majority of participants used the pinch gesture; a
gesture that wasn’t performed by British users in a 2010 study (Mauney et al 2010) and therefore
suggests that people can quickly adapt to different gestures for commonly used actions at this early
stage in tablet computing. Yet, the inability of Participant C to create gestures for these actions and
her concern for being unable to complete these tasks shows how the intimidating and inaccessible
nature of desktop systems still remain for tablet devices, in regards to basic computer users who lack
the experience of smartphone devices and aren’t influenced by the manufacturers’ adverts.
Chapter 4: Findings 21
4.2.6. Need for Context Sensitive Menus
For many actions, participants requested context sensitive menus that would be displayed following
a basic gesture. For the cut, copy and paste actions, 70% of participants asked for context sensitive
menus following gestures such as: tap and hold, multi-tap, and tap as shown in Figure 4.3.
6%13%
12%
6%
35%
29%
Tap and Hold Multi-Tap Tap Letters & Symbols Other No Attempt
The IUP study also recognised similar results as their ‘participants consistently requested [context
sensitive menus] for some actions, and they are particularly useful for novice users’ and therefore
they created two gestures - for certain gestures - for both novice and expert users.
While this study was trying to discover what gestures best suit certain actions, it is also important to
uncover what actions are difficult to convey through a gesture and what ones require additional on-
screen support to be completed.
4.3. Rating Study
4.3.1. Relationship between total number of similar gestures and user agreement
Prior to the Rating Study, I expected that the gesture types that were performed the most would be
rated with a higher score. While this was the case for the majority of actions, some saw the opposite.
Tapping the top-right and swiping were the most popular gestures for the Close Menu action;
accounting for over 60% of all created gestures. However, in the Rating Study, the Swipe gesture
received a score of 2.9 approval and Top-Right Tapping had an approval rating of 3.7, yet it was the X
gesture that received the highest approval score. Drawing an X on the menu received a score of 4.5,
even though only one participant in the Guessability Study choose that gesture.
Figure 4.3 Percentage of gesture types for cut, copy & paste actions
22 Guidelines for Gestural Tablet Interaction
0
1
2
3
4
5
6
Swipe ʻXʼ Letter Multi-Tap Pinch Tap Top-Right No AttemptAgreement Average Total No. of Gestures
When creating the set of gestures, issues like these drastically affect what gestures I should select for
a specific action. I can not create a gesture by combining two gestures as it will lead to a sign that
was neither created by users nor approved by them.
4.3.2. Advertising Influence
It was interesting to note the very high approval rating of 5 gestures across 5 actions; gestures that
are widely used in advertising smartphones and tablet devices. These gestures received an approval
rating of over 4.5 for their relevant action.
Action Dominant Gesture
Select single object Tap
Zoom-in Reverse-pinch
Zoom-out Pinch
Back Swipe
Forward Swipe
For example, the pinch and reverse-pinch gestures scored a 4.6 approval rating for the zoom-out and
zoom-in actions respectively; compared to the 1.5 approval rating average of the other two gestures
created by the Guessability Study participants.
This finding was also noted by the IUP study and they questioned ‘whether the gestures performed
were naturally instinctive’ due to the obvious influence from advertising. Moreover, it was these
gesture/action combinations that were popular among both experienced and inexperienced
participants; suggesting that people are able to learn and recall gestures that they have seen before;
even without prior experience of performing the gesture (Mauney et al 2010: 12).
Figure 4.4 Comparison between Agreement Averages and Total No. Of Gestures for Close Menu
Table 4.1 List of popular gestures among advertised gestures
Chapter 4: Findings 23
4.3.3. Approval of Symbols & Letters as Gestures
In section 4.2.2, I discussed my concern over some participants using letters as the default gesture
when they reached a difficult action; therefore not challenging themselves to create other options.
However, the Rating Study participants approved of the use of letters as gestures for certain actions
as shown in Figure 4.5 which details the approval ratings of the Paste action gestures.
0
2
4
6
8
Tap and Hold ʻVʼ Letter ʻPʼ Letter Curve Multi-Tap Tap No AttemptAgreement Average Score Total No. of Gestures
As witnessed with this example, the ‘P’ letter received a low approval rating of 1.3. However, if we
combine this information with data from the Print action, we discover that the Rating Study
participants rate letters associated with the relevant desktop shortcuts higher than ones that are not.
0
2
4
6
8
Tap and Hold ʻPʼ Letter Multi-Tap Tap Swipe
Agreement Average Score Total No. of Gestures
Figure 4.5 High approval rating of the ‘V’ Letter
Figure 4.6 High approval rating of the ‘P’ Letter for the Print action
24 Guidelines for Gestural Tablet Interaction
4.4. Section Summary
4.4.1. Guessability Study
• There seemed to be a reliance on desktop-based metaphors as 23% of all gestures were either
single-tap or double-tap.
• The use of letters and symbols for gestures was very popular. Associated letters were mainly based
on the desktop shortcut rather than the action word. Eg. V for the Paste action instead of P.
• For the Print action, one participant swiped the text towards the physical location of his printer in
their room.
• Many participants used gestures they have learnt from smartphone advertising campaigns as these
gestures are still relevant on tablet devices.
• The request for context-sensitive menus was overwhelming. For the cut, copy and paste actions,
70% of participants asked for a context-sensitive menu. This suggests that not all actions can be
easily defined by a single gesture.
4.4.2. Rating Study
• Certain gestures that were more popular for specific actions didn’t necessarily score a good
approval rating.
• Sometimes gestures that were created only once would score a high approval rating. Eg. ‘X’
gesture for the Close menu action was implemented by one participant in the guessability study,
yet scored an approval rating of 4.5 out of 5.
• Widely advertised gestures scored an approval rating of 4.5 out of 5 or above by the raters. Eg.
Pinch for the Zoom-out action and Swipe for the Back and Forward actions.
Chapter 4: Findings 25
5. The Gesture Set5.1. Introduction
Chapter 4 explored the results from both the guessability study and the rating study, examining the
most popular gestures created by participants and how popularity doesn’t necessarily improve a
gesture’s approval rating among other users. I discussed the role of advertising and how it could
have affected how instinctively the gestures were performed. Moreover, I discovered that using letters
as gestures for certain actions were very popular, yet are not adopted by current tablet devices.
Figure 5.1 details the guidelines for gestural tablet interaction that I created based on the findings
from the two prior procedures; the guessability study and then the following rating study.
5.2. Guidelines for Gestural Tablet Interaction
Action Proposed Gesture
Select single object Tap item
Select multiple objects Tap and hold one item and sequentially tap other items
Chapter 5: The Gesture Set 27
Action Proposed Gesture
Zoom-in Reverse pinch with two fingers on the canvas
Zoom-out Pinch with two fingers on the canvas
Help Draw symbol ‘?’ anywhere on the canvas
??Print Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘P’ anywhere on the canvas
Context-Sensitive Menu P
Context-Sensitive Menu P
28 Guidelines for Gestural Tablet Interaction
Action Proposed Gesture
Undo Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘Z’ anywhere on the canvas
Context-Sensitive Menu Z
Context-Sensitive Menu Z
Redo Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘Y’ anywhere on the canvas
Context-Sensitive Menu Y
Context-Sensitive Menu Y
Go home Swipe down with one finger on the canvas to reach the bezel of the device
Back Swipe with one finger from left to right on the canvas
Page 2 Page 1Page 2 Page 1
Chapter 5: The Gesture Set 29
Action Proposed Gesture
Forward Swipe with one finger from right to left on the canvas
Page 1 Page 2Page 1 Page 2
Rotate Tap and hold the corner of the item and curve in the desired direction
Hello
HelloHelloHello
HelloHello
Open menu Double-tap anywhere on the canvas to open the menu
MenuMenu
Close menu Draw letter ‘X’ inside of the menu box
MenuMenu
\ XMenuMenu
\ X
30 Guidelines for Gestural Tablet Interaction
Action Proposed Gesture
Move object Tap item and then drag to the desired location
Cut Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘X’ on the item to be cut
Context-Sensitive Menu X
Context-Sensitive Menu X
Copy Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘C’ on the item to be copied
Context-Sensitive Menu C
Context-Sensitive Menu C
Paste Double-tap anywhere on the canvas to open a context sensitive menu
Shortcut: Draw letter ‘V’ on the location where the item is to be pasted
Context-Sensitive Menu V
Context-Sensitive Menu V
5.2.1. Need for Shortcut options
As 6 actions required a context-sensitive menu to be displayed following a gesture, there was a need
to create shortcut gestures for these actions that would provide advanced users immediate access to
that specific feature. The shortcut gestures were based on the letters used in desktop shortcuts as
these are already well known and do not depend on cultural knowledge.
Chapter 5: The Gesture Set 31
5.3. Section Summary
• Created a set of guidelines for gestural tablet interaction.
• 18 gestures crafted for 18 actions.
• Based on results and findings from the guessability study and rating study.
• 6 actions have shortcut gestures associated with them as well because their primary gesture
launches a context-sensitive menu.
32 Guidelines for Gestural Tablet Interaction
6. Summative Investigation6.1. Introduction
The previous section introduced the set of guidelines for gestural tablet interaction that I had created
based on the guessability study and rating test. I created a set of 18 gestures that were individually
paired with a specific action; some functions requiring two gestures as the primary gesture only
displayed a context-sensitive menu.
As I have created the set of gestures, Chapter 6 details the findings and results from the validation
stage which utilised the gesture-meaning association testing method. The results from this section
will determine if the action/gesture relationships - and therefore the overall gesture set - are clear and
incontestable.
6.2. Gesture-Meaning Association Testing
As shown in Appendix D, I conducted a Gesture-Meaning Association test with three participants
where they paired 18 common computer actions to 18 gestures.
Figure 6.1 Raters during the Gesture-Meaning Association testing
Chapter 6: Summative Investigation 33
6.3. Alterations to Original Methodology
It was originally planned that all created gestures would be assessed through the Gesture-Meaning
Association method, however not all the gestures created can be adequately assessed using this
procedure. This concerns the gestures that require a context-sensitive menu to be displayed
following a double-tap gesture and therefore these will not be assessed during this stage. However,
all actions that utilised a context-sensitive menu, also had a shortcut gesture associated with it - for
example: the letter ‘P’ for printing - and these will be assessed.
6.4. Results
The results from the gesture-meaning association test are shown in Table 6.1 and it details the
number of participants who correctly matched the action/gesture relationship and the order in which
a pairing was created.
Participant 1Participant 1 Participant 2Participant 2 Participant 3Participant 3
Action/Gesture Pairings Matched? Order Matched? Order Matched? Order
Select single object ✔ 4 ✔ 1 ✔ 4
Select multiple objects ✔ 5 ✔ 9 ✔ 8
Zoom-in ✔ 1 ✔ 3 ✔ 2
Zoom-out ✔ 2 ✔ 4 ✔ 3
Help ✔ 8 ✔ 10 ✔ 5
Print ✔ 9 ✔ 11 ✔ 10
Undo ✔ 11 ✔ 8 ✔ 9
Redo ✘ - ✘ - ✔ 12
Go home ✔ 17 ✔ 16 ✘ -
Back ✔ 3 ✔ 5 ✔ 6
Forward ✔ 6 ✔ 6 ✔ 7
Rotate ✔ 10 ✔ 7 ✔ 13
Open menu ✔ 15 ✔ 13 ✘ -
Close menu ✔ 13 ✔ 12 ✔ 11
Move object ✔ 7 ✔ 2 ✔ 1
Cut ✔ 16 ✘ - ✔ 16
Copy ✔ 12 ✔ 14 ✔ 14
Paste ✔ 14 ✔ 15 ✔ 15
Table 6.1 Gesture-Meaning Association test results
34 Guidelines for Gestural Tablet Interaction
6.5. Findings
6.5.1. Gestures & Actions that were Easily Paired
Out of the 18 actions, 9 of them - shown in Table 6.2 - were very easily paired with the correct
gesture by all participants.
Easily Paired ActionsEasily Paired ActionsEasily Paired Actions
Select single object Select multiple objects Help
Zoom-in Zoom-out Rotate
Back Forward Move object
Table 6.2 Easily paired actions
While the majority of those gestures are already well known and accepted gestures - thanks to
manufacturers’ advertising - I was pleased to see the Help action be easily paired. This ? gesture is
currently not implemented by any major tablet OS - iOS, Android and WebOS - yet the universal
symbol seems to be recognised and supported by all participants - across all three stages - in this
report.
6.5.2. Difficult Actions & Gestures to Pair
During this process, I discovered that there were three actions that all participants had difficulty in
pairing with a gesture; Redo, Go home and Open menu. While the Redo and Open menu gestures are
founded in desktop-based knowledge, the Go home gesture is rather distinct, even if it mimics the
minimizing gesture on Windows and Mac OS X.
Figure 6.2 Minimizing effect on Mac OS X
Therefore, the Go home action was one of the final actions to be paired with a gesture; often being
chosen through a process of elimination. However, while discussing the action following the test,
participants understood the reasoning behind the gesture, but two testers admitted that they are
concerned by the hidden nature of the gesture.
All of the shortcut gestures created as part of the set of gestures were based on the desktop
shortcuts for that action, yet the ‘Y’ Redo shortcut is only known to Windows users as Mac OS X
utilises another letter for the Redo function on their computers. This issue was only raised because
two of the Gesture-Meaning Association participants were Mac OS X users and therefore were
unable to associate the ‘Y’ gesture to the Redo action.
6.5.3. Unfounded Concerns over Cut, Copy and Paste
Due to the trouble creating the gestures - both during the Guessability Study and the gesture
production stages - for Cut, Copy and Paste, I predicted that these three actions would be difficult to
pair up with the correct gestures. Yet, while these actions were some of the last to be paired -
Chapter 6: Summative Investigation 35
probably due to the simplicity of many other action/gesture pairings - two participants successfully
paired these actions and gestures together. The participants were pleased that the gestures utilised
their prior computing knowledge; echoing the support from a participant in the University of
Washington study who stated ‘anything I can do that mimics Windows—makes my life
easier’ (Wobbrock et al 2009: 8).
6.5.4. Confusion between Gestures & Actions
As participants always had access to every action and gesture, they were able to change their
decision at any time during the process. This allowed me to note any incident where gestures were
paired with incorrect actions, only to be later paired with the correct action; informing me of possible
confusion when the gestures are used as a group rather than as individuals.
There was slight confusion with the ‘P’ gesture, as two participants originally paired the gesture with
the Paste action instead of the correct Print action. This was before the participants realised that the
shortcut gestures were based on their desktop counterparts and therefore the pairings were rectified
before the end of the test.
36 Guidelines for Gestural Tablet Interaction
6.6. Section Summary
6.6.1. Alterations to Original Methodology
• When creating the set of guidelines for gestural tablet interaction, 6 gestures had a primary gesture
that displayed a context-sensitive menu. To assist advanced users, a shortcut gesture was
included for those actions.
• The primary gesture for the context-sensitive menu is difficult to validate because there is no direct
connection between the action and the gesture.
• The 6 gestures that displayed a context-sensitive menu were not validated, but the secondary
shortcut gestures were validated using the gesture-meaning association test.
6.6.2. Findings
• 9 action/gesture relationships were correctly paired very easily by all participants, 8 of which are
actively used in manufacturers’ advertising campaigns.
• The Redo, Go home and Open menu actions were very challenging for participants to correctly
pair. The Redo shortcut is based on a Windows-only shortcut that isn’t available on Mac OS X.
Other actions seemed to be paired by a process of elimination at the later stages of the gesture-
meaning association test.
• Cut, copy and paste actions were correctly paired by 2 participants. They were pleased they could
utilise their desktop knowledge when using a tablet device.
• Initial confusion between certain gestures and actions; two users initially placed the P gesture with
the Print action, before correcting their mistake and placing it with the Paste action.
Chapter 6: Summative Investigation 37
7. Discussion & Conclusion7.1. Conclusion
I have presented a study that examined and understood what gestures users associate with specific
actions, leading to the creation of a set of guidelines for gestural tablet interaction. During the
guessability study, I gathered 324 user-generated gestures across 18 actions, by 18 participants.
These were then examined and grouped by gesture type, and given an approval score based on the
appropriateness of the action/gesture relationship. Following this, I created a set of 18 action/gesture
pairings which were then validated by another set of participants by utilising the gesture-meaning
association method.
It is hugely important that we can establish a set of common gestures shared across platforms and
applications to eradicate the possibility of a fragmented interaction style on this new computing
paradigm. If tablets follow a similar path as desktop computers, then they have the possibility of
being in widespread usage for over 30 years. While desktop computers have improved their graphics
and industrial design, the fundamentals of desktop computing hasn’t changed since their formation
at Xerox PARC in the 1960s and 1970s. Therefore, it is pivotal that before we see widespread
adoption of tablet devices, we must create a set of common gestures that can last a generation.
7.2. Value & Benefit
With several manufacturers gearing up to launch and release their tablet devices this year, I have
created a set of platform-agnostic gestures, based on the input of 18 original participants who
produced over 320 individual gestures across 18 actions. This report has shown that there is no need
for platform-specific gestures and that a common set can be shared across a multitude of platforms.
This therefore will allow users to move from platform to platform and utilise their habitual knowledge
of a universal gesture set.
Throughout this report, I emphasised the importance of user involvement as it allowed me to
understand what gestures felt natural for them to perform, as well as understand participants’ level of
action/gesture pairing acceptance. Users being the driving force behind the creation of the set of
guidelines for gestural interaction should be an obvious choice, yet this approach is absent from
current tablet implementations.
Therefore, I involved current and potential tablet device owners, asking them to create, assess and
validate the set of 18 gestures. Yet, this report discovered more than just 18 gestures, but provided
information on action/gesture relationships that are not approved by users. This means we have the
knowledge of what gestures to avoid in future so users don’t learn incorrect gestures and develop
habit for bad practices.
It is important to note that the set of gestures has been validated by another set of participants,
which shows that these action/gesture relationships have been accepted and understood by a
Chapter 7: Discussion & Conclusion 39
number of participants. Interestingly, the University of Washington study ignored this step and just
released their set of gestures for surface computers based on a similar guessability study. While a
gesture might be natural to draw, it doesn’t necessarily mean that the gesture can be interpreted as a
depiction of a specific action.
7.3. Implications
Moreover, we witnessed gestures that are not included in current tablet devices; especially the use of
letter or symbols. This discovery allows for the possibility of a huge number of potential gestures
based on letters and symbols. It allows users to utilise their pre-existing computing and cultural
knowledge on tablet devices; reducing the learning process of new users. However, it must be noted
that while the majority of letter-based gestures created in this report are based on universally-
accepted desktop shortcuts, future letter-based gestures need to be tested for cultural differences.
During the Guessability Study, we discovered that many participants were asking for context-
sensitive menus to be displayed for certain actions; suggesting that not every function can be
depicted as a gesture. Therefore, there needs to be a common gesture that will provide further
actions and it could be argued that this gesture would be the most vital one as it will provide access
to a number of actions.
7.4. Challenges
7.4.1. Involvement of Users
My research was very user-driven, requiring them to participant throughout all three stages of the
project. This was very time-consuming and challenging for both the participants and myself, yet
without them, my report would have been worthless. Finding 18 participants for the guessability
study that were both available and fit the set criteria was difficult as many potential participants were
also completing academic projects.
7.4.2. Failed Pilot Study
The failed inter-rater agreement pilot study also added additional pressure as it required me to
research another method, design the test and then run it, thereby increasing the workload. This issue
wasn’t anticipated at the start of the project; only arising at the beginning of February. While I
overcame the issue - by finding raters eligible to participate within a week’s notice - it still brought
unneeded pressure on both the report and my other academic responsibilities.
7.5. Future Work
7.5.1. Individuality of Gestures
Even though we have created and validated individual gestures for this report, the gestures were not
tested as an overall set. Like the International Usability Partners study, some participants created
'similar, or even the same, gestures … for dissimilar actions' (Mauney et al 2010: 13) and this
therefore affected which gestures would be tied to a specific action - sometimes with the most
popular one not being selected.
40 Guidelines for Gestural Tablet Interaction
Even though the study was conducted in real-world environments - based on where the participants
felt comfortable and where they would consider using a tablet device - it remained platform and
application agnostic in an attempt to limit the affect of pre-learned skill. However, there is a need to
test the gesture set in real applications and with proper use cases to justify it's implementation in real
system. Testing the gesture set in real-world applications has the opportunity of yielding different
results to a study that was performed in isolation.
7.5.2. Different Types of Users
Many different subset of users are having their first computer experiences on tablet devices.
However, this study was based mainly on the limited age range of 20-25 - with a few exceptions -
and this ignores many other important age groups. Some old age pensioners have seen the potential
of tablet devices that portray a friendlier and more user-friendly image to that of standard desktop
computers (Wipf 2010), while the tactile nature of the devices are enticing toddlers to interact with
them (Stewart 2010). Moreover, further research should discuss accessibility issues and involve
disabled participants in creating and evaluating a gesture set. Even though basic applications have
been created for certain disabled users, there is a need for an accessible gesture set that spans
platforms rather than just individual applications (Apple 2011).
7.5.3. Further Analytical Work
As I was unable to fully utilise the inter-rater agreement method due to the inability of the raters to
complete the pilot task, the use of analytical data is very limited in this report. The replacement
method only asked the raters to rate the gestures based on the appropriateness of the action/gesture
relationship, and the three raters’ scores were then merged to create an average score. Future
research should discuss and examine the need for increased analytical data and how it could assist
in the creation of a more solid set of guidelines for gestural tablet interaction.
7.5.4. Reliance on Desktop Knowledge
Further research should also discuss the impact of users’ reliance on desktop-based metaphors. To
limit the affect of cultural differences, I based the shortcut gestures on the desktop-shortcut
equivalent and not on the action word; for example: V for the Paste action instead of P. However, we
need to examine whether this has any detrimental effects on tablet interaction and whether it limits
the overall scope of interaction styles and techniques on tablet devices.
Chapter 7: Discussion & Conclusion 41
8. Project Management8.1. Supervisor Meetings & Progress
The notes from the meetings with my supervisor can be found in Appendix F.
Prior to the Interim Report, I had scheduled 4 meetings with my supervisor to discuss the progress
that I had made on this final year project. We continued to have meetings throughout the academic
year; roughly every 2 weeks. These meetings were pivotal in ensuring that my project remained on
track as mini-deadlines were set up, so that my supervisor could see the project develop over time.
Throughout the year, I attended all scheduled meetings with my supervisor.
8.1.1. 26th January
Due to illness, I was unable to attend this meeting. However, I emailed my supervisor concerning the
progress that I had made; informing him of the slight additions to the guessability study and that I
was beginning the process of writing up the Methodology section.
8.1.2. 9th February
We discussed the final two stages of the research; inter-rater agreement and the validation test. I
raised concerns about the inter-rater agreement method as it would be rather time-consuming for the
participants because it would take several hours for an individual to complete. Therefore, we decided
to run a pilot test and John Halloran requested it to be completed by the next meeting.
The literature review - the results of which can be found in the Introduction & Background section -
was completed prior to this meeting. As stated in my Interim Report, it was a challenge to find
relevant literature as it is an emerging field. Therefore, I was limited to just two academic papers and
one published book that discussed relevant gestural systems. Yet, the first section of this project
benefits from the literature that discussed the forging of the desktop 30 years ago as many issues
facing that generation still concerns the fledging tablet market.
8.1.3. 23rd February
During this meeting, I brought back the results from the pilot study. As mentioned in Section 3, the
pilot study was unsuccessful and therefore I had to select another approval rating method. As I was
approaching the end of February, it was very concerning that I had to plan and complete another
approval rating test and therefore we decided to take a slightly more simpler approach where raters
would rate each gesture based on how well it represented the intended action.
Moreover, I also completed the first draft for the Introduction & Background section, along with the
Methodology section. My supervisor read through the draft during the meeting and informed me of
the importance of referring back to the purpose of this research; the purpose has to permeate
throughout the entire document.
Chapter 8: Project Management 43
8.1.4. 9th March
The results of the new approval test were explored during this meeting. I informed my supervisor of
the method utilised and how many participants were involved, as well as discussing how the results
would be expressed in the project documentation. My supervisor also asked for a draft project by the
next meeting in two weeks time.
8.1.5. 23rd March
My supervisor reminded me of the four components that make up this module; the Interim Report,
Presentation, Report, and Product Quality. As the Product Quality mark scheme wasn’t available
online, my supervisor told me to request the mark scheme from Richard Rider. The supervisor also
asked for a full draft of the report and that it be emailed to him for the 12th April.
I emailed Richard Rider, and he uploaded the mark scheme online for all students. Even though I
emailed my supervisor on the 11th April, he did not receive the report. Therefore, John Halloran
contacted me on the 14th April asking why he was yet to receive the document. Unfortunately, I do
not know why my supervisor didn’t receive the draft, yet we arranged to have a meeting on the 18th
April.
8.1.6. 18th April
For this meeting, I brought along the Results and Conclusion sections for my supervisor to review.
While reading these sections, my supervisor was concerned that it was difficult to see progression
from section to section and suggested that I had included a brief review and introduction at the start
of each section, along with a summary at a section end.
My supervisor asked me to implement these changes and send him a final draft over the next few
days for him to review and reply with feedback. On the 20th April, I emailed him with my final draft
and he stated that the report was progressing very well; only suggesting minor alterations. He
reiterated the importance of binding and submitting the report as early as possible to avoid any
issues during the final stages.
44 Guidelines for Gestural Tablet Interaction
ReferencesApple, 2011. iPad: Year One. [Online Video].
Available from: http://www.youtube.com/
watch?v=HpiVeC1Z3yI
[accessed 10 April 2011]
Apple, 2011. Apple Reports First Quarter
Results. [Webpage].
Available from: http://www.apple.com/pr/
library/2011/01/18results.html
[accessed 16 February 2011]
Apple, 2007. How To Advert. [Video].
Cupertino, California : Apple.
BBC News, 2010. Blackberry launches
Playbook rival to iPad. [Webpage].
Available from: http://www.bbc.co.uk/news/
technology-11416190
[accessed 16 February 2011]
Bloomberg, 2010. Apple Increases Tablet Share
to 95 Percent as Android Slips. [Webpage].
Available from: http://www.bloomberg.com/
news/2010-11-02/apple-increases-tablet-
share-to-95-percent-as-android-slips.html
[accessed 16 February 2011]
Ginsburg, S., 2010. Designing the iPhone User
Experience: A User-Centered Approach to
Sketching and Prototyping iPhone Apps. 1 ed.
Upper Saddle River, NJ: Addison Wesley.
Goodwin, K., 2009. Designing for the Digital
Age: How to Create Human-Centered Products
and Services. Indianapolis, IN: John Wiley &
Sons.
Google, 2011. Android Event. [Online Video].
Available from: http://www.youtube.com/
watch?v=RfJuigJebRg
[accessed 16 February 2011]
Horton, W., 1994. The Icon Book: Visual
Symbols for Computer Systems and
Documentation (Wiley Technical
Communication Library). 1 ed. New York: John
Wiley & Sons.
Mauney, D., Wirtanen, A., Capra, M., 2010.
Cultural similarities and differences in user-
defined gestures for touchscreen user
interfaces. CHI EA '10 Proceedings of the 28th
of the international conference extended
abstracts on Human factors in computing
systems.
Nielsen, J., 1995. Icon Usability. [Webpage].
Available from: http://www.useit.com/papers/
sun/icons.html
[accessed 9 February 2011]
Palm, 2011. HP webOS Think Beyond Event.
[Online Video].
Available from: http://www.youtube.com/
watch?v=VnjwG7Z8AM8
[accessed 16 February 2011]
RIM, 2010. Research in Motion to Acquire QNX
Software Unit from Harman International.
[Webpage].
Available from: http://press.rim.com/
release.jsp?id=3766
[accessed 16 February 2011]
References 45
Raskin, A., 2008. Don't Make Me Click. [Online
Video].
Available from: http://www.youtube.com/
watch?v=EuELwq2ThJE
[accessed 10 April 2011]
Raskin, J., 2000. The Humane Interface: New
Directions for Designing Interactive Systems. 1
ed. Reading, Mass.: Addison Wesley.
Rubin, J., 1994. Handbook of Usability Testing:
How to Plan, Design and Conduct Effective
Tests (Wiley Technical Communications
Library). New York: John Wiley & Sons.
Saffer, D., 2008. Designing Gestural Interfaces:
Touchscreens and Interactive Devices. 1 ed.
California, United States: O'Reilly Media.
Saffer, D., 2009. Designing for Interaction:
Creating Innovative Applications and Devices
(Voices That Matter). 2 ed. New Riders.
Stewart, T., 2010. The iPad as a serious
business tool for toddlers. [Webpage].
Available from: http://econsultancy.com/uk/
blog/6329-the-ipad-as-a-serious-business-
tool-for-toddlers
[accessed 10 April 2011]
Wipf, A., 2010. Over-65 Audience. [Webpage].
Available from: http://culturalfuel.net/
2010/04/20/ipad-a-push-for-silver-surfing-
testing-the-ipad-with-a-65-target-audience/
[accessed 10 April 2011]
Wobbrock, J. O., Ringel Morris, M., Wilson, A.
D., 2009. User-Defined Gestures for Surface
Computing. CHI '09 Proceedings of the 27th
international conference on Human factors in
computing systems.
Yahoo News, 2010. iPad sales cross million
twice as fast as original iPhone. [Webpage].
Available from: http://news.yahoo.com/s/
ytech_gadg/20100503/tc_ytech_gadg/
ytech_gadg_tc1901
[accessed 16 February 2011]
46 Guidelines for Gestural Tablet Interaction
BibliographyBenyon, D., Turner, P. & Turner, S., 2004.
Designing Interactive Systems: People,
Activities, Contexts, Technologies. 1 ed.
Harlow, England: Addison Wesley.
Cooper, A., Reimann, R. & Cronin, D., 2007.
About Face 3: The Essentials of Interaction
Design. 3Rev Ed ed. Indianapolis, IN: John
Wiley & Sons.
Kortum, P., 2008. HCI Beyond the GUI: Design
for Haptic, Speech, Olfactory, and Other
Nontraditional Interfaces (Interactive
Technologies). Amsterdam: Morgan Kaufmann.
Moggridge, B., 2006. Designing Interactions.
Cambridge, Mass.: MIT Press.
Nielsen, J., Mack, R. L., 1994. Usability
Inspection Methods. New York: John Wiley &
Sons.
Shneiderman, B., Plaisant, C., Cohen, M. &
Jacobs, S., 2009. Designing the User Interface:
International Version: Strategies for Effective
Human-Computer Interaction: Strategies for
Effective Human-Computer Interaction:
International Edition. 5 ed. Pearson Education.
Bibliography 47
Appendix AGuessability Study Documentation & Forms
Participant A
Appendix A 49
50 Guidelines for Gestural Tablet Interaction
Participant B
Appendix A 51
52 Guidelines for Gestural Tablet Interaction
Participant C
Appendix A 53
54 Guidelines for Gestural Tablet Interaction
Participant D
Appendix A 55
56 Guidelines for Gestural Tablet Interaction
Participant E
Appendix A 57
58 Guidelines for Gestural Tablet Interaction
Participant F
Appendix A 59
60 Guidelines for Gestural Tablet Interaction
Participant G
Appendix A 61
62 Guidelines for Gestural Tablet Interaction
Participant H
Appendix A 63
64 Guidelines for Gestural Tablet Interaction
Participant I
Appendix A 65
66 Guidelines for Gestural Tablet Interaction
Participant J
Appendix A 67
68 Guidelines for Gestural Tablet Interaction
Participant K
Appendix A 69
70 Guidelines for Gestural Tablet Interaction
Participant L
Appendix A 71
72 Guidelines for Gestural Tablet Interaction
Participant M
Appendix A 73
74 Guidelines for Gestural Tablet Interaction
Participant N
Appendix A 75
76 Guidelines for Gestural Tablet Interaction
Participant O
Appendix A 77
78 Guidelines for Gestural Tablet Interaction
Participant P
Appendix A 79
80 Guidelines for Gestural Tablet Interaction
Participant Q
Appendix A 81
82 Guidelines for Gestural Tablet Interaction
Participant R
Appendix A 83
84 Guidelines for Gestural Tablet Interaction
Appendix BGesture Screenshots
Participant A
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 85
Open menu Close menu Move object
Cut Copy Paste
86 Guidelines for Gestural Tablet Interaction
Participant B
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 87
Open menu Close menu Move object
Cut Copy Paste
88 Guidelines for Gestural Tablet Interaction
Participant C
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 89
Open menu Close menu Move object
Cut Copy Paste
90 Guidelines for Gestural Tablet Interaction
Participant D
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 91
Open menu Close menu Move object
Cut Copy Paste
92 Guidelines for Gestural Tablet Interaction
Participant D
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 93
Open menu Close menu Move object
Cut Copy Paste
94 Guidelines for Gestural Tablet Interaction
Participant E
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 95
Open menu Close menu Move object
Cut Copy Paste
96 Guidelines for Gestural Tablet Interaction
Participant F
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 97
Open menu Close menu Move object
Cut Copy Paste
98 Guidelines for Gestural Tablet Interaction
Participant G
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 99
Open menu Close menu Move object
Cut Copy Paste
100 Guidelines for Gestural Tablet Interaction
Participant H
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 101
Open menu Close menu Move object
Cut Copy Paste
102 Guidelines for Gestural Tablet Interaction
Participant I
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 103
Open menu Close menu Move object
Cut Copy Paste
104 Guidelines for Gestural Tablet Interaction
Participant J
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 105
Open menu Close menu Move object
Cut Copy Paste
106 Guidelines for Gestural Tablet Interaction
Participant K
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 107
Open menu Close menu Move object
Cut Copy Paste
108 Guidelines for Gestural Tablet Interaction
Participant L
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 109
Open menu Close menu Move object
Cut Copy Paste
110 Guidelines for Gestural Tablet Interaction
Participant M
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 111
Open menu Close menu Move object
Cut Copy Paste
112 Guidelines for Gestural Tablet Interaction
Participant N
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 113
Open menu Close menu Move object
Cut Copy Paste
114 Guidelines for Gestural Tablet Interaction
Participant O
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 115
Open menu Close menu Move object
Cut Copy Paste
116 Guidelines for Gestural Tablet Interaction
Participant P
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 117
Open menu Close menu Move object
Cut Copy Paste
118 Guidelines for Gestural Tablet Interaction
Participant Q
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 119
Open menu Close menu Move object
Cut Copy Paste
120 Guidelines for Gestural Tablet Interaction
Participant R
Select single object Select multiple objects Zoom-in
Zoom-out Help Print
Undo Redo Go home
Back Forward Rotate
Appendix B 121
Open menu Close menu Move object
Cut Copy Paste
122 Guidelines for Gestural Tablet Interaction
Appendix CRating Study Documentation & Forms
Appendix C 123
124 Guidelines for Gestural Tablet Interaction
Appendix C 125
Appendix DGesture Data Charts
Select single object
Tap
Multi-Tap
Circle
0 4 8 12 16Strongly Disagree Disagree Neutral Agree Strongly Agree
Select multiple objects
Multi-Tap Consecutive
Multi-Finger Simultaneously
Number Items
Circle
Tap Consecutive
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Appendix D 127
Zoom-in
Finger Spread
Zoom-Bar Swipe
Multi-Tap
No Attempt
0 2 4 6 8 10 12 14 16Strongly Disagree Disagree Neutral Agree Strongly Agree
Zoom-out
Finger Spread
Zoom-Bar Swipe
Tap
No Attempt
0 2 4 6 8 10 12 14 16Strongly Disagree Disagree Neutral Agree Strongly Agree
128 Guidelines for Gestural Tablet Interaction
Help
ʻ?ʼ Symbol
Tap
Multi-Tap
ʻHʼ Letter
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Tap and Hold
ʻPʼ Letter
Multi-Tap
Tap
Swipe
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Appendix D 129
Undo
ʻZʼ Letter
Swipe
ʻ^ʼ Symbol
Curve
Multi-Tap
Tap
Tap and Hold
Shake Device
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Redo
Swipe
Reverse ʻZʼ Letter
ʻ^ʼ Symbol
Curve
Multi-Tap
Tap
Tap and Hold
ʻYʼ Letter
Shake Device
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
130 Guidelines for Gestural Tablet Interaction
Go home
Swipe Down
ʻHʼ Letter
Swipe Left
Tap Top-Right
ʻXʼ Letter
5 Finger Pinch
Tap and Hold
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Back
Swipe Up
Swipe Left
Arrow
Tap Left
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Appendix D 131
Forward
Swipe Down
Swipe Right
Arrow
Tap Left
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Rotate
Multi-Finger Curve
Corner-Grab Curve
Single-Finger Curve
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
132 Guidelines for Gestural Tablet Interaction
Open menu
Multi-Tap
Tap and Hold
ʻMʼ Letter
Swipe
Tap
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Close menu
Swipe
ʻXʼ Letter
Multi-Tap
Pinch
Tap Top-Right
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Appendix D 133
Move object
Swipe
Tap and Hold > Swipe
Arrow Swipe
Tap and Hold > Tap
0 2 4 6 8 10 12 14 16Strongly Disagree Disagree Neutral Agree Strongly Agree
Cut
Tap and Hold
ʻXʼ Letter
ʻCʼ Letter
Swipe
Scrub
Multi-Tap
Tap
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
134 Guidelines for Gestural Tablet Interaction
Copy
Tap and Hold
ʻCʼ Letter
Swipe
Curve
Multi-Tap
Tap
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Paste
Tap and Hold
ʻVʼ Letter
ʻPʼ Letter
Curve
Multi-Tap
Tap
No Attempt
0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree
Appendix D 135
Appendix EGesture-Meaning Association Results
Participant 1Participant 1 Participant 2Participant 2 Participant 3Participant 3
Action/Gesture Pairings Matched? Order Matched? Order Matched? Order
Select single object ✔ 4 ✔ 1 ✔ 4
Select multiple objects ✔ 5 ✔ 9 ✔ 8
Zoom-in ✔ 1 ✔ 3 ✔ 2
Zoom-out ✔ 2 ✔ 4 ✔ 3
Help ✔ 8 ✔ 10 ✔ 5
Print ✔ 9 ✔ 11 ✔ 10
Undo ✔ 11 ✔ 8 ✔ 9
Redo ✘ - ✘ - ✔ 12
Go home ✔ 17 ✔ 16 ✘ -
Back ✔ 3 ✔ 5 ✔ 6
Forward ✔ 6 ✔ 6 ✔ 7
Rotate ✔ 10 ✔ 7 ✔ 13
Open menu ✔ 15 ✔ 13 ✘ -
Close menu ✔ 13 ✔ 12 ✔ 11
Move object ✔ 7 ✔ 2 ✔ 1
Cut ✔ 16 ✘ - ✔ 16
Copy ✔ 12 ✔ 14 ✔ 14
Paste ✔ 14 ✔ 15 ✔ 15
Appendix E 137
Appendix FSupervisor Correspondence
25th January - Thomas Davies to John Halloran // Subject: 26th January Attendance
John,
I'm sorry but I am not feeling too well and therefore I don't think I will be able to make it to
tomorrow's meeting.
This week I have started the guessability study and have completed four tests at the moment. As I
wasn't too sure about just having the before and after screenshot, I am showing participants an
animation of the action (Eg. Moving an object action will show the object moving across the screen)
prior to them creating the gesture. The gesture is still drawn on top of the before screenshot though.
The entire process is being recorded so I can ensure all gestures and the order in which they occur
are documented. I am also observing the participants and having them think aloud.
I have also nearly completed the methodology section of the write up.
Regards,
Thomas
Appendix E 139
9th February - John Halloran to Thomas Davies // Subject: 9th February Meeting Notes
IMAGE DEFINITION
inter-rater agreement
get a panel of people working independently
show them video of gestures
ask them to create clusters where they think the gesture is the same
get THEM to come up with clusters; diff raters' clusters should agree
ask raters for confidence levels when they put an item in cluster
(5 point Likert scale)
they can have a 'misc' pile
cronbach's alpha inter rater reliability (google)
PILOT STUDY
10 gestures
5 users
(raters would need to have some experience of this type of computing)
VALIDATION
image-meaning association - validation
aim for 2 weeks on the pilot
---------------------------------------------------
Dr John Halloran BSc, MSc, PhD
Department of Computing and the Digital Environment
Coventry University
Coventry CV1 5FB
john DOT halloran AT coventry DOT ac DOT uk
ext 7631
140 Guidelines for Gestural Tablet Interaction
23rd February - John Halloran to Thomas Davies // Subject: 23rd February Meeting Notes
change study
pairwise rater agreements within categories
18 categories
2 x 9
maximum of 9 users (say) - 2 each
- stats. difficult! needs SPSS; and what is the right stat.
- I'll do a bit of research - looks like fliess of cohen (not cronbach)
- work out the logic of the thesis
- we are looking for a standard set - intuitive and recognisable
- so what?
- supports the thesis that physical knowledge is basic, universal, not learned so easily deployed for
this interaction style and validates interaction style (lit about people who say it is 'natural' / innate vs
those who say rubbish it is all learned)
- support developers
- supports users
- where manufacturers deviate it is going to be harder to learn
- independent user test of the results
j
---------------------------------------------------
Dr John Halloran BSc, MSc, PhD
Department of Computing and the Digital Environment
Coventry University
Coventry CV1 5FB
john DOT halloran AT coventry DOT ac DOT uk
ext 7631
Appendix E 141
9th March - John Halloran to Thomas Davies // Subject: 9th March Meeting Notes
'Eyeball' shows inter-rater agreement within categories
'do you believe this gesture suits the desired action' - Likert 1-5
3 raters
we need a chart, as I scribbled
need a test for statistically significant agreement between raters
I do not know what this test is!
Next time 23rd March
- I want to see a draft dissertation
- if not done, put placeholder content e.g. 'What is going to appear here is as follows'
- Get the logic of the thesis and how the studies support it
- I am away 4th-11th inclusive (but not otherwise)
- aim at a longer supervision on 30th
1. we show that there is statistically significant agreement between raters
2. implement the gesture set
- description / storyboard
3. validate it
- 'image meaning test'
---------------------------------------------------
Dr John Halloran BSc, MSc, PhD
Department of Computing and the Digital Environment
Coventry University
Coventry CV1 5FB
john DOT halloran AT coventry DOT ac DOT uk
ext 7631
142 Guidelines for Gestural Tablet Interaction
23rd March - John Halloran to Thomas Davies // Subject: 23rd March Meeting Notes
Make sure there is a full draft dissertation on my desktop 9am Tues 12th April for comment
Keep hammering the research question, what it is, where we are with it, what got established so far
and what's next and why it's relevant
Then we will arrange a meet in week 18-21 if need be
Check the project guide esp the marking criteria
remember there are FOUR components - brief (10), poster (10), report (40), 'product/quality' (40)
what are the forms? request from Richard Rider
---------------------------------------------------
Dr John Halloran BSc, MSc, PhD
Department of Computing and the Digital Environment
Coventry University
Coventry CV1 5FB
john DOT halloran AT coventry DOT ac DOT uk
ext 7631
| DREAMSEQUENCE | THE ANNUAL CREATIVE COMPUTING SHOW | Friday 6 May 16:00, Humber
Theatre, Coventry University | www.dreamsequence.co.uk |
Appendix E 143
14th April - John Halloran to Thomas Davies // Subject: Project
Tom
I have had nothing from you, which is uncharacteristic. I am guessing this could be server rejection
but am a bit worried it might be illness for e.g.
Let me know what's happening. We'll probably need to meet next week
Regards
John
15th April - Thomas Davies to John Halloran // RE Subject: Project
John,
Thanks for emailing me. I emailed it in the early hours of Tuesday but it obviously didn't go through.
I'm not at home right now so I won't be able to resend it until the evening. I understand that it will be
difficult for you to read through it now though so don't worry.
In that email, I also asked if it would be possible for you to act as one of my references for my
application to City London. There are no immediate time constraints for the reference though. I've
asked Peter to be my second reference at the same time - he got my email!!
Thank,
Thomas
15th April - John Halloran to Thomas Davies // RE RE Subject: Project
OK. Yes happy to be a ref
What about meeting next week? Say Mon; I could do 15:00 or 12:00
Regards
John
15th April - Thomas Davies to John Halloran // RE RE RE Subject: Project
12:00 will be great.
Thanks,
Thomas
15th April - John Halloran to Thomas Davies // RE RE RE RE Subject: Project
OK confirmed
See you then
Regards
John
144 Guidelines for Gestural Tablet Interaction
20th April - Thomas Davies to John Halloran // Dissertation
John,
Here is the link to my dissertation - http://files.me.com/thomasdavies/e9jcls
Also, as you didn't receive my original email, here is also the reference document for the City
University post-grad course - http://files.me.com/thomasdavies/eautlv
Regards,
Thomas
20th April - John Halloran to Thomas Davies // RE Dissertation
just confirming I've received this - I can download the doic no problem
expect feedback by close of play tomorrow
Regards
John
20th April - John Halloran to Thomas Davies // RE RE Dissertation
Thomas
This is looking excellent. The only issue is the contents page doesn't match the sections - in
particular I can't see a project management section, and the page nos look wrong. Make sure the
project management section appears and that the sections in general are consistent with the
contents page.
I trust you to do this without further guidance. I suggest you try to get it printed and bound before
next week and submit asap which I guess will be Tuesday. I am confident that you know what the
procedure is.
Good Luck
Regards
John
Appendix E 145