Abstract

Computer-supported sketching-based design tools are becoming increasingly

available to aid designers as it bridges the gap between traditional design tools/media, such

as paper and pen and computer-aided design (CAD) software. However, there has been

little empirical research on the effects of using this type of informal design tool, and

almost none on the effects of beautification, using such tools, on the design process.

Beautification is described as the process of tidying up a hand-drawn design; and formality

is described as the outcome of beautification i.e. level of tidiness and professionalism

conveyed in the appearance of a design. The main purposes of this study were: 1) explore

the concept of beautification in the context of sketch-based design tools by examining the

dimensionality of beautification; and also 2) to investigate levels of formality of designs,

from hand-drawn (non-beautified) sketches to computer-rendered (beautified) diagrams,

and their effects of on design performance (i.e. number of changes made to designs

presented) during early stages of the design process. Results showed that: 1) as the level

of formality increases, the number of changes made (total, quality and expected changes)

decreases, and vice versa (i.e. a negative linear relationship between formality and design

performance); 2) experts performed at a higher level in comparison to novices’

performance across levels of formality; 3) subjects enjoyed working on designs that with

higher formality more than designs with a lower formality; 4) there was no difference

found in the preference between designing on paper compared to designing on the tablet

PC (InkKit) during the experiment; and 5) design tool preference(s) in real world design

situations was more diverse than the design medium/tool preferred in the experiment.

Important implications arose from this study include: 1) design education on the effects of

formality as a result of beautification; 2) improvements on the design process such as

easier preparation for client presentation and improved efficiency; and 3) sketch-based tool

development, in particular InkKit, to support more satisfying, natural designer-design tool

ii

interaction. Future directions in research such as replication and extension of the present

study, and methodological improvements are also recommended and discussed.

iii

Acknowledgement

First, I would like to thank my supervisors, Dr Brenda Lobb, Dr Beryl Plimmer and

Dr Doug Elliffe. Brenda, you helped open up my mind to see (and feel) answers from

different angles and be a critical reader, thinker and doer; most importantly, you pulled me

up and pushed me forward when I lost faith in my work and myself – thank you. Beryl,

thank you for making me believe in the computer science side of me again. Without your

encouragement and support at the beginning of this project, I would not have made it

through the programming days in the lab. Thank you Doug, for helping me refine the

experiment and data collection procedures; and for clearing up the grey clouds in my mind

on results analyzes.

I would also like to thank my family: Mum and Dad, for giving me life in the first

place and for your unconditional love and support, I love you; and Susan, my sister, my

best-friend, for being a good listener and advisor, and thank you for being true, I love you

too. Also my gym buddy and dining buddy, Iris, thanks for making me laugh when I don’t

smile, for keeping me positive, and for your encouragement throughout the year.

ii

Table of Contents

Abstract..........................................................................................................iiAcknowledgement.........................................................................................iiTable of Contents.........................................................................................iiiList of Tables...............................................................................................viiList of Figures...............................................................................................ixChapter 1. Introduction................................................................................1

1.1. Design Research: The design process...............................................................21.2. Design process as problem-solving...................................................................3

1.2.1. Factors affecting design performance.....................................................41.2.1.1. Expert vs. novice designers....................................................................41.2.1.1. Individual differences in problem-solving...................................71.1.2.2. Other factors affecting design performances.......................................11

1.1. Human-Computer-Interaction (HCI) and Design...........................................121.2. Prototypes, Prototyping and Prototyping tools...............................................14

1.4.1. Traditional Design tools for Prototyping..............................................181.4.1.1. Paper and Pen(cil)................................................................................181.4.1.2. Computer-Aided Design (CAD) Tools................................................191.4.1.3. Combination of Paper and Pen, and CAD...........................................211.4.1.4. Paper prototypes verses Digital prototypes..........................................23

1.3. Current Trend in Design Tools Research: Informal Sketch-based interface. .251.3.2. 2-Dimensional (2-D) sketch-based systems..........................................271.5.2. 3-Dimensional (3-D) Sketch-based systems.........................................281.5.3. On Improving Computer-Supported sketch tools.................................291.5.4. Bridging the Gap: A closer look at ‘Beautification’ (‘Formalization’)30

1.5.4.1. ‘Beautification’ versus ‘Formality’......................................................321.5.4.2. Practicality of Beautification in the design process.............................341.5.4.3. Beautification techniques and supporting systems..............................35

1.4. Related studies: Interaction with hand-drawn versus computer-rendered diagrams..........................................................................................................42

1.5. The Present study: Aims and hypotheses........................................................48

Chapter 2. Method......................................................................................51

2.1. Experimental Design.......................................................................................512.1.1. Independent Variable: Level of Formality............................................512.1.2. Dependent Variables: Functional changes...........................................52

2.2. Participants......................................................................................................552.3. Procedure.........................................................................................................562.4. Apparatus........................................................................................................59

2.4.1. Room Setup...........................................................................................592.4.2. The Tablet PC........................................................................................592.4.3. Morae Recorder (2004).........................................................................602.4.4. Inkit and the programming of beautification functions.........................60

2.4.4.1. Horizontal Alignment..........................................................................612.4.4.2. Vertical Alignment..............................................................................632.4.4.3. Standardization....................................................................................632.4.4.4. Line Smoothing...................................................................................64

2.5. Stimuli and Materials......................................................................................68

iii

2.5.1. Instruction Sheets..................................................................................682.5.2. The five designs each representing a different level of formality.........682.5.3. Post-task Questionnaire.........................................................................74

Chapter 3. Results.......................................................................................75

3.1. Data-screening of performance data...............................................................753.2. Analysis of performance data: One-way repeated measures ANOVA...........76

3.2.1. Analysis of “Total Changes” made across levels of formality..............773.2.1.1. Between-Subject Factors........................................................................79

3.2.1.1a. Design experience..............................................................................803.2.1.1b. Study major/specialization................................................................823.2.1.1c. Study Level........................................................................................83

3.2.1.2. Multiple Regression analysis..................................................................853.2.2. Analysis of “Quality Changes” made across levels of formality..........86

3.2.2.1. Between-Subject Factors........................................................................893.2.2.1a. Design Experience.............................................................................893.2.2.1b. Study major/specialization................................................................913.2.2.1c. Study Level........................................................................................93

3.2.2.2. Multiple Regression Analysis.................................................................953.2.3. Analysis of “Expected Changes” made across levels of formality.......97

3.2.3.1. Between-Subjects Factors.......................................................................993.2.3.1a. Design Experience.............................................................................993.2.3.1b. Study major/specialization..............................................................1013.2.3.1c. Study Level......................................................................................103

3.2.3.2. Multiple Regression Analysis...............................................................1043.3. Additional Analysis of performance data.....................................................106

3.3.1. Paired comparisons: Total, Quality and Expected changes................1063.3.2. Extra changes: Quality – Expected; and Total – Quality....................1073.3.3. Order Effect.........................................................................................109

3.4. Analysis of the “Overall Enjoyment” rankings of the five designs..............1133.4.1. Ranking according to design appearance (aesthetics).........................1143.4.2. Ranking according to perceived “effort required”..............................1153.4.3. Ranking according to perceived “fun and/or stimulating level”.........116

3.5 Design Tool Preference..................................................................................1173.5.1. Design tool preference in the experiment...........................................1173.5.2. Design tool preference in the experiment...........................................119

Chapter 4. Discussion................................................................................122

4.1. Effects of formality on design task performance..........................................1224.2. Between-subject effects: Expertise...............................................................132

4.2.1. Design experience................................................................................1324.2.2. Study major/specialization..................................................................1334.2.3. Study Level..........................................................................................134

4.3. Multiple Regression Analysis.......................................................................1354.4. Additional Findings.......................................................................................136

4.4.1. Relationship between total, quality and expected changes.................1364.5. “Overall Enjoyment” rankings of the five designs ranking of designs.........138

4.5.1. Rankings according to aesthetic aspects of designs............................1384.5.2. Rankings according to effort required.................................................1404.5.3. Rankings according to Stimulation/fun level......................................141

iv

4.6. Design tool preference..................................................................................1424.6.1. Design tool preference during the experiment....................................1424.6.2. Design tool preference in the real world.............................................145

4.7. Implications...................................................................................................1464.7.1. Implications on sketch-based tool development: Recommendations for

InkKit...................................................................................................1464.7.2. Improvements in the design process....................................................1484.7.3. Implications on design education........................................................149

4.8. Methodological issues and limitations..........................................................1514.9. Future research and directions......................................................................151

Chapter 5. Summary and Conclusion.....................................................155References..................................................................................................157Appendices.................................................................................................168

Appendix A. The five designs and the outline of design errors present in each design............................................................................................................169Appendix A1.1. Low formality design on paper – Online Magazine

subscription form.................................................................................170Appendix A.1.2. Online Magazine: Planned design errors...........................171Appendix A2.1. Low formality design on tablet PC – Samson’s Bank $1

million loan application form..............................................................172Appendix A2.2. Samson’s loan: Planned design errors................................173Appendix A3.1. Medium-low formality design on tablet PC – University of

Strawberries graduation application form...........................................174Appendix A3.2. University of Strawberries: Planned design “errors”.........175Appendix A4.1. Medium-high formality design on tablet PC – Dog

Registration Form................................................................................176Appendix A4.2. Dog Registration’s: Planned design “errors”.....................177Appendix A5.1. High formality design on tablet PC – America’s Next Top

Model application form.......................................................................178Appendix A5.2. America’s Next Top Model: Planned design errors...........179

Appendix B. Post-task Questionnaire..................................................................180Appendix C. Results of post-task questionnaire..................................................184Appendix D. Participant information sheets and consent forms..........................185Appendix E. Functional Aspects of Inkit............................................................192Appendix F. Instruction sheets containing the requirements and scenario

associated with each design..........................................................................193Appendix F1. Instructions including the requirements and the scenario for the

low formality (on paper) design..........................................................194Appendix F2. Instructions including the requirements and the scenario for the

low formality (on Tablet PC) design...................................................197Appendix F3. Instructions including the requirements and the scenario for the

medium-low formality design.............................................................200Appendix F4. Instructions including the requirements and the scenario for the

medium-high formality design............................................................203Appendix F5. Instructions including the requirements and the scenario for the

high formality design...........................................................................206Appendix G. Screen shots during font creation using My Font Tool for Tablet PC

(2004)............................................................................................................209Appendix H. Testing the normality assumption – Total number of changes made

across levels of formality..............................................................................211

v

Appendix I. Testing the normality assumption – Number of quality changes made across levels of formality..............................................................................218

Appendix J: Testing the normality assumption – Number of expected changes made across levels of formality....................................................................225

Appendix K. Mean total number of changes made across each level of formality – according to a combination of between-subjects factors (design experience, major/specialization and study level)............................................................232

Appendix L. Mean number of quality changes made across each level of formality – according to a combination of between-subjects factors (design experience, major/specialization and study level)............................................................233

Appendix M. Mean number of expected changes across each level of formality – according to between-subjects factors (design experience, major/specialization and study level)............................................................234

Appendix N. One-way ANOVA and post-hoc multiple comparisons between total, quality, and expected changes made across each level of formality....235

Appendix O: “Extra changes” made in designs...................................................236Appendix O1. “Extra changes” made in the Low Formality Design presented

on paper: International Online Magazine Subscription Form.............237Appendix O2. “Extra changes” made in the Low Formality Design on tablet

PC: Samson’s Bank $1 million Loan Application Form.....................240Appendix O3. “Extra changes” made in the Medium-Low Formality Design:

University of Strawberries Graduation Form......................................243Appendix O4. “Extra changes” made in the Medium-High Formality Design:

Dog Registration Online Form............................................................245Appendix O5. “Extra changes” made in the High Formality Design: 2007

America’s Next Top Model Online Application Form.......................247Appendix P. “Overall Enjoyment” rankings of the five designs across each level

of formality...................................................................................................249Appendix Q. Number of changes made in the five designs across levels of

formality by subjects whose “Overall Enjoyment” ranks was dependent on the appearance (aesthetics) of designs..........................................................251

Appendix R. Number of changes made in the five designs across levels of formality by subjects whose “Overall Enjoyment” ranks was dependent on perceived effort required...............................................................................252

Appendix S. Number of changes made in the five designs across levels of formality by subjects whose “Overall Enjoyment” ranks was dependent on the level of fun/stimulation when working on the designs...........................253

Appendix T. Subjects reasons for design tool preference during the experiment254Appendix U. Subjects reasons for design tool preference in real life design

situations.......................................................................................................256

vi

List of Tables

Table 1. Level of formality associated with each condition, and the medium used for the presentation and review of designs...........................................................1

Table 2. Taxonomy of beautification showing different variables associated with beautification..................................................................................................2

Table 3. Ratio of elements aligned (and its description) in each design representing a different level of formality.............................................................................20

Table 4. Systematic smoothing applied (% smoothed) to the original hand-drawn lines of textboxes, dropdown menus and radio buttons; and fonts used for labels to represent different levels of formality in the designs presented to the participant...............................................................................................21

Table 5. Mean and standard deviation for total changes made at each level of formality........................................................................................................27

Table 6. Mean differences and their significance at the .05 level in terms of total number of changes made between each condition........................................29

Table 7. Mean and standard deviation for total changes made, and the mean difference between groups, at each levels of formality according to subjects’ design experience (total n =30): none to some (non-CS/SE) design experience (n = 15) and CS/SE design experience (n = 15)........................30

Table 8. Mean and standard deviation for total changes made, and the mean difference between groups, at each level of formality according to subjects’ major/specialization in university (Total n =30): non-CS/SE related major (n = 10) and CS/SE related major (n = 20)..................................................32

Table 9. Mean and standard deviation for total changes made, and the mean difference between groups, at each level of formality according to subjects’ study level (total n=30): undergraduate (n=22) and graduate/postgraduate (n=8).............................................................................................................33

Table 10.1. R, Adjusted R Square and R Square change for total changes made across levels of formality, with formality level, design experience, study level and major/specialization as predictors entered...................................................36

Table 10.2. The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the mode for explaining total changes made......................................................36

Table 11. Mean and standard deviation for quality changes made at each level of formality .......................................................................................................36

Table 12. Mean differences and their significance at the .05 level in terms of the number of quality changes made between each condition...........................38

Table 13. Mean and standard deviation for quality changes made, and the mean difference between groups, at each level of formality according to design experience (total n=30): none to some (non-CS/SE) design experience (n=15) and CS/SE design experience (n=15) ..............................................39

Table 14. Mean and standard deviation for quality changes made, and the mean difference between groups, at each level of formality according to subjects’ major/specialization in Auckland University: Non-CS/SE related major (n = 10) and CS/SE related majors (n = 20)....................................................41

Table 15. Mean and standard deviation of quality changes made, and the mean difference between groups, at each level of formality according to subjects’ study level: undergraduate (n=22) and graduate/postgraduate (n=8).......43

Table 16.1. R, Adjusted R Square and R Square change for the number of quality changes made across levels of formality, with formality level, design

vii

experience, study level and major/specialization as predictors entered......45Table 16.2. The unstandardized and standardized regression coefficients, and the t-

value and significance of each between-subject variables included in the model for explaining quality changes made.................................................46

Table 17. Mean and standard deviation for expected changes made across levels of formality........................................................................................................46

Table 18. Mean differences and their significance at the .05 level in terms of the number of expected changes made between each condition.........................48

Table 19. Mean and standard deviation for expected changes made, and the mean difference between groups, at each level of formality according to design experience (Total n =30): CS/SE design experience (n = 15), none to some (non-CS/SE) design experience (n = 15)......................................................49

Table 20. Means and standard deviations for expected changes made, and the mean difference between groups, at each level of formality according to major/specialization (n=30): non-CS/SE related major (n=10); CS/SE related major (n=20.....................................................................................51

Table 21. Mean and standard deviation of expected changes made, and the mean difference between groups, at each level of formality according to study levels (n=30): undergraduate (n = 22); graduate/postgraduate (n = 8).....52

Table 22.1. R, Adjusted R Square and R Square change for the number of expected changes made across levels of formality, with formality level, design experience, study level and major/specialization as predictors entered......54

Table 22.2. The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the model for explaining expected changes made..............................................55

Table 23. The number of extra changes (quality – expected) made in each design, grouped according to the type of change......................................................57

Table 24. The number of extra changes (total – quality) changes made in each design, grouped according to the type of change......................................................57

Table 25. Mean ranks and standard deviation, in terms of overall perceived enjoyment and other underlying factors for subjects’ rankings (including appearance, perceived effort required, and perceived fun/stimulating level), when working on each design in comparison to other designs presented.............63

viii

List of Figures

Figure 1. Partial counterbalancing: orders of presentation of conditions..................1Figure 3. The practice design – presented to the participant prior the first experiment

condition.........................................................................................................7Figure 4. A bird-eye view of the room set-up for the experimental sessions...............10Figure 5. Horizontal alignment of elements.................................................................12Figure 6. Vertical alignment of elements......................................................................13Figure 7. Standardization of the size of objects............................................................14Figure 8. Levels of smoothing of hand-drawn objects (lines) to represent levels of

formality.....................................................................................................15Figure 9. The four fonts used to represent different levels of formality.......................17Figure 10.1. Low Formality (on paper) – Online Magazine subscription form..........22Figure 10.2. Low Formality (on Tablet PC) – Samson’s Bank $1 million loan

application form.....................................................................................22Figure 10.3. Medium-Low Formality – University of Strawberries graduation

application form.....................................................................................23Figure 10.4. Medium-High Formality - Dog Registration Form................................23Figure 10.5. High Formality design - America’s Next Top Model application form..24Figure 11. Multi-line graph showing mean total changes made across levels of

formality which is represented by the black bold line; each participant’s performance (in terms of total changes made across levels of formality) is also illustrated – see individual lines.........................................................28

Figure 12. Multi-line graph of mean total changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience.......................................31

Figure 13. Multi-line graph of mean total changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors...................................................33

Figure 14. Multi-line graph of mean total changes made across levels of formality according to subjects study level: undergraduate and graduate/postgraduate...............................................................................34

Figure 15. Multi-line graph showing mean quality changes made across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of quality changes made across levels of formality) is also illustrated – see individual lines.....................................................37

Figure 16. Multi-line graph of mean quality changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience.......................................40

Figure 17. Multi-line graph of mean quality changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors...................................................42

Figure 18. Multi-line graph of mean quality changes made across levels of formality according to subjects’ study level: undergraduate and graduate/postgraduate...............................................................................44

Figure 19. Multi-line graph showing mean expected changes across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of expected changes made across levels of formality) is also illustrated – see individual lines....................................47

Figure 20. Multi-line graph of mean expected changes made across levels of formality according to subject’s design experience...................................................50

ix

Figure 21. Multi-line graph of mean expected changes made across levels of formality according to subject’s major/specialization: non-CS/SE related major and CS/SE related majors.................................................................................52

Figure 22. Multi-line graph of mean expected changes made across levels of formality according to subjects’ study levels: undergraduate and graduate/postgraduate...............................................................................53

Figure 23. Multi-line graph of mean total changes, mean quality changes and mean expected changes across levels of formality...............................................56

Figure 24a. Mean total changes made across levels of formality according to the order of conditions presented.....................................................................59

Figure 24b. Mean total changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 directions (n=4)59

Figure 25a. Mean quality changes made across levels of formality according to the order of conditions presented.....................................................................60

Figure 25b. Mean quality changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 (n=4) directions60

Figure 26a. Mean expected changes made across levels of formality according to the order of conditions presented.....................................................................61

Figure 26b. Mean expected changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 (n=4) directions61

Figure 27. A bar graph showing mean rank and standard deviation, in terms of preference, according to the overall enjoyment, in working on each design with a different level of formality...............................................................62

Figure 28. Bar graph showing subjects’ design tool preference during the experiment67Figure 29a. Bar graph showing the proportion of subjects – according to study major:

CS/SE (n=20) and non-CS/SE majors (n=10) – preferring different design tools during the experiment (paper and pen; Tablet (Inkit); or no preference)..................................................................................................68

Figure 29b. Bar graph showing the proportion of subjects – according to study major: CS/SE design experience (n=15) and none to some non-CS/SE design experience (n=10) – preferring different design tools during the experiment (paper and pen; Tablet (Inkit); or no preference)...................68

Figure 30. Bar graph showing subjects’ design tool preference in real life design situations....................................................................................................69

Figure 31a. Bar graph showing the proportion of subjects – according to study major: CS/SE (n=20) and non-CS/SE majors (n=10) – preferring different design tools in real life design situations...............................................................70

Figure 31b. Bar graph showing the proportion of subjects – according to study major: CS/SE design experience (n=15) and none to some non-CS/SE design experience (n=10) – preferring different design tools in real life design situations....................................................................................................71

x

Chapter 1. Introduction

Computer-supported sketching-based design tools – also referred to as informal

design tools as they support natural human-computer interaction (i.e. sketching) – are

becoming increasingly available to aid designers across various design disciplines. The

main advantage of using such tools is that it bridges the gap between traditional design

tools/media, such as paper and pen and computer-aided design (CAD) software, such that

sketching (a natural, important design behavior) is supported, while also providing

additional functions such as editing and version control, as well as recognition and

beautification of pen input. However, there has been little empirical research on the

effects of using this type of informal design tool, and almost none on the effects of

beautification (design formality) using such tools, during the design process. In other

words, there is a need to examine the effects of design formality (appearance) as a result of

beautification (beautifying sketched content), on designers’ cognition, and hence, design

performance and outcome.

With this in mind, designers’ interaction with design tools/mediums and design

formality (as a result of beautification) were examined in the present study. The main

purposes of this study were to use an experimental approach to: 1) further explore the

concept of beautification in the context of sketch-based design tools by examining the

dimensionality of beautification; and also 2) to investigate levels of formality of designs,

from rough, (hand-drawn, non-beautified) sketches to formal (beautified) diagrams, and

their effects of on design performance during early stages of the design process.

This chapter of the thesis is organized into a few sections. First, research on the

design process and the manner in which it has been studied in the past are discussed, and

design as an emerging topic within the area of human-computer interaction is also

highlighted. The next section is on the development and the use of design tools, within

1

which, two traditional design tools (pen and paper and computer-aided design tools) are

compared, followed by a discussion on the recent trend of design tools with a sketching-

based interface. Furthermore, the concept of beautification (and formality) is described,

and research on informal sketch-based design tools that supports beautification is

reviewed. Research relevant to this study is also presented. Finally, in the last section of

this chapter, the aims and hypothesis of the present study are outlined.

1.1. Design Research: The design process

Design has long been of interest to many groups, from academics (e.g. researchers,

philosophers and psychologists) to practitioners (e.g. educators, engineers, architects) ever

since design activities began. It was during the late 1960s, that scientific research on

design began, according to Bayazit (2004), with different design research associations

founded across the globe during the time. For example, the Design Research Society,

which started the Design Studies journal – a journal dedicated to all design related studies

and research across multiple disciplines including architectural design, engineering design,

industrial design and software design. Since then, design research grew steadily

throughout the years (see Bayazit, 2004; Roth, 1999; and Downton, 2003 for fuller

historical accounts of design research and its current state). Cross (1999) distinguished

three categories of research on design, based on people, process and products:

Design epistemology – study of ways of knowing of designers

Design praxiology – study of practices and processes of design

Design phenomenology – study of the form and configuration of artifacts.

One of the major topics in design research is on the design process, which has been

much studied across disciplines, based on different approaches and perspectives, including

psychological, social, philosophical and mathematical. Researchers have described the

design process using different stage-process models. For example, Crampton-Smith and

2

Tabor (1996) described a generic design process that consists of understanding,

abstracting, structuring, representing and detailing. On the other hand, from the interviews

with eleven professional Web designers in their workplace, Newman et al. (2003) found

four main phases: discovery, design exploration, design refinement, and production. Such

pattern of iterative refinement was also discovered by Rowe (1987) who reviewed of a

number of staged-process models of design that were proposed in the early 1960s. Along

with the models to describe the design process, a wide range of studies were conducted

with focus on different aspects of the design process including the design process as a

whole (e.g. Atman, et al, 1999; Atman, et al, 2005); factors affecting the design process

(e.g. Darke, 1979; Naga & Noguchi, 2003; Ward, 1989); and design tools and

methodology used during the design process (e.g. Grosjean, & Brassac, 2000; Bilda, Gero,

& Purcell, 2006; Shneiderman, et al, 2006).

1.2. Design process as problem-solving

Design has been described as a complex and fastidious mental activity (Romer,

Leinert, & Sachse, 2000), which can be viewed as a kind of problem-solving

(Goldschmidt, 1997; Hegarty 1991; Rowe 1987; Smith and Browne 1993; Thomas &

Caroll, 1979). From a broader cognitive psychology perspective, problem-solving

encompasses a wide range of activities in which one is required to identify the solution to

a current problem (Green & Gilhooly, 2005); hence, the process of designing can be

viewed as part of problem-solving. As noted by Green and Gilhooly (2005), problem

solving is an activity that draws together the various different components of cognition –

for example, visual perception for the understanding a graphically presented problem and

for drawing a solution; as well as memory to recover any prior knowledge one might have

that could be relevant to solving a new problem; and attention which plays an important

role in all problem solving. Thus, the design process can be understood as a problem-

3

solving process. Furthermore, the major focus on research on problem-solving has been

on task performance (see Ericsson, 1991); hence, research on design as problem solving

has also as been on design performance (in terms of qualitative and quantitative

measurements such as time used, design quality and design outcome).

1.2.1. Factors affecting design performance

1.2.1.1. Expert vs. novice designers

Expertise in design has been viewed by many (e.g. Cross, 1999) as an important

factor that affects designers’ performance in the design process. One of the major

approaches towards exploring the concept of expertise within the context of problem-

solving (design) is to compare experts and novices. According to Cross (2004), novice

behaviour is usually associated with a ‘depth-first’ approach to problem solving, i.e.

sequentially identifying and exploring sub-solutions in depth, whereas the strategies of

experts are usually regarded as being predominantly ‘top-down’ and ‘breadth-first’

approaches. Many of the classic studies of expertise have been based on examples of

game-playing (e.g. chess), or on comparisons of experts versus novices in solving routine

problems (e.g. mathematics and physics). For example, early chess studies carried out by

De Groot (1946/1965) showed that instead of having superior information processing

capabilities, ‘experts’ players (grand masters) ultimately chose better moves than ‘novice’

(highly skilled) players; and that skill level was linked to the amount of information

remembered (i.e. better chunking). The results from Chase and Simon’s (1973) study, also

conducted on chess players, suggested that experts not only processes more knowledge

about their domain of expertise, but their knowledge was organized in more meaningful

and readily accessible ways. These studies provided a strong the basis for later studies on

problem solving as it showed that skill depended at least in part on the acquisition of

domain knowledge and stimulated a vast amount of research on the nature of expert

4

problem solving and the relationship between knowledge and skill. However, one must

point out that those studies used well-defined problems, whereas designers

characteristically deal with ill-defined problems (Cross, 2004; Romer, Leinert, & Sachse,

2000), therefore findings may not be generalizable to the design process.

There are some empirical studies specifically on design cognition, amongst which

have been studies on expert, or experienced designers, and comparisons of the processes

of novice and expert designers. The recent focus of such studies in design cognition has

been through the use of protocol analysis studies (Cross, Christiaans & Dorst, 1996).

Similar to the findings in Christiaans and Dorst’s (1992) analyses of the design protocol of

junior and senior industrial design students, Atman et al. (1999) found from their in-depth

protocol studies of twenty-six freshman engineering students (first year) and twenty-four

(fourth year) engineering students as they designed a playground for a fictitious

neighborhood, that there was a difference in terms of design performance and behaviour

between senior students and freshman in the design process. The results showed that the

seniors produced higher quality designs than freshman; in addition, the seniors gathered

more information, considered more alternative solutions, moved more frequently between

design steps and progressed further into the final steps of the design process. Atman et al.

(1999) suggested that the design processes the seniors learned in their four years

contributed to the differences found. Overall, it was an in-depth study, with carefully

controlled variables, as well as observations, video recordings and meticulously defined

elements in the protocol analysis, which in turn, enabled rich data to be collected and

analyzed.

As problem solving occurs over time, and the notion that experts become expert

through extensive practice – “practice makes perfect” (Green, & Gilhooly, 2005), follow-

up and longitudinal studies provided a richer understanding of expertise in design (verses

novice) over time. Adams, Turns and Atman (2003) conducted a longitudinal study on the

5

behaviours of freshman verses senior students during design. As the authors commented,

changes in individual students’ behaviours over time can be quite complex and variable –

as one would predict. Definite change was noticeable for some, however, for some of the

students, their behaviours did not appear to change at all and some simply spent more time

on design projects but without any qualitative behavioural changes. It also appeared that

students exhibited different behavioural changes for different types of design projects.

Similarly, Atman et al. (2005) conducted a follow-up study to their 1999 research,

in which verbal protocols was, again, collected and freshman and senior engineering

design processes to open-ended design problems was compared. The difference from the

earlier study was the increase in the number of subjects who participated: sixty-one seniors

(fourth year) engineering students and thirty-two freshman (first year) engineering

students; as they worked on two design problems (as opposed to one design problem in the

previous study). In addition, the study also included protocols for eighteen within-subjects

participants, who participated in the study first as freshmen and later as seniors, which

provided the rich data sources for the comparison of design process changes over time on

the individual student level. In terms of between-subject differences in design process

changes, results showed that seniors produced higher quality solutions, spent more time

solving the problem, considered more alternative solutions and made more transitions

between design steps than the freshmen to a greater extent, but comparable to results in

Atman et al. (1999). Similar to the findings for the general population of freshmen and

seniors, the within-subjects participants as a group showed differences between freshman

and senior participation, however, some participants did not exhibit growth on all

measures. Also, comparable to Adam, Tums and Atman’s findings in 2003, individual

students’ design behaviour varied by problem (i.e. was task dependent). Again, like the

study in 1999, this study was highly controlled and used coded variables to support the

analysis and interpretation from the rich data obtained.

6

However, results from the longitudinal study should be interpreted with caution as

the operational definition of change and the criteria for behavioural changes were

ambiguous. As the measure of behaviours over time can be complex and variable, one

must bear in mind that such study could not be conclusive with respect to expertise and

behavioural changes over time. Hence, in addition, the comparison of results found in

Adams, et al (2003) and Atman, et al. (2005) should be examined critically as the two

studies measured different aspects (design process as a whole verses behaviours during

design) and had different problem solving tasks over a different period of time.

1.2.1.1. Individual differences in problem-solving

As noted, research on design cognition often looks into design as problem solving,

particularly on expertise in design. Individuals are frequently categorized into experts and

novices and studies often focused on the between-group differences in the design process,

and sometimes non-differences were found. In addition to this, there has also been interest

in and increasing empirical research on the relationships between cognitive style, design

strategy and design performance (Cross, 1985; Kvan and Yunyan, 2005), which in turn,

affect the design process (i.e. iteration pattern and time spent on the different design

phases).

According to Lawson (1979) problem solving strategies can be categorized as

either ‘problem focused’ or ‘solution focused’, in which in a recent empirical study on

design protocol by Kruger and Cross (2006), were both found to be used by the nine

experienced industrial designers performing the same task, along with some sub-variants

(‘information driven’ and ‘knowledge driven’ respectively – see Kruger and Cross (2006)

for a more detailed description of the different cognitive strategies). Clear individual

differences between designers were found in most of the data relating to both design

process solution outcome, even though the designers performed the same tasks under the

7

same conditions. Kruger and Cross also found that most designers employed either

problem driven or a solution driven design strategy, and each were equally prevalent.

However, contrary to Lawson’s earlier claim in 1979, solution driven design was not used

as the dominant strategy. Furthermore, the different strategies used were related to the

overall solution quality (task outcomes) in a complex way. Designers who employed a

solution driven strategy was found to have lower overall solution quality, but higher

creativity scores. Where as, designers using a problem driven design strategy tended to

produce the best results in terms of the balance of both overall solution quality and

creativity. Overall, it was a rich descriptive study on design strategies and design

performance and outcome. However, it must be pointed out that, similar to many other

studies examining cognitive processes, the experiments were conducted as ‘think-aloud’

protocol studies (van Someren et al., 1994) i.e. designers were required to think aloud as

they were solving the problem – this may have affected designers’ performance in

detrimental manner, because problem-solving is a cognitive activity which requires

attention, and by talking out loud (attention on talking), it may distract and disturb the

flow of the problem-solving process.

Other studies examined individual differences in problem-solving with a slightly

different focus – styles of problem solving (as opposed to strategies used) and their

influence on the design process. For example, in Eisentraut’s (1999) empirical study,

fifteen engineering design students with different levels of study (varying from two to 14

semesters) and hence, different knowledge of design methodology, worked on three

problems – two computer simulated complex problems (non-design context) and the one

adaptive design problem. However, the participants were allowed five hours to finish the

allocated tasks, therefore, it was likely that participants’ concentration fluctuated, and

became fatigued near the end of the experiment. Like Kruger and Cross (2006), the

‘think aloud’ approach was also used, which may have affected participants’ design

8

performance, measured in terms of design quality, especially with such long experimental

period. A single case approach was used for data analysis and it was found individual

styles of problem solving determined the way in which designers organized their design

processes. It was observed that subjects exhibited stable problem solving behaviour (e.g.

amount of information gathered, speed of action). It was also noted that individuals who

worked on the three situations in the same way may have been successful in one situation,

but less successful in another, thus, Eisentraut concluded that the success in problem

solving depends on whether a style of problem solving meets the demands of the situation.

Eisentraut further suggested that diagnosing and training individual problem-solving

behaviour may play an important role in optimizing individual design processes and

therefore should be a part of design education

Similarly, Eisentraut and Gunther (1997) also empirically examined individual

styles of problem solving, but with a specific focus on their relation to representations in

the design process. Single case approach was chosen and analyses focused on the marks-

on-paper which were created and used by the designers (fifteen male engineering students

who worked on an adaptive design problem, as well as on two complex non-design

problems). The results suggested that the course of the design process in general and the

use of marks-on-paper in particular depended on an individual designer's style of problem

solving. However, participants’ background was not adequately described, and

furthermore, only four among fifteen cases were presented, therefore the generalizability

of the findings is questionable, and thus, results must interpreted with caution.

Nonetheless, Eisentraut and Gunther noted that drawings by hand and sketches turned out

to be important to the subjects during their design processes. It was also noted that more

than 70% of the documents produced in the experiments (illustration from the sketch in

principle to the sketch roughly to scale) cannot be created by a CAD-system. Eisentraut

and Gunther’s study yielded some important implications which are also relevant to the

9

present study, for example, as suggested, CAD-training should not replace drawing and

sketching in design education, and that the use of the ‘concreteness’ and ‘completeness’

(of sketches used by designers) as a tool in teaching the levels of abstraction in the design

process. Furthermore, because of the importance of sketches – a notion also agreed by

many other researchers (e.g. Goel, 1996; Goldschmidt, 2003) and supported by empirical

evidence (McGown, Green & Rogers, 1998; Schenk, 1991 Ullman, Wood & Craig, 1990),

it was suggested that CAD development should be directed towards systems that enable

the designer not only to create formal technical drawings, but also to easily make sketches

of different degrees of abstraction.

Although empirically studies have been conducted, more research is still needed to

develop a more comprehensive model that explains on individual differences in problem

solving in design more thoroughly.

1.1.2.2. Other factors affecting design performances

In addition to factors including expertise and individual difference in problem

solving, there are other factors that also affect the design process in the context of problem

solving. From the organizational and industrial psychology point of view, the physical

environment, for example work station design (Grandjean, Hunting & Pidermann, 1983)

and lighting (Stammerjohn, Smith & Cohen, 1981) in which an individual works in is an

important determinant in work (design) performance – hence, impacts on the design

process. The social context in which designers works in, for example individually or in a

group, also influences the design process (e.g. Goldschmidt, 1995; Gross et al., 1998;

Lahti, Seitamaa-Hakkarainen & Hakkarainen, 2004; Launis, Vuori & Lehtela, 1996;

Stempfle & Badke-Schaub, 2002). Time and resource constraints in design situations are

as recognized as major factors that shape the design process, in terms of designers’

10

motivation and decision making, and on design outcome (Burns & Vicente, 2000; Savage,

1998). In addition, research has also shown that the use of different design tools may

influence a designer’s attitude towards the design process. For example, Hanna and

Barber (2001) found that in addition to the difference in the design approach when using

computer-aided design (CAD) tools only as opposed to using both sketching and

computer-aided design tools, students’ attitudes towards the design process changed after

using CAD. Furthermore, different (combination of) design tools used have been shown

to affect the design process, namely, design activities (Holt, 1991; Lahti et al., 2004;

Shniderman et al., 2006) and design performance (Black, 1990; Jonson, 2005; Sachse,

Leinert & Hacker, 2001).

Overall, much research has been done on the design process and with so many

pieces of information grouped differently by different researchers in various disciplines,

one must bear in mind that to date, although attempts have been made to integrate existing

research into a more comprehensive model of the design process (e.g. Elias, 2005 –

cognitive model), there has not yet been a model that describes and explains the design

process as a whole in a systematic way. Before such model can be developed and tested

empirically, further understanding of the design process, particularly the designer within it

and his/her interaction with artifacts (design tools/mediums), is required.

1.1. Human-Computer-Interaction (HCI) and Design

Despite of some research in the design discipline, the study of designers’

interaction with design tools/media is still at its infancy and particularly critical with the

rapid technological development of design support tools. In order to examine such

relationships, it is important to understand how design research has become an emerging

branch within the domain of human-computer interaction, and especially, how the design

11

discipline was shaped by the emergence of new technologies and the corresponding

changes in designer capabilities and needs.

Human-computer interaction is a multidisciplinary field that incorporates research

and theories from computer science, psychology, engineering, anthropology, education,

design, mathematics, and even physics. Researcher have identified the fundamental aims

in HCI as to take a scientist-practitioner stance, to understand humans and their

interactions with computers, and the roles and impact of computers on humans; and

furthermore, to explore design techniques and methodologies with descriptive and/or

prescriptive approaches, as well as usability testing and evaluation; and thus, to design

effective products that are suitable for a wide range of human requirements and capacities

(Benyon, et al., 1993; Czerwinski, 2003; May, 2001)

Much of the research in HCI to date is concerned with design guidelines and

standards (e.g. Molich & Nielsen, 1990; Nielsen, 1994); effective design for diverse users

(e.g. Czaja, 200; Joiner, et al., 1998; Sear, et al., 2001); understanding of psychological

phenomena (cognition, perception, and action) in humans’ interaction with computers (e.g.

Reason, 1990; Neisser, 1976; Pancer, George, & Gebotys, 1992); as well as the study of

design methodologies including design tools techniques (e.g. Beaudouin-Lafon &

Mackay, 2003); and usability studies (e.g. Dumas & Redish, 1999, Lewis, 1998) .

However, in comparison to such research areas, it appears that the importance of research

on the design process has been largely neglected within the HCI discipline.

With the rapid technological development in design tools such as computer-aided

design (CAD), however, it was not until the last decade that design research became an

increasing interest to researchers in the HCI field. In 1991, a whole issue in the Journal of

Human-Computer Interaction was dedicated to the topic of design rationale i.e. the

reasons and the reasoning processes behind the design and specification of artifacts

(products), with a particular orientation to particular domain, namely computer and

12

information systems in the issue (e.g. Carroll & Rosson, 1991; Maclean, et al., 1991). A

few years later, in 1994, another issue of the journal was dedicated to design research, this

time with a focus on design context as it could open up new approaches and ways of

thinking for designers (Moran, 1994). However, it was only after more than a decade, in

2006, that another special issue on the topic of design appeared again in the Human-

Computer Interaction (HCI) journal. It was pointed out by Carroll (2006) that design is

appropriately one of the core topics of the HCI journal; however, as commented, there has

always been a lack of papers on design in the journal. Carroll also argued strongly that

design is the most fundamental topic in human-computer interaction because, “whatever

understanding we may achieve of human capabilities and preferences, the social and

cultural construction of activity, and the gamut of technological possibilities and

constraints, we still have to put it together in order to have any effect on the world. The

artifacts we design – infrastructures, systems and applications, policies and curricula – are

the most important results of our endeavors”, (p. 2). In other words, it is the actual

designers who design artifacts; and artifacts are all a result/product of design – by a

designer. Furthermore, even with all the required resources (information and tools)

available, whether a product is successful, in terms of quality and usability, is highly

dependent on the designer him/herself (and the design team); hence – the importance of

studying the designer-artifacts (i.e. design tools/media) interaction within the design

process.

1.2. Prototypes, Prototyping and Prototyping tools

The use of prototypes and different prototyping tools has been seen as playing an

important role in design cognition (in terms of problem solving, judgment and decision

making, and reasoning) in the design process. While some HCI handbooks has a strong

basis on empirical research (e.g. Helander, 1988; Jacko & Sears, 2003; van der Veer &

13

Mulder, 1988), many other HCI books, especially on (web) interface design are mainly

based on the design industry and practitioners’ experience (e.g. Brink, Gergle & Wood,

2002; Fowler & Standwick, 2004; Laurel & Mountford, 1990); and thus, guidelines on

the use of design tools in prototyping such as paper and pen, and computer software in the

design process are often prescriptive and do not necessary explain the underlying

mechanisms for using such tools. Therefore, in conjunction with HCI research, the design

literature was also turned to for a richer source of empirical studies on prototyping tools.

This was necessary in order to have a better understanding of the big picture of designer-

artifact interaction – and more specifically, on the use of different design tools for

supporting various stages in the design process (e.g. prototyping tools used during the

early phases of the design process).

Prototypes can be seen as a design tool and a prototype may be defined differently

in different design disciplines (e.g. in engineering, architecture and fashion design). In the

HCI context, in which this study focuses on, a prototype is defined and described, by

Beaudouin-Lafon and Mackay (2003), as a “concrete representation of part or all of an

interactive system. A prototype is a tangible artifact, not an abstract description that

requires interpretation. Designers, as well as managers, developers, customers, and end

users, can use these artifacts to envision and reflect on the final system” (p.1007). Not

only is prototyping a design process itself, prototyping is a also a social process, as

Helander (1988) discussed, and will provide the means by which that designer can make

his or her ideas explicit to the community and by which these ideas can be evaluated.

Studies have found that the use of prototypes can help refine a product’s functional

requirements very early in the design process (e.g. Yang, 2005).

While research has shown the benefits and problems associated with using

prototypes (see Alavi, 1984), prototypes have long been used in practical settings. In

Houde and Hilll’s (1997) study, it was suggested that prototype increases creativity, allow

14

early evaluation of design ideas help designers think through and solve design problems,

and support communication within multidisciplinary design teams. Also, a few studies

suggested that a development process based on prototyping during the early stages of a

major design project can significantly reduce the cost of the final software, typically found

in the early development phase. For example, in an early study, Boehm, Gray, and

Seewaldt (1984) found in a comparative experiment using computer science students that

a prototyping based design approach required approximately 45% less development time

than a more structured approach. Furthermore, the use of prototype is important as gives

an early picture of a design concept which can range from simplistic two-dimensional

sketches that represent design thinking and ideas (Ullman et al., 1990; Goel, 1995; Suwa

and Tversky, 1997) to more advance and sophisticated 3-dimensional mock-ups that are

hardly distinguishable from the real manufactured product. Hance, prototypes can be

different in nature and is dependent on the phase of design. Ullman (2003) described four

categories of prototypes based on their function and stage in product development: 1)

proof-of-concept prototype – used in the initial stages of design to better understand what

approach to take in designing a product; 2) proof-of-product prototype – used later in

design to clarify a design’s physical aspects and production feasibility; 3) proof-of-process

prototype – which shows that the desired product could be successfully produced with the

method and materials used; and 4) proof-of-production prototype – which demonstrates

that the full manufacturing process is effective and operational.

Furthermore, according to Beaudouin-Lafon and Mackay (2003), prototypes and

prototyping techniques can be analyzed in four dimensions: 1) representation – the form

of the prototype (e.g. sets of paper sketches or computer simulations); 2) precision – the

level of detail at which the prototype is to be evaluated (e.g. informal and rough or formal

and highly polished); 3) interactivity – the extent to which the user can accurately interact

with the prototype (e.g. watch only or fully interactive); and 4) evolution – expected life

15

cycle of the prototype (e.g. throw away or iterative). For the purpose of the present study,

the first two dimensions (representation and precision) of prototypes are more relevant and

will be mainly discussed.

As noted, prototypes aid designers in creating concrete representations of design

ideas and clarifying specific design directions, and also give designers (and others) an

early glimpse into how the new system will look and feel (Beudouin-Lafon & Mackay,

2001). Moreover, prototypes serve different purposes, and thus take different forms, for

example, series of quick rough sketches and a detailed computer simulation, all of which

are of help to the designer in different ways. Hence, Houde and Hilll (1997) suggested

that the designer must consider the purpose of the prototype at each stage of the design

process and choose the representation that is best suited to the current design situation.

Different levels of representations of prototypes can be achieved – for example, during the

early stages in the software design process, paper prototype such as paper sketches are

often used, as they can be created quickly, at a low cost, in comparison to using software

prototypes such as interactive interfaces usually used in later stages in the design process,

which is usually higher in cost and may require skilled programmers to implement

advanced interaction and/or visualization techniques or to meet tight constraints

(Beaudouin-Lafon & Mackay, 2003).

Moreover, prototypes vary in terms of precision – the relevance of details (content)

with respect to the purpose of the prototype (Beaudouin-Lafon & Mackay, 2003). For

example, when sketching a dialogue box (e.g. online (web) forms that requires user

interaction by input), the designer specifies elements size, positions of each field, and titles

of each label. The concept of precision is similar to the terms low-fidelity and high fidelity

of prototypes used in the literature to refer to the degree of relationship to the final system.

In addition to the theoretical and methodological research on prototypes, it would be

beneficial for future research to also explore designers’ interaction with different levels of

16

representation, (e.g. media, precision and formality) of prototypes, and whether such

factors may influence design performance and design outcome.

There are different techniques for prototyping. Some examples of prototyping

techniques include mock-ups, Wizard-of-Oz, storyboard (see Beudouin-Lafon & Mackay,

2001). It can be noted that such techniques uses different types of tools – for example,

paper and pencil (e.g. for prototyping a web interface), cardboard mock-ups (scaled three-

dimensional prototype) for a future building, video prototyping for story boards of a web

site. On the other hand, other prototyping tools involve software – for example,

Micromedia Director, Adobe Illustrator, Adobe Photoshop, Microsoft Publisher to create

non-interactive stimulations; Visual Basic.Net, and other programming languages such as

Java, C++, C#, to create interactive stimulations (e.g. user interface). However, as with

any tools in general, every tool has its strength and weaknesses, even in the various design

(prototyping) tools that are commonly used. Although one can say that design tools serve

to aid designers in the design process, however, it is also important that a designer uses, in

addition to the appropriate level of representation, appropriate design tools at different

stages in the design process and nature of the design task. The combination of useful

design tools employed at appropriate stages helps optimize the design process in terms of

efficiency, design quality and outcome. Hence, in the following section, the two main

tools/media used for prototyping (designing) and their effects on the designers during the

design process are discussed and compared.

1.4.1. Traditional Design tools for Prototyping

1.4.1.1. Paper and Pen(cil)

Paper and pencil are one the most commonly recommended set of tools to use in

the design process (e.g. Beaudouin-Lafon & Mackay, 2001; Brink, et al, 2002; Jacko &

Sears, 2003). Not only are paper and pencil inexpensive in comparison to other tools such

17

as computers and software, it also provides a medium for designers to freely explore

different design ideas through sketching (Newman et al, 2003). According to many,

sketching is one of the most important design activities as it facilitates reasoning (Akin,

1986; Goel, 1996; Goldschmidt, 1991, 1994; 2003; Tversky, 1999), problem-solving

(Romer, Leinert, &Sachse, 2000), memory and thinking (see discussion by Scaife &

Rogers, 1996), creativity (e.g. Goel, 1996, Goldschmidt, 2003; Hanna & Barber, 2001;

Kavakli, Scrivener, & Ball, 1998; Nakakoji, Tanaka, Fallman, 2006; Schenk, 1991;

Verstijnen, et al, 1998) and communication (Beudouin-Lafon & Mackay, 2001; Schenk,

1991; Van der Lugt, 2005); all of which are important aspects of design that affect design

outcome (see review on drawing and the design process in Purcell & Gero, 1998). In the

context of Web site design, Beudouin-Lafon & Mackay (2001) also agreed on the

importance of sketching on paper as it assists exploration of layout, content and aesthetics

of a design, and in addition, it is less likely that sketching on paper would constrain

designers’ thinking compared to using development environments, software and

programming languages. Such usefulness of paper and pencil as a design tool to support

sketching is reflected in a recent Web design practice study, in which Newman et al

(2003) found that paper prototypes (rougher, hand-drawn representation of the final

product) are frequently employed during the design process, especially at the early stages.

Beudouin-Lafon and Mackay (2001) argued, however, that although paper

prototypes have many advantages, they are not the answer to everything. In some

situations, paper prototypes are insufficient to fully evaluate a particular design idea. For

example, where interface requiring rapid feedback to users (e.g. Web site design), or

complex, dynamic visualization (e.g. engineering and architectural design usually require

software prototypes, created by using computer-aided design tools.

1.4.1.2. Computer-Aided Design (CAD) Tools

18

The technological advances which improved accessibility and lowered cost of the

use of computers was one of the major forces for the dramatic increase in research on

computer-aided design (CAD) tools in the past two decades. This had a major impact on

design representation and the design process, for example, influencing tool choices at

different stages, and faster and more complex design creation and evaluation in a dynamic

way. CAD tools assist architects, engineers and other design professionals in their design

activities, generally during the later stages in the design process, to create accurate

precision drawings and technical illustrations, hence, increase in fidelity and formality of

prototypes (a closer and more precise representation of the final design). As CAD enables

version control, editing and easy distribution (e.g. emailing) of designs, which has its

advantages over using paper and pen, the general argument for using CAD is to improve

design productivity, lower product development cost and shorten design cycle (Jacko, &

Sears, 2003). Additionally, Lahti, Seitamaa-Hakkarainen and Hakkarainen (2004) found

that computer-supported environment helped facilitate collaboration between individuals

in a group. In contrast, it is often claimed that the CAD system itself interferes with the

design work (Landauer, 1996; Luczak & Springer, 1997), especially during

conceptualization work (Lawson & Loke, 1997), as using CAD requires additional

cognitive workload, for example, skills to manipulate the computer objects on the screen

with the input devices and knowledge of commands to operate the system and input

drawing data (Khalid, 2001). However, Hamade, Artail and Jaber, (2005) found that

design behavior and performance using CAD tools can be increased through effective

CAD training, especially in novice users. Such findings may suggest that manipulation

and operation of the system (procedural knowledge) becomes automated, which in turn

may help decrease mental workload. Nonetheless, research on CAD tools has shown the

usefulness of such tools.

19

Despite of the opportunities and challenges that these CAD tools offer, their impact

on the design process has not been fully examined through empirical experimentation.

Nonetheless, it can be seen from a wider literature search, that theoretical and/or

descriptive work on CAD (e.g. Grekas & Frangopoulos, 2000, Tovey, 1997) is beginning

to bloom and mature, with an increasing number of empirically driven studies on user-

interaction with CAD tools (e.g. Hamade, Artail & Jaber, 2005; Khalid, 2001, Sachse,

Leinert, & Hacker, 2001).

1.4.1.3. Combination of Paper and Pen, and CAD

It is often that both pen and paper, and CAD tools are used in conjunction, and are

viewed as important tools in the design process as their strengths and weaknesses are

intertwined (Newman et al, 2003). According to Plimmer and Apperley (2001), designers

traditionally engage in sketching on paper and pencil, and ultimately move to computer-

based tools. Researchers have become increasingly interested in the non-traditional design

process in the digital age, and recent studies have compared the use of CAD tools and

sketching (drawing) on paper and pencil in different ways, including designers’ behaviors,

design outcome, design quality, and the design process, by using common methodologies

such as design protocol analysis – traditionally used studying human information

processing (problem solving) mechanisms (Newell, 1968). For example, Bilda, Gero, and

Purcell (2006) conducted think-aloud experiments with six expert architects (graduate

students with similar design knowledge and experience) to examine whether sketching is

essential for conceptual designing, based on a protocol analysis. Bilda et al. found no

significant difference between sketching (using paper and pencil) and not sketching (using

CAD) based on design outcome, cognitive activity and idea links, which led Bilda et al. to

conclude that sketching is not an essential activity for expert architects in the early phases

of conceptual designing. However, with extremely small number of subjects used in the

20

study, the generalizability of the results to professional architectural design situations is

questionable. Furthermore, thinking-aloud during the experiment may have disrupted the

natural flow of design activities, thus, results may have been polarized (hence, the number

of subjects).

In an experimental study with more subjects (seventy four undergraduates),

Sachse, Leinert and Hacker (2001), too, examined the use of computer and sketches

during design and found that sketching before and/or during the computer-aided design

resulted in a significantly reduced time taken for more complex task analyzed, despite of

the additional sketching time – this can be explained by the reduction of the number of

processing steps needed when sketching was allowed. Sachse et al.’s findings further

supported the perceived utility of sketching before and/or during the usage of CAD tools

(e.g. Newman et al., 2003). Bilda and Demirkan (2003), on the other hand, took a slightly

different stance, and aimed at gaining an insight on designers’ cognitive process while

sketching in traditional verses digital media, by using retrospective protocol analysis.

Their results showed that traditional media had advantages over the digital media, such as

supporting the perception of visual-spatial features, and organizational aspects of the

design, generation of alternative solutions and better conception of the design problem.

However, one remains concerned that the use of protocol analysis may not truly or

fully represent information process in the design process, as it is highly task-specific, and

may interfere with visual thinking, and distort the real design process (e.g. see discussions

in Ericsson & Simon, 1984; Llyod, Lawson & Scott, 1995). Despite of critiques on

protocol analysis, such studies yield valuable information about designers’ thought process

that is for the development and improvement of computer aids in (architectural) design to

support the conceptual phase of the design process.

Overall, although design education and design handbooks echoes the usefulness of

sketching on paper during early stages of design along with CAD tools (for in later

21

stages), they are often prescriptive in nature on the use of various design tools and derive

from assumptions and experience of the authors, and many of them do not necessarily

refer to such empirical evidence – which reflects the lack of conclusive research on the use

of (a combination of) different design tools/ mediums and their effects on the design

process. Moreover, it must be noted that although with their advantages and

disadvantages, the use and effectiveness of various design tools are also dependent factors

such as the design task (e.g. solution finding, development or refinement), design

complexity (e.g. simple or complex), and design scope (e.g. a whole Web site or one Web

page), as well as the designers’ very own personal preference and practice.

1.4.1.4. Paper prototypes verses Digital prototypes

In addition to the underlying effects of working with various design tools, one

closely related factor that also influences design outcome, is design representation and its

relative formality (on different media), created by using different design tools/media (e.g.

paper and pencil, and computer-based).

According to Brinck et al. (2002), prototypes can vary from very course, fuzzy

layouts of general page requirements done on paper (low-fidelity), to precise, highly

elaborate, refined and polished digital versions of the web site (high-fidelity). This range

provides the designer with levels of refinement and formality useful for testing and

exploring varying details of a given design. For example, paper prototypes maybe used to

gather feedbacks on basic functionality or visual layout in a quick and efficient way. In

contrast, for example in Web design, digital prototypes may be the only accurate way to

explore an online environment on issues such as colour and contrast.

There are some interesting differences among the perceptions conveyed via paper

and digital mockups. According to Brinck et al. (2002), it is better to use paper mockups

early in the design cycle for one major reason: clients tend to view paper mockups as a

22

conceptual rather than a finished product; and that the strength of this cannot be

overlooked. In addition, anecdotal evidence suggests that one of the biggest problems

encountered with digital mockups is that clients tend to view them as final, unchangeable

products, as opposed to paper mockups which are perceived as less polished and more

conceptual – also voiced by design professionals interviewed by Newman, et al. (2003) in

their study. Therefore, as these authors suggest, it can be seen that paper mockups tend to

provoke more comments; where clients and users could become more open in their

suggestions for change and the perceived inadequacies of the design (Beaudouin-Lafon &

Mackay, 2003) – in this way, paper mockups can be viewed as an agent in generating

useful feedback on boarder design issues. By contrast, clients viewing digital mockups

tend to focus on the details of the layout and on issues such as font choice, exact spacing,

label names, or colours; yet, it is often too soon to receive such advice in early stage in

design (Brink, et al., 2002; Newman et al., 2003). Thus, digital mockups tend to be

reserved for later in the prototyping process, nearer the finishing of the design. Hence,

mixing both types of prototypes (low-fidelity and high-fidelity) at different times can be

useful for revealing different types of problems (Brinck et al., 2002). This has an

implication on the importance of design representation – in particular, formality level of

designs (from informal, rough sketches to formal, computer-rendered representation of

design) used at different stages during the design process.

Again, the descriptive and prescriptive use of paper prototypes and digital

prototypes has been based mostly on anecdotal evidence from experts and professionals in

the design field. Hence, more understanding of cognitive and perceptual processes when

interacting with different types of prototypes may help the development of prototyping

tools and techniques, and moreover, the understanding of design representation and its

effects.

23

A deeper question yet to be answered, which arises, is whether designers will

perceive designs differently when presented on paper compared digital media. In other

words, whether the difference in media for representation would affect design

performance and outcome (e.g. error detection and design quality). Another factor that

further complicates such question is the introduction of design tools with sketch-based

interfaces – a computer-aided tool with a pen and paper feel (i.e. ‘digital ink’).

1.3. Current Trend in Design Tools Research: Informal Sketch-based interface

Along with strong anecdotal and empirical evidence on the effectiveness of

traditional design tools (pen and paper, and computer-based tools), the improvements and

popularity of tablet PC (pen-based input) also contributed to the significant increase in the

demand and usage of such sketch-based interfaces in diverse software applications (Pomm

& Werlen, 2004). This, in turn, has resulted in a major trend of research on sketching-

based design tools with an emphasis on the advantages of sketch-based interfaces, as these

“bridge the gap [between paper and pen, and computer-based design tools] by providing a

design-friendly computer-supported sketching environment and add a useful new

dimension to the design process” (p. 1337, Plimmer & Apperly, 2004). As opposed to

“formal” design tools such as CAD and programming languages like Java, C++ and Visual

Basic.Net, sketch-based design tools have been regarded as “informal” design tools

(Landay & Myers, 2001) as they support the ambiguity and informality of sketching

(Chung, Mirica, & Plimmer, 2005; Newman et al, 2003; Plimmer & Grundy, 2005). In

the following sections, related research on computer-supported sketch system

requirements is briefly described, followed by examples of current research on design,

implementation and (further) development of such systems; and finally, ‘beautification’ –

24

an important aspect (and functionality) of computer supported sketch-based tools – and its

related research, is further discussed.

So far, overall, research has shown strong evidence supporting the notion that

sketching (and its naturalness) plays an important role in design cognition and reasoning,

and that designers typically work with paper and pencil first, then later, transfer the paper

design onto formal computer-based tools (e.g. CAD software) – this formed the basis for

research on the design and development of computer-based sketching interfaces to support

the design process. For example, based on the importance of sketching in early stages of

design, Dickinson et al. (2005) discussed the potential of pen-tablet interface in

mechanical CAD modeling. Similarly, Plimmer and Apperly (2001) stressed on the

importance of drawing and how current computer interfaces interferes with the sketching

process; and further proposed requirements, in terms of functionality and usability, for an

ideal computer-aided sketch system to capture preliminary design – i.e. an interface that

facilitate direct, rapid drawing, while provide more functionality than paper or whiteboard.

Based on previous experimental research, Mulet and Vidal (2006) also suggested similar

functional requirements (i.e. sketch-supported interface) for improving on existing

computer-based design support tools. On the other hand, Fitzmaurice et al. (1999)

discussed design issues and requirements for rotating user interface (RUIs) to support

artwork orientation, with a more specific focus on an integral part of drawing – the

rotation of the piece of paper.

Many different types of ‘sketch systems’ have been implemented and developed as

a result of various studies highlighting the benefits of building computer-supported sketch

tools, and the associated design issues and requirements. Sketch systems that have been

developed so far have different capacities ranging from two-dimensional pen-input to

three-dimensional sketching and manipulation; as well as processing of sketched objects,

for example, by (domain-specific) sketch recognition and beautification.

25

1.3.2. 2-Dimensional (2-D) sketch-based systems

Many sketch-based systems have been developed to support sketching activities in

the design process, with earlier examples of systems that offered interaction in a 2-D

environment, including Saund and Moran’s (1994) “PerSketch” program and Kramer’s

(1995) “Architect’s Electronic Sketchboard”. Like Plimmer and Apperley (2003) had

proposed on design tools that “bridge the gap” between sketches on paper and pencil and

its digitized version in CAD system when near completion, one of the many current

systems that fitted this purpose was developed and implemented by Alvardo and Davis

(2006), called “magic paper”, with an emphasis on the mechanical engineering domain.

In addition to supporting natural sketching on paper, such system also recognizes and

interprets the sketched objects as the user draws, by resolving ambiguities in the 2-D

sketch; hence, by using such method of recognition, interference with the design process is

minimized. Similarly, a number of other sketch-based systems also involve two-

dimensional sketching environment which typically support manipulation and recognition

of sketched objects. For example, for supporting planning of a military course of action,

Forbus, Ferguson and Usher (2001) designed and developed a simple sketch system with

functionalities including recognition of symbols and supports interaction behaviours, also,

in a 2-D environment. A more sophisticated sketch-based computer system, and

frequently cited, is the Electronic Cocktail Napkin (Gross, 1996; Gross & Do, 1996). It

supports conceptual design through sketch recognition, interpretation and management of

2-D architectural drawings, as well as multiuser collaboration, which serves as an interface

for knowledge-based critiquing, simulation, and information retrieval (all especially

valuable in architectural design).

Besides 2-D domain-specific sketch tools, recent systems have been designed and

implemented in ways in which they become extensible and customizable to cater for

26

additional domain support and functionalities. For example, Kara and Stahovich (2004)

developed Sim-U-Sketch – again, a 2-D sketch-based interface, for Matlab’s Simulink

software package. Sim-U-Sketch enables a user to construct functional models by simply

drawing sketches on a computer screen, and also, interact with their sketches in real time

to modify and manipulate existing objects and add new ones. Moreover, the domain-

independent, trainable symbol recognizer levers the processing of unique drawing styles of

different designers, and thus, has the potential to support new domains such as architecture

drawings, in the system.

1.5.2. 3-Dimensional (3-D) Sketch-based systems

It is often, particularly in engineering and architectural design, that computer-aided

tools are used at a later stage in design to create three-dimensional shapes and objects (van

Dick & Mayer, 1997), so designers can explore the three-dimensional entity of objects.

Especially with arguments that commercial CAD systems for the manipulation of digital

models are still not adequately suited to support the design process (Cheutet et al., 2005) –

especially conceptual thinking (Oh, Stuerzlinger & Danahy, 2006), 2-D sketching-based

systems are now evolving into systems that offer design environment in a 3-D manner, and

support additional functionalities such as 3-D manipulation of sketched objects.

In 2005, Cheutet and his colleagues addressed the way designers express an object

shape in 2-D sketches through character lines and how these lines form a basis for

sketching shapes in 3-D; hence, the authors proposed a free-form deformation features

modeling method to enable 3-D objects handling extended from 2-D sketches.

Turner, Chapman, and Penn (2000) looked at developing a broader 3-D system

called Stilton, in which the user is presented with a 2-D drawing plane and drawings may

be reconstructed to form solid objects, and thus, allowing a user to navigate 3-D model in

different perspectives, such as panoramic photographs (i.e. “sketching out volume around

27

the user”). Turner et al. suggested that such feature of Stilton can greatly assist with some

aspects of familiarization and qualitative assessment of sites – particularly useful in

construction and architectural design. However, no end-user trials has yet been

undertaken to suggest the usefulness of such tool.

Although with the increasing number of systems with 3D sketching environment,

such systems require more sophisticated design and implementation to support better

usability and accessibility; thus, more work is still needed to address technical issues such

as visualization, mathematical and programming issues, as well as recognition and

interpretation of 3D sketches.

1.5.3. On Improving Computer-Supported sketch tools

While many researchers have focused on developing new types of computer-

supported sketch tools, whether 2-D or 3D, domain-specific or domain-independent,

knowledge-based or gesture-based, with different design support capabilities etc, others

have been motivated by the critical need to improve on the design, functionalities and

implementation techniques, and hence, effectiveness (and usability), of such systems.

Examples of current improvement work include, Arvo and Novins’ (2005) success on

improving ways of manipulating sketches by preserving sketch appearance, while Nealen

(2005), examined editing aspects of sketch (i.e. selection and moving a handle) in which

complex shape modeling tasks were greatly simplified. In his sketch-based system

(“Sketch Now”), Jonson (2005) attempted to stimulate the feel of using different kinds of

paper; however, as predicted, many difficult issues had to be addressed. Oh (2004) made

further developments (improvements) based on the Electronic Cocktail Napkin system

(Gross, 1996; Gross & Do, 1996), resulting a new sketch-based tool – Design Evaluator –

to support architectural design reasoning, but in a three-dimensional manner. Also in the

3-D sketching scene, Oh, Stuerzlinger and Danahy (2006) took a better approach towards

28

3-D conceptual design systems and developed SESAME (Sketch, Extrude, Sculpt, and

Manipulate Easily). Furthermore, the comparison of SESAME with a conventional CAD

package demonstrated the effectiveness of SESAME. Dijk and Mayer (1997) argued that

CAD systems lack flexibility to modify 3-D design objects – a major obstacle for effective

computer support during conceptual modeling; therefore, a new algorithm was developed

to allow faster and easier conversion from 2-D to 3-D model space, in which the 3D

surfaces can also be directly modified by sketching. Likewise, also with a mathematical

approach, Liu, Tang and Joneja (2005) focused more on the technical aspects and

presented an alternative method for sketch-based free-form shape modeling, and

demonstrated its usefulness and effectiveness by testing the technique with two design

applications.

On the whole, one can observe that the future of computer-supported “natural

design” can be hopeful with the on-going, rapid development and expansion of existing

digital-sketching systems and its related functionalities.

1.5.4. Bridging the Gap: A closer look at ‘Beautification’ (‘Formalization’)

Since nearly two decades ago, Thomassen, Teulings, and Schomaker (1988) had

already suggested that it could be an advantage if a single system could be designed that is

capable of dealing with different forms of graphemic, symbolic, and graphic output: i.e. a

system that would not only be able to read handwritten letters, digits, words, and text-

editing comments, but would also read non-alphanumeric characters and symbols, that

would interpret formulae and roughly sketched graphics, and that would render these in a

neat, orderly fashion, properly grouped and aligned. Maarse, Schomaker and Teulings

(1988) went on to further to give a hypothetical example of the automatic processing of

combined graphic and cursively written material entered on a graphic tablet. Lin and Pun

(1978) also proposed a similar interactive system for the interpretation of hand-sketched

29

line figures such as flow charts and logic circuit diagrams, in which hand-drawn diagrams

and hand-writing, are converted into computerized, standardized diagrams and hand-

writing.

Future development recommended over twenty years ago is now happening with

the emerging trend in sketch-based interfaces for design, and such computer-supported

sketch systems are now becoming more accessible, whilst being further explored and

improved. Not only do current computer-supported sketch tools bridge the gap between

paper and pen and computer-aided design tools, to support natural ways of designing (i.e.

sketching on paper), whilst providing the advantages of computer-based tools (e.g. design

transformation, editing support and version control); they also offer an interface for

processing sketched objects in a useful manner (e.g. Forbus, Ferguson & Usher, 2001;

Gross, 1996; Oh, 2004).

One of the fundamental approaches toward sketch processing is the recognition of

sketched shapes and objects, for example as predefined components – e.g. as Tang (2005)

noted, in the context of Web interface design, hand-written letters can be recognized as

“labels” and a hand-drawn circle can be recognized as a radio button (where as in the

mechanical engineering domain, a circle may be recognized as a wheel of an automobile).

Moreover, the increasing demand and research on digital sketch tools further points out to

the need for better recognition of sketched objects and hand-writing (e.g. Agar & Novins,

2003; Bo Yu, 2003, Rubine, 1991), to cater for other useful functionalities, especially for

sketch transformation and rendering (Bolz, 1993; Chung, Mirica & Plimmer, 2005; Hong

& Landay, 2006; Hse & Newton, 2005; Landay, 1996; Lin & Pun, 1978; Maarse et al.,

1988; Newman et al., 2003; Pomm & Werlen, 2004; Plimmer & Grundy, 2005;

Schweikardt & Gross, 2000; Shesh & Chen, 2004; Wang, Sun & Plimmer, 2005).

Thus, one of the major challenges embedded in the development of computer-

supported sketch-based design tools is the need to “beautify” sketched content (i.e.

30

beautification), both during sketching-based input and during conversion to computer-

rendered form (Plimmer & Apperley, 2004; Plimmer and Grundy, 2005).

1.5.4.1. ‘Beautification’ versus ‘Formality’

The term ‘beautification’ was originally used in urban planning, and now, in the

context of computer-supported sketch tools, beautification generally refers to the process

of tidying up a hand-drawn diagram (Plimmer & Grundy, 2005). In the wider context of

the design process, designers typically sketch on paper and later transfer the design onto

CAD tools, which can be viewed as a beautification process; however, it was criticized

that such transferring process from paper to the computer is error-prone and unproductive

(Tang, 2005; Young, 2005). In contrast, in terms of computer-supported sketch-based

design tools (with a fundamental goal to bridge the gap), such beautification process is

downsized into requiring only one design medium, and at the same time, eliminating the

transferring process – i.e. rough, untidy, sketched input can be transformed to appear

neater, tidier and more formal, by using a single design tool. The idea behind

beautification is similar to text recognition where hand-written input is recognized as text

form and could be transformed into computer fonts; hence, with sketched objects, hand-

drawn input is recognized and rendered into a higher level(s) of representation (e.g.

morphing of hand-written characters into fonts, straightening of lines, 3-D rendering from

2-D objects) – better referred to as ‘formality’ in the context of this study.

‘Formality’ can be described as the outcome of beautification, and incorporates

both concepts of precision and fidelity (as described in the section on prototypes).

Formality is defined, in this study, as the level of tidiness and professionalism conveyed in

the appearance of a design (prototype). This definition of formality is based on previous

studies on website design practices in which web site designers were observed to design

sites at different levels of refinement – site map, storyboard, and individual page – and that

31

designers sketch at all levels during the early stages of design, ranging from rough

sketches to tidied-up diagrams for (re)presentation to clients, to computer-rendered

designs (Lin et al., 2000; Newman et al. 2003). This concept can be illustrated in the

context of web interface design: when sketching a text box, the designer specifies the

element size (rough and uneven or same), position of each field (roughly spread or

aligned), and titles for each label and shape of each field (neatness of the hand-writing and

lines) all of which constitutes the overall appearance (tidiness/neatness) of a design. A

low-formality design would appear rough and sketchy (e.g. wiggly shape of a object), with

imprecision in the design (e.g. elements and labels unaligned) and would not look like the

final version of the design (low-fidelity); where as, a high-formality design would appear

tidy and professional (e.g. with straight smooth lines, alignment of labels and elements),

and with precision (e.g. location of elements) and would closely resemble the final version

of the design (high-fidelity). In addition to precision and fidelity, the definition of

formality also includes aspects of legibility and readability. According to Watzman’s

definition (2003), readability refers to the ability to find what you need on the page while

legibility refers to the ability to read it when you get there. Furthermore, topography could

make a design look more or less formal, by being more or less readable and legible.

Legibility is determined by many characteristics including typeface; letter spacing, word

spacing, line spacing; justified vs ragged columns; movement; colour; viewing

environment etc; thus, in the context of this study, a low formality design consist of untidy

hand-writing, as opposed to standardized, rendered text (e.g. fonts) in a high formality

design. As elements in the design can be tidied-up, based on principles of Gestalt

psychology (Koffka, 1935) of perceptual organization (e.g. grouping, proximity, similarity

among others), Brink et al. (2002) suggested that it can help an individual towards a

higher-level of comprehension of the display; hence, the increase in legibility and

readability – i.e. formality.

32

Overall, the level of formality of a design’s appearance is affected by the level

(extent) of beautification applied to the original design. In other words, one can increase

the formality of a design through beautification, so the beautified design appears tidier,

neater and comparatively more aesthetically pleasing (e.g. aligned elements and

straightened lines) than the non-beautified, original hand-drawn design.

In summary, beautification is a process, and formality is the outcome of

beautification; both of which are important concepts in computer-supported sketch-based

design tools.

1.5.4.2. Practicality of Beautification in the design process

Although the practicality of beautification in sketch-based interfaces is still yet to

be scientifically assessed in professional design settings, based on qualitative and

anecdotal evidence from design industry professionals, one can argue that beautification in

a computer-supported sketch-based environment can be useful.

A typical design situation, suggested by several authors (e.g. Brink et al., 2002;

Newman et al., 2003), is during the early stages in the design process of project

development when clients are involved – particularly, the presentation of ideas and

conceptual design to clients. Brinck, Gergle, Wood, (2002) pointed out that paper mock-

ups and/or prototyping may elicit clients’ negative responses such as “unprofessional”

look and feel of early design examples, and also, because early designs (prototypes) are

deliberately incomplete, clients can sometimes be distracted by the fact that not every

project requirements have been fully implemented, and may be led to conclude that

important elements have been forgotten. With this in mind, designs for client presentation

should appear processional, informal, but not too rough and sketchy.

In addition, the fact that designers do want clients to feel that a design finished, and

are reluctant to show their sketches (Newman et al., 2003), beautification can play an

33

important role in situations involving client presentation (as well as group collaboration).

For example, it can help tailor the needs for the presentation of designs with the desired

level of formality. In addition, efficiency is increased during the nearing of the deadline

by eliminating the needed to transfer paper design onto the computer (e.g. by scanning or

recreating the design from scratch), because hand-drawn-sketches are in the computer

(tablet) i.e. digitized; and in a few taps/clicks on the beautification icons in the sketch

system, rough sketches are then beautified (to the extent required).

1.5.4.3. Beautification techniques and supporting systems

In the discussion by Plimmer and Grundy (2005) on beautification issues in

computer-supported sketch-based design tools have illustrated different ways of

supporting beautification, as well as user-interface design, by implementing FreeForm and

SUMFLOW for testing and evaluation purposes. Plimmer and Grundy further identified

issues associated with sketch-time beautification (beautifying sketch content as user

draws) and formalization-time beautification (where almost all sketched content is

beautified), and pointed out to system requirements to support such approaches towards

beautification. These requirements include: alignment, resize formalize, direct

manipulation (e.g. drag and drop of sketched content), remove and replace, move and

resize element, move and resize group of elements. Based on Bolz’s (1993) claim that

fifty percent of the total time spent on creating drawings on a computer is on formalization

operations, Wang, Sun and Plimmer (2005) argued that the same results is achievable via

recognition and beautification techniques, and thus, save time. With this in mind, Wang et

al. explored sketch beautification techniques and its value in supporting the design process

by prototyping a grid-based, computer-supported sketch tool that incorporates several

techniques which were also discussed by Plimmer and Grundy (2005). An attempt to

determine the most suitable beautification technique and the stage in the drawing process

34

where it can be most effectively used, by testing six beautification techniques on a

continuum, ranging in nature and time suited for. Techniques consisted of four ink-

transformation (beautification) techniques including: manual, continuous morphing, corner

trigger and pen-up trigger; and two shadowing techniques: formal shadowing and sketch

shadowing (see Wang et al. 2005 for details of techniques). However, the study was

inconclusive regarding the use of beautification techniques at the right time, as the user

evaluation was informal, and the different types of techniques were not experimentally

tested to compare the (combined) effects. Furthermore, more work is required on the

recognition of sketched objects to support better beautification techniques tested, by

improving recognition algorithm to increase speed and accuracy. This need for accurate

recognition is also reflected in the difficulties faced in the development of pen-based

systems supporting smooth morphing of hand-writing (Pomm & Werlen, 2004) and

regular hand-drawn shapes (Arvo & Novins, 2000). One of the downsides of using such

tools is that it is restricted to small number of standard geometric shapes and recognizable

words. Taggart’s system is much simpler in comparison, to explore manual beautification

which converts sketchy lines into “intended” straight lines, and sketchy curves into

pointed corners (Taggart, 1975). Instead of turning curves into pointed corners, and

working with tools that only recognizes structured pen-input, Henzen and his associates

recently developed a mathematical approach towards free-form curve modeling in real

drawing situation (i.e. several lines drawn at the same area for purposes such as correction

and emphasis); which in turn, support curve smoothing as a beautification technique

(Henzen et al., 2005). Similarly, Sezgin, Stahovich, and Davis (2001) implemented a

system that combines multiple source of knowledge to provide robust early processing of

sketch input. Beautification in the system involves (minor) adjustments made to the

sketched input to make it look as “intended” – some examples include: adjusting slopes

(gradient) of the line segments to ensure that lines that were apparently meant to have the

35

same slope end up being parallel; adjustments made to the position of lines (in a sketched

object) to become aligned as intended in the layout, intended circles that have been

imperfectly drawn are adjusted to become an actual circle. Hence, the hand-drawn lines

are also ‘smoothed’. Sezgin et al. further conducted a user study with thirteen subjects

including computer science graduate students, computer programmers and architecture

students, to evaluate the system compared with Xfig (a Unix tool for creating diagrams).

All of the subjects reported that the use of the system being easier to use, efficient and

more natural. However, one must be cautious when interpreting such strong result,

because: 1) results came from only thirteen subjects, thus may biased; 2) the evaluation

study seemed informal, which could mean inconsistency in data collection procedures; 3)

very little details on experimental procedures were described which makes one question

about the robust user experience reported.

It has become apparent that recently, there has been an increasing interest in

developing tools and techniques to support the recognition and interpretation of 2-D

sketched shapes, which is then rendered and expanded into 3-D objects/models – for

example, Zenka and Slavik (2003) developed a system for creating ‘smart’ sketches of

free form 3D objects. With the help of the system, a 2-D sketch of an object is enriched

with information about its 3-D structure through rendering, and the user can further rotate

the created sketch of the object and view it from various angles. Zenka and Slavik argued

that their system is very easy to use since most of the user input is a traditional 2-D sketch,

and that it doesn’t limit the user’s creativity – based on the notion that sketching facilitates

creativity (e.g. Goel, 1995; Verstijnen, et al, 1998). A comparable sketch-based

application, SKETCH, was developed by Zeleznik, Herndon, and Hughes (2006) to

support an environment for sketching and editing 3D scenes. SKETCH uses simple non-

non-photorealistic rendering based on primitive line drawings, in which, due to the

gestural interface, operations can also be specified within the 3D environment. Other

36

recent example of 2-D to 3-D rendering techniques include 3-D botanical tree modeling

(e.g. Okabe, 2005); conversion of 2-D sketched objects into 3-D intended geometric

shapes/objects in engineering design (e.g. Kim & Kin, 2006; Shesh & Chen, 2004) based

on fuzzy logic (e.g. Qin, Wright, & Jordanov 2000; 2001); and 3-D models derived from

2-D free-hand drawings in architectural design (e.g. Schweikardt & Gross, 2000).

However, as many issues and problems are yet to be solved (e.g. accurate rendering,

efficient underlying mathematical algorithms, and requirements for sketch-based 3-D

environment); and thus, such work can be seen as the beginning for beautification

techniques that render 2-D sketches into 3-D models, and systems that support it.

Furthermore, a number of studies on the holistic development of computer-

supported sketch systems that undertake different forms of sketch beautification have also

shown some promising results. For example, earlier in 1993, Bolz focused on

implementing a tool, AssistenzComputer, that supports (semi-) automatic beautification of

drawings (Bolz, 1993), which was an extension to a conventional graphics editor.

AssistenzComputer analyses diagrams that consist of straight line segments using a user-

defined knowledge-based approach, and it also looks for flaws such as gaps and miss-

alignments, which are smoothed out in the beautified version of the sketch content. Hse

and Newton (2005) developed a complete sketch system for recognizing and beautifying

symbols, with its aimed to be simple and convenient by allowing the user to sketch the

desired shape directly and then replace it with a beautified symbol with the correct

transformation, all in one step. Interestingly Hse and Newton implemented such system as

an interface to the Microsoft Powerpoint application which makes it possible for a user to

sketch symbols directly onto a presentation slide.

SILK (Landay, 1996; Landay & Myers, 2001), a well-known sketch system, is a

tool aimed at the early stages of design, particularly for user interface design, when

designs are typically sketched rather than prototyped in software. Even in its sketched

37

form, the user interface is functional; for example, sketched buttons can be pressed and

tools can be selected in a toolbar. The sketch can also be transformed into an actual

interface where the design’s formality (appearance) is increased, and become a higher-

fidelity prototype (resembling the ‘look-and-feel’ of the final system (e.g. a user interface,

a HTML page etc). Thus, SILK enables the designer to test the design at any point, which

enables iterative design. In a similar manner as SILK, SATIN (Hong & Landay, 2006) is

a Java-based toolkit designed to support natural sketching in the design process. While

enabling beautification manipulation and rendering of objects by using specific libraries,

recognizers, interpreters, and multi-interpreters to handle pen-input, SATIN also provide

support for zooming and rotating objects, as well as switching between multiple views of

an object. Ideogramic (Damm & Hansen, 2002), a sketch-based case-tool, allows the user

to choose the level of beautification desired. The hand-drawn input can be left unaltered

or recognized and transformed into formal UML (Unified Modelling Language) shapes

such as classes (rectangle shapes with lines and text). Informal (hand-drawn) and formal

(computer-rendered, beautified) elements can coexist in the same diagram (design).

Additionally, the system also supports layout beautification (e.g. alignment).

In an extensive study, Newman et al. (2003) developed DENIM – an informal

sketch tool for Website design, based on the findings in their study to investigate on Web

site design practice (see discussion in Newman et al. 2003 for more details on their

findings). Overall, it supports sketching input, allows a designer to view and edit designs

at different levels, for example an individual Web page versus a whole Web site map,

through zooming. Designers are also able to interact with their sketched design as if in a

Web browser, thus allowing rapid creation and exploration of interactive prototypes.

Regarding beautification, DENIM recognizes simple symbols such as rectangles and lines,

which forms the basis for beautification – which varies depending on the level the user is

working at, for example in storyboard view, a line drawn to indicate navigation between

38

pages is smoothed and has a dot added to the source point and an arrow at the destination

point. Moreover, Newman and his colleagues had made various improvements in DENIM

based on an evaluation study with professional designers in which valuable feedback and

data was collected and analyzed.

InkKit, an informal sketch-based design tool was developed by Young (2005) and

Tang (2005) in collaboration – Young (2005) focused on the flexible and accurate sketch

recognition, while Tang (2005) examined beautification aspects of the system. According

to Tang and Young, the process of transferring a design from paper and pen to the

computer is error-prone and unproductive, and therefore the main aim of InkKit is to

address such issue. Additionally, beautification was referred to, by Tang, as “design

transformation” – a function which automates such transferring process by converting the

sketched input to produce the computer-equivalent output diagram for the appropriate

domain (e.g. interface design, floor plans, electrical circuits, flow diagrams). Usability

testing on InkKit showed positive results, showing that users were very satisfied with the

application based on learnability, understandability and user satisfaction. Chung, Mirica

and Plimmer (2005) further explored InkKit by adding more editing support and

beautification functions. In their version of InkKit, additional beautification functions

include: standardization of recognized components into predefined sizes based on

taxonomies specified by the user; vertical and horizontal alignment of elements in groups;

and components can be snapped to the grid. By default, all components of a sketch are

recognized and beautified as a whole; however, as noted by Chung et al., future

development of beautification techniques could include the beautification of selected

components only, by adjusting related algorithms. With its accessibility and much of the

basics done, hence, InkKit had form basis for exploring beautification this study.

On the whole, many computer-supported sketch systems have been developed to

bridge the gap between the use of paper and pen, and computer-based tools. Results from

39

Bailey and Konstan’s (2003) study further points out to the great future by showing that

the use of their informal design tool (DEMAIS) is more effective in comparison to using

pencil and paper and and Authorwware (CAD tool) for exploration and communication

behavior in early multimedia design. Also, sketch-based tools do not only support the

development of the final product, but it also aids the whole design process (e.g. needless

for the time-consuming, error-prone process of transferring paper designs onto the

computer); thus, beautification has become an important aspect to include in sketch-based

tools. It can be concluded that different types of sketch-based applications may require

different beautification techniques, which can be used at different phases during the design

process (Plimmer & Grundy, 2005). In other words, the design process can be enhanced

by tailoring the type of beautification and the time it is used according to the specific

nature of the design task.

1.4. Related studies: Interaction with hand-drawn versus computer-

rendered diagrams

Exploration of conceptual, functional and structural arrangements during early

stages of design occurs to eliminate guesswork early on, and also to detect errors at the

earliest stage possible. Therefore, in the context of beautification, it is important to look at

the effects of formality of a design (prototype) on the design process – particularly on

designers’ cognition process and design performance, for example, in terms of spotting

errors during the early stages of design – so that, as Brinck et al. (2002) noted, the end

results is “user-centered, cost-effective, high-quality, successful designs” (p. 216).

Much of the research on informal sketch-based design tools has been on the

development and implementation of such systems to bridge the gap between the paper and

pencil and computer paradigms; and thus, important aspects of sketch-based systems like

recognition and beautification have also been frequently examined. However, there has

40

been a lack of empirical research on designers’ interaction with such sketch tools and

almost none on the effects of design formality (beautification outcome) on designers’

cognition and performance in the design process. Current research on such area

predominantly consists of anecdotal and non-empirical work that are descriptive and

prescriptive in nature; however, most worryingly, only two directly related (empirical)

studies were found in the literature that fit into the topic of research on the effects of

design formality (i.e. Black, 1996; Plimmer & Apperley, 2004).

Research has shown that designers interact differently with hand-drawn diagrams

(low formality design) compared to tidy, formal, computer-rendered diagrams (high

formality design). As noted in the previous section on paper and digital prototypes,

anecdotal evidence from professional designers (e.g. Beaudouin-Lafon & Mackay, 2003;

Brink et al, 2002; Newman et al, 2003) suggested that paper prototypes (low formality) are

viewed as a conceptual, unfinished a product – thus, implying that paper prototypes tend

to provoke more comments (e.g. suggestions for changes, perceived inadequacies). In

contrast, digital prototypes (high formality) are typically perceived as a completed, final,

unchangeable product, and thus, clients/reviewers tend to question less and make fewer

suggestions for changes, especially on functional aspects of the product, and tend to focus

on the details related to the final product (e.g. aesthetics). Likewise, as Plimmer and

Apperley (2003) noted, a tidy design implies, perhaps incorrectly, that the design is

committed and complete. In addition, Wong (1992) and Wagner (1990) also noted that

informal, hand-drawn designs are better for eliciting comments from others, compared to

formal, computer-rendered designs.

Empirical evidence to support such anecdotal evidence is very limited – only two

relevant empirical studies were found that examined the effects of interaction with low

formality designs and high formality designs (i.e. Black, 1990; Plimmer and Apperley,

2004). The two studies formed the basis for the present study.

41

Results from Black’s (1990) study questionnaire indirectly confirmed that the

appearance (formality) of a design may affect designer’s performance, hence, design

outcome. In her study, Black compared visible planning on paper and on screen in

twenty-nine first year students (novice graphic designers), with a focus on students’

assessment of the experience and outcomes of their work. Following the completion of a

training session on relevant aspects of text manipulation as well as basic word processing

using specific Mac text editors, students completed a text design exercise in which

students planned the same text twice, working once on paper and once on screen. Half the

students worked on paper first and then moved on to screen, and the other half worked in

reverse order to control for order effects. The questionnaire was completed twice by

students: once immediately after the design exercise when students had had no formal

feedback from staff about the final drafts; and once after a class review in which students

had displayed their final drafts for discussion by their course tutor. One of the main

findings was that before the review of designs by the tutor, students were significantly

more satisfied with their final drafts on screen than their final drafts on paper, however,

after the review, students’ satisfaction with the paper design was higher than the screen

design. Such finding suggested that students may have perceived the final draft on screen

as a finished design that required no more improvements, as opposed to the paper design

drafts which appeared rough and informal, which may have been perceived as non-

finished designs that required more exploration and modification, thus students perceived

it as a less satisfying design compared to the design on screen. Furthermore, students’

exploration of ideas may have been limited by the ‘finished’ feedback (appearance of the

screen) they got from their screen draft.

However, a major limitation of the study was that it provided only an indirect

measure of the impact of medium (i.e. formality of designs) on visible planning (i.e.

designing; the design process). Future research to tackle such limitation might be to

42

objectively address the processes and products of designing on paper and screen and

computer, and more specifically, to examine and compare the number and quality of

solutions proposed in each medium, and maybe, the assessment of end product of

designing in each medium – such aspects will help unravel the effects of formality of

designs on designers’ performance in design (e.g. design decision making and design

quality). However, one must note that the process of designing in the two media may not

always be informative, as tasks performed during the use of the two media differ.

Furthermore, the problem of subjective definitions of what constitutes a design solution,

may affect the validity of the comparisons of the number and quality of design solutions

produced in each medium. Other minor limitations of the Black’s study include the lack

of experimental procedures to control for extraneous variables and the simplistic

questionnaire with only six questions that made one cautious about its validity. Moreover,

in the context of questionnaire studies, the number of subjects in Black’s study (n=25) was

small, thus, sufficient power for the study may not have been achieved, which in turn,

further limits the conclusiveness of such study.

Similarly, yet differently, another study examined and compared designers’

interaction with rough, sketchy informal diagrams and computer-rendered formal

diagrams (Plimmer & Apperley, 2004). Unlike Black (1990) who focused on traditional

media (pen and paper versus computer text editor), Plimmer and Apperley concentrated on

different design tools. In their study, interface designs were created for two applications: a

form for a book catalogue and a credit card application form. Each of the design was

presented in two ways: 1) a rough, hand-drawn version (low formality) of the design

created in Freeform – an informal sketch-based design tool; and 2) a formal, computer-

rendered version (high formality) of the design created with Visual Basic (VB) form

designer – a computer-based interface design tool. According to Plimmer and Apperley,

six small groups of subjects (two or three people all from a second year university

43

programming course) participated were asked to review the designs on the medium in

which it was presented on (either Freeform or the VB form design). For each application

design, subjects were given a brief problem statement (description and requirements of the

design) and two scenarios (what end-users might fill in on the two forms). Subjects were

asked to check the design given against the application description and requirements, and

to use the scenarios provided to fill in the form, as if they were the form-filler (subjects

were instructed that modifications to the designs were to be made within the same medium

the design was presented on). In addition, subjects’ skills were comparable – as noted, all

subjects were familiar with VB but not with Freeform (thus, received five minutes of on

the use the basic sketching features).

The applications (book catalogue and credit card forms) were carefully specified

by Plimmer and Apperley so that the number and type of elements in it were comparable

in each application. For example, each application included an option pair typically

represented by radio buttons along with their associated labels (i.e. the book catalog

included the specification of book binding as hardback or paper back, while the credit card

application form included applicant gender). Each application also included a selection

from a list set typically represented by a dropdown list (i.e. the book catalog specified a

book’s genre by selecting from a dropdown list, while the credit application specified

applicant income described in a set of specified ranges.

Finally, the main results found was that the number of changes made to the designs

was significantly different when using Freeform compared to VB – five out of six groups

made significantly more changes in Freeform than in VB, regardless of the application or

the order of the design exercise. Plimmer and Apperley further added that that nearly all

changes were improvements of the design. Another important finding in Plimmer and

Apperley’s study was that the time spent on different types of activity differed

significantly between the use of Freeform and VB, which was particularly true in terms of

44

the speed of making changes to the design. Although more skilled in using VB, it took

subjects longer to complete the design exercise in VB. Plimmer suggested that this may

be due to the disproportionate effort on keeping the design tidy by aligning, sizing

elements, as opposed to the focus on the design task (i.e. the design of the application)

during the use of FreeForm (informal-sketch based design tool).

However, although the design and manipulation of stimulus (i.e. the designs and

their specifications) was controlled, the design task was relatively simplistic, especially in

the context of group design, as the changeable aspects, particularly the number of ‘errors’

in designs presented to the subjects, were both little. Hence, the slight difference in the

number of change may indicate a significant real effect – plus the question of external

validity. Therefore, for future replications of Plimmer and Apperley’s study, it maybe

more effective to include a few more elements in the designs (stimulus) to create a more

stimulating design task, which also enables differences to be observed in a clearer and

more accurate manner. Another limitation of Plimmer’s study was the uncontrolled group

situations (groups of two or three people) when working on design tasks, which reflects to

the little attempt made to account for the group processes in the design tasks – perhaps

studying individual design process could provide more robust, reliable and interpretable

data.

Hence, along with its limitations and suggestions for further research, Plimmer and

Apperley’s (2004) study formed the basis for the present study – an extended version (also

an improved replicate) of their study, by not only examining two opposite levels of

formality (i.e. low formality designs that are hand-drawn and appear rough and sketchy;

and high formality designs that are computer-rendered and appear tidy and formal), but by

also examining the other levels of formality between the two ends. The present study also

questions how designers may be affected, in terms of design performance and design

outcome, when a design appears more or less formal. In other words, this study explores

45

different levels of formality that a design can appear (as a result of different degrees of

beautification) and the ways in which designers may be affected by more or less formal

looking designs. Thus, to address such questions (by using a within-subjects approach),

experimental procedures (as well as tools and material used) in this study are based on and

expanded from Plimmer and Apperley’s study. In addition, the present study addresses

the limitation of using two different tools (an informal sketch-tool and a formal VB tool)

for creating low formality designs (hand-drawn, rough and sketchy), and high formality

designs (computerized, formal and tidy); thus, this study uses only one tool (i.e. InkKit –

an informal sketch-based design tool) to create designs with varying levels of formality,

ranging from low to high by varying degrees of beautification. Moreover, a low formality

design presented on paper was also included in the experimental study, for comparing

whether designers/reviewers interact differently with a low formality design presented on

paper, and low formality design presented on the InkKit (informal sketching-based design

tool).

1.5. The Present study: Aims and hypotheses

On the whole, with the fundamental question of how design processes can be

optimized in mind (Eisentraut & Gunther, 1997), there exist many research projects on the

development of sketch-based tools to support the early stages in the design process (e.g.

Aliakseyeu, Martens, & Rauterberg, 2006; Chung, Mirica, & Plimmer, 2005; Henzen,

2005; Landay, & Myers, 2001). However, empirical research at the other end to examine

the effects on the designers’ cognition, perception, behaviour, work (design) performance

etc, of using such tools during the design process, is still at its infancy. Especially, the

lack of research on the extent of beautification that result in beautified designs which

appear more or less formal, and the effects of interacting with such designs on the design

process – will it affect design behavior, design performance and design outcome? Hence,

46

in addition to the need to further explore beautification techniques within the context of

sketch-based tools, there is a need to examine their effects on designers during the design

process.

The main purposes of this study were to use an experimental approach to: 1)

further explore the concept of beautification in the context of sketch-based design tools by

examining the dimensionality of beautification and its techniques; and 2) to also

investigate the effects of design formality (beautification outcome) – from rough (hand-

drawn, non-beautified) sketches to formal (beautified) diagrams – on design performance.

More specifically, this study explores the effects of different formality level of designs on

design performance, by measuring the number of changes made: total changes, quality

changes and expected changes made (described in Section 2: Method). Also taken into

account are factors that may play a role in affecting the relationships between formality

level of designs and design outcome, including expertise (e.g. design experience,

education, domain-specific knowledge), design perception and design medium. In

addition, the present study has a particular focus on the early stages of the design process

because of their determinant influence on design costs and design outcomes.

Hypothesis:

1) That the number of functional changes made (total, quality and

expected changes) will differ as levels of formality of a design increase (or

decrease).

2) That the number of functional changes made (total, quality and

expected changes) will differ between ‘experts’ and ‘novices’ as levels of formality

of a design increase (or decrease). More specifically:

47

a. That there is a difference in design performance between

subjects with task related design experience and subjects with no

design experience to some non-task related design experience.

b. That there is a difference in design performance between

subjects with task-related domain - specific knowledge (i.e. study

major/specialization) and subjects with no task-related domain-specific

knowledge to some task-related domain specific knowledge.

c. That there is a difference in design performance between

subjects with higher education (study) level (university

graduates/postgraduates) and subjects with lower education level in

comparison (university undergraduates).

3) That subjects will enjoy working on designs that appear more

formal (higher formality – i.e. more beautified) more than designs that appear less

formal, rougher with a sketchy look-and-feel (lower formality – i.e. less

beautified).

4) That there is no difference in preference between designing on

paper compared to designing on the tablet PC (InkKit).

5) That design medium/tool preference in real world design

situations would be more diverse than design medium/tool preference during the

experiment.

48

Chapter 2. Method

2.1. Experimental Design

A within-subject repeated measures design was used in this experiment to measure

the effects of formality on design performance. There were five conditions, all of which

were presented to each subject one after another. Latin square design, as shown in Figure

1, was used in to control order effects of the conditions presented (Heiman, 2001). In

addition, between-subject comparisons were also made to test Hypothesis 3 where experts

and novices may be affected by formality differently during the design process.

Subjects

Figure 1. Latin square design: orders of presentation of conditions i.e. rotation of conditions in two directions (from 1 to 5, and from 5 to 1), to control for practice effects.

2.1.1. Independent Variable: Level of Formality

There was one independent variable – levels of formality of a web interface design

prototype (HTML forms) with four levels: from low formality to high formality. There

were five conditions in the experiment, as shown in Table 1, with four conditions each

involving one HTML form design with a different level of formality presented on the

tablet PC, and for comparison, as interaction with digital and traditional media differs

(Bilda & Demirkan, 2003) which in turn, may affect design-decisions (Black, 1990), one

condition involved one HTML form design with low formality was presented on paper.Table 1.Level of formality associated with each condition, and the medium used for the presentation and review of designs.

Conditions Formality level Medium1 Low formality (totally hand-drawn) Paper (and pen)2 Low formality (totally hand-drawn) Tablet PC3 Medium-low formality Tablet PC4 Medium-high formality Tablet PC5 High formality ([totally] computer-rendered) Tablet PC

To produce different levels of formality, a taxonomy of beautification was

developed (see Table 2) based on previous work on beautification techniques (e.g. Chung,

Mirica, & Plimmer, 2005; Plimmer & Grundy, 2005; Pomm & Werlen, 2004; Wang, Sun

Order of conditions: from 1 to 51 2 3 4 52 3 4 5 13 4 5 1 24 5 1 2 35 1 2 3 4

Order of conditions: from 5 to 55 5 5 5 54 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 1

49

& Plimmer, 2005). Smoothness and alignment (vertical and horizontal) were varied

systematically as described in detail in section 2.5.2, while others were held constant. The

relationship between beautification and formality can be understood as: the higher the

degree of beautification, the higher the level of formality of a design (i.e. design appears

more formal) Table 2Taxonomy of beautification showing different variables associated with beautification.

2.1.2. Dependent Variables: Functional changes

Measurements were taken to explore the effects of design appearance (different levels of

formality) on the design performance – in this case, decision-making – during the early

stages of the design process. Participants were to improve the designs presented to them

by making changes to the designs; hence design performance (particularly decision

making) could be objectively measured by counting the different functional changes made.

Functional changes measured, in the context of web interface design, included:

o Adding:- an element to an item- an item to an item set- an item set to the design

o Deleting:- an element in an item- an item- an item set

o Changing: - a control from one type to another e.g. radio button to a textbox- a label e.g. spelling, words, grammar etc

o Resizing: - a control to suit the scenario e.g. resizing a text box to become bigger/smaller,

wider/longer to suit the scenarioo Relocating:

- an element / an item / an item set from one area to another area in the design to

suit logical flow of information.

VARIABLESHand drawn ----------------------continuum--------------------- Computer-generated(Low formality) (High formality)

Smoothness (objects, lines or characters) Rough (hand-drawn)

Computerized i.e. Smoothed, straight, formal

Size Inconsistant Exact, standardisedAlignment, vertical Inexact Exact, standardisedAlignment, horizontal Inexact Exact, standardisedSpacing, vertical (between objects) Irregular Exact, standardisedSpacing, horizontal Irregular Exact, standardised

50

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

N

N

N

N

N

N

N

54

In the context of this study of HRML form design, an ‘element’ is any single component

within a design. An element is part of an ‘item’ (a label and its associated control for user

input e.g. a label “Male”, associated with a radio button); and an item is part of to an “item

set” (which consists of one or more items e.g. Gender: Female (with a radio button) / Male

(with radio button). An item set is at the highest level within the hierarchy of functional

changes; hence, a functional change is counted at the highest level in the hierarchy. See

Figure 2 for an illustration of this tree of change.

The number of functional changes made at each level of formality was measured and

were categorized into three variables:

o Total Changes: any (functional) change made by the participant on the design

e.g. all changes mentioned above including adding, deleting, changing, resizing

and relocating elements, items, and/or item sets.

o Quality functional changes: the number of functional changes to elements, items

and/or item sets made by the participant that improved functionality in comparison

to the original version – according to Web Usability handbooks (e.g. Brinck,

Gergle, & Wood, 2002; Fowler & Stanwick, 2004), HTML design guidelines (e.g.

Shelly, Cashman, & Woods, 2005) and interface design principles (e.g. Watzman,

2003).

o Expected (planned) functional changes: the number of functional changes to

elements, items and/or item sets, made by the participant to ‘correct’ deliberate

design ‘errors’ planned by the experimenter – based on Web Usability handbooks

(e.g. Brinck, Gergle, & Wood, 2002), HTML design guidelines (e.g. Shelly,

Cashman, & Woods, 2005) and interface design principles (e.g. Watzman, 2003).

As the number of deliberate errors was the same in each design presented to the

participants, the number of corrections (expected changes) made in each design was

measured to allow for controlled comparisons between conditions, hence, to explore the

effects of formality on design-decisions during early stages of the design process (i.e.

during early prototyping). In addition to expected changes, quality changes and total

changes made were also measured as it was anticipated that along with expected changes,

participants would made other changes to the design that were not deliberate errors (refer

to Appendix A for the outline of design errors that was present in each design). The three

measurements (total changes, quality changes, and expected changes) were important for

the assessment of validity and reliability of the experimental stimuli – the five designs

presented to the participants i.e. whether the number of quality changes and expected

55

changes were similar; and whether the total number of changes made was statistically

different from the number of quality changes. If the number of changes made were not

statistically different, then it could be statistically reasoned, that the five designs were

equivalent, and that the results was due to the experimental manipulation of the

independent variable.

Participants were directed to make functional changes by the instruction to

“improve the design[s] to better serve its [their] purposes”, therefore, participants did not

have to make any beautification changes such as alignment of elements (see other

examples in Table 2) to ‘tidy-up’ the design. Thus, beautification changes made by

participants were not counted as a functional change. Furthermore, beautification changes

were not measured as a dependent variable it was predicted that the process of tidying up a

design was time consuming (e.g. Newma, et al., 2003) especially in an experiment with

only 11 minutes for each condition.

In addition, a post-task questionnaire was used to record the following variables

(see Appendix B for response options details in the questionnaire):

o Post-task rankings of overall enjoyment of designs in the order from the most-liked

design (1) to the least liked design (5). Reasons for rankings were also recorded.

o Preference for design medium in the experiment. Response options were

preference for pen and paper; preference for the tablet PC; or no preference.

o Preference for design medium in the real world. Response options were open.

o Demographics including (open response) :

Design experience.

Study specialization/major.

Study Level.

2.2. Participants

Thirty adults – sixteen male (mean age of 22.81, SD = 5.87) and fourteen female

(mean age of 21.14, SD = 1.03) between 18 and 44 years of age (total mean age of 22.03,

SD = 4.36) volunteered to participate in the study.

Participants (n=20 who majored/specialized in Computer Science (CS) or Software

Engineering (SE) in their study, as well as participants with non-CS/SE study backgrounds

(n=10) were recruited from the University of Auckland. All participants were current

students/recent graduates from the University of Auckland (mean years of study at

university: 3.20, SD = 1.186). Out of thirty, twenty-five participants were students

56

(twenty-two undergraduates and three post-graduates), five participants, recruited through

researcher’s personal contacts, were recent graduates from the University of Auckland in

2006, all currently working in computer/software engineering-related industries. Overall,

there were twenty-two undergraduates, and eight graduates/postgraduates.

Papers taken most frequently during the course of study were the computer science

stage one papers including CS101. All participants (n=22) who majored/specialized in

computer science(CS)/software engineering(CS) had taken or taking CS101 and only two

out of ten participants who majored in non-CS/SE subjects (including other engineering,

business and information-system students) had taken CS101. Relevant papers taken by

participants who majored in CS/SE were: CS101 and CS105 (stage one); CS230 (stage

two); CS345 (stage three) and SE SE450 (equivalent to CS345). Others papers taken by

participants with non-CS/SE majors included: information system, engineering,

biomedical science, psychology and business papers. Fifteen participants had CS and/or

SE design experience such as HTML design, website and interface designs and software

design, whereas, the other fifteen participants’ design experience ranged from no design

experience to some non-CS/SE related design experience.

All participants were exposed to Inkit (the design tool used in the experiment) for

the first time. All participants had normal eyesight or corrected-to-normal by spectacles or

contact lenses. For the summary of demographics, see Appendix C.

Participants were each reminded not to discuss the experiment with their peers, and

were thanked with $2 worth of chocolate as a token of appreciation as well as entering the

draw to win $50 cash.

2.3. Procedure

Study approval was obtained from the University of Auckland Human Participants

Ethics Committee (UAHPEC). Each participant took part in a single session

approximately one hour long. An experimental protocol was produced to standardize

experimental procedures and thus, help minimize experimental errors and variability.

Participants were first instructed to make changes to improve each design presented to

them. Five early designs of online forms were then presented to the participants one after

another – four designs were presented on the tablet PC and one was presented on paper

(refer to Table 1). Upon completion of experimental tasks, participants filled in the post-

task questionnaire. The detailed procedure is described next.

57

Upon arrival, the participant was asked to read the information sheet and to sign

the consent form (see Appendix D). The participant was then asked to adjust the work

station, to suit him or her, including the chair height and its distance to the desk, the desk

height (by adjusting the lever), screen angle (of the tablet PC) and the positioning of the

mouse. The experimenter further checked if the participant was comfortable with the

lighting level and room temperature, and whether there was anything else the participant

needed to do before starting the experiment, to minimize disruption of the experimental

procedures.

InKit, an informal design tool (i.e. the experimental apparatus described below in

section 2.4.4.) was presented to the participant on the tablet PC (described in section 2.4.3)

during which a brief introduction to Inkit – its authors and purposes – was given. The

practice design, as shown in Figure 3, containing sketches of four common types of

elements found in HTML forms (i.e. text boxes, dropdown menus, labels and radio

buttons) was presented and a description of each element was given in terms of usage and

functionality.

Figure 3. The practice design – presented to the participant prior the first experiment condition. It contains the four main elements (label, textbox, dropdown menu, and radio button) used in the to-be-given designs.

Instructions were explained again, informing the participant that he or she could

make any changes to the design. In addition to the types of changes described earlier (p.2)

including adding, deleting, changing, resizing and relocation of any elements in a design,

participants were instructed that annotation was also acceptable (e.g. make notes,

explanation, draw arrows etc) to indicate changes that should be made.

While explaining each type of change, the experimenter also demonstrated how the

change could be made in Inkit with pen-input, on the tablet PC (see Table 1 in Appendix

58

05 for the demonstration details). Although passively shown already in the previous steps,

the participant was informed explicitly on the three ‘modes’ in Inkit (see Table 2 in

Appendix 05), and how to change modes (i.e. by tapping on the icon associated to the

mode, and an exclusively highlighted icon meant that desired mode has been successfully

selected).

The next five to ten minutes was the guided familiarization process in which

participants were able to get familiar with Inkit. At the end of the period, participants

showed that they could draw (add and annotate), erase (delete), select and move

(relocation), and select and resize (resize), and hence, changing modes (drawing, erasing

and selecting). The instruction sheet (described in section 2.5.1 below) containing the

requirements and the scenario for the first condition was then presented. Instructions were

read out-loud by the experimenter and subjects were to ask questions if they were unclear

(see Appendix F for each set of requirements and scenario associated with each design).

Before starting the experiment, participants were explicitly told to stay within the

application and to use only the three functionalities (draw, erase, select) at all times. The

designs were then presented successively to the participant depending on the randomly

selected order of conditions for the particular participant (refer to Figure 1). The five

designs presented (one design in each condition) are described below in section 2.5.2.

The first form design was presented either on paper or the tablet PC according to

the condition order. Each subject was given 11 minutes in each condition – 1 minute for

reading the instructions and 10 minutes for working on the design. At the end of eleven

minutes, the participant was asked to stop making changes or to finish off any changes

they were making. One extra minute was given if the participant had asked for more time.

When the changed design file was saved to the participant’s unique folder, the next design

was presented (again, either on paper or the tablet PC according to the condition order)

together with its associated instruction sheets.

With the design presented on paper, the participant was given a sheet of blank A4

paper, along with the design and the instruction sheets which also contained the

requirements and the scenario. The participant was instructed to make changes on the

original design using the blue ball-point pen given, and to use the blank paper if more

space was needed. With designs presented on the tablet PC in Inkit, the participant used

the specialized pen for the tablet PC to draw directly on the tablet screen (with immediate

input feedback – i.e. what you draw is what you see) as if drawing on a piece of paper.

While the participant was working on the design, the experimenter monitored the

process on the display panel output on the other side of the room, and recorded (on paper)

59

any observations such as design behaviors, usability issues, timing issues such as over-

time or under-time etc that were of interest. This information was also recorded on the

tablet PC using Morae Recorder (2000), as described in section 2.4.3.

When the last condition has ended, the participant was asked to fill in the

anonymous paper-and-pencil questionnaire (refer to Appendix B) and to notify the

experimenter when page 3 in the questionnaire was reached so that the experimenter could

show him/her the five designs that they had worked on (in their order of presentation) to

facilitate answering of subsequent questions.

2.4. Apparatus

2.4.1. Room Setup

Each experiment session involved only one subject which took place in a quiet

room in the Department of Computer Science building, at the University of Auckland.

The room was set up so that neither the subject nor the experimenter could see each other

during each experiment condition (see Figure 4) to minimize effects of observation.

2.4.2. The Tablet PC

The experiment was conducted using a Toshiba Tablet PC (Edition 2005, Intel®

Pentium® M, 1600MHz, 590MHz, 512RAM, Microsoft Windows XP Operating system).

The stimuli were presented on a 15” CRT colour (LCD) screen on the Tablet PC with

1280 X 1024 pixel resolution. Colour quality was set to the highest at 32 bits and

brightness level was set to the maximum to ensure clarity and recordability. Screensaver

was turned off. There was no glare or reflections on the screen since there were no bright

lights in subjects’ visual field. Although there was a window near the workstation, the

Venetian blind with half closed slits were pulled down throughout the experiment to filter

out direct sunlight.

60

Figure 4: A bird-eye view of the room set-up for the experiment sessions where the

experimenter (separated from the participant by the dividing wall) was able to view the exact screen display that the participant was seeing and working on.

2.4.3. Morae Recorder (2004)

Morae Recorder (2004) on the Tablet PC was used to record and save all actions

performed by the participant on the computer including input from the mouse, pen and/or

keyboard, visible on the screen.

2.4.4. Inkit and the programming of beautification functions

The computer program that was used in the study is called Inkit – an informal

design tool developed by the Human Computer Interaction Group in the Computer

Science Department at Auckland University (see Chung, Mirica, & Plimmer, 2005; Tang,

61

2005; Young, 2005; for more detailed descriptions of Inkit, its architecture and codes).

Since the study was exploratory in nature, despite that Inkit was, at the time of the study,

still at the development and testing stage for more complex features (other than stable

basic computer functions such as save, open, copy, paste, drag and drop, resize etc), Inkit

served as a useful tool for the main purpose of the study. And thus, beautification

functionalities required in informal sketch-based tools based on previous research on

beautification techniques (e.g. Chung, Mirica, & Plimmer, 2005; Damm & Hansen, 2002;

Plimmer & Grundy, 2005; Pomm & Werlen, 2004; Sezgin, Stahovich, & Davis, 2001;

Wang, Sun & Plimmer, 2005), were explored and implemented with codes written in C#,

using Inkit (March, 2006) as the fundamental building block. During beautification,

objects (drawn ‘shapes’) are first recognized by ‘recognition engine’ to identify what type

of ‘things’ they are – for example, in terms of user interface, shapes could be recognized

as textboxes, labels, dropdown menus, radio buttons, buttons etc. A blue rectangle that

wraps tightly around the object would then appear, and beautification occurs at this point

by manually selecting (and combining) beautification functions listed on the menu: 1)

Horizontal Alignment; 2) Vertical alignment; 3) Standardization; and 4) Line smoothing at

33%, 50%, 66%, and 100% (some codes available upon request).

2.4.4.1. Horizontal Alignment

The horizontal alignment function aligns objects by first, categorizing which

horizontal group each object belongs to, then calculating the average bottom point of each

horizontal group according to bottom point of the objects’ surrounding rectangle within

the same group, and finally moving the objects to its aligned positions. So at the end,

objects belonging to the same horizontal group would sit on the same point on the y-axis,

with its original points on the x-axis retained. See Figure 5 below for an illustration of

horizontal alignment.

62

Figure 5. Horizontal alignment of elements: (A) the original hand drawn elements; (B) the blue rectangle that wraps tightly around each object indicate that each object has been recognized as an user interface (UI) element; (C) the elements are aligned horizontally when the “horizontal alignment” button is clicked – the bottom point of the blue rectangle is aligned to the same point on the y-axis.

2.4.4.2. Vertical Alignment

(A)

(B)

(C)

63

The same goes to the vertical alignment function but with the reversed concept –

each object is categorized into its vertical group, then the average left point of each

vertical group according to the left point of the objects’ surrounding rectangle within the

same group, and objects were moved to its aligned positions. So at the end, objects that

belong to the same vertical group would be aligned to the left on the same point on x-axis,

with its original points on the y-axis retained. See Figure 6 below for an illustration of

vertical alignment.

Figure 6. The same steps in Figure 111 occur in vertical alignment where objects are first recognized by the recognition engine, and recognized objects are shown within the blue rectangular bounding box. The elements are aligned vertically when the “vertical alignment” button is clicked, shown in the diagram where the bottom point of the blue rectangle is aligned to the same point on the x-axis.

2.4.4.3. Standardization

The size (height and width) of each object that belongs to the same object group

(for example, textbox, dropdown menu, radio button, and label) would be the closely

approximated after standardization. An algorithm for standardization was designed where

the sizes of objects in each group are averaged and also calculated by a specific numerical

factor for normalization purposes. See Figure 7 for an illustration of standardization of the

size of objects.

64

Figure 7. Standardization of the size of elements: (A) the original hand drawn elements; (B) the blue rectangle that wraps tightly around each object indicate that each object has been recognized as an UI element; (C) the elements are standardized when the “standardize” button is clicked – the elements are standardized to a closely approximated size (height and width) within the blue rectangle; (D) additionally, vertical alignment and horizontal alignment are applied.

2.4.4.4. Line Smoothing

Smoothing of lines was achieved through designing algorithms that involved

identifying all the points on the line, flattening (smoothing) the line by mathematically

shifting the points (in terms of x and y coordinates) to a closer approximation to the

corresponding point on the line of the mathematically generated object based on the

corners. In other words, the smoothing function systematically straightens lines of hand-

drawn objects at different levels – 33.3%; 66.6% and 100%. One of the assumptions

made, as described below in 2.5.2, was that as the lines look smoother (straighter and more

computerized), the diagram will appear more formal. In this study, smoothing was

possible for textboxes, dropdown menus and radio buttons. See Figure 8 for an illustration

of different levels of smoothing hand-drawn lines objects to represent different levels of

formality.

(A) (B)

(C) (D)

65

Figure 8. Levels of smoothing of hand-drawn objects (lines) to represent levels of formality: (A) the original hand-drawn objects, representing low formality; in (B), (C) and (D), lines in grey denotes the original hand-drawn objects and lines in black are the original lines that have been 33.3%, 66.6% and 100% smoothed, respectively. (A) Represents low formality; (B) represents medium-low formality; (C) represents medium-high formality; and (D) represents high formality.

However, there has been very little methodological support (in terms of mathematical

and computer programming) for rendering hand written electronic ink to computer fonts

still in the form of electronic ink. Despite the lack of research, attempts to smooth

(morph) characters and/or words in a label were made through techniques such as

mathematically mapping and transforming points, and also manipulating a label’s entity –

but results of such techniques did not reach a satisfactory level for the experiment. Like

Pomm and Werlen’s (2004) exploration on hand writing morphing techniques, beautifying

hand-written texts could be done as an individual study. This is particularly true given the

tight time frame for this study. An alternative method was used to achieve different levels

of formality of text for the purpose of this study by programming specified fonts.

(A)(A) (C)(C)

(B)(B) (D)(D)

66

For the lowest level of formality, the experimenter’s ‘normal’ hand writing using

pen-input was used (i.e. characters not exactly aligned horizontally and vertically, spacing

between characters and words was not exact and each character looked slightly different

every time). As for achieving medium-low formality, a new font – the experimenter’s

computerized (standardized) handwriting – was created by using My Font Tool for Tablet

PC (Lanier, 2004). Handwriting of individual letters from ‘a’ to ‘z’ both in capital and

lower case letters, as well as numbers and some common symbols such as comas, full

stops, exclamation marks, question marks etc, were recorded through pen-input. Character

spacing, word spacing, and line spacing were adjusted, and the final step was to compile

the data to create the new standardized handwriting font to be installed on the tablet PC

(see Appendix G for some screen shots during font creation). This tool ensured that, when

the labels were created, characters were aligned exactly according to the base-line, mid-

line and roof-line horizontally and vertically; character spacing and word spacing were

exact horizontally and vertically, and each character was standardized so that the same

look-and-feel of the character was used. With such mathematical and physical

manipulation of font properties, visually, the computerized characters (words) looked

more formal than the handwritten characters (words).

Although there has been no direct and conclusive empirical evidence to determine

‘font formality’, much of the research on fonts has been on text/font legibility and

readability. For example, Arditi (2004) looked at the effects of customized fonts (varying

in size, serifs and san serifs) by forty visually impaired users and found that different

individuals produce different distinct fonts that resulted in enhanced legibility. However,

no comparisons in legibility were made between the customized fonts and the highly

legible standard fonts such as Times New Roman. In other studies, for example Arditi and

Cho (2005), font properties were manipulated by the experimenters. In Arditi and Cho’s

(2005) experiment, lower-case fonts varying only in serif size (0%, 5%, and 10% cap

height) were used, and legibility was accessed using size thresholds and reading speed. It

was found that serif fonts were slightly more legible than sans serif, but had no effect on

reading speeds (rapid serial visual presentation and continuous reading speed). No

difference in legibility was found when typefaces differed only in the presence or absence

of serifs. Other studies on the effects of font typography examines cognitive performance

such as information recall (Gasser, et al., 2005); visual search and information retrieval in

web pages (Ling & van Schaik, 2006); letter recognition (Sanocki, 1998) and classic

experiments on reading comprehension (Poulton, 1965). Furthermore, web usability

studies on fonts also examined legibility and performance. For example, Bernard, et al.

67

(2001) examined popular online fonts – ornate fonts (e.g. Bradley and Corsiva), sans serif

fonts (e.g. Arial, Verdana, Tahoma etc) and serif fonts (e.g. Courier New, Georgia, Times

New Roman, etc) – and found, also, no difference in legibility between the font types,

however, performance, in terms of reading time, was different. Bernard, et al. also

examined perception of various fonts and found that Courier, Commic, Verdana, Georgia,

and Times New Roman were perceived as the most legible font types. In addition,

aesthetic appeal related to specific font types were also explored, and found that Courier

and Time New Roman (serif fonts) were perceived as being the most business-like,

whereas Comic (sans serif font) was perceived as being the most fun and youthful. In the

field of advertisement, it was suggested that san serifs (non-serif) fonts looked more

playful, youthful and fun (also suggested by Bernard, et al. 2001), and have now become

more popular in terms of its use in logos for multi-national brands (e.g. fast-food

companies) to small businesses (e.g. retail shops).

In addition to the low formality and medium-low formality fonts represented by

non-computerized and computerized handwriting on the tablet PC, the fonts to represent

medium-high formality and high formality were also determined. With no conclusive

evidence on font formality, it was reasoned, therefore, an uncommon, san serif font (in this

case, Gulim) was to be used to represent beautified handwriting at medium-high level of

formality; and as there were some findings supporting the use of Times New Roman as a

common legible typeface compared to san serif fonts, Times New Roman was used for

representing fully beautified handwriting at high level formality. See Figure 9 below for

an illustration of the four fonts used to represent different levels of formality.

How are you? (Times Roman Numeral)

How are you? (Gulim) How are you? (Experimenter’s standardized Handwriting)

Figure 9. The four fonts used to represent different levels of formality: Times Roman Numeral – represents high formality; Gulim – representing medium-high formality; standardized handwriting – representing medium-low formality; and un-standardized handwriting on the tablet PC – representing low formality.

There are many variables that play a role in producing different typeface. While

every effort was made to control for common variables affecting typeface, as suggested by

68

Watzman (2005), including size, letter spacing, word spacing, line spacing and line

justification in all five designs, we can not be sure that the fonts used in this study to

represent the various levels of formality were an accurate representation of the levels.

This is acknowledged as a limitation of the study.

2.5. Stimuli and Materials

2.5.1. Instruction Sheets

In each condition, subjects were given a design of an online form to work on and to

make any changes to improve the design in terms of functionality and its purpose. In each

condition, an instruction sheet containing the requirements and the scenario (see Appendix

F for each set of instruction sheets associated with each design) was given before

presenting the associated design to the participant. The same format was used for all five

instruction sheets: (identical) instructions were printed at the beginning on each instruction

sheet, followed by the requirements and scenario associated with the design presented.

The purpose of including both the requirements and the scenario for each condition was to

help produce a more realistic design situation – requirements collected (Maybew, 2003)

and scenarios formed (Rosson & Carroll, 2003) in the early stages of the design process to

help shape the final product (although in a laboratory environment). In particular, with

respect to the conditions, the requirements helped the participant to identify whether the

correct information was being ‘collected’ from the end-user (who would be filling in the

HTML form), which in turn, guided the participant to add, delete and relocate

elements/items and/or item sets appropriately. The scenario, in addition, points out

whether an element was of the appropriate type (change element) and size (resize).

2.5.2. The five designs each representing a different level of formality

There were a total of five equivalent designs, and hence, a total of five conditions.

Variables had to be controlled between the five designs to increase validity of results so

we are measuring the effects of formality on design decisions. Hence, the designs were

made as equivalent as possible, in terms of: 1) the purpose of the forms – requiring users

to fill in personal information – hence, common online forms; 2) order of elements in the

design (i.e. textboxes, radio buttons, dropdown menus, and labels in the same order in

each design); 3) the balance of types of element (i.e. 12 textboxes, 10 radio buttons, 5

dropdown menus and 31 labels in each design), and thus, 4) the number of elements in

69

each design (i.e. total of 58). Within the design, the sizes of elements were also

controlled for by having closely approximated physical measurements i.e. textboxes all

had an approximate height of 1cm and a width of 8.5cm (and 5 textboxes with half the

width) at 100% screen radio; similarly for dropdown menus, with a triangle with a height

and width of 50mm on the right hand side within the dropdown menu; all radio buttons

had an approximate diameter of 1cm (sizes could not be perfectly exact due to different

levels of smoothness/roughness of beautified lines); labels had an approximate height of

1cm, however, the width of labels was uncontrollable as word length and the number of

words contained in a label varied. Mathematical standardization of element groups was

achieved by applying the standardization function in the beautification menu list.

Additionally, all labels were programmed to space 50 pixels apart vertically (on the y-

axis) and their associated controls (textbox, dropdown menus and radio buttons) were

aligned to the labels horizontally (on the x-axis) at 30 pixels apart.

The beautification variables shown in Table 2 were either controlled (size, spacing)

or systematically varied (alignment, smoothness). Alignment and smoothness of hand

drawn elements were combined and varied systematically, as described below, to produce

different levels of formality.

1) Alignment (vertical and horizontal)

According to perceptual theorist (e.g. Gestalt, Marr, Gibson, Rock), factors such as

orientation and grouping of stimuli, affect visual perception of a form. For example, axis-

alignment affects perceptual grouping (Boutsen & Humphreys, 1999); the importance of

balance of objects as an organizing design principle (Locher, Stappers, & Overbeeke,

1998). Therefore, by varying the extent of alignment, elements of the same type (e.g.

textboxes, dropdown menus, radio buttons, and labels) would appear differently grouped

and organized. In addition, according to web usability handbooks and guidelines based on

Gestalt principles (e.g. Brink, 2002), groups of elements and information should be

aligned for grouping purposes and easier comprehension – for example, alignment of

textboxes to textboxes; labels to labels; and “submit” buttons at the centre-bottom of the

page. If elements are unaligned, information becomes scattered, creating complexity and

additional visual features within the design. Brink (2002) such problems will make the

display look excessively cluttered and unprofessional - hence, it will affect the appearance

of a design (the more formal, the more professional a design appears).

Therefore, an assumption was made that different ratios of alignment would

produce different levels of formality of a design. In other words, the more elements

70

aligned the more formal the design will appear. To create different levels of formality in

the designs presented to the participants, elements in the design were aligned

systematically according to mathematic principles of ratios as shown below in Table 3.

Table 3. Ratio of elements aligned (and its description) in each design representing a different level of formality.

Formality Ratio Alignment descriptionLow formality (paper) 3:0 No elements are aligned exactlyLow formality (tablet PC) 3:0 No elements are aligned exactlyMedium-low formality 3:1 From top to bottom, and from left to right, every third

element is aligned to the element at the top that belong to the same group e.g. label to label, textbox to textbox, radio button to radio button, label to label)

Medium-high formality 3:2 From top to bottom, and from left to right, every second element out of three elements are aligned to the element at the top that belong to the same group e.g. label to label, textbox to textbox, radio button to radio button, label)

High formality 3:3 Every element aligned exactly according to its type

2) Smoothness of lines (lines that make up textboxes, dropdown menus, radio buttons

and labels)

Although there has been some studies on beautification of hand-drawn sketches in

electronic ink (e.g. Hse & Newton, 2005; Pomm & Werlen, 2004; Ženka & Slavík, 2003),

no direct empirical evidence regarding smoothness of lines affecting formality was found.

However, from everyday examples, it can be noticed that smoothness of lines can be an

important factor that influence the appearance of objects. For example, it is natural to want

to see objects such as a black board, a table, walls, with smooth edges (aesthetically

pleasing), rather than rough, uneven edges, which may appear to be an unfinished product;

in addition, printed text may appear to be more formal than hand written text, which may

appear to be unfinished).

Therefore, a rational and valid assumption was made – that a computerized straight

line (without bumps) appear more formal than a hand drawn line (with bumps), and hence,

that the smoother the line (including lines that make up text), the more formal it will

appear. To create different levels of formality in the designs presented to the participants,

lines were smoothed systematically, as shown below in Figure 4.

Table 4. Systematic smoothing applied (% smoothed) to the original hand-drawn lines of textboxes, dropdown menus and radio buttons; and fonts used for labels to represent different levels of formality in the designs presented to the participants.

71

Formality Smoothing applied to lines (%)

Fonts representing beautified (smoothed) hand-writing (labels)

Low formality (paper) 0.0 Non-beautified original hand writing on paperLow formality (tablet PC) 0.0 Non-beautified original hand writing on the tablet PCMedium-low formality 33.3 Standardized original hand writing on the tablet PCMedium-high formality 66.6 GulimHigh formality 100.0 Times New Roman

When the two beautification variables described above – alignment and smoothing

of lines, were combined and varied systematically, and keeping the other two

beautification variables constant (size and spacing), equivalent designs (as described

above) that appeared more or less formal were created. In other words, different levels of

formality were produced by systematically varying alignment and smoothness of lines,

and hence, beautification taxonomy was successfully developed and two beautification

variables were tested and validated. Figure 10.1, 10.2, 10.3, 10..4 and 10.5 below shows

each design representing a different level of formality, with one low formality design

presented on paper, and four designs from low formality to high formality presented on the

tablet PC. Each design was presented one after another (according to the order of

presentation) to the participants to work on, with the instruction to improve each design by

making changes to them.

72

Figure 10.1. Low Formality (on paper) – Online Magazine subscription with 0% smoothed lines i.e. all original hand-drawn design on paper; non-computerized ‘normal’ ‘natural’ writing (of the researcher’s); some form of rough non-computerized vertical and horizontal ‘alignment’ with 0:3 ratio of exact (computerized) alignment of elements.

Figure 10.2. Low Formality (on tablet PC) – Samson’s Bank $1 Million Loan Application; 0% smoothed lines i.e. all original hand-drawn input, non-computerized ‘normal’ ‘natural’ writing (of the researcher’s); some form of rough non-computerized vertical and horizontal ‘alignment’ with 0:3 ratio of exact (computerized) alignment of elements

73

Figure 10.3. Medium-Low Formality – Graduation Application Form with 33% smoothed lines; characters in standardize hand writing (of the researcher) created using My Font Tool For Tablet PCs (2005); font size 18; vertical and horizontal alignment ratios of 1:3 in arithmetic order.

Figure 10.4. Medium-High Formality - Dog Registration Form with 66% smoothed lines; characters in Gulim font i.e. without serifs; font size at 18; vertical and horizontal alignment ratio of 2:3 in arithmetic order

74

Figure 10.5. High Formality design - America’s Next Top Model Application Form with: 100% smoothed lines i.e. perfect straight lines; characters in Times new Roman font i.e. With Serifs; font size at 18; vertical and horizontal alignment ratio of 3:3 (all elements aligned)

2.5.3. Post-task Questionnaire

A post-task questionnaire (see Appendix B) was used for recording participants’

demographic information such as age, gender, education level, programme studied, papers

taken, occupation, and design experience. Preference for design tools (pen and paper

verses the tablet PC) during the design tasks and in real-world design situation, as well as

participant’s “overall enjoyment” when working on each design in comparison to another

(by ranking from 1 to 5, from the most-liked design to the least-liked design; and the

reasons for the rankings) were also recorded in the questionnaire. Such information was

used to explore whether performance – in this case, design decisions to improve a design

at different levels of formality, was affected by factors such as design experience (e.g.

Cross, 2004,; Kavakli & Gero, 2002), study major/specialization and study level (e.g.

Atman, et al., 2005; Atman, et al., 1999) and design medium preference (e.g. Bailey &

Konstan, 2003; Black, 1990; Hann & Barber, 2001; Newman, et al., 2003), during the

design process.

75

Chapter 3. Results

For the purpose of analysis, design performance was measured in terms of number

of (functional) changes made. In other words, one dependent variable with three levels

(the total number of changes, quality changes and expected changes) was measured at

each level of the independent variable (formality). Subjective measures included overall

enjoyment rankings of designs, design tool preference. Data were analyzed using SPSS

for Windows version 14.0 (SPSS Inc.). Analysis of variance (ANOVA) with repeated

measures and unplanned pair-wise comparisons were conducted to analyze the effects of

formality on outcome measures (number of changes made). Between-subject effects were

also analyzed. Friedman’s rank test for several related samples were conducted to analyze

subjective measures including rankings of designs.

3.1. Data-screening of performance data

In order to test whether the data satisfied the normality assumptions for a

parametric repeated samples t-test (see Cohen, 1988), histograms, homogeneity of

variance, the skewness and kurtosis statistics, and normality tests and plots for the scores,

as well as assumptions of sphericity were examined. For all normality information on

total changes, quality changes and expected changes, see Appendixes H, I and J

respectively.

The histograms with normal curves were created for the mean scores of each of the

three dependent measures: 1) total changes (see Appendix H), 2) quality changes (see

Appendix I) and 3) expected changes (see Appendix J), across all five levels formality.

Initial visual inspection showed roughly normal distributions and a few slightly skewed

distributions. However, it was inadequate to conclude that the distribution was non-

normal from the skewed data from a small sample of n = 30 (Cohen, 1988). Thus, all data

were also plotted against the standardized version of the data and showed that the scores

were normally distributed at each level of formality i.e. a roughly linear relationships (see

normal Q-Q plots for level of formality in Appendix H, I and J).

The comparison of variance between levels of formality in each category of change

made, indicated that the scores were similar enough (Coakes & Steed, 2001), therefore the

homogeneity of variance assumption for each measure group (DV) was not violated. See

variance across level of formality in Appendix H, I and J.

The 95% confidence interval around the skewness and kurtosis scores of neither

cell included zero indicating that the scores were not normally distributed, with data

76

skewed both ways in positive and negative directions, with negative and positive values

for kurtosis indicated leptokurtic and platykurtic distribution (Heimanz, 2001). However,

looking at the values for skewness and kurtosis in each level of formality, the absolute

values of skewness and kurtosis statistics were almost always smaller than the standard

errors, indicating that the skewness and kurtosis were comparable with the zero value in a

normal distribution.

In order to examine the relationships between the skewness, kurtosis and variance,

the Kolomogorov-Smirnov (KS) and Shapiro-Wilk (SW) tests of normality were both

conducted (see the test of normality in Appendix H, I and J). KS tests (with Lilliefors

Significance Correction) assessed the kurtosis and skewness of each data group,

demonstrating that data were suitable for parametric testing (i.e. normally distributed) as

KS statistics at each level of formality showed no significance (p > .50). SW, calculated if

sample size is less than fifty (Coakes & Steed, 2001), also showed similar values, further

indicating that the data did not violate the normality assumption of parametric tests.

Moreover, there were no missing data, and there were no outliers except for one in

the low formality scores (participant 10) in the quality changes data group. However, this

outlier was included in the analysis as it fell within the upper quartile range in the other

levels of formality (see Figure 5 in the analysis section for quality changes for the box plot

of quality changes made at each level of formality).

Overall, therefore, the data were reasonably normally distributed, hence, normality

assumptions not violated and it was justifiable to use parametric analysis.

3.2. Analysis of performance data: One-way repeated measures ANOVA

One-way ANOVA with repeated measures were conducted to examine each

dependent variable (total change, quality changes and expected changes) under each

independent variable (level of formality). An alpha level of 0.5 was used for all statistical

tests.

Sphericity

Before analyzing one-way ANOVA with repeated measures for each type of

change made across levels of formality, sphericity had to be examined. According to

Brace, Kemp and Snelgar (2006), the assumption of sphericity is that the correlations

between all of variables are roughly the same – in other words, the null hypothesis in the

sphericity assumption is that the correlations among the number of changes made in each

77

level of formality are equal. Moreover, Tabachnick and Fidell (2001) suggested when

there are more than two levels of IV (in this case, formality) the test of sphericity must be

conducted to decide which test should be used to interpret significance. Therefore, the

Mauchly’s test of sphericity was conducted on each data group. The test was significant

for total change (Approx. Chi-square = 24.644, p < .05), indicating that the sphericity

assumption was violated. On the other hand, the test yielded no significance in both

quality changes (Approx. Chi-square = 10.004, p = .351) and expected changes (Approx.

Chi-square = 5.170, p = .820), meaning that the null hypothesis of sphericity was

accepted. Hence, the assumptions of sphericity were met and the normal within-subjects

ANOVA was not violated for both data groups.

3.2.1. Analysis of “Total Changes” made across levels of formality

Table 5 shows mean and standard deviation of total changes made at each level of

formality.

Table 5Mean and standard deviation for total changes made at each level of formality

Mean Std. Deviation1. Low formality (on paper) 18.73 6.572. Low formality (on tablet PC) 15.17 4.143. Medium-low formality 14.00 4.074. Medium-high formality 13.13 3.865. High formality 11.27 3.51

Since the Mauchly’s test of sphericity (for the data group of total change) was

significant, an alternate, multivariate approach was adopted (Brace, Kemp & Snelgar,

2006). The results for the ANOVA indicated a significant main effect of formality on the

total changes made to the designs, Wilk’s Lambda = .265, F (4.26) = 17.99, p < .001,

multivariate partial η2 = .74. A strong significant linear trend was also found, F (1, 29) =

59.59, p < .001, partial η2 = .67, over the mean value of total changes made at each level of

formality (illustrated in Figure 11). A weaker but significant cubic trend was also found, F

(1, 29) = 8.529, p < .01, partial η2 = .23, suggesting that overall, the number of total

changes made were the highest when participants were presented with the low-formality

design on the paper, and decreased as formality increased. Subjects’ performance was

slightly unstable, however, suggesting that order effects (discussed later) may have played

a role in contributing to the combination of linear and cubic trends.

78

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formality

Tota

l cha

nges

Mean total change

Figure 11. Multi-line graph showing mean total changes made across levels of formality which is represented by the black bold line; each participant’s performance (in terms of total changes made across levels of formality) is also illustrated – see individual lines.

As there was lack of empirical research with only one previous study (Plimmer,

2002) on the effects of formality on the design process, one could not precisely predict

what conditions would differ from each other and in what direction – therefore, as Brace et

al. (2006) suggested, unplanned pair-wise comparisons were conducted to examine the

differences between the mean total changes at each level of formality.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons)

showed that the total number of changes made was significantly lower when participants

were presented with the high formality design, compared to other designs with lower

levels of formality: medium-high formality; medium-low formality; low formality on the

tablet PC and low formality on paper. Differences increased as the level of formality

decreased, as shown in Table 6. On the other hand, the total number of changes made in

the low formality design presented on paper was significantly higher than other levels of

formality presented on the Tablet PC: low formality on the Tablet PC; medium-low

formality; medium-high formality and high formality. Differences increased as the level

of formality increased, also shown in Table 6. Interestingly, even though there were two

79

low formality conditions, one presented on paper and one presented on the tablet, the total

number of changes made still differed significantly between these conditions – the mean

difference was 3.57 as can be seen in Table 6. This was also shown in Figure 11 where

the mean number of total changes made was much greater when made on paper than on

the Tablet PC. Furthermore, as shown in Table 6, no significant difference was found, in

terms of mean total changes, between medium-high formality and medium-low formality;

and between medium-low formality and low formality on the Tablet PC. However, the

total number of changes made at low formality was significantly higher than at medium-

high formality.

Table 6Mean differences and their significance at the .05 level in terms of total number of changes made between each condition.

(I) Factor 1 (J) Factor 1Mean Difference

(I-J)Low formality (on paper) Low formality (on Tablet PC) 3.57* Medium-low formality 4.73* Medium-high formality 5.60* High formality 7.47*Low formality (on Tablet PC) Low formality (on paper) -3.57* Medium-low formality 1.17 Medium-high formality 2.03* High formality 3.90*Medium-low formality Low formality (on paper) -4.73* Low formality (on Tablet PC) -1.17 Medium-high formality 0.87 High formality 2.73*Medium-high formality Low formality (on paper) -5.60* Low formality (on Tablet PC) -2.03* Medium-low formality -0.87 High formality 1.87*High formality Low formality (on paper) -7.47* Low formality (on Tablet PC) -3.90* Medium-low formality -2.73* Medium-high formality -1.87*

* The mean difference is significant at the .05 level.

3.2.1.1. Between-Subject Factors

In order to examine whether other factors affected the total changes made at each

level of formality, between subject effects including design experience,

major/specialization and study level were explored. Furthermore, each between-subject

factor had only two levels therefore no post-hoc tests were necessary.

3.2.1.1a. Design experience

Subjects’ design experience was examined first as it was hypothesized that there

would be a difference in the total number of changes made across levels of formality

80

between subjects who had more or less design experience. The subjects were categorized

into two groups: 1) subjects with no experience or some non-computer science/software

engineering design experience (n = 15); and 2) subjects with computer science (CS) /

software engineering (SE) design experience (n = 15). Table 7 shows mean and standard

deviation of total changes made at each level of formality according to subjects’ design

experience.

Table 7Mean and standard deviation for total changes made, and the mean difference between groups, at each levels of formality according to subjects’ design experience (total n =30): none to some (non-CS/SE) design experience (n = 15) and CS/SE design experience (n = 15)

Results from the ANOVA with design experience as the between-subject factor

showed that there was a significant formality-by-design experience interaction effect, F (4,

112) = 6.24, p < .001, partial η2 = .18, and a significant between-subject effect of design

experience, F (1, 28) = 8.49, p < .01, partial η2 = .23, on the total changes made across

levels of formality. This indicated that the total changes made across levels of formality

differed between subjects with no experience to some non-CS/SE design experience and

subjects with CS/SE design experience – and more specifically, subjects with CS/SE

design experience made consistently more changes across levels of formality compared to

subjects with no experience or some non-CS/SE experience (see Figure 12).

Design Experience(X) None to some (non-

CS/SE) design experience(Y) CS/SE design

experience

MeanStd.

Deviation MeanStd.

DeviationMean Difference

(Y-X)1. Low formality (paper) 15.00 5.37 22.47 5.54 7.472. Low formality (tablet) 13.20 3.75 17.13 3.64 3.933. Medium-low formality 13.27 4.76 14.73 3.24 1.464. Medium-high formality 11.47 3.44 14.80 3.61 3.335. High formality 10.47 3.34 12.07 3.61 1.60

81

0

2

4

6

8

10

12

14

16

18

20

22

24

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n to

tal c

hang

es

None to some (non-CS/SE)design experienceCS/SE design experience

Figure 12. Multi-line graph of mean total changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience

A significant linear trend was also found when the effects of formality and design

experience were combined, F (1, 29) = 10.43, p < .005, partial η2 = .27. This suggested

that the linear trend was also significant in both groups, where subjects made less (more)

changes as the level of formality increased (decreased), regardless of magnitude

differences. There were between-group differences across levels of formality, as shown in

Table 7, and differences tend to decrease as formality increased (refer also to Figure 12).

There was also a significant but weak formality-by-design experience quadratic trend, F

(1, 29) = 4.41, p < .045, partial η2 = .14, and cubic trend, F (1, 29) = 4.50, p < .043, partial

η2 = .14. Such trends are also illustrated in Figure A, where the total changes increased

gradually from high to low formality on the Tablet but more markedly higher at low

formality on paper (more detectable in subjects with CS/SE design experience); points of

increase are detectable in the negative linear trend from low formality to high formality.

Two other between-subject factors – major/specialization and study level, were

explored mainly through visual inspection due to various reasons: the number of subjects

in each group could not be balanced; there was overlapping of subject factors, i.e. explicit,

isolative (i.e. nested) grouping of subjects was near impossible in the current study as

major/specialization, study level and design experience were all intimately-correlated, and

even if it were possible, a much larger sample would have been needed – therefore

82

subjects were grouped according to one factor only, and grouped data was examined

through multi-line graphs.

3.2.1.1b. Study major/specialization

Since the experimental task involved HTML (web) form design, it was of interest

to see whether total changes made across levels of formality differed between subjects

who had more or less HTML knowledge. Therefore to explore such between-subject

effect, subjects were grouped into two groups: 1) subjects with a non-CS/SE related major

(n = 10); and 2) subjects with a CS/SE major (n = 20). Table 8 below shows mean and

standard deviation of total changes made in each group across levels of formality.

Table 8Mean and standard deviation for total changes made, and the mean difference between groups, at each level of formality according to subjects’ major/specialization in university (Total n =30): non-CS/SE related major (n = 10) and CS/SE related major (n = 20)

Results from the one way ANOVA with study major as the between-subject factor showed

that there was a significant formality-by-major/specialization effect, F (4, 112) = 4.10, p

< .01, partial η2 = .13, as well as significant formality-by-major/specialization trends:

linear trend, F (1, 28) = 5.09, p = .032, partial η2 = .15, and quadratic trend, F (1, 28) =

5.37, p < .028, partial η2 = .16; however, no significant between-subject effect was found

(illustrated in Figure 13). Visual inspection of Figure 13 suggested that there was linear

trend across levels of formality, and the total changes made increased more rapidly at the

lower formalities in the CS/SE major group, while the other group showed a less

consistent linear trend. Interestingly, at medium-low formality subjects performed at the

same level – the total changes made was similar in the CS/SE major group and the non-

CS/SE major group. This could also explain the non-significant results from the between-

subject effects tests. Overall, the between-groups difference decreased as the formality

level increased (see mean differences in Table 8) – the gap between the two lines was

smaller at the higher levels of formality compared to lower levels of formality, as

explained by the significant formality-by-major/specialization interaction.

Major/Specialization(X) Non-CS/SE related major (Y) CS/SE related major

MeanStd.

Deviation MeanStd.

Deviation Mean Difference

(Y-X)1. Low formality (paper) 15.20 5.67 20.50 6.39 5.302. Low formality (tablet) 13.50 3.47 16.00 4.28 2.503. Medium-low formality 14.30 4.52 13.85 3.94 0.454. Medium-high formality 12.10 3.38 13.65 4.06 1.555. High formality 10.70 4.08 11.55 3.27 0.85

83

0

2

4

6

8

10

12

14

16

18

20

22

24

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n to

tal c

hang

es

Non-CS/SE related major

CS/SE related majors

Figure 13. Multi-line graph of mean total changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors

3.2.1.1c. Study Level

As study level may have also played a role in producing particular trends among different

groups, subjects were classified into two groups: 1) undergraduates (n = 22); and 2)

graduates/post-graduates (n = 8). Table 9 below shows mean and standard deviation of

total changes made in each group across levels of formality.

Table 9 Mean and standard deviation for total changes made, and the mean difference between groups, at each level of formality according to subjects’ study level (total n=30): undergraduate (n=22) and graduate/postgraduate (n=8).

No significant statistics, such as formality-by-study level effect and trends, were

found from the results of ANOVA with study level as the between-subject factor, except

for between-subjects effects, F (1, 28) = 6.10, p = 0.02, partial η2 = .18. However,

Study level(X) Undergraduate (X) Undergraduate

MeanStd.

Deviation MeanStd.

Deviation Mean Difference

(Y-X)1. Low formality (paper) 17.32 6.20 22.63 6.30 5.312. Low formality (tablet) 14.32 3.59 17.50 4.90 3.183. Medium-low formality 13.60 4.28 15.13 3.40 1.534. Medium-high formality 12.00 3.28 16.25 3.77 4.255. High formality 10.36 3.23 13.75 3.20 3.39

84

examining statistics alone was not conclusion there was an unbalanced number of subjects

in each group. Visual inspection of Figure 14 suggested that there was a strong linear

trend in the undergraduate group, while the graduate/postgraduate group showed a weaker

linear trend with a slight increase in the mean total change at medium-high formality, after

the medium-low formality condition. It was also visible that the total changes made

increased rapidly at low formality (on paper) in both groups. Also, the between-subject

differences appeared to be greater nearer the two ends of the formality spectrum: low

formality on paper (mean group difference = 5.55) and low formality on the Tablet PC

(mean group difference = 3.49); and medium-high formality (mean group difference =

4.43) and high formality (mean group difference = 3.30); and the smallest between-group

difference at medium-low formality (mean group difference = 1.33) – see mean

differences in Table 9. This also suggested that there was some formality-by-study level

interaction. Although differing in magnitude, overall, a rough linear trend was visible for

both groups – as formality increased (decreased), total changes made decreased

(increased).

0

2

4

6

8

10

12

14

16

18

20

22

24

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n to

tal c

hang

es

Undergraduate

Graduate/postgraduate

Figure 14. Multi-line graph of mean total changes made across levels of formality according to subjects study level: undergraduate and graduate/postgraduate.

3.2.1.2. Multiple Regression analysis

The similar trends with the three factors further suggested that they were closely

related. The data set was re-grouped according to a combination of design experience,

study level and major/specialization (see Appendix K), and multiple regression analysis

85

was then conducted to examine and separate individual effects that contributed to the

overall effect of formality on the total changes made. In other words, these analyses

sought to discover how much each between-subject factor helped explain the effect of

formality on the total changes made.

Formality and the three between-subjects variables (design experience and study

level, and major/specialization) were entered one after the other respectively into SPSS.

Before looking at the actual results, in addition to the data screening earlier for normality

and outliers, multicollinearity was first examined. According to Brace et al. (2006), the

closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the

relationship between this and the other predictor variables; and the higher the VIF value

(value from 1.0), the stronger relationship is between predictor variables; and such values

becomes a worry. However, results indicated high tolerance values (over .90), and low

VIF values (less than 1.08), therefore there was no multicollinearity issues.

Using the stepwise method, a significant model which included formality, design

experience and study level, emerged, F (3, 31) = 31.67, p < .0001. The model explained

73%% of the variance (Adjusted R2 = .730). Table 10.1 shows the adjusted R square and

change statistics of each predictor when added to the model. Formality level (model 1)

accounted for 36.1% of the variance (Adjusted R2 = .361, p <.0001), and the inclusion of

design experience in model 2 resulted in an additional 30.1% of the variance being

explained (R2 change = .301, F (1, 32) = 30.13, p < .0001). Study level helped explained a

further 7.4% of the variance when added upon formality and design experience (R2 change

= .074, F (1, 31) = 9.95, p = .005). However, study major/specialization was excluded

from the model as it did not have a significant impact when added (R2 change = .00, F (1,

30) = .003, p = .96) – hence, not a good predictor to explain total changes made across

levels of formality.

Table 10.1. R, Adjusted R Square and R Square change for total changes made across levels of formality, with formality level, design experience, study level and major/specialization as predictors entered

Model RAdjusted R

SquareStd. Error of the Estimate Change Statistics

R Square Change F Change

Sig. F Change

1 .616(a) .361 2.27 .379 20.18 .0002 .825(b) .660 1.65 .301 30.13 .000

86

3 .868(c) .730 1.47 .074 9.27 .0054 .868(d) .721 1.50 .000 .003 .960

a Predictors: (Constant), Formality Levelb Predictors: (Constant), Formality Level, Design experiencec Predictors: (Constant), Formality Level, Design experience, Study leveld Predictors: (Constant), Formality Level, Design experience, Study level, Major/specialization

Table 10.2 gives information for the predictor variables (formality and between-

subject variables) included in the significant model. The result suggests that formality

alone (the manipulated variable) has a strong significant impact on the total number of

changes made (β = -.62, t = -6.92, p < .0001). The negative statistics further suggests that

as formality level increases, the total changes decreases. The results for design experience

(β = .5, t = 5.57, p < .0001) and study level (β = .28, t = 3.05, p < .005) further indicates

that on top of the effects of formality on total changes made – people with more design

experience and/or at a high level of study (e.g. graduates) are more likely to make greater

number of changes than those with less design experience and/or at a lower level of study

(e.g. undergraduates).

Table 10.2 The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the mode for explaining total changes made.

B Std Error B β tFormality -1.217 .176 - .616** -6.92Design experience 2.837 .510 .503** 5.57Study Level 1.553 .510 .275* 3.05

*p = .005, ** p < .0001

3.2.2. Analysis of “Quality Changes” made across levels of formality

Table 11 shows mean and standard deviation of quality changes made at each level

of formality.

Table 11Mean and standard deviation for quality changes made at each level of formality.

Formality level Mean Std. Deviation1. Low formality (paper) 15.73 5.502. Low formality (on tablet PC) 13.05 4.013. Medium-low formality 12.90 3.814. Medium-high formality 10.80 3.925. High formality 9.02 3.54

Since the Mauchy’s test of sphericity was not significant, the traditional test of

within-subjects effect was conducted and the results from ANOVA showed that there was

87

a significant main effect of formality on the number of quality changes made, F (4, 116) =

31.763, p < .001, partial η2 = .48. A significant linear trend was also found, F (1, 29) =

76.91, p < .001, partial η2 = .73, over the mean quality changes at each level of formality,

indicating that subjects made most quality changes in the low formality design on paper,

followed by low formality design on the Tablet PC, and the numbers dropped as formality

increased – see Figure 15; and also refer to the bold line for the means across levels of

formality. However, no significant quadratic, cubic nor order 4 trends were found.

0

2

4

6

8

10

12

14

16

18

20

22

24

26

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formality

Qua

lity

chan

ges

Mean quality changes

Figure 15. Multi-line graph showing mean quality changes made across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of quality changes made across levels of formality) is also illustrated – see individual lines.

As noted before, due to the lack of previous empirical research, unplanned pair-

wise comparisons were conducted to examine the difference in the mean quality changes

between levels of formality.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons)

showed that the number of quality changes made was significantly lower when

participants were presented with the high formality design, compared to the designs with

lower levels of formality: medium-high formality; medium-low formality; low formality

on the tablet PC and low formality on paper. Difference increased as the level of formality

decreased, as shown in Table 12. On the other hand, the mean number of quality changes

88

made in the low formality design presented on paper was significantly higher than all

other levels of formality presented on the Tablet PC: low formality on the Tablet PC;

medium-low formality; medium-high formality and high formality. Difference increased

as the level of formality increased, also shown in Table 6. The increasing and decreasing

differences further emphasized the significant linear trend found. Furthermore, the mean

number of quality changes made was different between each level of formality, except

between medium-low formality and low formality on the Tablet PC (mean difference

= .33) –illustrated in Figure 15. It was also interesting to note that, similar to the data for

total changes, even though there were two low formality conditions, one presented on

paper and one presented on the tablet, the number of quality changes made still differed

significantly between these conditions – the mean difference was 2.68 as can be seen in

Table 12. This was also shown in Figure 15 where the mean number of quality changes

made was higher when made on paper than on the Tablet PC.

89

Table 12

Mean differences and their significance at the .05 level in terms of the number of quality changes made between each condition.

(I) Factor 1 (J) Factor 1Mean Difference

(I-J)Low formality (on paper) Low formality (on Tablet PC) 2.68* Medium-low formality 2.83* Medium-high formality 4.93* High formality 6.72*Low formality (on Tablet PC) Low formality (on paper) -2.68* Medium-low formality 0.15 Medium-high formality 2.25* High formality 4.03*Medium-low formality Low formality (on paper) -2.83* Low formality (on Tablet PC) -0.15 Medium-high formality 2.10* High formality 3.88*Medium-high formality Low formality (on paper) -4.93* Low formality (on Tablet PC) -2.25* Medium-low formality -2.10* High formality 1.78*High formality Low formality (on paper) -6.72* Low formality (on Tablet PC) -4.03* Medium-low formality -3.88* Medium-high formality -1.78*

* The mean difference is significant at the .05 level.

3.2.2.1. Between-Subject Factors

In order to examine whether other factors affected the quality changes made at

each level of formality, between subject effects including design experience, study level

and major/specialization were explored. Furthermore, each between-subject factor had

only two levels therefore no post-hoc tests had been conducted.

3.2.2.1a. Design Experience

Subjects’ design experience was examined first as it was hypothesized that there

will be a difference in the number of quality changes made across levels of formality

between subjects who had more or less design experience. Thus, subjects were

categorized into two groups: 1) subjects with no experience or some non-computer

science/software engineering design experience (n = 15); and 2) subjects with computer

science (CS) / software engineering (SE) design experience (n = 15). Table 13 shows

mean and standard deviation of quality changes made at each level of formality according

to subjects’ design experience.

Table 13

90

Mean and standard deviation for quality changes made, and the mean difference between groups, at each level of formality according to design experience (total n=30): none to some (non-CS/SE) design experience (n=15) and CS/SE design experience (n=15)

Results from ANOVA with design experience as the between-subject factor

showed that there was a significant formality-by-design experience interaction effect, F (4,

112) = 4.07, p < .005, partial η2 = .13, along with the significant between-subject effects of

design experience, F (1, 28) = 7.31, p < .02, partial η2 = .21, on the number of quality

changes made across levels of formality. This indicated that the quality changes made

across levels of formality differed between subjects with no experience or some

non-CS/SE design experience and subjects with CS/SE design experience. More

specifically, subjects with CS/SE design experience made consistently more quality

changes across levels of formality compared to subjects with none-to-some (non-CS/SE)

experience (see Figure 16).

0

2

4

6

8

10

12

14

16

18

20

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n qu

ality

cha

nges

None to some (non-CS/SE)design experienceCS/SE design experience

Figure 16. Multi-line graph of mean quality changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience

Design Experience(X) None to some (non-

CS/SE) design experience(Y) CS/SE design

experience

MeanStd.

Deviation MeanStd.

Deviation Mean Difference

(Y-X)1. Low formality (paper) 12.77 4.83 18.70 4.53 5.932. Low formality (tablet) 11.20 3.59 14.90 3.62 3.703. Medium-low formality 12.40 4.47 13.40 3.09 1.004. Medium-high formality 9.37 2.95 12.23 4.33 2.865. High formality 8.00 3.34 10.03 3.54 2.03

91

Significant trends were also found with the combined effects of formality and

design experience: linear trend, F (1, 28) = 7.04, p < .03, partial η2 = .20; as well as a weak

quadratic trend, F (1, 28) = 4.57, p < .05, partial η2 = .14 – illustrated in Figure 16 where

quality changes increased rapidly at low formality (on paper). This suggested that there

was a linear trend in both groups regardless of magnitude differences, where subjects

made less (more) changes as the level of formality increased (decreased). In addition to

the statistics, Figure 16 shows that there was a stronger linear trend across levels of

formality in the subjects CS/SE design experience however, the linear trend was less

consistent in subjects with none to some (non-CS/SE) design experience. At medium-low

formality, there was an increase in the mean quality changes made by subjects with non-

to-some (non-CS/SE) design experience and thus, between-group difference at such level

of formality was the smallest compared to other levels. The between-group differences at

low formality on paper was the largest (mean difference = 5.20), followed by low

formality on the Tablet PC (mean difference = 3.50), and the mean differences between

groups tended to decrease as formality increased (refer to Table 13 and Figure 16) – this

further highlighted the formality-by-design experience interaction.

Two other between-subjects factors – major/specialization and study level, were

explored primarily through visual inspection of multi-line graphs due to various reasons:

the number of subjects in each group could not be balanced; there were overlapping of

subject factors, i.e. explicit, isolative (i.e. nested) grouping of subjects was near

impossible in the current study as major/specialization, study level and design experience

were all intimately-correlated, and even if it was possible, a much larger sample was

needed – therefore subjects were grouped according to one factor only.

3.2.2.1b. Study major/specialization

Since the experimental task involved HTML (web) form design, it was of interest

to see whether quality changes made at each level of formality differed between subjects

who had more or less HTML knowledge. Therefore subjects were grouped into two

groups: 1) subjects with a non-CS/SE related major (n = 10); and 2) subjects with a CS/SE

major (n = 20). Table 14 shows the mean and standard deviation of quality changes made

in each group across levels of formality.

Table 14

92

Mean and standard deviation for quality changes made, and the mean difference between groups, at each level of formality according to subjects’ major/specialization in Auckland University: Non-CS/SE related major (n = 10) and CS/SE related majors (n = 20)

Results from ANOVA with study major/specialization as the between-subject factor

indicated that there was a significant formality-by-major interaction, F (4, 112) = 3.00, p <

.025, partial η2 = .10, and weak significant trends including a linear trend, F (1, 28) = 4.23,

p < .05, partial η2 = .13, a quadratic trend, F (1, 28) = 4.76, p < .05, partial η2 = .15, on the

number of quality changes made across levels of formality. However, no between-

subjects effects were found. Figure 17 highlighted the significant statistics and showed

that there was a strong linear trend of quality changes made across levels of formality and

a rapid increase at the low formality (on paper) in the CS/SE major group; where as, in the

non-CS/SE major group, there was a weaker linear trend with one non-linear point with

respect to other points. Interestingly, at medium-low formality subjects performed at the

same level – the mean quality changes made was similar in the CS/SE major group and the

non-CS/SE major group. Overall, subjects who majored in CS/SE made more quality

changes than subjects who majored in non-CS/SE areas of study. Between-group

difference (see mean differences in Table 14) was the largest at low formality presented on

paper (mean difference = 3.78) and decreased at low formality present on the Tablet PC

(mean difference = 2.05). Next, as formality level increased, the between-group

differences decreased – the gap between the two lines was smaller at the higher levels of

formality compared to lower levels of formality which further suggested that there was

some interaction (illustrated in Figure 17).

Major/Specialization(X) Non-CS/SE related major (Y) CS/SE related major

Mean Std. Deviation Mean Std. Deviation Mean Difference

(Y-X)1. Low formality (paper) 13.10 5.18 17.05 5.29 3.952. Low formality (tablet) 11.40 3.21 13.88 4.19 2.483. Medium-low formality 13.30 4.70 12.70 3.40 - 0.604. Medium-high formality 10.25 3.76 11.08 4.07 0.835. High formality 8.30 4.24 9.38 3.19 1.08

93

0

2

4

6

8

10

12

14

16

18

20

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n qu

ality

cha

nges

Non-CS/SE related major

CS/SE related majors

Figure 17. Multi-line graph of mean quality changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors

3.2.2.1c. Study Level

As study level may have also played a role in producing particular trends among

groups, subjects were classified into two groups: 1) undergraduates (n = 22); and 2)

graduates/post-graduates (n = 8). Table 15 shows mean and standard deviation of quality

changes made in each group across levels of formality.

Table 15Mean and standard deviation of quality changes made, and the mean difference between groups, at each level of formality according to subjects’ study level: undergraduate (n=22) and graduate/postgraduate (n=8).

Although no significant formality-by-study level interaction was found from the

ANOVA results with study level as the between-subject factor, the between-subjects tests

was significant, F (1, 28) = 6.87, p < .015, partial η2 = .20, suggesting that number of

Study Level(X) Undergraduate (X) Undergraduate

Mean Std. Deviation Mean Std. Deviation Mean Difference

(Y-X)1. Low formality (paper) 14.64 5.19 18.75 5.52 4.112. Low formality (tablet) 12.39 3.70 14.88 4.52 2.493. Medium-low formality 12.43 3.83 14.19 3.68 1.764. Medium-high formality 9.41 3.00 14.63 3.73 5.225. High formality 8.07 3.08 11.63 3.57 3.56

94

quality changes made at each level of formality between undergraduates and

graduates/post-graduates differed significantly. Figure 18 shows that the number of

quality changes made by graduates/postgraduates at each level of formality was

significantly higher compared to undergraduates. Visual inspection of Figure 18 also

suggested that there was a roughly linear trend in both the undergraduate and

graduate/postgraduate. However, trend tests showed that there was a no linear but a

significant cubic trend, F (1, 28) = 6.34, p < .02, partial η2 = .19, demonstrated in Figure 18

where points of inflection are noticeable e.g. quality changes made by: undergraduate

subjects at medium-low formality; and graduate/postgraduate subjects at medium-high

formality.

Furthermore, according to study level, the overall trend of quality changes made

across levels of formality was similar to previous between-subjects trends (design

experience and major/specialization) – refer to Figure 16, 17 and 18 – all with an increase

in the number of quality changes made at medium-low formality occurring after a drop at

the low formality condition presented on the Tablet PC. The smallest between-group

difference was at medium-low formality (mean difference = 2.06), where as the largest

between-group difference was at medium-high formality (mean difference = 4.39) – see

Table 15 for mean differences across levels of formality. Moreover, the non-parallel lines

further suggested that there was some (small) formality-by-study level interaction.

Although differing in magnitude, the overall linear trend was visible for both groups of

individuals – as formality increased (decreased), the number of expected changes made

decreased (increased).

95

0

2

4

6

8

10

12

14

16

18

20

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n qu

ality

cha

nges

Undergraduate

Graduate/postgraduate

Figure 18. Multi-line graph of mean quality changes made across levels of formality according to subjects’ study level: undergraduate and graduate/postgraduate

3.2.2.2. Multiple Regression Analysis

The similar trends with the three factors further suggested that they were closely

related. The data set was re-grouped according to a combination of design experience,

study level and major/specialization (see Appendix L), and multiple regression analysis

was then conducted to examine and separate individual effects that contributed to the

overall effect of formality on the quality changes made. In other words, the analyses

sought to discover how much each between-subject factor helped explain the effect of

formality on the number of quality changes made.

Formality and the three between-subjects variables (design experience and study

level, and major/specialization) were entered one after the other respectively into SPSS.

Before looking at the actual results, in addition to the data screening earlier for normality

and outliers, multicollinearity was first examined. According to Brace et al. (2006), the

closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the

relationship between this and the other predictor variables; and the higher the VIF value

(value from 1.0), the stronger relationship is between predictor variables; and such values

becomes a worry. However, results indicated high tolerance values (over .90), and low

VIF values (less than 1.08), therefore there was no multicollinearity issues.

96

Using the stepwise method, a significant model which included formality, design

experience and study level, emerged, F (3, 31) = 27.15, p < .0001. The model explained

69.8% of the variance (Adjusted R2 = .698). Table 16.1 shows the adjusted R square and

change statistics of each predictor when added to the model. Formality level (model 1)

accounted for 37% of the variance (Adjusted R2 = .370, p <.0001), and the inclusion of

design experience in model 2 resulted in an additional 24.7% of the variance being

explained (R2 change = .247, F (1, 32) = 21.75, p < .0001). Study level helped explained a

further 7.4% of the variance when added upon formality and design experience (R2 change

= .074, F (1, 31) = 9.95, p = .005). However, study major/specialization was excluded

from the model as it did not have a significant impact when added (R2 change = .005, F (1,

30) = .54, p = .47) – hence, not a good predictor to help explain the number of quality

changes made across levels of formality.

Table 16.1Adjusted R Square and R Square change

Model RAdjusted R

SquareStd. Error of the Estimate Change Statistics

R Square Change F Change

Sig. F Change

1 .623(a) .370 2.74703 .388 20.960 .0002 .797(b) .613 2.15246 .247 21.749 .0003 .851(c) .698 1.90283 .088 9.947 .0044 .854(d) .693 1.91710 .005 .540 .468

a Predictors: (Constant), Formality Levelb Predictors: (Constant), Formality Level, Design experiencec Predictors: (Constant), Formality Level, Design experience, Study leveld Predictors: (Constant), Formality Level, Design experience, Study level, Major/specialization

Table 16.2 gives information for the predictor variables (formality and between-

subject variables) included in the significant model. The result suggests that formality

alone (the manipulated variable) has a strong significant impact on the total number of

changes made (β = -.62, t = -6.61, p < .0001). The negative statistics further suggests that

as formality level increases, the number of quality changes made decreases. The results

for design experience (β = .45, t = 4.68, p < .0001) and study level (β = .30, t = 3.15, p

< .005) further indicates that on top of the effects of formality on quality changes made –

people with more design experience and/or at a high level of study (e.g. graduates) are

more likely to make greater number of quality changes than those with less design

experience and/or at a lower level of study (e.g. undergraduates).

Table 16.2The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the model.

97

B Std Error B β tFormality -1.50 .227 -.623 -6.61Design experience 3.08 .659 .447 4.68Study Level 2.08 .659 .302 3.15

** p < .0001, * p < .005

3.2.3. Analysis of “Expected Changes” made across levels of formality

Table 17 below shows mean and standard deviation of expected changes made at each

level of formality.

Table 17Mean and standard deviation for expected changes made across levels of formality

Formality level Mean Std. Deviation1. Low formality (paper) 13.55 4.242. Low formality (on tablet PC) 11.18 3.253. Medium-low formality 10.22 3.344. Medium-high formality 9.02 3.455. High formality 8.00 3.30

In order to test whether formality had an effect on the number of expected changes

made, one way ANOVA with repeated measures was conducted. Results showed that

there was a significant main effect of formality on the number of expected changes made

to the designs, F (4, 116) = 29.28, p > .001, partial η2 = .50. As one could not tell how

levels of formality affected the number of expected changes made, trends test were also

examined. A significant linear trend was found, F (1, 29) = 92.70, p < .001, partial η2

= .76, over the mean value (expected change) for each level of formality.

Figure 19 shows that, overall, participants made the most expected changes in the

low-formality design on the paper, and the numbers dropped as formality increased.

However, no significant quadratic, cubic nor order 4 trends were found.

98

0

2

4

6

8

10

12

14

16

18

20

22

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formality

Exp

ecte

d ch

ange

s

Mean expected changes

Figure 19. Multi-line graph showing mean expected changes across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of expected changes made across levels of formality) is also illustrated – see individual lines.

Again, due to the lack of previous empirical research, unplanned pair-wise

comparisons were conducted to examine the differences in the mean quality changes

between levels of formality.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons)

revealed that participants made significantly more expected changes when they were

presented with the low formality paper design, compared to designs with other levels of

formality presented on the Tablet PC: low formality on the Tablet PC; medium-low

formality; medium-high formality and high formality. Difference increased as the level of

formality increased, as shown in Table 18. On the other end, participants made

significantly fewer expected changes when they were presented with the high formality

design, compared to designs with medium-low formality; low formality on the tablet PC

and low formality on paper. Differences increased as the level of formality decreased;

however, no significant difference was found between high formality and medium-high

formality. Furthermore, as shown in Table 18, no significant difference was found

between medium-high formality and medium-low formality; and between medium-low

formality and low formality on Tablet PC. This suggested that subjects’ performance (in

terms of making expected changes) was comparable when they were presented with

99

designs with higher formalities (high formality and medium-high formality), and similarly

in designs with lower formalities on the Tablet PC (low formality and medium-low

formality; plus medium-low formality and medium-high formality).

Similar to total changes and quality changes, interestingly, although there were two

low formality conditions, one presented on paper and one presented on the tablet, the

number of expected changes made still differed significantly – the mean difference was

2.37 as can be seen in Table 18. This was also shown in Figure 19 where fewer expected

changes was made on the Tablet PC than on paper.

Table 18Mean differences and their significance at the .05 level in terms of the number of expected changes made between each condition.

(I) Factor 1 (J) Factor 1Mean Difference

(I-J)Low formality (on paper) Low formality (on Tablet PC) 2.37* Medium-low formality 3.33* Medium-high formality 4.53* High formality 5.55*Low formality (on Tablet PC) Low formality (on paper) -2.37* Medium-low formality 0.97 Medium-high formality 2.17* High formality 3.18*Medium-low formality Low formality (on paper) -3.33* Low formality (on Tablet PC) -0.97 Medium-high formality 1.20 High formality 2.22*Medium-high formality Low formality (on paper) -4.53* Low formality (on Tablet PC) -2.17* Medium-low formality -1.20 High formality 1.02High formality Low formality (on paper) -5.55* Low formality (on Tablet PC) -3.18* Medium-low formality -2.22* Medium-high formality -1.02

3.2.3.1. Between-Subjects Factors

In order to examine whether other factors affected the number of expected changes

a participant made, between subject effects including design experience, study level and

specialization were explored.

3.2.3.1a. Design Experience

Subjects’ design experience was examined first as it was hypothesized that there

will be a difference between the numbers of (expected) changes made by subjects who

have more or less design experience. The subjects were grouped into two groups: 1) no

experience or some non-CS/SE design experience (n = 15); and 2) CS/SE design

100

experience (n = 15). Table 19 below shows mean and standard deviation of expected

changes made at each level of formality according to subjects’ design experience.

Table 19Mean and standard deviation for expected changes made, and the mean difference between groups, at each level of formality according to design experience (Total n =30): CS/SE design experience (n = 15), none to some (non-CS/SE) design experience (n = 15).

ANOVA with design experience as the between-subject factor was conducted to

examine whether there was a difference in subjects’ performance according to design

experience. Results showed that in addition to the significant main effect of formality, F

(4, 116) = 29.28, p > .001, there was also a significant between-subject effect, F (1, 28) =

7.64, p < .01, partial η2 = .21, suggesting that subjects with CS/SE design experience made

more changes across levels of formality compared to subjects with no experience or some

non-CS/SE experience as can shown in Figure 20. Furthermore, a weak formality-by-

design experience linear trend was found (although not strictly ‘statistically significant’ at

the alpha level of .05), F (1, 28) = 4.15, p = .051, partial η2 = .13. Visual inspection of

Figure 20 further suggested that there was a stronger linear trend across levels of formality

in subjects with CS/SE design experience than subjects with no experience or some non-

CS/SE design experience – the mean number of expected changes made by subjects with

no experience or some CS/SE design experience at low formality was the same in

medium-low formality. However, on the whole, there was still a linear trend where

subjects made fewer (more) changes as the level of formality increased (decreased).

Additionally, as illustrated in Figure 20, there were magnitude differences between the

two groups across levels of formality. Although, no statistically significant formality-by-

design experience interaction was found, Figure 20 shows that the two lines, representing

subjects with different design experience, appeared to be non-parallel, and thus, there was

some interaction.

Design Experience(X) None to some (non-

CS/SE) design experience(Y) CS/SE design

experience

Mean Std. Deviation Mean Std. Deviation Mean Difference

(Y-X)1. Low formality (paper) 11.50 4.31 15.60 3.09 4.102. Low formality (tablet) 9.47 2.89 12.90 2.69 3.433. Medium-low formality 9.47 3.72 10.97 2.83 1.504. Medium-high formality 7.73 2.96 10.30 3.51 2.575. High formality 7.07 3.65 8.93 2.72 1.86

101

0

2

4

6

8

10

12

14

16

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n ex

pect

ed c

hang

es

None to some (non-CS/SE)design experienceCS/SE design experience

Figure 20. Multi-line graph of mean expected changes made across levels of formality according to subject’s design experience

Two other between-subjects, major/specialization and study level, were explored

through visual inspection due to various reasons: the number of subjects in each group

could not be balanced; there were overlapping of subjects i.e. explicit, isolative (i.e.

nested) grouping of subjects was near impossible in the current study as

major/specialization, study level and design experience are all intimately-correlated, and

even if it was possible, a much larger sample would be needed – therefore subjects were

grouped according to one factor only.

3.2.3.1b. Study major/specialization

Since the experimental tasks involved HTML (web) form design, it was of interest

to see whether the number of changes made across the levels of formality differed between

subjects who had more or less HTML form knowledge. Therefore to explore such

between-subject effects, subjects were grouped into two groups: 1) those with non-CS/SE

related major (n = 10); and 2) those with CS/SE major (n = 20). Table 20 shows mean and

standard deviation of expected changes made at each level of formality according to

design experience.

Table 20Means and standard deviations for expected changes made, and the mean difference between groups, at each level of formality according to major/specialization (n=30): non-CS/SE related major (n=10); CS/SE related major (n=20)

102

No significant effects were found after conducting one way ANOVA with study

major/specialization as the between-subject factor. However, since the number of subjects

in each group was not balanced (n = 20, n = 10), statistics produced was not conclusive.

Visual inspection of a multi-line graph (Figure 21) suggested that the number of expected

changes made by subjects who majored/specialized in CS/SE was higher than subjects

who majored in non-CS/SE subjects across the levels of formality, except at medium-low

level of formality, where there was no significant mean difference (mean difference =

0.13) between the two groups – refer also to Table 20. Furthermore, there was a strong

linear trend in the CS/SE major group, while the other group showed a less consistent

linear trend with a sudden increase in expected changes at medium-low formality. In

addition, this indicated that there was also some formality-by-major/specialization

interaction as the lines were non-parallel – gaps bigger at lower formalities compared to

higher formalities – illustrated in Figure 21.

0

2

4

6

8

10

12

14

16

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n ex

pect

ed c

hang

es

Non-CS/SE related major

CS/SE related majors

Major/Specialization(X) Non-CS/SE related major (Y) CS/SE related major

Mean Std. Deviation Mean Std. Deviation Mean Difference

(Y-X)1. Low formality (paper) 11.60 4.20 14.53 4.01 2.932. Low formality (tablet) 9.85 3.05 11.85 3.22 2.003. Medium-low formality 10.30 4.28 10.18 2.88 - 0.134. Medium-high formality 8.80 3.79 9.13 3.36 0.335. High formality 7.30 4.26 8.35 2.77 1.05

103

Figure 21. Multi-line graph of mean expected changes made across levels of formality according to subject’s major/specialization: non-CS/SE related major and CS/SE related majors

3.2.3.1c. Study Level

As study level may have played a role in producing particular trends among different

groups, subjects were also categorized into two groups: 1) undergraduates (n = 22); and 2)

graduates/post-graduates (n = 8). Table 21 below shows mean and standard deviation of

expected changes made in each group at each level of formality.

Table 21Mean and standard deviation of expected changes made, and the mean difference between groups, at each level of formality according to study levels (n=30): undergraduate (n = 22); graduate/postgraduate (n = 8).

Results from the ANOVA with study level as the between-subject factor showed

no statistically significant formality-by-study level interaction, but a significant between-

subjects effect was found, F (1, 28) = 5.18, p < .03, partial η2 = .16, suggested that the

number of expected changes made across levels of formality between the undergraduates

and graduates/post-graduates differed in magnitude – graduates/postgraduates made more

expected changes across the levels of formality compared to undergraduates (illustrated in

figure U). Although no statistically significant linear trend found, a weak significant cubic

trend was detected, F (1, 28) = 4.12, p = .052 (slightly above the alpha level of .05).

Visual inspection of Figure 22 further indicated that the weak cubic trend was contributed

by the strong linear trend in the undergraduate group, and a less consistent linear trend in

the graduate/postgraduate group with a steep dip at the low formality condition presented

on the tablet PC, followed by a slight increase in the mean expected changes at medium-

high formality. The two lines representing undergraduates and graduates/postgraduates

also appeared to be non-parallel, suggesting that there may be some formality-by-study

level interaction. On the whole, the main linear trend was still visible for both groups but

differing in magnitude – as formality increased (decreased), the number of expected

changes made decreased (increased).

Study Level(X) Undergraduate (X) Undergraduate

MeanStd.

Deviation MeanStd.

Deviation Mean Difference

(Y-X)1. Low formality (paper) 12.89 4.11 15.38 4.29 2.492. Low formality (tablet) 10.80 3.09 12.25 3.66 1.453. Medium-low formality 9.66 3.16 11.75 3.53 2.094. Medium-high formality 7.96 2.86 11.94 3.40 3.985. High formality 7.21 2.79 10.19 3.79 2.98

104

0

2

4

6

8

10

12

14

16

Lowformality(paper)

Lowformality

Medium-lowformality

Medium-highformality

Highformality

Levels of formality

Mea

n ex

pect

ed c

hang

es

Undergraduate

Graduate/Postgraduate

Figure 22. Multi-line graph of mean expected changes made across levels of formality according to subjects’ study levels: undergraduate and graduate/postgraduate

3.2.3.2. Multiple Regression Analysis

The similar trends with the three factors further suggested that they were closely

related. The data set was re-grouped according to a combination of design experience,

study level and major/specialization (see Appendix M), and multiple regression analysis

was then conducted to examine and separate individual effects that contributed to the

overall effect of formality on the expected changes made. In other words, these analyses

sought to discover how much each between-subject factor helped explain the effect of

formality on the number of expected changes made.

Formality and the three between-subjects variables (design experience and study

level, and major/specialization) were entered one after the other respectively into SPSS.

Before looking at the actual results, in addition to the data screening earlier for normality

and outliers, multicollinearity was first examined. According to Brace et al. (2006), the

closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the

relationship between this and the other predictor variables; and the higher the VIF value

(value from 1.0), the stronger relationship is between predictor variables; and such values

becomes a worry. However, results indicated high tolerance values (over .90), and low

VIF values (less than 1.08), therefore there was no multicollinearity issues.

Using the stepwise method, a significant model which included formality, design

experience and study level, emerged, F (3, 31) = 25.15, p < .0001. The model explained

105

68.1% of the variance (Adjusted R2 = .681). Table 22.1 shows the adjusted R square and

change statistics of each predictor when added to the model. Formality level (model 1)

accounted for 41% of the variance (Adjusted R2 = .41, F (1, 33) = 24.60, p <.0001), and

the inclusion of design experience in model 2 resulted in an additional 21.9% of the

variance being explained (R2 change = .219, F (1, 32) = 19.87, p < .0001). Study level

helped explained a further 6.2% of the variance when added upon formality and design

experience (R2 change = .062, F (1, 31) = 6.62, p = .015). However, study

major/specialization was excluded from the model as it did not have a significant impact

when added (R2 change = .019, F (1, 30) = 2.043, p = .16) – hence, not a good predictor to

help explain expected changes (but a better predictor than in total and quality changes)

made across levels of formality.

Table 22.1Adjusted R Square and R Square change

Model RAdjusted R

SquareStd. Error of the Estimate Change Statistics

R Square Change F Change

Sig. F Change

1 .654(a) .410 2.87575 .427 24.602 .0002 .804(b) .624 2.29388 .219 19.865 .0003 .842(c) .681 2.11552 .062 6.623 .0154 .853(d) .691 2.08079 .019 2.043 .163

a Predictors: (Constant), Formality Levelb Predictors: (Constant), Formality Level, Design experiencec Predictors: (Constant), Formality Level, Design experience, Study leveld Predictors: (Constant), Formality Level, Design experience, Study level, Major/specializationTable 22.2 gives information for the predictor variables (formality and between-

subject variables) included in the significant model. The result suggests that formality

alone (the manipulated variable) has a strong significant impact on the total number of

changes made (β = -.65, t = -6.74, p < .0001). The negative statistics further suggests that

as formality level increases, the number of quality changes made decreases. The results

for design experience (β = .43, t = 4.34, p < .0001) and study level (β = .25, t = 2.57, p

< .015) further indicates that on top of the effects of formality on quality changes made –

people with more design experience and/or at a high level of study (e.g. graduates) are

more likely to make greater number of expected changes than those with less design

experience and/or at a lower level of study (e.g. undergraduates).

Table 22.2The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the model.

B Std Error B β tFormality -1.705 .253 -.654 -6.742**Design experience 3.178 .733 .426 4.336**Study Level 1.886 .733 .253 2.574*

106

**p < .0001, *p < .005

3.3. Additional Analysis of performance data

3.3.1. Paired comparisons: Total, Quality and Expected changes

One-way ANOVA was conducted to compare the three variables (the number of

total, quality and expected changes) across the five conditions, and also to check for

internal validity – whether total number of changes and the number of quality changes are

comparable with the number of expected changes across the five conditions and that the

designs presented to the participants were valid across all five conditions.

Post-hoc paired comparisons were conducted after finding significant differences,

from the one-way ANOVA results (p < .003), between total changes, quality changes and

expected changes across all levels of formality (see Appendix N). Post-hoc paired

comparisons revealed that there was no significant difference between total changes and

quality changes at: low formality (paper), low formality (on Tablet PC), medium-low

formality and medium-high formality; however, there was a significant difference at high

formality (mean differences = 2.26, p = .04). Between quality changes and expected

changes, there was no significant differences at: low formality (paper), low formality (on

Tablet PC), medium-high formality and high formality; however, there was a significant

difference at medium-low formality (mean difference = 2.68). Furthermore, the difference

between total changes and expected changes was significant at every level of formality.

Figure 23 below shows the total, quality and expected changes made across levels of

formality. Overall, the numbers in total, quality and expected changes were different

across levels of formality, as can be seen in Figure 22, but relatively comparable in terms

of the general negative linear trend and the of number of changes made.

107

0

2

4

6

8

10

12

14

16

18

20

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formality

Mea

n nu

mbe

r of c

hang

es

Expected changes

Quality Changes

Total changes

Figure 23. Multi-line graph of mean total changes, mean quality changes and mean expected changes across levels of formality.

3.3.2. Extra changes: Quality – Expected; and Total – Quality

After statistical analysis of the three types of changes (total, quality and expected), it

was of interest to re-visit the data to count all other changes made to explore the different

types of changes made in addition to the expected changes. Therefore, the number of

extra changes made, in addition to expected changes (correction of deliberate “errors”),

that were of quality (i.e. changes that improved the design) was recorded. Changes that

were not expected, nor were considered quality changes were also recorded i.e. “other”

changes (total – quality). Each extra change found was grouped according to its type –

similar to the main types of functional changes (add, delete, change element type, resize,

relocation (refer to Figure 2 in the method section); but for the purpose of such

investigation, the “change of element” category was separated into two categories: change

of text in labels; and change of element type. Table 23 below shows the number of

individual quality changes made (quality – expected changes) in each category of change,

in each design.

Table 23. The number of extra changes (quality – expected) made in each design, grouped according to the type of change.

Category of change Low formality (on paper)

Low formality

Medium-low formality

Medium-high formality

High formality

108

Change of text in labels 9 16 11 4 2.5change of element type 5 3.5 6.5 4 2.5adding elements 15.5 15.5 9 11.5 10.5deleting elements 1 1 0.5 1 0relocation of elements 4 4 5 3 1resizing of elements 1 3.5 2.5 1 2.5miscellaneous changes 0 1 1 1 0

Similarly, Table 24 below shows the number of “other” individual changes made

(those did not count as quality changes or expected changes i.e. total – quality changes) in

each category of change, in each design. For more details on the extra individual changes

made in each design (quality – expected; total – quality) and the number of participants

making such changes, see Appendix O1, O2, O3, O4 and O5 (from low formality to high

formality designs).

Table 24. The number of extra changes (total – quality) changes made in each design, grouped according to the type of change.

Low formality (on paper)

Low formality

Medium-low formality

Medium-high formality

High formality

Change of text in labels 23 26 16 6 9change of element type 13 7 9 12 7adding elements 29 23 11 20 22deleting elements 8 6 4 5 4relocation of elements 32 18 16 13 10resizing of elements 2 3 5 1 7miscellaneous changes 0 1 1 1 1

Overall, visual inspection of Table 23 and Table 24 suggested that type of quality

changes made were mostly the addition of elements, and rarely deleted items; and the

changes (total minus quality changes) made were mostly relocation and addition of

elements, followed by change of text in labels and change of element type. The deletion

of elements was mostly considered as extra changes and only a few were of quality (as

reflected in Table 23 and Table 24). However, interestingly, the proportion of addition of

elements was relatively higher in the high formality design, which also reflects the low

number of other types of changes made. In other words, it was likely that subjects would

add elements to a design with high formality than making other types of changes.

3.3.3. Order Effect

By counterbalancing the order of presentation of the five conditions (presentation

of designs with different levels of formality – refer to Figure 1 in the methods section),

order effects was controlled for, and it was expected that overall, order effects would not

influence results.

109

Nonetheless, although order effects was controlled for, according to Heiman

(2000), order effects may reduce internal validity; therefore this was also examined as it

may have played a role in producing a (weaker/stronger) trend of changes made across

levels of formality. However, as there was only a small sample (N = 30), and that it was

not the purpose of the study to test for order effect, no statistical tests were performed to

search for significant order effects. Instead, graphs were used for visual inspection. Data

were grouped and examined according to the order of conditions presented (e.g. 54321,

12345, 43215, 23451…etc), one multi-line graph for total changes (Figure 24a); quality

changes (Figure 25a); and expected changes (Figure 26a). Visual inspection of the Figure

24a, 25a and 26a suggested that generally, although with different orders of conditions

presented, there was still a linear trend found in the number of total, quality and expected

changes made across levels of formality – also identified in the trends test in one-way

repeated measures ANOVA. There was little to some order effect – this was more

noticeable in orders 54321 and 12345. In order 54321, added with the practice effect over

the five conditions, such order levered the main effect of formality – less changes made in

the first condition (high formality), but as formality decreased, the number of changes

increased – also seemed to be affected by practice effect. However, order 12345 worked

against the main trend – in other words, with practice effect present, and at its strongest

during the fifth condition (high formality), the main effect of formality on the number of

changes made was weakened by the opposing force – practice effect. Figure 24b, 25b and

26b illustrates the contrast between the curves of orders 54321 and 12345.

The weaker or stronger trends shown in Figures 24a, 24b, 25a, 25b, 26a and 26b,

must be interpreted with extreme caution as there were only four subjects (n=4) in each

order group. Overall, as expected, order effect did not influence the results significantly

as linear trends were present regardless of the order of presentation of conditions, which

further highlights the general significant linear trend.

110

0

5

10

15

20

25

30

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formality

Mea

n To

tal c

hang

es54321

43215

32154

21543

15432

12345

23451

34512

45123

45123

Figure 24a. Mean total changes made across levels of formality according to the order of conditions presented.

0

5

10

15

20

25

Low formality(paper)

Low formality Medium-lowformality

Medium-highformality

High formality

Levels of formailty

Mea

n to

tal c

hang

es

54321

12345

Figure 24b. Mean total changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 directions (n=4).

111

0

5

10

15

20

25

Low form ality(paper)

Low form ality Medium -lowform ality

Medium -highform ality

High form ality

Levels of formality

Mea

n Q

ualit

y C

hang

es54321432153215421543154321234523451345124512351234

Figure 25a. Mean quality changes made across levels of formality according to the order of conditions presented.

0

2

4

6

8

10

12

14

16

18

20

22

24

Low form ality(paper)

Low formality Medium-lowformality

Medium-highform ality

High form ality

Levels of formality

Mea

n qu

ality

cha

nges

5432112345

Figure 25b. Mean quality changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 (n=4) directions.

112

0

2

4

6

8

10

12

14

16

18

20

Low formality(paper)

Low form ality Medium-lowform ality

Medium-highformality

High form ality

Levels of formality

Mea

n E

xpec

ted

Cha

nges

54321432153215421543154321234523451345124512351234

Figure 26a. Mean expected changes made across levels of formality according to the order of conditions presented.

0

2

4

6

8

10

12

14

16

18

20

Low form ality(paper)

Low form ality Medium -lowform ality

Medium -highform ality

High form ality

Levels of formality

Mea

n E

xpec

ted

Cha

nges

54321

12345

Figure 26b. Mean expected changes made across levels of formality according to the presentation of conditions in the 54321 (n=4) and 12345 (n=4) directions.

3.4. Analysis of the “Overall Enjoyment” rankings of the five designs

113

Subjects’ rankings of the five designs with different levels of formality were

examined (see Appendix P). “Overall enjoyment” was defined as the subjects’ ranking of

the five designs in the order from the “most-liked” design with a rank of 1 (i.e. the design

they enjoyed working on the most in comparison to other the other designs presented) to

the “least-liked” design with a rank of 5 (i.e. the design they enjoyed working on the least

in comparison to the other designs presented). Figure 27 shows the overall ranking, from

the most liked (lower scores) to the least liked (higher scores), when working on each

design in comparison to other designs in terms of enjoyment.

3.37(sd = 1.35)

4.2(sd= 1.35)

3.17(sd = 1.15)

2.53(sd = 0.90)

1.7(sd = 1.02)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

High formality Medium-highformality

Medium-lowformality

Low formality Low formality(paper)

Levels of formality

Mea

n ra

nk

Figure 27. A bar graph showing mean rank and standard deviation, in terms of preference, according to the overall enjoyment, in working on each design with a different level of formality.

Overall, the high formality design was ranked most frequently as the “most liked”

design (by seventeen out of thirty, or 56.7% of participant). Most participants (twenty out

of thirty, or 66.7%) ranked the low formality design presented on the Tablet PC as the

least liked design. Interestingly, however, fourteen out of thirty (46.7%) participants

ranked the low formality design presented on paper as the second least liked. In other

words, when given low formality designs, participants liked (enjoyed) working on the

design presented on paper more than working on the design presented on the Tablet PC.

This was reflected in participants’ preference for design mediums in the experiment:

thirteen out of thirty (43.3%) participants preferred using paper and pen, while fifteen out

of thirty (50%) participants preferred using the Tablet PC, and two participants (6.6%)

indicating no preference of one medium over another medium.

114

In addition to the overall enjoyment, the underlying reasons for the rankings

expressed by the subjects were also investigated. Subjects’ responses were generally

categorized into three groups – subjects who ranked according to (a combination of): 1)

aesthetics of the design; 2) effort required to improve the design; and 3) level of

fun/stimulation when working on the design. One must note that some subjects’ response

overlapped into the groups (i.e. subjects ranked according to a combination of the two or

three factors); however, for the purpose of the analysis, the response were grouped and

analyzed in separate tests. Table 25 shows the mean and standard deviation of rankings of

the five designs with respect to overall enjoyment, aesthetics, perceived fun/stimulating

level, and perceived effort required when working with a design.

Table 25. Mean ranks and standard deviation, in terms of overall perceived enjoyment and other underlying factors for subjects’ rankings (including appearance, perceived effort required, and perceived fun/stimulating level), when working on each design in comparison to other designs presented.

Overall enjoyment Aesthetics

Perceived effort required

Perceived fun/stimulating level

MeanStandard deviation Mean

Standard deviation Mean

Standard deviation Mean

Standard deviation

High formality 1.70 1.02 1.33 .577 1.55 0.67 2.86 1.35Medium-high formality 2.53 0.90 2.62 .740 2.55 1.04 2.71 1.11Medium-low formality 3.17 1.15 3.14 .910 3.82 1.25 1.86 0.90Low formality (tablet) 4.20 1.35 4.62 .973 3.91 1.45 3.29 1.70Low formality (paper) 3.37 1.35 3.29 1.38 3.09 1.38 4.29 1.11

3.4.1. Ranking according to design appearance (aesthetics)

Most participants (twenty-one out of thirty, or 70%,) ranked according to the

aesthetic properties of the designs (e.g. tidiness of lines and fonts, alignment of elements,

and whether it was easy to follow). See Table 25 which shows the mean rank of designs

with high formality to low formality according to aesthetics factors. To determine the

effect of formality level (aesthetics) on participants’ enjoyment in working on the design

(indicated by rankings), Friedman’s rank test for several related samples was used as

proposed by Winer, Brown, and Michels (1991) and Howell (2002). Analysis of the

ranked data showed that the subjects’ rankings of the five designs according to aesthetics

were significantly different, χ2 (4, N = 21) = 47.43, p < .0001, and the Kendall coefficient

of concordance of .57 indicated strong differences among the rankings. Post hoc analysis

by Wilcoxin signed-rank test, indicated that participants enjoyed working on the high

formality design (lower rankings) significantly more than the other four designs, all with p

< .0001, as participants indicated that the designs appeared “tidy”, “pretty” and “[format

is] easy to follow”. Results also showed that participants enjoyed working on the low

115

formality design on Tablet PC the least as it had the highest mean rank, and was

significantly higher when compared to the other four designs: low formality on paper (p

= .006); medium-low formality (p = .001); medium-high formality (p < .0001) and high

formality (p < .0001), as many participants had noted that the design appeared “untidy”,

aesthetically “unattractive", and that the layout was “hard to follow”. Interestingly

participants enjoyed working on the low formality design presented on paper significantly

more than the low formality design presented on the tablet PC (p = .006). No significant

differences were found between rankings of other pairs of designs (medium-high

formality, medium-low formality and low formality on paper) in terms of enjoyment when

a design appeared to be more or less formal (beautified).

Additionally, the overall rankings of designs according to appearance (aesthetics)

of designs were negatively associated with the trend of the number of changes made

across levels of formality (refer to Appendix Q for subjects’ performance – those who

ranked according to the aesthetics factor – across levels of formality). The number of

changes made increased linearly as formality decreased, and similarly, rankings increased

as formality decreased – i.e. subjects liked designs that appeared more formal than designs

that appeared less formal, but on the other hand, subjects made fewer changes to the

designs that they liked (higher formality designs) and made more changes to designs that

that liked less (lower formality designs). However, interesting, unlike subjects’

performance, the low formality designs presented on paper was ranked lower than the low

formality design presented on the tablet PC i.e. subjects enjoyed working on the low

formality design presented on paper than on the tablet PC.

3.4.2. Ranking according to perceived “effort required”

Moreover, eleven out of thirty (23%) participants ranked according to the

perceived effort (input) required to improve designs. The mean ranks from high formality

to low formality were 1.55, 2.55, 3.82, 3.91, 3.09 (see Table 25). Subjects’ reasons

associated with rankings indicated that a higher rank (e.g. 5) meant that the design

“required lots of changes” and therefore more effort is needed to improve the design; and a

lower rank (e.g. 1) meant that the design was generally “good” and “required fewer

changes” (i.e. least effort required).

As suggested by Howell (2002) and Winer, Brown, and Michels (1991),

Friedman’s rank test for several related samples was employed to determine the effect of

formality on participants’ perceived effort required when working on the design (which in

116

turn affected the overall enjoyment rankings). Analysis of the ranked data showed that the

subjects rankings of the five designs, according to perceived effort required, were a

significantly different, χ2 (4, N = 11) = 16.16, p < .003, with the Kendall coefficient of

concordance of .37 indicating fairly strong differences among the rankings. Post hoc

analysis by Wilcoxin signed-rank test, (and controlling for the Type I errors across these

comparisons at the .05 level using the LSD procedure) further suggested that the

participants enjoyed working on the high formality design (lowest mean ranking)

significant more than the other four designs: medium-high formality (p = .039); medium-

low formality (p = .004); low formality on Tablet PC (p = .009) and low formality on

paper (p = .015) – some participants reasoned that the high formality design had “fewer

mistakes” and “not many changes needed” compared to other designs which participants

indicated that they “wanted to change more” in the design. Results also suggested that

subjects enjoyed working on the medium-high formality design significantly more than the

medium-low formality design (p = .046) as well as the low formality design on the Tablet

PC (p = .034). No significant differences were found between rankings of other pairs of

designs (medium-low formality; low formality on Tablet PC and low formality on paper)

in terms perceived effort required when working on a design (i.e. the overall enjoyment

rankings)

Additionally, the overall rankings of designs according to the perceived effort

required was negatively related to the number of changes made across levels of formality

(refer to Appendix R for subjects’ performance – those who ranked according to effort

required – across levels of formality). The number of changes made increased (decreased)

linearly as formality decreased (increased) (see Appendix R for subjects performance),

and similarly, rankings increased as formality decreased. However, the rankings of low

formality designs – one presented on paper and the other presented on the tablet PC – was

not in linear order expected – refer to Table 25.

3.4.3. Ranking according to perceived “fun and/or stimulating level”

Few participants (seven out of thirty, or 23%) ranked according whether designs

were fun or stimulating to work on during the task. The mean rank of the designs from

high formality to low formality was in the order of 2.86, 2.71, 1.86, 3.29 and 4.29 (see

Table 25). Participants’ reasons associated with rankings indicated that a rank of 5 meant

that the design was the least interesting and stimulating to work on; whereas, a rank of 1

meant that the design that was the most fun and/or stimulating to work on out of the five

designs given. To explore the whether there were any differences in the overall enjoyment

117

rankings of the five designs in terms of level of fun/stimulation, Friedman’s rank test for

several related samples was conducted. Results showed that the ranked data did not differ

significantly, χ2 (4, N = 7) = 8.80, p = .66, which suggested that participants found the five

designs similar in terms of the level of fun/stimulation when working on the designs.

Visual inspections suggested that subjects enjoyed working on the medium-low formality

design as it was more fun/stimulating than other designs (mean rank of 1.86 – lowest

compared to other designs); but found the low formality design least fun/stimulating to

work on (mean rank of 4.29 – highest in relation to other designs). The fun/stimulating

level of other designs did not differ very much for the subjects, which was shown in the

similar rankings (2.86, 2.71, and 3.29 – refer also to Table 25 for variance). However,

results were only preliminary and inconclusive as the number of subjects was very small

(n= 7).

Additionally, although with varying perception of whether a design was fun and/or

stimulating to work on (see Table 25), it made no difference to the effect of formality – the

number of changes made as formality increased (decreased) still decreased (increased) –

see Appendix S for subjects’ performance across levels of formality – those who ranked

according to the fun/stimulation level of working on the designs. This may suggest that

the perception of fun/stimulation level when working on the designs did not correlate

strongly to the number of changes made to the designs; however, since fun/stimulation

level of designs varies in different individuals, and that there were only seven subjects

examined, interpretations must be made with caution.

3.5 Design Tool Preference

3.5.1. Design tool preference in the experiment

Subjects’ design tool preference in the experiment was examined to determine

whether subjects preferred using pen and paper, and/or the Tablet PC or whether there was

no preference between the two tools during the design tasks.

Figure 28 shows that overall, 50% of the participants preferred using the tablet PC

(Inkit), while thirteen (43.3%) participants preferred using paper and pen during the design

tasks; and two participants responded with no particular preference between using pen and

paper, and the Tablet PC during the design tasks. See Appendix T for the detailed

responses from subjects on design tool preference during the experiment.

118

0

10

20

30

40

50

60

No preference Preferred paper &pen

Preferred Tablet(Inkit)

Design tool preference during the experiment

Per

cent

age

(%)

Figure 28. Bar graph showing subjects’ design tool preference during the experiment.

Additionally, factors that were likely to affect design tool preference during the

experiment were briefly examined, including study major/specialization (see Figure 29a)

and design experience (see Figure 29b) – also refer to Appendix T for indication of

subjects’ study and design backgrounds along with detailed responses.

Visual inspection of the Figure 29a revealed that in the CS/SE major group, more

subjects preferred using the Tablet (55% or eleven out of twenty) than using paper and pen

(35% or seven out of twenty subjects), while two subjects had no preference; and in

contrast, in the non-CS/SE majors group, more subjects preferred using paper and pen

(60% or six out of ten) than using the Tablet (40% or four out of ten); and hence, with a

smaller number of subjects in the group, interpretation of group differences must be made

with extreme caution. Similarly, such contrasting pattern was also found in terms of

design tool preference between subjects with different design experience – in the CS/SE

design experience group, more subjects preferred using pen and paper (46.7% or seven out

of fifteen) than using the Tablet (40%, six out of fifteen); and in contrast, more subjects

with none-to-some non-CS/SE design experience preferred using the Tablet (60%, nine

out of fifteen) than using paper and pen ( 40% or six out of fifteen). With respect to no

particular preference for either of the two design tools used in the experiment, not much

could be concluded as there were only two subjects responding this way, and both with

CS/SE major and design experience.

119

0

10

20

30

40

50

60

70

No preference Preferred paper &pen

Preferred Tablet(Inkit)

Design tool preference during the experiment

Per

cent

age

(%)

CS/SE majorNon-CS/SE majors

0

10

20

30

40

50

60

70

No preference Preferred paper &pen

Preferred Tablet(Inkit)

Design tool preference during the experiment

Per

cent

age

(%)

CS/SE design expereince None to some non-CS/SE design experience

Figure 29a. Bar graph showing the proportion of subjects – according to study major: CS/SE (n=20) and non-CS/SE majors (n=10) – preferring different design tools during the experiment (paper and pen; Tablet (Inkit); or no preference).

Figure 29b. Bar graph showing the proportion of subjects – according to study major: CS/SE design experience (n=15) and none to some non-CS/SE design experience (n=10) – preferring different design tools during the experiment (paper and pen; Tablet (Inkit); or no preference).

3.5.2. Design tool preference in the experiment

Subjects’ design tool preference in the “real world” was also examined for

comparisons with design tool preference during the experiment – the similarities and

differences. Figure 30 shows that overall, most participants expressed that he/she would

prefer using pen and paper first then move on to using computer if they were in real life

design situations. On the other hand, many participants preferred using computers (with

other popular applications such as Photoshop, Visual Basic.Net). The number of subjects

expressing other preferences was similarly low. See Appendix U for the detailed

responses from each subject.

120

Figure 30. Bar graph showing subjects’ design tool preference in real life design situations

Factors that were likely to affect design tool preference in real life design situations

were briefly examined, including study major/specialization (see Figure 31a) and design

experience (see Figure 31b) – also refer to Appendix U for indication of subjects’

background along with the detailed responses.

Visual inspection of the Figure 31a revealed that subjects who majored in CS/SE had

a wider variety of preferences for a single and/or a combination of design tools, compared

to subjects with non-CS/SE majors. There was also a greater proportion of subjects with

non-CS/SE major preferring paper and pen than subjects with a CS/SE major.

Interestingly, no subjects with non-CS/SE major preferred using tablet (Inkit) if they were

doing design in the real world, compared to a small number of subjects with CS/SE major

who would prefer (choose) tablet as their design tool. When subjects were examined

according to their design experience, a different pattern of preference was found – as

shown in Figure 31b – there was a distinct preference for particular design tool(s) between

subjects with CS/SE design experience and subjects with none to some non-CS/SE design

experience. The majority of subjects with CS/SE design experience preferred using

computer (that has other design tools such as VB.net and Photoshop) as opposed to the

majority of subjects with none to some non-CS/SE design experience preferring the use

pen and paper first, then computer (software). Furthermore, the use of pen and paper, then

computer was the second most popular preference in the CS/SE design experience group;

however, as the number of subjects in the group with none to some non-CS/SE design

experience, no such statement could be made.

121

0

10

20

30

40

50

60

Nopreference

Paper andpen

Tablet PC(Inkit)

Computer(Othertools)

Pen andpaper, thenTablet PC

Pen andpaper, thenComputer

Design tool preference in real life design situations

Per

cent

age

(%)

CS/SE majorNon-CS/SE Major

Figure 31a. Bar graph showing the proportion of subjects – according to study major: CS/SE (n=20) and non-CS/SE majors (n=10) – preferring different design tools in real life design situations.

0

10

20

30

40

50

60

70

80

Nopreference

Paper andpen

Tablet PC(Inkit)

Computer(Othertools)

Pen andpaper, thenTablet PC

Pen andpaper, thenComputer

Design tool preference in real life design situations

Per

cent

age

(%)

CS/SE design experience

None to some non-CS/SE design expereince

Figure 31b. Bar graph showing the proportion of subjects – according to study major: CS/SE design experience (n=15) and none to some non-CS/SE design experience (n=10) – preferring different design tools in real life design situations.

Chapter 4. Discussion

There was significant effect of formality on design task performance, in terms of

total, quality and expected changes made; significant linear trends were also found,

indicating that as formality increases, the number of changes made decreases. Significant

effect of expertise, particularly design experience, study major/specialization and study

level on the number of changes made across levels of formality. The overall enjoyment

rankings of the five designs were ranked differently by different subjects, in three

categories, including those who ranked according to aesthetics and fun/stimulating level of

the design and perceived effort required to improve the design. No statistical analyses of

subjective measurement of design tool preference during the experiment, and design tool

preference in real design situations. The simple visual analyses served as an indication of

different tool preference in the laboratory experiment as opposed to the real world, and

whether it affected design performance in any ways. No difference in preference was

found between designing on paper compared to designing on the tablet PC (InkKit) in the

experiment. As expected, preference in real world design situations was more diverse

compared to preference in the experiment – this gave a brief indication of whether InkKit

will be an effective tool, in terms of usability – whether people will actually use it. The

following sections will discuss the findings in a more detailed manner.

4.1. Effects of formality on design task performance

In the following section, the mean total number of changes is referred to as “total

changes”; the mean number of quality changes made is referred to as “quality changes”;

and the mean number of expected change made is referred to as “expected changes”.

Overall, there was an effect of formality on the number of changes made (in

terms of total, quality and expected changes), and more specifically, the number of

changes made decreased as the designs’ level of formality increased; hence, a negative

relationship between formality and design performance. This suggests that formality plays

an important role in affecting the subjects’ design performance; especially on decisions on

making changes to improve the designs presented (in terms of functionality and usability).

Field’s (2004) findings showed that aesthetics play an important role in problem

solving, which further support the findings of this study that formality (as a kind of

aesthetics, a strong emotional response) play an important role design performance (as

problem solving performance). Furthermore, as design can be seen as a kind of problem

solving by many (e.g. Goldschmidt, 1997; Smith and Browne 1993; Thomas & Caroll,

1979), the underlying visual mechanisms, for example, maybe visual attention in diagram-

based problem-solving (e.g. Grant & Spivey, 2003) may help explain why designers make

more changes to a design that is hand-drawn and appear rough and sketchy (lower levels

of formality), compared to tidier designs, that appear more precise, polished and formal

(higher levels of formality) – maybe more visual attention is required to first understand

and then work on a design with lower levels of formality than designs with higher levels

of formality. Although no research directly supported this claim (question), Grant and

Spivey (2003) measured eye movements in problem solving, suggesting that visual

attention was an important factor in problem solving and highly correlated to the

frequency of correct solutions; in this case, visual attention may play a role in design

(problem-solving) performance, in terms of improving a design by making changes. A

factor that affects visual attention is perceptual grouping of elements (an aspect of Gestalt

psychology – Koffa, 1935) – in the context of the present study, in the low formality

designs, no elements are aligned exactly, therefore more elements are scattered, thus, more

attention and eye fixations (and time) are needed to process the initial design presented;

however, in high formality designs, more elements are aligned (i.e. perceptually grouped),

which allows easier and fast scanning of the design, hence, less attention on details. In

addition, in higher formality designs, lines are straightened and text is presented in

standardized computer fonts, thus allowing even fast scanning and better readability

compared to low formality designs with squiggly lines and handwriting, which maybe

harder to follow, thus requires even more time and attention is to process the initial design

presented; thus requires more attention on details.

Such difference in attention required to process a design presented may be useful

to explain the effects of formality on design performance – as the level of formality of

increases, the design may become easier to process (e.g. reading, scanning) as less

attention is required on details, thus, maybe fewer errors are noticed; and as for low

formality designs, as more detailed attention is required, more errors maybe noticed during

the course of processing of the design. Hence, design performance – number of changes

made (total, quality and expected changes) to improve the design presented – may be

affected in such way.

There are other alternative explanations for design performance across levels of

formality. Skeptically, it can be argued that the effect of formality on the design process,

and more specifically on the number of changes made, was simply due to the

experimenter’s subconscious bias during the course of designing of the five online forms

to be presented to the participants. It was possible that the higher formality designs were

subconsciously designed as being more “correct” than designs of lower formalities – such

effect could be seen as one of the limitations of the study. However, optimistically, it is

unlikely that results from the study was purely due to experimental bias as there were

statistically significant results and noticeable trends shown in the graphs including

significant negative linear trend as formality increases in total changes, quality changes

and expected changes decreases; plus differences in terms of number of changes made

between designs with different levels of formality. Furthermore, as the independent

variable (formality) was carefully manipulated and controlled for, and that the number of

planned deliberate “errors” to be corrected was the same in each design, the analysis of

expected changes enabled a controlled and systematic way of examining the differences in

subjects’ performance across levels of formality; and indeed, significant formality effect

and a linear trend were found.

On top of this, it is also debatable whether the subjects noticed more errors in the

low formality designs than higher formality designs (effects of formality); or whether the

errors in the low formality designs were just easier to detect or easier to improve than

errors in the high formality designs – in other words, varying difficulty in improving the

five designs (an experimental confound). The latter of the two arguments, can be viewed

as one of the uncontrollable limitations in the study even though all objective measures

have been taken to make each design as similar as possible. It is inevitable that perception

of task difficulty is subjective and personal for each individual, as such variable is

dependent on personal experience, skills and ability – according to Spector (2006), in the

context of Industrial and organizational psychology, work/task performance is dependent

on such factors. It becomes the question of interaction rather than causation when it

comes to debating about the effects of formality on errors being noticed in the designs

presented – results could be interpreted as: 1) formality affecting subjects’ visual attention

on a design, for example, more eye fixations in stimulus that appeared more complex

(Grant & Spivey, 2003) which could be the case with the lower formality designs as it was

scattered and less clustered and aligned as the high formality design). From aesthetics

approach in art (see Levinson, 2003 for a fuller account and discussion), it may be that the

more formal (or aesthetically pleasing) a design appears, the fewer errors one may notice,

as beauty may have ‘blinded’ the eyes (and the mind).

Linear Trends

The significant linear trend found in terms of the total changes, quality changes

and expected changes made in designs across levels of formality suggests that as the

formality of a design increases (i.e. as a design appears more formal, pretty and tidy): 1)

the number of changes made in attempt to improve the design (i.e. total changes)

decreases; 2) quality changes decreases; and 3) expected changes (i.e. the number of

‘planned errors’ corrected) decreases. Vice versa, as the formality of a design decreases

(as a design appears rougher, less tidied-up and sketchy): 1) the number of changes made

in attempt to improve the design (i.e. total changes) increases; 2) quality changes

decreases; and 3) expected changes (i.e. the number of ‘planned errors’ corrected)

increases. Moreover, the significant linear trend in the expected changes made across

levels of formality showed a more robust effect of formality on design performance. Even

when each design presented contained the same number of “planned errors” for

corrections, the number of corrections made to those errors (i.e. expected changes) still

differed significantly in a linear manner as the design looked more or less formal. Such

findings support the underlying concepts in design education (e.g. in design classes) and

design process prescription in design handbooks (e.g. Fowler & Stanwick, 2004; Brinck,

Gergle, & Woods, 2002) that low formality designs (e.g. rough, hand-drawn, unfinished

looking design) should be used for early stages in design to facilitate exploration and for

catching early errors; where as, higher-fidelity and higher-formality prototypes (computer-

rendered, design) should be created at later stages for refinement. The effects of formality

found in this study suggests that although it maybe helpful for designers to use

beautification functions in sketching-based design tools to provide a quick glimpse of the

(near) finish product in just a few clicks, designers, however, should not beautify a design

until the very last stage of the process, as the results showed that subjects’ performance

was poorer (in terms of improving the design) as the level of formality of the design

presented increased. In addition, the linear trend in design performance in terms of the

number of changes made, further linked previous findings (Black, 1990; Plimmer and

Apperley, 2004), that reviewers/designers interact with low formality designs (rough,

hand-drawn) and high formality designs (tidied-up, computer-rendered) – suggesting that

the relationship (i.e. the effect of formality on deign performance) can still be found at

other levels of formality.

High formality Versus Low formality designs on Tablet PC

Total, quality and expected changes made were significantly lower when subjects

were presented with the high formality design on the tablet PC, compared to other four

designs with lower levels of formality. On the other hand, total, quality and expected

changes made were significantly higher when subjects were presented with the low

formality design on the tablet PC, compared to other designs with higher levels of

formality on the tablet PC.

The same trend was found in previous studies (e.g. Black, 1990; Plimmer and

Apperley, 2004) that subjects interacted differently with hand-drawn designs versus

computer-rendered designs, which suggested that different problem-solving mechanisms

and strategies used in different design media, when a design appears rough and sketchy

(e.g. on pen and paper, or on a sketch-based interface) or computer-rendered and formal

(e.g. on computer using different software). Moreover, Black (1990) found that subjects

(students) were more satisfied with their design on computer (high formality design) than

with the design on paper (low formality design). However, after a review with their tutor,

interestingly, subjects indicated in the questionnaire that they were more with the design

on paper than on computer (screen). This suggested that after the class review students

noticed that their design on paper (low formality design) was indeed better than their

design on the computer. Hence, in the context of this study, Black’s results suggested that

design performance (in terms of decision making and error detection) maybe better when

working with low formality designs better than with high formality designs. However, it

must be noted that Black’s results was only a subjective and indirect indication of design

performance. Also, in her study, the participants created the designs from scratch; where

as in the present study, participants were given already-designed forms to improve on (i.e.

reviewing/editing the designs) – different cognitive processes may have played a role

depending on the nature of the design task (whether designing from scratch or reviewing).

In another study, Plimmer and Apperley (2004) found also, that subjects

interacted with hand-drawn, sketchy designs differently from computer-rendered, formal

looking designs. Plimmer and Apperley’s results showed that subjects made fewer

changes to the high formality design (computer rendered, formal and tidy design created in

VB.Net) compared to the low formality design (hand-drawn and sketchy design created in

FreeForm, an informal sketch-based tool), in which most changes were quality changes

and that the two deliberate errors in the design were mostly detected and corrected.

However, one must be bear in mind that in Plimmer and Apperley’s study, no distinctions

were made between different types of changes made to a design – any changes (total

changes); quality changes that improved the design; or expected changes (corrections of

planned errors) for controlled comparisons between designs with a different level of

formality. Furthermore, Plimmer used a different application for each design task, and

both tasks involved a desk top computer with the standard input devices including mouse

and keyboard, as opposed to the present study which involved Tablet PC with pen-input

used to present four designs with low to high formality, plus one low formality design

presented with for comparison.

Previous studies (i.e. Black, 1990; Plimmer & Apperley, 2004) only looked at

two levels of formality – low formality and high formality; where as, the present study

examines and compared several levels of formality systematically form low to high, on the

same tool (plus one low formality on paper for comparison). Therefore, comparison of

results from the current study with previous studies must be interpreted with extreme

caution.

Low formality VS low formality designs: Paper versus Tablet PC

Furthermore, total, quality and expected changes made were (much) higher in the

low formality design (on paper) than all other levels of formality on the tablet PC. Hence,

the significant difference of the number of changes made between the two low formality

designs – one presented on paper and one presented on the Tablet PC – the number of

changes made on the design presented on paper was much higher than the design

presented on the tablet PC. Both designs had the same formality according to the same

beautification criteria – non-beautified hand-drawn design (refer to Table 2, 3, 4). This

raises curiosity about how much difference there is between seeing and perceiving a

design on paper compared seeing and perceiving the same design on the tablet PC; and

whether or not, the ‘gap’ between paper and pen and computer has been really ‘bridged’

and in an effective way. Or it maybe that the difference in performance was simply due to

the difference in medium, as using the tablet PC introduces formality itself, and even a

different dimension of formality. In addition, research have shown that information

processing on screen differs from information processing on paper, for example,

differences in handling and manipulation, display size, angle of view and differences in

cognitive requirements such as short term memory required to remember other

information outside of the screen, compared to the wide perceptual field supported by

using paper (for a fuller review of the differences, see Dillion, 2003). All these factors

may have contributed to the differences in design performance between the two designs

with the same level of formality, but on a different medium.

Moreover, there is an implication on sketch-based design tools that sketching on

paper is still not being modeled and supported on current computer interface in a true

manner. Hence, it maybe useful for future research on sketch-based design tools. to focus

on improving such aspects, to enable users to experience better and more natural human-

computer interaction.

There is no previous research on comparing different levels of formality of

designs in one study, therefore, conclusive statements could be made. However, there is a

promising future in research concerning beautification and formality in conjunction,

towards a better understand of beautification approaches (e.g. what level of formality

should a hand-drawn diagram beautify to become) and how it will affect designers in an

unobtrusive way.

What happens in the middle, when a design is more or less formal?

Total changes

Along with the significant linear trend, although subjects’ performance, in terms

of total changes made, differed significantly when working on designs with low formality

(on paper and on tablet PC) and high formality, subjects’ performance did not differ

significantly when they were presented with designs with medium-high and medium-low

formality, and designs with medium-low and low formality on the Tablet PC. However,

subjects’ performance was visually different on the graphs (see Figure 11).

The non-significant differences between the three levels of formality could be

attributed to the significant (weaker) cubic trend found, p < .01, partial η2 = .23, in

addition to the linear trend. Moreover, low formality design on paper and low formality

design on the tablet PC may not belong on the same continuum (levels of formality) –

maybe a different dimension; and even so, it is likely that the low formality design lies

further towards the low formality end of the continuum. In comparison, low formality

design on the tablet PC was systematically created with respect to other levels of

formality, so that each level of formality lies next to each other in an approximate interval

within the same continuum. This was one of the main limitations in the present study –

although including the low formality designs presented on paper provided useful a

comparison, it may have affected the statistical analysis as it took into account the changes

made at every level of formality. The number of changes made in the low formality design

on paper was much higher than designs with other levels of formality, hence, it maybe

because of that, that when the number of changes made at each level of formality was

compared, no significant differences were found. However, optimistically, the high

number of changes made in the low formality design on paper may do very little damage

to affect the overall results. Therefore with this in mind, ANOVA with repeated measures

of the performance data was conducted again, this time including only four levels (from

low formality to high formality on the tablet PC). As predicted, however, the same results

were found.

Another explanation for the non-significant differences between the middle levels

of formality is that it was impossible to predict every change that was going to be made,

hence ‘total changes’ made (i.e. any changes made) was measured, for interesting

comparisons between quality changes and expected changes made. Plus, as participants

were free to make any changes, total (as well as quality) changes made across levels of

formality could not be measured in a systematic and controlled way, like expected changes

made.

The non-significant difference found between the medium-high, medium-low,

and low formality design may also indicate a small effect, or small significant difference,

that was not shown statistically. As this study is the first, known so far in the literature, to

explore different levels of formality of designs and their effects on design performance,

more research is needed to improve such methodological issues, and most importantly,

future replications of this study and/or variations of the current study are needed to be

more conclusive and to further interpret the findings and implications of the effects of

different levels of formality, particularly on the less extreme levels of formality where

there is a lack of research on – whether there is a true difference in performance when a

design appears more or less formal; and whether there are major implications of such

findings on beautification techniques.

Quality changes

Different to results in total changes, subject’s performance in terms of quality

changes made, was significantly different at each level of formality except in one occasion

where no significant difference was found between quality changes made in the medium-

low formality design and quality changes made in the low formality design (on tablet PC).

One explanation for this is the familiarity of design content in the medium-low formality

design (i.e. University Graduation Application form) which may have resulted in an

increase in quality changes made with respect to quality changes other designs. In

contrast, subjects may have been less familiar content in the low formality design on tablet

PC (i.e. Bank Loan Application form), hence, fewer-than-expected number of quality

changes were made – this also highlight the results on individual preference and overall

enjoyment ratings for designs (see later sections). However, content was impossible to

control for, as mentioned later that, because different individuals will have different

exposure to and personal experience with a variety of fill-in forms, and reactions to

different forms will differ (see later sections on “overall enjoyment”).

Other than the higher number quality changes made in the medium-low formality

design as anticipated, it can be said that even for the medium levels of formality, the linear

trend still occurs in a significant manner – that design performance, in terms of quality

changes made, decreases at a significant level as the level of formality increase.

Expected changes

Similar to total changes made, although subjects’ performance, in terms of

expected changes made, differed significantly when working on designs with low

formality (on paper and on tablet PC) and high formality, subjects’ performance did not

differ significantly when they were presented with designs with high formality and

medium-high formality, medium-high and medium-low formality, and designs with

medium-low and low formality on the Tablet PC. However, subjects’ performance was

visually different on the graphs (see Figure 19). Unlike total changes (uncontrolled

measurement of any changes made), expected changes were planned deliberate errors for

subjects to correct, with each design containing the same number of errors (twenty-three

correctable errors); however, unexpectedly, there was no significant incremental

differences between each level of formality. This suggested that subjects’ performance (in

terms of expected changes made) was comparable when they were presented with designs

with higher formalities (high formality and medium-high formality), and similarly in

designs with lower formalities on the Tablet PC (low formality and medium-low

formality; and medium-low formality and medium-high formality).

Although with the significant linear trend found, the relationship between the mid

levels of formality is still unclear. It maybe that the differences in the designs’ levels of

formality was too little to make a difference in affecting subjects’ design performance in

making expected changes; or it could be that the differences are significant, but was not

detected in this particular study due to the simplistic nature of the design tasks conducted

in the laboratory, far from a real design situation, which left little space for subjects to do

real design work.

It must be noted that during results analysis, it was ambiguous in deciding

whether parametric or non-parametric analysis should be employed. Although the

independent variable – formality – was mathematically and systematically manipulated

with three assumptions made (refer to methods section 2.5.2) while controlling for other

variables, it was debatable whether that the level of measurement in terms of levels of

formality was interval. In other words, whether there was a real difference in appearance

(formality) between designs across the levels of formality (based on the assumptions that it

is an interval level of measurement as the variable was meticulously manipulated); or

whether the design appearance could only be ranked from low formality to high formality,

where one design is higher in formality than another; hence, units between levels of

formality not exactly equal (ordinal level of measurement). On the other hand, with the

dependent variable being the number of changes made in the designs across levels of

formality, interval level of measurement was achieved. There was slight non-normal

(skewness and kurtosis) distribution, but was statistically justified (homogeneity of

variance, distribution normality, skewness and kurtosis statistics compared against the

errors and sphericity) to be reasonable for a parametric test involving one-way repeated

measures ANOVA. Such ambiguities was addressed by first choosing the appropriate

approach guided by justification (parametric approach), then double checking by

comparing parametric results with non-parametric outcome of analyzes. Friedman’s rank

test (the parametric equivalent of one-way repeated measures) was also conducted for

total, quality and expected changes made. However, the non-parametric results were not

reported, as it yielded the same significant results as the parametric tests including

significant differences and significant trends found. The results from post hoc Wilcoxin’s

sign rank test was also the same as the parametric post hoc comparisons i.e. differences

were found in the same pairs of variables

Overall, with the strong effect of formality on design performance, as well as

significant linear trends found in terms of total, quality and expected changes made across

levels of formality; the results found in previous studies (i.e. Black, 1990; Plimmer and

Apperley, 2004), that was found in the present study, that design performance differs

when working with low formality designs compared to high formality designs; this study

is the first, known so far in the literature, to explore different levels of formality of designs

and their effects on design performance. Therefore, future replications of this study and/or

variations of the current study are needed, in order to improve methodological issues, to

further interpret the findings, and to be more conclusive on the effects of levels of

formality of designs, particularly the less extreme levels of formality where there is a lack

of research on. For example, one could examine whether there is a true difference in

design performance when a design appears more or less formal; and whether there are

major implications of such findings on beautification techniques in sketching-based tool.

4.2. Between-subject effects: Expertise

Overall, results from between-subject analyses on expertise suggest that as

formality increase, the differences in terms of performance between groups (i.e. experts

and novice) tend to decrease (i.e. performance at a similar level); and vice versa where the

differences in performance between groups increase as formality decreases (i.e. more

variability in performance). Such finding suggests that no matter what level of design

experience, major/specialization or study level one has (i.e. expertise), formality still has

an impact on design performance, and more specifically, on the number of changes made

to improve the design (total, quality and expected changes made).

4.2.1. Design experience

As hypothesized, design experience was found to affect design performance

across levels of formality. The significant effect of design experience (the between-

subject factor) on the number of changes made (total, quality and expected changes)

across levels of formality indicated that subjects with Computer Science (CS)/ Software

Engineering (SE) design experience performed better (i.e. made more changes) across

levels of formality compared to subjects with no experience or some non-CS/SE

experience.

Significant linear trends were found which further indicated that regardless of

magnitude differences (the level of performance), the effects of formality on design

performance (total, quality and expected changes made) still existed when design

experience was taken into account.

The significant formality-by-design experience interaction showed that, in

addition to the effect of formality, design experience also affected design performance (i.e.

total changes and quality changes made) in a non-parallel manner. Although, no

statistically significant formality-by-design experience interaction was found on expected

changes made across levels of formality, it was observable that the difference in

performance decreased as the level of formality increased; and vice versa, where the

difference in performance increased as the level of formality decreased. Hence, there was

some formality-by-design interaction. This further suggests that individuals with different

design experience are affected by formality in different ways (magnitude), which in turn

influence design performance.

Moreover, as level of formality decreased, the larger differences in performance

(total, quality and expected changes made) were particularly noticeable when the two low

formality designs were presented; and more specifically: 1) between-subject performance

difference at each level of the two low formality design; and 2) within-subject

performance difference between working on the two low formality designs presented on a

different medium. Such big differences suggest that the change of media seem to have a

greater impact on those who had CS/SE design experience. Hence, the slope representing

the number of changes made was steeper (i.e. more changes) for subjects with CS/SE

design experience compared to others with no CS/SE design experience to some CS/SE

design experience.

4.2.2. Study major/specialization

For comparing between-subjects effect in design experience, subjects were

categorized into two even groups with fifteen subjects in each group, and data were

analyzed statistically to test Hypothesis 2a. However, study major/specialization and

study level were explored only mainly through visual inspection of graphical output due to

the various reasons as discussed in the results section (Chapter 3), for example,

unbalanced number of subjects in each group, problems with grouping of subjects and the

relatedness of the between-subject factors.

No significant effect of study major (the between-subject factor) was found on

the number of changes made (total, quality and expected changes) across levels of

formality; which indicated that performance level was similar in the two groups. Linear

and quadratic trends were found in total changes and quality changes, however not in

expected changes made across levels of formality. Furthermore, significant formality-by-

study major interaction was found in total changes and quality changes, and again not in

expected changes, across levels of formality. Such results suggest that the effect of

formality on design performance is still true for subjects with different

major/specialization (i.e. as the level of formality increase, the number of changes made

decreases); however, the extent to which they are affected by formality may differ. It

appears that results found in statistical analyses and information conveyed through

graphical output showed conflicting effects. In addition to the imbalanced number of

subjects in each group, (n=10 and n=20) that may have contributed to the conflicting

results; performance of subjects with a non-CS/SE major at the medium-low formality

condition of was, somehow, unexpected. Subjects in both group seemed to have

performed at the same level – the total, quality and expected changes made was similar in

both the CS/SE major group and the non-CS/SE major group. This could also help

explain the non-significant results from the between-subject effects tests. Furthermore,

subjects who majored in information system and other engineering specializations such as

computer system and electrical engineering may have had taken some CS papers (e.g.

most commonly CS101 and 105), and/or already had some knowledge of website design

and HTML components, but those subjects were also grouped into the non-CS/SE major

group. Thus, such factors may have played a role in producing the trend in the

non-CS/SE major group – i.e. similar level of performance as those in the CS/SE major

group. Hence, one must be extra cautious when interpreting such results.

4.2.3. Study Level

On the whole, the between-subjects results found in this study showed that in the

context of levels of formality of designs, there is a strong relationship between expertise

and design performance; which supports findings from previous studies comparing experts

and novices in the design process, for example, Christiaans and Dorst (1992) who

compared junior and senior industrial design students, and Atman et al. (1999) who also

compared students (first year and fourth year engineering students) and both studies

showed that there was a difference in terms of design performance and behaviour between

the two groups of ‘experts’ and ‘novices’ in the design process. Furthermore, observations

in the experiment revealed that some subjects with CS/SE design experience (‘experts’)

went through the whole design first to fix the ‘errors’, for example, changing an element to

the appropriate element, then went onto the detailed work; where as, ‘novices’ such as

bioscience and psychology students with no design experience searched for problems one

by one. Such finding further supports the research findings that novice and expert uses

different problem-solving strategies (e.g. Ho, 2001 who found that expert designer used

explicit problem solving strategies, which the novice appeared not to possess, but both

expert and novice used similar, bottom-up, or working-backward, problem solving

strategies.)

Overall, results from comparing subjects with different design experience seemed

to be the more reliable for interpretation in comparison to study major/specialization and

study level, and in addition, the number of subjects grouped in each group was equal (i.e.

fifteen subjects), and adequate for achieving reasonable statistical power according to

Cohen (1988). However, one of the limitations of the current study was that the subjects

had different design experience and came from a range of different (design) disciplines.

Hence, results could be improved by having participants with similar design backgrounds

(i.e. design experience, major/specialization and study level) to reduce variability to yield

stronger results. It may also be more useful to include a larger sample size in the two

groups; however, one must assess practicality of testing a greater number of subjects.

Moreover, the definition of expertise still remains one of the limitations in many studies –

although expertise was explained in the present study by including design experience,

domain-specific knowledge and education level; however, there was a considerable

amount of overlapping between the three between-subjects factors as they are highly

correlated.

4.3. Multiple Regression Analysis

As there was overlapping between the between-subject factors, the contribution

of variables to the overall effect was explored – how much formality and each between-

subjects factors account for the variance and in the end how much overall? Results from

Multiple Regression Analysis showed that design experience explained as much as

formality in the total number of changes made across levels of formality, which suggests

that the more design experience a participant has, the more changes he/she attempts to

make to improve the design. For explaining quality changes made across levels of

formality, design experience and study level explained the variance in addition to the main

effect of formality. Here in the context of this study, participants may have design

experience but their study level may be different, therefore, if participants is at a higher

study level, their design experience and related knowledge is likely to be better than those

who are at a lower level of study. However, it can also be observed in the experiment that

software engineering graduates with design experience may have made more changes to a

design, however, those changes did not necessary improve the design. A classic example

would be the radio button – according to web design guideline, the radio button should be

on the left side to the label (text) it is associated to. However, some participants followed

the original deliberate “error” that was presented in the design where, for example, a radio

button was deliberately located to the right of the associated text, and would draw the

radio button to the right of the label when a participant adds an item. Moreover, some

participants explicitly changed a radio button to the right of the associated label.

Formality was correlated higher, and explained more in expected changes made. One of

the reasons that variance explained by formality was much higher was mostly likely that

each design had the same number of “planned errors” for participants to change, therefore,

changes made could be compared between different levels of formality in a more

controlled manner compared to total and quality changes made.

As multiple regressions analysis showed that the level of formality correlates

fairly strongly with the number of changes made (total and quality changes, and especially

expected changes made), future research should explore more dimensions of beautification

(in terms of HTML designs) such as colour (multiple dimension and a huge area of study

itself), texture, shading, the (2.5-Dimensional) ‘look-and-feel’ (e.g. buttons that one could

“click” on and textboxes that one could “type” into a textbox) etc, and its effects on the

design process. How adding or removing a dimension may affect the design process

would be of interest to examine also, e.g. the number of changes made; perception of

attractiveness; and overall enjoyment etc. From the current study, it maybe hypothesized

that as the number of dimensions of increase, formality will increase, and thus, the number

of changes people make will differ (decrease), and the effects would be greater as more

dimensions are added and at a higher level, for example, more colours, controls (e.g.

buttons, textboxes, drop down menus) with shading, 3-D ‘look-and-feel’.

4.4. Additional Findings

4.4.1. Relationship between total, quality and expected changes

Quality changes derived from total changes, and expected changes derived from

quality changes. Guidelines are useful in many ways for example, standardizing and

learning, and in this case, helped design the five HTML forms that were presented to the

participants. Guidelines were used to produce designs contained deliberate design ‘errors’

based on HCI handbooks and design guidelines. Furthermore, the changes that were

counted as quality changes were based also, on design guidelines, to decide whether a

change was of quality.

However, one must point out the problem with the large number of assumptions

made by guidelines and handbooks that are sometimes arbitrary but these are rules to help

standardize designs e.g. like platforms for different operating systems. Such that websites

have behaved similarly, for example, clicking a blue underlined ‘link’ will lead to you the

site you clicked on; and only one item could be chosen within the dropdown menu.

Humans have learned to use the internet and have become used to the conventions that are

used to build websites and forms containing HTML controls e.g. textbox, radio buttons,

tick boxes, buttons etc. Assumptions used in guidelines, in turn, forms the basis for

evaluating whether a design is good. In this study, finding criteria for good design of

HTML forms based on experimental studies about errors and usability, and whether the

form has been filled-in correctly was difficult. Thus, it was even more difficult to decide

whether a change made by the participant was really a quality change (ones that will

improve the design).

The counting and analysis of the extra changes that participants made (total

minus quality changes; and quality minus expected changes) further questioned the

usefulness of (web) design guidelines. Why did people make those extra changes? Why

were the changes not included in the criterion for good designs after all? Even software

engineers (fresh graduates and employed) made some changes to the designs that did not

confirm to the guidelines and standards. It was the most difficult to decide whether a

change should be counted as quality change, or just a change.

It was also noticed from re-counting and re-analysis of different changes, that the

‘extra changes’ (total minus quality) are mostly changes in the flow of information i.e.

relocating items to another area. Furthermore, relocate of elements and items seemed to

increase as it formality level decrease. Flow of information and logic is subjective to each

individual therefore, only the relocation of items that made a difference (i.e. improvement)

to the design was counted as a quality change. Also, the exact changes made were highly

individualized – some changes were similar but most often, the changes were not the

same. This further reflects the individual differences on what is important to change and

what is not, and also, what is considered an improvement in a design is different to

different individuals. Maybe most guidelines are subjective in the end? But on the whole,

the “extra changes” that did not qualify as quality changes could be interpreted in a light-

hearted manner, as subjects had varying levels of design experience, domain-based

knowledge (major/specialization) and study level, which may have contributed to the high

number of extra changes?

Looking at this from another angle, why did people not make some of the

changes (total and quality changes) i.e. those made by other participants? Motivation,

boredom and concentration may have played a role here. Software engineers (graduates)

made significantly more changes across all levels of formality compared to undergraduates

(with CS design experience and majors) – it appeared that software engineers may have

put in more effort. It was also noticeable that some subjects had lower motivation level,

for example, making only a few changes, stopped and indicated that he/she had finished.

Also, near the fourth and fifth conditions, it was sometimes noticeable that some subjects’

concentration level decreased – stopping at 8-9 minutes during the 11-minute condition.

4.5. “Overall Enjoyment” rankings of the five designs ranking of

designs

Participants were asked to rank the five designs in the order of overall enjoyment

when working on each design from the most-liked design to the least-liked design. Also,

due to its nature, it was impossible to infer a magnitude of differences in the overall

enjoyment from such responses.

Overall, participants indicated that they enjoyed working on the higher formality

designs more than low formality designs, and the rankings increased (i.e. less liked) as the

levels of formality of designs increased. In other words, participants liked working on

lower formality designs less. Interestingly, subjects enjoyed working on the low formality

design on paper more than working on the low formality design on the tablet PC –

indicated by the difference in rankings. Participants’ preference for design media may

have played a role affecting subjects’ enjoyment ranking. The underlying reasons for the

rankings were also examined.

4.5.1. Rankings according to aesthetic aspects of designs

Analysis of the ranked data showed that the subjects’ rankings of the five designs

according to aesthetics were significantly different (p < .0001), and the Kendall coefficient

of concordance of .57 indicated strong differences among the rankings. Post-hoc tests

were also conducted.

Most participants ranked according to the appearance of the designs (twenty-one

subjects) and these subjects indicated in the questionnaire that a design was easier to

follow and comprehend when it looked “nice and tidy” with all the “elements aligned”

with “tidy lines and fonts” compared designs that were “untidy”; hence, subjects enjoyed

it more when working on the designs that appeared aesthetically pleasing than designs that

looked rough and “sketchy". Hence, participants indicated that they enjoyed working on

designs that appeared tidier and nicer, compared to designs that looked less attractive and

untidy.

Importantly, such results also helped validate the levels of formality of the

designs – the design with high formality appeared to be more formal, and aesthetically

pleasing, than the medium-high formality design; medium-high formality design appeared

to be more formal and aesthetically pleasing than medium-low formality; and low

formality (on he tablet PC) looked the least formal, with untidy and unaligned elements

compared to other designs.

Moreover, in the present study, the aim was not to show which design was liked

the most and which design was liked the least, and why. Rather, of additional interest to

the present study was the question of whether there were individual differences or a

common pattern of perception when working on designs with different levels of formality,

that influenced whether a person liked working on a design (overall enjoyment); which

may have played a role in affecting the number of changes made in a design, across levels

of formality. From the results where subjects response were grouped into three main

groups, it could be seen that the pattern of overall enjoyment ranking with the underlying

ranking factor being design aesthetics, is somehow correlated with the effects of formality:

as formality decreases, the less a participant enjoyed working on a design; and as formality

decreases, the number of changes increase. However, the pattern at the low formalities, the

pattern did not match exactly i.e. the number of changes made increases from low

formality on tablet to low formality on paper compared to the rank pattern being the

opposite, where the low formality design on tablet PC was less (least) liked compared to

the low formality design on paper, often scoring a higher (highest) rank (i.e. least liked).

From the design perspective, the findings of the current study suggests that it is

better to work with informal designs with sketchy properties (non-beautified), as it is more

likely that change to a design would be greater which may improve the quality of a design

– shown in the results where all three types of changes (total, quality and expected)

increased as a design appeared less formal and sketchy. However, despite such advantage

of working with rough and sketchy (low formality) designs, findings from the current

study suggest that people still like working on designs that appear more formal i.e. humans

like beautiful products (e.g. Hassenzahl, 2004; Overbeeke & Wensveen, 2004; Tractinsky,

Katz & Ikar, 2000), just like a (web) designer likes formality in their design – participants

liked working on design that appeared most aesthetically pleasing i.e. the high formality

design that looked tidy, with elements aligned, line straightened, and text in a clear legible

font (Roman Times Numeral). Thus conflicting mechanisms occur: people’s perception or

their likes of things do not match their mind’s workings – as if both working against each

other. On one side, one would prefer and/or like working on a design that appears formal

(beautified) but on the other, the mind works the best when a design looks rough and

sketchy. And even though, overall, participants did not enjoy the working on the lower

formalities, they still performed better compared to their performance when working with

a high formality design, where fewer changes were made.

Finally, as research has shown that aesthetics affect the consumers’ (users’)

perception on usability of the product, it raises the question of whether aesthetics will also

affect the designers (and their clients) during the design process – are designers affected

by aesthetic, just like the end-users, during the design process? Will they make fewer

changes to a design as it appears more attractive/aesthetically pleasing, and therefore

associate the design as a “good [enough] design” that requires little changes? Or are the

designers immune to the effects of aesthetics as they are designing the product, given the

requirements.

4.5.2. Rankings according to effort required

Rank test for several related samples found that there was an effect of formality

on participants’ perceived effort required when working on the designs, which in turn

affected the overall enjoyment rankings – hence rankings were significantly different and

with the Kendall coefficient of concordance of .37 indicating fairly strong differences

among the rankings.

The trend of participants’ overall enjoyment rankings according to perceived

effort required were comparable with the rankings according to aesthetics of designs – the

high formality design had the lowest score (i.e. participants enjoyed working on the high

formality design most) and as formality decreased, the scores (ranks) increased (i.e.

participants enjoyed working on the designs less and less as formality decreased). In

addition, similar to trend of rankings according to aesthetics, low formality design on

Tablet PC had the higher (highest) score compared to low formality design on paper

(which was comparable with medium-low formality design), indicating that the

participants enjoyed working on the low formality design on paper more than the low

formality design on the tablet PC. Such results suggest that when a design appears more

formal (beautified), it may affect participants’ perception of working on the design, and

that when media of presentation differs, perception may differ. The rankings also reflect

the relationship between levels of formality and perceived effort required – the more

formal a design appears, the less (perceived) effort required; however, no causation effect

could be concluded as the two elements affect each other.

4.5.3. Rankings according to Stimulation/fun level

No differences found on the effect of formality on participants’ perceived level of

fun/stimulation when working on the designs, which in turn reflected the overall

enjoyment rankings. This suggests that participants found the designs similar in terms of

level of fun/stimulation. On the other hand, such results also suggest there was great

individual variability which was also noticeable when participants’ rankings were

examined individually. This suggests that perception of fun/stimulating level was

subjective and varied across individuals. Maybe it was a confounding factor that

individuals found it more interesting to work on a particular design therefore put more

effort in to improving a design. One could also argue that it was probably the level of

formality that affected participants’ interest to work on a design (i.e. whether a design was

fun/stimulating in comparison with other designs) or it was maybe just the context of the

design alone – e.g. America’s next top model, dog registration form, university graduation

form, bank loan application and subscription to an magazine online. For example, paper

being rated as the least interesting maybe contributed to its similarity with online

subscription forms that internet users are frequently faced with, therefore (some)

participants may have felt that it was the least interesting to work on out of the five

designs. Thus, motivation maybe affected – when one is bored, it is likely that one would

make fewer changes as opposed to when one is motivated and interested in the design. It

is also likely that more attention and effort is put into improving a design – i.e. more

changes made and therefore, it is likely (but not necessarily) that more quality changes are

made (compared to when little changes are made). This suggests that although every

effort had been made to control for such effect (see discussion in methods section 2.5.2.)

to obtain results that reflect the effects of formality (and not others), subjects reacted to the

content different designs differently – each individual has different preferences, exposure,

background and experience (in life).

On the other hand, it may also be that for those participants who ranked according

go level of stimulation of designs – instead of formality, the perceived level of

fun/stimulation of designs may have been the main factor that affected the number of

changes made in different designs. However, although with varying perceptions of

whether a design was fun to work on, it made no difference to the effect of formality – the

number of changes made as formality increased (decreased) still decreased (increased).

However, as there was only a few number of participants (N = 7) who ranked according to

this factor, results was only preliminary and hence, not conclusive; hence, it was only an

indication of the underlying reasons (and was not the underlying reason itself) that

affected the overall enjoyment rankings.

Overall, the different enjoyment rankings suggest that whether one likes working

on a design more or less, the effects of formality still exist – as formality increases, the

number of changes made (design performance) decreases.

4.6. Design tool preference

Design tool preference during the experiment and design tool preference in real

world design situations were indicated by participants. Due to the impracticality, no

statistical analysis was used, and visual inspection of graphs was more useful for the

purposes of examining design tool preference.

4.6.1. Design tool preference during the experiment

Design medium preference(s) during the experiment was important to at least

touch on as they may act as a mediator/moderator on the effects of formality on the

number of changes made in a design when presented on a different medium. However, as

subjects were presented with four designs on the Tablet PC and only one design one the

paper, design medium preference(s) as a mediator/moderator, therefore, could not be

statistically examined in a satisfactory manner. It would be more feasible in the future, to

include two sets of designs, from low formality to high formality, one set presented on the

tablet PC and the other set presented on the paper. It is then, that design medium

preference could be examined and compared as a mediator/moderator.

The null hypothesis in Hypothesis 4 was rejected as no significant difference was

found in design tool preference in during the experiment – the number of subjects who

preferred using paper and pen (thirteen participants, 43.3%) and the number of subjects

who preferred using the tablet PC (fifteen participants, 50%) was very similar. This

further supports that InkKit (tablet PC) bridges the gap between paper and computer, by

providing a sketching-based interface (as well as other additional functionalities) and thus,

subjects’ interaction with InkKit was comparable with pen and paper.

Factors relating to expertise that were likely to affect design tool preference were

examined briefly, including study major/specialization and design experience. Result

suggests that those who majored in CS/SE (who are likely to be more computer oriented)

are more likely to prefer using the tablet PC (Inkit) – a medium with conventional

computer functions such as selection, resizing, deletion, copy and paste, drag and drop,

undo, redo; as opposed to the traditional design tool/medium: pen and paper. Such result

also indicates that those who majored in non-CS/SE subjects are more likely to prefer

paper and pen over the use of tablet (Inkit).

However, as seen from the reasons for preference, subjects either preferred one

tool over the other because he/she liked using it better (the positive features of the

preferred tool were mentioned) or because he/she disliked using the ‘other’ tool or found

using the other tool dissatisfactory (the negative features of the non-preferred tool

mentioned); and thus, the preferred tool was only an indication of the better tool out of the

two. Therefore, it is difficult to judge objectively which design tool a participant really

preferred – i.e. did a participant prefer a particular tool because the other was bad or did

he/she prefer the tool that has the more advantages over the other tool?

Regarding no preference for either design tool used in the experiment, it can be

interpreted from the results such that subjects with CS/SE major and experience will tend

to notice the advantages and disadvantages of each tool (and the suitability for different

tasks, as one of the subjects had expressed – see Appendix 21); therefore, some subjects

indicated no particular design tool preference for one tool over the other as both tools had

their advantages.

Furthermore, design tool preference according to a subjects’ study major – one

must interpret the results with caution as the number of subjects in each group was

unbalance and therefore, only a proportion of subjects (within each group) preferring a

particular tool was shown. Additionally, in some cases with few subjects in the group, a

slight difference in the number of subjects (for example, a difference of one subject)

preferring one tool over another may have resulted in a more observable difference in

percentage. In comparison, the results showing subjects design tool preference according

to design experience was less unambiguous to interpret as the number of subjects was

balanced in each group. Hence, overall results are not conclusive to suggest whether there

is a difference in tool preference in subjects with different study majors and design

experience. To be more conclusive about design tool preferences (overall as well as

between groups), a more balanced number of subjects in each group and a greater number

of subjects on the whole would be needed.

On the whole, no direct relationships could be concluded that design tool

preference in the experiment affected the number of changes made across levels of

formality as the there was only one condition with paper and pen as opposed to four

conditions that required participants to use the Tablet PC along with the Tablet pen –

therefore comparisons could not be made explicitly; hence, future research. The greater

number of changes using paper and pen compared to using Tablet PC and the tablet pen

suggested that the two design tool do not belong on the same continuum in terms of

interaction with a design – two different dimension of design interaction mechanism.

Furthermore, even with some subjects having preference for one tool over another, the

main trend (number of changes made across levels of formality) still exists regardless of

design tool preferences – as formality increases, the number of changes made decreases.

This could be interpreted as: fewer design “errors” corrected (expected changes); fewer

improvements (quality changes), and generally fewer attempts made (total changes) to

improve the design – evidently, with respect one’s design expertise suggested by the

between-subjects effects. From another angle, as formality of a design decreases, the

number of changes made increases (and much greater in low formality design on paper)

regardless of design tool preference, which further suggests as designs appeared less

formal, more design errors were corrected (expected changes), more improvements were

made (quality changes), and attempts to improve the design increased (total changes)

4.6.2. Design tool preference in the real world

Again, due to the impracticalities and purpose of the study, no statistical analysis

was conducted. In comparison to design tool preference during the experiment, design

tool preference in the real world was more diverse and variable as subjects were free to

indicate different tools they may prefer to use if they were in real world design situations.

This also points out that different individuals may be more/less experienced with and

exposed to different tools, and whether they have access to it to increase experience is also

a factor that could affect an individual’s preference for a design tool(s). In real world

design situations, subjects indicated that they would prefer using: paper and pen; tablet PC

(InkKit); computer (other computer applications); pen and paper and tablet PC; pen and

paper, and computer (other computer applications), and one subject also indicated no

preference for different design tools.

Particularly, with the finding that most participants would prefer using both pen

and paper and (then) tablet PC, results supports and confirm previous findings that

conventionally, professionals (and students) often design with pen and paper first, then

move onto using computer to make the design look more formal (i.e. beautification). This

also reflects design education where students are taught to use paper and pen first, and

computer applications second. Furthermore, one is constantly reminded of the advantages

of using pen and paper, for example, during planning; essay plan; art; mathematics;

architecture; graphics design; engineering etc, by different disciplines in the education

sector (e.g in the context of human-computer interaction, Beudouin-Lafon & Mackay,

2001; Goel, 1996; Helander, 1988; Laurel & Mountford, 1990). Such teachings are

derived from the findings of many studies that suggest the sole use of computer as being

disruptive to the natural design process (e.g. Black, 1990; Goel, 1996) and may affect the

design outcome (e.g. Black, 1990; Goel, 1996); and the use of paper as being a greater tool

for sketching – one of the most important design activities during the early stages of the

design process (Black, 1990; Goel, 1996; Goldschmidt, 1991, 1994; 2003, Tversky, 1999;

Van der Lugt, 2005). However, in spite of the emphasis of and findings supporting the

advantages of using paper and pen in design (especially during early stages), due to the

resource constraints such as time and efficiency of transferring a paper design onto a

computer, (student) designers often choose not to use paper, but to design using a

computer at the early stages of the design process and some may use a computer from the

very beginning. For example, Newman et al. (2003) found in their web design practice

study that designers often move, from working with pen and paper, onto working on the

computer earlier than they wish to, but due to time constraints, they had to.

Additionally, preference in the “real world” could also be affect by the nature of

the design task – whether early stages or the refinement stages in the design process or

whether it is simple or complex design task, for example, simple tasks where not much

planning is needed (especially for expert designers, for example, designing a simple online

form that requires only the users’ name, address and a submit button) verses complex

designs where planning of website architecture, information design, and individual pages’

content and layout etc, maybe needed. In addition to factors such as familiarity and

competency, exposure and access to design tools, and whether a new design tool has better

usability, learnability, and functionality; after all, in a sense, it might just crumbles down

to personal preference for which tool to use – no matter how good a mobile phone is with

all the functions possible such as digital cameras, recorders, calculators etc that clearly has

its advantages, one might still choose to use the conventional mobile phone where making

and receiving phone calls, texting and receiving texts is good enough, and will do the job.

4.7. Implications

4.7.1. Implications on sketch-based tool development: Recommendations for InkKit

This study also served as an informal, preliminary usability analysis of Inkit.

Ultimately, InkKit has the potential to become an “all-in-one” design tool that supports

sketch-based input; while providing sketch editing functions (e.g. select and move,

relocation, resize, deletion, etc) as well as beautification capabilities (e.g. alignment, line

smoothing, standardization etc). Moreover, it is possible that in the future, Inkit could

support exporting of high-fidelity (high-formality) prototypes, in just one click of a button,

whether as a word document, excel, or Visual Basic.Net, or as HTML (as ink input data is

stored as XML files); allowing a prototype to be worked on using different applications.

The current version of InkKit, although still in the development stage, is showing some

promising results, where a story board, consisting of pages of hand-drawn sketches

connected to each other in the desired way, can be exported as an HTML page, where the

designer can navigate through it as if he/she was using it on the internet.

Tthere is also an implication on the types of beautification functionalities to be

included in sketching-based design applications and also, if beautification is used – at

what stages would it be most useful and effective? It could be useful to have different

levels of beautification to produce prototypes with different levels of formality for

different purposes – for presentation, review, or to serve as a display of layout. Future

development of beautification functionalities could also involve colour combinations,

HTML look-and-feel, shadows for buttons, and textboxes to allow input for testing and

evaluation. Furthermore, this means that if the preliminary design is primarily for internal

use, designers may want to skip step the using higher levels of beautification and use low

to medium levels of beautification to just to slightly tidy up a designer, but when

presenting a design to clients or a review committee, then higher levels of beautification

may be useful.

From observations and comments expressed by participants, there are some

recommendations for Inkit, in terms of usability improvement and technical development:

1) Speed of moving ink strokes around – sometimes very slow when moving or

resizing after a while of working on a design – buffer problem for moving data? It is likely

in applications such as Inkit, that a user would sketch and manipulate the content

frequently; therefore, one of the future development of Inkit should be on improving its

support for vast amounts of data manipulation – so that the simultaneous feedback can be

achieved.

2) One of the many difficulties faced in this study was the programming of

beautification functions in Inkit and one of them being hand-writing beautification – no

methods exists for mapping text in computerized fonts into ink strokes, which could be

manipulated as an ink stroke, as if it was an ink input on the screen. Hence, the back-door

method was used to creating different font that appeared to be more or less formal

objectively, by producing the text in the design as “labels” (programmed in Inkit to show

at the desired location on the designs). Future development of Inkit could also look at

overcoming such problem by

3) The previous point leads to the subject of reverse engineering where one can

take a webpage, and apply reverse engineering methods, to “de-beautify” the web page

and end up with a less formal design, for example, becomes mono-coloured with only

black as the ink colour (like the designs in the study). However, what level of formality a

webpage should be strip down to is still the fundamental question yet to be answered by

future research. Reverse engineering could be useful for redesign / review purposes as

suggested by the findings from the current study – more changes made (hence, higher

proportion of quality changes i.e. improvements) in designs with lower formality

compared to high formality designs; people may not think a web page needs to be changed

as it looks formal and nice already, but maybe the prettiness has blinded the designer from

usability problems etc.

4) Changing of modes.. – practice effects? Maybe a tablet rubber? Like a pen,

pencil and rubber interaction? Will it be the same? What’s are the effects? How will it

compare with the pen tap to change modes?

There are also implications on “rapid beautification” (during sketching) and

“smart beautification” (during formalization-time after sketching). Rapid beautification, if

inappropriately implemented and used, could interrupt the design process itself which is,

in turn, destructs the main purpose of beautification in informal tools. This interruption

could also induce annoyance and frustration when unnecessary and/or unwanted

beautification appears. The findings from the present study suggest that rapid (immediate)

recognition and beatification may have a negative effect in the design process.

Furthermore, even with beautification at formalization time, as a design appear prettier,

and more formal, fewer changes would be made to a design – even during the early stages

of the design process, in which informal properties of designs have shown to be important

in this study; thus, it may jeopardize the continuous, iterative process of and activities

during design, and ultimately, the design quality of the end product (starting from early

stages of the design process as informal design tools are used during that stage). As

Plimmer and Grundy (2005) agreed, developing sketching-based design tools that provide

different type of beautification is not straightforward. Therefore, more research is

required in the future to explore the effectiveness of using different beautification

techniques at different times in the design process, and design tasks in different domains

4.7.2. Improvements in the design process

Beautification in sketching-based design tools bridges the gap between paper and

pen and the computer, thus, the conventional transferring process of the design from paper

to the computer (a beautification process itself) is eliminated. This, in turn, could result in

an increase in efficiency (and also motivation?), which means more time for other design

activities – as designers can concentrate on sketching and ‘designing’, and beautify deigns

in just a few moments.

There is another implication from such findings on the design process in practical

work situations. As Newman et al. (2003) found in their web design practices study, in

practical design settings, particularly for client presentation – designers often want to

present to their client a professional looking design; thus, designers have to transfer their

paper design onto the computer by recreating the designs from scratch in common

computer applications such as photoshop. However, as Brink et al. (2002) suggested,

(also confirmed in Newman et al.’s (2003) study) one frustrating problem with client

presentation in the early stages of the design process is often the level of formality of the

design to be presented – designers want their designs to appear professional, yet not

wanting the client to view it as a finished design. With tools currently used in the

professional domain (i.e. pen and paper and computer software), to create designs that

appear unfinished and conceptual (informal) yet looking professional, medium-levels of

beautification could be useful in this sense to create designs that appear not too rough

(hand-drawn, sketchy and untidy) or too formal (computer-rendered, finished-looking

designs created by computer applications such as Photoshop, Word, Publisher etc.).

Generally, in the context of beautification in early stages of the design process, as

formality increased, people noticed fewer ‘errors’ in the designs. There are implications

for the design process and the final product – the use of beautification should be

considered carefully, especially during the early stages of the design process when

exploration of ideas occurred, as the results form this study suggest that the more a design

is beautified, the fewer changes people make to correct design errors. Hence, it is costly

for design companies if ‘errors’ are noticed late in the production phases

4.7.3. Implications on design education

There are implications from the results of the study, particularly the effects of

formality on quality and expected changes, on design education that are inline with the

implications from design education research – that students should be taught to design

(creation, review and edit) with paper and pen first during early stages of design, as

opposed to using a computer, because paper and pen supports sketching activities which

have been shown to be one of the most important design activities during the early stages

of the design process (e.g. Bilda, Gero, & Purcell, 2006; Goel, 1996; Lugt, 2005;

McGown, Green, & Rodgers, 1998; Ullman, Wood, & Graig, 1990; Verstijnen, et al,

1998).

Likewise, with sketch-based interfaces becoming more popular as pointed out by

Pomm and Werlen (2004), along with beautification (and recognition) techniques being a

explored (Qin, Wright & Jordanov, 2000; Plimmer & Grundy, 2005; Pomm & Werlen,

2004; Shesh & Chen, 2004), eventually, it is likely that such tools would be accessible for

design students (as well as design educators and professionals) from different disciplines –

only if and when usability testing shows good results which further facilitates

commercialization of such products for a wider user population. Hence, should sketch-

based design tools becomes one of the main stream design tools along with paper and pen,

and computers (using applications such as Computer-Aided Design (CAD), Microsoft

Photoshop, Fireworks etc), the effects of formality (beautification) should be emphasize

through design education, to help minimize the likelihood of error-filled designs lasting

through to the later stages in the design process, for example, in domains such as web

interface design; flow diagrams in mechanical design; and floor plans in architectural

design. It may not be too much of a problem in design projects and assignments for

students, but in the design industry, such mistake could be exceedingly costly for the

design company responsible. The impact could be even more profound in the design of

complex systems where errors/mistakes that are unnoticed until the later stages in the

design cycle (depending on its severity) could ultimately put the design at risk of being

redeveloped – i.e. back to the early stages of design; which could mean extra stress for the

designers and his/her company, as well as critical strains on the already-limited resources,

i.e. mainly time (and money – such as in physical product design when prototypes are

required). Hence, design education and training must address the effects of formality on

design performance, and thus, increase caution when beautifying designs.

Although, the findings from this study suggests that an active approach should be

taken towards addressing the issues with levels of beautification and formality in sketch-

based design tools in the design process – i.e. minimizing possible undesired effects of

formality in early design – there are other factors that may affect the use of beautification

and its effectiveness. For example: the nature and scope of the design task – whether it is a

complex design (e.g. a simple website with three to four pages or a whole information

system) or a simple design (e.g. a simple online form on a design that only requires a user

to fill in their name and address, or a complex online form that requires more extensive

user input); and the level(s) of formality one wants to achieve; here, recommendations for

different situations regarding the use of beautification could be useful.

4.8. Methodological issues and limitations

In addition to the limitations discussed earlier including: small and heterogeneous

sample; content in the five design; possible experimenter’s bias in the five forms designed;

and levels of formality created and their intervals; the other limitation of this study was

that InkKit was (and is) still in its development stage. Although it closely matches the

Tablet Diary (Tablet Edition, 2005) program in which users are able to write/draw

anything (using the pen), several participants commented that InkKit was “a bit hard to

use” in the experiment as they had to switch between different modes i.e. Inking mode,

Eraser mode, Selection mode by pointing the pen tip to the icons at the top of the screen.

According to Fitts Law (with respect to VDU display design), the larger the distance from

one object to another on the screen, the longer it takes for the eyes to fixate from one

object to another object on the screen. This suggests that, as the hand movement is guided

by the eyes, the time it takes for the hand to guide the cursor from one object to another

also increases. Hence, the design task in the experiment required participants to make

changes to an existing design, by deleting, adding, resizing, relocating etc, therefore,

participants had less freedom to stay in one mode. This mode switching may have

affected participants’ willingness to make changes; hence, results on the effects of

formality on the number of changes made (design performance) should be interpreted with

caution. An implication of this on interface design is to better accommodate switching of

modes to support natural human-computer interaction.

4.9. Future research and directions

This study is one of the first to examine the different levels of formality and their

effects on design performance, and thus, the findings from the present study raised many

specific as well as wider questions on the effects of formality on design performance;

which in turn, forms the basis for future direction in similar research. The limitations and

methodological issues found in the present study also provide useful resources for

improvements in methodological aspects in similar research in the future. In addition to

the research recommendations discussed earlier along with the limitations of the study, for

example, exploring more dimension and levels of formality, future research could also

explore the following.

More laboratory studies are needed to further explore the effects of sketch-based

design tools, particularly the effects of levels of formality of designs (as a result of

different degrees of beautification), on the design performance, behaviour, cognition and

perception during the design process. For example, more levels of beautification could be

explored by including aspects such as the HTML ‘look and feel’ (e.g. of dropdown menus,

textboxes, buttons, radio buttons); simple colours to represent differences between

components (Brinck, et al., 2002); hence, whether the negative linear relationship (levels

of formality and design performance) will still exist becomes an important question to be

answered. Field studies should also be done to check external validity to support the

(non)practicality and usefulness of beautification in real world situations such as

presentation to clients in early stages in the design process (Newman, et al, 2003).

As beautification has been regarded as an important aspect of informal sketching-

based tools (Plimmer and Apperley, 2004), different levels of beautification was explored

in the present study. It would be useful, in terms of design education, for future research

to further explore the effects of different degrees of beautification at other stages in the

design process. In addition, evaluation studies on informal sketching-based tools that

provide beautification functionalities might help with design tool development. Such

studies will evaluate whether beautification functions can be used effectively, and if

included in an informal sketching-based tool, at what level and to what extent should it be;

and whether it is plausible or practical? Furthermore, it would also be valuable to examine

the effects and usefulness of beautification in other (different) types of applications e.g.

programs that support pen-input of music scores, mathematic formulas, simple drawings,

and even kids drawing program (which has been an interest to many researchers since the

sketch-based applications became popular). In addition to different applications, the

effects and the effectiveness of beautification at different levels for different groups of

users could also be investigated. Moreover, one question yet to be answered is whether

there is an optimal level of beautification to create the appropriate level of formality of

designs; hence, future extensions to the present study to provide a more comprehensive

picture of relationships between different levels of beautification, formality and design

performance.

The present study examined individual performance in a laboratory setting only;

however, in real design situations, designers often work in teams (Stempfle & Badke-

Schaub, 2002). Thus, the present study could be expanded to examine the effects of

formality on team performance. However, results from such study may be difficult to

interpret and examine as variability within a team may differ between teams – a

combination of qualitative and quantitative data maybe useful for a fuller account and

analyzes.

One underlying (and fundamental) question raised from the present study, besides

the effects of formality (as a result of beautification) on design performance, is the effects

of using different design mediums. Therefore, future research should explore the effects

of formality on the design process using different design medium, in a more systematic

manner, for example, by comparing subjects between two groups (with similar design and

study backgrounds) where subjects in one group works on designs with different levels of

formality presented on the tablet PC; while subjects in the other group works on designs

with different levels of formality presented on paper (printed versions of the designs on

the tablet PC). Thus, differences and/or similarities of using different design mediums

combined with the effects formality could be explored.

Results from the present study suggest that individual differences played a role in

affecting the overall number of changes made across designs with different levels of

formality. It was observed that some subjects just make more changes overall in

comparison to others – maybe they are more motivated to make changes. Some subjects

tries hard and some subjects did not seem to exert too much effort – shown in the

consistent high number of changes compared the consistently low number of changes as

seen in individuals regardless to his/her design experience, study level or study

major/specialization. It was observed also, that some participants were able to concentrate

the whole way throughout the five conditions, utilizing every minute during the tasks as

opposed to other participants who made few changes and stopped before the 10minutes

time. This was also noticeable in the recorded on-screen actions throughout the five

conditions – some participants had little on-screen actions compared to others who’s

screen constantly changes; hence, the more changes on a screen (e.g. movements) the

larger the video file. Such observations supports that motivation and personality play a

significant role in influencing (work) performance, especially in the field of work,

industrial and organizational psychology as shown in many psychological studies reported

by Spector (2006). Future research on motivation and personality in the context of design

performance will contribute to a greater understanding of designers’ interaction with

different elements and artifacts within the design process – a long term research direction.

Furthermore, findings from such studies could be useful to the design industry, for

purposes such as personality screening during the recruitment process and motivation

increase to help improve performance (e.g. design quality, design decisions, innovation

and efficiency).

Chapter 5. Summary and Conclusion

There has been little of research on designers’ interaction with informal sketching-

based tools, particularly, within the context of beautification – an important aspect in such

tool. Moreover, the effects of beautification on the designers (e.g. behavior, perception,

cognition, performance), have been largely neglected as many researchers have been

focusing on improving recognition and beautification techniques in sketching-based tools.

Overall, the present study provided a strong starting point in a new path of sketch-based

tools research by exploring different levels of formality (created by applying different

degrees of beautification) and their effects on designers’ performance in the early stages of

the design process.

Beautification was explored in the present study, by developing a taxonomy of

degrees of beautification, which was validated by producing designs that appeared more,

or less formal (i.e. with different levels of formality). Furthermore, the designs produced

by systematically varying the degree of beautification, were also used for exploring the

effects of formality on design performance – measured in terms of number of changes

made to a design presented – during the early stages of the design process.

The findings in the present study confirmed previous findings by Black (1990) and

Plimmer and Apperley (2004) that reviewers/designers interact differently with designs

that appears untidy, sketchy, rough and informal (i.e. low formality designs) and designs

that appears tidy, computer-rendered, and formal (i.e. high formality designs). In addition,

the current study examined not only the two levels of formality that Black and Plimmer

and Apperley looked at, but also the other levels of formality in between – where a design

appeared more or less formal; hence, significant negative linear trend found.

Results showed that formality of a design affects design performance, such that as

the level of formality increases, the number of changes made (total, quality and expected

changes) decreases, and vice versa; demonstrating a negative linear relationship between

formality and design performance. Moreover, results showed that formality of designs

affected both the experts and novices, and that experts performed at a higher level in

comparison to novices’ performance (demonstrated by the significant between-subjects

effects). Subjective measurements included overall enjoyment and design tool

preferences. Subjects enjoyed working on designs that appeared more formal (higher

formality – i.e. more beautified) more than designs that appeared rougher and less formal

(lower formality – i.e. less beautified). There was no difference found in the preference

between designing on paper compared to designing on the tablet PC (InkKit) during the

experiment. On the other hand, results showed that design tool preference(s) in real world

design situations was more diverse than design medium/tool preference during the

experiment.

Important implications arose from this study include: 1) design education on the

effects of formality as a result of beautification, and the caution one should take when

using beautification functions in informal sketching-based tools such as InkKit; 2)

improvements on the design process such as easier preparation for client presentation and

improved efficiency which could leave more time for actual ‘designing’; and 3) informal

sketching-based tool development, in particular InkKit, to support more satisfying, natural

designer-design tool interaction.

As beautification has been regarded as an important aspect of informal sketching-

based tools (Plimmer and Apperley, 2004), and that designs can be beautified to different

extents, thus appearing more or less formal; one fundamental question that arise, is

whether there is an optimal level of beautification to create the appropriate level of

formality of designs – this is particularly important to understand for designers in practical

settings as client presentation is often frustrating; as one of the participants in Newman et

al.’s (2003) study as a design process itself. Hence, future replications and extensions to

the present study will provide a more comprehensive model of relationships between

different levels of beautification, formality and design performance. This, in turn, will

help nurture our understanding on designers’ interaction with informal sketch-based tools

in the context of beautification; hence, a new path and angle towards design process

research.

References

Adams, R. S., Turns, J., & Atman, C. (2003). What could design learning look like? In N. Cross and E. Edmonds (Eds.), Expertise in Design. University of Technology, Sydney, Australia: Creativity and Cognition Press.

Agar, P., & Novins, A. (2003). Polygon recognition in sketch-based interfaces with immediate and continuous feedback. Paper presented at the Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East Asia, Melbourne, Australia.

Akin, O. (1986) Psychology of Architectural Design, Pion, London.

Akin, O., & Lin, C. (1995). Design protocol data and novel design decisions. Design Studies: Analysing Design Activity, 16(2), 211-236.

Alavi, M. (1984). An assessment of prototyping approach to information systems development. Communications of the ACM, 27(6), 556-563.

Aliakseyeu, D., Martens, J.-B., & Rauterberg, M. (2006). A computer support tool for the early stages of architectural design. Interacting with Computers, 18(4), 528-555.

Alvarado, C., & Davis, R. (2006). Resolving ambiguities to create a natural computer-based sketching environment. Paper presented at the International Conference on Computer Graphics and Interactive Techniques, Boston, Massachusetts.

Arditi, A. (2004). Adjustable typography: An approach to enhancing low vision text accessibility. Ergonomics, 47(5), 469-482.

Arditi, A., & Cho, J. (2005). Serifs and font legibility. Vision Research, 45, 2926–2933.

Arvo, J., & Novins, K. (2000). Fluid Sketches: Continuous recognition and morphing of simple hand-drawn shapes. Paper presented at the Proceedings of UIST ’00, San Diego, 73-80.

Arvo, J., & Novins, K. (2005). Appearance-preserving manipulation of hand-drawn graphs. Paper presented at the Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, Dunedin, New Zealand.

Atman, C. J., Cardella, M. E., Turns, J., & Adams, R. (2005). Comparing freshman and senior engineering design processes: an in-depth follow-up study. Design Studies, 26(4), 325-357.

Atman, C. J., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A comparison of freshman and senior engineering design processes. Design Studies, 20(2), 131-152.

Bakera, E. L., & O’Neil Jr, H. F. (2002). Measuring problem solving in computer environments: current and future states. Computers in Human Behavior, 18(6), 609–622.

Bailey, B. P., & Konstan, J. A. (2003). Are Informal Tools Better? Comparing DEMAIS, pencil and paper, and authorware for early multimedia design. CHI '03 Proceedings of the SIGCHI conference on Human factors in computing systems 5(1), 313-320.

Beaudouin-Lafon, M., & Mackay, W. (2003). Prototyping tools and techniques. In J. A. Jacko & A. Sears (Eds.), The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (pp. 1-1277). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Benyon, D., Davies, G., Keller, L., Preece, J., & Rogers, Y. (1993). A Guide to Usability: Human Factors in computing. Workingham, England: Addison-Wesley.

Bernard, M., Mills, M., Peterson, M., & Storrer, K. (2001, July). A comparison of popular online fonts: Which is best and when? Usability News, 3(2). Retrieved June 13, 2006, from, http://psychology.wichita.edu/surl/usabilitynews/3S/font. htm

Bilda, Z., & Demirkan, H. (2003). An insight on designers' sketching activities in traditional versus digital media. Design Studies, 24(1), 27-50.

Bilda, Z., Gero, J. S., & Purcell, T. (2006). To sketch or not to sketch? That is the question. Design Studies, In Press, Corrected Proof.

Black, A. (1990). Visible planning on paper and on screen: The impact of working medium on decision-making by novice graphic designers. Behaviour & Information Technology, 9(4), 283-296.

Boehm, B. W., Gray, T. E., & Seewaldt, T. (1984). Prototyping verses specifying: A multiproject experiment. IEEE Transactions on Software Engineering, SE-10(3), 290-303.

Bolz, D. (1993, February). Some aspects of the user interface of a knowledge based beautifier for drawings. Paper presented at the Proceedings of the 1st international conference on Intelligent user interfaces Orlando, Florida, United States, 45-52.

Bo-Yu, S. C. (2003, February). Sketch: A domain-independent system for sketch recognition. Paper presented at the Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East Asia, Melbourne, Australia

Boutsen, L., & Humphreys, G. W. (1999). Axis-alignment affects perceptual grouping: Evidence from simltanagnosia. Cognitive Neuropsychology, 17(7), 655-672.

Braisby, N., & Gellatly, A. (Eds.). (2005). Cognitive Psychology. Oxford: Oxford University Press.

Brink, T., Gergle, D., & Wood, S. D. (2002). Design web site that works: Usability for the web. San Francisco: Morgan Kaufmann Publishers.

Burns, C. M., & Vicente, K. J. (2000). A participant-observer study of ergonomics in engineering design: How constraints drive design process. Applied Ergonomics, 31(1), 73-82.

Carroll, J. M. (2006). Introduction to this special issue on foundation of design in HCI. Special Issue of Human-Computer Interaction, 21(1), 1-3.

Carroll, J. M., & Rosson, M. B. (1991). Deliberated evolution: Stalking the view matcher in design space. Human-Computer Interaction, 6(3/4), 197-200.

Carroll, J M., Thomas, J C & Mallaotra, A. (1980). Presentation and representation in design problem-solving. British Journal of Psychology, 71, 143-153.

Cheutet, V., Catalano, C. E., Pernot, J. P., Falcidieno, B., Giannini, F., & Leon, J. C. (2005). 3D sketching for aesthetic design using fully free-form deformation features. Computers & Graphics, 29(6), 916-930.

Christiaans, H & Dorst, C. (1992). Cognitive models in industrial design engineering: A protocol study. In D. L. Taylor & D. A. Stauer (Eds.), Design Theory and Methodology - DTM92. American Society of Mechanical Engineers, New York, USA.

Chung, R., Mirica, P., & Plimmer, B. (2005, July 6-8). InkKit: a generic design tool for the tablet PC Paper presented at the Proceedings of the 6th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: making CHI natural CHINZ '05, Auckland, NZ.

Crampton Smith, G., & Tabor, P. (1996). The role of the artist-designer. In T. Winogard (Eds.), Bringing design to software (pp. 37-57). New York: ACM Press.

Cross, N. (1985). Styles of learning, designing and computing. Design Studies, 6, 157-162.

Cross, N. (2004). Expertise in design: an overview. Design Studies: Expertise in Design, 25(5), 427-441.

Cross, N., Christiaans, H., & and Dorst, K. (1994). Design expertise amongst student designers. Journal of Art and Design Education, 13(1), 39–56.

Cross, N., Christiaans, H., & Dorst, K. (Eds.) (1996). Analysing design activity. John Wiley & Sons Ltd., Chichester, UK.

Czaja, S. J., Sharit, J., Ownby, R., Roth, D. L., & Nairs, S. (2001). Examining age differences in performance of a complex information search and retrieval task. Psychology and Aging, 16, 564-579.

Czerwinski, M. (2003). Humans in human-computer interaction. In J. A. Jacko & A. Sears (Eds.), The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (pp. 1-1277). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Damm, C. H., & Hansen, H. R. (2002). Ideogramic. http://www.ideogramic.com/ Retrieved 27 April, 2006.

Darke, J. (1979). The primary generator and the design process. Design Studies, 1(1), 36-44.

Dickinson, J. K., Yu, Z., Zeng, Y., & Antunes, H. (2005). Pen-tablet as a CAD interface alternative. Robotics and Computer-Integrated Manufacturing, 21(4-5), 465-474.

Dillon, A. (2003). Designing usable electronic text (2nd ed.). New York: Taylor & Francis.

Downton, P. (2003). Design research. Melbourne: RMIT University Press.

Dumas, J., & Redish, G. (Revised Edition) (1999). A practical guide to usability testing. London: Intellect Books.

Eisentraut, R. (1999). Styles of problem solving and their influence on the design process. Design Studies, 20(5), 431-437.

Eisentraut, R., & Gunther, J. (1997). Individual styles of problem solving and their relation to representations in the design process. Design Studies: Descriptive models of design, 18(4), 369-383.

Elias, J. S. (2005). The engineering design mind: A cognitive model and a case study. Dissertation Abstracts International: Section B: The Sciences and Engineering, 66, 11B (UMI No. 3194788).

Ericsson, K. A., & Simon, H. A. (1984). Protocol Analysis. Cambridge: MIT Press.

Fields, C. (2004). The role of aesthetics in problem solving: Some observations and a manifesto. Journal of Experimental and Theoretical Artificial Intelligence, 16(1), 41-55.

Fitzmaurice, G. W., Balakrishnan, R., Kurtenbach, G., & Buxton, B. (1999, May). An Exploration into Supporting Artwork Orientation in the user interface. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit Pittsburgh, Pennsylvania, United States

Forbus, K. D., Ferguson, R. W., & Usher, J. M. Towards a computational model of sketching, Intelligent User Interfaces, 77-83.

Fowler, S., & Stanwick, V. (2004). Web application design handbook: Best practices for web-based software. Boston: Morgan Kaufmann Publishers.

Gasser, M., Boeke, J., Haffernan, M., & Tan, R. (2005). The influence of Font type on information recall. North American Journal of Psychology, 7(2), 181-188.

Goel, V. (1996). Sketches of thoughts. Cambridge, Massachusetts: The MIT Press.

Goldschmidt, G. (1991). The dialectics of sketching. . Creativity Research Journal, 4(2), 123-143.

Goldschmidt, G. (1994). On visual design thinking: The vis kids of architecture, Design Studies, 15, 158-174.

Goldschmidt, G. (1995). The designer as a team o f one. Design Studies: Analysing Design Activity, 16(2), 189-209.

Goldschmidt, G. (1997). Capturing indeterminism: representation in the design problem space. Design Studies, 18(4), 441–445.

Goldschmidt, G. (2003). The backtalk of self-generated sketches. Design Issues, 19(1), 72-88.

Grandjean, E., Hunting, W., & Pidermann, M. (1983). VDT workstation design: Preferred settings and their effects. Human Factors, 25(2). 161-175.

Grant, E. R., & Spivey, M. J. (2003). Eye movements and problem solving: Guiding attention guides thought. Psychological Science, 14(5), 462-466.

Green, A. J. K., & Gilhooly, K. (2005). Problem solving. In N. Braisby & A. Gellatly (Eds.), Cognitive Psychology (pp. 344-381). Oxford: Oxford University Press.

Grekas, D. N., & Frangopoulos, C. A. (2000). A heuristic algorithm for drawing of a flow diagram. Advances in Engineering Software, 32(3), 239-253.

Grosjean, S. F., P; Brassac, C. (2000). Those "psychological tools" inside the design process. Knowledge-Based Systems, 13(1), 3-9.

Gross, M. D. (1996). The electronic cocktail napkin: A computational environment for working with design diagrams. Design Studies, 17, 53-69.

Gross, M. D., & Do, E. Y.-L. (1996, April 13-18). Demonstrating the electronic cocktail napkin: a paper-like interface for early design. Paper presented at the Conference on Human Factors in Computing Systems - CHI'96 Proceedings.

Gross, M. D., Do, E.Y.-L., McCall, R.J., Citrin, W.V., Hamill, P., Warmack, A., Kuczun, K.S. (1998). Collaboration and coordination in architectural design: approaches to computer mediated team work. Automation in Construction 7(6), 465-473.

Hamade, R. F., Artail, H. A., & Jaber, M.Y. (2005). Learning Theory as Applied to Mechanical CAD: Training of Novices. International Journal of Human-Computer Interaction, 19(3), 305-322.

Hanna, R., & Barber, T. (2001). An inquiry into computers in design: attitudes before-attitudes after. Design Studies, 22(3), 255-281.

Hannah, G. G. (2002). Elements of design: Rowena Reed Kostellow and the structure of visual relationships . New York: Princeton Architectural Press.

Hassenzahl, M. (2004). The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 19(4), 319-349.

Hegarty M (1991) Knowledge and processes in mechanical problem solving. In: Sternberg, R. J., Frensch, P. A. (Eds.), Complex problem solving: Principles and mechanisms. Hillsdale: Erlbaum.

Helander, M. (Ed.). (1988). Handbook of Human-Computer Interaction. New York: Elsevier Science Publishers.

Henzen, A., Ailenei, N., Fiore, F. D., & Patterson, F. V. R. J. (2005). Sketching with a low-latency electronic ink drawing tablet. Paper presented at the Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia Dunedun, New Zealand.

Ho, C.-H. (2001). Some phenomena of problem decomposition strategy for design thinking: Differences between novices and experts. Design Studies, 22, 27-45.

Holt, K. (1991). The impact of technology strategy on the engineering design process. Design Studies, 12(2), 90-95.

Hong, J. I., & Landay, J. A. (2006). SATIN: a toolkit for informal ink-based applications. Paper presented at the International Conference on Computer Graphics and Interactive Techniques, Boston, Massachusetts.

Houde, S & Hill, C. (1997). What do prototypes prototype? In M Helander, T Landauer and P Prabhu (2nd

ed. rev.), Handbook of human-computer interaction (pp367-381). Elsevier Science: Amsterdam.

Hse, H. H., & Newton, A. R. (2005). Recognition and beautification of multi-stroke symbols in digital ink. Computers & Graphics, 29(4), 533–546.

Jacko, J., & Stephanidis, C. (Eds.). (2003). Human-Computer Interaction: Theory and practice (Part 1) (Vol. 1). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Jacko, J. A., & Sears, A. (Eds.). (2003). The Human-Computer Interaction handbook: Fundamentals, evolving technologies and emerging applications. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Joiner, R., Messer, D., Light, P., & Littleton, K. (1998). It is best to point for young children: A comparison of children’s pointing and dragging. Computers in Human Bahaviour, 14, 513-529.

Jonson, B. (2005). Design ideation: the conceptual sketch in the digital age. Design Studies, 26(6), 613-624.

Kara, L. B., & Stahovich, T. F. (2004). Sim-U-Sketch: a sketch-based interface for SimuLink. Paper presented at the Proceedings of the working conference on Advanced visual interfaces, Gallipoli, Italy.

Kavakli, M., Scrivener, S. A. R., & Ball, L. J. (1998). Structure in idea sketching behaviour. Design Studies, 19(4), 485-517.

Khalid, H. M. (2001). Modelling and analysis of CAD expert behaviour in using manual input devices. International Journal of Industrial Ergonomics, 27(2), 79-92.

Kim, D. H., & Kim, M. J. (2006). A new modeling interface for the pen-input displays. Computer-Aided Design, 38(3), 210-233.

Koffka, K. (1935). Principles of Gestalt psychology. London: Routledge.

Kramer, A. (1995, May 7-11). Supporting design activities in the written medium. Paper presented at the CHI' 95 Mosaic of Creativity, Denva, Colorado, USA.

Kruger, C., & Cross, N. (2006). Solution driven versus problem driven design: strategies and outcomes. Design Studies, In Press, Corrected Proof.

Kvan, T., & Yunyan, J. (2005). Students’ learning styles and their correlation with performance in architectural design studio. Design Studies, 26, 19-34.

Lahti, H., Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2004). Collaboration patterns in computer supported collaborative designing. Design Studies, 25(4), 351-371.

Landauer, T. K. (1996). The trouble with computers: Usefulness, usability and productivity. Cambridge MA: MIT Press.

Landay, K. A. (1996, April). SILK: Sketching interfaces like kraz. Paper presented at the Conference on Human factors in computing systems, Vancouver, British Columbia, Canada

Landay, J. A., & Myers, B. A. (2001, March). Sketching interfaces: Toward more human interface design. IEEE Computer, 34(3), 56-64.

Lanier, P. (2004). My Font Tool for Tablet PC (Version 1.0) [Computer software]. Microsoft Corporation.

Laurel, B., & Mountford, S. J. (1990). Art of HCI Interface design. New York: Addison-Wesley Publishing Co.

Launis, M., Vuori, M., & Lehtela, J. (1996). Who is the workplace designer? -- Towards a collaborative mode of action. International Journal of Industrial Ergonomics Computer-aided Ergonomics and Safety, 17(4), 331-341.

LaViola, J. J., & Zeleznik, J. R. C. (2006). MathPad2: a system for the creation and exploration of mathematical sketches. Paper presented at the International Conference on Computer Graphics and

Interactive Techniques, Boston, Massachusetts.

Levinson, J. (Ed.). (2003). The oxford dictionary of aesthetics. New York: Oxford University Press.

Lawson, B., & Loke, S. M. (1997). Computers, words and pictures. Design studies, 18(2), 171-183.

Lewis, J. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57-78.

Lin, J., Newman, M. W., Hong, J. I., & Landay, J. A. (2000, 1-6 April). DENIM: Finding a tighter fit between tools and practice for Web site design. Paper presented at the Proceedings of the SIGCHI conference on Humana factors in computer systems, The Hague, The Netherlands.

Ling, J., & Van Schaik, P. (2006). The influence of font type and line length on visual search and information retrieval in web pages. International Journal of Human-Computer Studies, 64(5), 395-404.

Liu, Y.-J., Tang, K., & Joneja, A. (2005). Sketch-based free-form shape modelling with a fast and stable numerical engine. Computers & Graphics, 29(5), 771-786.

Lloyd, P., Lawson, B., & Scott, P. (1995). Can concurrent verbalization reveal design cognition? Design Studies, 16(2), 237-259.

Locher, P. J., Stappers, P. J., & Overbeeke, K. (1998). The role of balance as an organizing design principle underlying adults' compositional strategies for creating visual displays. Acta Psychologica, 99(2), 141-161.

Luczak, H., & Springer, J. (1997). Ergonomics of CAD systems. In M. G. Helander, Landauer, T.K., Prabhu, P.V. (Ed.), Handbook of Human Computer Interaction (pp. 1349-1394). Amsterdam: North-Holland.

Van der Lugt, R. (2005). How sketching can affect the idea generation process in design group meetings. Design Studies, 26(2), 101-122

Maarse, K. F., Schomaker, L. R. B., & Teulings, H.-L. (1988). Automatic identification of writers. In G. C. c. d. Veer & G. Mulder (Eds.), Human-Computer Interaction: Psychonomic Aspects (pp. 335-352). Berlin, Heidelberg: Springer-Gerlag.

MacLean, A., Young, R. M., Bellotti, V. M. E., & Morgan, T. P. (1991). Questions, options, and criteria: Elements of design space analysis. Human-Computer Interaction, 6, 319-355.

May, J. (2001). Human-Computer Interaction. In International Encyclopedia of the Social & Behavioral Sciences (pp. 7031-7035). Oxford: Pergamon.

Maybew, D. J. (2003). Requirements specification within the usability engineering life cycle. In J. A. Jacko & A. Sears (Eds.), The Human-Computer Interaction handbook: Fundamentals, evolving technologies and emerging applications (pp. 264-285). Mahwah, New Jersey: Lawrence Erlbaum Asssociates.

McGown, A., Green, G., & Rodgers, P. A. (1998). Visible ideas: information patterns of conceptual sketch activity. Design Studies, 19 (4), 431–453.

Molich, R., & Nielsen, J. (1990). Improving a human-computer dialogue. Communications of the ACM, 33, 338-348.

Morae Recorder (Version 1.1.0.) [Computer software]. (2004). TechSmith Corporation.

Mulet, E., & Vidal, R. (2006). Functional requirements for computer-based design support systems, derived from experimental studies. Knowledge-Based Systems, 19(1), 32–42.

Mullen, M. (1991). PICTIVE: An exploration in participatory design. Proceedings of ACM Conference on Human Factors in Computing System (CHI’ 91) (pp. 225-231). New York: ACM Press.

Naga, Y., & Noguchi, H. (2003). An experimental study on the design thinking process started from difficult keywords: modeling the thinking process of creative design. Journal of Engineering Design, 14(4), 429–437.

Nakakoji, K., Tanaka, A., & Fallman, D. (2006, April). "Sketching" nurturing creativity: commonalities in art, design, engineering and research. Paper presented at the CHI '06 extended abstracts on Human factors in computing systems CHI '06 Montréal, Québec, Canada.

Nakarada-Kordic, I. (2004). Web aesthetics and usability : The effect of perceived attractiveness of web interface design on visual search of web sites. Unpublished Master of Science Thesis, University of Auckland, Auckland, NZ.

Nealen, A., Sorkine, O., Alexa, M., & Cohen-Or, D. (2005). A sketch-based interface for detail-preserving mesh editing. Paper presented at the International Conference on Computer Graphics and Interactive Techniques, LA, California.

Neisser, U. (1976). Cognition and reality: Principles and implications of cognitive psychology. San Francisco: Freeman.

Newman, M. W., Lin, J., Hong, J. I., & Landay, J. A. (2003). DENIM: An informal web site design tool inspired by observations of practice. Human-Computer Interaction, 18(3), 259-325.

Newell. A. (1968). On the analysis of human problem solving protocols. In J. C. Gardin & B. Jaulin (Eds.). Calcul et formalization dans les sciences de l’homme (pp. 146-185). Paris: Centre National dela Recherche Scientific.

Nielsen, J., & Mack, R. L. (Eds) (1994). Usability inspection methods. New York: John Wiley.

Overbeeke, K., & Wensveen, S. (2004). Beauty in use. Human-Computer Interaction, 19(4), 367-369.

Oh, J.-Y. (2004). Design Evaluater: Critiquing freehand sketches. Masters of Science thesis, Design Machine Group, Department of Architecture. University of Washington, Seattle.

Oh, J.-Y., Stuerzlinger, W., & Danahy, J. (2006). SESAME: Towards better 3D conceptual design systems. Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, DIS, 2006, 80-89.

Okabe, M., Owada, S., & Igarashi, T. (2005). Interactive design of botanical trees using freehand sketches and example-based editing. Eurographics, 24(3), 487-496.

Pancer, S. M., George, M., & Gebotys, R. J. (1992). Understanding and predicting attitudes toward computers. Computers in Human Behavior, 8, 211-222.

Plimmer, B., & Apperly, M. (2001). Computer-Aided sketching to capture preliminary design. Paper presented at the Third Australasian User Interfaces Conference (AUIC2002), Melbourne, Australia. In J. Grundy & P. Calder (Ed.), Conferences in Research and Practice in Information Technology (pp 9-12). Australian Computer Society.

Plimmer, B. E., & Apperley, M. (2003). Software for Students to Sketch Interface Designs. In Proceedings of Interact, Zurich, 73-80.

Plimmer, B., & Apperley, M. (2004). Interacting with sketched interface designs: An evaluation study. Paper presented at the CHI '04 Conference on Human Factors in Computing Systems archive, Vienna, Austria.

Plimmer, B. E, & Grundy, J. (2005, January). Beautifying sketching-based design tool content: issues and experiences. Paper presented at the Proceedings of the Sixth Australasian conference on User interface (AUIC2005), Newcastle.

Pomm, C., & Werlen, S. (2004, May). Smooth morphing of handwritten text. Paper presented at the Proceedings of the working conference on Advanced visual interfaces, Gallipoli, Italy.

Poulton, E. C. (1965). Letter differentiation and rate of comprehension in reading. Journal of Applied Psychology, 49(5), 358-362.

Purcell, A. T., & Gero, J. S. (1998). Drawings and the design process: A review of protocol studies in design and other disciplines and related research in cognitive psychology. Design Studies, 19(4), 389-430.

Qin, S. F., Wright, D. K., & Jordanov, I. N. (2000). From on-line sketching to 2D and 3D geometry: a system based on fuzzy knowledge. Computer-Aided Design, 32(14), 851-866.

Qin, S. F., Wright, D. K., & Jordanov, I. N. (2001). A conceptual design tool: a sketch and fuzzy logic based system. Proceedings of the Institution of Mechanical Engineers - Part B: Engineering Manufacture, 215(1), 111-116.

Reason, J. (1990). Human error. Cambridge: Cambridge University Press.

Romer, A., Leinert, S., & Sachse, P. (2000). External support of problem analysis in design problem solving. Research in Engineering Design: Theory, Applications, and Concurrent Engineering, 12(3), 144–151.

Rosson, M. B., & Carroll, J. M. (2003). Scenario-based design. In J. A. Jacko & A. Sears (Eds.), The Human-Computer Interaction handbook: Fundamentals, evolving technologies and emerging applications (pp. 264-285). Mahwah, New Jersey: Lawrence Erlbaum Asssociates.

Rowe, P. G. (1987). Design thinking. MIT Press, Cambridge, MA.

Rubine, D. (1991). Specifying gestures by example. Computer Graphics (SIGGRAPH '91 Proceedings), 24(5), 329-337.

Rupp, M., Burg, A., & Beck, E. (2003). Rapid prototyping for wireless designs: the five-ones approach. Signal Processing, 83(7), 1427-1444.

Sachse, P., Leinert, S., & Hacker, W. (2001). Designing with computer and sketches. Swiss Journal of Psychology, 60 (2), 65–72.

Sano, D. (1996). Designing large-scale Web sites: A visual design methodology. New York, Wiley.

Sanocki, T. (1998). Font regularity constraints on the process of letter recognition. Journal of Experimental Psychology: Human Perception and Performance, 14(3), 472-480.

Saund, E., & Moran, T. P. (1994). A perceptually-supported sketch editor. Paper presented at the ACM Symposium on User Interface Software and Technology, Marina del Ray. CA.

Savage, J. C. D., Miles, C., Moore, C. J., & Miles, J. C. (1998). The interaction of time and cost constraints on the design process. Design Studies, 19(2), 217-233.

Scaife, M. and Rogers, Y. (1996). External cognition: How do graphical representations work? International Journal of Human-Computer Studies, 45, 185-213.

Schenk, P. (1991). The role of drawing in the graphic design process. Design Studies, 12(3), 168-181.

Schweikardt, E., & Gross, M. D. (2000). Digital clay: deriving digital models from freehand sketches. Automation in Construction, 9(1), 107-115.

Sear, A, Karat, C.-M., Oseitutu, K., Karimullah, A., & Feng, J. (2001). Productivity, satisfaction, and interaction strategies of individual with spinal cord injuries and traditional users interacting with speech recognition software. Universal Access in the Information Society, 1, 1-12.

Sezgin, T. M., Stahovich, T., & Davis, R. (2001, November). Sketch based interfaces: early processing for sketch understanding. Paper presented at the Proceedings of the 2001 workshop on Perceptive user interfaces, Orlando, Florida.

Shelly, G. B., Cashman, T. J., & Woods, D. M. (2005). HTML: Complete concepts and techniques (Third Ed.). Shelly Cashman Series, Australia.

Shesh, A., & Chen, B. (2004). SMARTPAPER: An interactive and user friendly sketching system. Eurographics, 23(3), 301-310.

Shneiderman, B., Hewett, T., Fischer, G., Jennings, P., Czerwinski, M., Kules, B., et al. (2006). Creativity support tools: Report from a U.S. National Science Foundation sponsored workshop. International Journal of Human-Computer Interaction, 20(2), 61-77.

Smith, G.F., & Browne, G.J. (1993). Conceptual foundations of design problem solving. IEEE Trans Systems, Man and Cybernetics, 23(5), 1209–1219.

Spector, P. E. (Ed.). (2006). Industrial and organizational psychology: research and practice: Hoboken, New Jersey: Wiley & Sons.

Stammerjohn, L. W., Smith, M. J., & Cohen, B. G. F. (1981). Evaluation of work station design factors in VDT operations. Human Factors, 23(4), 401-412.

Stempfle, J., & Badke-Schaub, P. (2002). Thinking in design teams - an analysis of team communication. Design Studies, 23(5), 473-496.

Taggart, J. (1975). Sketching, an informal dialogue between designer and computer. In N. Negroponte (Ed.). Reflections on computer aids to design and architecture (pp 147-162). New York: Petrocelli Charter.

Tang, G. L. (2005). Ink Kit: The user interface of the generic design transformational tool. Honours project report, University of Auckland, New Zealand.

Thomassen, A. J. W. M., Teulings, H.-L., & Schomaker, L. R. B. (1988). Real-time processing of cursive writing and sketched graphics. . In G. C. c. d. Veer & G. Mulder (Eds.), Human-Computer Interaction: Psychonomic Aspects (pp. 335-352). Berlin, Heidelberg: Springer-Gerlag.

Tractinsky, N., Katz, A. S., & Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13(2), 127-145.

Tovey, M. (1997). Styling and design: intuition and analysis in industrial design. Design Studies, 18(1), 5-31.

Turner, A., Chapman, D., & Penn, A. (2000). Sketching space. Computers & Graphics, 24(6), 869-879.

Tversky, B. (1999). What does drawing reveal about thinking? Visual and Spatial Reasoning in Design, MIT.

Ullman, D. G., Wood, S., & Craig, D. (1990). The importance of drawing in the mechanical design process. Computers & Graphics, 14(2), 263-274.

Van Dijk, C. G. C., & Mayer, A. A. C. (1997). Sketch input for conceptual surface design. Computers in Industry, 34(1), 125-137.

Van Someren, M W, Barnard, Y., & Sandberg, J. (1994). The think aloud method: A practical guide to modeling cognitive processes. Academic Press, London, UK.

Van der Veer, G. C. c. d., & Mulder, G. (Eds.). (1988). Human-Computer Interaction: Psychonomic Aspects. Berlin, Heidelberg: Springer-Gerlag.

Verstijnen, I. M., Van Leeuwen, C., Goldschmidt, G., Hamel, R., & Hennessey, J. M. (1998). Sketching and creative discovery. Design Studies, 19(4), 519-546.

Wagner, A. (1990). Prototyping: A day in the life of an interface designer. In B. Laurel (Eds.) The art of human-computer interface design (pp. 79-84). Addison-Wesley: Reading MA.

Wang, B., Sun, J., & Plimmer, B. (2005, July 6-8). Exploring sketch beautification techniques. Paper

presented at the Proceedings of the 6th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: making CHI natural CHINZ '05, Auckland, New Zealand.

Ward, A. (1989). Phenomenological analysis in the design process. Design Studies, 10(1), 53-66.

Welch, M. (1999). Analyzing the tacit strategies of novice designers. Research in Science & Technological Education, 17(1), 19-34.

Wilpert, B. (2006). Psychology and design processes. Safety Science, In Press, Corrected Proof.Wong, Y.Y. (1992). Rough and ready prototypes: Lessons from graphic design. Paper presented at the

Proceedings of Human Factors in Computing Systems CHI '92, Monterey, 83-84.

Yang, M. C. (2005). A study of prototypes, design activity and design outcome. Design studies, 26, 649-669.

Young, M. (2005). Ink Kit: The back end of the generic design transformation tool. Honours project report, University of Auckland, New Zealand.

Zeleznik, R. C., Herndon, K. P., & Hughes, J. F. (2006). SKETCH: an interface for sketching 3D scenes. Paper presented at the International Conference on Computer Graphics and Interactive Techniques, Boston, Massachusetts.

Zenka, R., & Slavík, P. (2003, April). Rendering: New dimension for sketches Paper presented at the Proceedings of the 19th spring conference on Computer graphics.

Appendices

Appendix A. The five designs and the outline of design errors present in each design

Appendix A1.1. Low formality design on paper – Online Magazine subscription form

Appendix A.1.2. Online Magazine: Planned design errors

Item Change from: Change to:1 Title Label: “Subsciption” “Subscription”2 Title Label: “Interntional” “International”3 Login Name Login Name item None4 Re-enter Password No item (encrypted) textbox + label:

“Re-enter password” item5 Contact No. 1 Textbox 2 Textboxes (1 smaller one for

the area code)6 Ethnicity Ethnicity Item No Item7 Preferred viewing

languageNone - Item with Dropdown menu +

label: “Preferred viewing language” OR- radio buttons

8 Payment 2 items - 3 items: Radio buttons + labels: “Cheque”/”Bank deposit”/”Credit card”

9 Payment Label on left & radio button on the right

Change sides: radio button on left, label on the right

10 Favorite singer Dropdown menu Textbox11 Favorite movie Dropdown menu Textbox12 Daily/weekly/yearly

magazine. Item setTextboxes in item set

Radio buttons

13 Daily/weekly/yearly magazine. Item set

Labels in item set: “dd”, “mm”, “yyyy”

Labels: “daily($price)”, “weekly($price)”, “yearly($price)”

14 Mailing address: Dropdown menu Textbox15 Would you like a

hardcopy of the magazine?

No item Item with a Question with 2 radio buttons + labels: “Yes”, “No”

16 Full Name 1 Textbox 2 Textboxes & Labels: “First Name”, “Last Name”

17 Age Textbox Dropdown menu18 Viewing interests Radio buttons in

item setCheckboxes in item set

19 Viewing interests: Weather

“Weather” item No item

20 Viewing interests No item “Other” item with a textbox + label: “Other”

21 Dating option “Yes, proceed” “Yes”22 Submit form None “Submit” button

Appendix A2.1. Low formality design on tablet PC – Samson’s Bank $1 million loan application form

Appendix A2.2. Samson’s loan: Planned design errors

Item Change from: Change to:1 Title Label: “Samsons” “Samson’s”2 Full name 1 textbox 2 textbox + labels (“First

names” and “Last Name”)3 Passport no. Label: “Passport no.” “(NZ) Passport no.”4 Home Number/

Mobile Number. 1 Textbox 2 Textboxes (1 smaller one

for the area code)5 Address: Town/City Textbox Dropdown menu6 Bankruptcy item set Label on left & radio

button on the rightChange sides: radio button on left, label on the right

7 Bankruptcy Label: “Agree”, “Not Agree”

“Yes”, “No”

8 Partner’s weekly income

None Add item: Label + textbox

9 Date of loan Textboxes in item set Dropdown menus in item set10 Weekly income range Textbox - Dropdown menu OR

- Radio buttons11 Yearly income Yearly income item No Item12 Personal Assets Dropdown menu - Text(box) Area OR

- Checkboxes (+ labels)13 Housing status item

setNone Radio buttons + labels

(“Rent” or Mortgage”)14 Purpose of loan Textbox Text Area15 IRD number Dropdown menu Textbox 16 IRD number 1 Textbox Controls for 9 digit input e.g.

3 small textboxes17 Any Questions Dropdown menu Text(box) Area 18 Any Questions Radio button on Left

hand side None

19 Any Questions Within the ‘check’ area

Separation from check items e.g. Align to left to separate from check items

20 Check items Radio buttons in item set

Checkboxes in item set

21 Submit application Question type item “Submit” button22 Submitt “submitt” “submit”

Appendix A3.1. Medium-low formality design on tablet PC – University of Strawberries graduation application form

Appendix A3.2. University of Strawberries: Planned design “errors”Item Change from: Change to:

1 Title Label: “Univerity” University2 Title Label: “Applicaiton” Application3 Full Name 1 Textbox 2 Textboxes & Labels: “First

Name”, “Last Name”4 2nd Department

nameNo item Dropdown (Label: “2nd

Department, if any”)5 2nd Degree name No item Dropdown: (Label “2nd Degree,

if any”)6 Department name Textbox Dropdown menu7 Degree name Textbox Dropdown menu8 Graduation

year/semesterTextbox Dropdown

9 Graduation Year/Semester

1 control Splitting “Year” and “Semester” – 2 controls (dropdowns)

10 Graduating in: “Person” “Absentia”11 Graduating in:

(item set)Label on left & radio button on the right

Change sides: radio button on left, label on the right

12 Contact No. Dropdown menu Textbox13 Contact No. 1 Textbox Split into 2 Textboxes (1 smaller

one for the area code)14 Ethnic group Ethnicity item No Item15 Last day of

courseTextboxes in item set Dropdown menus in item set

16 Give speech “Yes, I want…” & “No, I don’t want…”

No Item

17 Payment No Item - Dropdown menu + labels OR- Radio Buttons + labels

18 Mailing Address: House/Street

Dropdown Textbox

19 Want to borrow: Radio buttons in item set

Check boxes in item set

20 Hood color 1 Dropdown 2 dropdown 21 And the hood

color isRadio button on left hand side

None (just label and dropdown)

22 Continue to next page

Question type “Proceed to next page” button

Appendix A4.1. Medium-high formality design on tablet PC – Dog Registration

Form

Appendix A4.2. Dog Registration’s: Planned design “errors”

Item Change from: Change to:1 Title Label: “Registation” Registration2 Title Label: “Applicaiton” Application3 First owner’s name One textbox 2 textboxes for First name and

Last name4 2nd owner’s name One textbox 2 textboxes for First name and

Last name5 Address 2 One line of input 2 lines of input: Item

“Town/City” with a textbox6 Contact No. No item Add Contact Number item (with

2 textboxes)7 Sterilized “Yes”, “No” Add “Don’t know” item8 Sterilized Label on left & radio

button on the rightChange sides: radio button on left, label on the right

9 Gender Dropdown menu Radio buttons & labels: “ Female” and “Male”

10 Age Long dropdown Shorter dropdown11 Appearance Textboxes in item set Dropdown menus in item set12 Registration with vet Textbox - Radio Buttons & labels: “Yes”

and “No” OR- Checkbox

13 Registered with Dog lovers society

Existing item None

14 Dog’s special Conditions

Dropdown - Text Area OR- checkboxes

15 Purpose of the dog Textbox - Dropdown OR- Radio buttons

16 Breed 1 dropdown - 2 dropdown (for crossed breed) OR- Textbox(s)

17 Where did you get dog?

Label “..? Can choose more than 1 option”

None – can only choose one option

18 Brought from pet shop

“Brought from from pet shop”

“Brought from pet shop”

19 The shop is: Dropdown Textbox20 The shop is: Radio button on left

hand sideNone (just label and dropdown)

21 Proceed to next page No item “Proceed to next page” Button(s) 22 Dog’s information Scattered around Group dog’s information with

label(s) to separate from owner’s info

Appendix A5.1. High formality design on tablet PC – America’s Next Top Model application form

Appendix A5.2. America’s Next Top Model: Planned design errorsItem Change from: Change to:

1 Title “American’s” “America’s”2 Title “Applicattion” “Application”3 Full Name 1 Textbox 2 Textboxes & Labels: “First Name”,

“Last Name”4 Town/City Textbox Dropdown menu5 State Textbox Dropdown menu6 Contact No. 1 Textbox 2 Textboxes (1 smaller one for the

area code)7 Status “Mr”, “Mrs” “Married”, “Single” or “In a

relationship”8 Status Label on the left &

radio button on the right

Radio button on left, label on the right

9 Status Only 2 choices Add one more item to choose “Married”/ “Single”/ “In a relationship”

10 Gender Gender Item None11 Age Long dropdown menu Shorter menu for 2 digit number12 Date of birth Textboxes in item set Dropdown menus in item set13 Height Textbox Dropdown menu14 Weight Textbox Dropdown menu15 Why Enter Dropdown menu Text Area 16 Occupation Textbox Dropdown17 Experience Dropdown menu Text Area18 Experience “Experience” “Modeling experience”19 Heard from Radio buttons in item

setCheck boxes in item set

20 Other reasons Label: “other reasons” - “other sources” OR- “other”

21 Other reasons Dropdown menu Textbox22 Submit? Submit question Submit button

Appendix B. Post-task Questionnaire

Exploring the effects of different degrees of beautification on the design process

Post-task Questionnaire

Please answer the questions below by ticking the appropriate box or writing in your answer. Do not write your name anywhere on this form as your recorded information should remain anonymous. Take as much time as you need to answer those questions.

Part 1

1a. What is your age (years)? _________________ 1b. Gender: Female Male

2. What is your highest / current education level (please circle appropriately)?

Secondary

Other technical or professional training (please specify): _____________________

Tertiary (please specify the following if you have graduated):

Graduated at: _________________________ In year: ____________

Major/specialization:_______________________________________

3. What is your occupation?

a. Student: University of Auckland

Other Institution (please specify):__________________________

Study Programme: _________________________________________

What is your major/specialization? ____________________________

How many years have you been studying at university/other

institutions for? _______________

(Please continue to page 2)

Please indicate what paper(s) you have taken and/or are taking currently (please tick appropriately). Or write paper name(s) at the space provided below the table.

Tick Paper(s) (or Equivalent)SOFTENG 450 – Software Development Methodologies

COMPSCI 101 - Principles of Programming

COMPSCI 105 - Principles of Computer Science

COMPSCI 230 - Software Design and Construction

COMPSCI 280 - Applications Programming

COMPSCI 345 - Human Computer Interaction

OTHER papers: __________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

3. What is your occupation? (continued)

b. Other (please specify): ____________________________________________

4. Have you had any previous design experience? If yes, please describe briefly.

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

Please notify the experimenter that you have finished filling in this page.

(Please continue to page 3)

5. Please rank, in order, which design you enjoyed working on the most to the design you enjoyed working on the least. i.e. put “1” as the most-liked design, “2” as the second most-liked design…and so on, to “5” as the least-liked design. Please also state the reason if you can. The Experimenter will now show you the 5 designs that you have worked on.

Forms Rank Reasons

1st Form

2nd Form

3rd Form

4th Form

5th Form

6. In this experiment, did you prefer working on the Tablet PC or on paper? Why?

___________________________________________________________________________

___________________________________________________________________________

7. If you were designing something in real life, would you prefer to use paper / a computer?

___________________________________________________________________________

___________________________________________________________________________

8. Any comments on the experiment?

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

That’s all for now! Thank you very much for participating! Please give this to the researcher when you have finished.

APPROVED BY THE UNIVERSITY OF AUCKLAND HUMAN PARTICIPANTS ETHICS COMMITTEE on 13/04/2006 for a period of 3 years from 13/04/2006. Reference 2006 / 045.

Appendix C. Results of post-task questionnaireDescriptive statistics of participant’s characteristics (N=30)

Measure Items Frequency PercentGender: Male (Mean age = 22.81) 16 53.3

(SD = 5.87)Female (Mean age = 21.14) 15 46.7 (SD = 4.36)

Age (years): 19 3 10.020 4 13.321 11 36.722 6 20.023 4 13.325 1 3.344 1 3.3

Mean 22.03Std deviation 4.36

Occupation: Employed in SE/CS fields 5 16.7Student 25 83.3

Education (current and completed)

Graduate/Postgraduate 8 26.7

Undergraduate 22 73.3Years of study at tertiary 1.0 3 10.0

1.5 4 13.32.5 1 3.33.0 3 10.03.5 6 20.04.0 8 26.74.5 5 16.7

Mean 3.20Std deviation 1.19

Tertiary Institution of study Auckland University 30 100Other institutions 0 0

Study Major/Specialization: Computer science (CS)/software engineering (SE) majors 20 66.7

Non-CS/SE related majors 10 33.3Design Experience: CS/SE design experience 15 50.0

None to non-CS/SE design experience 15 50.0

Preference of design medium: In the experiment No preference 2 6.6

Pen and paper 13 43.3Tablet PC 15 50.0

In the real world No preference 1 3.3Paper and paper 3 10Tablet PC (Inkit) 3 10Computer (Other tools) 8 26.7Pen and paper, then Tablet PC 1 3.3Pen and paper, then Computer 14 46.7

Corrected vision Yes 30 100No 0 0

Appendix D. Participant information sheets and consent forms

Department of PsychologyThe University of AucklandTamaki CampusPrivate Bag 92019Auckland

Tel: 09 373 7599 ext 86870

PARTICIPANT INFORMATION SHEET: STUDENT

Title: Investigation of the form design process.

To participants:

My name is Louise Yeung and I am doing my thesis in partial fulfilment of the requirements for the Master of Science in the Department of Psychology at The University of Auckland. Together with my supervisors, Drs Brenda Lobb, Beryl Plimmer and Douglas Elliffe, I am investigating how people design the forms we have to fill in so often during our lives, for example, to order a book, apply for a driver’s licence or register a dog.

You are invited to participate in our research and we would appreciate any assistance you can offer us, although you are under no obligation to do so: your participation is voluntary and neither your grades nor academic relationships with the Departments of Psychology or Computer Science or members of staff will be affected whether or not you participate.

Participation involves one visit to our laboratory at The University of Auckland, for approximately one hour, and requires that your eyesight is normal or corrected-to-normal by spectacles or contact lenses. If you agree to participate, I will ask you to be seated at a desk and using a tablet (a specialized pen-driven computer), present you with five different form designs, one after the other. I will ask you to check each design using the scenario provided (e.g. register a dog, order a book, apply for credit) and make any changes you think necessary to improve the form. The changes you make and the time you spend working on each form’s design will be recorded automatically by the tablet. At the conclusion, you will be asked to note your age, education level and design experience on the tablet.

All the information you provide remains anonymous. I will give your information a code number; your name will not be recorded and no-one will be able to identify you from any of the recorded data. Your consent form will be held in a secure file for 6 years, at the end of this time it will be shredded. Your name will not be used in any reports arising from this study. Each participant will have a separate set of data, you will not see other participants’ data and they will not see yours. The computer is password protected to protect this data. The anonymous information collected during this study may be used in future analyses and publications and will be kept indefinitely. When it is no longer required all copies of the data will be destroyed. At the conclusion of the study, a summary of the findings will be available from the researchers upon request.

If you don’t want to participate, you don’t have to give any reason for your decision. If you do participate, you may withdraw at any time during the session. You can also ask for the information you have provided to be withdrawn at any time until 1/8/2006, without explanation and with penalty, by contacting me (details overleaf). If you choose not to participate, or to withdraw yourself or your information, your grades or academic relationships with the Departments of Psychology or Computer Science or members of staff will not be affected.

If you agree to participate in this study, please first complete the consent form attached to this information sheet. Your consent form will be kept separately from your data so that no-one will be able to identify you from the information you provide.

Thank you very much for your time and help in making this study possible. If you have any questions at any time you can phone me [cell phone no. to be inserted here], my supervisor Dr Brenda Lobb (09-373-7599 ext. 86870) or the Head of Department, Associate Professor Fred Seymour (3737599 ext 88414), or you can write to us at:

Department of Psychology, Tamaki CampusThe University of AucklandPrivate Bag 92019Auckland.

For any queries regarding ethical concerns, please contact The Chair, The University of Auckland Human Participants Ethics Committee, The University of Auckland, Research Office - Office of the Vice Chancellor, Private Bag 92019, Auckland. Tel. 3737599 ext 87830.

APPROVED BY THE UNIVERSITY OF AUCKLAND HUMAN PARTICIPANTS ETHICS COMMITTEE on for a period of 3 years from 2006. Reference 2006/045.

Department of PsychologyThe University of AucklandTamaki CampusPrivate Bag 92019Auckland

Tel: 09 373 7599 ext 86870

CONSENT FORM: STUDENT

THIS CONSENT FORM WILL BE HELD FOR A PERIOD OF AT LEAST SIX YEARS

Title: Investigation of the form design process

Researchers: Louise Yeung, Dr Brenda Lobb, Dr Beryl Plimmer, Dr Douglas Elliffe

I have been given and understood an explanation of this research project. I have had an opportunity to ask questions and have them answered. I understand that at the conclusion of the study, a summary of the findings will be available from the researchers upon request.

I understand that the anonymous data collected from the study will be held indefinitely and may be used in future analyses.

I understand that I may withdraw myself or any information traceable to me at any time up to 1stAugust, 2006 without giving a reason, without any penalty.

I understand that my grades and relationships within the Departments of Psychology and/or Computer Science will be unaffected whether or not I participate in this study or withdraw my participation during it.

I agree to take part in this research by completing the laboratory session.

I confirm that my eyesight is normal or corrected-to-normal.

Signed:

Name: (please print clearly)

Date:

APPROVED BY THE UNIVERSITY OF AUCKLAND HUMAN PARTICIPANTS ETHICS COMMITTEE on 13/04/06 for a period of 3 years from 2006. Reference 2006/045.

Department of PsychologyThe University of AucklandTamaki CampusPrivate Bag 92019Auckland

Tel: 09 373 7599 ext 86870

PARTICIPANT INFORMATION SHEET

Title: Investigation of the form design process

To participants:

My name is Louise Yeung and I am doing my thesis in partial fulfilment of the requirements for the Master of Science in the Department of Psychology at The University of Auckland. Together with my supervisors, Drs Brenda Lobb, Beryl Plimmer and Douglas Elliffe, I am investigating how people design the forms we have to fill in so often during our lives, for example, to order a book, apply for a driver’s licence or register a dog.

You are invited to participate in our research and we would appreciate any assistance you can offer us, although you are under no obligation to do so.

Participation involves one visit to our laboratory at The University of Auckland, for approximately one hour, and requires that your eyesight is normal or corrected-to-normal by spectacles or contact lenses. If you agree to participate, I will ask you to be seated at a desk and using a tablet (a specialized pen-driven computer), present you with five different forms, one after the other. I will ask you to check each design using the scenario provided (e.g. register a dog, order a book, apply for credit) and make any changes you think necessary to improve the form. The changes you make and the time you spend working on each form’s design will be recorded automatically by the tablet. At the conclusion, you will be asked to note your age, education level and design experience on the tablet.

All the information you provide remains anonymous. I will give your information a code number; your name will not be recorded and no-one will be able to identify you from any of the recorded data. Your consent form will be held in a secure file for 6 years, at the end of this time it will be shredded. Your name will not be used in any reports arising from this study. Each participant will have a separate set of data, you will not see other participants’ data and they will not see yours. The computer is password protected to protect this data. The anonymous information collected during this study may be used in future analyses and publications and will be kept indefinitely. When it is no longer required all copies of the data will be destroyed. At the conclusion of the study, a summary of the findings will be available from the researchers upon request.

If you don’t want to participate, you don’t have to give any reason for your decision. If you do participate, you may withdraw at any time during the session and you can also ask for the information you have provided to be withdrawn at any time until 1/8/2006, without explanation and without penalty, by contacting me (details overleaf).

If you agree to participate in this study, please first complete the consent form attached to this information sheet. Your consent form will be kept separately from your data so that no-one will be able to identify you from the information you provide.

Thank you very much for your time and help in making this study possible. If you have any questions at any time you can phone me (021- [cell phone number to be inserted here ], my supervisor, Dr Brenda Lobb (09-373-7599 ext. 86870) or the Head of Department, Associate Professor Fred Seymour (3737599 ext 88414), or you can write to us at:

Department of Psychology, Tamaki CampusThe University of AucklandPrivate Bag 92019Auckland.

For any queries regarding ethical concerns, please contact The Chair, The University of Auckland Human Participants Ethics Committee, The University of Auckland, Research Office - Office of the Vice Chancellor, Private Bag 92019, Auckland. Tel. 3737599 ext 87830.

APPROVED BY THE UNIVERSITY OF AUCKLAND HUMAN PARTICIPANTS ETHICS COMMITTEE on for a period of 3 years from 2006. Reference 2006/045.

Department of PsychologyThe University of AucklandTamaki CampusPrivate Bag 92019Auckland

Tel: 09 373 7599 ext 86870

CONSENT FORM

THIS CONSENT FORM WILL BE HELD FOR A PERIOD OF AT LEAST SIX YEARS

Title: Investigation of the form design process

Researchers: Louise Yeung, Dr Brenda Lobb, Dr Beryl Plimmer, Dr Douglas Elliffe

I have been given and understood an explanation of this research project. I have had an opportunity to ask questions and have them answered. I understand that at the conclusion of the study, a summary of the findings will be available from the researchers upon request.

I understand that the anonymous data collected from the study will be held indefinitely and may be used in future analyses.

I understand that I may withdraw myself or any information traceable to me at any time up to 1stAugust, 2006 without giving a reason, without any penalty.

I agree to take part in this research by completing the laboratory session.

I confirm that my eyesight is normal or corrected-to-normal.

Signed:

Name: (please print clearly)

Date:

APPROVED BY THE UNIVERSITY OF AUCKLAND HUMAN PARTICIPANTS ETHICS COMMITTEE on 13/04/06 for a period of 3 years from 2006. Reference 2006/045.

Appendix E. Functional Aspects of Inkit

Table 1. Steps taken to demonstrate each type of change to the participant. Change DemonstrationAdd In “Drawing Mode” – a text box, a dropdown menu, a radio

button, and a label (“Name”) were drawn using pen-input on the Tablet PC screen.

Delete In “Eraser Mode” – a text box, a dropdown menu, a radio button, and a label (“Name”) were deleted by ‘brushing’ through the unwanted object.

Resize In “Selection Mode” – an element was selected by ‘drawing’ (pointing the cursor) around the element desired on the screen in a continuous motion, then let go. The rectangle box around the element indicated that it was selected. The element was then resized by dragging a (any) corner.

Relocate While still in “Selection Mode” – a text box was selected. It was dragged (by pointing to the object on the screen and moving the pen along the screen to the desired location) and dropped (by taking the pen off the screen) to the desired place (right hand corner)

Change by annotation

In “Drawing Mode” – an arrow was drawn to indicate where an element should be placed instead of the existing location. Another arrow was drawn into the space between two elements, to indicate the addition of an element to the specified space (arrow head).

Table 2.Description of the functionality of different modes, and the visual feedback of the icon selected and highlighted (in orange). Modes Icon highlighted

(visual feedback)Functionality

Inking/drawing

Ink icon Enables the user to use the pen to ‘draw’ or ‘ink’ onto the opened page (screen) as if they are drawing on paper.

Eraser Eraser icon Enables the user to erase anything that they have sketched on the page

Selection Selection icon Enables user to select elements they have drawn on the page. When an element is selected, the user can then resize or relocate the element.

Appendix F. Instruction sheets containing the requirements and scenario associated with each design

Appendix F1. Instructions including the requirements and the scenario for the low formality (on paper) design.

Instructions:

Use the requirements stated in Part A and scenarios provided in Part B to check the design you have been given. Change the design to provide a better interface to serve its purpose.

Part A: Requirements for the form

Design an interface for subscribing the International Online Magazine. The subscriber must know that it is the International Online Magazine subscription page. The online form should be easy to fill in. The following information should be obtained from the subscriber:

Login name – the user would use this login name to get to the online magazine they are subscribed to

Password – must ensure the user correctly inputs the password that they want as the password would be encrypted when as they type it in. ‘

Email address Contact number Viewing language: English, Chinese Simplified, Chinese traditional, Thai, French,

German, Spanish, Italian, Japanese Payment – credit card/cheque/bank deposit Favourite singer Favourite movie Do they want the magazine daily, monthly, or yearly (different price) Subscriber’s full name Age – for statistical purposes and to restrict viewing options if they are under 18yrs

of age. They are to indicate whether they would like a hard copy of the magazine they are

subscribed to. Mailing address – if they want a hard copy What type of things the subscriber are interested to view in their magazine

(customizing purposes): adult; movies; music; news; sports; other. Subscribers are to choose whether they want the “dating” option.

When they have finished filling in the form, they should be able to submit the form to subscribe for the magazine immediately.

Part B: Scenario

Use the scenario below to CHECK that you have a ‘control’ (e.g. textboxes, radio buttons, dropdown menus) for each item, and that each ‘control’ is of the appropriate type and size.

Login name – “pinkyWinky666” Password – “privateMary asd kjbJerry” Email address – privateMaryJerry@crazymail.com Contact number – +64 219237846 Viewing language: Chinese Traditional Payment – “Bank Deposit” Favourite singer – “Spice Girls” Favourite movie – “Titanic” Subscriber’s full name – “Mary Jerry” Age – “18” “Yes, I want a hard copy of the magazine”. Mailing address – “833 Crazy Avenue, Mamalada Land, New Brealand” Interested in: “adult”; “movies”; “other: children” Dating option: “Yes”

Appendix F2. Instructions including the requirements and the scenario for the low formality (on Tablet PC) design.

Instructions:

Use the requirements stated in Part A and scenarios provided in Part B to check the design you have been given. Change the design to provide a better interface to serve its purpose.

Part A: Requirements for this form

Design an interface for the $1 million Loan application (Samson’s Bank – NZ branch). The applicant must know that it is the Samson’s Bank (NZ) page. The online form should be easy to fill in. The information listed below should be obtained from the applicants accurately:

Full Name Passport No. – New Zealand Home Number Mobile Number Address – the town or city must be within the designated areas i.e. Auckland,

Hamilton, Wellington, Rototua, Dunedin and Christchurch Bankruptcy – to check if they were / are bankrupt. Occupation Status – married/single/in a relationship – if married, then what’s the partner’s

weekly income The date they want the loan Applicant’s Weekly income Other personal assets of value – e.g. house, car, stock etc Is the applicant renting a house or on Mortgage Purpose of the loan – whether applicant is suitable IRD number – security reasons Applicants must check whether they have all the items ready: i.e. person

verification, income verification (from inland revenue), past 10 years bank record, 10 official character reference

Applicants may include some questions in the applicationWhen the applicant has finished filling in the form, they should be able to submit the form.

Part B: The Scenario

Use the scenario below to CHECK that you have a ‘control’ (e.g. textboxes, radio buttons, dropdown menus) for each item, and that each ‘control’ is of the appropriate type and size.

Full Name – “Mary Carey” Passport No. – “FZ 1055872” Home Number – “09 3487533” Mobile Number – “027 13049773” Address – “10 Symonds street, Auckland CBD. Occupation – “Personal Assistant” Status – married Partner’s weekly income – “$1000” The date they want the loan – 07/011/2006 Applicant’s Weekly income – “$300” Other personal assets of value – “car” House – “Renting house” Purpose of the loan – “I want to be able to spend all my money because I have

an uncurable disease and I want to live my life happily in the time left. Then my partner will pay them off for me”

IRD number – 123-321-123 Yes, have all the items ready: person verification, income verification (from

inland revenue), past 10 years bank record, 10 official character reference Question: What is the interest rate? Because if it’s too high then I shall find

Samuel for their $2million loan”Submit application – “Next page”

Appendix F3. Instructions including the requirements and the scenario for the medium-low formality design.

Instructions:

Use the requirements stated in Part A and scenarios provided in Part B to check the design you have been given. Change the design to provide a better interface to serve its purpose.

Part A: Requirements for this form

Design an interface for the University of Strawberries (NZ) Graduation application form. The applicant must know that it is the University of Strawberries (NZ) application page. The online form should be easy to fill in. The information listed below should be obtained from the students accurately:

Full Name Preferred Name – to be called at the graduation ceremony Student ID – 10 digits ID Programme Name – e.g. Bachelor of Science, Master of Commerce etc. Department Name – e.g. Computer Science, Health Science, Psychology etc. Preferred graduation year and semester Whether he/she want to graduate in person or in absentee If absent, where they want the certificate to be sent to – address If the address is not in New Zealand, then postage fee must be paid – by

eftpos/credit card/cheque. Contact Number The last day of their course The things they want to borrow for the graduation ceremony: trencher, gown,

black suit, black dress, hood (and what colour hood)?The applicant should be able to proceed to the next page when he/she has finished filling in this page.

Part B: The Scenario

Use the scenario below to CHECK that you have a ‘control’ (e.g. textboxes, radio buttons, dropdown menus) for each item, and that each ‘control’ is of the appropriate type and size.

Full Name – “Cheng Lim Wang” Preferred Name – “Cheng Wang” Student ID – “9994443331” Programme Name – “Bachelor of Science, Bachelor of Arts” Department Name – “Psychology and philosophy.” Preferred graduation year and semester – “year 2007, semester 1” Whether he/she want to graduate in person or in absentee – “In absentee” If absent, where they want the certificate to be sent to – “19/F, XYZ Building,

XYZ road, Kowloon, Hong Kong” Postage payment – “Credit card” Contact Number – “021 1873009” The last day of their course – “June,10, 2006” Want to borrow – trencher, gown, hood (hood colours: pink and blue) Continue - “To Next page”

Appendix F4. Instructions including the requirements and the scenario for the medium-high formality design.

Instructions:

Use the requirements stated in Part A and scenarios provided in Part B to check the design you have been given. Change the design to provide a better interface to serve its purpose.

Part A: Requirements for this form

Design an interface for the New Zealand Dog Registration Online Application form. The applicant must know that it is the New Zealand Dog Registration Page. The online form should be easy to fill in. The information listed below must be obtained:

Dog’s Name Owner’s Name Address Contact Number Second Owner – if there is one Second Owner’s Address Is the Dog sterilized – for statistical and population control purposes Gender Age – dog’s age Appearance of dog – height, weight, colour Is the dog registered with a vet – for health and safety Dog’s special condition(s) – e.g. blind, disabled, violent history etc. Purpose of dog – e.g. guide dogs, police dogs, pet etc. Breed Where did they get the dog? : On the street, from friend(s) and/or family,

SPCA, own dog’s offspring, from a pet shop. – And if it’s from a shop, which shop.

An agreement to love his/her dog.The applicant should be able to proceed to the next page when he/she has finished filling in this page.

Part B: The Scenario

Use the scenario below to CHECK that you have a ‘control’ (e.g. textboxes, radio buttons, dropdown menus) for each item, and that each ‘control’ is of the appropriate type and size.

Dog’s Name – “Dinky” Owner’s Name – “Peter-Andrea Smith” Address – “8 Sunny Bay, Whangarei.” Contact Number – “027 1234567” Second Owner – “Shelly Brown” Second Owner’s Address – “14 Sunny Bay, Whangarei” Is the Dog sterilized – Don’t know Age – 4 years Appearance of dog – height = “55cm”, weight = “30kg”, colour = golden

white Is the dog registered with a vet –“No” Dog’s special condition(s) – None Purpose of dog – pet Breed – Pure Labrador Where did they get the dog? : “From SPCA” An agreement to love his/her dog. – “No”

Appendix F5. Instructions including the requirements and the scenario for the high formality design.

Instructions:

Use the both the requirements stated in Part A and the scenario provided in Part B to check the design you have been given. Change the design to provide a better interface to serve its purpose.

Part A: Requirements for this form

Design an interface for the 2007 America’s Next Top Model secured online application form. The applicant must know that it is the America’s Next Top Model application form. The online form should be easy to fill in, with minimal room for input errors from applicants. The information listed below must be obtained from the applicants:

Full Name Home Address – it must be an American Address Contact Number Email address Status – married, in a relationship, single Date of Birth – to check whether they are between 18 – 28 yrs old. Age – second check for age Height (cm) Weight (kg) Occupation – for fast grouping purposes Modelling experience – to identify those who has more experience Why do they want to enter – to screen for individuals who stands out. Where they hear about this America’s next top model form: Radio, TV,

newspaper, modelling agencies, online and/or other – for statistical and future promotional purposes.

When the applicant has finished filling in the form, they should be able to submit the form.

Part B: The Scenario

Use the scenario below to CHECK that you have a ‘control’ (e.g. textboxes, radio buttons, dropdown menus) for each item, and that each ‘control’ is of the appropriate type and size.

Full Name – “Mary-Jane Louisa Love-Hewitt” Home Address – “156, Mysterious Lane, New York, Washington D.C.” Contact Number – “09982 12345667” Email address – “maryJane009@xmail.com” Status – “in a relationship” Date of Birth – “06/06/1985” Age – “21” Height (cm) – “180” Weight (kg) – “47” Occupation – “student” Modelling experience – “3 years part-time modelling at Elle Magazine, 2

years part-time cat walk experience, model for a new underwear brand.” Why do they want to enter – “I want to become America’s next top model.

Modelling is my dream career, I want it so bad, I’ve been dreaming about it ever since I was like 3 years old. I am very into fashion, and I want to proof to the world that blondes are just as smart and beautiful as brunettes”

Hear about it from – TV, radio, modelling agencies, online, magazines. Submit form

Appendix G. Screen shots during font creation using My Font Tool for Tablet PC (2004).

Figure G1. Font creation using My Font Tool for the Tablet PC (2004): hand-writing input at the designated spaces for each character in the alphabet, and numbers from 0-9, as well as some common symbols such as full-stops, commas, exclamation marks etc.

Figure G2. From the original handwriting illustrated in Figure G1, the data is

compiled and the new font called “LY_Handwriting” is then created, as shown, in different sizes.

Figure G3. An illustration of the Gulim typeface in different sizes in My Font Tool for the Tablet PC

Figure G4. An illustration of the Times New Roman typeface in different sizes in My Font Tool for the Tablet PC.

Appendix H. Testing the normality assumption – Total number of changes made across levels of formality

Statistic Std. ErrorLow formality (on paper) Mean 18.733 1.1996 95% Confidence

Interval for MeanLower Bound 16.280

Upper Bound21.187

5% Trimmed Mean 18.815 Median 19.500 Variance 43.168 Std. Deviation 6.5702 Minimum 5.0 Maximum 30.0 Range 25.0 Interquartile Range 9.5 Skewness -.108 .427 Kurtosis -.680 .833Low formality (on Tablet PC) Mean 15.167 .7567 95% Confidence

Interval for MeanLower Bound 13.619

Upper Bound16.714

5% Trimmed Mean 15.074 Median 15.000 Variance 17.178 Std. Deviation 4.1447 Minimum 7.0 Maximum 26.0 Range 19.0 Interquartile Range 6.0 Skewness .381 .427 Kurtosis .525 .833Medium-low formality Mean 14.000 .7428 95% Confidence

Interval for MeanLower Bound 12.481

Upper Bound15.519

5% Trimmed Mean 13.944 Median 13.500 Variance 16.552 Std. Deviation 4.0684 Minimum 7.0 Maximum 23.0 Range 16.0 Interquartile Range 6.3 Skewness .063 .427 Kurtosis -.560 .833Medium-high formality Mean 13.133 .7042 95% Confidence

Interval for MeanLower Bound 11.693

Upper Bound14.574

5% Trimmed Mean 13.111 Median 13.500 Variance 14.878 Std. Deviation 3.8572 Minimum 7.0 Maximum 20.0 Range 13.0 Interquartile Range 6.3 Skewness -.051 .427 Kurtosis -1.165 .833High formality Mean 11.267 .6414 95% Confidence

Interval for MeanLower Bound 9.955

Upper Bound12.578

5% Trimmed Mean 11.259 Median 11.500 Variance 12.340 Std. Deviation 3.5129 Minimum 6.0 Maximum 17.0 Range 11.0 Interquartile Range 6.3 Skewness -.054 .427 Kurtosis -1.369 .833

Tests of Normality

Kolmogorov-Smirnov(a) Shapiro-Wilk Statistic df Sig. Statistic df Sig.Low formality (on paper) .083 30 .200(*) .976 30 .703Low formality (on Tablet PC) .116 30 .200(*) .982 30 .873Medium-low formality .103 30 .200(*) .976 30 .725Medium-high formality .152 30 .073 .946 30 .133High formality .124 30 .200(*) .931 30 .052

* This is a lower bound of the true significance.a Lilliefors Significance Correction

Low formality (on paper)

30.025.020.015.010.05.0

Low formality (on paper)

6

4

2

0

Freq

uenc

y

Mean =18.733Std. Dev. =6.5702

N =30

Histogram

Low formality (on paper) Stem-and-Leaf Plot

Frequency Stem & Leaf

1.00 0 . 6 8.00 1 . 00011344 6.00 1 . 678999 9.00 2 . 011122244 6.00 2 . 567899

Stem width: 10.0 Each leaf: 1 case(s)

3530252015105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Low formality (on paper)

Low formality (on Tablet PC)

25.020.015.010.0

Low formality (on Tablet PC)

10

8

6

4

2

0

Freq

uenc

y

Mean =15.167Std. Dev. =4.1447

N =30

Histogram

Low formality Stem-and-Leaf Plot

Frequency Stem & Leaf

2.00 0 . 78 12.00 1 . 112222334444 13.00 1 . 5557777889999 2.00 2 . 12 1.00 2 . 6

Stem width: 10.0 Each leaf: 1 case(s)

252015105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Low formality (on Tablet PC)

Medium-low formality

20.015.010.0

Medium-low formality

6

4

2

0

Freq

uenc

y

Mean =14.0Std. Dev. =4.0684

N =30

Histogram

Medium-low formality Stem-and-Leaf Plot

Frequency Stem & Leaf

4.00 0 . 7888 13.00 1 . 0000122344444 10.00 1 . 5566777899 3.00 2 . 013

Stem width: 10.0 Each leaf: 1 case(s)

242118151296

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Medium-low formality

Medium-high formality

18.015.012.09.06.0

Medium-high formality

6

4

2

0

Freq

uenc

y

Mean =13.133Std. Dev. =3.8572

N =30

Histogram

Medium-high formality Stem-and-Leaf Plot

Frequency Stem & Leaf

7.00 0 . 7788899 10.00 1 . 0111123334 12.00 1 . 556666666788 1.00 2 . 0

Stem width: 10.0 Each leaf: 1 case(s)

20.017.515.012.510.07.5

Observed Value

2

0

-2Ex

pect

ed N

orm

al

Normal Q-Q Plot of Medium-high formality

High formality

17.515.012.510.07.5

High formality

6

4

2

0

Freq

uenc

y

Mean =11.267Std. Dev. =3.5129

N =30

Histogram

High formality Stem-and-Leaf Plot

Frequency Stem & Leaf

.00 0 . 6.00 0 . 666777 6.00 0 . 888999 3.00 1 . 011 4.00 1 . 2233 8.00 1 . 44455555 2.00 1 . 67 1.00 1 . 8

Stem width: 10.0 Each leaf: 1 case(s)

17.515.012.510.07.55.0

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of High formality

Appendix I. Testing the normality assumption – Number of quality changes made across levels of formality

Statistic Std. ErrorLow formality (on paper) Mean 15.733 1.0045 95% Confidence

Interval for MeanLower Bound 13.679

Upper Bound17.788

5% Trimmed Mean 15.824 Median 16.500 Variance 30.271 Std. Deviation 5.5019 Minimum 4.0 Maximum 25.0 Range 21.0 Interquartile Range 9.4 Skewness -.156 .427 Kurtosis -.716 .833Low formality (on Tablet PC) Mean 13.050 .7324 95% Confidence

Interval for MeanLower Bound 11.552

Upper Bound14.548

5% Trimmed Mean 12.963 Median 13.500 Variance 16.092 Std. Deviation 4.0115 Minimum 6.0 Maximum 22.0 Range 16.0 Interquartile Range 4.3 Skewness .176 .427 Kurtosis -.200 .833Medium-low formality Mean 12.900 .6958 95% Confidence

Interval for MeanLower Bound 11.477

Upper Bound14.323

5% Trimmed Mean 12.741 Median 13.000 Variance 14.524 Std. Deviation 3.8111 Minimum 7.0 Maximum 23.0 Range 16.0 Interquartile Range 5.5 Skewness .544 .427 Kurtosis .223 .833Medium-high formality Mean 10.800 .7162 95% Confidence

Interval for MeanLower Bound 9.335

Upper Bound12.265

5% Trimmed Mean 10.722

Median 10.750 Variance 15.390 Std. Deviation 3.9230 Minimum 4.0 Maximum 19.0 Range 15.0 Interquartile Range 7.3 Skewness .153 .427 Kurtosis -.721 .833High formality Mean 9.017 .6462 95% Confidence

Interval for MeanLower Bound 7.695

Upper Bound10.338

5% Trimmed Mean 9.019 Median 9.250 Variance 12.526 Std. Deviation 3.5391 Minimum 3.5 Maximum 14.5 Range 11.0 Interquartile Range 6.0 Skewness -.042 .427 Kurtosis -1.290 .833

Tests of Normality

Kolmogorov-Smirnov(a) Shapiro-Wilk Statistic df Sig. Statistic df Sig.Low formality (on paper) .119 30 .200(*) .969 30 .522Low formality (on Tablet PC) .113 30 .200(*) .963 30 .377Medium-low formality .110 30 .200(*) .963 30 .378Medium-high formality .100 30 .200(*) .966 30 .447High formality .116 30 .200(*) .940 30 .092

* This is a lower bound of the true significance.a Lilliefors Significance Correction

Low formality (on paper)

25.020.015.010.05.0

Low formality (on paper)

8

6

4

2

0

Freq

uenc

y

Mean =15.733Std. Dev. =5.5019

N =30

Histogram

Low formality on paper Stem-and-Leaf Plot

Frequency Stem & Leaf

.00 0 . 4.00 0 . 5799 9.00 1 . 000122234 10.00 1 . 6677777799 7.00 2 . 0012223

Stem width: 10.0 Each leaf: 1 case(s)

252015105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Low formality (on paper)

Low formality (on Tablet PC)

20.015.010.05.0

Low formality (on Tablet PC)

6

4

2

0

Freq

uenc

y

Mean =13.05Std. Dev. =4.0115

N =30

Histogram

Low formality Stem-and-Leaf Plot

Frequency Stem & Leaf

.00 0 . 2.00 0 . 67 4.00 0 . 8888 5.00 1 . 01111 5.00 1 . 23333 9.00 1 . 444444555 1.00 1 . 7 3.00 1 . 999 1.00 Extremes (>=22)

Stem width: 10.0 Each leaf: 1 case(s)

242118151296

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Low formality (on Tablet PC)

Medium-low formality

20.015.010.0

Medium-low formality

6

4

2

0

Freq

uenc

y

Mean =12.9Std. Dev. =3.8111

N =30

Histogram

Medium low formality Stem-and-Leaf Plot

Frequency Stem & Leaf

8.00 0 . 78889999 12.00 1 . 011223333344 9.00 1 . 555567889 1.00 2 . 3

Stem width: 10.0 Each leaf: 1 case(s)

\242118151296

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Medium-low formality

Medium high formality

18.016.014.012.010.08.06.04.0

Medium-high formality

10

8

6

4

2

0

Freq

uenc

y

Mean =9.017Std. Dev. =3.4503

N =30

Histogram

Medium-high formality Stem-and-Leaf Plot

Frequency Stem & Leaf

.00 0 . 2.00 0 . 45 6.00 0 . 666677 3.00 0 . 889 6.00 1 . 000111 6.00 1 . 222333 4.00 1 . 4444 2.00 1 . 66 1.00 1 . 9

Stem width: 10.0 Each leaf: 1 case(s)

15105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Medium-high formality

High formality

14.012.010.08.06.04.0

High formality

5

4

3

2

1

0

Freq

uenc

y

Mean =9.017Std. Dev. =3.5391

N =30

Histogram

High formality Stem-and-Leaf Plot

Frequency Stem & Leaf

3.00 0 . 333 4.00 0 . 4445 6.00 0 . 667777 4.00 0 . 8999 5.00 1 . 00111 7.00 1 . 2222233 1.00 1 . 4

Stem width: 10.0 Each leaf: 1 case(s)

1412108642

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of High formality

Appendix J: Testing the normality assumption – Number of expected changes made across levels of formality

Statistic Std. ErrorLow formality (on paper) Mean 13.550 .7736 95% Confidence

Interval for MeanLower Bound 11.968

Upper Bound15.132

5% Trimmed Mean 13.657 Median 14.750 Variance 17.954 Std. Deviation 4.2373 Minimum 4.0 Maximum 20.5 Range 16.5 Interquartile Range 7.1 Skewness -.400 .427 Kurtosis -.554 .833Low formality (on Tablet PC) Mean 11.183 .5938 95% Confidence

Interval for MeanLower Bound 9.969

Upper Bound12.398

5% Trimmed Mean 11.231 Median 12.000 Variance 10.577 Std. Deviation 3.2523 Minimum 4.5 Maximum 17.0 Range 12.5 Interquartile Range 5.1 Skewness -.225 .427 Kurtosis -.644 .833Medium-low formality Mean 10.217 .6092 95% Confidence

Interval for MeanLower Bound 8.971

Upper Bound11.463

5% Trimmed Mean 10.130 Median 9.500 Variance 11.132 Std. Deviation 3.3365 Minimum 4.5 Maximum 18.0 Range 13.5 Interquartile Range 5.6 Skewness .270 .427 Kurtosis -.707 .833Medium-high formality Mean 9.017 .6299 95% Confidence

Interval for MeanLower Bound 7.728

Upper Bound10.305

5% Trimmed Mean 8.954 Median 9.750 Variance 11.905 Std. Deviation 3.4503 Minimum 3.0 Maximum 17.0 Range 14.0 Interquartile Range 5.4 Skewness .161 .427 Kurtosis -.469 .833High formality Mean 8.000 .6027 95% Confidence

Interval for MeanLower Bound 6.767

Upper Bound9.233

5% Trimmed Mean 8.009 Median 9.000 Variance 10.897 Std. Deviation 3.3010 Minimum 2.0 Maximum 14.5 Range 12.5 Interquartile Range 5.5 Skewness -.243 .427 Kurtosis -.782 .833

Tests of Normality

Kolmogorov-Smirnov(a) Shapiro-Wilk Statistic df Sig. Statistic df Sig.Low formality (on paper) .134 30 .179 .969 30 .513Low formality (on Tablet PC) .132 30 .190 .965 30 .411Medium-low formality .130 30 .200(*) .956 30 .242Medium-high formality .112 30 .200(*) .977 30 .739High formality .152 30 .073 .957 30 .263* This is a lower bound of the true significance.a Lilliefors Significance Correction

Low formality (on paper)

20.017.515.012.510.07.55.0

Low formality (on paper)

10

8

6

4

2

0

Freq

uenc

y

Mean =13.55Std. Dev. =4.2373

N =30

Histogram

Low formality (on paper) Stem-and-Leaf Plot

Frequency Stem & Leaf

1.00 0 . 4 6.00 0 . 678999 9.00 1 . 011233444 13.00 1 . 5555666778899 1.00 2 . 0

Stem width: 10.0 Each leaf: 1 case(s)

2015105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Low formality (on paper)

Low formality (on Tablet PC)

17.515.012.510.07.55.0

Low formality (on Tablet PC)

8

6

4

2

0

Freq

uenc

y

Mean =11.183Std. Dev. =3.2523

N =30

Histogram

Low formality (on Tablet PC) Stem-and-Leaf Plot

Frequency Stem & Leaf

2.00 0 . 45 4.00 0 . 7777 3.00 0 . 999 6.00 1 . 000111 7.00 1 . 2222222 6.00 1 . 445555 2.00 1 . 67

Stem width: 10.0 Each leaf: 1 case(s)

17.515.012.510.07.55.0

Observed Value

2

0

-2Ex

pect

ed N

orm

al

Normal Q-Q Plot of Low formality (on Tablet PC)

Medium-low formality

17.515.012.510.07.55.0

Medium-low formality

8

6

4

2

0

Freq

uenc

y

Mean =10.217Std. Dev. =3.3365

N =30

Histogram

Medium-low formality Stem-and-Leaf Plot

Frequency Stem & Leaf

1.00 0 . 4 5.00 0 . 66667 8.00 0 . 88888999 4.00 1 . 0111 8.00 1 . 22333333 2.00 1 . 45 2.00 1 . 67

Stem width: 10.0 Each leaf: 1 case(s)

17.515.012.510.07.55.0

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Medium-low formality

Medium-high formality

18.016.014.012.010.08.06.04.0

Medium-high formality

10

8

6

4

2

0

Freq

uenc

y

Mean =9.017Std. Dev. =3.4503

N =30

Histogram

Medium-high formality Stem-and-Leaf Plot

Frequency Stem & Leaf

2.00 0 . 33 5.00 0 . 44555 3.00 0 . 667 4.00 0 . 8888 9.00 1 . 000000111 5.00 1 . 22333 1.00 1 . 4 1.00 1 . 7

Stem width: 10.0 Each leaf: 1 case(s)

15105

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of Medium-high formality

High formality

15.012.510.07.55.02.5

High formality

8

6

4

2

0

Freq

uenc

y

Mean =8.0Std. Dev. =3.301

N =30

Histogram

High formality Stem-and-Leaf Plot

Frequency Stem & Leaf

.00 0 . 3.00 0 . 222 5.00 0 . 44555 5.00 0 . 66677 6.00 0 . 899999 8.00 1 . 00000011 2.00 1 . 22 1.00 1 . 4

Stem width: 10.0 Each leaf: 1 case(s)

15.012.510.07.55.02.5

Observed Value

2

0

-2

Expe

cted

Nor

mal

Normal Q-Q Plot of High formality

Appendix K. Mean total number of changes made across each level of formality – according to a combination of between-subjects factors (design

experience, major/specialization and study level)

Design experienceMajor/

specialization Study levelFormality

level Mean Std. ErrorNone to non-CS/SE design experience

Non-CS/SE related majors

Undergraduate 1 13.17 2.28

2 12.33 1.52 3 14.17 1.77 4 11.17 1.37 5 10.50 1.34 Graduate/

postgraduate1 17.00 3.96

2 13.50 2.63 3 12.50 3.07 4 12.00 2.37 5 11.00 2.32 CS/SE major Undergraduate 1 16.00 2.11 2 13.86 1.41 3 12.71 1.65 4 11.57 1.27 5 10.29 1.24 Graduate/

postgraduate 1 .(a) .

2 .(a) . 3 .(a) . 4 .(a) . 5 .(a) .CS/SE design experience

non CS/SE related majors

Undergraduate 1 20.00 5.59

2 19.00 3.72 3 16.00 4.34 4 14.00 3.36 5 6.00 3.28 Graduate/

postgraduate1 19.00 5.59

2 15.00 3.72 3 17.00 4.34 4 16.00 3.36 5 16.00 3.28 CS/SE Undergraduate 1 21.25 1.98 2 15.63 1.31 3 13.63 1.54 4 12.75 1.19 5 10.88 1.16 Graduate/

postgraduate1 25.60 2.50

2 19.60 1.66 3 15.80 1.94 4 18.00 1.50 5 14.40 1.47

a. This level combination of factors is not observed, thus the corresponding population marginal mean is not estimable.

Note: Formality Level: 1 = Low formality (on paper); 2 = Low formality (on Tablet PC); 3 = Medium-low formality; 4 = Medium-high formality; 5 = High formality.

Appendix L. Mean number of quality changes made across each level of formality – according to a combination of between-subjects factors (design

experience, major/specialization and study level)

Design experienceMajor/

specialization Study levelFormality

level MeanStd.

ErrorNone to non-CS/SE design experience

Non-CS/SE related majors

Undergraduate 1 11.42 2.01

2 10.67 1.52 3 12.92 1.66 4 8.92 1.28 5 7.50 1.35 Graduate/

postgraduate1 13.50 3.47

2 10.50 2.64 3 12.25 2.88 4 10.50 2.22 5 9.25 2.34 CS/SE major Undergraduate 1 13.71 1.86 2 11.86 1.41 3 12.00 1.54 4 9.43 1.18 5 8.07 1.25 Graduate/

postgraduate 1 .(a) .

2 .(a) . 3 .(a) . 4 .(a) . 5 .(a) .CS/SE design experience

non CS/SE related majors

Undergraduate 1 17.50 4.92

2 16.00 3.73 3 14.00 4.08 4 13.00 3.13 5 5.00 3.31 Graduate/

postgraduate1 18.00 4.92

2 13.00 3.73 3 17.00 4.08 4 15.00 3.13 5 14.50 3.31 CS/SE Undergraduate 1 17.50 1.74 2 13.69 1.32 3 12.25 1.44 4 9.31 1.12 5 8.88 1.17 Graduate/

postgraduate1 21.00 2.20

2 17.00 1.67 3 14.40 1.82 4 16.20 1.40 5 12.00 1.48a. This level combination of factors is not observed, thus the corresponding population marginal mean is

not estimable.

Note: Formality Level: 1 = Low formality (on paper); 2 = Low formality (on Tablet PC); 3 = Medium-low formality; 4 = Medium-high formality; 5 = High formality.

Appendix M. Mean number of expected changes across each level of formality – according to between-subjects factors (design experience, major/specialization and

study level)

Design experienceMajor/

specialization Study levelFormality

level Mean Std. ErrorNone to non-CS/SE design experience

Non-CS/SE related majors

Undergraduate 1 10.50 1.64

2 9.25 1.21 3 9.42 1.43 4 7.33 1.23 5 6.50 1.31 Graduate/

postgraduate1 11.75 2.84

2 8.25 2.09 3 10.25 2.47 4 9.00 2.13 5 8.25 2.27 CS/SE major Undergraduate 1 12.29 1.52 2 10.00 1.12 3 9.29 1.32 4 7.71 1.14 5 7.21 1.22 Graduate/

postgraduate 1 .(a) .

2 .(a) . 3 .(a) . 4 .(a) . 5 .(a) .CS/SE design experience

non CS/SE related majors

Undergraduate 1 14.50 4.02

2 14.50 2.95 3 12.00 3.49 4 12.00 3.01 5 5.00 3.21 Graduate/

postgraduate1 15.00 4.02

2 12.00 2.95 3 14.0 3.49 4 14.00 3.01 5 12.50 3.21 CS/SE Undergraduate 1 15.00 1.42 2 12.19 1.04 3 9.88 1.23 4 8.13 1.06 5 8.00 1.14 Graduate/

postgraduate1 16.90 1.80

2 13.90 1.32 3 11.90 1.56 4 12.70 1.35 5 10.50 1.44a. This level combination of factors is not observed, thus the corresponding population marginal mean is

not estimable.

Note: Formality Level: 1 = Low formality (on paper); 2 = Low formality (on Tablet PC); 3 = Medium-low formality; 4 = Medium-high formality; 5 = High formality.

Appendix N. One-way ANOVA and post-hoc multiple comparisons between total, quality, and expected changes made across each level of formality

Table N1. Significant between group differences and linear trends found after conducting one-way ANOVA on total, quality and expected changes.

df F Sig.Low formality (paper) Between Groups (Combined) 2 6.67 .002* Linear Term Contrast 1 13.23 .000**Low formality (on Tablet PC) Between Groups (Combined) 2 8.15 .001* Linear Term Contrast 1 16.28 .000**Medium-low formality Between Groups (Combined) 2 8.08 .001* Linear Term Contrast 1 15.26 .000**Medium-high formality Between Groups (Combined) 2 9.10 .000* Linear Term Contrast 1 18.08 .000**High formality Between Groups (Combined) 2 7.03 .001* Linear Term Contrast 1 13.43 .000**

* The between group difference is significant at the .05 level.** The linear trend (linear contrast across levels of formality) found is significant at the .05 level.

Table N2. Post-hoc multiple comparisons between total, quality, and expected changes made across each level of formality

Dependent Variable(I) Total, Quality,

Expected (J) Total, Quality,

Expected

Mean Difference

(I-J)Std.

Error Sig.Low formality (paper) Total Changes Quality Changes 3.00 1.43 .115 Expected Changes 5.18* 1.43 .001 Quality Changes Total Changes -3.00 1.43 .115 Expected Changes 2.18 1.43 .387 Expected Changes Total Changes -5.18* 1.43 .001 Quality Changes -2.18 1.43 .387Low formality (on Tablet PC) Total Changes Quality Changes 2.12 .99 .104 Expected Changes 3.98* .99 .000 Quality Changes Total Changes -2.12 .99 .104 Expected Changes 1.87 .99 .186 Expected Changes Total Changes -3.98* .99 .000 Quality Changes -1.87 .99 .186Medium-low formality Total Changes Quality Changes 1.10 .97 .777 Expected Changes 3.78* .97 .001 Quality Changes Total Changes -1.10 .97 .777 Expected Changes 2.68* .97 .021 Expected Changes Total Changes -3.78* .97 .001 Quality Changes -2.68* .97 .021Medium-high formality Total Changes Quality Changes 2.33 .97 .054 Expected Changes 4.13* .97 .000 Quality Changes Total Changes -2.33 .97 .054 Expected Changes 1.78 .97 .207 Expected Changes Total Changes -4.12* .97 .000 Quality Changes -1.78 .97 .207High formality Total Changes Quality Changes 2.25* .89 .040 Expected Changes 3.27* .89 .001 Quality Changes Total Changes -2.25* .89 .040 Expected Changes 1.02 .89 .772 Expected Changes Total Changes -3.27* .89 .001 Quality Changes -1.02 .89 .772

* The mean difference is significant at the .05 level.

Appendix O: “Extra changes” made in designs.

Appendix O1. “Extra changes” made in the Low Formality Design presented on paper: International Online Magazine Subscription Form

Types of changesT – Q(no. of

changes)

Q-E(no. of

changes)Change of text in label“Do you want a daily/weekly/yearly mag?” text changed to “How often do you want the mag?”

2 2

“Do you want a daily/weekly/yearly mag?” text changed to “I want to receive my mag…”

1 1

“Do you want a daily/weekly/yearly mag?” text changed to “How do you want it to be sent…”

1

“Do you want a daily/weekly/yearly mag?” text changed to “Do you want the magazine daily/weekly/yearly mag?”

1

At “Mailing address” item: label in first line changed to “Street no/Street” 1 1“What type of things…” text changed to “Interest…” 3“What type of things…” text changed to “Magazine content preference…” 1“What type of things…” text changed to “What genres are you interested in viewing in your magazine…?”

1

“Full Name”: text changed to “Subscriber’s Full Name” 2“Age ”: text changed to “Age years old (for statistical and content restriction purposes”

1

“Age”: label changed to “DOB” 3“Login Name” text changed to “Preferred Login Name” 1 1“Login Name” text changed to “Choose a Login Name” 1 1“Password” text changed to “Preferred password” 1 1“Contact no.” text changed to “Preferred Contact no.” 1 1“Contact no.” text changed to “Phone number” 1“Contact no.” text changed to “Number” 1“Email” text changed to “Email address” 2“Do you want the dating option?” “Would you like the dating option?” 1 1“(Please Choose….)” text changed to “(Please tick….)” 1At “Address” item set: labels changed to “Address Line1, Address Line2, Address Line3…

1

“Subscription to International Online Magazine” text changed to “Subscription Form to International Online Magazine”

1

“Payment” text changed to “Payment type” 1 1Relocation of elements/items/item setsAt “Date of loan” item set: items moved from horizontally aligned to vertically aligned

5

At “Payment” item set: items moved from horizontally aligned to vertically aligned

1 1

At “Interest” item set: items listed both horizontally and vertically instead of listing downward:

3 3

At “Dating option”: radio buttons moved from the left to the right of the label 1

“Age” (or “DOB”) item(s): moved below “Address” item set 1“Payment” item set: moved below “Daily/weekly/yearly mag” item set 4 4“Payment” item set: moved to the bottom, below “Viewing Interest” item set, above “Dating option” item.

2

“Payment” item set: moved to the bottom before the “Submit” item 1“Full Name”, “Age” and “Address” items/set below “Ethnicity” item 1“Full Name”, “Age” and “Address” items/set moved to the top as the first few items below the main heading

2

“Full Name”, “Age” and “Address” items/set below “Email” item 1

“Full Name” and “Age” items: moved to the top as the first item 3“Full Name” and “Age” items: moved below “Password” item 1“Full Name” item: moved to the top as the first item 1“Age” item: moved next to “Contact no” item 1“Email” item: moved above “Password” item (i.e. swapped location) 1“Email” item: moved below “Ethnicity” item 1“Email” item: moved to the right of “Contact no.” item 1“Contact” item: moved above “Email” item (i.e. swapped location) 2“Contact” item: moved below “Country” item within the “Address” item set 2“Favourite singer” and “Favourite movie” items: moved above “Viewing Interests” item set

3 3

“Favourite singer” and “Favourite movie” items: moved below “Viewing language” item, above “Daily/weekly/yearly mag” item set

1

“Favourite singer” and “Favourite movie” items: moved to the bottom of everything

1

“Favourite movie” item: moved to the right of “Favourite singer” item: 1“Daily/weekly/yearly mag” item set: moved below “Viewing Interests” item set. 1“Login Name” and “Password” items: moved below “Payment” item set 1“Address” item: moved below “Contact no.” item 1“Daily/weekly/yearly mag” item set: moved above “Payment” item set 1“Daily/weekly/yearly mag” item set: moved below “Country” item within the “Address” item set

1

“Daily/weekly/yearly mag” item set: moved above “Favourite singer” and “Favourite movie” items

1

“Country” item: placed first in the “Address” item set 1“Password” item: placed next to “Login Name” item 1Change of element(s) type in an item/item setAt “Dating option” item: question type changed to“ I would like the dating option”

2 2

1

1

At “Age” item: changed to 1 1At “Email” item: 1

At “Ethnicity” item: 8 8 **At “Payment option” item: items merged into one item with a dropdown menu 5At “Daily/weekly/yearly mag” item set: items merged into one item with a dropdown menu

3

At “Town/City” item within the “Address” item set:

4 4

At “Country” item within in the “Address” item set:

3

At “Viewing Interests” item set: all radio buttons merged into one item 1“Dating option” item set and “Submit” item set merged 1 1Adding an Element/Item / Item setAdded “ Required fields” at the top right corner below the main heading 1 1Added reminder “ Remember to check everything before you submit the form” 1 1Added instruction “Fill in details below” at the top below the main heading 1 1Added headings for each section e.g. “Login info”, “Payment Options”..etc 1Following the “Do you want a hard copy” item: added “If yes, mailing address” or similar at above “Mailing address” item set.

8 8

Added “Confirm Login name: ” below “Login Name” item 1Added “Submit” Question: “Do you want to submit? ”

1 1 *, **

Added “Submit” Question: “Do you want to submit? ”

1 1 *, **

Added “ Submit” item at the end of the form 1 1 *, **

Added “Verify Email ” below “Email” item 1Added group box around “Mailing Address” item set 3 3Added group box around “Login Name” and “Password” items 1 1Added heading “Address” at the “Mailing address” item set 1 1At “Mailing Address ” item within the Address” item set: split into two items: No. Street

1

Added extra line of input in “Address” item set: “Suburb ”

2 2

Added “ Credit card no. ” to the “Credit card” item within the “Payment” item set

7 7

Added “Account no. ” to the “Bank deposit” item within the “Payment” item set

1

Added “Postcode ” below “Address” item set 2 2Added “Others ” to the right of “Favourite Singer” item 5 5 *, **Added “Others ” to the right of “Favourite Movie” item 5 5 *, **Added “Other ” within the “Viewing Interests” item set 1 1 *, **Added “Date of Birth ” to the right of “Age” item 1Added “Gender ” at the top of the design below the main heading 1Added instructions “Password must be xxx characters min and xxx characters max containing xxx characters” etc at the “Password” item

2 2

Added “ Others” at the “Viewing interests” item set 2 2 *, **Added “Start date ” after “Daily/weekly/yearly mag” item set 1Added “Start date ” after “Daily/weekly/yearly mag” item set 1

Added “Are you currently a mail subscriber? ” after “Daily/weekly/yearly mag” item set

1

Added “Check availabilty” button at “Login Name” item 2 2Deleting an Element/ Item / Item setDeleted “Age ” item 1 At “Daily/weekly/yearly mag” item: textboxes combined to become one dropdown menu

1

Deleted “Full Name ”item 1Deleted “Favourite singer ” item 1

Deleted “Favourite movie ” item 1Deleted “ ” at “Weather” item within the “Viewing Interests” item set 3 3Deleted “Payment” item set 1Deleted “Dating Option” item set 1Resizing an Element(s)At “Full Name” item: longer textbox 5 5 *, **

At “Age” item: smaller textbox 2 2 *, **Note: T – Q = Total changes – Quality changes made; Q – E = Quality changes – Expected changes made

* scored 0.5 out of 1 the particular quality change due to some incorrectness according to the criteria for quality changes

** scored 0.5 out of 1 for the particular expected change due to some incorrectness according to the criteria for expected changes.

Appendix O2. “Extra changes” made in the Low Formality Design on tablet PC: Samson’s Bank $1 million Loan Application Form

Types of changesT – Q(no. of

changes)

Q-E(no. of

change)Change of text in label“Address: House/Street” text changed to “Address: Street no./Street” 1 1“Address: House/Street” text changed to “Address” 2“When do you want your loan?” text changed to “Loan date” 1 1“Any Questions?” text changed to “Any Questions? Please specify” 1 1“Any Questions?” text changed to “If you have any questions, please state”

1 1

“Any Questions?” text changed to “Questions?” 3At the “Check “ item set: “Income verification” text changed to “Income verification from Inland Revenue”

3 3

At “Submit” item set: “Yes” and “Next page” text changed to “Yes, next page” and “No, go back”

1 1 *, **

“Weekly income range” text changed to “Weekly income” 2“Weekly income range” text changed to “Your Weekly income” (with a )

1 1

“Weekly income range” text changed to “Your Weekly income range” 2 2“Status” “Marital Status” 5 5“Person verification” text changed to “Proof of ID” 1 1 *“Income verification” text changed to “Proof of income” 1 1 *“Past (10 years) bank records” text changed to “Past (10 years) bank statements”

1 1 *

“Home no.” text changed to “Home phone no.” 3 3“Reasons for loan” text changed to “Purpose of your loan: Please describe”

1 1

“Reasons for loan” text changed to “Reasons of your loan: Please describe”

1 1

“Other Personal Assets” text changed to “Personal Assets of Value” 1 1“Other Personal Assets” text changed to “Other Personal Assets of Value” 1 1“When do you want your loan” text changed to “Date of loan” 1“IRD number” text changed to “IRD number (for security reasons)” 1“Home no.” and “Mobile” text changed to “Contact (home)” and “Contact (mobile)”

1

“Mobile” text changed to “Mobile no.” 1 1“Personal Asset” item: text changed to “Personal asset worth in total” i.e. changed meaning

1

Relocation of elements/items/item set“Income range” item: moved above “Date of Loan” item set, below “Status” item

1

“IRD” item: moved below “Date of Loan” item, before “Personal Assets” item

1 1

“IRD” item: moved below “Contact numbers” items 1“IRD” item: moved below “Status” item, above “Date of loan” item 1“IRD” item: moved next to “Personal Assets” item 1“Passport no.” item: moved below “IRD” number items, above “Address” item set

1

“Passport no” item: moved below “Address” item set 1“Weekly income range” item: moved above below “Occupation” item, above “Status” item

2 2

“Address” item set, moved below “Full Name” item 2 2“Mobile no.” item: moved next to “Home no.” item horizontally 2“Mobile no.” item: moved above “Home no.” item (i.e. swapped location) 1“Date of loan” item set: moved above “Check” item set 1“Date of loan” item set: moved below “Personal Assets” item, above “Reasons for Loan” item

1 1

At “Date of loan” item set: labels below elements moved to the right hand side of the elements: dd mm yyyy

1

“Address” item set: moved to the right of “Full Name” item 1“Status” item: moved to the right of “Occupation” item 1At “Date of loan” item set: items moved from horizontally aligned vertically aligned

1

Change of element(s) type in an item/item setAt “Occupation” item: 8At “Yearly income range” item: 10 10 **At “Yearly income range” item: multiple 2 2 **At “Reasons for loan” item: 6, 12, 22At “Status” item: multiple radio buttons 6 6At “Personal Assets” item: 1 1 **At “Any Questions” item: 1 1 *Adding an Element/Item / Item setAt “Loan date” item set: added slashes / / 1 1Added “Suburb ” below “Address (House/ Street) item within the “Address” item set

3 3

Added “Renting: Yes ” at the “income range” items 1 1 **Added “$” at “Weekly income range” item:

1 1

Added “ Expiry date: ” next to “Passport Number” item 1At “Address(House/Street) ” item within the “Address” item set: split into two items: No. Street

1

Added “Email ” item at “Contact number” items 2 2Added instruction “Please fill in details below. ** are compulsory fields” at the top below the main heading

1 1

Added “Address” heading to the “Address” item set 1 1Added “Personal details” heading above the “Full name” item 1 1Added “Other ” at “Reasons for loan” item to the right of the dropdown menu (changed earlier)

3 3*, **

Added “Other ” at “Personal assets” item 2 2Added “ ” to the right of “Any Question ” item

1 1*, **

Added “ ” to the right of: “Occupation ”

2 2*

Added ‘area code element’ at “Home no.” item: “Home no. ”

1 1

Added “Workplace no ” 1Added below “Bankruptcy” item set:“Reason(s) for bankruptcy, please describe: “

1 1

Added instruction “Please provide hard copy of bank statements” at “Check” item set

1 1

Added numbers to each question 1 1Added instruction “If married, fill in xxxx” at the “Income” items 1 1Added “Reset” and “Clear” buttons in addition to the “Submit” item 1Added “Section lines” to separate sections in the form 1 1Added “If others, specify please: ” at “Status” item 1Deleting an Element/ Item / Item set 1Deleted “ Do not agree ” in “Bankruptcy” item set. 1 1 *Deleted “Any Questions” item 1Deleted “Date of loan” item set (but maybe due to redoing the design from scratch)

1

Delete “Loan Agreement contract” item at “Check” item 1Added “Status” item below “Address” item set, in addition to the an existing item further down

1

Deleted “ ” and “ ” at “Submit” item set (left with “Submit Application” label)

1 1 *, **

Resizing an Element(s)Resized “Day” and “Month” items only within the “Date of Loan” item set.

1 1 **

“Date of Loan” item set: smaller textboxes

2 2

Larger main heading 1 1At “Full Name” item: longer textbox 1 1 *, **

Other changes made (not counted as functional changes)Aligned all on left (indicated) and indention 1 1Note: T – Q = Total changes – Quality changes made; Q – E = Quality changes – Expected changes made

* scored 0.5 out of 1 the particular quality change due to some incorrectness according to the criteria for quality changes

** scored 0.5 out of 1 for the particular expected change due to some incorrectness according to the criteria for expected changes

Appendix O3. “Extra changes” made in the Medium-Low Formality Design: University of Strawberries Graduation Form

Types of changesT – Q(no. of

changes)

Q-E(no. of

changes)Change of text in label“Mailing address: House/Street” text changed to “Mailing address: No./Street”

1 1

“Student ID” text changed to “University of Strawberry Student ID” 1“Contact No” text changed to “Contact No. (NZ)” 1 1At “Continue to next page” item set: “Home” and “Next page” text changed to “Yes, next page” and “No”

4 4*, **

At “Last day of your course” item set: “dd”, “mm”, and “yyyy” text changed to “Day”, “Month”, and “Year”

1 1*

At “Last day of your course” item set: “dd”, “mm”, and “yyyy” text changed to “dd”, “month”, and “yyyy”

1 1

“Degree Name” text changed to “Program/Degree Name” 1 1“Degree Name” text changed to “Degree” 1 1At “Borrow items” item set: “You want to borrow: ” text changed to “You want to borrow for the graduation ceremony: ”

1 1

“Preferred Name” text changed to “Preferred name (for the graduation ceremony)

1 1

“Preferred Name” text changed to “Preferred name (called at the graduation ceremony)”

1 1

“Graduation Year/Semester” text changed to “Preferred Graduation Year/Semester”

1 1

“Graduation Year/Semester” text changed to “Preferred Graduation time” 1 1*“Contact No.” text changed to “Number you preferred to be contacted” 1“Degree Name” text changed to “Program name” 1 1At the main heading: “….graduation…..” text changed to “…….Graduation…..”

1 1

Relocation of elements/items/item set“Gender” item moved below “Age” and “Appearance” 1Moving “Graduating in” item set above “Mailing Address” item set (or vice versa i.e. moving “Mailing Address” below Graduating in” item set

14 14

“Degree name” and “Department” items swapped location 2“Graduation year/semester” item: moved below “Graduating in” item set 1“Graduation year/semester” item: moved below “Ethnicity” item and above “Last day of your course” item set

1

“Contact No.” and “Ethnicity” items moved away so that graduation items next to above and below each other

2 2

“Contact No.” item: moved below “Mailing Address” item set. 2 2“Contact No” item: moved below “Last day of your course” item set. 2“Last day of your course” item set: moved below “Department” item and above “Graduation year/semester” items

1

At “Last day of your course” item set: labels below elements moved to the top of the elements dd mm yyyy dd mm yyyy

1

“Mailing Address” item set: moved up to “Graduating in” item set 2 2“Hood colour” item moved to the right of “Hood” item within the “Borrow items”

3 3

“Preferred Name” item: moved to the right of “Full Name” item 1“Degree Name” item: moved to the right of “Student ID” item 1“Graduation Year/Semester” item: moved to the right of “Department” item 1“Ethnicity” item: moved to the right of “Contact No.” item 1Change of element(s) typeAt “Hood colour” item within the “Things to borrow” item set:

1 1*

At “Ethnicity” item: 1 1**

At “Speech” item set: combined items changed to“I want to give a speech: ”

1 1**

At “Speech” item set: changed to (radio button) 12 12**At “Speech” item set: changed (tick box) 2 2At “Country” item within the “Mailing Address” item set: changed to

1

At “Town/City” item within the “Mailing Address” item set:

2 2

At “Ethnicity” item: multiple radio buttons 1 1**At “Country” item within the “Mailing Address” item set:

1

Adding an Element/Item / Item setAdded heading “Degree Name” 1 1Added “Other, please specify: ” to the right of “Ethnicity” item. 2 2**Added “If absent” to the left to the “Mailing address” label 15 15Added group box around “Address” item set 1 1Added “Hood Colour” item “auto-filled” 1 1Added “Pronunciation of preferred name ” below “Preferred Name” item

1 1

Added “Account Number” item in addition to the “Postage Payment” item required

2 2

At “Country” item within the “Mailing address” item set: split into: NZ

1 1

Added “Suburb ” at “Mailing Address” item set 1 1Added another “Address” item set in addition to the “Mailing Address” item. 1Added control properties i.e. “Speech” item set kept ‘hidden’ if the item before (“Graduating in” item) is not chosen

1 1

Deleting an Element/ Item / Item setAt “Speech” item set: deleted “ No, I don’t want to give a speech” item

1 1*, **

At “Borrow items” item set: deleted “Hood colour” item 1Deleted “Graduating in” item set (indicated no need but not counted) 1Indicated no need to have “Borrow items” item 1Resizing an Element(s)“Student ID” item: smaller textbox or 8 8“Last day of your course” item: smaller textboxes 1 1Larger main heading 1 1At “Full Name” item: longer textbox 1 1*, **

At “Preferred Name” item: shorter (smaller) textbox 1

Other changes made (not counted as functional changes)Alignment of elements, especially in the first half of the design (not counted as change)

6 6

Note: T – Q = Total changes – Quality changes made; Q – E = Quality changes – Expected changes made

* scored 0.5 out of 1 the particular quality change due to some incorrectness according to the criteria for quality changes

** scored 0.5 out of 1 for the particular expected change due to some incorrectness according to the criteria for expected changes

Appendix O4. “Extra changes” made in the Medium-High Formality Design: Dog Registration Online Form

Types of changesT – Q(no. of

changes)

Q-E(no. of

changes)Change of text in labelAt “Appearance” item set: “Height” and “Weight” text changed to “Height (cm)” and “Weight (kg)”

13 13

“I love my dog with all my heart” text changed to “Love the dog?” 1At “Second owners address” item set : “Address” changed to “Second owners address”

4 4

“Address” text changed to “Address: no./street” 1 1“Address” text changed to “Address: Street” 1 1 “Dog’s name” text changed to “Name of the dog” 1Relocation of elements/items/item setAt “Appearance” item set: items moved from horizontally aligned vertically aligned

3 3

At “Appearance” item set: labels below elements moved to the left of the elements: “Height Weight Colour ”

1

At “Agree to Love dog” item: changed sides of radio buttons from to

1

“Dog name” item moved down to dog information area above “Age” and “Appearance”

2 2

“Gender” item moved below “Age” and “Appearance” 1“Breed” item moved below “Age” and above “Appearance” 1 1“The Shop is” item within “Where did you get your dog” item set moved next to “Brought from pet shop” item

3

Second owners item sets moved to the right side to First owner’s item sets

2

“Age” item moved to the right of “Gender” item 1“Register with vet” item moved to the right of “Appearance” item set 1“Register with Dog lovers’ society” moved below “Register with vet” item

1

“Purpose of dog” item moved next to “Dog’s special conditions” item

1

At “I Agree to love dog” item set: “ Yes No ” items moved to the right of “I agree to love my dog…” label

1

Change of element(s) typeAt “Age” item: changed to a 10At “Registered with Dog Lovers society” item: 11 11 *, **

At “Registered with Vet” item: changed to 3 3 *, **At “Gender” item: changed to 7At “Gender” item: changed to “Male Yes” 1At “Where did you get dog” item set: all radio buttons changed to tick boxes

5

At “Where did you get your dog” item set: list of items changed to a single line (dropdown menu)

1

“I agree to love dog” item set: changed to “ I agree to love the dog with all my heart ”

1 1

At “Town/city” item within the “Address” item set: changed to

1 1

At “Purpose of dog” item: textbox changed to text area 1At “Sterilized” item: radio buttons changed to 1 “Age” item: changed to “ Date of Birth ” 1Adding an Element/Item / Item set

Added instruction “ required fields” at the top below the heading 1 1Adding “if there’s a 2nd owner” text (or equivilent and/or with a tick box or radio button) above “Second owner” item set

6 6

Added “ If other, please specify: ” next to the “And the Shop is: ” item

1

Added “ Other, specify: ” after the options available at “Where did you get the dog?” item

5 5

At “the Shop is ” item within the “Where did you get dog” item set: “Shop is: ” split into two items: “Name: Area: ”

1 1

Added group boxes to separate first and second owner 3 3Added “Appearance: ”

Separated from “Height”, “Weight” and “Colour” items

1

Added “Other: ” at the “Appearance” item set 1Added “Owners Details” heading at the top below the main heading of “Dog registration form”

3 3

Added “Suburb: ” item to “Addresss” item set 1 1Added “Postcode: ” item to “Address” item set 1 1 “Age” item split into: “Month: Years ” 1At “Dog’s special conditions” item: added extra dropdown menus 1Added “Are you moving out soon? Yes No ” and “If yes, specify ” below “Address” item set

1

Added “If registered with vet, vet name: ” at “Registration with vet” item

1

Added “Proceed to next page: Yes No ” at the end of the page 1 1 *, **Added “Up-Load a picture of dog” item set 1Added numbers to each items e.g. 1, 2, 3a, 3b, 4, 5…etc 2 2Added “ If others, please specify ” next to “Purpose of dog ” item

1 1

Added a heading at each section (in addition to “Dog’s information” and “Owner’s information” headings )

1 1

Deleting an Element/ Item / Item setDeleted “Appearance” item set 1Deleted “Second Owner’s Address” item set 1At “Sterilized” item set: deleted “ ” item 1At “I agree to love dog” item set: deleted “ ” item 1 1At “Where did you get the dog” item set: deleted “And the shop is” item 1Resizing an Element(s)“Address” item longer textbox 1 1Other changes made (not counted as functional changes)Aligned everything (but not counted) 3 3

Note: T – Q = Total changes – Quality changes made; Q – E = Quality changes – Expected changes made

* scored 0.5 out of 1 the particular quality change due to some incorrectness according to the criteria for quality changes

** scored 0.5 out of 1 for the particular expected change due to some incorrectness according to the criteria for expected changes

Appendix O5. “Extra changes” made in the High Formality Design: 2007 America’s Next Top Model Online Application Form

Types of changesT – Q(no. of

changes)

Q-E(no. of

changes)Change of text in label“Status” text changed to “Title” 3At “Submit” item: changed text from “I agree” and “I do not agree”; to a variation of “Yes, submit” and “No”

5 5 **

“Why Enter” text changed to “Reasons for entry” 1“Status” text changed to “Marital Status” 3 3 “Where did you hear this?” changed to “Where did you hear about this competition?”

1

“Date of Birth” text changed to “DOB” 1“Address” text changed to “Home address” 1“Address: Street” text changed to“Address” 1“Email” text changed to “Email address” 1Relocation of items/item set“Address” item set moved below email & contact number 1“Email” item moved above contact number (swapped places) 1“Date of birth” item moved above “Gender” & “Age” items 1Items(set): “Status”, “gender”, “age” and “date of birth” moved below the “name” item

1

“Weight item” moved below “Height” item 1“Why enter” item moved below “Occupation” and “Modeling experience” items

9 9

“Why enter” item moved below “Occupation” item (i.e. swapped locations)

1

“Date of birth” item moved above “Age” item 3“Status” item below “Age” item 1At “Date of birth” item set: items moved from horizontally aligned vertically aligned

1

Change of element typeAt “Age” item: the dropdown menu changed to textbox 1At “Gender” item: dropdown menu changed to radio buttons associated with “male” and female”

5 5 **

At “Status” item: radio buttons changed to a textbox 1At “Email” item: the textbox changed to: 1

At “Submit” item: radio buttons associated to “I agree” and “I do not agree” changed to buttons

2 2 *, **

At “Why enter” item: textbox changed to a textbox 6 6 *, **At “Experience” item: dropdown menu changed to a textbox 3 3 *, **Adding an Item / Item setAdded “Others ” item to the right of “Occupation” item 1 1Added “Others, please specify ” item to the right of “Occupation ” item.

1 1

Added a group box around “Address” item set 1 1Added a group box around “Name”, “Address”, “Contact number” and “Email” items(sets)

1 1

Added “Mobile phone no.” item below “Contact No.” item 1 1

Added “Title: ” item at “Status” item 1At “Street” item within the “Address” item set: “Street split into two items:

1

Added: “ I agree with the terms and conditions” item set at the end (including a “terms and conditions link”)

1 1

Added: “Conditions link” item at the “Submit” item 1 1Added “Country ” in “Address” item set 1

Added one more item within the “Address” item: “Address 2: ”

1

Added text “(must be in America)” at “Address” item set. 1Added “Personal detail” heading at the top below the main heading “America’s next top model”

1 1

Added instruction “ important details to fill in” at the top right hand corner

1 1

Added “ ” to the right of “Modeling experience” item

1

At “Modeling experience” item: added next to the element

1

Added “Years of related experience” item to the right of “Modeling Experience” item

1 1

Added “Other ” to the right of “Status” item 1Added “Please confirm age ” item at “Age” item 1At “Modeling Experience” item: added“Describe ” (text area) next to it

1 1 *

Added “Or other reasons: ” next to “Why enter” item 1Deleting an element/ Item / Item setAt “Other reason” item within the “Heard from” item set: deleted the radio button / check box on the left associated to the item

2

At “Other reason” item within the “Heard from” item set: deleted the label “Other reason” left with “ ” only

1

At “Submit” item set: deleted the item “ I do not agree” 1Deleted “Age” item 1Resizing an Element(s) in an item / item set“Date of birth” item set smaller textboxes 1 1“Gender” item a smaller dropdown menu 1“Town/City” item within the “Address” item set smaller textbox 2“Email” item longer textbox 1“Contact No.” item smaller textbox 2“Full name” item longer textbox 2 2*, **Larger heading “2007 America’s Next Top Model” 2 2Other changes madeAligned everything (counted as a functional change – easier to follow) to the left

2

Note: T – Q = Total changes – Quality changes made; Q – E = Quality changes – Expected changes made.* scored 0.5 out of 1 the particular quality change due to some incorrectness according to the

criteria for quality changes** scored 0.5 out of 1 for the particular expected change due to some incorrectness according to

the criteria for expected changes

Appendix P. “Overall Enjoyment” rankings of the five designs across each level of formality

Table Z1. Percentage and frequency of rankings of designs with different levels of formalityLevel of formailty Ranking from 1 – 5 Frequency PercentLow formality (on paper) 1 - Most liked 5 16.7 2 - 3 10.0 3 - 3 10.0 4 - 14 46.7 5 - Least liked 5 16.7Low formality (on Tablet PC) 1 - Most liked 3 10.0 2 - 1 3.3 3 - 3 10.0 4 - 3 10.0 5 - Least liked 20 66.7Medium-low formality 1 - Most liked 3 10.0 2 - 4 13.3 3 - 12 40.0 4 - 7 23.3 5 - Least liked 4 13.3Medium-high formality 1 - Most liked 3 10.0 2 - 13 43.3 3 - 9 30.0 4 - 5 16.7

5 - Least liked 0 0.0High formality 1 - Most liked 17 56.7 2 - 8 26.7 3 - 3 10.0 4 - 1 3.3 5 - Least liked 1 3.3

Table Z2. Individual subjects’ rankings of the five designs in the order from 1 (“most-liked design”) to 5 (“least-liked design”). Underlying reasons for the rankings (indicated by the subjects) are also shown: (A) = Aesthetics; (B) = Perceived effort required; and (C) Fun/stimulating level.

SubjectsHigh

formality

Medium-high

formality

Medium-low

formality

Low formality (on Tablet

PC)

Low formality (on paper)

Factors:(A) (B) (C)

1 1 2 3 5 4 1 12 1 3 2 5 4 13 1 2 3 5 4 14 2 3 4 5 1 1 15 1 2 3 4 5 16 5 4 3 1 2 17 1 3 2 5 4 18 1 3 4 5 2 1 19 1 2 4 5 3 1

10 1 2 4 5 3 111 1 4 3 5 2 1 112 3 2 1 5 4 113 4 1 2 3 5 114 2 3 1 4 5 1 1 115 2 1 5 3 4 116 3 2 5 1 4 117 3 4 2 1 5 1 118 1 2 3 5 4 119 2 3 1 4 5 120 2 4 3 5 1 121 2 3 4 5 1 122 1 2 3 5 4 1 123 1 2 3 5 4 124 2 3 4 5 1 125 1 2 5 3 4 1 126 1 2 3 5 4 127 1 4 5 2 3 128 1 2 3 5 4 129 2 3 4 5 1 130 1 1 3 5 4 1

Mean 1.7 2.53 3.17 4.2 3.37 N = 21 N = 11 N = 7

Appendix Q. Number of changes made in the five designs across levels of formality by subjects whose “Overall Enjoyment” ranks was dependent on

the appearance (aesthetics) of designs.

Table Q1. Total number of changes made across levels of formality by subjects who ranked according to the apperance of the designs (n = 21).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 11.33 0.81 3.71Medium-high formality 13.86 0.88 4.03Medium-low formality 14.00 0.85 3.91Low formality 15.57 0.98 4.48Low formality (paper) 19.95 1.52 6.98

Table Q2. Number of quality changes made across levels of formality by subjects who ranked according to the apperance of the designs (n = 21).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 9.31 0.82 3.76Medium-high formality 11.48 0.91 4.17Medium-low formality 12.79 0.79 3.64Low formality 13.43 0.92 4.21Low formality (paper) 16.62 1.27 5.83

Table Q3. Number of expected changes made across levels of formality by subjects who ranked according to the apperance of the designs (n = 21).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 8.23 0.74 3.40Medium-high formality 9.62 0.81 3.70Medium-low formality 10.07 0.73 3.34Low formality 11.48 0.76 3.50Low formality (paper) 14.07 0.96 4.38

Appendix R. Number of changes made in the five designs across levels of formality by subjects whose “Overall Enjoyment” ranks was dependent on

perceived effort required

Table R1. Total number of changes made across levels of formality by subjects who ranked according to percieved effort required to improve designs (n = 11).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 11.91 0.97 3.21Medium-high formality 14.64 1.07 3.56Medium-low formality 16.55 1.11 3.67Low formality 16.91 1.08 3.59Low formality (paper) 19.55 1.76 5.84

Table R2. Number of quality changes made across levels of formality, by subjects who ranked according to percieved effort required to improve designs (n = 11).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 9.18 0.86 2.86Medium-high formality 12.23 1.08 3.57Medium-low formality 15.36 1.15 3.82Low formality 14.77 1.11 3.70Low formality (paper) 16.96 1.54 5.11

Table R3. Number of expected changes made across levels of formality, by subjects who ranked according to percieved effort required to improve designs (n = 11).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 8.46 0.90 2.98Medium-high formality 10.64 1.07 3.56Medium-low formality 12.05 0.97 3.21Low formality 12.55 0.85 2.82Low formality (paper) 14.73 1.15 3.81

Appendix S. Number of changes made in the five designs across levels of formality by subjects whose “Overall Enjoyment” ranks was dependent on

the level of fun/stimulation when working on the designs.

Table S1. Total number of changes made across levels of formality by subjects who ranked according to the level of fun/stimulation when working on the designs (n = 7).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 11.14 1.39 3.67Medium-high formality 11.86 1.39 3.67Medium-low formality 13.14 1.60 4.22Low formality 12.86 1.10 2.91Low formality (paper) 15.71 2.31 6.10

Table S2. Number of quality changes made across levels of formality by subjects who ranked according to the level of fun/stimulation when working on the designs (n = 7).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 8.57 1.13 2.98Medium-high formality 9.21 1.20 3.17Medium-low formality 12.14 1.36 3.59Low formality 10.71 0.87 2.29Low formality (paper) 13.00 1.89 5.00

Table S3. Number of expected changes made across levels of formality by subjects who ranked according to the level of fun/stimulation when working on the designs (n = 7).

MeanStd.

Deviation Statistic Std. Error StatisticHigh formality 7.64 1.03 2.73Medium-high formality 7.79 1.04 2.75Medium-low formality 9.50 0.87 2.31Low formality 9.36 0.75 1.97Low formality (paper) 11.29 1.67 4.41

Appendix T. Subjects reasons for design tool preference during the experiment

Table T1. Reasons by participants who preferred using the tablet (Inkit)CS/SE major

CS/SE design experience Reasons for preference

* * can make changes quickly and easily without having to erase things manually: 1) just select and drag; 2) easy to use because easy to make changes, can move things around, resize, erase etc

* * easier to make changes but need more practice on it* easy to use, simple interface* * quicker changes and can move things around, resize.

can move things, fun, tidier* like computers & faster changes

because its easier to show where you want to move things to, compared to paper – needs arrows & crossing out etc

* * fast changes, and move things around and transform thingseasier to change

* * Liked working on the tablet (but no particular preference if have to choose between 2, I don’t mind)

* * easier to make changes but need more practice on it* it was fun, and easy to make changes, rather than crossing out* faster, easier, click instead of writing* was good but with more practice - would prefer a lot more than paper

Table T2 Reasons by participants who preferred using the tablet (Inkit)CS/SE major

CS/SE design experience Reasons for preference

* * liked paper more coz tablet = cumbersome and annoying to click on select/draw/erase buttons

* * Not really like working with tablet (but I suspect it's lack of familiarity) so preferred paper

* * easier to make changes, faster, just crossing out stuff, and done.* * Preferred/liked paper more than tablet coz easier to draw and write on

paper, right in front of you, no scrolling etceasier but maybe computer is good to make things look good, and move things aroundNo preference after getting used to the pen on tablet, but would prefer paper before

* Liked working on paper better – not used to pen as my handwriting is terrible to follownice to change from computer to paper but preferred paper so it's in front of you - easier visualizationeasier to make changes using a pen on paper, tablet = hard to do selecting, erase, draw etc

* * faster coz can draw faster, and crossing out, mental iteration etc* because Tablet's interface was bad, and the changing of modes was

annoying (draw, erase, select), doesn't have the freedom like paper* because on tablet, had to change modes between draw, select, erase,

tiring, and annoying. If short cut, then better* easier with pen and paper

Table T3 Reasons by participants who had no particular preference in design tool CS/SE major

CS/SE design experience Reasons for preference

* * For Dog and Model - wouldn't mind which to design with - good to use tablet so can move things with (reordering) but with designs that required lots of changes[Online Magazine and Loan Application], I would prefer to use pen and paper

* * Paper = more accurate, can draw things better compared to the tablet. But with Tablet - it's easier to change, delete and modify.

Appendix U. Subjects reasons for design tool preference in real life design situations

Table U1. Reasons by participants who had no preference(s)

CS/SE majorCS/SE design Experience Reasons

^ Depends: weather design is complex or simple

Table U2. Reasons by participants who preferred using paper and pen

CS/SE majorCS/SE design Experience Reasons

Paper: more control, faster at drawing and writingProbably paper

^ * Paper - faster, pen and paper infront of you, more freedom

Table U3. Reasons by participants who preferred using tablet (Inkit)

CS/SE majorCS/SE design Experience Reasons

^ * Tablet, definitely coz got the parts from both worlds - comp and paper^ * tablet everytime - easier to work with + easier to erase mistakes

completely cf paper - even when rubbed out, deisgn mistakes are often sill visable and can be distracting

^ tablet, coz easier to edit stuff

Table U4. Reasons by participants who preferred using computer (other tools)

CS/SE majorCS/SE design Experience Reasons

^ * PC programs but tablet if it's easier to use - eg not to move too much with hand

^ * On PC: .Net - just need to drag and drop, tablet is too clumpsy, draw, select, erase, etc)

^ * PC: other programs such as fireworks, photoshop etc, - easy editing and not tablet coz not user friendly, hands moving too much.

^ * ComputersComputer - software, but easier than inkit (depend on software). Don't use paper.

^ * Computer more than paper and tablet* Will Computer (something like VB.net, drag and drop) wouldn't use

paper first, straight to comp. * Compter - can do direct changes on the computer.

Table U5. Reasons by participants who preferred using pen and paper, then tablet

CS/SE majorCS/SE design Experience Reasons

^ * If working on designs like dog or antm, then would use tablet. If beginning of a design, then would use pen-paper. Paper if still draft like, tablet if near finished.

Table U6. Reasons by participants who preferred using paper and pen, then continue on PC

CS/SE majorCS/SE design Experience Reasons

paper - faster and easier with paper and pen, then to PC^ * paper over tablet at the moment, coz like using paper and pen/ use PC.

Would choose to use mouse and keyboard coz not used to tablet's pen-inputpaper - coz technology (inkit) is hard to use, not user friendly, paper, then put it on computer

^ * Computer but not necessary a tablet - because not used to tablet, and have different modes: select, erase, draw. Would use paper first then computer.

^ Paper first then computer^ Paper first, then computer. And probably not Tablet, coz hard to draw

on…unless user friendly programPaper first, then computer. Need to get used to tablet if going to usePorbably paper to do a draft, then put onto computerDraw on paper first to sketch out idea first then computer

^ * Would use paper if start on scratch, if got everything there, would use computer - would be good if can just transform into nice looking after you've drawn it

^ * Draft on paper first (from rough to finalized) then use the final version from paper and transfer to computer and then some more finalizing on computer

^ Paper and pen, then transfer to computer^ Paper first then computer^ Paper first for rough copy because so many modifications are made,

then computer for drafts