Upload
zhiyuan
View
217
Download
0
Embed Size (px)
Citation preview
Collaborative sketching in crowdsourcing design: a newmethod for idea generation
Lingyun Sun • Wei Xiang • Shi Chen • Zhiyuan Yang
Accepted: 18 August 2014� Springer Science+Business Media Dordrecht 2014
Abstract Design integrates concepts and solves problems. Crowdsourcing design
imports vast knowledge and produces creative ideas. It publishes design tasks, collects
dozens of contributors’ ideas and reward the best. Contributors in crowdsourcing design
work individually when generating ideas. However, those who collaborate could make
better use of crowd’s knowledge, which might produce ideas of higher quality. By ana-
lyzing the advantages and disadvantages of crowdsourcing design, this article proposes a
collaborative crowdsourcing design method that integrates crowd’s sketching processes.
This method uses a tree to arrange crowd’s ideas and enable flexible modifications of prior
ideas. A demonstration system named Sketchfans is developed, and the results of a
crowdsourcing sketching experiment using this system are presented. The experimental
results validate that this method is effective; participants rely heavily on the idea tree for
inspiration, and the best ideas appear around the ends of branches on the idea tree.
Moreover, participants displayed unique development patterns. They often developed high-
quality ideas from initial ideas that were regarded as poor quality. The demonstration
system Sketchfans, supports this method well. Finally, we optimize Sketchfans based on
analysis of participants’ activities and feedback.
Keywords Idea generation � Conceptual design � Collaborative sketching �Crowdsourcing
L. Sun � W. Xiang � S. Chen (&) � Z. YangModern Industrial Design Institute, Zhejiang University, Hangzhou, Chinae-mail: [email protected]
L. Sune-mail: [email protected]
W. Xiange-mail: [email protected]
Z. Yange-mail: [email protected]
123
Int J Technol Des EducDOI 10.1007/s10798-014-9283-y
Introduction
The internet enables people around the world to cooperate and collaborate easily.
Crowdsourcing design recruits contributors from various backgrounds to submit solutions
for design tasks, and makes use of the vast knowledge and experiences residing in crowd at
a low cost. 99Designs as a typical crowdsourcing design website holds small design
contests that collect on average 86 ideas for a task (Wooten and Ulrich 2011), while only
rewarding the best.
Researchers have proposed multiple methods to improve the quality of ideas in
crowdsourcing design (Wooten and Ulrich 2011; Morris et al. 2013), but they have not
changed the way of idea generation. Contributors still work individually to produce ideas.
Design is an iterative process during which ideas are constantly evaluated and reformulated
(Restrepo and Christiaans 2004); the crowd’s knowledge and perspectives are beneficial to
contributors’ idea evaluation and idea generation processes. Studies of group design also
propose methods that integrate group members’ design processes and enhance the design
quality (Shah et al. 2001). Therefore, people who collaborate to design might produce high
quality ideas. Besides, crowd who collaborate could learn from others’ design processes
and improve their design ability, and this enables sustainable crowd work (Kittur et al.
2013).
This article integrates people’s sketching processes (thus constituting the crowd in
crowdsourcing) to explore a new method of crowdsourcing design. It proposed a collab-
orative sketching method that employed a tree structure to arrange crowd’s sketches,
developed a demonstration system named Sketchfans, and conducted an experiment to
examine the quality of ideas produced by collaborative sketching. Sketchfans consisted of
a task module to publish and participate in tasks, and a sketching module to support
sketching and arrange people’s ideas into idea trees. People in different locations could use
Sketchfans to scan existing idea trees and go on sketching. Sketchfans thus enabled distant
and unsynchronized collaborative sketching, which might be beneficial for crowdsourcing
design. This experiment instructed participants to undertake design tasks using Sketchfans,
and recorded their activities. Participants’ ideas were assessed, and their suggestions for
this collaborative sketching process were transcribed. We compared the usefulness and the
originality of the ideas generated by multiple periods of crowdsourced sketching, and
summarized the development patterns between ideas and their antecedents to examine how
the ideas evolve. Sketchfans is then optimized on account of participants’ activity patterns
and suggestions.
Previous studies
Design as an evolution process
Design involves a unique way of thinking. It starts from vague problems and aims to fulfill
ambiguous goals. Designers need to gather a lot of information to structure problems and
goals, and find paths that connect them (Goldschmidt 1997). They achieve this by con-
tinually refining their solutions. These solutions import new concepts, reveal design situ-
ations, and enrich designers’ understanding of current problems (Sim and Duffy 2003).
Problems are thus re-formulated and new design constraints are added, inducing solutions
that are more suitable than before (Restrepo and Christiaans 2004). The evolution of
solutions distinguishes design from other domains (Cross 2006).
L. Sun et al.
123
A typical industrial design process involves the four stages of conceptual sketching,
drawing, modelling, and prototyping (Pei 2009). Sketching is the main method of idea
generation. As a quick visual representation, sketches comprise both form and function
(Temple 1994; Taura et al. 2012). They provide external memories of designers’ thoughts,
enabling functional issues to be recalled at any time and communicated with others (Suwa
et al. 1998). Moreover, sketches are more ambiguous than drawings. Because designers’
understanding of problems and goals changes during the sketching process, ambiguous
sketches induce new interpretations of existing configurations, and these are the driving
force behind the discovery of further solutions (Pache 2005; Suwa et al. 2000). Therefore,
sketching largely supports the evolution of solutions in the design process.
Crowdsourcing design
Crowdsourcing represents the act of assembling large numbers of people via an open call
to complete a task (Howe 2006a, b). Four key components constitute a crowdsourcing
process: pre-selection of contributors, accessibility of peer contributions, aggregation of
contributions, and remuneration for contributors (Doan et al. 2011; Geiger et al. 2011).
Malone et al. also identified the goal, participant, process, and incentive as core organi-
zational genes (2009).
The advantage of crowdsourcing derives from the knowledge and experiences residing
in contributors that were previously unknown and unavailable (Bogers and West 2012;
Malone et al. 2009). Contributors provide varied perspectives and content, and work in a
divide-and-conquer format, supporting a wide and quick exploration of problems (Erickson
2011; Geiger et al. 2012). The variety of contributors also brings challenges. Crowd-
sourcing initiators must offer effective incentives, propose proper tasks, manage multiple
submissions of varying quality, and handle unpredictable actions (Malone et al. 2009; Jain
2010). Another challenge is that varied contributors usually participated individually at
different times during crowdsourcing. Crowdsourcing is thus an unsynchronized and dis-
tant collaborative process, which might reduce the quality of collaborative works.
Crowdsourcing supports design. The vast knowledge imported by crowds facilitates a
wide exploration of design ideas. Companies built communities to collect feedback and
creative ideas for their products (Bayus 2013); they also post design tasks and choose
solutions from crowd’s submissions. Design contest is the main form of crowdsourcing
design. Websites including Designcrowd (http://www.designcrowd.com/) and 99designs
(http://99designs.com/) hold multiple small-scale design contests, recruit contributors to
submit ideas, and reward the best. Apart from the best idea, other submitted ones provide
initiators with fresh insights and commitment toward the central design problem (Tidball
et al. 2011).
Researchers have employed various methods to support crowdsourcing design. Giving
direct feedback to contributors enhances the quality and number of submissions (Wooten
and Ulrich 2011). Dontcheva et al. (2011) recommended that work group support,
incentives align with desired behavior, appropriate wording, and monitor use of other
sources fostered creativity.
These methods enhanced the quality of ideas, but didn’t change the way of idea gen-
eration in design contests; contributors still work individually to generate and revise their
ideas. This individual work has two drawbacks. First, design requires problem exploration
and thorough consideration of alternatives, and so hastily generated ideas submitted for
relatively low chances of reward may not meet the quality standards required for a mature
design. Second, these forms of crowdsourcing design only collect the crowd’s final ideas.
Collaborative sketching in crowdsourcing design
123
Designers have their specific ‘‘lenses’’ to interpret the design task and refine ideas (Lehoux
et al. 2011). A collaborative crowdsourcing design method that aggregates these ‘‘lenses’’
should be more appropriate.
Studies on group design provide valuable references to crowdsourcing design. Memory
is an associative network, once a concept is activated, its closely related concepts are
retrieved more easily than weakly connected concepts (Raaijmakers and Shiffrin 1981).
The ideas proposed by a member in design group activated others’ weakly connected
concepts that were hard to retrieve, resulting in better design (Linsey et al. 2011). Group
design methods were developed accordingly. The method called C-sketch passes sketches
sequentially through the design group so that designers could add and modify others’ ideas
(Shah et al. 2001); Gallery Method sets times for group discussion of ideas (Pahl et al.
2007); Filter Mediated Design filters the modified part of design ideas and integrates them
in a cohesive way for further modification (Haymaker et al. 2000). These methods facilitate
the reinterpretation and improve the quality of ideas.
Group design methods suggest that integrating group members’ design processes is
helpful. However, the unsynchronized and distributed crowd in crowdsourcing design do
not meet the features of design groups. During collaborative crowdsourcing design, each
contributor design for a short time, and then leaves. Contributors face multiple ideas
generated by unfamiliar contributors from the beginning of the design process. They must
understand these ideas and find valuable information to support their explorations.
Moreover, contributors are not familiar with each other, their actions are unpredictable,
and the quality of their ideas are varied. Simply applying these group design methods in
crowdsourcing design process might result in a mess.
Nickerson and Sakamoto (2010) and Yu et al. (2011) have explored a collaborative
crowdsourcing design method that split the design process into basic steps. Using this
method, ideas are first submitted, and better ideas are chosen as stimulation and given to
another group, inducing next-generation ideas. Every step calls for an individual crowd-
sourcing process. This method improves design quality. Even when the next-generation
ideas are a simple combination of prior ideas, their quality scores still increase.
Another possible method is arranging crowd’s sketching processes effectively. Sketch
records designers’ train of thoughts and helps idea generation; crowd facing well-organized
sketches of prior contributors can interpret these sketches to understand ideas and generate
new ones. Moreover, well organized sketching processes enable crowds to continue
sketching from any stages of prior sketching process, which mitigate the effect of low
quality ideas. Sun et al. (2014) used a tree to organize the ideas and proved to be useful in
individual sketching. The branches on the tree represent the design paths during sketching
processes. This article proposed a method that employed this tree structure to arrange
crowd’s ideas and display the design paths, thus supporting collaborative sketching in
crowdsourcing design.
A method of collaborative sketching
The collaborative sketching method involved a sketching module and a task module
(Fig. 1). In sketching module, contributors studied the prior ideas organized as an idea tree,
sketched new ideas which were recognized as new nodes of the tree. Initiators published
their tasks in the task module; contributors searched through design tasks in this module,
downloaded the latest idea trees, and submitted their ideas back to this module. The task
module then integrated these ideas into the idea tree. During these processes, incentives
L. Sun et al.
123
encouraged contributors to participate in tasks and develop ideas cooperatively. A col-
laborative sketching demonstration system, named Sketchfans, was then developed fol-
lowing the framework of this method.
Sketching module
The sketching module supported contributors’ idea generation processes and arranged their
ideas into an idea tree. The display of the idea tree was optimized to highlight inspirational
ideas.
Idea tree
An idea tree that showed the connections among ideas was developed referring to the tree
structure proposed by Sun et al. (2014). The first node on the tree is a description of the
design task, and following nodes are ideas that present solutions. Every branch is an
evolving path on which following ideas are developed from prior ones. The tree grows via
the sketching process, with branches extending as contributors develop ideas on the
branches. When designers develop new ideas that solve design tasks from new perspec-
tives, new branches linked to the task description node are established. Therefore, an idea
tree sorts piles of ideas into several branches that clearly display the evolution process.
Subsequent contributors could then choose idea nodes on one branch and scan sketches
drawn during the growth of this branch, which might enhance their understanding of ideas.
The sketching module generated the idea tree from contributors’ sketching processes.
An algorithm has been proposed that recognizes creative ideas (Sun et al. 2014), but the
unpredictability of the sketching activities in crowdsourcing results in low recognition
precision. The sketching module thus encouraged contributors to mark their creative ideas
after contributors have sketched them. Explorations that had not generated an idea formed
temporary nodes on the idea tree.
A typical sketching process for a task is shown in Fig. 2. The task was to design
communal facilities for elderly people’s recreation, and this had produced a tree containing
twelve ideas. During this process, the contributor scanned three branches involving cal-
ligraphy (idea 1), recreational projection (idea 2, idea 3, idea 5, idea 7, idea 8, and idea 9),
and a treadmill (idea 4, idea 6, idea 10, idea 11, and idea 12). When choosing idea 9, which
used funny shadows, the contributor found a new way of producing shadows with sunlight
(idea 13). Idea 1, based on writing brushes, also attracted his attention, inducing the floor of
calligraphy (idea 14); the elderly could teach their grandchildren to write while strolling.
Next, the contributor explored idea 14 for a while (temporary node), checked the treadmill
branch back and forth, and chose idea 11 to develop the cooperative treadmill into a
competitive one (idea 15). Idea 11 then have two developments.
Fig. 1 Collaborative sketching method. (Color figure online)
Collaborative sketching in crowdsourcing design
123
Display optimization of idea tree
Crowdsourcing sketching often collects lots of ideas while contributors may only scan a
few, and so there is a large probability that valuable ideas will be neglected. Therefore, two
rules were set to optimize the display of the idea tree accordingly.
The first rule is ‘‘Ideas having multiple following ideas are enlarged.’’ This rule pre-
supposes that those ideas that have induced multiple following ideas are inspirational and
easily stimulate new thoughts in the continuing collaborative sketching process. The
second rule is ‘‘Ideas on long branches are enlarged.’’ This rule emphasizes the evolution
of ideas; those on long branches have been widely endorsed and consistently enhanced,
making them more likely to trigger new and better ideas.
The size of the idea nodes is adjusted according to these two rules. Ideas having
multiple following ideas and those on longer branches are bigger. We use a weight to
balance the effect of the two rules. The idea tree in Fig. 2 could thus be optimized as
shown in Fig. 3.
Task module
The task module displayed tasks and integrated the ideas that had been submitted. During
collaborative sketching, multiple contributors downloaded the same idea tree, and then
Fig. 2 Idea tree grows during a contributor’s sketching process (numbers display the sequences of ideas,grey rectangle represents a temporary node, and orange borders mark the ideas sketched by this contributor.Ideas 3, 5, 6, 7, 8, and 10 are omitted). (Color figure online)
L. Sun et al.
123
submitted their ideas at different times. All new idea nodes and branches were linked to
prior nodes. The task module thus received new idea nodes and corresponding sketches,
and linked these new nodes to the idea tree. In this way, contributors could view the idea
tree with all submitted ideas.
Sketchfans
This study developed Sketchfans for the iPad. Compared with a professional sketching
environment involving a computer and tablet, the iPad has a wide range of users, and offers
portable hardware that allows contributors to sketch freely. This demonstration system
contained an online task module and an offline sketching module. The task module dis-
played thumbnails of design tasks and idea trees, and the sketching module provided basic
sketching functions, arranged contributors’ ideas into idea trees, and uploaded new ideas to
the task module. The default size of the biggest idea node on the tree is set to 25 9 25
pixels to display the full structure of the tree, and it was possible to zoom in or out to view
details of the tree.
Contributors used Sketchfans to participate in collaborative sketching. As shown in
Fig. 4a, b, a contributor could check the tasks in which he had participated and search for
other tasks. In this example, he chose the already-participated in task to examine the latest
Fig. 3 Display optimization of the idea tree in Fig. 2. (Color figure online)
Collaborative sketching in crowdsourcing design
123
idea tree (Fig. 4c). By zooming in on the idea tree and clicking relevant nodes, the
contributor selected some exciting ideas and downloaded the tree to work on improving
them. Using the sketching module of Sketchfans, the contributor clicked idea nodes that he
was interested in, sketched new ideas, and pressed the ‘‘add’’ button to mark these ideas
(Fig. 5). The sketching module then linked these marked ideas as new nodes in the idea
tree. After sketching, the contributor submitted the updated tree to the task module by
clicking the ‘‘upload’’ button.
Initiators also used Sketchfans to publish collaborative sketching tasks. An initiator
clicked the ‘‘add’’ button in the sketching module to build a new idea tree for the design
task, described the task in the first node, and uploaded this one-node tree. The new task was
then published in the task module to await others’ participation.
A test of collaborative sketching method
Collaborative sketching integrates crowd’s thoughts and perspectives, which might be
beneficial to the generation of ideas. This section reports a collaborative sketching
experiment, and evaluates the generated ideas to further explore the effectiveness of col-
laborative sketching in crowdsourcing design.
Hypotheses
The first hypothesis is that the originality and the usefulness of ideas increase during
collaborative sketching. Subsequent participants scan a lot of ideas and generate more
original and useful ideas.
The second hypothesis is that the originality of ideas increases during the growth of
branches on the tree. Following contributors retrieve weak connected concepts when
interpreting the prior ideas proposed by ‘‘strangers,’’ thus producing more original ideas.
The third hypothesis is that the best ideas appeared near the end of branches. The
originality and the usefulness of ideas improve with the modifications. Therefore, ideas
near the end of long branches would be better.
The fourth hypothesis is that the ideas having multiple following ideas and ideas
inducing long branches are of higher originality. Participants choose inspirational ideas and
develop them. Therefore, these ideas are more original than ideas with only one
development.
Fig. 4 Interface of task module: a tasks in which the contributor has participated, b all tasks, and c previewof the idea tree. (Color figure online)
L. Sun et al.
123
Participants and procedure
A sketching experiment was conducted. Twenty-three participants (14 female, 9 male)
involved in this experiment. Among these participants, twenty students had an average of
3.6 years’ learning experience in industrial design, one had learned marketing for 3 years,
one had learned environmental design for 3 years, and one had worked as an industrial
designer for 2 years. People in crowdsourcing are usually physically separated and do not
know each other. To simulate this, participants in this experiment came and completed the
task separately, and did not know each other during the whole experiment. Seven of the 23
participants were randomly chosen, and sketched twice at different times to simulate the
repeated participation during actual crowdsourcing design process. The others only sket-
ched once during the experiment. Participants used the new iPad and a stylus to draw
sketches (Fig. 6). They were instructed to design communal facilities for elderly people’s
recreation, among which the best idea would be rewarded with 100 yuan (approximately
US$16), and ideas on the path to the best idea would share 50 yuan. Participants in design
experiments usually got 15–20 yuan, and so the much higher rewards of this experiment
ensured participants’ substantial effort.
The entire experiment consisted of three stages:
1. Introduction of sketching process. The participant was introduced to the experiment,
and allowed to familiarize himself/herself with the Sketchfans software and iPad.
2. Individual sketching process. The participant first downloaded and examined the idea
tree. Initially, the idea tree only contained one node that described the design task.
After examination, the participant was instructed to sketch three further ideas using
Sketchfans. She/he pressed a button each time she/he finished sketching an idea to add
this idea on the idea tree. The scanning and sketching time was not limited. The
participant uploaded the three new ideas when finished.
Fig. 5 Interface of sketching module (left column contains functional buttons, right column displays theidea tree). (Color figure online)
Collaborative sketching in crowdsourcing design
123
3. Idea explanation. After sketching, the participant explained her/his ideas and those that
she/he chose to develop, reported any obstacles during the process, and gave some
suggestions about Sketchfans. These reports were transcribed, and suggestions were
categorized.
Data analysis
The ideas participants sketched are numbered according to their chronological order. The
majority of participants spent less than 20 min to scan the idea tree and less than 20 min to
sketch three ideas. Participants generated 90 ideas and produced a tree with 20 branches.
The number of ideas is typical for design contests hold by crowdsourcing websites like
99designs and CrowdSpring (Wooten and Ulrich 2011).
Each idea on a branch changes parts of the prior idea. An idea consists of four hier-
archies involving the goal, function, structure, and detail. The goals of ideas refer to the
benefits they can bring to people, the functions refer to what the facilities do to meet goals,
the structures refer to what the facilities use to implement functions, and the details refer to
components of structures. Ideas having new goals built new branches. The developments of
ideas on a branch were categorized into three categories: physical principles, working
principles, and embodiment according to the article of Shah et al. (2003). Physical prin-
ciple developments refer to new functions used to satisfy one goal, for example, using
shadows (function: produce shadows) rather than videos to guide exercise (goal: guide
exercise). Working principle developments are new structures used to satisfy one function,
for instance, producing shadows (function: produce shadows) using the sun (structure:
hollowed plate) rather than a lamp. Embodiment developments refer to new details, such as
new figures of shadows (detail: shape of the hollow). The idea 15 in Fig. 2 is thus working
principle development because it uses a new structure, i.e., the opposite treadmill, to meet
the run function. Two researchers with 4 years’ learning experience in industrial design got
familiar with the categories, coded these developments individually, and settled eight
inconsistencies by discussion.
Fig. 6 Participant sketching on an iPad. (Color figure online)
L. Sun et al.
123
Sketch quality influences the perception of idea quality (Kudrowitz et al. 2012). Par-
ticipants’ varied sketching skills might influence the idea evaluation. To analyze the
changes in idea originality and usefulness more accurately, we exclude such effects by
redrawing all sketches. A graduate student with 4 years’ industrial design experience
redraws all ideas after all sketching processes. He copied these sketches and smoothed the
lines according to participants’ explanation of their ideas. As shown in Fig. 7, the graduate
student connected the detached lines of the chair while keeping the relative location of
these lines. Therefore, the re-drawing adjusted the qualities of two left sketches in Fig. 7 to
the same and did not change the ideas.
Existing methods of idea evaluation mainly focus on two aspects: whether the idea is
unusual and original, and whether the idea could be implemented and have practical
benefits (Nelson et al. 2009; Dean et al. 2006; Oman et al. 2013; Kudrowitz and Wallace
2013). We thus evaluated the originality and the usefulness of ideas, and selected the best
ideas according to the sum of these two averaged scores. The original ideas are those that
are unusual and rare. The useful ideas are those actually solve problems and are easy to
implement. For example, the idea involving four balls and four holes offers new recreation
method (the lower pictures in Fig. 7), but whether the elderly people likes it, and whether it
is easy to maintain in public space are questionable. Therefore, it would get a high orig-
inality score and a medium usefulness score. A relaxing chair is easy to implement and
offer a place to sit on, but there have been plenty of similar relaxing chairs; thus it is
unoriginal but useful.
Shah et al. evaluated idea originality based on the frequency of ideas; the more times
one idea was proposed by participants, the less original the idea was (2003). However,
during the collaborative sketching experiment, one original idea proposed by a participant
might be appreciated and developed by subsequent participants, thus lowering the origi-
nality score of this idea. Therefore, this article employed judges to evaluate these ideas. Six
graduate students with 4.8 years’ industrial design experience participate in idea evalua-
tion. These ideas are randomly sorted and evaluated in terms of their usefulness and their
originality using five-point scales, where five denoted highly useful and very original, and
one represented useless and unoriginal. The inter-rater agreement is acceptable with a
Kendall’s W of 0.623 for originality score and 0.665 for usefulness score. The originality
score and the usefulness score of each idea are then averaged, and the best idea and ideas
along the path to this idea are rewarded.
Results
The distribution of ideas on the tree was polarized (Fig. 8). Ten branches had only one idea
each, whereas the longest branch had 14 ideas. When viewing the tree, participants tended
to develop ideas on longer branches, and they also attended to other inspirational ideas.
The first idea proposed in the experiment was not developed any further until the 33rd idea.
Another branch containing 12 ideas was built in the middle of experiment, and grew fairly
quickly.
Participants could build a new branch and develop the existing branches during the
experiment. The proportion of ideas that build new branches was not affected by time.
Twelve of the first half of the 90 ideas built new branches, and eight of the second half built
new branches.
Participants employed more embodiment developments and less working principles in
the second half of 90 ideas (v2 = 8.34, p = 0.015). Seven ideas in the first half of 90 ideas
Collaborative sketching in crowdsourcing design
123
employed embodiment developments, while the amount is eighteen in the second half of
90 ideas. The proportion of ideas with physical principles reminded the same.
Originality and usefulness of ideas
The ideas received a mean score of 2.74 (SD = 0.59) for originality and 2.84 (SD = 0.61)
for usefulness. The best idea received an overall score of 8 (originality score = 3.5,
Fig. 7 Examples of the sketches before and after re-drawing. (Color figure online)
Fig. 8 Distribution of thenumber of idea nodes on abranch. (Color figure online)
L. Sun et al.
123
usefulness score = 4.5), and the worst received only 3.83 (originality score = 2.33, use-
fulness score = 1.5). Of the 90 ideas, the latter 45 were more useful than the former
(t = 2.06, p = 0.043). Their originality scores were not significantly higher.
The originality and usefulness of ideas on the first half of a branch were compared with
those on the second half. Each half of a branch had half the quantity of ideas. Ideas on the
second halves were more original than those on the first (Z = 3.99, p = 0.003), with an
average originality score of 2.96 compared to 2.50. The average usefulness of ideas on
each half did not differ. The change of originality and usefulness scores of ideas as the
branches grew was further analyzed. As shown in Fig. 9, the originality scores increase,
whereas the usefulness scores reach their peak around the 5th and 9th idea, and then
decrease.
The ideas were studied to determine the reasons for this change in usefulness scores.
Developments before the middle of a branch concentrated on fixing flaws with the initial
ideas, with typical examples being ‘‘ergonomic support for the elderly’’ and ‘‘circular table
with chessboard for multiple people.’’ On the second half of branches, developments
brought in new functions and features, such as ‘‘running competition’’ and ‘‘vertical
chessboard.’’ Participants explored new possibilities after improving the initial functions,
thus scoring lower for usefulness and higher for originality. This could also be confirmed
by the 13th idea, where further fixation on new ideas improved the usefulness score
(Fig. 9). Therefore, usefulness fluctuated, whereas originality increased.
The best ideas and the worst ideas
The features of the four best ideas, the four most original ideas, and the four most useful
ideas were analyzed. These ideas appeared in long branches after several rounds of
development (Fig. 10). The left best idea that appeared in the second branch from the left
was proposed late in the experiment (the 81st of the 90 ideas), which might have not
enough time to be developed. This article also analyzed the features of the four worst ideas,
the four least original ideas, and the four least useful ideas. The worst ideas appeared on
long branches while they appeared during the early developments. Ideas that were not quite
good attracted lots of developments. It was surprising that one branch could have both the
best ideas and the worst ideas. The third branch from the left had three least original ideas
and two most original ideas. The second branch from the right had one best idea and one
worst idea, and one most useful idea and one least useful idea.
Features of ideas inducing long branches and those having multiple following ideas
The originality and usefulness scores of the first ideas on long branches did not differ from
those on short branches (originality: t = -0.96, p = 0.351; usefulness: t = -0.11,
p = 0.916). Ideas on long branches had the same originality and usefulness as those on
short branches (originality: t = -0.64, p = 0.528; usefulness: t = 0.32, p = 0.754).
However, the ideas on long branches had more varied originality scores and more stable
usefulness scores than those on short branches. They had a standard deviation of 0.61 on
originality and 0.58 on usefulness, while the ideas on short branches had a standard
deviation of 0.54 on originality and 0.74 on usefulness.
Participants did not choose high-quality ideas to develop. Rather, they chose to develop
ideas that were not so good. Ideas having multiple (more than two) following ideas had the
same usefulness as other ideas, and slightly lower originality (t = -1.78, p = 0.080).
Collaborative sketching in crowdsourcing design
123
They were followed by larger modifications than ideas with few following ideas
(v2 = 9.73, p = 0.008). Indeed, 17 out of 38 following ideas made physical principle
developments, whereas for ideas having one or two following ideas, only five out of 32
following ideas were made on physical principles.
Usability of Sketchfans
All participants successfully finished the sketching task and reported that the other sketches
were helpful. As sketches involved both text and drawings, participants could easily report
the exact meaning of the ideas that they chose to develop. They reported that the tree
structure assisted the understanding of ideas and stimulated new thoughts, but also men-
tioned that some sketches were messy and difficult to understand. Participants explored the
idea tree when searching for inspiration. They would generally view ideas, and relied
heavily on the thumbnails on the idea tree to search for what they wanted. Therefore, a
large proportion of suggestions were related to the display of the idea tree. Ten participants
felt that the default size of the idea tree was too small to view thumbnails and click on idea
nodes, and six participants had difficulty finding ideas that they had scanned before. Other
Fig. 9 Originality andusefulness of ideas during thegrowth of branches. (Color figureonline)
Fig. 10 Positions of the ideas on branches (other branches have been omitted. The lower an idea on abranch, the later it was proposed during the sketching process). (Color figure online)
L. Sun et al.
123
suggestions called for a more fluid loading of the idea tree and a clearer understanding of
others’ ideas.
Discussion
Collaborative sketching allows a balanced exploration of design tasks. Participants propose
a broad range of ideas, and develop several of them thoroughly. They easily examine and
develop ideas on long branches, without fixating on them. Among the seven branches that
had more than five ideas, two contain only one idea until the middle of experiment.
Participants collaborate well during the exploration. Even though they can not contact each
other, participants implement prior ideas step-by-step and developed some great ideas.
Test of collaborative sketching method
The collaborative sketching process method is effective. Hypothesis one and hypothesis
two are partly supported by the results. Subsequent ideas tend to be more useful than prior
ideas during collaborative sketching; while they have the same originality. Originality
constantly increases along the branches, and usefulness increases when participants fix
flaws in the prior ideas on the branches. Collaborative sketching performs two kinds of idea
improvements. Scanning many ideas enables a thorough consideration of the design tasks,
producing more useful ideas; while for ideas on the same branch, constant exploration on
the design path supports further breakthroughs and induces more original ideas modified
from prior solutions.
Hypothesis three is supported by the results. During collaborative sketching, high-
quality ideas still come from the modification rather than a burst of inspiration. The best
ideas appear on the end of branches and the worst ideas appear on the former part of
branches. Participants might come up with all sorts of ideas at the beginning of a branch,
but following developments effectively enhance these ideas, as shown in Fig. 10, even the
best ideas could be produced from the worst ones.
Hypothesis four is contrary to the results. It is interesting that participants are not
inspired by highly original ideas, and would rather develop ideas of lower originality. Two
factors may contribute to this. First, because collaborative sketching focuses on the con-
ceptual stage of design, the original ideas might be too original for other participants to
build a connection from these concepts to their own knowledge. Therefore, participants
will find it hard to generate new ideas from these original ideas. Second, participants know
little about each other during collaborative sketching. They only scan others’ ideas for a
limited time, and have no opportunity for group discussion or debate. The time duration is
too short for an unfamiliar original idea to be fully understood and stimulate new thoughts.
The less original ideas, on the contrary, are ‘‘usable new thoughts’’ that offer a starting
point for participants’ development. Originality enhancements are achieved by exploring
from a familiar zone, i.e., significant developments of less original ideas, rather than
developing original ideas in a new zone.
Sketchfans supports the collaborative sketching process. Participants are easily able to
find thumbnails of the experimental task, understand relations among ideas on the idea tree,
and click on idea nodes. The idea tree enables more functions than we first thought. It not
only displays relations among ideas, but also acts as a quick index that helps participants
search and recall the ideas they have viewed. Another notable feature is that participants
Collaborative sketching in crowdsourcing design
123
prefer to choose clear ideas to develop. As Sketchfans only offers free drawing function,
ideas produced by participants with relatively poor drawing skills might be ignored, nar-
rowing the scope of suitable contributors.
This study tests the effectiveness of the collaborative sketching method in crowd-
sourcing design, rather than the effects of collaborative sketching components. Thus, two
minor components of collaborative sketching were adjusted to observe the process more
clearly. Each participant sketched three ideas; participants sketched in a controlled envi-
ronment for this experiment, rather than in their familiar workspace. At the same time, we
kept the key features of collaborative sketching: participants were recruited and rewarded;
they sketched on others’ sketches and did not know the identities of other participants; their
sketching activities and the time spent on sketching were not restricted, and the number of
ideas was typical for a crowdsourcing design contest. These key features ensure that the
experiment is a reliable simulation of the real collaborative sketching process.
Optimization of sketchfans
Participants frequently used the idea tree during sketching. The display of the idea tree was
modified and new functions were added. As participants examined ideas on branches,
Fig. 11 Participants could mark one idea for later use: a the button to mark an idea, b the marked idea, andc the button to unmark an idea. (Color figure online)
Fig. 12 Sketching interface with marked ideas. (Color figure online)
L. Sun et al.
123
rather than the entire structure of the idea tree, we enlarged the default size of the idea
nodes and shortened the distance between them to make it easier to preview and click.
Following a brief test using three participants, the default node size was set to 40 9 40
pixels, and the distance between the centers of two adjacent ideas was set to 60 pixels.
A ‘‘favorite’’ function was added to the idea tree. This function marks ideas for later
use. As shown in Fig. 11, when an idea node was pressed for 3 s, a button would pop up to
turn this idea node into a circle. Circles among squares clearly display participants’
favorite ideas on the idea tree (Fig. 12), offering a quick search and recall experience.
Participants could only see their own marks during the sketching process. These marked
idea nodes denote those that caught the participant’s interest during the sketching process,
and offer valuable data for adjusting the size of the idea nodes.
Conclusion
This article proposes a new crowdsourcing design method that recruits crowds to collab-
oratively sketch for design tasks. This method arranges contributors’ ideas in an idea tree,
showing their relations and helping subsequent contributors to develop those ideas. A
demonstration system named Sketchfans is developed, and this method is examined
experimentally. Finally, the demonstration system is optimized according to participants’
sketching activities and feedback.
The experimental results reveal that the collaborative sketching method is effective in
enhancing the quality of contributors’ ideas. Subsequent ideas during the sketching pro-
cesses are more useful than prior ideas, and ideas on the second half of branches got higher
originality scores than those on the first half of branches. The best ideas appear around the
end of branches, and some of them are even developed from the worst ideas. Participants
make balanced explorations. They largely develop familiar ideas and gradually implement
ideas during collaborative sketching. This method supports the participants’ sketching
well; participants seldom feel hindered, and rely heavily on the idea tree for inspiration.
The method also displays some shortcomings. Insufficient communication and discus-
sion make it hard to design for tasks that have complex constraints and details. Since
participants collaborate in this method and interactions between participants are significant
for collaboration, a module that supports unsynchronized communication should be
included in this method. This article presents an initial exploration of collaborative
sketching in crowdsourcing design. Further studies will optimize the arrangement and
display of ideas to improve design quality, and develop a collaborative sketching system
that enables communication and discussion.
Acknowledgments This paper is supported by the National Natural Science Foundation of China(61004116) and the Zhejiang Provincial Natural Science Foundation of China (LY13E050005,LY13F030002).
References
Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analysis of the Dell IdeaStormcommunity. Management Science, 59(1), 226–244.
Bogers, M., & West, J. (2012). Managing distributed innovation: Strategic utilization of open and userinnovation. Creativity and Innovation Management, 21(1), 61–75.
Cross, N. (2006). Designerly ways of knowing (1st ed.). London: Springer.
Collaborative sketching in crowdsourcing design
123
Dean, D. L., Hender, J. M., Rodgers, T. L., & Santanen, E. L. (2006). Identifying quality, novel, and creativeIdeas: Constructs and scales for idea evaluation. Journal of the Association for Information Systems,7(10), 646–698.
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web.Communications of the ACM, 54(4), 86–96.
Dontcheva, M., Gerber, E., & Lewis, S. (2011). Crowdsourcing and creativity. In The ACM CHI conferenceon human factors in computing systems: Workshop on crowdsourcing and human computation,Vancouver, Canada, May 7–12 2011.
Erickson, T. (2011) Some thoughts on a framework for crowdsourcing. In The ACM CHI conference onhuman factors in computing systems: Workshop on crowdsourcing and human computation, Van-couver, Canada, May 7–12 2011.
Geiger, D., Rosemann, M., Fielt, E., & Schader, M. (2012) Crowdsourcing information systems-definition,typology, and design. In International conference on information systems, Orlando, USA December16–19 2012 (Paper 53).
Geiger, D., Seedorf, S., Schulze, T., Nickerson, R. C., & Schader, M. (2011) Managing the crowd: Towardsa taxonomy of crowdsourcing processes. In Americas conference on information systems, Detroit,Michigan, August 4–8 2011 (Paper 430).
Goldschmidt, G. (1997). Capturing indeterminism: Representation in the design problem space. DesignStudies, 18(4), 441–455. doi:10.1016/S0142-694X(97)00011-2.
Haymaker, J., Keel, P., Ackermann, E., & Porter, W. (2000). Filter mediated design: Generating coherencein collaborative design. Design Studies, 21(2), 205–220.
Howe, J. (2006a). Crowdsourcing: A definition. Crowdsourcing: Tracking the rise of the amateur. http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html.
Howe, J. (2006b). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.Jain, R. (2010). Investigation of governance mechanisms for crowdsourcing initiatives. In Americas con-
ference on information systems, Lima, Peru, August 12–15 2010 (Paper 557).Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., et al. (2013). The future of
crowd work. Paper presented at the the 16th ACM conference on computer supported cooperative workand social computing, San Antonio, USA, February 23–27.
Kudrowitz, B., Te, P., & Wallace, D. (2012). The influence of sketch quality on perception of product-ideacreativity. AI EDAM, 26(Special Issue 03), 267–279. doi:10.1017/S0890060412000145.
Kudrowitz, B., & Wallace, D. (2013). Assessing the quality of ideas from prolific, early-stage productideation. Journal of Engineering Design, 24(2), 120–139.
Lehoux, P., Hivon, M., Williams-Jones, B., & Urbach, D. (2011). The worlds and modalities of engagementof design participants: A qualitative case study of three medical innovations. Design Studies, 32(4),313–332.
Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., & Markman, A. B. (2011). Anexperimental study of group idea generation techniques: Understanding the roles of idea representationand viewing methods. Journal of Mechanical Design, 133(3). doi:10.1115/1.4003498.
Malone, T., Laubacher, R., & Dellarocas, C. (2009). Harnessing crowds: Mapping the genome of collectiveintelligence. Research Paper No. 4732-09, MIT.
Morris, R. R., Dontcheva, M., Finkelstein, A., & Gerber, E. (2013). Affect and creative performance oncrowdsourcing platforms. In Affective computing and intelligent interaction (ACII), 2013 humaineassociation conference on, 2013 (pp. 67–72), IEEE.
Nelson, B. A., Wilson, J. O., Rosen, D., & Yen, J. (2009). Refined metrics for measuring ideation effec-tiveness. Design Studies, 30(6), 737–743. doi:10.1016/j.destud.2009.07.002.
Nickerson, J., & Sakamoto, Y. (2010). Crowdsourcing creativity: Combining ideas in networks. In Work-shop on information in networks, New York, USA, September 24–25 2010.
Oman, S. K., Tumer, I. Y., Wood, K., & Seepersad, C. (2013). A comparison of creativity and innovationmetrics and sample validation through in-class design projects. Research in Engineering Design, 24(1),65–92.
Pache, M. W. (2005). Sketching for conceptual design. Empirical results and future tools. Munich: Tech-nischen Universitat Munchen.
Pahl, G., Wallace, K., & Blessing, L. T. M. (2007). Engineering design: A systematic approach. Berlin:Springer.
Pei, E. (2009). Building a common language of design representations for industrial designers & engi-neering designers. Loughborough: Loughborough University.
Raaijmakers, J. G., & Shiffrin, R. M. (1981). Search of associative memory. Psychological Review, 88(2),93–134. doi:10.1037/0033-295X.88.2.93.
L. Sun et al.
123
Restrepo, J., & Christiaans, H. (2004). Problem structuring and information access in design. Journal ofDesign Research, 4(2), 1551–1569.
Shah, J. J., Smith, S. M., & Vargas-Hernandez, N. (2003). Metrics for measuring ideation effectiveness.Design Studies, 24(2), 111–134. doi:10.1016/S0142-694X(02)00034-0.
Shah, J. J., Vargas-Hernandez, N., Summers, J. D., & Kulkarni, S. (2001). Collaborative sketching (C-Sketch)—An idea generation technique for engineering design. The Journal of Creative Behavior,35(3), 168–198.
Sim, S. K., & Duffy, A. H. (2003). Towards an ontology of generic engineering design activities. Researchin Engineering Design, 14(4), 200–223.
Sun, L., Xiang, W., Chai, C., Wang, C., & Huang, Q. (2014). Creative segment: A descriptive theory appliedto computer-aided sketching. Design Studies, 35(1), 54–79. doi:10.1016/j.destud.2013.10.003.
Suwa, M., Gero, J. S., & Purcell, T. A. (1998). The roles of sketches in early conceptual design processes InProceedings of the twentieth annual conference of the Cognitive Science Society.
Suwa, M., Gero, J., & Purcell, T. (2000). Unexpected discoveries and S-invention of design requirements:Important vehicles for a design process. Design Studies, 21(6), 539–567.
Taura, T., Yamamoto, E., Fasiha, M. Y. N., Goka, M., Mukai, F., Nagai, Y., et al. (2012). Constructivesimulation of creative concept generation process in design: A research method for difficult-to-observedesign-thinking processes. Journal of Engineering Design, 23(4), 297–321.
Temple, S. (1994). Thought made visible: The value of sketching. Co-Design Journal, 1, 16–25.Tidball, B., Mulder, I., & Stappers, P. J. (2011). Online design contests: A network of inspiration for
designers. Paper presented at the proceedings of the 4th world conference on design research, IASDR,2011, Delft, The Netherlands, October 31–November 4.
Wooten, J., & Ulrich, K. (2011). Idea generation and the role of feedback: Evidence from field experimentswith innovation tournaments. Available at SSRN 1838733.
Yu, L., & Nickerson, J. V. (2011). Cooks or cobblers? Crowd creativity through combination. In Pro-ceedings of the SIGCHI conference on human factors in computing systems, 2011 (pp. 1393–1402),ACM.
Collaborative sketching in crowdsourcing design
123