Transcript
Page 1: Analysing participation in collaborative design environments

The proliferation of computer media and networking has the poten-

tial to make fundamental changes in the methods, models and tech-

niques employed to educate and train design students and pro-

fessionals. The easy access to the internet requires a reconsideration of

the desktop Computer Aided Instruction and Computer Aided Learning

(CAT/CAL) paradigms. Analyzing the role and impact of computers in the

on-going changes in education, Lockard, Abrams and Many1 specify two

types of CAI/CAL applications. Applications classified as Type I employ

1 Lockard, J, Abrams P D,and Many, W A Microcomputersfor twenty-first century educatorsAddison-Wesley, Reading, MA,USA (1994)

www.elsevier.com/locate/destud0142-694X/00 $ - see front matterDesign Studies21 (2000) 119–144PII: S0142-694X(99)00043-5 119 2000 Elsevier Science Ltd All rights reserved Printed in Great Britain

Analysing participation incollaborative design environments

Simeon J. Simoff and Mary Lou Maher, Key Centre of DesignComputing and Cognition, University of Sydney, NSW-2006, Australia

Computer-supported collaborative design can be realised by a broadrange of collaborative environments, each facilitating a different kind ofcollaboration. Understanding the style of collaboration and the potentialfor each environment is important when choosing a particulartechnology. We have developed a virtual world approach to teachingdesign computing in which students learn through traditional lectures,online seminars, and collaborative design projects. The environmentintegrates both synchronous and asynchronous communication as wellas shared documentation. One side effect of using this environment isthe incremental development of a record of the communication andcollaboration. This record can be the basis for the analysis ofparticipation in collaboration. We show how text analysis as a part ofdata mining can be used to analyse different aspects of participation.Specifically, we analyse participation in synchronous communication toevaluate individual contribution. We then analyse asynchronouscommunication to evaluate the extent of collaboration. The methodspresented can be an automated part of the collaborative environmentproviding information for student evaluation in an educationalenvironment or individual contribution in a professional environment. 2000 Elsevier Science Ltd. All rights reserved

Keywords: collaborative design, computer supported design, designeducation, data mining

Page 2: Analysing participation in collaborative design environments

2 Streibel, M J ‘Instructionalplans and situated learning: thechallenge of Suchman’s theoryof situated action for instructionaldesigners and instructional sys-tems’ in Anglin, G J (ed) Instruc-tional technology, past, present,and future Libraries Unlimited,Englewood, CO, USA (1995) pp117–1323 Savery, J R and Duffy, T M‘Problem based learning: aninstructional model and its con-structivist framework’Educational Technology Vol 35No 5 (1995) pp 31–384 Mitchell, W J and McCul-lough, M Digital design mediaVan Nostrand Reinhold, NewYork, USA (1995) pp 441–462

computing resources to do things educators have previously done withoutcomputers. These applications improve such aspects of teaching like prep-aration of teaching materials and student management. Generally, thisapproach does not change the teaching strategies and schemata, thus TypeI automation reduces the technical efforts but does not result in more effec-tive teaching. The other group of applications, labeled as Type II are ori-ented towards bringing teaching and learning methods and experiencesimpossible without computers. We explore this type of environment andits use in design education and the profession.

Most of the early Web-mediated on-line environments deliver variousmultimedia learning materials and give students access to an enormousresource of information. However, the on-line materials alone are not suf-ficient for evaluation and assessment since the instructors cannot trace stud-ent participation, in the sense of interaction with course materials, otherstudents, and tutors. The absence of an overall learning model, combinedwith restricted interaction and single direction of the information flow,show that these initial efforts were able to utilise only a small part fromthe potential of internet technologies. On-line learning methods can facili-tate three components of course administering — (i)course materialstobe delivered to the students; (ii)communicationbetween students and edu-cators, and (iii)management and assessmentof students. In this paperwe focus on automating the analysis of communication and collaboration,providing a methodology for analysing participation in any collaborativedesign environment.

1 Computer-supported collaborative designeducationDesign, as an educational subject, is characterised by the lack of clearseparation between theoretical knowledge and practical skills. In addition,design projects require intensive collaboration of numerous specialists. Theinstructional strategy in design studios is based on constructivist principlesin which a student actively constructs an internal representation of knowl-edge by interacting with the material to be learned. The model implementsthe principles of situated cognition2 and problem-based learning3. Accord-ing to these theories, both social and physical interaction enter into thedefinition of a problem and the construction of its solution. Neither theinformation to be learned, nor its symbolic description, is specified outsidethe process of inquiry and the conclusions that emerge from that process.

Mitchell and McCullough4 present the general idea of the virtual designstudio as an extension of conventional studios, discussing differentenabling technologies and corresponding studio scenarios. Maher, Simoff

120 Design Studies Vol 21 No 2 March 2000

Page 3: Analysing participation in collaborative design environments

5 Maher, M L, Simoff, S J andCicogniani, A ‘Potential andlimitations of virtual design stu-dio’. Interactive Construction On-line January al. (1997b)6 Simoff, S J and Maher, M L‘Design education via web-based virtual environments’ InAdams, T (ed), Computing inCivil Engineering Proceedings ofthe Fourth Congress of Comput-ing in Civil Engineering, ASCE,New York, USA (1997a) pp418–4257 Simoff, S J and Maher, M L‘Web-mediated courses: the rev-olution in on-line design edu-cation’ In Ashman, H, Thistle-waite, P, Debreceny, R andEllis, A (eds) Into the main-stream — the Web in AustraliaSouthern Cross UniversityPress, Lismore, Australia(1997b) pp 143–154

and Cicogniani5 developed these ideas into a Web-based Virtual DesignStudio (VDS) as a distributed environment for teaching students about col-laborative design. The modification of the workspace in the VDS and thereplacement of the physical space by a computer mediated educationalenvironment enhances the studio-style learning by releasing geographicaland time scheduling constraints.

Our experience in developing learning materials for virtual design studios6,7

show that the design and management of Web-mediated courses involvesmore than converting lectures and exercise notes to a collection of linkedweb pages or providing tools for communication. We identify two issuesin developing effective collaborative design environments:

I the need to create a sense of place and community among the parti-cipants in the studio and

I the need for interactive and/or automatic assessment.

We address these needs in the development of our Virtual Campus(http://www.arch.usyd.edu.au:7778/). This is an educational environmentwhich uses the metaphor of a physical campus as a collection of rooms toorganise the learning and community activities. Rooms in the virtual cam-pus are grouped according to their type of use, such as classroom, office,or conference room. The rooms provide some of the functions of a physicalroom, such as privacy, security, and a place for various activities. Theclassrooms are equipped with a slide projector, a recorder, whiteboard,notice board and other facilities that help the student with a particularlearning task. The basic communication mode is text-based, though thecampus has a Web-based interface for navigation through a 3D visualisa-tion of the facilities (using VRML).

The interface, shown in Figure 1, provides facilities for viewing the partof the campus where the person is located, various tools for using thecampus and a line input for communication, navigation, and creatingobjects. Figure 1 shows the practice studio for preparing a presentation.The room is accessible from the conference room. The top part of thewindow shows the room description, with a tool bar for looking at thewhiteboard, the slide projector, the course materials (created using WebCTas described below), and other tools for using the campus. The bottom partof the window shows the text description of the room and responds toinput from the user. The input can vary from commands to show who isin the room to communication commands such as “say”, “whisper”, and“emote”. This environment provides real time communication similar to achat window, with the additional feature of providing a permanent place

Analysing participation in collaborative design environments 121

Page 4: Analysing participation in collaborative design environments

8 Sutherland, J ‘The Java rev-olution’ Sun Expert Vol 8 No 1(1997) pp 43–54

to store information and other objects. This environment effectivelyaddresses the first need — to create a sense of place and community.

The second issue is the need to provide for interactive and automatedassessment. Since the students are not in a physical classroom, in-classtests do not make sense. There are several difficulties with implementinginteractive assessment within a Web-mediated course delivery mechanismin comparison to stand-alone CAI/CAL programs. The main disadvantagecomes from the batch mode of information exchange. This makes theimplementation of genuine, immediate and meaningful feedback problem-atic. Typically, a form is filled out then sent to the server which processesit and sends back another form or web page. Sutherland8 presents newarchitectures of enhanced Java/CGI implementations and Applet/Servletremote method invocations, which have the potential to overcome thisproblem of the client/server model. These innovations allow greater inter-

122 Design Studies Vol 21 No 2 March 2000

Figure 1 The Web interface to the Virtual Campus.

Page 5: Analysing participation in collaborative design environments

activity and permit more complex client operations. However, this is at thetechnological front end and, consequently, there is a lack of tools to sim-plify the implementation.

The response to this need is a new generation of Web-authoring environ-ments, which provide a set of tools for student communication and toolsfor course designers to create and structure courses. There are numerous“working” pioneers, among which WebCT(http://homebrewl.cs.ubc.ca/webct/webct.html), developed at the Univer-sity of British Columbia, has been successfully employed for design coursemanagement and delivery at the University of Sydney. Figure 2 illustratesWebCT in course design mode. These new environments include on-line

Analysing participation in collaborative design environments 123

Figure 2 WebCT: designing a Web-based course on Design Computing.

Page 6: Analysing participation in collaborative design environments

quizzes, student participation and progress tracking facilities and enhancedcourse management facilities. These features are mainly based on theanalysis and bar-graph visualisation of server statistics. They do not ana-lyse and assess student collaboration — an essential part of project-baseddesign courses.

The combination of the WebCT course design environment with the VirtualCampus provides a comprehensive environment for collaborative learning.The combined environment is illustrated in Figure 3. The WebCT courseserver includes a bulletin board system with archiving facility and the cam-pus classrooms provide a place for meetings, presentations, and seminars.Project-based courses, conducted in this environment, use a variety ofsynchronous and asynchronous computer-mediated teaching techniques.Transcripts of on-line subject discussions, conducted in the Virtual Campusand messages on bulletin boards are a rich source of data. We show how

124 Design Studies Vol 21 No 2 March 2000

Figure 3 Integrated Web-based design educational environment.

Page 7: Analysing participation in collaborative design environments

9 Fayyad, U M, Piatetsky-Shapiro, G and Smyth, P ‘Fromdata mining to knowledge dis-covery: An overview’. In Fayyad,U M, Piatetsky-Shapiro, G,Smyth, P and Uthurusamy, R(eds) Advances in knowledgediscovery and data mining, AAAIPress, Boston, MA, USA (1996)pp 1–3410 Chen, M S, Han, J and Yu,P S ‘Data mining: an overviewfrom a database perspective’Knowledge and Data Engineer-ing Vol 8 No 6 (1996) pp 866–883

text analysis and data mining methods can be used for the analysis andevaluation of student participation and collaboration.

2 Evaluating individual contribution using textanalysis and data miningData mining (DM), known also as knowledge discovery (KD), is the over-all process of examining a data source for implicit information and rec-ording this information in explicit form, in other words, the extraction ofa high-level knowledge from a low-level data. The overall process of datamining is shown in Figure 4. DM involves the identification of potentiallyuseful and understandable patterns in this data9, spanning the entire spec-trum from discovering information of which one has no knowledge towhere one merely confirms a well known fact. Data mining methods havebeen developed in machine learning, statistics, data visualization anddeductive databases to examine the content of large databases10.

Historically, data mining was initiated in large databases. The number ofrecords in such databases can be estimated in millions, and the informationis well-structured in table form according to a particular data model. How-ever, a vast amount of design knowledge is coded in machine-readable textform or as electronic dictionaries, manuals and references, as CAD draw-ings or digital images. Most of the design data is unstructured, in compari-son with the large databases used in conventional data mining. A character-istic of data mining and knowledge discovery is that the source data wascreated for purposes other than knowledge discovery. In our case, thesource data is the text transcripts from the seminar discussions in the vir-tual campus.

The on-line discussions in the virtual campus are based on the idea of acomputer-mediated moderated discussion. A computer-mediated moder-ated discussion can be held both in synchronous and asynchronous mode.Asynchronous discussions can be realised through a computer-mediated

Analysing participation in collaborative design environments 125

Figure 4 The data mining lifecycle.

Page 8: Analysing participation in collaborative design environments

bulletin board, a discussion list-server, or even by e-mail. Here we focuson synchronous seminar discussions taking place in the virtual campusclassroom. Each seminar is devoted to a particular theme as part of thecourse, complementary to lectures or course material, which students havepreviously attended or read. Figure 5 illustrates a virtual seminar scenario.

Discussions in the virtual campus have an initial start-up period, whenparticipants are getting comfortable. Then the discussion continues as in aphysical seminar, except the participation rate tends to be higher and itcan be more difficult to control the topic. The role of the instructor(s) isto prevent distractions and maintain the focus of the discussion. One of theinstructors (usually the specialist in the topic and the potential moderator inthe discussion) prepares a preliminary agenda, corresponding to the courseobjectives. Mentors also watch that all students have the opportunity toparticipate in the discussion.

Online seminars can form part of the student’s assessment by includingthe amount and content of the student’s participation. The virtual campusprovides a means for recording in explicit and descriptive form all activitiesduring the discussion in a format suitable for further quantitative and quali-tative analysis. Thus we have the ability to analyse students’ and instruc-tors’ contribution, providing feedback in both directions.

A virtual seminar can be represented as a sequence of activities, as shownin Figure 6. Each activity is described by an expression. An expression

126 Design Studies Vol 21 No 2 March 2000

Figure 5 Virtual seminar in virtual campus environment.

Page 9: Analysing participation in collaborative design environments

consists of a subject, who performs the action or the utterance in theactivity, a verb, which describes the action or utterance, an object towardswhich the action is directed and the content of the action. Using this for-malism we can represent, analyse and compare virtual seminarsimplemented through different underlying environments. In this paper weconsider only the virtual seminars implemented via text-based environ-ments. A raw excerpt from the transcript of a seminar discussion on cyber-space is shown in Figure 7.

Analysing participation in collaborative design environments 127

Figure 6 Formal represen-

tation of virtual seminar.

Figure 7 An unprocessed excerpt of a seminar discussion on cyberspace.

Page 10: Analysing participation in collaborative design environments

As a description of the sequence of activities during the seminar, the tran-script is a mixture of quantitative and qualitative data. Initially, we formatthe transcript in a way that each line corresponds to a single activity. Thensuch characteristics as the total number of lines, the number of lines, whichstart with the same subject, are examples of quantitative data. The conceptsthat constitute the content of each activity are an example of qualitativedata. These data, provided in the records of the virtual seminar, are a usefulsource for investigating the way the seminar has been run, students’ andmoderator’s participation, the appropriate discussion styles and other inter-esting patterns that will help to understand and better apply virtual seminarsin design education.

The raw seminar transcript may make sense to the participants in the dis-cussion and not to someone analysing the content. An observer would notunderstand the flow of the conversation in Figure 7, or who Charles is inFigure 8. Is Charles the moderator of the seminar, or an external expertinvited for a discussion on a particular topic? The names of the participantsdo not indicate their roles in the discussion.

Each transcript undergoes preliminary processing to create the correspond-ing data sets for statistical and content analyses, and a more meaningfulcopy of the transcript archived and used for further research and analysis.The preprocessing includes several stages. It starts with initial cleaning ofsome data headers and service information, and automatic numbering ofall lines in the remaining body of the transcript. On the one hand, thenumbering provides a reference label for each line, i.e. seminar activity,assuming that we have already done the formatting step discussed before.These numbers then allow us to refer to a particular part of the discussion.For instance, the line numbers in Figure 8 show that this is part of thebeginning of the seminar. On the other hand, the total number of linesgives us an idea about the length of the seminar discussion as a sequenceof activities, rather than as a time interval. Considering that the length ofthe seminar is restricted in time by the course schedule, the number oflines also provides some idea about how active the participants were inthe discussion.

128 Design Studies Vol 21 No 2 March 2000

Figure 8 Who is Charles?

Page 11: Analysing participation in collaborative design environments

The next step is connected with the line structure and content of the tran-script record. In this paper we consider only transcripts which consist ofnatural language expressions. Some expressions can include one or morecompleted sentences, some only part of a sentence. Following the modelin Figure 6, we split each expression into two parts. The subject, the verband the object are labeled as aleft-hand sideand the content remains inthe right-hand side. This is a conventional separation, merely for dis-tinguishing between descriptive and actual content data.

From the initial transcript we generate two additional data sets: one inwhich we modify only the left-hand side, that gives us a clear idea whois who through the whole seminar. We keep the right-hand side unchangedfor quantitative analysis. The second data set is derived from the first oneby normalising and coding the right-hand side in a way that both sidesbecome consistent. The data transformation process includes two basictechniques: reference normalisation and role coding techniques.

2.1 Reference normalisationWe identify all login names used by each participant and all referenceswithin the discussion for each participant. After the name reference normal-isation, each participant is represented by only one name spelling in theseminar. The case of reference normalisation within the content of thediscussion (the right-hand side) is illustrated in Figure 9, where the personparticipating under the name “davidS” (line 129) is referred in the contentof the discussion by another participant as “david” (line 132).

Another part of the normalisation process is converting, where necessary,the right-hand side of each expression to the structure “subject-verb-object”. For example, lines 4 and 7 in Figure 10 show that Tim talks tothe people in the room and then to a particular person, respectively. Bothlines are examples of the same activity:

Analysing participation in collaborative design environments 129

Figure 9 The name reference within the discussion can differ from the login name.

Page 12: Analysing participation in collaborative design environments

Line 4: Subject{tim} - Verb(says) -Object {} - Content(Yes, I guessso, but I’m not sure)Line 7: Subject{tim} - Verb () -Object(Prof) - Content(can we bringCharles in here?)

This activity is explicitly specified in line 4 by the verb “says”, when inline 7 the verb is omitted. The object in line 4 is the group of peoplein the seminar, thus it is also omitted. Formally, normalisation requiressubstitution of the missing components so that we have a uniform datastructure. Another type of normalisation is the substitution of a word fora set of separated symbols. Line 436 in Figure 10 is an example of sucha case, where we need to replace “•s O” with “thinks”. Normalisationalso includes the correction of typing errors.

Normalisation considerably improves the structure of the transcript andpartially its readability. However, it does not analyse the content of theseminar. The reader probably has noticed that the lines in Figure 8 are partof the excerpt presented in Figure 10. Yet, who is Charles?

130 Design Studies Vol 21 No 2 March 2000

Figure 10 Various constructs which require normalisation and role coding.

Page 13: Analysing participation in collaborative design environments

2.2 Role codingOnce the normalization is done, we code the reference to each participantwith their formal role. The aim of this operation is to obtain qualitativedata sets, which, on the one hand, are suitable for quantitative analysis,and, on the other hand, are readable and meaningful to researchers, anddo not violate the anonymity of the data source. We define two sets ofrole categories: basic and special.

The basic set encompasses the role categories, which are common for thecourse discussions. In the case of virtual seminars, which are part of aregular course program, the basic set of role categories includesstudent,moderator, instructor and expert. The last category accommodates externalprofessionals and world-known experts, both from industry and academiathat could participate in the virtual seminar. Each participant is assigneda name according to the following format: “Category Number”, forexample, “Instructor 2”, “Student 1”, “Student 2”. In practice, onlyone person is assigned the role of a moderator, thus in our coding weuse “Moderator” instead of “Moderator 1”. Such a naming conventionexplicitly shows the role of each participant and easily allows the auto-mation of analysis procedures whether we make investigations at individualor category level. The basic set of roles provides the foundation for analysisboth within a single course and across courses. The special set includescategories, which reflect the roles required by the specific features of eachcourse program.

Now a simple glance at Figure 11 is enough to unravel the mystery ofCharles. The discussion, devoted to the principles of knowledge-baseddesign, included an on-line design of a touring robot for a text-based virtualenvironment. The data in Figure 11 also gives far better vision of what ishappening at each particular stage of the discussion.

2.3 Data mining through text analysisThe data from the transcripts can be analysed in at least two ways: con-firmatory and exploratory. When the initial hypothesis shapes the schemaof the confirmatory analysis, the exploratory data analysis (EDA) and datamining (DM) are performed to discover interesting patterns or structuresthat require explanation. Exploratory data analysis and data mining sharethe same functional objective — the discovery of unknown patterns andrelationships in collected data, thus we use joint EDA/DM abbreviation.EDA/DM provides us with information for generating hypothesis for laterconfirmatory study. Exploratory analysis is suitable for new areas ofresearch. This section illustrates how we can analyse the data sets that wehave constructed from the transcript of a virtual seminar. Our aim is topresent the methodology rather than the results of complete research.

Analysing participation in collaborative design environments 131

Page 14: Analysing participation in collaborative design environments

11 Sudweeks, F and Simoff,S J ‘Complementary explorativedata analysis: the reconciliationof quantitative and qualitativeprinciples’ In Jones, S (ed)Doing internet research SagePublications, Thousand Oaks,CA, USA (1999) pp 29–55

Earlier we have emphasized that the transcripts can be viewed both asquantitative and qualitative data. Sudweeks and Simoff11 introduced theComplementary Explorative Data analysis (CEDA) methodology, whichprovides a framework for combining quantitative and qualitative methodsfor doing research in internet-mediated communications. CEDA employsquantitative methods to extract reliable patterns, whereas qualitativemethods are incorporated for content analysis of the essence of phenomena.In virtual seminars we can combine statistics derived from the quantitativedata with results obtained by text data mining and content analysis of thequalitative data from expressions.

An essential part of the quantitative analysis of the transcript data is thedevelopment of the coding scheme. Current coding schemes used in theresearch in application of virtual environments in education are stand-alonesingle level coding schemes. They are designed to extract informationabout considered features at a particular level of details. Such schemes donot support drill-down analysis — a popular data mining techniques, usedin new research areas when there is a lack of knowledge about the objectof investigation.

132 Design Studies Vol 21 No 2 March 2000

Figure 11 Normalised and role-coded version of the transcript in Figure 10.

Page 15: Analysing participation in collaborative design environments

12 Maher, M L, Cicogniani, Aand Simoff, S J ‘ An experi-mental study of computermediated collaborative design’International Journal of DesignComputing Vol 1 (1997a).13 Maher, M L, Simoff, S Jand Cicognani, A ‘Observationsfrom an experimental study ofcomputer-mediated collaborativedesign’ In Maher, M L, Gero, J Sand Sudweeks, F (eds) Formalaspects of collaborative CADPreprints of the TFIP WG 5.2Workshop on Formal Aspects ofCollaborative Computer-AidedDesign, Jenolan Caves, Aus-tralia (1997c) pp 165–185

To be able to implement drill-down data mining techniques consistentlywe propose to use open hierarchical coding schema, designed to conductinvestigations on increasing level of detail and utilise the results obtainedon previous levels. The initial step in this approach is similar to the initialdata-driven coding of the two-step coding schema used for the analysis ofthe documented designs in computer-mediated collaborative design experi-ments12,13. However, its further steps are in some sense the opposite to thetwo-step schema, in which the coded output of the first step is regroupedaccording to the coding schema using in the next step.

The coding schema shown in Figure 12 illustrates this idea. The seminaractivities described in a transcript can be classified initially into two generalcategories: ones that are related to the theme of the seminar and ones thatare unrelated (like ‘passing a mug of coffee’). This classification is suf-ficient if we are interested only in a measure of whether the participantwas able to stay on the topic of the seminar. The automation of this pro-cedure uses sorting and text analysis.

We investigate the transcripts of three seminars within the elective course“Computer-Mediated Communications for Designers”. In this example, thecourse included four students and three instructors, labeled “Student 1”,$, “Student 4” and “Instructor 1”,$, “Instructor—3”, respectively.For each seminar students were given preliminary readings — a majorpaper on the topic and some additional selected readings. In each seminarone of the instructors act as a moderator, having the freedom to conductthe discussion in accordance with the topic. The other instructors partici-pated with their expertise.

Table 1 represents the results of the coding to three seminars according tothe first level of the coding scheme in Figure 12. We consider only thelines between two margin activities performed by the moderator: the open-ing and the closing of the seminar. The shaded cells indicate the instructor

Analysing participation in collaborative design environments 133

Figure 12 An example of open hierarchical coding schema.

Page 16: Analysing participation in collaborative design environments

who was the moderator of the corresponding seminar. Null values in col-

umn “Related” indicate that the person has not participated in the seminar.

A simple bar-graph visualisation of the data in Table 1 is presented in

Figure 13. It provides a rough pattern of the concentration of each partici-

pant during the seminar. All three seminars are intensive, with the leader-

ship of the third seminar and the smallest concentration during the second.

We consider only the left branch in the coding scheme further, considering

the activities related to the topic of the seminar. The “discussion pies” in

Figures 14 and 15 illustrate the individual (a) and categorised (b) partici-

pation. The visual balance of Figure 14a indicates that the participation in

the discussion by Student 1, Student 2 and the Moderator

(Instructor 2) are nearly the same. As shown in Figure 15a the partici-

pation of the other instructors is relatively small compared to the Moder-

ator’s contribution. Patterns in Figure 14b indicate the different character

of the discussion during Seminar 2. Student 2 continues to be the most

active among the students. Figure 15a shows that more than two thirds of

the utterances came from the students; Figure 15b indicates that student

participation was only a bit more than 50%; and Figure 15c indicates that

134 Design Studies Vol 21 No 2 March 2000

Table 1 Seminar activities categorised into “Related” and “Unrelated” to the seminar theme.

Participants Seminar 1 Seminar 2 Seminar 3

Related Unrelated Related Unrelated Related Unrelated

Instructor 1 3 0 79 0 86 0Instructor 2 109 1 36 3 28 0Instructor 3 20 7 41 4 0 0Student 1 102 6 40 12 47 0Student 2 144 4 67 2 35 0Student 3 38 10 60 10 0 0Student 4 0 0 15 7 0 0Total 416 28 338 38 196 0

Figure 13 Participants profile: related versus unrelated seminar activities.

Page 17: Analysing participation in collaborative design environments

student participation was less than 50%. This could be a warning sign thatstudent participation is dropping.

Considering the role and participation of the instructors and moderators isalso of interest. Instructor 3 has considerably less participation than theother instructors. A possible hypothesis is that the moderator didn’t haveenough experience, thus leaving the initiative to the other instructors. Con-sidering the effectiveness of the moderator by comparing the pie graphs,we see that Figure 14a and Figure 14b show more uniform participationfrom the students. Consequently, we can state that the amount of controlimposed from the moderator shapes the visual pattern of the discussion,therefore, the visual pattern of the discussion allows us to infer the experi-ence and quality of moderation.

Relying on line-based estimators of the participants’ activity could be mis-leading in cases when participants used either very short expressions —one or two words, as in lines 411–413 in Figure 10, or very long(approximately more than 10–15 words), for example, when defendingparticular design solutions. The estimators can be corrected by introducingweights, based on the average length of expressions and length variance.However, word-based estimators, derived from the total amount of words,alphanumeric and other characters provide an idea about individual andgroup activities. We illustrate this approach using the number of words inthe seminars, presented in Table 2.

Figures 16 and 17 provide visualization of the participants’ activity basedon the data in Table 2. Note that segmentations of the pie graphs in Figures14 and 16, and Figures 15 and 17, respectively, are very similar, indicating

Analysing participation in collaborative design environments 135

Figure 14 Line-based estimate of individual activity.

Figure 15 Line-based estimate of group activity.

Page 18: Analysing participation in collaborative design environments

that the average number of words per expression across the seminars is

neither in the “too short” area, nor on the “too long” side. The estimated

average values are presented in Table 3. Thus, visual patterns of discussion

are applicable for assisting on-line student assessment and

instructor/moderator adjustment. The information about the amount of

other characters used in expressions gives an idea about the use of punctu-

ation and other symbols in expressing additional cues in the text. The rela-

tive usage based on the data in Table 2 is shown in Figure 18.

The exploration of the discussion structure and collaboration during the

seminar requires the use of the next level in the coding scheme in Figure

12. In each seminar, from the activities related to the discussion, the lines

that respond to a specific person are classified as local threads, the rest are

136 Design Studies Vol 21 No 2 March 2000

Figure 16 Word-based estimate of individual activity.

Figure 17 Word-based estimate of group activity.

Table 2 Number of words used during the seminars.

Participants Seminar 1 Seminar 2 Seminar 3

Words Alpha- Other Words Alpha- Other Words Alpha- Othernumeric characters numeric characters numeric characters

characters characters characters

Instructor 1 80 345 3 462 1877 28 728 2783 39Instructor 2 715 3179 135 145 647 13 215 886 32Instructor 3 181 723 32 275 1205 24 0 0 0Student 1 754 3169 155 304 1386 54 228 919 69Student 2 1157 4946 203 503 2213 47 271 1046 31Student 3 203 836 12 252 1113 10 0 0 0

Page 19: Analysing participation in collaborative design environments

marked as global. Shaded lines 248, 250 to 254 in Figure 18 are examplesof local threads. The other threads we label as global.

The result of the coding is presented in Table 4. The shaded cells indicatethe instructor who was the moderator of the corresponding seminar.

Figure 19 illustrates the results. Figure 19a shows very few local threads,which we consider to be an indicator of tight collaboration during thediscussion. An increase in the amount of local threads can be caused byproblems in moderation (Figure 19b) or if the seminar theme includes anumber of issues and the discussion of these goes in parallel (Figure 19c).Sometimes it could be a result of preferred style of behaviour during thevirtual seminar.

The analysis above does not provide much information about whether thediscussion was related to the topic, or how much particular participant

Analysing participation in collaborative design environments 137

Table 3 Average number of words per expression.

Participant Seminar1 Seminar2 Seminar3

Instructor 1 8.0 8.4 8.5Instructor 2 6.6 4.1 7.7Instructor 3 7.9 6.7 0.0Student 1 5.5 7.4 4.9Student 2 7.0 7.6 7.5Student 3 5.1 4.5 0.0Seminar Average 6.7 6.5 4.8

Figure 18 An example of a long local thread which can dominate the seminar.

Page 20: Analysing participation in collaborative design environments

contributed to the topic of the discussion or not. Text-based virtual sem-inars provide the necessary data for extracting such information.

Computer-based content analysis is a state-of-the-art activity, considerablyimproved with the overall progress in computing. In virtual seminars itprovides a quantitative analysis of the qualitative seminar data, offeringobjective results to supplementing instructor’s subjective evaluation of thediscussion. The content analysis of seminar transcripts starts with the speci-fication of a list of words that distort the content of the transcripts. The

138 Design Studies Vol 21 No 2 March 2000

Figure 19 Participants profile: Related activities: Local versus global threads in seminar discussions.

Table 4 Related seminar activities classified into “Local” and “Global” threads.

Participants Seminar 1 Seminar 2 Seminar 3

Global Local Global Local Global Local

Instructor 1 3 0 24 17 56 30Instructor 2 98 11 73 6 15 13Instructor 3 17 3 27 9 0 0Student 1 102 0 38 2 39 8Student 2 139 5 55 12 11 24Student 3 38 0 47 13 0 0Student 4 0 0 15 0 0 0Total 397 19 279 59 121 75

Page 21: Analysing participation in collaborative design environments

14 Berthold, M R, Sudweeks,F, Newton, S and Coyne, R‘Clustering on the Net: Applyingan autoassociative neural net-work to computer-mediated dis-cussions’ Journal of ComputerMediated Communication, Vol 2No 4 (1997) http://www.ascusc.org/jcmc/vol2/issue4/berthold.html.15 Berthold, M R, Sudweeks,F, Newton, S and Coyne, R ‘Itmakes sense: Using an autoas-sociative neural network toexplore typicality in computermediated discussions’ InSudweeks, F, McLaughlin, Mand Rafaeli, S (eds) Networkand netplay: virtual groups onthe Internet, AAAI/MIT Press,Menlo Park, CA, USA (1998) pp191–220

list of words, excluded from the analysis, consists of prepositions, conjunc-tions and disjunctions, articles, pronouns, and other particular words. Thelast group is used for analysing the content. Content analysis softwarecounts the occurrences of the remaining words and excludes theinfrequently occurring words. The investigator defines the threshold andalso a list of thematic key words, whose occurrence has to be consideredeven if it is below the threshold. In addition to term frequencies, automatedcontent analysis investigates the co-occurrence of the frequently occurringwords and their clustering based on these co-occurrences.

One of the problems in applying content analysis methods to virtual sem-inar data is the relatively low word frequencies. Qualitative techniques areusually applied for the analysis of focused interviews. The word fre-quencies in such interviews are about 100–300. In a seminar transcriptthese frequencies are usually no greater than 10.

3 Evaluating collaboration through data analysisand visualisationCollaborative design denotes design activity when more then one personworks on a single design problem, having a common goal or intent. Collab-oration is possible when the collaborators share activities and informationto achieve the common goal. Effective collaboration occurs when the col-laborators share design tasks, communication, representation and docu-mentation5.

To facilitate collaboration between team members, the virtual campus pro-vides each design team with a separate bulletin board and design logsviewer. These bulletin boards are independent of the major studio bulletinboard. To protect the privacy of teamwork and information exchange, eachbulletin board and design log viewer were password protected. The man-agement of these components is implemented through the VDS Team man-ager, whose front panel is shown in Figure 20.

We assume that the major exchange of information between designers isrealised through the bulletin board. Then we consider that both the contentand the pattern of the messages on the bulletin board reflect team collabor-ation. In this paper we consider message patterns assuming that the contentanalysis has established a correspondence between the subject and the con-tent of the message.

The messages on the board are grouped in threads. A group ofresearchers14,15 proposed a threefold split of the thread structure of e-mail

Analysing participation in collaborative design environments 139

Page 22: Analysing participation in collaborative design environments

messages in discussion archives in order to explore the interactive threads.It included (i) reference-depth: how many references were found in asequence before this message; (ii)reference-width: how many referenceswere found, which referred to this message; and (iii)reference-height: howmany references were found in a sequence after this message. These para-meters are important for a detailed analysis of the messages and the struc-ture of the discussion. For qualitative judgements we introducethread-length of a thread as the maximum of the sum of reference-height andreference-width over the messages in the thread, andthread-widthas themaximum among the sums of reference-widths at each level.

Further developments included the time variable explicitly11, in additionto the threefold split. Each message is completely identified by two indi-ces — one for its level and one for its position in time in the sequence ofmessages at this level. Thetime envelopeis defined as the time intervalbetween the first and the last messages in the thread. Figure 21 shows thevisual representation of the thread reference tree. “M” and “t” denote mess-age and time points, respectively. This model allows the comparison ofthe structure of discussion threads both in a static mode (for example, theirlength and width at corresponding levels) and in a dynamic mode (forexample, detecting moments of time when one thread dominates anotherin multi-thread discussions).

140 Design Studies Vol 21 No 2 March 2000

Figure 20 The “front panel” functionality of Team manager.

Page 23: Analysing participation in collaborative design environments

16 Knuth, D E The art of com-puter programming, Vol 1: Fun-damental algorithms Addison-Wesley, Reading, MA, USA(1997) pp 311–312

We categorise bulletin board messages in project-based design courses intotwo major groups: related to design tasks and management. Task-relatedmessages consider a particular topic of the project, when managementmessages are connected with the arrangement of meetings, changes inschedules and places.

The graph structure reference in Figure 21 shows the type of design collab-oration that occurs in a team environment. Once the representation is struc-tured hierarchically there are several techniques to display its structure.Donald Knuth, in the first volume of his “bible for programmers” describedfew notations for visualising tree structures — graph tree, idention graph,and a nested set16. The graph visualisation is more helpful for smallerthreads and when we are interested in quantitative information. Graph treeis also useful when we consider the dynamic properties of the discussion.A nested set visualisation provides a better illustration when we are inter-ested more in the type of collaboration and can omit the time.

Collaboration on a shared design task can be considered at different levelsof abstraction and “degrees” of task sharing. Following this view,researchers identify two extreme approaches to sharing design tasks duringcollaboration:single task collaborationandmultiple task collaboration.

During single task collaboration the resultant design is a product of a con-tinued attempt to construct and maintain a shared conception of the design

Analysing participation in collaborative design environments 141

Figure 21 Visualisation of a thread reference tree11.

Page 24: Analysing participation in collaborative design environments

task. In other words each of the participants has his/her own view overthe whole design problem and the shared conception is developed, roughlyspeaking, by the “superposition” of the views of all participants. This typeof collaboration is reflected in the bulletin board message structure by theexistence of task-related threads running in parallel. The collaboration iscloser when there are large thread widths and lengths. The time envelopesalso can be larger, compared to the average time envelope — participantsusually revise some of their earlier views. The bulletin board also includesseveral long threads usually connected with crucial issues in the project.

During multiple task collaboration the design problem is divided amongthe participants so that each person is responsible for a particular portionof the design. Thus, multiple task collaborative design does not necessarilyrequire the creation of a single shared design conception, though designerswork cooperatively in a common electronic workspace. This collaborationstyle is reflected by isolated task-related messages or short threads. Mostof the messages or the threads are related to the management of the project.

Student collaboration in course design projects is usually a combinationof these approaches. Some of the teams usually followed the single taskcollaboration until they created a joint understanding and view on the prob-lem and then they had some separation of the implementation tasks. Otherteams started with distributing the initial research tasks with some degreeof single task collaboration during the composition of the final design.

Figure 22 illustrates a bulletin board of a single task collaborating team.Graph-tree and nested set visualisation of this bulletin board fragment isshown in Figure 23a and b, respectively. The graph-tree in Figure 23aillustrates the idea of parallel threads corresponding to different project

142 Design Studies Vol 21 No 2 March 2000

Figure 22 Bulletin board fragment with task-related messages, visualised as indention graph.

Page 25: Analysing participation in collaborative design environments

tasks, although the threads are relatively short (the greatest thread-lengthis 3, but for most of them the thread-length is 2) and “narrow” (thread-width for each one is 1). The nested set in Figure 23b provides a betterqualitative picture of collaboration. That is, more nested rectangles indi-cates more collaboration.

4 SummaryCreating collaborative design environments usually involves the consider-ation of the technology that can enable communication and informationsharing. These environments can provide for both synchronous and asyn-chronous communication. We demonstrate how the record of the communi-cation can be used to analyse participation and contribution to the collabor-ative project. We present our Virtual Campus which provides aneducational environment for on-line seminars and threaded discussions onelectronic bulletin boards.

The analysis of the record of on-line seminars can show who has partici-pated and the extent of their participation. Evaluating individual partici-pation can identify not only the amount of contribution, but the content ofthe contribution. The analysis of the bulletin board discussion can provideindications of the type of collaboration and the extent of the interactionon ideas and management. Although we present the benefits of this kindof analysis for educational environments, the analysis of professional col-laboration can provide feedback on project participation and the effective-ness of the collaborative team.

The method for analysing participation is based on text analysis and datamining. The analysis framework provides extensive support for research

Analysing participation in collaborative design environments 143

Figure 23 Formal visual-

isation of the bulletin board

fragment in Figure 22. (a)

Graph-tree, (b) nested set.

Page 26: Analysing participation in collaborative design environments

both in educational and collaborative design environments. Presented tech-niques can be employed in the protocol analysis of collaborative design inWeb-based design environments. Current environments do not include on-line data analysis and analytical processing features. Data analysis, miningand visualisation could become an integral part of design educational andcollaborative environments. Providing means for on-line analysis of com-munication and collaboration is invaluable for the active learning andresearch in computer-mediated design.

144 Design Studies Vol 21 No 2 March 2000


Recommended