The proliferation of computer media and networking has the poten-tial to make fundamental changes in the methods, models and tech-niques employed to educate and train design students and pro-fessionals. The easy access to the internet requires a reconsideration ofthe desktop Computer Aided Instruction and Computer Aided Learning(CAT/CAL) paradigms. Analyzing the role and impact of computers in theon-going changes in education, Lockard, Abrams and Many1 specify twotypes of CAI/CAL applications. Applications classified as Type I employ
1 Lockard, J, Abrams P D,and Many, W A Microcomputersfor twenty-first century educatorsAddison-Wesley, Reading, MA,USA (1994)
www.elsevier.com/locate/destud0142-694X/00 $ - see front matter Design Studies 21 (2000) 119144PII: S0142-694X(99)00043-5 119 2000 Elsevier Science Ltd All rights reserved Printed in Great Britain
Analysing participation incollaborative design environmentsSimeon J. Simoff and Mary Lou Maher, Key Centre of DesignComputing and Cognition, University of Sydney, NSW-2006, Australia
Computer-supported collaborative design can be realised by a broadrange of collaborative environments, each facilitating a different kind ofcollaboration. Understanding the style of collaboration and the potentialfor each environment is important when choosing a particulartechnology. We have developed a virtual world approach to teachingdesign computing in which students learn through traditional lectures,online seminars, and collaborative design projects. The environmentintegrates both synchronous and asynchronous communication as wellas shared documentation. One side effect of using this environment isthe incremental development of a record of the communication andcollaboration. This record can be the basis for the analysis ofparticipation in collaboration. We show how text analysis as a part ofdata mining can be used to analyse different aspects of participation.Specifically, we analyse participation in synchronous communication toevaluate individual contribution. We then analyse asynchronouscommunication to evaluate the extent of collaboration. The methodspresented can be an automated part of the collaborative environmentproviding information for student evaluation in an educationalenvironment or individual contribution in a professional environment. 2000 Elsevier Science Ltd. All rights reserved
Keywords: collaborative design, computer supported design, designeducation, data mining
2 Streibel, M J Instructionalplans and situated learning: thechallenge of Suchmans theoryof situated action for instructionaldesigners and instructional sys-tems in Anglin, G J (ed) Instruc-tional technology, past, present,and future Libraries Unlimited,Englewood, CO, USA (1995) pp1171323 Savery, J R and Duffy, T MProblem based learning: aninstructional model and its con-structivist frameworkEducational Technology Vol 35No 5 (1995) pp 31384 Mitchell, W J and McCul-lough, M Digital design mediaVan Nostrand Reinhold, NewYork, USA (1995) pp 441462
computing resources to do things educators have previously done withoutcomputers. These applications improve such aspects of teaching like prep-aration of teaching materials and student management. Generally, thisapproach does not change the teaching strategies and schemata, thus TypeI automation reduces the technical efforts but does not result in more effec-tive teaching. The other group of applications, labeled as Type II are ori-ented towards bringing teaching and learning methods and experiencesimpossible without computers. We explore this type of environment andits use in design education and the profession.
Most of the early Web-mediated on-line environments deliver variousmultimedia learning materials and give students access to an enormousresource of information. However, the on-line materials alone are not suf-ficient for evaluation and assessment since the instructors cannot trace stud-ent participation, in the sense of interaction with course materials, otherstudents, and tutors. The absence of an overall learning model, combinedwith restricted interaction and single direction of the information flow,show that these initial efforts were able to utilise only a small part fromthe potential of internet technologies. On-line learning methods can facili-tate three components of course administering (i) course materials tobe delivered to the students; (ii) communication between students and edu-cators, and (iii) management and assessment of students. In this paperwe focus on automating the analysis of communication and collaboration,providing a methodology for analysing participation in any collaborativedesign environment.
1 Computer-supported collaborative designeducationDesign, as an educational subject, is characterised by the lack of clearseparation between theoretical knowledge and practical skills. In addition,design projects require intensive collaboration of numerous specialists. Theinstructional strategy in design studios is based on constructivist principlesin which a student actively constructs an internal representation of knowl-edge by interacting with the material to be learned. The model implementsthe principles of situated cognition2 and problem-based learning3. Accord-ing to these theories, both social and physical interaction enter into thedefinition of a problem and the construction of its solution. Neither theinformation to be learned, nor its symbolic description, is specified outsidethe process of inquiry and the conclusions that emerge from that process.
Mitchell and McCullough4 present the general idea of the virtual designstudio as an extension of conventional studios, discussing differentenabling technologies and corresponding studio scenarios. Maher, Simoff
120 Design Studies Vol 21 No 2 March 2000
5 Maher, M L, Simoff, S J andCicogniani, A Potential andlimitations of virtual design stu-dio. Interactive Construction On-line January al. (1997b)6 Simoff, S J and Maher, M LDesign education via web-based virtual environments InAdams, T (ed), Computing inCivil Engineering Proceedings ofthe Fourth Congress of Comput-ing in Civil Engineering, ASCE,New York, USA (1997a) pp4184257 Simoff, S J and Maher, M LWeb-mediated courses: the rev-olution in on-line design edu-cation In Ashman, H, Thistle-waite, P, Debreceny, R andEllis, A (eds) Into the main-stream the Web in AustraliaSouthern Cross UniversityPress, Lismore, Australia(1997b) pp 143154
and Cicogniani5 developed these ideas into a Web-based Virtual DesignStudio (VDS) as a distributed environment for teaching students about col-laborative design. The modification of the workspace in the VDS and thereplacement of the physical space by a computer mediated educationalenvironment enhances the studio-style learning by releasing geographicaland time scheduling constraints.
Our experience in developing learning materials for virtual design studios6,7show that the design and management of Web-mediated courses involvesmore than converting lectures and exercise notes to a collection of linkedweb pages or providing tools for communication. We identify two issuesin developing effective collaborative design environments:
I the need to create a sense of place and community among the parti-cipants in the studio and
I the need for interactive and/or automatic assessment.
We address these needs in the development of our Virtual Campus(http://www.arch.usyd.edu.au:7778/). This is an educational environmentwhich uses the metaphor of a physical campus as a collection of rooms toorganise the learning and community activities. Rooms in the virtual cam-pus are grouped according to their type of use, such as classroom, office,or conference room. The rooms provide some of the functions of a physicalroom, such as privacy, security, and a place for various activities. Theclassrooms are equipped with a slide projector, a recorder, whiteboard,notice board and other facilities that help the student with a particularlearning task. The basic communication mode is text-based, though thecampus has a Web-based interface for navigation through a 3D visualisa-tion of the facilities (using VRML).
The interface, shown in Figure 1, provides facilities for viewing the partof the campus where the person is located, various tools for using thecampus and a line input for communication, navigation, and creatingobjects. Figure 1 shows the practice studio for preparing a presentation.The room is accessible from the conference room. The top part of thewindow shows the room description, with a tool bar for looking at thewhiteboard, the slide projector, the course materials (created using WebCTas described below), and other tools for using the campus. The bottom partof the window shows the text description of the room and responds toinput from the user. The input can vary from commands to show who isin the room to communication commands such as say, whisper, andemote. This environment provides real time communication similar to achat window, with the additional feature of providing a permanent place
Analysing participation in collaborative design environments 121
8 Sutherland, J The Java rev-olution Sun Expert Vol 8 No 1(1997) pp 4354
to store information and other objects. This environment effectivelyaddresses the first need to create a sense of place and community.
The second issue is the need to provide for interactive and automatedassessment. Since the students are not in a physical classroom, in-classtests do not make sense. There are several difficulties with implementinginteractive assessment within a Web-mediated course delivery mechanismin comparison to stand-alone CAI/CAL programs. The main disadvantagecomes from the batch mode of information exchange. This makes theimplementation of genuine, immediate and meaningful feedback problem-atic. Typically, a form is filled out then sent to the server which processesit and sends back another form or web page. Sutherland8 presents newarchitectures of enhanced Java/CGI implementations and Applet/Servletremote method invocations, which have the potential to overcome thisproblem of the client/server model. These innovations allow greater inter-
122 Design Studies Vol 21 No 2 March 2000
Figure 1 The Web interface to the Virtual Campus.
activity and permit more complex client operations. However, this is at thetechnological front end and, consequently, there is a lack of tools to sim-plify the implementation.
The response to this need is a new generation of Web-authoring environ-ments, which provide a set of tools for student communication and toolsfor course designers to create and structure courses. There are numerousworking pioneers, among which WebCT(http://homebrewl.cs.ubc.ca/webct/webct.html), developed at the Univer-sity of British Columbia, has been successfully employed for design coursemanagement and delivery at the University of Sydney. Figure 2 illustratesWebCT in course design mode. These new environments include on-line
Analysing participation in collaborative design environments 123
Figure 2 WebCT: designing a Web-based course on Design Computing.
quizzes, student participation and progress tracking facilities and enhancedcourse management facilities. These features are mainly based on theanalysis and bar-graph visualisation of server statistics. They do not ana-lyse and assess student collaboration an essential part of project-baseddesign courses.
The combination of the WebCT course design environment with the VirtualCampus provides a comprehensive environment for collaborative learning.The combined environment is illustrated in Figure 3. The WebCT courseserver includes a bulletin board system with archiving facility and the cam-pus classrooms provide a place for meetings, presentations, and seminars.Project-based courses, conducted in this environment, use a variety ofsynchronous and asynchronous computer-mediated teaching techniques.Transcripts of on-line subject discussions, conducted in the Virtual Campusand messages on bulletin boards are a rich source of data. We show how
124 Design Studies Vol 21 No 2 March 2000
Figure 3 Integrated Web-based design educational environment.
9 Fayyad, U M, Piatetsky-Shapiro, G and Smyth, P Fromdata mining to knowledge dis-covery: An overview. In Fayyad,U M, Piatetsky-Shapiro, G,Smyth, P and Uthurusamy, R(eds) Advances in knowledgediscovery and data mining, AAAIPress, Boston, MA, USA (1996)pp 13410 Chen, M S, Han, J and Yu,P S Data mining: an overviewfrom a database perspectiveKnowledge and Data Engineer-ing Vol 8 No 6 (1996) pp 866883
text analysis and data mining methods can be used for the analysis andevaluation of student participation and collaboration.
2 Evaluating individual contribution using textanalysis and data miningData mining (DM), known also as knowledge discovery (KD), is the over-all process of examining a data source for implicit information and rec-ording this information in explicit form, in other words, the extraction ofa high-level knowledge from a low-level data. The overall process of datamining is shown in Figure 4. DM involves the identification of potentiallyuseful and understandable patterns in this data9, spanning the entire spec-trum from discovering information of which one has no knowledge towhere one merely confirms a well known fact. Data mining methods havebeen developed in machine learning, statistics, data visualization anddeductive databases to examine the content of large databases10.
Historically, data mining was initiated in large databases. The number ofrecords in such databases can be estimated in millions, and the informationis well-structured in table form according to a particular data model. How-ever, a vast amount of design knowledge is coded in machine-readable textform or as electronic dictionaries, manuals and references, as CAD draw-ings or digital images. Most of the design data is unstructured, in compari-son with the large databases used in conventional data mining. A character-istic of data mining and knowledge discovery is that the source data wascreated for purposes other than knowledge discovery. In our case, thesource data is the text transcripts from the seminar discussions in the vir-tual campus.
The on-line discussions in the virtual campus are based on the idea of acomputer-mediated moderated discussion. A computer-mediated moder-ated discussion can be held both in synchronous and asynchronous mode.Asynchronous discussions can be realised through a computer-mediated
Analysing participation in collaborative design environments 125
Figure 4 The data mining lifecycle.
bulletin board, a discussion list-server, or even by e-mail. Here we focuson synchronous seminar discussions taking place in the virtual campusclassroom. Each seminar is devoted to a particular theme as part of thecourse, complementary to lectures or course material, which students havepreviously attended or read. Figure 5 illustrates a virtual seminar scenario.
Discussions in the virtual campus have an initi...