Incidence of group awareness information on students' collaborative learning processes

Incidence of group awareness information onstudents’ collaborative learning processesM. Pifarré,* R. Cobos† & E. Argelagós**Faculty of Education, Universitat de Lleida, Lleida, Spain†Department of Computer Science, Universidad Autónoma de Madrid, Madrid, Spain

Abstract This paper studies how the integration of group awareness tools in the knowledge manage-ment system called KnowCat (Knowledge Catalyser), which promotes collaborative knowl-edge construction, may both foster the students’ perception about the meaningfulness ofvisualization of group awareness information and promote better collaborative processes aswell as enhance better task performance. Forty-seven university students participated in aresearch study, where one group of 23 students used KnowCat without the awareness console(non-awareness group); the other 24 students used KnowCat with the awareness console(awareness group). Both groups used KnowCat during one semester. Data analysis revealedthat the awareness group means were higher than those of the non-awareness group in termsof participation, cognitive and metacognitive learning activities, and task performance. More-over, students revealed that knowing what, where and how much their classmates werecontributing acted as positive feedback by encouraging participation and orienting their ownbehaviour and their contribution to the collaborative work. In this paper, we claim that thevisualization group awareness information in KnowCat, a utility allowing students to visualizeand track what, where, how much and how often other participants contributed to theKnowCat knowledge area, had a positive impact on the students’ collaborative behaviour.

Keywords collaborative learning processes, computer-supported collaborative learning, groupawareness, instructional guidance, metacognition, peer feedback.

Introduction

Work groups need information about one another,about shared artefacts and about group processes. Thisinformation is often associated with group awareness(Briggs, 2006). It refers to being informed about spe-cific aspects of group members such as other partici-pants’ locations in the shared area (where are theyworking), their actions (what are they doing), theirinteraction history (what have they already done), their

intentions (what are they going to do next) and theirknowledge (Bodemer & Dehler, 2011; Gross, Stary, &Totter, 2005; Gutwin & Greenberg, 2002).

In collaborative learning scenarios, awareness ofpartners’ cognitive and social activities is a relevantvariable because awareness tools may help collabora-tors to overcome problems related with communica-tion, engagement and coordination of the group work(Carroll, Neale, Isenhour, Rosson, & McCrickard,2003). With a view to doing research on group aware-ness, computer-supported collaborative learning –henceforth, CSCL – aims at developing tools that couldeffectively support group members and studying therelationships between awareness and learning (Buder,2011).

Accepted: 29 August 2013Correspondence: Manoli Pifarré, Faculty of Education, Universitatde Lleida, Avinguda de l’Estudi General 4, Lleida E-25001, Spain.Email: [email protected]

bs_bs_banner

doi: 10.1111/jcal.12043

Original article

© 2013 John Wiley & Sons Ltd Journal of Computer Assisted Learning (2014), 30, 300–317300

mailto:[email protected]

To this end, the study described in this paper appliesgroup awareness tools to a knowledge managementsystem called KnowCat (Knowledge Catalyser)(Cobos, 2003) to help group members raise awarenesson the other’s activities in a collaborative learning envi-ronment. The ultimate aim of our research is to studyhow the integration and use of group awareness tools ina CSCL environment affects the students’ collaborativelearning activities and task performance.

The possibilities of group awareness tools insupporting collaborative learning

Research on group awareness in CSCL environments isemerging judging by the broad scope of group aware-ness applications existing nowadays. Bodemer andDehler (2011) established three essential group aware-ness types for effective collaborative learning, namelysocial, cognitive and behavioural awareness.

Social awareness indicates the functioning of thegroup as perceived by the collaborators. This type ofawareness informs of the presence or availability ofother members of the community and assesses socialbehaviour patterns such as cooperativeness, friendli-ness, influence or reliability. Some studies indicate thatsocial processes may allow group members to knowand understand each other in order to become a com-munity of learning and a better development team tocollaboratively solve the problem (Kreijns, Kirschner,& Jochems, 2003). In this respect, Phielix, Prins,Kirschner, Erkens, and Jaspers (2011) expanded theconcept of a CSCL environment by means of a peerfeedback tool (radar) and a reflection tool (reflector).They examined the incidence of a CSCL environmenton both perceived social and cognitive group behaviourand on the social and cognitive performance of thegroup. The study concluded that the combination ofpeer feedback to foster interpersonal behaviour and thereflection of feedback enhanced group process satisfac-tion and social performance.

Cognitive awareness refers to the knowledge ofgroup members. It gives information about self andpartners’ self-assessed knowledge, cues about knowl-edge of the other group members, type of knowledgecontributed by others or distribution of knowledge incollaborative learning scenarios. Dehler, Bodemer,Buder, and Hesse (2011) proved that tools providingpartners’ knowledge were based on the availability of

learning content, performance in a knowledge test oropinion ratings in a questionnaire. The study also dem-onstrated that reliable knowledge group awarenesscould promote better management of the diversity ofthe group mates’ knowledge in order to successfullyreach the purposes of the collaborative task.

In this line of research, Romero (2011) reported onimproved students’ knowledge group awareness whenusing a CSCL system called EuroCat. This system con-sists of an interface which allows students to state theirstatus and knowledge of the different aspects of thetask. The study showed how unfamiliar students, thatis, having zero-history group with the online mastercourse, managed to evaluate their group mates’ knowl-edge more accurately and increased their knowledgegroup awareness as a result of using the knowledgeawareness tools provided by the learning environmentEuroCat.

Behavioural awareness indicates the learners’ activ-ities within the CSCL environment such as what theparticipants are doing, how much, how often andwhere. One of the main approaches to developingbehavioural awareness is to visualize important infor-mation that can help individuals to know what, howand how much the other group members are contribut-ing to solve the collaborative task. Visualizations candisplay and facilitate the collection and interpretationof crucial information for better collaboration becausethey create a complex conceptual structure representedexternally in a visual display (e.g., Jiang, Elen, &Clarebout, 2009; Kimmerle & Cress, 2009).

Several studies have implemented visualizationsabout different aspects of individual contributions tothe collaborative task and have analysed their impacton the collaborative processes and task performance. Inthis line of argument, Janssen, Erkens, and Kirschner(2011) investigated the potential of a participationawareness tool, which visualized the group members’relative contribution in an online writing collaborativetask. The study argued that whenever students partici-pated actively during online collaboration and partici-pation was divided equally among group members,every single member of the group had the opportunityto contribute to the problem-solving process, partici-pate in knowledge construction, give or request expla-nations, and use and refine his/her skills. This studyalso reported on the higher incidence of students usingthe participation awareness tool in online discussions

Incidence of group awareness information 301

© 2013 John Wiley & Sons Ltd

during the collaborative writing process. The participa-tion awareness tool also promoted equal participationwithin the group, and better coordination and regula-tion of group activities in the relational area. However,the increase in awareness participation of this study didnot guarantee better group performance.

Other studies have emphasized the fact that behav-ioural awareness tools can generate better externalfeedback among group members because they providegroup members with feedback on how adequately theyare collaborating. Educational literature highlights theimportance of scaffolding processes among peers,especially the processes for giving and receiving assis-tance to favour better collaborative learning (e.g.,Scardamalia & Bereiter, 1994; Hakkarainen, Lipponen,& Järvelä, 2002; Pifarré, 2007).

An example of how group awareness tools can beembedded in CSCL environments to give feedback togroup members has been extracted from research byZumbach, Reimann, and Koch (2006). These authorsembedded a collaborative environment in management-based scaffolds whose collaborative online learningenvironment was enriched by functions as tracking,analysing and feeding back parameters of participation,collaboration, motivation and emotional state of groupmembers. Zumback et al. showed that a positive influ-ence of feeding back the groups and its reuse on par-ticipants influenced the learners’ group behaviour, aswell as their problem-solving, emotional and motiva-tional parameters.

To sum up all the above, enhancement of students’group awareness by means of tools providing informa-tion about key aspects of group collaboration can, inturn, help students monitor their collaborative behav-iour in order to better contribute to solve a collaborativetask. However, as pointed out by Buder (2011), furtherresearch is required on such issues as (i) what displaymethod of the group collaborative behaviour is mostappropriate in a given context, (ii) what informationshould be displayed, and (iii) what collaborative pro-cesses should be monitored. Our study attempts then todelve further into these issues. This paper gives adetailed account of this study.

Description of the KnowCat system

KnowCat is a fully consolidated and thoroughly testedand validated CSCL system which has been developed

at Universidad Autónoma de Madrid (Spain) in activeuse since 1998 (Alamán & Cobos, 1999; Cobos, 2003).The main aim of this system is to generate qualityeducational materials as the automatic result of studentinteractions with the materials, by catalysing the crys-tallization of knowledge. More specifically, the systemis based on a mechanism called ‘Knowledge Crystalli-zation’. The mechanism supplies evidence for what thebest contributions are, in the users’ opinion, throughinteraction with the system. A full description of thesystem and the experimental studies carried out so farcan be consulted in Cobos (2012) but a brief descrip-tion can be found below.

KnowCat enables to build up community knowledgesites called ‘KnowCat sites’ or ‘KnowCat nodes’. Eachknowledge site is organized around the followingknowledge elements: (1) The knowledge tree, based ona hierarchical structure of topics, displays the organi-zation of the knowledge site in several topics. (2) Eachtopic contains a set of mutually alternative documentsthat describe the topic. (3) At any time, the author of adocument can contribute with a new version of his/herdocument. (4) Each document can receive annotations– or notes, for short –, with a review of the informationpresented in a document. Each note belongs to a typethat determines its purpose. The three note types are:clarification, support and review. (5) Finally, eachdocument can receive assessments. An assessment rep-resents a ‘weight assertion’ which can be utilized by theusers in order to determine the quality (with a valuefrom 1, minimum value, to 10, maximum value) of aspecific aspect (i.e., correctness, innovation, etc.) of aspecific part of a document (i.e., introduction, refer-ences, etc.).

The main aim of this paper is to show how KnowCatcan be improved when using group awareness tools. Tothis end, six group awareness tools were integrated inKnowCat, as shown in Figure 1. These tools werecalled Registered Users, Online Users, Radar View,Notes View, History View, and Participation View.These KnowCat awareness tools are briefly describedbelow.

(a) Registered Users provides brief information aboutthe registered users in KnowCat; in other words,who are my partners?

(b) Online Users provides information about thecontact data and the current location of a selected

M. Pifarré et al.302


user; in other words, who is online and where ishe/she working?

(c) Radar View provides the locations of the onlineusers through a replicated knowledge tree of theKnowCat site (Figure 2). In each topic of the rep-licated knowledge tree, the number of online usersand who is interacting in each topic is shown; in

other words, where and what are the connectedusers doing?

(d) Notes View provides graphical information aboutinteraction among users in the annotating task.Users can consult which document was annotated,who annotated it and what was annotated(Figure 3). With this tool, the users can see

Figure 1 Screenshot of the KnowCat Site Called ‘Psychopedagogy Intervention’; the Awareness Console is Shown in the GroupInformation Part

Figure 2 Example of Radar View Awareness Tool



graphically how the annotation process is going onamong the different group members – who by, whatabout and who to was a document annotated. It isexpected that knowing how the feedback processhas been performed with the small groups maypositively affect a group’s collaborative behaviour.

(e) Participation View displays in a table the numberof contributions of each participant for eachknowledge element, namely the number of docu-ments added, the number of new document ver-sions, the number of notes added and the number ofvotes. Besides, this tool also displays the commu-nity average for each knowledge element. In addi-tion, the participation view offers the option todisplay the members of the community arranged inascending or descending order depending on thenumber of contributions made in a specific knowl-edge element (bottom part in Figure 1).

(f) History View displays chronically and graphicallyinformation about the tasks realized by each user.Furthermore, as Figure 4 indicates, on the left sideof the history view are displayed the milestonesagreed by the community in order to fulfil collabo-rative work. Therefore, students can compare theirown work with that done by other group membersand with what they should have done in order toreach effective collaboration. This tool may guidestudents through the progress of their own

behaviour and motivate them to further contribute tothe collaborative process.

Moreover, all the information visualized in all KnowCatawareness tools can be clicked on and students can trackdown and consult the classmates’ knowledge submittedto the system: documents, notes or versions. From ourperspective, the possibility to easily track down theknowledge submitted to the collaborative knowledgearea might provide valuable group awareness about howknowledge is collaboratively constructed and distrib-uted in the system. Furthermore, this awareness mightgive users many opportunities to monitor and regulatetheir own actions.

Research questions

The four initial research questions asked in our studywere the following:

1. How meaningfully do students perceive the infor-mation provided by the different group awarenesstools in KnowCat for effective collaborative learn-ing – in terms of social and behavioural awareness?

2. Does the use of group awareness tools in KnowCatencourage students to give more feedback – behav-ioural awareness – in terms of sending more notes totheir peers, compared with the use of the systemwithout such tools?

Figure 3 Example of a Part of the NotesView Visualization Tool



3. Does the use of group awareness tools in KnowCatpromote better students’ collaborative learningactivities – in terms of affective, cognitive andmetacognitive activities – compared with the use ofthe system without such tools?

4. Does the use of group awareness tools in KnowCatenhance better students’ task performance com-pared with the use of the system without such tools?

Method

Participants

Forty-seven university students participated in thestudy. They were divided into two groups. One groupof 23 students, called ‘non-awareness group’, usedKnowCat without the awareness console. The remain-ing 24 students, called ‘awareness group’, usedKnowCat with the awareness console. Both groupsconsisted of the natural class for the academic termand used KnowCat during a one-term period of theregular university course called psycho-pedagogicalintervention in developmental disorders (Degree inPsychopedagogy at the Universitat de Lleida, Spain).Both groups followed the same instructional process,as described in the next section.

In order to guarantee the reliability in the selectionprocess of participants from both awareness and non-awareness groups, we analysed the students’ marksobtained in the previous curricular course – calledInstructional Psychology. The Mann–Whitney non-parametric test (95% significant level) revealed no sta-tistical significant differences between both groups(non-awareness group: N = 23, M = 7.78, SD = 0.89;awareness group: N = 24, M = 7.39, SD = 2.03;p = 0.888). It can be noted that the awareness group hasa higher standard deviation than the non-awarenessgroup because 78% of the marks of the non-awarenessgroup ranged from 7 to 8.9 points (in a range from 0 to10) and 75% of the marks of the awareness groupstudents ranged from 7 to 10. Despite that, we couldconsider both groups as statistically comparable.

Procedure

Students of both groups followed the course face-to-face and they solved online two authentic problem-based tasks using KnowCat. The tasks presented realcases of children with learning difficulties, to whichstudents had to design specific educational interventionin order to better overcome these difficulties andpromote the children’s better development. The

Figure 4 Example of the History View



non-awareness group used the KnowCat versionwithout the awareness console while the awarenessgroup used the KnowCat version with the awarenessconsole.

In order to assist all the students in the use ofKnowCat to construct knowledge collaboratively, that is,the use of KnowCat notes as improved scaffolds thatcould help their classmates improve their documents, wedesigned a specific educational process integrating peda-gogical prerequisites pointed out in CSCL literature.

To solve both tasks, students worked with the assis-tance of KnowCat at two collaborative levels: in smallgroup level in phase 1, and in class group level in phase2 (see Figure 5). The main aim of the students’ workwith KnowCat in phase 1 – ‘working in small groups’– was to elaborate an educational group interventionreport that responded to a real educational case pre-

sented in the task. The steps of the methodology of thecollaborative small group procedure in phase 1 werethe following:

(a) Each student wrote an individual report containingthe individual resolution of the real case and sub-mitted it as a document to the KnowCat platform.

(b) The other members of the small group read eachpeer’s report separately and annotated them – thatis, gave assistance – with a view to helping a fellowclassmate improve on them.

(c) The author of document then read the notes con-cerning his/her own report taking into account theclassmates’ notes and documents, rewrote his/herown document and submitted it back to the systemas a new document version. When students rewrotetheir documents, they could introduce ideas

Figure 5 Pedagogical Intervention and Use of the KnowCat System



included in their classmates’ reports because theobjective of rewriting the document was to elabo-rate collaboratively the best educational groupintervention for each real case.

(d) The members of each group voted for the bestreport which, in their opinion, contained theadequate response to a real educational case. Thenthe best socially acceptable report (calculated withthe KnowCat Knowledge Crystallization mecha-nism) was submitted to the system in the ‘classsection’ as a group report, and shared it withthe other (small) groups of the class. The educa-tional project presented the voting process as acollaborative action rather than a competitive one.At the beginning of the project, the students sharedthe common objective of collaboratively findingand arguing the best and adequate response to realproblems presented in each task by giving individ-ual feedback to their peers in the form of notes andvotes.

The aim of the students’ work with KnowCat in phase 2– ‘working with the whole class’ – was to decide whichsmall group report was, in the opinion of the classcommunity, the best educational intervention for eachreal case. The procedure in phase 2 was as follows:

(a) Students read all the reports submitted by the sixsmall groups and voted for the best one. Studentsargued their reasons in the voting process.

(b) KnowCat Knowledge Crystallization mechanismhelped select the best socially accepted documentof the whole class.

Instruments and data analyses

With a view to providing a solution to our first researchquestion – to study the students’ perception about themeaningfulness of the information provided by theKnowCat group awareness tools for effective collabo-rative learning, in terms of social and behaviouralawareness – at the end of the semester, awareness stu-dents group was administered a survey questionnaire tocollect data on their collaborative work with thesystem. This questionnaire had been previously testedin some of our research studies (Cobos, 2012; Cobos,Claros, & Moreno, 2009). The questionnaire wasdivided into three sections:

Section 1: Questions related with time invested in usingthe knowledge elements of the system such as docu-ments, versions and notes.Section 2: Students’ perception about how the differentknowledge elements (documents, notes, assessmentsand votes) promoted collaborative learning activities.Section 3: Students’ perception about each awarenesstool in relation with usability, presentation, frequencyof use, and usefulness in promoting collaboration andcollaborative learning activities. The development ofthe items of this section was based on work presentedby Gutwin and Greenberg (2002).

The questionnaire had a total number of 40 questions.The students answered the questionnaire using a4-point Likert scale in which students were asked touse a punctuation between 1 and 4 (1, minimum valueand 4, maximum). The answer format was adapted tothe content of the questions of the questionnaire. Forexample, the answer format for the frequency questionswas: 1 = never, 2 = rarely, 3 = occasionally, 4 = fre-quently; the answer format for quality questions was:1 = poor, 2 = barely acceptable, 3 = good, 4 = verygood. The percentages of the students’ answers werethen calculated for each question. Besides, the studentshad a blank area for comments and arguments on theiranswers.

Two examples of section 3 of the questionnaire areshown below:

The information displayed in the Participation Viewencourages you to contribute with a new element inKnowCat (submit a new note, new assessment or newdocument).1 = never; 2 = rarely; 3 = occasionally; 4 = frequentlyThe information displayed in the History View was(a) Usefulness1 = poor; 2 = barely acceptable; 3 = good; 4 = very good(b) Well-formatted presentation and easy to read1 = poor; 2 = barely acceptable; 3 = good; 4 = very good(c) Frequency of use during the collaborative resolution

of the task1 = never; 2 = rarely; 3 = occasionally; 4 = frequently(d) Useful for the collaborative resolution of the task1 = poor; 2 = barely acceptable; 3 = good; 4 = very good

As regards our second and third research questions – toanalyse the impact of the implementation of groupawareness tools in KnowCat on collaborative behav-iour and on the promotion of collaborative learning



activities compared with the use of the system withoutawareness tools – detailed content analyses of the notessubmitted to the system were carried out.

A coding scheme was used to study possible differ-ences in the content of the notes and in the learningactivities involved in writing the notes during thesolution of the problem-based task between theawareness and non-awareness groups. The codingscheme was based on the categories developed byVeldhuis-Diermanse (2002), which had been used insome of our previous studies (Pifarré, 2007; Pifarré &Cobos, 2009, 2010). The scheme established threegeneral types of learning activities (or categories) andnine subcategories: (1) cognitive activities – containingthe following three subcategories: debating ideas, usingexternal information and experiences, and linking orrepeating internal information; (2) metacognitive activ-ities – containing the following three subcategories:planning, keeping clarity and monitoring; and (3) affec-tive activities – containing the following three subcat-egories: general reaction, asking for general feedback,and chatting or social talk. Appendix I includes adescription and examples of each subcategory of thecoding scheme.

The coding process consisted of two steps: (a) divid-ing the messages into meaningful units, and (b) assign-ing a code to each unit. We decided to segment thenotes into units of meaning by using semantic featuressuch as ideas, argument chains, and discussion topics,or by regulating activities such as making a plan,asking for an explanation or explaining unclearinformation.

To ensure objectivity in the coding process, validityand reliability aspects were considered in the study.Two evaluators of our research group with experiencein this type of coding processes participated in thesegmentation and categorization process. In the firststep, both evaluators categorized 5% of the total notesseparately. In order to develop the coding rules andachieve reliability, from those notes which the evalua-tors categorized differently, a common view was nego-tiated. In the second step, both evaluators categorized25% of the total notes separately. The Cohen’s κ coef-ficient for both was as high as 0.92. The rest of thenotes were coded by one evaluator.

Finally, in order to answer our fourth research ques-tion – to examine the impact of the implementation ofgroup awareness tools in KnowCat on students’ per-

formance compared with the use of the system withoutawareness tools – both instructors of the course markedthe quality of the individual report that each studentrewrote after the annotation group collaborationprocess with KnowCat. Although this report wasrewritten and submitted individually, it had to includethe best ideas supplied by the members of the smallgroup – both in their individual report and in the notes– in an attempt to solve the task better (phase 1 ofFigure 5). Each instructor marked one of the two taskssolved in collaboration by using KnowCat because ofhis/her expertise on the topic of the report – a specificchild learning difficulty. Each report was marked usinga punctuation mark between 0 and 10 and both instruc-tors agreed on the indicators to assess the quality of thereports. These indicators referred to the adequacy ofthe subject contents and how students built adequateand convincing arguments to justify the educationalintervention proposed to solve the tasks.

The Mann–Whitney independent non-parametrictest (95% significance level) was run in order toanalyse whether there were statistical differencesbetween the awareness and non-awareness group inrelation with the variables: (a) learning activities (orcategories), namely cognitive, metacognitive, andaffective, and (b) task performance.

Results

Students’ perceptions about the meaningfulness ofvisualization of group awareness information

The questionnaire was answered by 17 out of 24 stu-dents of the awareness group. In the questionnaire,63% of students answered that visualizing their class-mates’ participation motivated them to contribute withknowledge in KnowCat (e.g., uploading documents,contributing with notes). Furthermore, students pointedout that frequently (76%) or occasionally (24%) com-pared their participation with that of their classmates.These results showed that knowing what and howmuch their classmates were contributing acted as posi-tive feedback which both encouraged students’ partici-pation and monitored their behaviour.

As regards the frequency of use of the group aware-ness console, calculated on a scale between 1 (never)and 4 (frequently), students reported an average of 2.5(Figure 6). The awareness tools most used by thestudents because of their useful information in



accomplishing the collaborative task were Participa-tion View, History View and Notes View. It shouldbe noted that all of these tools provided graphicalinformation about their classmates’ contributions.Kimmerle and Cress (2009) pointed out the potential ofa graphical approach in designing specific groupawareness tools.

Our study will take into consideration the weakpoints reported by students on such areas as usabilityand information presentation of the different awarenesstools (see Figure 6). Results showed then that theRadar View was the awareness tool less valued by thestudents. This result could be partially explainedbecause KnowCat is an asynchronous system that doesnot supply ways of communicating and interactingamong community members. Thus, knowing who andwhere a team partner is working online is not relevantin fulfilling the collaborative task.

The use of awareness tools in KnowCat encouragesstudents to give more feedback to their peerscompared with the use of the system withoutawareness tools

In order to know whether the use of group awarenesstools in KnowCat encourages students to give morefeedback to their peers by posting a higher number ofannotations on classmates’ reports, we compared thenumber of notes posted by the awareness group withthose posted by the non-awareness group (researchquestion 2). The awareness group posted a totalnumber of 559 notes (M = 23.29; SD = 9.16) whilethe non-awareness group posted only 371 notes(M = 17.52; SD = 4.71). The Mann–Whitney inde-pendent non-parametric test (95% significance level)was run in order to analyse whether there were sta-tistically relevant differences between both groups of

1

2

3

4

Usability Presenta on Frequency of use U lity

Registered users

Online users

Radar view

Par cipa on view

History view

Notes view

Figure 6 Students’ Opinion about the Awareness Tools Provided in the KnowCat System



students in relation to the number of notes posted. Nostatistically significant differences were found(p = 0.082). However, it should be noted that the dif-ferences between both groups were close to be statis-tically significant in favour of the awareness group. Inaddition, it should be considered that the awarenessgroup students reported in the questionnaire that theawareness information motivated them to participatein the system. It can therefore be concluded that thegroup awareness tools allowing students to visualizewhat, where, how much and how often other partici-pants contributed in the KnowCat knowledge area hada positive impact on the students’ collaborativebehaviour as it increased feedback.

The use of awareness tools in KnowCat promotes thedevelopment of better collaborative learningactivities compared with the use of the systemwithout awareness tools

In order to study the incidence of group awarenessinformation on the content of the notes and the learningactivities required for writing the notes between theawareness and non-awareness groups (research ques-tion 3), we analysed the content of the notes written bythe students during phase 1 – ‘working in small groups’(see Figure 5). To this end, we used the coding schemepresented in Instruments and Data Analyses sectionand shown in Appendix I.

As a new dependent variable, we calculated anindividual score of the proportion of affective, cogni-tive and metacognitive meaningful units. Thus, theindividual score of each student showed in which pro-portion he/she developed affective, cognitive andmetacognitive learning activities in his/her notes. As aresult, the proportion of affective, cognitive andmetacognitive learning activities reached 100% foreach student. This individual score compensated thefact that awareness group students posted more notesthan the non-awareness group.

Awareness group showed a higher number of cogni-tive and metacognitive learning activities than the non-awareness group. By contrast, the non-awarenessgroup presented a higher number of affective activitiesthan the awareness group. In order to study if therewere statistical differences between the awareness andnon-awareness groups in relation to their individualscore – indicating for each student the proportion of

cognitive, metacognitive and affective learning activ-ities developed during the project – we ran the Mann–Whitney independent non-parametric test (95%significance level). This test revealed that the meanvalues were statistically different in awareness andnon-awareness groups in affective units (p = 0.037)and nearly statistically significant in metacognitiveunits (p = 0.051) while cognitive units (p = 0.328) werestatistically irrelevant. Figure 7 represents the meanvalue of the dependent variable, that is, individualscore of the proportion of cognitive, metacognitive andaffective learning activities.

A detailed analysis of the cognitive learning activ-ities developed by both groups of students showed thatawareness group students showed a higher number ofsubcategories than non-awareness group, especiallywhen discussing ideas and linking internal information.However, such differences were not statistically rel-evant (Figure 8).

With regard to the detailed analysis of themetacognitive activities, the data revealed that theawareness group students developed a higher numberof subcategories, namely those related to planning andkeeping clarity (Figure 8). The Mann–Whitney inde-pendent non-parametric test revealed that the differ-ences between both groups in the subcategoriesincluded in the metacognitive category were statisti-cally relevant (planning, p = 0.02; keeping clarity,p = 0.028).

The data of subcategories included in the affectiveactivities revealed that the non-awareness groupshowed a higher number of subcategories than theawareness group specially those subcategories relatedto general reaction and chatting (Figure 8). The Mann–Whitney independent non-parametric test revealed thatonly the differences between both groups in the chat-ting subcategory were statistically relevant (p = 0.48).

The use of awareness tools in KnowCat enhancesbetter task performance compared with the use ofthe system without awareness tools

We analysed how the group awareness informationmight improve the students’ performance. To this end,the quality of the individual rewritten report submittedto the system by each student was evaluated (phase 1,Figure 5). The Mann–Whitney independent non-parametric test revealed that the difference between the



mean values of the quality of individual reports wasstatistically significant; awareness students solvedtasks better (non-awareness group: N = 23, M = 5.83,SD = 1.78; awareness group: N = 22, M = 8.19,SD = 1.25; p = 0.006). In this analysis, two awarenessgroup students did not finally submit the rewrittenreports despite having participated in the annotationprocess.

Discussion and conclusions

Students’ perceptions about KnowCat groupawareness information

Students reported that the information provided by theKnowCat group awareness tools increased the students’participation in online collaboration. These resultscoincide with previous research studies confirming thatindividuals orientate their own behaviour towards thatof the other members of the group (Cobos et al., 2009;Janssen et al., 2011; Kimmerle & Cress, 2009). Theresults also confirm that the behavioural awareness pro-vided by the KnowCat awareness tool of ParticipationView and History View, which indicates the classmates’participation in the collaborative knowledge area, had a

positive impact on the students’ behaviour. This, inturn, results in active and equal participation of thestudents to solve a collaborative task.

In addition, the KnowCat group awareness toolscreate opportunities for social comparison. Thus, thestudents pointed out that the KnowCat awareness toolsthey most frequently used were Participation View,History View and Notes View (Figure 6). This meansthat, by comparing themselves to other groupmembers, students were motivated to set themselveshigher standards for collaborative task resolution. Thisconfirms previous research on the positive effect ofpromoting social comparison by providing groupawareness information. This results in increased stu-dents’ motivation in the resolution of collaborativetasks and in higher participation rates (Michinov &Primois, 2005; Zumbach et al., 2006).

KnowCat group awareness information and thepromotion of students’ feedback

The results obtained in our research showed that theawareness group presented higher participation ratesthan the non-awareness group by posting a highernumber of annotations on classmates’ reports. In our

0

10

20

30

40

50

60

AFFECTIVE COGNITIVE METACOGNITIVE

PERCENTAGE

Learning Activities

No Awareness

Awareness

Figure 7 Mean Value on the Individual Score of the Proportion of Cognitive, Metacognitive and Affective Learning Activities inStudents’ Notes



work, the pedagogical use of KnowCat notes enhancedthe use of KnowCat notes as a tool to give explicit peerassistance in task resolution. Therefore, we can claimthat the awareness information helped better promotethe students’ involvement in a shared task in whichexplicit peer assistance emerged. These results coin-cide with those obtained in previous research studiesconcluding that awareness information has a positiveeffect on students’ collaborative behaviour and motiva-tion (Janssen et al., 2011).

KnowCat group awareness information and thepromotion of collaborative learning activities

Students’ higher participation of awareness groupwas aimed at cognitive and metacognitive actions tosolve a collaborative task better. It can also be claimedthat the behavioural group awareness tools imple-mented in KnowCat helped students change theircollaborative behaviour in favour of metacognitiveprocesses.

Cognitive activities have been described in educa-tional literature as the thinking processes to learn

content and to attain learning goals (Vermunt, 1998). Inthis study, the term cognitive category alluded to aseries of processes related to managing and supplyinginformation about the topic documents written by stu-dents. Accordingly, meaning units coded under thiscategory were strongly task oriented. One explanationfor the fact that cognitive activities were the most fre-quent learning activity in both experimental groupscould be found in the design of the collaborative task.Students were asked to help their group mates rewriteand improve their documents about a problem-basedactivity which in turn referred to a specific coursetopic. It seemed logical then that the students’ contri-butions revolved around discussing ideas, contributing,referring or repeating internal information about thecontent of a problem.

Nevertheless, it is group awareness information thathas a higher impact on metacognitive learning activ-ities. Group awareness students were aware of theimpact of planning skills. Furthermore, students regu-lated their peers’ problem-solving processes either byproviding alternative procedures or solutions or byreflecting on how to solve the task more effectively,

0

10

20

30

40

50

60

General reac on

Asking for general

feedback

Cha ng or social talk

Discussing ideas

Using external

informa on and

experiences

Linking or repea ng internal

informa on

Planning Keeping clarity

Monitoring

AFFECTIVE COGNITIVE METACOGNITIVE

PERCENTAGE

Learning Ac vi es Subcategories

No Awareness

Awareness

Figure 8 Mean Value on the Individual Score of the Proportion of the Subcategories of Cognitive, Metacognitive and AffectiveLearning Activities in Students’ Notes



which approach or procedure was the best one toaccomplish the task more effectively.

Another relevant result is the tendency of awarenessgroup students to be more active in monitoring theirunderstanding and strategy use by asking questions –keeping clarity subcategory. Previous research hadshown how asking each other questions about the task-resolution processes and self-questioning constitutesuccessful scaffolds in promoting the development ofmetacognitive skills (Kramarski & Gutman, 2005).

Research on metacognition has emphasized the viewof metacognition as an essential part of socially shareddiscussions. It proves that the others (both adults andpeers) play a major role in recent research onmetacognition seen as part of a collaborative learningsituation where metacognition regulation is also con-sidered as a group level activity rather than an indivi-dual’s performance (Goos, Galbraith, & Renshaw,2002; Zimmerman, 2000). From this perspective, thesocial context that supports and frames the learningtask becomes a core mechanism to understand thedevelopment of students’ self-regulation processesrelated to task definition, goal setting, planning, enact-ing and evaluation (Hadwin, Wozney, & Pontin, 2005).

The small-scale research project reported here pro-vides some evidence that awareness tools in KnowCatcan make a difference in the social and learning contextthat supports students’ collaborative behaviour. Thisbehaviour promotes responsibility and awareness ofthe processes involved in regulating learning and tasksand in promoting the use of scaffolds. These findingscoincide with those obtained in prior studies conclud-ing that awareness group information promoted regu-lation of mutual performance, co-reflection oncognitive performance, setting goals and formulatingplans to enhance group performance cognitive perfor-mance (Dehler et al., 2011; Fransen, Kirschner, &Erkens, 2011; Phielix et al., 2011).

However, in our study group awareness informationdid not stimulate social awareness. The awarenessgroup showed a lower percentage of affective learningprocesses compared with that of the non-awarenessgroup. This result can be justified in that the instruc-tional use of KnowCat was mainly task related whereasthe information provided in the different KnowCatgroup awareness tools was knowledge related. Accord-ingly, students were probably more focused on thecognitive and metacognitive aspects of their collabora-

tive work rather than on the social aspects of theircollaboration. Furthermore, the collaborative task waspart of a regular university course, while research inhigher education has reported that learning teams inhigher education tend to be pragmatic by focusing pri-marily on task aspects of performance and less on teamaspects (Fransen et al., 2011).

On the other hand, it can be argued that the socialpresence awareness information provided by KnowCatawareness tools – such as Online Users or Radar View– was sufficient to make all members of the small groupfeel like a learning community which gave them suffi-cient and accurate feedback to better solve the task.Nevertheless, we agree with previous research studiesthat emphasize the role of social awareness in promot-ing better group processes and performance. Thisaspect will be further pursued in our research agenda aswe believe the social awareness tools in KnowCatcould be better developed.

KnowCat group awareness information andstudents’ performance

It can be noticed that group awareness information hada positive impact on task performance. These resultsoutline an encouraging path in the field of CSCLbecause, in contrast to other previous studies, in ourresearch, the instructional use of KnowCat with groupawareness information helped students solve the taskbetter if compared with the use of KnowCat withoutgroup awareness information (Dehler et al., 2011;Janssen et al., 2011). However, these results must beinterpreted carefully as the sample size of the popula-tion considered in this study might be too limited toextend the results to a wider segment of population.

Furthermore, it should be noted that the nature of thecollaborative instructional design used in this studymay have had an influence on our results. There arethree characteristics of the instructional use ofKnowCat that might have increased the potentialitiesand the impact of group awareness information on col-laborative activities and performance. First, the CSCLenvironment was used in a long-term instructionalprocess – one semester; this extended time periodmight have been long enough for establishing acommon shared area. Second, students worked at twolevels of collaboration: small-group and class-groupcollaboration; the former enhanced collaboration and



increased participation and the latter, based on theKnowCat voting system and the innovative ‘Knowl-edge Crystallization’ mechanism of KnowCat,enhanced collaborative decision-making about the bestsolution to a real case in the community’s opinion.Third, the external guidance of course instructorsduring the process may have also contributed to devel-oping collaboration activities. Instructors developedtwo key guidance actions: on the one hand, they sup-plied information to students, at the beginning of thestudy, on the collaborative purpose of the tasks. Thiscoaching highlighted the importance of scaffoldinggroup partners during the learning processes to solvetasks better by means of joint and group effort; on theother hand, the instructors set the milestones to beaccomplished through group collaboration. The impactof the instructors’ external guidance on the students’collaborative tasks might have been increased inawareness group students because the milestones toaccomplish by the group were registered in the aware-ness tools provided in KnowCat – History View. Stu-dents were given graphical information comparing howthe group was actually contributing and how it shouldhave been contributing; this information stimulated themonitoring of the students’ collaborative behaviour.

Unsolved questions and further research

Nevertheless, some questions about group awarenesstools and their impact on collaborative learning activ-ities and outcomes still remain unanswered in ourstudy. First, our study aimed to examine how the inte-gration of different awareness tools in KnowCatincreased their potential for collaborative learning.However, we ignore which of the incorporated aware-ness tools had a bigger impact on the students’ collabo-rative processes and task performance. The students’responses to the questionnaire survey gave only generalinformation about their increased perception of themeaningfulness of Notes View, History View and Par-ticipation View as more useful awareness tools. Furtherresearch should be done to verify how and to whatextent each individual awareness tool can provide valu-able information for enhancing effective collaboration.

Second, our study only partially analysed how thestudents’ use of a specific awareness tool had an influ-ence on their collaboration. Our study displayed differ-ent types of awareness information and revealed to

what an extent students used it. However, no effort wasinvested on promoting reflection and co-reflection fromthis information. Providing group members with infor-mation on their cognitive and social behaviour is notenough to positively alter their behaviour. Groupmembers also need to process this information and askthemselves whether they understand, accept and agreewith the feedback (Phielix et al., 2011).

In view of this, our next logical step on this line ofresearch may probably focus on studying how studentsinterpret and reuse the awareness information given. Tothis end, it will probably be necessary to do furtheranalyses on how learners perceive process, interpretand use group awareness information. This futureresearch will probably make use of qualitative methodssuch as key event recall interviews in which studentscan explicitly explain such aspects as: whether or notthey understood the tool, how they valued the tool, theirmotives for using the tool, their interpretation of theinformation given, how they used the awareness infor-mation to improve collaboration (Beers, Boshuizen,Kirschner, Gijselaers, & Westendorp, 2008).

Third, our study showed that providing awarenesstools had a positive impact on the students’ task per-formance. However, the sample size considered in thisstudy is too small to allow us to draw a more generalconclusion. A larger sample size should be consideredto obtain further data that can confirm or dismiss this,so far, encouraging positive effect on learning.

Fourth, we agree with Phielix et al. (2011) that mostCSCL environments – as KnowCat – focus on support-ing cognitive or task-related processes and provide fewpossibilities to exchange socio-emotional or affectiveinformation through other channels of communicationrather than text that might better enhance to form, buildand maintain crucial social relationships for collabora-tive learning.

With a view to strengthening social group aware-ness, research is currently being done on designing andexperimenting with three new awareness tools inKnowCat: (a) a Chat Service, which will allow users toexchange comments with an online community; (b) aCommunity Wall Service, which will allow users toshare, by means of posts, their own ideas and concernsin a common shared area (this service was inspiredby the Facebook wall posts mechanism). It is expectedthat these two new tools will promote, among KnowCatusers, a better and easier way to communicate and



interact with other members of a community, a needindicated by awareness group students; and (c) theMotivation Booster, which provides users with feed-back information about the progress of their contribu-tions in KnowCat comparing them against themilestones agreed on by the community and givessome recommendations about how users can improveand adjust their behaviour to achieve efficient collabo-ration. The main aim of this service is to motivatestudents to work regularly with the KnowCat platformthrough motivational messages (recommendation andcongratulation messages).

Besides, the Motivation Booster provides theinstructors with a valuable synthesis report of updatedstudents’ work in KnowCat. Thus, the instructor canexamine possible students’ difficulties to reach col-laborative objectives and decide if he/she has to inter-vene in the community in adjusting common learningobjectives. In addition, the instructor can decide ifhe/she has to help individual students to better contrib-ute to the community.

To sum up, the awareness information helped stu-dents orientate, monitor and regulate their collaborativeprocesses. Our future research will focus on applyingthis exploratory result on a larger study group andassessing fully the students’ work using the KnowCatsystem and, also, employing qualitative methods toverify how the use of awareness information improvedtheir collaborative processes and group performance.

Acknowledgements

This research was partly funded by the SpanishNational Plan of R + D, project numbers: EDU2009–11656, EDU2012–32415, TIN2011–24139, and by theCAM (Autonomous Community of Madrid), projectnumber S2009/TIC-1650.

References

Alamán, X., & Cobos, R. (1999). KnowCat: A web applica-tion for knowledge organization. In P. P. Chen (Ed.),Lecture notes in computer science 1727 (pp. 348–359).New York: Springer.

Beers, P. J., Boshuizen, H. P. A., Kirschner, P. A., Gijselaers,W., & Westendorp, J. (2008). Cognitive load measure-ments and stimulated recall interviews for studying theeffects of information and communications technology.Educational Technology Research and Development, 56,309–328.

Bodemer, D., & Dehler, J. (2011). Group awareness in CSCLenvironments. Computers in Human Behaviour, 27, 1043–1045.

Briggs, R. O. (2006). On theory-driven design and deploy-ment of collaboration systems. International Journal ofHuman-Computer Studies, 64, 573–582.

Buder, J. (2011). Group awareness tools for learning: Currentand future directions. Computers in Human Behaviour, 27,1114–1117.

Carroll, J., Neale, D., Isenhour, P., Rosson, M., &McCrickard, S. (2003). Notification and awareness:Synchronizing task-oriented collaborative activity.International Journal of Human-Computer Studies, 58,605–632.

Cobos, R. (2003). Mechanisms for the crystallisation ofknowledge, a proposal using a collaborative system (Doc-toral dissertation). Universidad Autónoma de Madrid.

Cobos, R. (2012). Knowledge crystallisation supported bythe KnowCat system. In H.-T. Hou (Ed.), Knowledgemanagement/book 3 (pp. 185–200). Croatia: Intech OpenAccess Publisher. ISBN 979-953-307-744-3.

Cobos, R., Claros, I., & Moreno, J. (2009). A proposal ofawareness services for the construction of quality commu-nity knowledge supported by the knowledge managementsystem KnowCat. Proceedings of International Confer-ence Human-Computer Interaction, HCI 2009. San Diego,July, 2009. pp. 365–374.

Dehler, J., Bodemer, D., Buder, J., & Hesse, F. W. (2011).Guiding knowledge communication in CSCL via groupknowledge awareness. Computers in Human Behavior, 27,1068–1078.

Fransen, J., Kirschner, P., & Erkens, G. (2011). Mediatingteam effectiveness in the context of collaborative learning:The importance of team and task awareness. Computers inHuman Behavior, 27, 1103–1113.

Goos, M., Galbraith, P., & Renshaw, P. (2002). Sociallymediated metacognition. Creating collaborative zones ofproximal development in small group problem solving.Education Studies in Mathematics, 49(2), 193–223.

Gross, T., Stary, C., & Totter, A. (2005). User-centeredawareness in computer-supported cooperative work-systems: Structured embedding of findings from socialsciences. International Journal of Human-ComputerInteraction, 18(3), 323–360.

Gutwin, C., & Greenberg, S. (2002). A descriptive frameworkof workspace awareness for real-time groupware. Com-puter Supported Cooperative Work, 11, 411–446.

Hadwin, A. F., Wozney, L., & Pontin, O. (2005). Scaffoldingthe appropriation of self-regulatory activity: A socio-cultural analysis of changes in teacher-student discourseabout a graduate research portfolio. Instructional Science,33, 413–450.



Hakkarainen, K., Lipponen, L., & Järvelä, S. (2002). Epis-temology of inquiry and computer-supported collaborativelearning. In T. Koshman, N. Miyake, & R. Hall (Eds.),CSCL2: Carrying forward the conversation. Mahwah, NJ:Erlbaum.

Janssen, J., Erkens, G., & Kirschner, P. (2011). Group aware-ness tools: It’s what you do with it that matters. Computersin Human Behaviour, 27, 1046–1058.

Jiang, L., Elen, J., & Clarebout, G. (2009). The relation-ships between learner variables, tool-usage behaviour andperformance. Computers in Human Behavior, 25, 501–509.

Kimmerle, J., & Cress, U. (2009). Visualization of groupmembers’ participation. Social Sciences ComputerReview, 27(2), 243–261.

Kramarski, B., & Gutman, M. (2005). How can self-regulatedlearning be supported in mathematical elearning environ-ments? Journal of Computer Assisted Learning, 22(1),24–33.

Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identi-fying the pitfalls for social interaction in computer-supported collaborative learning environments: A reviewof research. Computers in Human Behaviour, 19, 335–353.

Michinov, N., & Primois, C. (2005). Improving productivityand creativity in online groups through social comparisonprocess: New evidence for asynchronous electronic brain-storming. Computers in Human Behaviour, 21, 11–28.

Phielix, C., Prins, F., Kirschner, P., Erkens, G., & Jaspers, J.(2011). Group awareness of social and cognitive perfor-mance in a CSCL environment: Effects of a peer feedbackand reflection tool. Computers in Human Behaviour, 27,1087–1102.

Pifarré, M. (2007). Scaffolding through the network: Analys-ing the promotion of improved online scaffolds among

university students. Studies in Higher Education, 32(3),389–408.

Pifarré, M., & Cobos, R. (2009). Evaluation of the develop-ment of metacognitive knowledge supported by theKnowCat system. Educational Technology ResearchDevelopment, 57(6), 787–799.

Pifarré, M., & Cobos, R. (2010). Promoting metacognitiveskills through peer scaffolding in a collaborative computer-based environment. International Journal of Computer-Supported Collaborative Learning, 5(2), 237–253.

Romero, M. (2011). The collaborative learning activityreflect on the computer learning environment and itsimpact on. In R. Wegerif (2001) (Ed.) Education for aglobal networked society. Proceedings of EARLI Confer-ence 2011, Book of abstracts and extended summaries. pp.1091–1092.

Scardamalia, M., & Bereiter, C. (1994). Computer supportfor knowledge-building communities. The Journal of theLearning Sciences, 3, 265–283.

Veldhuis-Diermanse, A. E. (2002). CSCLearning? Participa-tion, learning activities and knowledge construction incomputer-supported collaborative learning in higher edu-cation (Doctoral dissertation). Veenendaal, WageringenUniversity.

Vermunt, J. D. (1998). The regulation of constructive learn-ing processes. British Journal of Educational Psychology,68, 149–171.

Zimmerman, B. (2000). Attaining self-regulation. A socialcognitive perspective. In M. Boekarts, P. Pintrich, & M.Zeidner (Eds.), Handbook of self-regulation. San Diego:Academic Press.

Zumbach, J., Reimann, P., & Koch, S. (2006). Monitoringstudents’ collaboration in computer-mediated collaborativeproblem-solving: Applied feedback approaches. Journal ofEducational Computing Research, 35(4), 399–424.



Appendix I: Coding Scheme

Categories Subcategories Subcategory description Subcategory example

Cognitiveactivities

Discussingideas

Students raise a problem, reason outa solution, make an evaluation ofan idea contained in the work of aclassmate. Students contribute tocreate a shared understanding ofthe problem.

– I agree that there are aspects that relateto the contents of the subject article . . .

– I liked the use of conceptual maps andtables to make understandable the mainconcepts of the article.

– I think that your conclusions are goodand clear.

Using externalinformationandexperiences

Students contribute with newinformation sources outside thespeech-based or with related andrelevant experiences.

– . . . To do that, you can rely on thetheoretical framework presented in thedocument of our colleague John.

– I think that you should cite Shuell (1996)in his introduction about . . .

Linking orrepeatinginternalinformation

Students repeat or link ideas alreadycontained in the dialogue.

– You considered necessary aspects of thepractice and the evaluation of . . .

– The use of oral language is veryimportant to reach a consensus amongthe different members of the group, asdialogic language is a mediator.

Metacognitiveactivities

Planning Students present or ask for anapproach or procedure to carry outthe task.

– You should add a scheme withinformation on the following points.

– You could divide your explanation indifferent sections to improve thepresentation of your ideas.

Keepingclarity

Students ask for an explanation,synthesis of information,clarification or illustration.

– You say that to carry out this experimentin Primary Education more teacherswould be required. Do you think that thisis the only point to keep in mind?

– Could you provide examples?Monitoring Students remember the original

planning or aim. The studentsmention the work done by theirclassmates and suggest ways ofimproving on it.

– You’ve taken a good scheme, but youcould improve your contribution if youadd another column in which you tackleon.

– If you work in Secondary Education, whatideas of this article would you use?

Affectiveactivities

Generalreaction

Students emotionally react towardstheir classmates’ work and give apositive general opinion.

– First of all, congratulations on your work.– Good job!

Asking forgeneralfeedback

Students ask for general impressionsor views to their classmates.

– Do you think that I propose adequaterecommendations?

– What would you do in my shoes?– Let me know what do you think about

my new ideas.Chatting or

social talkStudents make notes of an informal

or social nature to develop positiveaffective relationships, groupcohesiveness and trust.

– Hi, how are you?– Don’t lose heart!– Nothing more, it continues as well!



Documents

Incidence of group awareness information on students' collaborative learning processes