Davinia Hernández-Leo, Eloy D. Villasclaras,Davinia Hernández-Leo, Eloy D. Villasclaras,Juan I. Asensio-Pérez, Yannis DimitriadisJuan I. Asensio-Pérez, Yannis Dimitriadis

GSIC / EMIC groupGSIC / EMIC groupUniversity of Valladolid, SpainUniversity of Valladolid, Spain

TENCompetence Workshop

Barcelona, June 22nd 2007 http://gsic.tel.uva.es

Some findings Some findings from the evaluation of from the evaluation of Collage authoring toolCollage authoring tool

2

Our approach (I)Our approach (I)

ChallengesChallenges1. Designing scripts that potentially elicits the

desired interactions

How can How can teachersteachers create “ create “potentially effectivepotentially effective” ” ““computer-interpretablecomputer-interpretable” collaboration scripts ” collaboration scripts according to the necessities of their situation?according to the necessities of their situation?

3. Hiding computational representations of scripts to teachers

2. Formalizing the scripts so that they are computer-interpretable

3

Our approach (II)Our approach (II)

1. Not easy to design a script that potentially elicits the desired interactions

Reusing the “essence” of commonly used CSCL scripts (formulated as patterns)CLFPs, Collaborative Learning Flow Patterns:Broadly accepted techniques repetitively used by practitioners when structuring the flow of types of (collaborative) learning activities (E.g. Jigsaw, Pyramid (Snowball) CLFPs)

4

Our approach (IV)Our approach (IV)

2. Formalizing the scripts so that they are computer-interpretable

IMS Learning Design (LD)IMS Learning Design (LD)

5

Our approach (V)Our approach (V)

3. Computational representations are not familiar to educators

Integrate in authoring tools “a pattern-based design process for CSCL macro-scripts

computationally represented with IMS LD” which allows teachers to create their own computer-interpretable script by reusing CLFPs as visual LD templates that can be assembled and refined

COLLAGECOLLAGE (COLaborative LeArning desiGn Editor)(COLaborative LeArning desiGn Editor)

(Based on Reload Editor. IMS-LD level A compliant)

6

Evaluation methodology (I)Evaluation methodology (I)

Multicase studyMulticase study (Stake, 2005)(Stake, 2005)

Authentic experiences with different audiences (contextual issues, achievement Authentic experiences with different audiences (contextual issues, achievement of educational benefits, impact of software systems)of educational benefits, impact of software systems)

Cross-case analysisCross-case analysis

Not aiming at accurate generalizations but at valid conclusionsNot aiming at accurate generalizations but at valid conclusions

Mixed method combining qualitative and quantitative data gathering Mixed method combining qualitative and quantitative data gathering

techniquestechniques (Martínez et al., 2003)(Martínez et al., 2003)

Triangulation: Comparative analysis of the data obtained with the different Triangulation: Comparative analysis of the data obtained with the different techniquestechniques

7

Evaluation methodology (II)Evaluation methodology (II)

Multicase studyMulticase study (Stake, 2005)(Stake, 2005)

QUITAIN: The proposed pattern-based design process for CSCL macro-scripts computationally represented with IMS LD

ISSUE: Does the design process implementedin Collage facilitate the reuse of CLFPs

in the creation of particularized LD-represented CSCL scripts in a way

that allows teachers to focus on the CSCL critical elements?

Creating CSCL scripts based on

CLFPs using Collage

ISSUE: Can we useCollage for creating a script representing a scenario proposed

by a third-party?

Solving a third-party scenario

ISSUE: Can we use CSCL scripts

created with Collage in real situations ?

Putting into practice a CSCL

script created with Collage

“Collage workshops” Case Study

Lab. uni. of Valladolidand Cádiz, Spain

Hands-onsessions

Mixed method Questionnaires

ObservationsFocus groups

Mini-cases

“Planet game” Case Study

ICALT conference

Pre-workPresentingDiscussing

Achieved CSCL script, papers, discussion

“Network Managment” Case Study

Two f2f anda distancesession

Lab. Uni. OfValladolid

Mixed method

8

Evaluation methodology (III)Evaluation methodology (III)

Two main experiences with the target audienceTwo main experiences with the target audience (teachers with interest in CL and CS)(teachers with interest in CL and CS)

University of Cádiz, Spain [University of Cádiz, Spain [UCAUCA], ], University of Valladolid, Spain [University of Valladolid, Spain [UVAUVA]]

Creation of the script (for Network Management course)Creation of the script (for Network Management course)

Two smaller experiencesTwo smaller experiences (mini-cases): (mini-cases): CSCL practitioners [CSCL practitioners [GSIC/EMICGSIC/EMIC], ],

Educational technologist [Educational technologist [UNFOLDUNFOLD]] Several different scriptsSeveral different scripts

DataData: Scripts, questionnaires, observations, discussion : Scripts, questionnaires, observations, discussion groupsgroups

9

Sample creation of a script (I)Sample creation of a script (I)

““CTM2” script CTM2” script (applied in the “Network Management” case (applied in the “Network Management” case study)study)

Optional undergraduate course on Network Management Optional undergraduate course on Network Management technologiestechnologies

TeacherTeacher

I want to design a collaboration script that guides the students in the

collaborative understanding of a collaborative understanding of a complex long technical papercomplex long technical paper that that

can be divided into 3 different can be divided into 3 different sectionssections (3 versions of a network (3 versions of a network management protocol). I want the management protocol). I want the

students to discuss and reach students to discuss and reach agreement on the main ideas of the agreement on the main ideas of the

paper…paper…

(Evaluation methodology)(Evaluation methodology)

10

Sample creation of a script (II)Sample creation of a script (II)

Checking educational Checking educational benefits, types of benefits, types of problems, complexityproblems, complexity

Selecting the CLFPs

Reading information and Reading information and examplesexamples

(Evaluation methodology)(Evaluation methodology)

11

Authoring a CLFP-based

LD

Combining the Combining the CLFPsCLFPs

Sample creation of a script (III)Sample creation of a script (III)(Evaluation methodology)(Evaluation methodology)

12

Sample creation of a script (IV)Sample creation of a script (IV)

Authoring a CLFP-based

LD

(Evaluation methodology)(Evaluation methodology)

Refining the Refining the

CLFPsCLFPs

13

Evaluation methodology (IV)Evaluation methodology (IV)

ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements?

Pattern-Pattern-based based design design processprocess

Focus on Focus on CSCL CSCL critical critical elementselements

Use of Use of CollageCollage

Characteristics of Characteristics of the potential the potential audienceaudience

Four topics of interestFour topics of interest

14

Evaluation resultsEvaluation results

Pattern-based Pattern-based design processdesign process

Is the selection of the Is the selection of the CLFP-based LD templates CLFP-based LD templates and their representation and their representation useful and satisfactory?useful and satisfactory?

Does the design process achieves a satisfactory trade-off Does the design process achieves a satisfactory trade-off between the reuse of the CLFPs and the creation of between the reuse of the CLFPs and the creation of scripts contextualized according to the situational needs?scripts contextualized according to the situational needs?

Support data

“… a minimum formation of patterns is necessary, for which Collage is helpful [UVA-focus]”

Selection phase is critical and promotes the understanding of the patterns

“Collage systematizes the selection of the patterns [UCA-quest-final]”

All the GSIC/EMIC, 5 UNFOLD, 3 UVA and 10 UCA partici-pants agree on that the CLFPs are significant. 1 UVA and 3 UCA interviewees state that they are other relevant techniques but these are OK. The rest (2 UNFOLD, 1 UVA and 1 UCA participants) indicate that they do not know any CL technique.

The CLFPs are significant but there are more well-known strategies (e.g. for assessment)

Support data

“… I miss a pattern that I employ in my classes, which is randomly selecting a member of a groupto explain what the group has done [UVA-quest-final]”

The interviewees’ ideas of the CLFPs coincide with what is presented in Collage

Support data

“… I think that the patterns are perfectly transferredto the user workspace, reproducing the needed roles and activities for their execution [GSIC/EMIC]”

The combinations of patterns provides design flexibility

Support data

“… it would be nice if other patterns could be added.However, as patterns can be combined, thesealready offer quite a lot of flexibility [UNFOLD]”

Support data

“… no example is directly transferable…”“… the examples are useful especially if results areavailable… as in a community of practice…”[UVA-focus]

Pattern-based templates are probably more useful in the processof customizing a new situation thanready-to-run templates, but completeexamples are also useful

Support data

“… no example is directly transferable…”“… the examples are useful especially if results areavailable… as in a community of practice…”[UVA-focus]

Satisfactory trade-off between flexibility, keeping the essence captured in the CLFPs, hiding LD-specific technologicaldetails and providing a clear (but limited) set of design options

“… it saves the teacher a lot of specification workload [UVA-quest-final]”“I do not find the process too contrained since it enablesthe combination of patterns… and I think that it is easierfor a novice to have an already structured model [UVA-focus]”

Support data

15

Evaluation resultsEvaluation results

Use of Use of CollageCollage

Can the teachers use successfully Collage?Can the teachers use successfully Collage?

How can CollageHow can Collagebe improved?be improved?

Support data

Most of the participants find Collage user-friendly and intuitive and are able to create partly completed examples during the workshops. Very few problems appear by they point out the importance of applying the scripts in real situations.

“I am satisfactorily astonished by the user-friendliness of the tool; I thought it was going to be more difficult [UCA-quest-final]”“.. the participants create the designs without difficulties… [UVA-observer]”

Collage is easier to use, specific to CL andit is the first IMS-LD editor providing pattern-based templates

“… at last, patterns in practice!” “… less complicated and more intuitive than Reload on its own…”“makes Reload much easier to use for specific purposes”[UNFOLD]

Support data

Incorporating more CLFPs and other types of patterns (also non-pattern based elements), which would provide more flexible design options. (Level B and C of IMS-LD) “Preliminary view” of the script. The integration of Collage with instantiation and delivery systems should support flexibility.

“… it may happen that advanced users cannot specifythings that they would like [UVA-quest-final]”

“… the incorporation of activities that are not part of a pattern but that may be complementary [GSIC/EMIC]”“… if some students abandon the course in the third

session, then I need to re-structure [UVA-focus]”

Support data

16

Evaluation resultsEvaluation results

CharacteristicsCharacteristicsof the potential of the potential audienceaudience

[UCA-focus-group]

Which are the Which are the characteristics andcharacteristics andmotivations of themotivations of thepotential audiencepotential audienceof Collage?of Collage?

-Interested in designing CL processes to be used with an LMS in f2f, distant or blended situations-teachers, novice or with experience in CL-some experience and interest in the use of ICT,but it is not necessary to be expert (IMS-LD) technologist

“… it is going to be necessary to use this type of tools in our educational system. The EEES will also push to use these tools [UCA-quest-final].”

“… for non-experts on CL this is really an eye-opener [UNFOLD]”

Support data

17

Discussion (I)Discussion (I)

A systematic multicase study A systematic multicase study Focus on Focus on Collage workshopsCollage workshops (2 experiences (2 experiences

with target audience and 2 mini-cases), using with target audience and 2 mini-cases), using a mixed evaluation methoda mixed evaluation method

Other two cases:Other two cases:

ICALT 2006 workshop (third-party scenario)ICALT 2006 workshop (third-party scenario)

Experience with students in a real situation Experience with students in a real situation (Hernández-Leo et al., IEEE Transactions on (Hernández-Leo et al., IEEE Transactions on Education, to be published in November 2007)Education, to be published in November 2007)

18

Discussion (II)Discussion (II)

Positive results with respect to:Positive results with respect to: Reuse and contextualizationReuse and contextualization Focus on CSCL critical elementsFocus on CSCL critical elements Support of practitioners (teachers)Support of practitioners (teachers)

But also pointers for several But also pointers for several enhancements and new research effortsenhancements and new research efforts

http://gsic.tel.uva.es/collagehttp://gsic.tel.uva.es/collage

Davinia Hernández-LeoDavinia Hernández-Leodavher@tel.uva.esdavher@tel.uva.es

20

Evaluation resultsEvaluation results

Focus on CSCLFocus on CSCLcritical elementscritical elements

Does the design process:Does the design process:help to determine the help to determine the learning objectives, task-type and expected interactionlearning objectives, task-type and expected interaction that will be developed? that will be developed?

help to understand and determine the help to understand and determine the structure regarding the structure regarding the flow of activities and hierarchy flow of activities and hierarchy of groupsof groups??

support also the definition of support also the definition of group-size, resource distribution, computer support and structure within group-size, resource distribution, computer support and structure within activitiesactivities?? The “selection phase” helps to determine

the objectives related to CL that will be promoted and to select the task-type that will be solved by the students

Support data“… it helps to determine the CLFP more suitable to foster particular objectives [UVA-quest-final]”“… it helps to envisage how the interactions willdevelp [UCA-quest-final]”

Many arguments support that the design process is very satisfactory concerning this element

Support data“I think that the most important aspect of Collage is that ithelps to structure and systemtize the different types of activities… Collage has helped me to structure what I already do but I do not organize well [UCA-focus]”

Support data“… it hightlights the interrelations among activities”“… the structure of the (learning) process helps to check the roles that each group plays in each moment” [UCA-quest-final] “… Collage helps to conceptualize the groups [UVA-focus]“

I helps to specify these elements, but it doesNot provide suggestion of recommendedcontent or tools and the eventual structure within activities is textually-defined (or supported by tools)

“… Since Collage guides the design process…, we are pushed to reflect on each element that shapes the activity [UVA-quest-final]”“the design implies the previous determination of allthese elements and their organization [UCA-quest-final]”

Support data