Evalueren van Groupware

J.H. Erik Andriessen

Technische Universiteit Delft

Examples of ICT supported groups

• Teams of consultants in Germany

• Aviation design teams

• Home nursing in Sweden

• Unilever communities

• Ambulance ‘teams’

New work arrangements

Virtual offices

–Telework / Telecommuting

–Hotdesking / Hoteling

–Home workers

–Mobile work

Virtual groups

–Virtual fixed teams

–Virtual mobile teams

–Virtual communities

Developments in work settings

• Many modern organisations are “networking organisations”. – Many problems/tasks require people with different skills and experiences– Workforces are more distributed, and organizations are becoming more

“virtual”– Much work is done in groups– Groups often work inefficient

• Dispersed cooperation is possible, but to make such teams effective, knowledge is needed about: – the functioning of groups in general, – virtual groups, and – the possibilities of ICT support

Virtual teams

So: Virtual teams have advantagesBut: Virtual teams have also problems:

– Interpersonal communication is more difficult : No non-verbal signals; no unplanned encounters; no context awareness

– And therefore : Difficulty in all kinds of group processes: collaboration, coordination, developing trust, exchanging knowledge

– ICT tools have to match with the type of group and its task; And they have to be supportive for the group processes

Global virtual teams

GVT’s have even more problems:• Time zone differences • Language differences \ Cultural differences

Research suggests: In individualistic cultures (EU, USA) people prefer direct

expression of opinions; In virtual groups they prefer synchronous communication, through telephone, video and chat.

In collectivist cultures (Asia, Africa) people are sensitive to non-verbal signals and group relations. In virtual groups people prefer asynchronous communication, to be able to express themselves more carefully: e-mail

What is “Groupware?”• Tools (hardware, software, processes) that support

person-to-person collaboration• This can include e-mail, bulletin boards,

conferencing systems, decision support systems, video and workflow systems, etc…

• Some common groupware acronyms:– Group Support Systems (GSS)– Group Decision Support Systems (GDSS)– Electronic Meeting Systems (EMS)– Bulletin Board Systems (BBS)– Group Collaboration Systems (GCS) – Computer-Supported Cooperative Work (CSCW) systems

Three types of interaction

All collaboration technology always implies:• 1. Human - computer interaction• 2. Human - database interaction: (information

seeking) • Internets• Intra-net• Group networks

• 3. Mediated interpersonal interaction (communication)

A Simple Classification of Groupware (adapted from Johansen, 1991)

Time

Location

Same Different

Same

Different

GDSS; Support for FtF Meetings,

Email, Bulletin board,

Comp. Conf

Teleconferencing,Instant Msg

Chat, Whiteboard,Video

Email, Bulletin board,

Comp. ConfWeb based CS

A GDSS Example

Video Conferencing

Groupware system

Five Basic group processes

communication

co-operation co-ordination

learning byknowledgesharing

social interactionteam building

Outcomes

Organisationaloutcomes

Group vitality

Individualrewards

Processes

communication

cooperation coordination

learningsocialinteraction

Individual interpretation and performance

Organisational environment Changes in organisational setting

Emerging structures

Groupcharacteristics

Technology

Persons Task

Formal structure Reflection

LearningAppropriation

Lifecycles

Group

Culture

Group

Physical setting

Dynamic Group Interaction model

Basic Principles

• The effectiveness of a group can be expressed in terms of three types of outcomes, i.e. (quality and quantity of the )products, individual ‘rewards’ and vitality of the social relations.

• Effectiveness depends on the quality of the individual preformance and six group processes, which have to match

• The quality of the group processes depends on the support of six conditions, and on the interaction with the environment.

• The six aspects of the context-of-use have to fit to each other.

• Groups develop and tools become adopted and adapted to, through interaction processes and feedback.

SUPPORT – MATCH – ADAPTATION

To evaluate the role of groupware tools.

Evaluate• Technical properties• Degree of fit to task users, group, setting,

other characteristics• Degree of support for processes • Effects on outcomes• Possibilities to adapt

Organisationaloutcomes

Group vitality

Individualrewards

communication

cooperation coordination

learningsocialinteraction

TOOL

Persons Task

Formal structure Reflection

LearningAppropriation

Lifecycles

Group

Culture Physical setting

Evaluation issues in more detail1. Describe the tool characteristics: reliability, portability,

maintainability, network performance, costs, infrastructural quality, security/privacy and evaluate whether this is adequate (ISO-9126)

2. Describe the functionalities3. Analyse the task and evaluate whether the functionalities fit the task 4. Analyse the users and evaluate whether the tool fits the users (usability)5. Analyse the group (structure, culture, setting) and evaluate whether the tool fits

the group6. Evaluate whether the tool supports (or at least does not hinder) the group

processes: communication, co-operation, co-ordination, learning, social interaction.

7. Evaluate whether the tool contributes to (or at least does not hinder) individual, group, organisational outcomes.

8. Evaluate to which extend the tools can be adapted to learning and new uses

Evaluation principles

• - Evaluation should be integrated in the design process from the very beginning.

• - The design and the evaluation process should be iterative and stakeholder centred; critical success factors of stakeholders should be formulated.

• - Evaluation can take place at three periods in the design life cycle – a. Concept evaluation

– b. Prototype evaluation

– c. Operational assessment

Analysis of task, contexttechnology, stakeholders

Development of Future Usage Scenarios

Identification of Stakeholders’success factors

Choice of Evaluation approach

Iterative tests of prototypes

Designrequirements

Iterative design and implementation

Identification of technologicaland organisational options

Use, experience, adaptation of new system

ConceptEvaluation’

Prototype Evaluation

OperationalEvaluation

Analysis of potential impacts

Lessons learned (1)

1.     Groupware is part of a social system. Design not for a tool as such but for a new socio-technical setting.

2.    Design for several levels of interaction, i.e. for user friendly human computer interaction, adequate interpersonal communication, group co-operation and organisational functioning.

3.    Design in a participative way, i.e. users and possibly other stakeholders should be part of the design process from the beginning.

4.   Analyse carefully the situation of the users. Success of collaboration technology depends on the use and the users, not on the technology. Introduction should match their skills and abilities, and also their attitudes, otherwise resistance is inevitable.

5. Analyse carefully the context, since success of collaboration technology depends on the fit to that context. The more a new setting deviates from the existing one the more time, energy and other resources should be mobilised to make it a success.

Lessons learned (2)

6. Introduce the new system carefully. Apply proper project management, find a champion, try a pilot, inform people intensively

7. Train and support end-users extensively

8. Measure success conditions and success criteria before, during and after the development process. Only in this way you can learn for future developments.

9. Plan for a long process of introduction, incorporation, evaluation and adaptation. Groupware is not a quick fix.

10. Despite careful preparations groupware is appropriated and adapted in unforeseen ways. Keep options open for new ways of working with the groupware, because this may result in creative and innovative processes.

Evaluation methods

1. Inspection methods

• Heuristic Evaluation:

2. Performance analysis

• Human Reliability Analysis

3. Behaviour analysis

• Diagnostic Recorder for Usability Measurement (DRUM)

4. Effort and satisfaction

• NASA-Task Load IndeX (NASA-TLX)

• Measuring the Usability of Multi-Media Systems (MUMMS)

• MultiMedia Communication Questionnaire (MMCQ)

Evaluation Methods (2)

5. Task aspects and relations

• Extended Delft Measurement Kit

6. Network performance

7. System usage and interaction registration

• Automatic registration of the use of the system

• Coding schemes for communication content

Person Task

Interpretation : •personal goals / task and perceived usefulness of tools •social relations and norms •perceived ease of use of tools and perceived situational constraints

Organisationalenvironment Outcomes

Tool

Setting Motivation to act and to choose and use tools

Tool Choiceand Use

Task Performance

Situational constraints

ReflectionLearning

Appropriation

Diversity Hypothesis

Fully Dispersed Three Subgroups Two Subgroups

Most ConflictLeast Trust

Least ConflictMost Trust