1

Sundar Gopalakrishnan, Guttorm Sindre, and John Krogstie:Adapting UML activity diagrams for mobile work process modelling: Experimental comparison of two notation alternatives

PoEM 2010 Delft, 9.-10. Sept.

2

Agenda

• Motivation• Background• Research method• Results• Discussion of threats to validity• Summary and further work

3

Traditional perspectives to modeling• Structural• Behavioral• Functional• Goal and rule-oriented• Object-oriented• Social communication• Actor/role-oriented

What about location/place/space... ?

4

Zachman Framework

21e.g. DATA

ENTERPRISE ARCHITECTURE - A FRAMEWORK

Builder

SCOPE(CONTEXTUAL)

MODEL(CONCEPTUAL)

ENTERPRISE

Designer

SYSTEMMODEL(LOGICAL)

TECHNOLOGYMODEL(PHYSICAL)

DETAILEDREPRESEN- TATIONS(OUT-OF- CONTEXT)

Sub-Contractor

FUNCTIONINGENTERPRISE

DATA FUNCTION NETWORK

e.g. Data Definition

Ent = FieldReln = Address

e.g. Physical Data Model

Ent = Segment/Table/etc.Reln = Pointer/Key/etc.

e.g. Logical Data Model

Ent = Data EntityReln = Data Relationship

e.g. Semantic Model

Ent = Business EntityReln = Business Relationship

List of Things Importantto the Business

ENTITY = Class ofBusiness Thing

List of Processes theBusiness Performs

Function = Class ofBusiness Process

e.g. Application Architecture

I/O = User ViewsProc .= Application Function

e.g. System Design

I/O = Data Elements/SetsProc.= Computer Function

e.g. Program

I/O = Control BlockProc.= Language Stmt

e.g. FUNCTION

e.g. Business Process Model

Proc. = Business ProcessI/O = Business Resources

List of Locations in which the Business Operates

Node = Major BusinessLocation

e.g. Business Logistics System

Node = Business LocationLink = Business Linkage

e.g. Distributed System

Node = I/S Function(Processor, Storage, etc)Link = Line Characteristics

e.g. Technology Architecture

Node = Hardware/SystemSoftware

Link = Line Specifications

e.g. Network Architecture

Node = AddressesLink = Protocols

e.g. NETWORK

Architecture

Planner

Owner

Builder

ENTERPRISEMODEL

(CONCEPTUAL)

Designer

SYSTEMMODEL

(LOGICAL)

TECHNOLOGYMODEL

(PHYSICAL)

DETAILEDREPRESEN-

TATIONS (OUT-OF

CONTEXT)

Sub-Contractor

FUNCTIONING

MOTIVATIONTIMEPEOPLE

e.g. Rule Specification

End = Sub-condition

Means = Step

e.g. Rule Design

End = ConditionMeans = Action

e.g., Business Rule Model

End = Structural AssertionMeans =Action Assertion

End = Business ObjectiveMeans = Business Strategy

List of Business Goals/Strat

Ends/Means=Major Bus. Goal/Critical Success Factor

List of Events Significant

Time = Major Business Event

e.g. Processing Structure

Cycle = Processing CycleTime = System Event

e.g. Control Structure

Cycle = Component CycleTime = Execute

e.g. Timing Definition

Cycle = Machine CycleTime = Interrupt

e.g. SCHEDULE

e.g. Master Schedule

Time = Business EventCycle = Business Cycle

List of Organizations

People = Major Organizations

e.g. Work Flow Model

People = Organization UnitWork = Work Product

e.g. Human Interface

People = RoleWork = Deliverable

e.g. Presentation Architecture

People = UserWork = Screen Format

e.g. Security Architecture

People = IdentityWork = Job

e.g. ORGANIZATION

Planner

Owner

to the BusinessImportant to the Business

What How Where Who When Why

John A. Zachman, Zachman International (810) 231-0531

SCOPE(CONTEXTUAL)

Architecture

e.g. STRATEGYENTERPRISE

e.g. Business Plan

TM

5

Motivation

• ’Where’ is increasingly relevant– Outsourcing– Supply Chain Management/Logistics– Virtual organization– Mobile applications and information systems

• And it is possible to utilize ’where’ to a larger degree (also real time) to know where users, equipment and goods should be, are or where at a certain time

– Tracking (RFID, UWB, GPS, GSM, WiFi, Ultrasound…) – Internet of Things (IoT)

• This paper: – Presents some different notation alternatives based on UML activity diagrams vs. Modeling mobile

information systems– Summarizes an analytical evaluation from a previous paper (I-ESA’10)– Makes an experimental comparison of the two most promising ones

6

Possible notations looked at(I-ESA’10)• Standard UML, using annotation boxes to indicate

context or location• Redefining swimlanes to indicate context / location• Using colour for context / location

• These three were compared analytically, using a home care case supported by a mobile IS as an example

7

Alt 1: Using annotation boxes

• Advantages– Smallest deviation from standard

UML AD– Enhanced understandability for those

who already know AD

• Disadvantages– Greatly increases # nodes in diagram

(poor expressive economy)– May be confusing if you also need to

use notes for something else in addition to context / location

8

Alt 2: Swimlanes for where• Advantages

– Shifting ”who” to stick figures: AD more uniform with UCD

– Swimlanes intuitively indicate location

• Disadvantages– Many lines from stick figures to

activities, poor readability (would be even worse with bigger example)

– New usage of swimlanes may confuse those already familiar with AD

9

Alt 3.: using colour• Advantages

– No increase in # nodes or lines, better readability

– With two process design alternatives beside each other, it is easy to spot differences in location

• Disadvantages– Larger deviation from

standard UML

– Must add legend

– Possible challenge for colour blind users

10

Analytical comparison

Analytical comparison (I-ESA’10):• Alt 1. (trad. UML with annotations) and Alt 3.

(Colours) came out as the two most promising• Proposed further work: Make experimental

comparison

Notation Minimal deviation from

standard

Expressiveness Intuitive / Easy to

read

Less Complexity

Simple Large Simple Large Simple Large Simple Large

Annotated + + + + - - - + - -

Location Swimlanes - - + + - -- + - -

Colours - - + + + + + + + + + +

11

Experimental design• Compare two diagram alternatives, annotation and colour• Controlled experiment looking at the participants’

– Performance using the notation

– Opinion about the notation

• Within-subjects design (Latin squares)– Controls better for selection bias

– ”Doubles” the N

• Measured variables:– Performance:

• Understanding: score on 12 True/False questions about a case after reading textual description and seeing diagram

• Error_detection: score on identifying 5 deliberately seeded errors in a diagram relative to textual description

– Opinion: • Perceived Ease of Use, Perceived Usefulness, and Intention to Use, measured by

answers to a TAM-inspired questionnaire w 14 questions

12

Hypotheses

• Since the colour notation was best in the analytical comparison, this was hypothesized to have advantages, i.e.:– H1: understanding scores will be better for the colour notation than

for the annotated notation

– H2: error detection scores will be better for the colour notation than for the annotated notation

– H3: participants’ opinion about the colour notation will be more positive than for the annotated notation

13

Experimental tasks1. Answering a pre-experiment questionnaire investigating

relevant competence

2. Reading tutorial about first diagram notation (annotatated or colour depending on group)

3. Reading text and diagram for case (home care or traffic control), answering 12 T/F questions

4. Answering post-task questionnaire giving opinion about the notation

5. Repeating steps 2-4 with the opposite case and notation

6. Repeating 2-6 with the error detection task

14

Latin squares designGroup Id (Understanding+ TAM

factor) Questionnaire onError Identification Questionnaire on

Group A Annotated Home Care +Colour Traffic Control

Annotated traffic Control + Colour Home Care

Group B Colour Traffic Control +Annotated Home Care

Colour Home Care +Annotated traffic Control

Group C Annotated traffic Control + Colour Home Care

Annotated Home Care +Colour Traffic Control

Group D Colour Home Care +Annotated traffic Control

Colour Traffic Control +Annotated Home Care

15

Results• 46 students participated, randomly assigned into the

four Latin squares groups• Clear advantage for colour notation in performance

– 3 students performed very poorly in error detection (not doing a serious job); might be considered outliers?

• Slight, but not significant advantage for colour when it comes to opinion

Compared variable (N=46)

Coloured diagram

Annotated diagram

Diffe-rence

EffectSize

Sign.? Y/N(p-value)Mean SD Mean SD

Understanding 0.960 0.057 0.926 0.078 0.0344 0.51 Y (0.01)Error detection 4.50 1.34 3.93 1.17 0.57 0.45 Y (0.001)Errors (w/o outliers, N=43)

4.77 0.53 4.16 1.04 0.60 0.77 Y (0.001)

16

Conclusion on hypotheses

• H1: coloured notation would be better than annotated for understanding (answering T/F questions): CONFIRMED

• H2: coloured notation would be better than annotated for detecting errors: CONFIRMED

• H3: participants’ opinion about the coloured notation would be more positive: REJECTED

17

Threats to validity

• Conclusion validity: significant results but small to moderate effects, should have had larger N to make a strong claim about results

• Construct validity: many other ways of understanding a model than answering T/F questions, and many other work tasks than identifying errors. But at least, this is a relevant task, and the ability to answer questions correctly should indicate to some extent whether a model has been understood

18

Threats to validity, cont.• Internal validity: Latin squares design and pre-exp.

questionnaire should control very well for any selection bias. Notations were presented in equal detail and style in tutorials, and no preferred or hypothesized outcome was signalled to the students

• External validity: The biggest challenge:– Students are not practitioners and motivation may be limited in an

experiment which has no impact on their job or a delivered product. But the comparative nature of the experiment should mean that performance with both notations are equally hurt by low competence or motivation.

– Small experimental tasks are not representative of the more complex tasks in ”real” mobile IS development.

19

Further work

• Also experiment with different pattern fills instead of colour (e.g. better for colour blind users)

• More experiments, possibly including practitioners and/or including collaboration among several persons instead of just individuals answering questions

• Larger industrial case studies, to try out alternative notations with larger and more realistic work tasks