WG3 Interplay between Design and Evaluation, Quality Models and

Standards

Twintide Coimbra, Portugal March, 2013

Ebba Þóra Hvannberg

WG3a Qualities in use

• Identify which software qualities in use (e.g., trust, creativity, automaticity, security, safety, sociability, usability, reliability, evolvability - (related to standards, e.g., ISO 9241, 27477))

– are differentiated by sector-dependent or sector-independent criteria and describe connections

– vary over lifecycle – are traded-off against quality attributes– are prioritized and realized in the process of

systems development in different sectors/disciplines

WG3c design-evaluation feedback

• Understand how iterative design-evaluation-redesign feedback cycles operate for– computing systems in different sectors– relative to specific evaluation criteria: social,

economic, technical etc.– traceability through cycle

Past activities

• Quality attribute maps of relations between domains

– Created initially (Bertinoro 2011)– Refined ( London, 2012)– Consolidated (Skopje, 2012)

Analysis of Transfer of Quality Attributes

• The goal was to strengthen the quality attribute map

• Papers selected with the following criteria– Quality attribute(s), Sector/Domain, Method – Had to have empirical data

• Papers analysed– 9 members contributed to the analysis of 17

papers

Stories of transfer described

• Stories on transfer described

• As a <role> I want to <task> so that I can do <need>

• Storytellers were asked to think about method transfer when describing a task

Prescribed format (from facilitator)

• 20 stories were collected in London, March 2012

• The stories have been classified and cleaned

• A set of 12 stories remain that are related to some kind of transfer of method from one „method application“ to another

Forms of transfer

• Transfer between domains– E-Learning to Games

• Transfer from generalization to specialization– Heuristics evaluation for Virtual Environments

• Adoption of a technique from another field followed by adaptation – E.g. Laddering from marketing

Stories analysed and grouped

• Generalization to specialisation• Specialisation• Transfer between domains• Transfer between similar domains• Adaptation

Open questions

• How can these results be verified or expanded– In the literature, look for similar patterns

• Case studies • Knowledge on transfer • Collect more transfer stories from literature

– In practice• Designers and developers change jobs often• They may be able to tell us transfer stories, i.e. how they

transfer knowledge/methods between sectors

Copenhagen 2012 results

• Tell a story of how you have carried out an evaluation, e.g. in a project and contrast it to another instance of an evaluation.

• 18 cases of transfer stories were collected from Twintide members

• Results are here

Overview of form of transfer

• Overview

Revising stories and further analysis

1. Read and amend story1. Typos2. Specify domain of story 3. Review stories4. Review transfer between stories 5. Add publication if available

Identify method resources used in the stories *

1. Participant-recruitment 2. Task selection3. Reporting format4. Problem identification5. Problem classification6. Analysis7. Heuristics8. Thinking aloud protocol 9. Please add more as needed

*Law, E.L.-C., Hvannberg, E., Vermeeren, A.P.O.S., Cockton, G., Jokela, T.: Made for Sharing: HCI Stories of Transfer, Triumph and Tragedy. CHI 2013 Extended Abstracts. ACM, Paris, France (2013)**Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human-Computer Interaction 27 (2011) 940-970

Identify contextual factors*

1. Business goal2. Design purpose3. Development context4. Team skills5. Corporate culture/values6. Please add more as needed

*Law, E.L.-C., Hvannberg, E., Vermeeren, A.P.O.S., Cockton, G., Jokela, T.: Made for Sharing: HCI Stories of Transfer, Triumph and Tragedy. CHI 2013 Extended Abstracts. ACM, Paris, France (2013)**Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human-Computer Interaction 27 (2011) 940-970

Contrast Method resources

Item Story 1 Story 2 Contrast/ Type of transfer

Participant recruitmentTask selection

etc.

Contrast Contextual Factors

Item Story 1 Story 2 Type of transfer

Business goal

Design purpose

Development contextTeam skills

Corporate culture

Schedule

Task Duration Time

Presentation of previous results

30 minutes 14:00

Read and amend stories 30 minutes 14:30

Identify method resources and contextual factors

30 minutes 15:00

Coffee break 30 minutes 15:30

Contrast stories 30 minutes 16:00

Report method resources and contextual factors

30 minutes 16:30