Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices

1/37

CLIHC 2013

DeustoTech - Deusto Institute of Technology, University of Deusto

http://www.morelab.deusto.es

December 6, 2013

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android

Devices

Ivan Pretel, Ana B. Lago

CLIHC 2013

2/37

CLIHC 2013

Outline

Introduction

Usability, Quality and Effectiveness

Context model focused on Mobile Devices

Context model Capturer

System Validation

Conclusion

3/37

CLIHC 2013

Introduction




System Validation

Conclusion

4/37

CLIHC 2013

5/37

CLIHC 2013

Introduction (I)

► According to the last Ericsson Mobility Report (November 2013)

►By Q3 2013, total mobile subscriptions were around 6.6 billion. By the end of 2019, they are expected to reach around 9.3 billion.

Introduction

6/37

CLIHC 2013

Introduction (II)

► Testing tools have changed and have been focusing on the web domain

► But, the web domain is focused not only on its own but also on the mobile one

► Mobile device -> More contexts than ever

► Issue

►The quality of mobile applications fluctuates depending on the context in which they are used

►When we try to capture it, we modify it

Introduction

7/37

CLIHC 2013

Introduction (III)

► Approach

► A mobile-based tool to automatically evaluate the effectiveness of interactions and capture metrics of the surrounding context without adding disturbing elements

Introduction

8/37

CLIHC 2013

Introduction




System Validation

Conclusion

9/37

CLIHC 2013

Usability, Quality and Effectiveness (I)

► Usability

► ISO 9241-11

► “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” – “Effectiveness is the level of accuracy and completeness with

which users achieve specified goals”

► ISO 9126-4

► “Usability is the capability of the software product to be understood, learned, used and attractive to the user, when is used under specified conditions”

– “Effectiveness is the capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use”


10/37

CLIHC 2013

Usability, Quality and Effectiveness (II)

► Quality according to ISO 9126 ► ”Represents a property of the software product defined in terms of a set of

interdependent attributes (usability, security, reliability, performance, complexity, readability, reusability) expressed at different levels of detail and also taken into account the particular context of use”

► Can be measured by 3 quality aspects

► Internal Quality

– From an internal view (e.g. spent resources)

►External Quality

– Its behaviour (e.g. number of wrong expected reactions)

►Quality in Use

– “The quality of software that user can perceive when the software is used in an explicit context of use”

– It is measured by 4 main capabilities

» Effectiveness, satisfaction, productivity and safety


11/37

CLIHC 2013

Usability, Quality and Effectiveness (III)

► Effectiveness metrics (ISO/IEC 9126)

► Task Effectiveness

►What proportion of the goals is achieved?

– Measures the quantity of the goals achieved by a user

– 0<=TE<=1

► Task Completion

►What proportion of the tasks is completed?

– Measures the level of success the user achieves performing tasks

– 0<=TC<=1

► Error Frequency

►What is the frequency of errors?

– Measures the number of times that an error is made within a given period

– 0<=EF<=1


12/37

CLIHC 2013

Introduction




System Validation

Conclusion

13/37

CLIHC 2013

14/37

CLIHC 2013

Context model focused on Mobile Devices (I)


► It has been taken into account and has been defined several times

► Physical location

► Weather attributes

► Involved communities & stakeholders

► Goals

► …

► Tries to answer

► Who are the users?

► What are they doing?

► Where?

► Which resources are nearby?

15/37

CLIHC 2013

Context model focused on Mobile Devices (II)

► Nuevo modelo basado en tres pilares


► Context of Use model based on 3 main components

16/37

CLIHC 2013

Introduction




System Validation

Conclusion

17/37

CLIHC 2013

18/37

CLIHC 2013

Context model Capturer - Lab vs Field (I)


► Lab-based

►Log information in a highly controlled environment using specific devices and users

– Control: high

– Context variables: limited

– Objectivity: low

► Field-based

►Log using added agents and invasive evaluation methods

– Control: very low

– Context variables: unlimited

– Objectivity: depends on the added agents

► Capturing tools

19/37

CLIHC 2013

Context model Capturer - Lab vs Field (II)

► If you do not want to influence the context, you have to remove all the biasing elements which can influence the captured data and use only the elements which is composing it

Capture using only the mobile device

► This tool should capture the context model via the built-in mobile sensors and logging interaction events


20/37

CLIHC 2013

Context model Capturer - Purposed System


► The purposed system is formed by a tiny Android library and a server to store and log the performed interactions

► 1. The app-to-test (ATT) should be integrated with the library

► 2. This library automatically captures context and interaction information and stores it in a local database

► 3. When the device has internet connection and its owner wants, all the information is uploaded

21/37

CLIHC 2013

Context model Capturer – Library (I)

► Interaction attributes

► Using one command ►log(event_type, interface, object)


22/37

CLIHC 2013

► Static attributes

► Captured during the registration

Context model Capturer – Library (II)


► Automatic ► Manual

23/37

CLIHC 2013

► Device Sensors

► Android API ► External Services

Context model Capturer – Library (III)


► Dynamic attributes

► Captured during the interaction (all automatically)

24/37

CLIHC 2013

Introduction




System Validation

Conclusion

25/37

CLIHC 2013

System Validation – Experiment (I)

► 4 subjects

► A tiny memory game

► 1 day registering interactions

► Contexts

► At home [H]

► Public Transport [PT]

► Walking Down the street [C]

► At work (office) [T]

System Validation

26/37

CLIHC 2013


► Steps ► Download the Android app

– From http://qualityinuseserver.appspot.com

► Sign up on the testing platform through the app

► Play the game

► Upload info to server

System Validation

http://qualityinuseserver.appspot.com/

27/37

CLIHC 2013


► Focusing on the effectiveness measurement

► Task: To end up with all of the cards flipped face up in less than 15 turns

►More than 15 turns: task is not completed

► 8 pairs of cards

► It means the best round is made by only 8 interactions and 0 errors

System Validation

28/37

CLIHC 2013

System Validation

► Effectiveness metrics ► Task Effectiveness (TE)

►TE = 1 – ΣAi * error weight

►The weight of an error is 1/7 = 0.143

– Max number of errors is 6 with 8 correct interactions

– If you make 7 errors

» (1- (7*1/7) = 0)

► Task Completion (TC)

►TC = won rounds / played rounds

► Error Frequency (EF)

►EF = Number of errors / Total turns

System Validation

29/37

CLIHC 2013

System Validation – Results

► The outdoor contexts (PT and S) affect subjects and lead them to be less effective than in the indoor contexts (H and W).

► TE indicates that the S context is the context where subjects have the worst results

► The most efficient context is W, although EF exposes that is where subjects make more mistakes

System Validation

30/37

CLIHC 2013

Introduction




System Validation

Conclusion

31/37

CLIHC 2013

Conclusion

► The effectiveness and the new context model, which can bias the usability and the quality of mobile applications, can be automatically captured and measured by the own mobile device minimizing the interaction bias

Conclusion

32/37

CLIHC 2013

Conclusion

► Found issues

► Security –android.permission.INTERNET android.permission.ACCESS_FINE_LOCATION

–android.permission.ACCESS_NETWORK_STATE android.permission.ACCESS_WIFI_STATE

–android.permission.MODIFY_AUDIO_SETTINGS android.permission.RECORD_AUDIO

–android.permission.WRITE_EXTERNAL_STORAGE

► Weather variables capture depends on the Internet connection

► Developers have to specify where are the errors

► Next steps

► Solve the found issues

► Increase the number of users

► Calculate correlations

Conclusion

33/37

CLIHC 2013

Thank you for your attention

¡Pura vida!

34/37

CLIHC 2013

DeustoTech - Deusto Institute of Technology, University of Deusto

http://www.morelab.deusto.es

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices

Iván Pretel

{[email protected]}

35/37

CLIHC 2013

All rights of images are reserved by the original owners*, the rest of the content is licensed under a

Creative Commons by-sa 3.0 license.

* • http://www.fotopedia.com/items/flickr-2830319467 • http://www.fotopedia.com/items/flickr-4609827016 • http://en.wikipedia.org/wiki/File:Skyline_oklahoma_city.JPG • http://www.flickr.com/photos/grimsanto/6190478735/ • http://commons.wikimedia.org/wiki/File%3AAstronomy_Amateur_3_V2.jpg • http://commons.wikimedia.org/wiki/File%3AAldrin_Apollo_11_cropped.jpg