The product of evaluation is knowledge. This could be knowledge about a design, knowledge about the user or knowledge about the task.

Two types of evaluation:1. Summative Evaluation2. Formative Evaluation

Is evaluation necessary?

When do we need it?

Quantitative vs. Qualitative Data

To increase the quality of research, we must avoid the following effects during the observational study:

1. Hawthorne effect2. Observer effect3. Halo effect

Verbal Protocols

Designing Observations:◦ Writing a verbal protocol◦ How to conduct a session◦ Analyzing a protocol/transcript

We ask users to speak out load and mention why they had done an action.

2 components:◦ Talk aloud: verbalize silent decision◦ Think aloud: verbalize whatever thoughts occur

during this task.

Select:◦ Tasks◦ Users◦ Environment◦ One or two significant functional requirements

Observe (at least 3 users)

Give users their set of tasks to complete

Conduct a think-aloud study

Keep protocols (transcripts for each user)

Record users comments, etc. (p. 141)

Description of the environment List of tasks completed by the user Users’ background & demographic details Record and write up users’ comments, body

language, facial expressions The aspects of the interface that these

responses relate to should also be detailed.

Choose the tasks Select users – wrong users lead to

misleading information Explain the purpose to the users Conduct the evaluation – for example:

◦ What are you looking at now?◦ What just happened?◦ What are you going to do next? Why?

The data that you will have at the end of a session is known as the transcript, which details the physical actions and verbal commentary that the user has made.

When analyzing a transcript of an evaluation session, the aim is to categorize the comments according to:◦ Frequency◦ Fundamentality

Experiments define a hypothesis

2 stages of experiments:◦ Implementing the experiment◦ Analyzing the results

Advantages: systematic with a repeatable approach to testing based on scientific rigor.

Disadvantages: includes a reduced consideration of specific variables, questions which are hard t relate to real-world holistic problems and an artificial setting which may lack real-world validity.

In testing we strive for realism

CW is an approach to formative evaluation without users.◦ Preparing a CW◦ Conducting a CW◦ Experiments in Support of Design◦ Dependent and Independent Variables◦ Assigning Subjects◦ Statistics◦ Summary of user experimentation

Interviews◦ Structured interview◦ Unstructured interview

Questionnaires:◦ Open Q’s◦ Closed Q’s:

Simple checklist: Y/N or N/A Ranked order: (select your preference) Multi-point rating scale: strongly agree/disagree