1

Living the user experience Gary P. Smith

EG A/S Industrivejsyd 13c

7400 Herning, Danmark Telephone number, +45 99 28 32 51

gary.smith@eg.dk ABSTRACT In this paper I will discuss the lived experience in design ethnography and use context from the perspective of the observer shadowing the observer and how living the user experience can bring you closer to the use context.

First I will argue that although video shadowing can capture a wealth of data about users and use context in a design ethnographic field study, I will show through examples, that there is an experience beyond what the camera can see, in relation to a plethora of different aspects of individual, social and cultural experiences.

Second, what does this mean for a designer to have been there where things happened?

Author Keywords Design, ethnography, video, user, sensory, context

INTRODUCTION Within software UI design, interaction design plays an important role in conjunction with supporting activities; we need to identify the users of a proposed interactive system. Ethnography has become popular within interaction design because if products are to be used in a wide variety of environments, a designer must know the context and ecology of those environments.

“By bringing the user into the design problems we can focus on the users needs, both general and specific.” (Newman & Lamming, 1995:29) One of the problems with traditional ethnographic studies is that they take months or years to complete where as, “Design ethnographers count their field studies in days rather than months” (Buur & Ylirisku, 2007:46) During a recent field study our design team carried out the method of video shadowing (Buur & Ylirisku, 2007:65), which was spread over a one-week period, following each day a group of different users. The field studies focus was “working with the body”, this could be people that work with there own bodies, carpenters and an electrician or work with other peoples bodies, such as a pathologist or a spinning instructor.

By capturing user experiences through this method our design team was able to gather a record of what they do. “Observing is an intense and tiring activity, but however tired you are it is important to write up experiences and observations at the end of each day. If not valuable information is lost as the next day’s events override your previous days impressions” (Preece, Rogers, Sharp 2007:328)

In contrast as software UI designer much of the theory and methods implemented within my work are based around HCI theory, which revolves around information and cognitive data gathering. “Focus groups uncover people’s /perceptions/ about their needs and their values. This does not mean that they uncover what people /actually/ need or what really /is/ valuable to them-however-knowing perceptions of needs is as important as knowing the needs themselves” (Preece & Rogers, 2007) The insight it gave about users and its effectiveness as a design tool, compared to personas, which we have used in the past, from my experience, are more than often very open for interpretation by the design team and even more so by the developers.

Although the method of video shadowing had been effective and had brought me closer to end users, it began to highlight some concerns I have had with HCI methods for data gathering; Much of my previous work has been on desktop computing, but my current and future work is evolving within the mobile space, which has a much wider context. So this will become an important issue in the future, when convincing management about using these new methods. Controlling the environment? I am often testing people’s interactions with new interfaces, but these are confined to environments, which are often artificially constructed and controlled. In direct contrast to my first field study with crafts people, which was dynamic and unconstrained. Although this may seem trivial or even obvious now, at the time it did not. I arrived at 07.00am ready to begin the days video shadowing; it started slowly by following the crafts person while he stocked his van in preparation for the day’s activities. At around 09:00am I began to miss the convenience of the office coffee machine with the constant bombardment of body signals telling me that I had missed my first cup of coffee. Although I was experiencing my body’s apparent erg for the office coffee machine, at the time I just ignored it as being a consequence of doing the shadowing method, and thus being too occupied with the task at hand to have time. It was only when we reached 12:00am and parked into a small lay by on our way to the next customer that Peter said “OK, now we eat, its lunch time”. It was at this point my body signals, which I had been ignoring, caught up with me and I really began to observe my self and my experiences within the environment. My environment had changed and new rules applied. Was my preparation poor? Had I forgotten something so basic as a sandwich box? What had I just experienced?

2

First Living Experience Within the first hour of my video shadow I followed an electrician on his daily rounds to a private residents, he was carrying a ladder, a toolbox and materials to carry out his job.

This was my first field experience with a crafts person and I was not prepared for something so simple, as taking off a pair of shoes, when entering the house. I was so busy trying to observe him and capture him on video that I was left with no alternative; I pulled them off by using my feet.

Fig 1.0 Hands free environment

This action would become a problem later in the video shadow when on numerous occasions we would have to go back and fourth between his van to pick up extra equipment, it is not easy putting your shoes on and tying your laces while holding a camera with one hand. “I learned but could not alter my situation” and adapt to the environment.

At some point during this taking off and putting on of my shoes I began to reflect over my own experience within the context.

At the time it was a more passive, if not subconscious checklist or active self-monitoring of my own experience in relation to my surrounding context.

The electrician I was observing was guiding me through his context. Observing the video only, without living the experience shows the electrician carrying his tools.

But although he actually takes his shoes off when entering the house, it is not captured on film. Is this interaction obvious or more importantly is data lost? Although this may seem obvious it is part of the tradesman’s context and it highlights something even more important. The craftsman’s context has a working code of ethics. Some of these ethics would unfold as the day progressed…

• When and how do you go to the toilet?

• Do you use the customers, or do you wait and go elsewhere?

Peter said that he would only ever use a customer’s toilet in an absolute emergency and then only to take a pee, otherwise he would drive back to use his office.

By observing myself, and my own experiences from the environment I was able to experience and learn from it that simple things like tying your shoelaces and going to the toilet in this context were in fact a hindrance, which they are not when working in the comfort of an office. But it did not stop hear, I began thinking about how more practical the electricians clothing was compared to my own.

Although I was not carrying out the work I was already learning how to adapt to this new environment, by observing how he navigated his context, through my self-awareness.

I was holding a camera, and he was holding tools. I began to think how his pair of clogs would solve my problem of untying my shoelaces, because my hands were full. This is when I realized I was having a living experience.

Self-observation: Different people, different cultures, if English people do not wear clogs? What do they wear?

DEGREE OF PARTICIPATION Something that became apparent after my field studies was the degree of participation one has when observing others within there own environments. From the outset I was worried that the camera would be very intrusive and taping a complete working shift might be too much.

What was interesting was how the days video shadowing unfolded, at the start of the recording I felt a sense of self-awareness that he was very aware that he was being filmed, such as a quick look directly into the camera.

At the start when meeting fellow crafts people who were unprepared by my presence which for them was out of the ordinary could be sensed and seen by the way they stared into the camera on our first encounters or moved slowly out of the lenses view, Peter would quickly react almost as a reflex on each encounter and explain who I was and what I was doing and then quickly proceed about his daily routine.

“Whether the ethnographer is doing a form of participant observation or interviewing, she or he is still participating in a material, sensorial and social environment” (Pink 2009:105)

What was interesting about these encounters was how they changed with each set of new encounters, Peters explanations of my presence got shorter and until eventually he no longer gave an explanation he just mumbled out a non-intelligible word or sentence as he proceeded about his chores, he had avoided answering there question entirely, this had a kind of numbing effect, and seemed to play down my presence.

“Video may provoke some people to make faces, others to clean up their speech, and yet others to move cautiously in front of the camera. This effect – what scientists call bias – may wear off as people become familiar with the presence of the camera” (Buur, Ylirisku 2007:48) My previous video observation work has been carried out in test labs, these are controlled environments, and much of the work is as a passive observer where we are interested in seeing how a user will interact with a specific interface. Fig 2 How much control do we have over our participation

This was different I was telling myself that I had to remain a fly on the wall and keep a passive observer roll, but the fact that I was there and following the crafts person, I was participating.

3

“It is difficult to be a truly passive observer if you are in the field, simply because you cant avoid interaction” (Sharp, Rogers, Preece 2007:326)

I was seeing how he carried out the job, at times he would even explain to me in detail about specific methods or why he did the things he did.

I was learning, I felt like an apprentice on his first day at work, this was when I realized I was participating without even realizing it

Video Rec. vs. Lived Experience On another occasion while shadowing an electrician outside fixing a street lamp onto the side of a building, I remember how bitterly cold it was standing and holding onto a camera. I remember alternating the camera from one hand to the other while placing my free hand in my coat pocket to warm it up. At the same time complaining about my own discomfort and fatigue of holding a small video camera.

Fig 3 Sensing: Heavy, Cold, Tired and Balancing The design team when reviewing the video could see the crafts person standing on a ladder fixing the lamp in place, but really could not comprehend or appreciate the weight of the object he was holding, how heavy it became while he was holding it over an extended period of time, effected by the objects temperature, or the chill factor as the wind blows while standing still and confined to the top of a ladder, and yet still maintaining a steady balance.

Experience beyond what the camera can see In our design team sessions while discussing and reviewing the video footage I found that my own experiences during the field study more immersive than footage recorded in the field by other team members, there clips felt more detached and distanced. I felt that something was missing; I had not lived their experience. I also began to observe an apparent enthusiasm for there own clips, in contrast to other member’s clips. Was this detachment from each other’s clips mutual? , And did they feel that there was a connection between this detachment and the lived experience. Would living each other’s experiences reduce or remove this feeling of detachment. This apparent detachment I was observing, made me ask the following questions to the design team:

• Do you have a stronger attachment to your own clips? • Why do you feel a better attachment to your own clips,

than those of other team members?

Members of my design team said they did feel a feeling of detachment while watching other member’s clips compared to there own, and interestingly they did feel that it was easy to recall details from there own lived experiences. It became apparent to us all that the video seemed not only to be a memory trigger of our lived experiences, but it allowed us to recall personal experiences, details that could be recollected which were beyond what the camera lens had actually recorded. From living the experience it became clear that we could use the video as a form of Dictaphone, so that you could use it to index personal experiences and recall them later. We also discussed in a future field study, how it would have been interesting to carry out a video shadow over a number of days following each others users, so that each member of the design team could experience a similar lived experience, to see if this would close the detachment between each other’s clips.

Living Experience – Sensory ethnography During the review of my video footage, while preparing clips for a video card game (Buur, Ylirisku 2007:105), I felt a strong need to explain my experiences to the design team; I felt that the short clips did not convey enough empathy. Before I carried out the field study I had already envisioned how an electrician went about his daily work, I believed at the time they worked with electrical cables and fuses all day. I never realized how far from the truth this was. I decided to produce a video portrait to allow other members of my design team, to feel empathy for an electrician, by putting them into his shoes, and to show them how physically hard his work was. It was during the presentation of this video portrait and the playing of the video card game, that it became apparent that the lived experience had been a multisensoriality of experience, perception, knowing and practice. Which I felt had not been captured by the video camera. As an interactions designer I decided to contact an anthropologist for current background literature for doing sensory ethnography as I thought it was important to understand and enrich my knowledge about the senses.

Sensory ethnography Pink 2009, suggests an approach to doing sensory ethnography where “vision does not dominate the way we experience our environments” (Pink 2009:14), however (Pink 2009:28), explains that there is some disagreement among scholars of the senses regarding how phenomenological understandings might be employed.

• What human perception involves • Interconnections between the senses • Relationship between perception and culture

This is due to how neurobiologists are challenging conventional thinking, that are senses are not modular functions but in fact are cross-modal interaction between the senses. Much of the theory concentrates on re-thinking the ethnographic process through reflexive attention to the ‘sensoriality’ of the experience, by outlining a sensory ethnographic methodology. Where “Embodiment” implies an integration of mind and body,

4

and “Emplacement” is used to suggest the interrelationship of body, mind and environment. Pink 2009, suggests that often moments of sensory learning are not planned, “Rather, these are often unplanned instances whereby the researcher arrives at an understanding of other peoples memories and meaning through their own embodied experiences and/or attending to other people’s practices, subjectivities and explanations.”(Pink 2009:65), this is the case during my field study example described in “Video Rec. vs. Lived Experience” of alternating my hands between my warm pocket and the cold camera, while recording the electrician standing on a ladder, it was through this engagement that I was able to reflect, and conceptualize meaning through my own emplacement. The fact that the camera felt cold by physical sensation of touch bridged a gap beyond my own physical touch and touching a conscious connection with the crafts person’s. “However, an understanding of the senses as essentially interconnected suggests how (audio) visual images and recordings can evoke, or invite memories of the multisensoriality of the research encounter.” (Pink 2009:101)

Fig 4 Confined, heavy, large, difficult to hold I agree with Pink (2009:107) that the power of video as a technique is invaluable to help researches use video to communicate aspects of these experiences to others, or from my own experience of how the video can evoke, or invite memories from the lived experience, by observing my own ego, through act of becoming aware of my self, I am able to use video as an expressive medium for personal learning. Although it would be impossible to cover in such a short paper, Pink explains how we in the modern western culture have divided the senses into vision, hearing, touch taste and smell, where as other cultures do not necessarily divide them in the same way. She goes on to explain a method which researchers can develop so that they can be “of two sensorial” about things. “Being able to operate complete awareness in two perceptual systems of sensory orders simultaneously (the sensory order of ones own culture and that of the culture being studied)” (Pink 2009:51) The idea behind this is to break down our experiences into sensory categories, and if we are aware of which categories we are using they can help us to understand and give meaning to different types of sensation.

“It does nevertheless imply that in applied research attention to the senses can lead to an appreciation of what is important to others”(Pink 2009:59) Although these methods can help me in a field study to observe users, I feel that there is a greater issue to address on how to bridge what I will call a “gap of detachment” between members of the design team who have lived the experience and those who have not. I will come back to this later in a proposal to “Annotating the experience”.

Living Experience – Mobile UI for crafts people? The software company I work for has developed an ERP1 system over the last 25 years called ASPECT4. It is used within many different industries. The front end to this system is designed for a Windows desktop environment, not a mobile platform.

While writing this paper I was approached by our builders supply department who have been thinking about moving parts of there product into the mobile space.

He said: “I have spoken with your department head and he has recommended that we speak to you about a new mobile UI we are in the process of designing, which will be used for crafts people to register there hours, through a web based solution using the mobile browser opera.”

Fig: 5 Hands free… workflow is never interrupted

It was during his informal presentation, I asked him “have you ever been out in the field with a crafts person while they are working.”

I said to him from my lived experience, I could not see crafts people using such a solution. “My lived experience….”,

It was at this point I began to experience that I had moved beyond my previous personal role as a designer thinking in information and cognition theory, founded in my HCI background, and was reflecting over the lived experience, which was a much stronger and a more personal experience, which I could now relate to.

In Fig 4.0 the electrician is busy working when he receives a call, he takes the call but in no way does it become a hindrance in his workflow he just continues working, while taking the call.

He then needs to check a date, so he says he will phone the person back later, he then rips a piece of cardboard off of a screw box,

1 ERP – Enterprise resource planning

5

writes down there telephone number and details. This is put into his back pocket and he continues working.

Later on in his workflow, on returning to his van, the cardboard in his pocket becomes a form of physical post-it which reminds him to check his calendar, with a mobile PC mounted in the vans cabin; the cardboard is used as a physical reference, and like a post-it is thrown away when finished with.

Although the video documents him speaking and working at the same time, I kept reflecting over the tying of shoes laces as described in one of the my previous living experience, it then be came apparent to me that this was a central aspect of this crafts person’s context, and I could relate and begin to understand the importance it meant to him that his hands are constantly free at all times to carry out his work.

What was even more interesting was that I could explain this lived experience without even showing the video footage to my colleague I felt comfortable when I said to him that I could not see crafts people using such a solution.

As an interesting side note to this, I am in the process of building a new house, and when I attended one of the building project meetings, I asked several of the crafts people, including funny enough the electrician from my field study if they would use such a system, and they all said “no”. When asked why they said that from their experience they were more of a hindrance because they got in the way of doing the job. They said: “Pen and Paper is easy and reliable”.

Although the lived experience had highlighted how important a hands free environment was for a craftsman, the personal benefit as a designer to make a cost effective decision was even greater.

Annotating the experiences During much of this paper I have focused on the living experience, and explained how the design team had experienced the same detachment while reviewing clips by other members.

So how can I reduce this gap of detachment between my lived experience and how the design team perceives my clips when viewing them?

After much reflection I wondered if it would be possible in some way to annotate parts of my lived experiences onto the video clips, allowing the design team to experience some of the researchers embodied experience and practice, and vice versa when viewing there clips.

I propose the following two methods as a possible solution to reducing some of the detachment experienced by the design team:

• Annotating the video during the field study

• Annotating the video after the field study

Annotating during a field study In the field studies I have carried out, using video shadowing (Buur, Ylirisku 2007:65) it was done by using a single camera and a built in microphone, I propose an alternative to this setup where an extra microphone is added.

The first microphone would be used to capture the dialog of the user being shadowed and the second microphone would capture the observers dialogue. The observer would then record their

thoughts/observations/experiences in the way one would use a Dictaphone to record notes, by speaking aloud.

This would obviously require that the observe: • Becoming aware, of ones own thoughts and actions. • Catching them self’s in the act of thinking and/or

reacting • Comprehending the source and utility of such thoughts

and actions

The users and observers dialogue would be recorded on separate tracks with the same time code, this would allow the design team to swap between these tracks, or hear both tracks at the same time.

Advantages:

• The observer would describe their embodied experiences and practices.

Disadvantages:

• Would an observer have the same problems associated with users during a speaking-aloud test, where the cognitive load could become too large and the observer stops talking.

Annotating after a field study Pink 2009 - “Visualising Emplacement”, I could not help thinking about a question which had been raised about, to what extent embodied experiences and practices can be represented with (audio)visual media. An example is given about a researcher who films an actor’s transfiguration when putting on makeup, the lens only captures surface elements, but it is represented powerfully in the following reflexive text:

“The color, smell, and textures of these objects [the make-up and accessories] were familiar to me I moved so that I had a direct view of Iemoto’s face reflected in the mirror. As I watched her put on a habutai (a purple silk fabric to cover the hair under a wig) and begin to apply layers of makeup, an empathetic sensation came over me as if I could feel the makeup on my face too.” (Pink 2009:104,Hahn 2007:147)

If this can be written as a reflexive text, then why can this not be annotated as a separate audio track over the original video clip?

I have seen this method used to great effect on DVD’s by directors, when they explain to film enthusiasts, what they where thinking when filming specific shots, and what they wanted to convey to the audience.

When an observer has completed their field study they could annotate the video with a separate audio track this would be done during the reviewing of the footage.

They would then speak aloud-about there thoughts and memories invoked by watching the clips.

Advantages:

• After completing a field study you may see things that at the time were disjointed, or at the time unexplainable.

• The cognitive load would be less than out in the field.

Disadvantages:

• Data will be lost; we have to rely on their memory.

6

CONCLUSION Using the method of video shadowing alone is an effective method to gather a record of what users do, when edited together with a video interview it has aloud me to create a video portrait to provoke empathy to other members of my design team.

It has also shown me that the video captured contains a trigger to invoke memories from the field study and that even more importantly that there is an experiences even beyond what the camera can see.

This living experience is so profound that I was shocked that only one days filming could give me so much insight into a particular use context.

It has also shown me that an observer requires a better understanding about how to observe their senses.

Therefore preparation is required by any observer on how tap into this resource, pink 2009 chapter on “Preparing for sensory research” is a good starting point.

What does this mean for a designer to have been out there where things happen? Personally as a designer it has only enforced the importance of doing field studies, and getting away from the clinical environment of the office, because living the experience allows a greater and wider knowledge to understanding what users really do. In the long term this knowledge will allow for better innovation because my ideas will be based on user knowledge and experience. The mobile UI is a good example from my own working environment where time and money can be saved, based on a personal experience. From my field study experiences I believe that annotating the video would allow the possibility for members of a design team who were not involved in the actual field study to obtain a better understanding of the users context by observing the observer through there audio dialogue. Thus reducing some of the detachment experienced by the design team.

ACKNOWLEDGMENTS I would like to thank Wendy Gun for literature from an anthropology perspective. Thank you to Peter Gull for allowing me to follow him while he was working, using the video shadow method and permission to use the material in this paper. I would like to give a big thanks to Henrick, Louise and Kim the members of my field study group for their knowledge and input. Thank you prof. Jacob Burr, for your patients and knowledge, your book was inspiring, and your video methods are now a part of a new and developing tool case. Finally thanks to my wife and boys for their support, for the many hours I have been away from home.

REFERENCES

[1] Interaction design – Sharp Rogers Preece

ISBN 978-0-470-01866-8.

[2] User Centered Product Design – J. Buur, T.Binder [3] Designing with video – Salu Ylirisku, Jakob Buur

ISBN 978-1-84628-960-6

[4] Design of Everyday things - Donald A. Norman ISBN 13-978-0-465-06710-7

[5] Pink,S.2009. Doing Sensory Ethnography. London: Sage Publications. Literature background reading Russell, C. 1999. Experimental ethnography: The Age of Film in the Age of Video, Chapter 3.The body as main attraction USA: Duke University Press.