'The Connect gallery – using prototype evaluation as a strategic tool in the design process of interactives.'

IntroductionThis paper introduces the prototyping process and its benefits in the design process of interactives. Through a case study from National Museums Scotland, it exemplifies how going to the trouble of prototyping can really pay off in terms of not ending up with an interactive that no one likes or understands.

Background to Connect

Connect is a permanent interactive science and technology gallery, situated on the ground floor of the National Museum of Scotland. The overall theme of the gallery is Creativity, Discovery and Innovation, inspired by the science and technology icons selected for the gallery. These objects are key examples of innovation and creativity in their time and provide inspiration for discovery and innovation in the future. One such example is Dolly the Sheep, a key icon of twenty-first century scientific achievement.

Within the gallery there are five sub topics: genetics (‘Me2’), robotics (‘Robots’), Space (‘Blast Off’), energy and power (‘Power Up’) and transport (‘Move It’). A series of interactives are co-located with the relevant objects for each topic, and provide visitors with opportunities to interact physically, emotionally and mentally through challenges, puzzles, and physical activities. The objects serve as a focus for the interactives.

There are a total of 19 interactives in the gallery, of which about half are mechanical and half are ICT interactives, as well as touch screen information stations around the gallery. Seven of the ICT interactives and the information stations, all of which were specially designed for the gallery, underwent prototyping during the design process.

Figure 1: Connect gallery floor plan showing the five sub topics

The Prototyping Process

First, the ICT developers produced the first prototypes. These mainly just contained the ‘core interaction’ of the interactives, designed to get the main messages and navigation across. Most graphics were rough ‘placeholders’ in case substantial redesign was needed, and paper storyboards alone were not acceptable.

The next step was to test the prototypes in the museum to see how visitors responded to them. I’ll say a bit more about this step in a moment, but basically it consisted of two parts – observation, followed by a short interview.

The responses from the testing were then analysed and fed back to the developers. This consisted of a written report and a face-to-face meeting. The reports detailed problems with the functionality of the prototypes and suggestions for improvements, and also covered any content or design issues that needed attention.

Finally, the developers made relevant changes to the prototypes according to the feedback. Then the process started again.

Figure 2: The Prototyping Process

Each interactive went through 3 phases of evaluation before final delivery. The second round of prototypes incorporated recommendations from the earlier feedback, and most sections incorporated final graphics or graphic styles so the museum could confirm it was happy with that. The final version was slightly different in that the testing with visitors step was replaced with robustness testing, where basically all available staff tested theinteractives for bugs, typos and other weaknesses.

We worked to a very tight schedule. We had one week per prototype interactive for the testing, and a further week to gather the results, with the feedback meetings with the developers taking place two weeks after submission of the relevant prototypes. As I said before, 7 interactives and the information stations were tested and evaluated, which made a total of 8 test subjects. We worked in three 4-week blocks, testing two prototype interactives per week. The ICT developers were paid by the museum in stages which tied into the delivery phases of the prototypes for evaluation.

Testing the Prototype Interactives

I want to say a little bit more about the step where we tested the prototypes with visitors. Like I said, it consisted of two parts. After approaching visitors in the museum and inviting them to take part in testing, we observed them trying out the prototype interactives to find out what seemed to be working or not working. Specific things that the observers were looking out for included whether visitors seemed to have any problems with the instructions, understanding what they were meant to do, as well as difficulties with the navigation.

The observation was followed by a short interview with the relevant visitor. If it was a group of visitors being observed, then just one member of the group was interviewed. The sample size for each prototype interactive was 20 groups/ interviewees. Questions that were asked in the interviews included:

- What did you think the interactive was about?

- Did you find out anything new you didn’t know before?

- How would you describe this interactive to a friend? and

- What did you like most/ least about the interactive?

as well as practical things such as how easy or difficult they found the language to understand, and their opinion on the length of the interactives.

1. Invite visitors to take part in testing

2. Observe visitors using prototype interactives

3. Short interview with visitors to get responses/ reactions on prototype interactives

Figure 3: Steps in testing the prototype interactives

Why Prototype?

So what are the benefits of prototyping? Well, it gives developers a chance to experiment with new and different designs, and it formalises the process by which the museum feeds back to the developers – feedback is sent a fixed number of times and in a written document everyone agrees on.

More importantly, the museum can ensure that visitors

- understand the interactive (understanding)

- are able to operate the interactive (ergonomics), and

- enjoy the interactives (enjoyment).

Case Study: GM Foods

So, to end with I have an case study from our testing for the Connect gallery that exemplifies how going to the trouble of prototyping can really pay off in terms of not ending up with an interactive that no one likes or understands.

The ICT interactive I would like to present to you focuses on the GM food debate. It has a 17’’ touch screen, however, the prototype was operated by mouse. The interactive has an introduction explaining GM, followed by a game-play based challenge, and then an opportunity to vote on some topical GM issues at the end. The key messages the interactive is intended to convey are the many different purposes of GM technology, and that society must decide how best to use if for the future, i.e. a focus on the ‘pros and cons’ of GM and the surrounding debate.

The target audience for the interactive are schools and families with children age 11 and above, as well as independent adults. I would like to focus on the middle part of the interactive, the game-play based challenge1.

In the first prototype, visitors were given three scenarios to choose from – to create foods that contain medicine, crops resistant to weedkiller, or foods that last longer on the supermarket shelf. On choosing a scenario, visitors got some more information about that scenario and were then invited to accept their challenge. On accepting their challenge, visitors then had to choose a plant to modify. When individual plants were selected, three characteristics popped up for each plant that visitors had to ‘accept’ or ‘decline’, based on the information they had been given on the previous screen. Once visitors had identified the correct plant for their challenge they could then modify it.

Feedback from the testing of the first prototype

So, did it work? Overall visitors seemed to like the interactive and enjoy the experience, but that seemed to be mainly restricted to the colourful graphics, and perhaps the novelty of being involved in the testing and getting a special preview. The understanding and ergonomics showed a different picture.

1 Please note that the screenshots from the case study prototypes referred to in the original presentation at

the VSG Summer School were not available for inclusion in this paper.

As I already said, most of the visitors seemed to enjoy themselves, although half of them found something they disliked. In terms of understanding of the key messages, only 55% mentioned GM or genetic modification, and no one mentioned anything about choice or debate. 40% found the language either quite difficult or very difficult, and overall 30% said the interactive had taught them a lot.

The most significant result was in terms of ergonomics – 60% had a problem using the interactive, including initial difficulties in understanding the challenge, to not really understanding the interactive at all or why they were getting answers right or wrong.

Prototype 1 Prototype 2

Enjoyment 50% found something they disliked

Understanding 55% mentioned GM or genetic modification

0% mentioned choice or debate

40% found the language quite or very difficult

30% said it taught them a lot

Ergonomics 60% had a problem using the interactive

Figure 4: Feedback from the testing of the first prototype

Prototype redesign

We decided to throw the baby out with the bathwater and completely revisit the game-play of this interactive, concentrating more on ‘choice’ and ‘debate’.

The new prototype showed a card game after the introductory screens. It gives the visitor 10 pairs of picture cards they have to turn over two at a time and try to match. On correctly matching a pair, a fact about GM food appears on screen relating to the matched picture. The picture is then assigned to either a ‘Pro’ or ‘Contra’ column (unfortunately this screen shot does not show the columns as no cards have been turned over yet).

When we tested this new prototype, we found a great improvement in both understanding and ergonomics.

Only 45% found something they disliked, so slightly fewer than before. In terms of understanding, however, the results differed significantly. All visitors mentioned GM or genetic engineering, and half of them mentioned it was about the pros and cons of GM food. Only 10% found the language quite difficult, which included foreign visitors whose first language was not English. No one found the language very difficult. A total 80% said

they had learned something new. In terms of ergonomics, the results also differed significantly – only 10% had any difficulties using the interactive.

Prototype 1 Prototype 2

Enjoyment 50% found something they disliked

45% found something they disliked

Understanding 55% mentioned GM or genetic modification

100% mentioned GM or genetic modification

0% mentioned choice or debate

50% mentioned it was about pros and cons of GM foods

40% found the language quite or very difficult

10% found the language quite difficult

30% said it taught them a lot 80% said they had learned something new

Ergonomics 60% had a problem using the interactive

10% had a problem using the interactive

Figure 5: Feedback from the testing of the second prototype

Conclusion

However, we still hadn’t got it quite right yet. Half of the visitors felt the game was too long. And half of the visitors clicked on the cards in the game before the cards were ready to turn over. We therefore recommended to the developers that the number of card pairs should be reduced, and to make the cards react quicker.

The final version has 8 pairs of picture cards instead of 10, and you can also see the ‘pros and cons’ columns in this screen shot, which we renamed ‘for and against’. The cards also respond much faster, which I can’t show you here but if you’re ever in Edinburgh I recommend you visit the Connect gallery and try it out for yourself. Thank you.

Jenni Fuchs

Visitor Studies Officer

National Museums Scotland

June 2007