“Wonder Turner” and “The Amazing Cinemagician” Augmented Reality and Mixed Reality Art Installations

Helen Papagiannis

York University, Toronto, Canada

ABSTRACT This paper discusses the approach, evolution and exhibition of the augmented reality art installation Wonder Turner and mixed reality work The Amazing Cinemagician at the Ontario Science Centre Idea Gallery from May 29 through September 6, 2010 in Toronto, Canada, by artist and researcher Helen Papagiannis. Both interactive artworks are featured in the exhibition entitled, The Amazing Cinemagician: New Media Meets Victorian Magic by Helen Papagiannis in association with the Augmented Reality Lab at York University. KEYWORDS: Augmented Reality, Mixed Reality, Art, exhibition, interactive, RFID, FogScreen. INDEX TERMS: J.5 [Art and Humanities]: Fine Arts; H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systms – augmented reality.

1 INTRODUCTION I am an artist and researcher who has been exploring augmented reality (AR) as a creative medium in artistic practice and storytelling since 2005. My work in AR has evolved over the past five years as the technology has advanced, with constraints and new possibilities of the technology guiding my work and process. My approach to working with AR has been one of starting with the technology first, rather than beginning with content. I first study the traits of the technology by experimenting with the medium to comprehend what it does well and how it works best. This exploration then guides the content development, with a story emerging that evolves from the technology as opposed to beginning with a story that I wish to adapt to the technology. I find this approach to be critical in understanding a new medium. Once the characteristics of the technology are grasped, experimentation can push the technology in novel forms. I am presently completing my PhD in Communication and Culture at York University, focusing on the creative possibilities for AR in storytelling, art, and cinematic experiences, and am also a senior research associate at the AR Lab in the Faculty of Fine Arts, Department of Film, York University.

2 ONTARIO SCIENCE CENTRE IDEA GALLERY I was invited by the Ontario Science Centre (OSC) in 2009 to create an exhibition for the Idea Gallery. The OSC is a groundbreaking science museum located in Toronto, Canada, with a hands-on approach to science and technology. The OSC stimulates the imagination, encourages creative thinking and makes science and technology fun through interactive exhibits,

with over 1 million visitors passing through its doors every year. The mandate of the Idea Gallery is to provide opportunities for talented young and emerging artists and researchers to display innovative projects that blur traditional boundaries between art, science, design and technology.��� I had previously exhibited at the OSC in Fusion 2007, Art, Science, Design and Technology Festival, for which I was awarded first prize in the innovation competition for my AR projects.

Figure 1: Gallery visitors interacting with the Amazing Cinemagician installation at the Ontario Science Centre, Idea Gallery. Photo © Helen Papagiannis

3 WONDER TURNER In considering which projects to exhibit, I immediately thought

of my AR work, Wonder Turner, which I planned to grow in scale ���for a large public installation. My original prototype for Wonder Turner was a handheld object measuring 24 inches tall by 4 inches by 4 inches (Figure 2), and consisted of a series of three stacked rotating wooden cubes with AR markers on each side of the cube [1]. The participant could turn each of the individually

E-mail: hpapa@yorku.ca

27

IEEE International Symposium on Mixed and Augmented Reality 2010Arts, Media, & Humanities Proceedings13 -16 October, Seoul, Korea978-1-4244-9342-5/10/$26.00 ©2010 IEEE

stacked cubes 360 degrees to form new compositions in an ‘exquisite corpse’[2] style with multiple choices for heads, torsos and feet, appearing as augmented imagery. Participants could create wondrous new characters including such combinations as a surgeon’s head, belly-dancing torso and penguin feet to a llama head, guitar-playing torso and trampoline feet. My ambition for this work, once tested and successful at a small scale, was to enlarge the piece to a larger human scale, so that the participant could see their body alongside the virtual bodies they chose to compose.

Figure 2: First Wonder Turner prototype, hand-sized. Photo: Eric White

Working with the OSC, we fabricated a large model of Wonder

Turner to 5 feet tall by 10 inches by 10 inches (Figure 3) for children, teenagers and adult visitors to interact with in the Idea Gallery. The small hand-held size of the original prototype permitted only one individual to interact with the work at a time. Now, with the larger model, participants were able to interact in larger groups. Multiple people could turn the cubes together to choose between various augmented imagery and create new compositions collaboratively. The Wonder Turner was placed in front of a large screen with a web camera where participants could see their own body mirrored alongside the transformed AR imagery layered atop the Wonder Turner.

Figure 3: Wonder Turner installation at the OSC Idea Gallery, ‘human sized’. AR imagery of videos of various characters’ body parts appeared on the screen as participants turned each of the large stacked cubes to create new compositions. Photo © Helen Papagiannis

Figure 4: Participant interacting with the Wonder Turner installation at the OSC Idea Gallery. Photo © Helen Papagiannis

28

The role of physical interaction was an important aspect to this work, of being able to physically turn and position the cubes adorned with AR markers to reveal a new scenario. The viewer had to become an active participant to explore the Wonder Turner, unveiling a surprise moving virtual image with each AR marker. Chance also came into play when spinning the cubes in multiple whirls to see where they would stop, with the character combinations resulting in a surprise. The role of the body furthered a connection between the real and the virtual by positioning one’s self in the scene in both mirrored image and physical movement.

Drawn in through a sense of play and the curiosity of seeing their own image reflected on the screen, participants were invited to discover the Wonder Turner. The comical augmented imagery of various body parts further welcomed the viewer as a fun event, not as a technologically intimidating experience. Play is an important aspect of my work in AR; playfulness in the types of imagery I incorporate, and also in practice, in the physical involvement of play and exploring the work.

3.1 Wonder Turner Approach and Process As previously noted, my approach to working with AR begins with the technology first, as opposed to content, with story evolving from and being guided by the technology. Wonder Turner is a project that grew out of my experiments using multiple AR markers, particularly my AR Joiners series, inspired by artist David Hockney (b.1937) [3]. The AR Joiners were influenced by Hockney’s photocollages, referred to as Photo Joiners, where multiple photographic prints were arranged to form a total composition. Taking Hockney’s concept from still photography to a layered virtual moving image, I physically layered multiple AR markers, showing individual 2D video clips simultaneously to create a single 2D video composition. One of my AR joiners, Paris, Eiffel Tower, 2008, was key in inspiring Wonder Turner. Paris was composed of three AR markers arranged vertically atop each other. Each AR video clip was running on a different timeline, causing the collaged image of the Eiffel Tower to appear as a whole, yet composed of varying pieces. I began to think of how this effect could be applied to other work, of multiple images that could be combined to create an identifiable whole. The concept of an exquisite corpse from the Surrealists immediately came to mind.

Instead of paper markers, I envisioned a scenario in which cubes could be used with multiple combinations available to choose from in a secret and hidden way, echoing the style of an exquisite corpse in which the final combination was a surprise with the previous contributor not being able to see what the first or next was going to add. This spirit of surprise is translated in the technology, of the viewer not knowing what they will unveil, with the AR markers appearing as an arbitrary black and white symbol to the naked eye, and the augmented content only revealed on the screen by the computer software.

Wonder Turner has also been applied as a sound installation where each AR marker presents a different looping audio clip connected to a pulsating image of a tonality of colours fading in and out. The participant can rotate each cube to hear a different sound and mix the various sides of the cubes to create new audio and colour compositions. Further, the participant can use their hand or body to interrupt, stop, and replay the sounds by physically blocking the AR marker. In this sense, Wonder Turner can be played like a musical instrument, such as a harp, by concealing and revealing each augmented musical note with the physical placement of one’s hands.

3.2 Wonder Turner Participant Interaction It was interesting to observe participant behaviours while interacting with the Wonder Turner installation at the OSC. A graphic panel with information on the Wonder Turner was placed next to the large screen. The text read:

Wonder Turner Turn the cubes slowly. What do you see on the screen? Wonder how it works? Wonder Turner is an Augmented Reality (AR) experience in an exquisite corpse format that lets you mix and match video clips of unexpected images. When you face the cubes, you only see black and white symbols – but when the cubes face the camera, software transforms the symbols into moving images onscreen.

Although instructions were provided, viewers tended to approach the Wonder Turner object itself first before reading the information text. The participant interaction typically followed this order of events: 1. The participant approached the Wonder Turner, looking at the curious nature of the structure and symbols 2. The participant looked at the screen and was surprised to see their own image mirrored back alongside other video elements of moving body parts, which appeared on the Wonder Turner 3. The participant had a moment of realization that the video on the screen of the various characters was not actually present in the participant’s physical space 4. This was followed by questioning what was happening, which caused the participant to touch the Wonder Turner and discover that it spins to reveal further videos of different characters caused by the black and white symbols 5. The participant then began to play with various combinations of AR markers, revealing and selecting different combinations of characters 6. Participants then play between the virtual elements on the screen and their own physical presence and image on the screen, posing with their favourite character combination and incorporating their own body: ranging from hugging and kissing the Wonder Turner and in turn virtual character which appeared on the screen, to hiding behind the Wonder Turner and extending their own arms or legs to become part of the virtual character. Participants then also take photos of themselves appearing with the character on screen, using their personal camera or phone.

Figure 5: Participants using their physical bodies to become a part of the scene appearing on the screen alongside the virtual character compositions in Wonder Turner installation. Photo © Helen Papagiannis

29

Figure 6: Participants posing with their favourite character combinations in the Wonder Turner installation. Photo © Helen Papagiannis

Figure 7: A participant uses her iPhone to take a photograph of her friend posing with the character on screen. Photo © Helen Papagiannis

4 THE AMAZING CINEMAGICIAN, FOGSCREEN AND RFID Also featured in the OSC exhibition is The Amazing Cinemagician (Figure 8), which integrates Radio Frequency Identification (RFID) technology with the FogScreen, a translucent projection screen consisting of a thin curtain of dry fog. The Amazing Cinemagician speaks to technology as magic, linking the emerging technology of the FogScreen with the pre-cinematic magic lantern and phantasmagoria spectacles of the Victorian era. The Amazing Cinemagician was created in homage to the great French filmmaker and magician Georges Méliès (1861- 1938). Méliès, often referred to as the ‘father of special effects’ in cinematography, became famous for the trick-film, utilizing a stop-motion and substitution technique through the splicing of film, multiple exposures and various other methods. Méliès was a stage magician before being introduced to cinema at a preview of the Lumiere brothers’ invention, where he is said to have exclaimed, “That’s for me, what a great trick” [4].

Figure 8: The Amazing Cinemagician installation at OSC featuring RFID and FogScreen. Photo © Helen Papagiannis

The Amazing Cinemagician installation is based on a card-trick,

using physical playing cards as an interface to interact with the FogScreen. RFID tags are hidden within each physical playing card. Part of the magic and illusion of this project was to disguise the RFID tag as a normal object, out of the viewer’s sight. Each of these tags corresponds to a short film clip by Méliès, which is projected onto the FogScreen once a selected card is placed atop the RFID tag reader. The RFID card reader is hidden within an antique wooden lectern (adding to the aura of the magic performance and historical time period), which in turn is connected to a Sony VAIO microcomputer running MAX MSP [5] also out of view. The VAIO is then connected to the ceiling-mounted projector, which projects the corresponding film clips onto the FogScreen. To further heighten the mood and echo the Victorian era, artefacts from the OSC’s archives were featured in a display case behind the lectern, including a magic lantern, stereoscope and large film splicer, adding to the aura of the installation (Figure 9).

Figure 9: The Amazing Cinemagician antique lectern and artifacts including stereoscope, film splicer and magic lantern added to the aura of the installation. Photo © Helen Papagiannis

30

The following instructions were provided to the participant: “Pick a card. Place it here. Prepare to be amazed and entertained.” (Figure 10) Once the participant places a selected card atop the designated area on the lectern (atop the concealed RFID reader), an image of the corresponding card is revealed on the FogScreen, which is then followed by one of Méliès’ films. The decision was made to provide visual feedback of the participant’s selected card to add to the magic of the experience and to generate a sense of wonder, similar to the witnessing and questioning of a magic trick, “How did you know that was my card? How did you do that?” This curiosity would ideally inspire further exploration of each of the cards (and in turn, Méliès’ films) to determine if each of the participant’s cards could be properly identified.

Figure 10: The Amazing Cinemagician installation, top of lectern with playing cards shown and FogScreen projection. Photo © Helen Papagiannis

Unlike the Lumieres’ approach to cinema, which was rooted in “documenting everyday actualities”, Méliès was interested in the fantastical illusions cinema could give form to [6]. In the article “Méliès the Magician: The Magical Magic of the Magic Image” (2007), André Gaudreault discusses Méliès’ use and refinement of the stop-camera technique for its magic effects, “it is employed to produce appearances, disappearances and sleight-of- hand-tricks” [7]. Méliès’ film Les Cartes Vivantes, or The Living Cards (1904), is included in this installation as one of the films projected onto the FogScreen. Méliès appears in the film as a magician performing a card trick; however, this magic trick is far from ordinary. In addition to objects shifting forms, appearing and disappearing, in this magic performance the inanimate becomes animate, with images of figures on cards depicted as coming to life. With the use of stop-substitution, a live woman dressed in costume emerges from the Queen of Hearts card (Figures 11-12). The trick continues with the King of Diamonds, using the same effect. Méliès uses the newfound medium of cinema to extend magic into novel, seemingly impossible visualities.

Figure 11: Still from Méliès’ film, “Les Cartes Vivantes”, or “The Living Cards”, 1904.

Figure 12: Méliès takes on the role as magician in the film bringing the cards to life, as shown with the Queen of Hearts.

In The Amazing Cinemagician we see the conjuring not of

literal ghosts, but rather the ghost of early cinema re-appearing in the form of Méliès’ films projected onto fog. Now having looked back to the past of cinema, what might the future of FogScreen hold for the future of cinema and emerging technologies? As we look ahead, what role will tactility and materiality play? The Amazing Cinemagician is a combination of new and old media, bridging the tactile with the digital. Including Méliès’ work in this installation also speaks to tactility. In addition to the notion of technology as magic, perhaps best described by Arthur C. Clarke, "Any sufficiently advanced technology is indistinguishable from magic", Méliès’ work speaks to a tactile exploration of media. For as Gaudreault writes:

“Indeed the essential quality of Méliès’ magic is produced, in a remarkable and primordial way, by fragmentation, cutting, and breaks in continuity. In his films, Méliès’ magic wand was, first and foremost, a pair of scissors.” [8]

“Contrary to what we have been told, trick effects by means of stop-camera technique in order to carry out a substitution are accompanied, in Méliès’ work by a touching up carried out after the fact, using scissors and glue.” [9].

The FogScreen enables possibilities for a new kind of materiality and tactility both in programming interaction via software in touching the FogScreen as well as movements which are not programmed, but left to the participant, including the ability to walk-through the image projected on to the fog, place one’s hands through the image, and even blow on the image to break it apart. The ethereal quality of these floating ghostly

31

images renders them virtual, yet there is a peculiar tactility that the fog brings to these images allowing the participant to touch the virtual and even move it around with one’s hands or breath. I’d like to propose that this work is a mixed reality in this regard, where a new kind of environment is created where physical and digital/virtual/imagined objects co-exist and interact. This is seen in the RFID tags embedded in the playing cards to conjure the digital projection in real time on the FogScreen as well as the dual nature of the virtual tangible fog projections.

5 AUGMENTED REALITY EXHIBITION TAKEAWAY Viewers of the OSC exhibition were invited to take a postcard

from the exhibition home with them, which featured an AR marker and a website for viewers to engage in an AR experience outside of the gallery. The takeaway featured an AR Magic Tunnel Book, where viewers could hold up the AR marker to a webcam and explore multiple AR scenes via the website using the same single marker on the postcard, or download and print one from the website [10]. I designed the first scene, entitled Magic Theatre (Figures 13, 14), which echoed the themes of the exhibition featuring a mutli-layered stage with magicians performing magic tricks that the viewer could turn and explore from various angles. Playing cards flew overhead, circling the stage, with Méliès’film clip, Les Cartes Vivantes, playing at the back of the stage. Three other scenes by Andrew Roth and AR Lab Director Dr. Caitlin Fisher were included that the viewer could cycle through by selecting “next scene”.

Figure13: Magic Theatre OSC Exhibition Takeaway, AR Magic Tunnel Book experience. Photo © Helen Papagiannis

Figure 14: Alternate angle. Photo © Helen Papagiannis

6 CONCLUSION This paper has discussed the evolution of the augmented reality artwork Wonder Turner from a hand-sized prototype to a large human scale installation where the participant is able to see one’s body next to the virtual characters that appear on the screen. This larger installation at the OSC, as opposed to the earlier hand held prototype, also welcomes multiple participants to engage in the experience in groups; whereas the smaller scale was designed for a single user at a time. Wonder Turner applies an exquisite corpse format to unveil surprise augmented videos for the participant, who rotates the large cubes with AR markers to create new combinations of characters. The role of chance comes into play as participants can spin the cubes to create random combinations or selectively turn through the cubes to arrive at their desired combination. Participants often have a favourite composition among the various options and enjoy posing with their character, behaving playfully and taking photographs of the screen. The FogScreen and RFID installation, The Amazing Cinemagician combines old and new media using a series of playing cards as an interface to project film clips by ‘the amazing cinemagician’ himself, Méliès. The participant is left wondering how the magic of the card trick is performed, with their selected card appearing from the fog. The aura of the installation is further heightened by incorporating an antique weathered lectern into the exhibit as well as authentic objects from the OSC’s archives including a magic lantern, large film splicer and stereoscope.

REFERENCES [1] Wonder Turner uses AR software, SnapDragon, developed by our

AR Lab at York University by Andrew Roth, Andrei Rotenstein, and Mikhail Sizintsev, utilizing MAX MSP and the AR library by Dr. Mark Fiala.

[2] An ‘Exquisite Corpse’ is a method invented by the Surrealists by which a collection of images or words is assembled. The sequence of images or words is added to by multiple collaborators, with each person unaware of what the previous person contributed. The final result is a collective work that often results in a surprise and/or humorous combined image or string of words.

[3] Papagiannis, Helen. “Augmented Reality (AR) Joiners, A Novel Expanded Cinematic Form.” 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Orlando, Florida, Oct 19-22, 2009.

[4] Petro, Patrice. “Fugitive Images: From Photography to Video.” Indiana University Press, 1995. p.62.

[5] The application this project uses in MAX MSP was programmed by Andrew Roth, Technology Manager, Augmented Reality Lab, York University.

[6] Tafler, David. “When Analog Cinema Becomes Digital Memory.” Wide Angle 21.1 (1999): 181-204. p.184.

[7] Gaudreault, André. “Méliès the Magician: The Magical Magic of the Magic Image.” Early Popular Visual Culture 5.2 (2007): 167-174. p.171.

[8] Ibid. 167 [9] Ibid. 168 [10] Architecture created by Andrew Roth using FlarToolkit.

32