Immersed gaming in Minecraft

Milan Loviska1, Otto Krause1, Herman A. Engelbrecht2, Jason B. Nel2,Gregor Schiele3, Alwyn Burger3, Stephan Schmeißer3, Christopher Cichiwskyj 3,

Lilian Calvet4, Carsten Griwodz4,5, Pål Halvorsen4,5,1Territorium-Kunstverein, Austria, 2University of Stellenbosch, South Africa

3University of Duisburg-Essen, Germany, 4Simula Research Laboratory, Norway, 5University of Oslo, Norwaymilan@loviska.com, info@ottookrause.com, {hebrecht,jason}@ml.sun.ac.za,

{gregor.schiele, alwyn.burger, stephan.schmeisser, christopher.cichiwskyj}@uni-due.de, {lcalvet,griff,paalh}@simula.no

ABSTRACTThis demonstration will showcase mixed reality technologiesthat we developed for a series of public art performances inVienna in October 2015 in a collaboration of performanceartists and researchers. The focus of the demonstration is onnatural interaction techniques that can be used intuitivelyto control an avatar in a virtual 3D world. We combinevirtual reality devices with optical location tracking, handgesture recognition and smart devices. Conference atten-dees will be able to walk around in a Minecraft world byphysically moving in the real world and to perform actionson virtual world items using hand gestures. They can alsotest our initial system for shared avatar control, in which auser in the real world cooperates with a user in the virtualworld. Finally, attendees will have the opportunity to giveus feedback about their experience with our system.

Categories and Subject DescriptorsH.4 [Information Systems Applications]: Miscellaneous

KeywordsMinecraft; position tracking; head tracking; gestures

1. INTRODUCTIONMedia content should be presented to an audience in the

most immersive manner. The means of achieving this differvastly. Nothing prevents a gamer from becoming totally im-mersed in the symbolic world of Dwarf Fortress, whereas asimple sharpening algorithm can hinder the biggest fan of aTV series from becoming immersed in a 4K movie on the lat-est TV set. The pinnacle of immersion is putatively achievedby allowing a user to interact freely in a virtual world by con-trolling every bodily aspect of their avatar through motionsthat feel natural and don’t have to be trained for the specificactivity in the virtual world.In recent times, a number of systems have reached the

mass market that enable such a direct control over avatars

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

MMSys’16 May 10-13, 2016, Klagenfurt, Austria

c© 2016 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-4297-1/16/05.

DOI: http://dx.doi.org/10.1145/2910017.2910632

Figure 1: Collaborative action. Foto: http://eSeL.at

in some manner. A commercial success was triggered bythe physical interaction options provided by the input con-trollers of the Nintendo Wii console. Motion sensors embed-ded in the controller allow players to perform movements of,e.g., tennis rackets or car wheels. The Microsoft Kinect re-peated this success in a different way, by interacting throughgestures, without manipulating a physical device. In paral-lel to these new input devices, there is a re-awakening of theconcept of virtual reality (VR) glasses (a 1960s concept [3]).These provide an immersive experience in terms of visualinput by giving the user a first-person 3D view into a vir-tual world, while shutting out the real world. By combiningthese input and output devices, it is possible that we willachieve an unprecedented level of immersion.Still, open questions remain. Current systems restrict the

user to a relatively small physical area, and moving aroundin a large virtual world therefore requires different means,e.g., using a classical controller. Interactions require specificgestures that are often totally different from movement foran interaction in real life. As an example, jumping is usuallyachieved by pressing a button on a controller or by makinga special gesture. This requires training, especially if veryaccurate inputs for complex interactions are needed. It canlead to totally different bodily experiences for the user andbreak immersion.Our work aims at offering users in a virtual world a more

natural experience with more freedom, e.g., allowing themto move larger distances without resorting to buttons ona device and enabling more complex interactions withoutlengthy training. In our demo, we invite one attendee ata time to explore a large Minecraft world that was createdfor a series of art performances in the Third Life Project.They can control their avatars through physical head andbody movements, and manipulate the virtual world throughhand gestures. Other attendees are not restricted to anobserver role, but they are invited to perform real-world

actions that exert additional influence onto these avatars.They can actively help or hinder the attendee exploring theworld. Figure 1 shows a moment of collaborative dancingfrom the performance. To get an impression of the user’sexperience, a video that summarizes the performances inwhich the system was used can be accessed here: http://thirdlifeserver.org/media.html.In the remainder of this demo paper, we first discuss the

context of our work, the Third Life project. Then we presentthe user interactions of our system and how each one of themis realised. After the artists’ experiences during the perfor-mances, we elaborate further on the proposed demonstrationand conclude the paper with a short summary and outlook.

2. THE THIRD LIFE PROJECTThe Third Life Project1 is an ongoing cooperation be-

tween performance artists and computer science researcherson the topic of "third life" – exploring the potential of vir-tual actions to transgress directly into reality and performextravirtual actions.Avatars afford us already the opportunity to live a “sec-

ond life” in a virtual environment. Thanks to the psycholog-ical relativity of human perceptions, a “human mind largelytreats virtual people just like physical ones” [1, p.84]. There-fore, avatars can address the issue of dual identities, exer-cising the gap between online persona and offline self andprovide us tools to reconsider our own real life authenticity.The Third Life Project aims to extend the remarkable ca-pacity of this technology and reduce the gap by inventingand implementing strategies and technology for direct en-gagement with elements of real environments through ele-ments of the virtual ones. Positioning such a project withinthe context of contemporary digital and live performanceart practice not only creates a platform for confronting andsharing ideas between groups that might not have an oppor-tunity to come together otherwise, but through the inclusionof and direct interaction with larger audiences as well grantsus a unique opportunity to examine playfully the social ef-fects of representational practices in virtual environments.The demonstration focuses on a single part of the Third

Life Project, the direct influences that a performer’s actionsin the real world can have onto the virtual world by directlycontrolling a first-person avatar through a wide variety ofmovements. Attendees in the demonstration will performthis role. They will experience several aspects of the per-formance, namely (1) viewing the virtual environment, (2)interacting with virtual objects, (3) natural movement inthe virtual environment, and (4) a multi-player control overa single avatar.

2.1 Virtual environmentWe developed two Minecraft worlds, designed to showcase

different actions like jumping, dancing, opening and clos-ing of doors, as well as allowing the avatar to roam freelythrough different settings like indoor spaces, narrow hall-ways, staircases, tunnels, and large scale outdoors environ-ments. The worlds help to establish a basic storyline whilesupporting the implementation of the used technologies ina way that can be intuitively understood by an observingaudience. We chose Minecraft because of our previous ex-perience, but applications in other types of virtual world1http://www.thirdlifeserver.org

would be possible as well. Both worlds contain two differ-ent virtual representations of the WUK performance venue2,which serve as an entry and exit point from the real worldto the virtual one and back. To break the logic of the realworld, the first world around WUK is an open space thatmashes up greenery with a desert environment, and containsa village, a huge eyeball hanging in the sky above and pro-gramming code flying loosely in the air. In the desert, onemeets a giant server representing Kubrick’s Space Odyssey2001 monolith with a floating foetus inside. It contains aherd of non-player characters, virtual pigs, which can be re-leased and guided back to the WUK in the course of thepresentation. The second world is darker and fantasy-likewith mushroom forests, cobwebs, water and lava beams thatone can observe while travelling in a mine cart. The longrailroad passes along the virtual upside-down version of theMuseum of Modern Art in New York City (MoMA)3 andleads to a discotheque, where the avatars can dance andafterwards get teleported back to WUK.

2.2 Viewing the virtual environmentFor immersing a user in a virtual environment, it is im-

portant to embed the senses of the user into the 3D envi-ronment in a manner that is comfortable and natural. VRhardware is currently very popular and affordable as evi-denced by the many companies developing VR gear. Wechose to use the well-known head-mounted display (HMD)Oculus Rift DK2 (Oculus) for enabling the user to view thevirtual environment. The advantage of using the Oculus isthat the user is able to change his view of the world by simplymoving their head. The Oculus promises low latency, withthe goal of minimising simulation sickness for the user [4].Another motivation for using the Oculus was the existenceof an open-source project4 that had already modified theMinecraft client to support the Oculus.

2.3 Interacting with virtual objectsUsing the Oculus allows the user to see the virtual envi-

ronment in a natural manner, but not to interact with vir-tual objects. It blocks out the real world, which introducesa barrier to the use of traditional input devices (keyboards,mice, etc.). To re-introduce interaction, and to make a nat-ural experience, we want the user to use his hands instead ofsuch devices. We use an off-the-shelf motion controller, theLeap Motion5 to recognise hand gestures, effectively turningthe user into the input device. The Leap Motion containsa pair of infrared cameras and its software recognises theuser’s hands in the cameras’ field of view, which can thenbe used for gesture recognition.We mount the Leap Motion on the front of the Oculus (us-

ing a mount that can be obtained from Leap Motion specif-ically for this purpose) and integrate it into the Minecraftclient. Gestures are mapped to emulate keyboard or mousebutton input, allowing users to interact with virtual objects.The users interact with virtual objects within range by aim-ing the cursor at the object with their right hand and thenperform a gesture to interact with the object, e.g., by form-ing a fist. To select objects and place them in the virtual2http://www.wuk.at3Minecraft was added to the video games collection of theMoMA in 2013.4https://github.com/mabrowning/minecrift5https://www.leapmotion.com

environment, the users need to navigate and select items inan inventory. They interact with in-game menus (and theinventory) as if the menus are virtual blackboards hangingin front of them. It is noteworthy that the cursor can bemoved independently of the users’ looking direction.

2.4 Moving in the virtual worldThe Oculus Rift allows the user to look around in the

virtual world, but we wanted to achieve immersion beyondthis. It should be possible to translate movement in the realworld into movement in the virtual world.Our approach uses a single camera worn by the user in

addition to the Oculus and Leap Motion, which observes aset of pre-installed markers. These markers are CCTags6 [2],due to their proven resistance to motion blur and partial oc-clusion. After reconstructing the placement of the markersin space, we can track the position and orientation of theuser’s torso, and translate this into movement in the vir-tual world. However, the real-world movement is naturallylimited by the demo space. The user movement in most ofthe demo area is translated directly into movement in thevirtual world with a 1:2 ratio, which was determined em-pirically as a compromise between movement accuracy andfaster movement. However, the outer edge of the demo areais turned into a scrolling area. As soon as the user entersthis region of the demo area, the avatar starts to move con-tinuously in the direction that the user is facing. The usercan change the movement direction by turning his body. Tostop continuous movement, the user has to step back intothe inner demo area.To provide a haptic feedback for the transition from ab-

solute movement area to scrolling area, we defined the innerarea by a soft carpet, distinct from the hard, flat surface ofthe scrolling area. This allows a user to feel when he entersor leaves the inner area.In the scrolling area, we use body orientation to determine

movement direction, although we contemplated using abso-lute position for scrolling direction. With respect to move-ment direction, scrolling is a movement mode that allowsthe user to cross long distances in the virtual world. Theindependent movement of the view that is provided by theOculus provides freedom, similar to looking out of a driving

6https://github.com/poparteu/CCTag

Figure 2: Moving in Minecraft, Foto: http://eSeL.at

car. Observing a changing situation while scrolling, how-ever, will eventually create the desire to change direction.With the position-based approach, this requires a step backonto the carpet, a move across the carpet, which leads to un-intended local movement, and leaving the carpet for anotherscrolling direction. Instead, we found that relying on torsoorientation, achieved better immersion. The user can turninside the scrolling area, and changes scrolling direction inthe virtual world. This is experience in way similar to, e.g.,using roller skates and thus feels more natural. In essence,this allows the user to switch between something similar tonormal pedestrian movement and floating.

2.5 Shared avatar controlExtending on the aforementioned interaction techniques,

we also experimented with shared avatar concepts. A sharedavatar is controlled by multiple users at the same time.This way each user can concentrate on different aspects ofthe avatar behaviour and can use optimised input devices.This requires the users to coordinate their actions with eachother but has the potential to perform more complex avatarbehaviours. In our experiment we let two users share oneavatar. The first user was fully immersed into the virtualworld and was moving and interacting using the techniquesdiscussed before, i.e. an Oculus, a Leap Motion Controller,and our location tracking system. The second user waspresent in the real world and experienced the virtual worldusing a traditional 2D screen. This gave him the freedomto move around quickly in the real world and perform ac-tivities that would be difficult or potentially dangerous forthe user immersed in the virtual world. We chose the sec-ond user to control avatar jumping and teleportation. Torealise a natural interaction also for this user, these avatarbehaviours were initiated by performing analogue activitiesin the real world. As an example, to make the avatar jump,the second user jumped on a real trampoline with embed-ded sensors. We also used this to let the avatar dance. Toteleport the avatar to different virtual world locations, thesecond user carried a physical block to different locationson stage, similar to carrying the avatar to different targetlocations on a miniature map. Again, embedded sensors de-tected the block movement and reported it to our system.Overall, this can allow the second user to act as a kind ofguardian angel for the first one. If the avatar is stuck in ahole in the virtual world, the users can combine their actionsto jump the avatar out of the hole. If the avatar cannot getfree or needs to get away quickly from a dangerous situa-tion, the second user can initiate a teleport. Depending onthe application, this may lead to interesting power dynamicsbetween the users, like the second user playing with the firstone by deliberately misusing his avatar behaviours.Clearly, we only tried initial experiments with the shared

avatar idea. Still, the potential for novel interactions in thevirtual world as well as between users is very high. It shouldbe examined in more detail.

3. PERFORMER’S EXPERIENCEAfter describing how a user can perform different interac-

tions, we now discuss how our performers experienced thissystem on stage. In general, immersion was greatly en-hanced by giving the performer the ability to act naturallyand experience the virtual world using different senses.Due to tests and rehearsals, our performer was using our

system for several hours per day, often with only a few shortbreaks. We expected that wearing the Oculus Rift DK2 withits relatively low-resolution display for such extended peri-ods would lead to simulation sickness. This was true forour performer when wearing the Oculus Rift while sittingin front of a computer, even for short periods of use. How-ever, when moving on the carpet in the physical space of thetheatre stage with the headset on, no such condition was ex-perienced, regardless of the duration of the interaction. Thehuman body seems to be capable of adapting better if thevisual input is accompanied by correspondent motoric activ-ity. Please note that this seems to be true even if the user’smovement in the real world is not perfectly reflected in thevirtual world. Our system induced noticeable delay and –especially during tests – miscalculated movements.Another observation is that our simple way of providing

the performer some haptic feedback on his location in thereal world (using the carpet) worked very well. Being “cutoff” from the physical surrounding increases the danger ofpotentially loosing the orientation and location in the phys-ical space. Walking barefoot on a carpet allowed the per-former to be constantly reminded of the borders of his phys-ical area. Besides, the square shape of the carpet relativelymatched the blockiness of the Minecraft world and that ge-ometrical correlation smoothed his orientation significantly.The performer could probe with his foot to find the carpetto prepare leaving the scrolling area. More sophisticated op-tical feedback like virtual shadows could be useful but couldalso break optical immersion.An unsolved problem with the Leap Motion gesture recog-

nition is that it requires the hands to be in its cameras’field of view to track gestures. This sometimes confused theperformer, especially for abstract gestures. When trying tograb a virtual object, the performer intuitively looked at theobject, thus causing no problem. However, we also experi-mented with hand gestures for more general activities, e.g.,sending a signal to the outside world. This caused confu-sion since after a very short time the performer tried to sendsignals while looking around in the world. Not seeing theposition of his own hands, he failed to notice that they werenot visible to the Leap Motion camera, and assumed his sig-nal was not received or ignored. In general, not being able tointeract with objects without looking at them does not cor-respond with our real-life experience. Thus, mounting theLeap Motion on the Oculus Rift seemed to be disruptingimmersion. This needs to be explored further.Finally, we found that interactions that required both per-

formers to cooperate with each other was very engaging forboth of them. One performer could teleport the other’savatar from one virtual environment to another. By jump-ing on a real trampoline he could make the avatar jumprepeatedly (dancing in the discotheque) or get him out ofplaces he had fallen into. Especially the jumping case re-quired exact coordination between the performers, becauseone performer had to walk in a certain direction when theother started jumping. This shows that the real-life multi-user cooperation can enrich immersion by giving the interac-tions with virtual environments an interpersonal dimension.As a side note, the performer in the real world also felt moreimmersed because he didn’t feel like a mere observer.

4. DEMONSTRATIONAfter conducting several artistic performances, we want to

give users the possibility of experiencing our system them-selves. The user of the demo can either enter the virtualworld by wearing the camera rig with Oculus and Leap Mo-tion, or interfere with another user’s actions, by use of thetrampoline. After making some progress since the perfor-mances, the rig can be used wirelessly in the demo. Theuser can explore the Minecraft world freely and interactwith it. The demo is rather resource-intensive. Whereaswe will bring all equipment, there is a considerable demandfor space of at least a 5x5m2 area to experience both walkingand scrolling operation, and the installation of the CCTagsbenefits from a ceiling height of at least 5m.

5. CONCLUSION AND FUTURE WORKOur results so far are promising. We were able to create

a high level of immersion by combining a number of – in-dividually well known – interaction techniques into a singlesystem. Extending sophisticated technology like VR HMDsand gesture tracking with haptic feedback from a passivecarpet as well as the idea of a scrolling area, our performerscould control their avatar with relative ease and accuracywithout extended training sessions.In future work we plan to continue our collaboration and

extend our system with new and enhanced interaction fea-tures. First, we plan to optimise the user experience byreducing system latency and giving the user better feedbackabout the current localisation accuracy. Second we want toextend the system with novel ways of how actions in thevirtual world might influence the real world. So far, this isrestricted to controlling the lighting on stage. Finally, wealso plan to increase audience immersion by giving audiencemembers more abilities to interact with the virtual worldthemselves.

6. ACKNOWLEDGMENTThe Third Life Project was supported by City of Vienna’s

Department of Cultural Affairs, the Arts Division; the Artsand Culture Division of the Federal Chancellery of Austria;the FP7 project no 609757, FiPS; the H2020 project no644874, POPART; Stellenbosch University in South Africa.The authors thank the LABO Mixed Realities and the Uni-versity of Bielefeld for their contributions.

7. REFERENCES[1] J. Blascovich and J. Bailenson. Infinite Reality:

Avatars, Eternal Life, New Worlds, and the Dawn ofthe Virtual Revolution. William Morrow, New York,NY, USA, 2011.

[2] L. Calvet, P. Gurdjos, and V. Charvillat. Cameratracking using concentric circle markers: Paradigmsand algorithms. In Image Processing (ICIP), 2012 19thIEEE International Conference on, pages 1361–1364,Sept 2012.

[3] M. L. Heilig. Stereoscopic-television apparatus forindividual use, Oct. 1960. US Patent 2,955,156 A.

[4] E. Regan and K. Price. The frequency of occurrenceand severity of side-effect of immersion virtual reality.Aviation Space and Environmental Medicine,65(3):527–530, 1994.