A Mobile Augmented Reality Environmentfor Collaborative Learning and Training

Reiner WichertZGDV, Computer Graphics CenterMobile Information Visualization

Darmstadt, Germanyreiner.wichert@zgdv.de

Abstract: The following work focuses on the requirements for a Collaborative AugmentedReality System with the usage of Internet technologies to create new possibilities forcollaborative teaching. In most learning environments, learning is imparted through seeingand hearing. In this paper, Collaborative AR Discussion and personal experience are alsointegrated in order to achieve a quicker transmission of study material and an increase inmemory efficiency. We have analyzed the scenarios occur most frequently in service andmaintenance and also the literature of Tele-Teaching. The needed requirements were derivedand as a result integrated in a test system. This collaborative game simulates a web-basedtraining and teaching environment in Collaborative AR as a new approach for teachers andtrainees and describes scenarios and research perspectives for distance education and trainingat the same location. The game was used as a testbed to clarify the problems associated withAR and Collaborative Learning.

Introduction

Through the integration of Multimedia, study material can be transmitted much faster. In 1965,William Glasser described what influences a student’s ability to retain information and ideas in his bookControl Therapy in the Classroom. Respective to this thesis, people remember 10% of what they read, 20% ofwhat they hear, 30% of what they see, 50% of what they see and hear, 70% of what they discuss with others,80% of what they personally experience and 95% of what they teach others (Glasser 65). With CollaborativeAR, learners communicate directly with each other in a face-to-face communication, but also interact withremote learners located in other learning rooms. According to Glasser, and as it relates to the interactive aspectin the learning environment of a Collaborative AR, people can remember information in a very efficient way,because users can discuss and have experiences by interacting with the augmented world in the collaborativelearning and training environment.

Tele-presence as a multimedia aspect and the access to worldwide distributed information enable anew way of teaching and learning. By combining learning and multimedia, a couple of net-based teaching andtraining systems have already been realized. By Tele-Teaching, we mean the distribution of knowledge in asynchronous Online Scenario or net of computers, in which the teachers and students work simultaneously. ByTele-Training, we mean the provision of study materials on network servers. These materials are retrieved bystudents asynchronously. Tele-Teaching is usually associated with university study, while Tele-Training ismore suited for further professional training in a cooperate setting (Effelsberg).

Augmented Reality (AR) has become an important part of computer graphics. “Unlike Virtual Realitywhere the physical world is completely replaced with synthetic environments, in Augmented Reality environ-ments, 3D computer graphics objects are mixed with physical objects to become part of the real world”Billinghurst says (Billinghurst 00). Computer Supported Cooperative Work (CSCW) has also been expanded asan identifiable interdisciplinary research field. According to Wilson, CSCW is a generic term, which combinesthe understanding of the way people work in groups with the enabling technologies of computer networking,and associated hardware, software, services and techniques (Rodden 91). By combining CSCW withAugmented Reality, a new collaboration - Collaborative AR - became possible. Collaborative AR is defined byReitmayr and Schmalstieg, where co-located users can experience a shared space that is filled with real andvirtual objects (Reitmayr 01). The use of CSCW technologies in combination with Tele-Teaching and Tele-Training opened a new application area known as Computer Supported Cooperative Learning (CSCL)(McCONNEL 94). But it is a new idea to take CSCL into Collaborative AR. With Collaborative AR, groupdiscussions are possible, where each user can interact and attain personal experience. In reference to Glasser, itis the best way to achieve a quicker transmission of study materials and an increase in memory efficiency.

Findings

Today, the ability to communicate with almost anyone, to exchange knowledge, solve problems, andlearn new things via computer networks is widespread. We know that we learn most effectively if we learntogether with others in groups. The Internet allows cooperative learning independent of time and place (IPSI).

Almost all of the existing e-learning systems do not support AR. So, one of the advantages of thecomponent-based System will be to support individual users in their simultaneous interaction with augmentedobjects by using Collaborative AR while exchanging knowledge.

To specify the requirements for a cooperative AR learning environment, the scenarios that occur mostfrequently in service and maintenance have been analyzed and two standard situations were identified:1. cooperative interaction in a master / trainee scenario at the same place with e.g. skilled workers and a

teacher having different views to the augmented world, standing and discussing in front of a machine;2. direct interaction with a remote expert using AR Technologies like interactive video and having the same

view as the skilled worker. The expert can interact with the skilled worker and offer support.From these standard situations we have derived the following needed technical and interactive require-

ments. A collaborative AR system should give support and assistance to a learning group in their commonwork. Multiple users can be at the same place, at the same time, interacting together directly in the augmentedworld. Furthermore, group discussions are supported within the field of Augmented Reality. A common centralview is established for each participant in order to get a defined understanding of the problem and to meet therequirements for this specific problem. Another aspect of teaching and collaboration is that an AR system has toprovide remote support with the help of Augmented Reality over great distances. However, with a mobileaspect, the system would give multiple users access to information and help at any given time. It should clarifytechnical aspects, such as how to distribute the functionality in a learning collaboration and which methods canbe used to get visual or haptic feedback. To meet the requirement of interaction by a collaborative group, agame could be implemented using the Collaborative AR Prototype System with the advantage of 3D groupinteraction and individual views for multiple users co-operating at the same place or with a connection over theInternet to remote users. As a result, a 3D Tetris clone was implemented as a testbed for different scenarios.

The Tetris game poses a good possibility for realizing the requirements of a collaborative AR learningenvironment, because it can be played at the same place or from a remote location. It is possible to logindynamically and share the AR Tetris game with other participants. It simulates e.g. the collaboration of skilledproduction workers and the connection of technicians to a remote expert for support over the Internet.Exploring this research topic within a game presents a significant challenge.

The Tetris game is split in the functionality realized by the Tetris Application and the visualization bythe rendering component of the AR system. Both are embedded inside the web page, which displays points, thenext piece, and the augmented view of the game (Fig. 1). The Tetris application can be started by any playerand other users can login dynamically and share the AR Tetris withthese participants. To realize this aspect, the status of the game istransmitted into the collaboration component. The first player whostarts the game has the role of the master. The other players do notstart another game, but get this status at the beginning of theirsession and initialize their game with these settings. If a playerrotates a piece, the orientation of the actual piece will be set into thesystem. If a translation is done, the new position of the center of thepiece is set in the system. A video server captures the picture of thecamera and provides it to the rendering component and to themarker-based tracking system. A component was needed to handlethe communication between the main components of the base ARsystem and to notify other components or other clients co-operatingtogether when values are changing.

Figure 1: 3D AR Tetris clone

In the case of the collaborative learning environment, the values to transmit are the orientation of thescene, the names of the virtual objects used and also the actions like translations or rotations. The values arepresent on both the client and server side and are compared by way of a Push Mechanism. The PushMechanism has been conceived to forward a notification to the clients immediately after an event occurs. Bothclient- and server-sided components can be informed via notification by the Event Mechanism. Another

component is responsible for controlling the collaboration. If two users are changing a value at the same time,this component has to recognize and regulate the collision detection and the common concurrent interaction.

Scenarios

This Collaborative AR System can be used in a wide range of educational, e.g. in the field ofcommerce, to help users in their homes by way of a furniture company consultant, or in the industrial field, tosupport skilled workers standing in front of a machine by enlisting an expert. By using Collaborative AR totransmit study materials, a Cooperation at the same place is enabled by transmission of the scene, the actionsand the orientation of the virtual objects to all participating users in the group; a remote cooperation over greatdistances is also enabled, requiring additionally the camera image of one of the cooperating users. Thus,training in a collaborative group interacting together at the same place or via distance education can beportrayed in a lot of scenarios. Both forms of cooperation can also be transmitted to the participantssynchronously, as well as asynchronously. With an asynchronous connection, a later alteration of theaugmented world is possible through assigning and storing scene, actions and orientation to a specific frame inthe videostream. In this way, a later correction can be carried out to better portray an error or problem and toheighten the learning experience. In contrast, with synchronous learning in groups, collision recognition interms of a simultaneous access of the virtual object is necessary. By implementing a Collaborative AugmentedReality learning environment, applications in the field of Tele-Teaching and Tele-Training become both syn-chronously and asynchronously realizable. For a remote synchronous learning environments, the remote usercan have the same model of the real world, which has to be augmented, and can interact the same way as acollaborative user at the same site. In the case of the Tetris game, both users have the model of a pen or ajoystick for interaction and the board of the game. Another synchronous possibility is to transmit the videoimages of one user to the remote user. The interaction can be done with an interactive graphic and will be sentthrough the event mechanism, as well. In the case of face-to-face collaboration, a pen with markers is used for3D interaction. This allows a very natural interaction mechanism with the game. In a later version, speechrecognition will be integrated to enable a multi-modal interface and, according to Glasser, we can improve theeffectiveness of remembering information with the aspect of speech.

But it is also possible to realize asynchronous teaching and training collaborations. We need to storethe camera images of at least one user and also the actions like translations or rotations of the scene, theorientation of the virtual objects and the name of the scene dependent on the frame number of the images. If westore the images of more than one user, we can also switch between the views of the users. In this asynchronousscenario, the images are retrieved from memory, are identified by the corresponding frame number and aresuperimposed with the formerly valid objects. However, the virtual objects can be altered later as to position ororientation, or other virtual objects can be loaded. In this way, errors can be better explained and understood.This can occur on-site in the real world by projecting the virtual objects time-dependent to the recording in thereal world with the help of a head-mounted display (HMD), or in a remote scenario, displayed on the desktoptogether with the video content. Discussions concerning a particular action could be subsequently initiated andproblems better be solved. Explanations could be offered as to the point in time when something was donewrong and how it could be made better could be visualized.

The Tetris game supports multiple users playing one game. This simulates group discussions and theconnection to a remote teacher collaborating with a student while assembling a mechanical engine. It alsohandles the problem of common competing interaction. The actual field of view of other players and theirdifferent games can be integrated into the view of a teacher. One special view can be selected and theinteraction can be switched to the selected game. This represents the scenario of a master who supervisesseveral trainees learning the AR system or being instructed how to handle a machine. The master can select theview of a trainee and can give him advice and support in his augmented environment. As a result, a master cansupervise a few learners.

Another aspect is the view of common and private data by the collaboration participants dependent onthe expertise of the users. Thus, each user of the learning collaboration can customize the view to his needs orto see different aspects of the same thing adapted to the circumstance. Converted to the Tetris game, the usersplay the same game and get a private view of the game and their interactions. The common data is the nextpiece and the score. The users can add and customize visual aspects to their needs like the deleted levels or thetime played.

How to visualize the result of cooperation by a learning group will be a further step in this work. Ifmany people play the same game remotely, a player wants to know why the piece moves to the left even though

he/she pushed the piece to the right. Therefore, the representation of the results of the interaction is animportant field. The best way to do this will be to use multiple indications like color, arrows and numbers (Fig.2) or the player can get haptic feedback.

42%42%37%37%

21%21%

Figure 2: Visualization of results Figure 3: Collision detection and Semaphores

There are a lot of difficulties to solve in sharing virtual learning environments. Mutual exclusions have to beimplemented and collision detection is needed. Semaphores have to control the access to interactionmechanisms. This problem can be approached in one game for two players, where two pieces are falling down(Fig. 3). Each person can interact with one piece, gets personalized information and has to coordinate his workwith the other player while getting information when collisions are detected.

Conclusions

We have analyzed specific scenarios. The needed technical and interactive requirements for thesescenarios have been derived. A collaborative AR environment was suggested and realized prototypically as atest system for a Tetris game. The system can be used in a wide range of educational settings and it allows anasynchronous, as well as a synchronous access e.g. to a learning environment that can be used remotely fordistance education or in a collaborative group interacting together at the same place in a master – traineescenario. It is possible to login dynamically to the same group and share the augmented space with otherparticipants at any given time. It was observed that the fast event mechanism can provide the information toother participants in real time by pushing the information. A further step for the future is to evaluate theoutcome of this cooperative game and to transfer these initial results to education applications. But until then, alot of experiences can be garnered and further research work completed.

References

Glasser, W. (1965). Control Therapy in the Classroom, Harper & Row: NY, 1986; Reality Therapy:A NewApproach to Psychiatry, Harper & Row, NY.

Wolfgang Effelsberg, Lehrstuhl für Praktische Informatik IV, Universität Mannheim, Lehren und Lernen imInternet, http://www.igd.fhg.de/igd-a6/itti1.html

Billinghurst, M., Poupyrev, I., Kato, H., May, R. (2000). Mixing Realities in Shared Space: An AugmentedReality Interface for Collaborative Computing, ICME2000, New York, USA.

Rodden, T. (1991). A survey of CSCW systems, Interacting with computers - the interdisciplinary journal ofhuman-computer interaction, 3(3), pp.319-353.

Reitmayr, G., Schmalstieg, D. (2001). Mobile Collaborative Augmented Reality, ISAR2001, New York.

McCONNEL, D. (1994). Implementing Computer Supported Cooperative Learning, Kogan Page Publishing.

Fraunhofer IPSI, Integrated Publication and Information Systems, Darmstadt, http://ipsi.fhg.de/CSCL/

ARVIKA, Augmented Reality for Development, Production and Servicing, Germany, http://www.arvika.de/