Augmenting paper based learning materials A pragmatic approach

Sandra Maathuis-SmithSchool of Social Sciences

Open Polytechnic of New ZealandLower Hutt, New Zealand

sandra.smith@openpolytechnic.ac.nz

Gary MershamSchool of Social Sciences

Open Polytechnic of New ZealandLower Hutt, New Zealand

gary.mersham@openpolytechnic.ac.nz

Abstract—Some distance education institutions have invested heavily in print-based resources, and learners still show a preference for print; but some concepts and practical or skills based courses, for example science or engineering, do not translate well into the print medium. This paper presents Augmented Reality (AR) as a possible solution and discusses the development and implementation of an AR application into traditional print learning materials.

Keywords-Distance education; mixed reality; augmented reality; ODL

I. ENHANCIING PAPER BASED LEARNING MATERIALS WITH AUGMENTED REALITY

International evidence (from publications and strategy documents from JISC and EURODL, and the development of user experience design UXD) suggests that in the future. Open and Distance Learning (ODL) will not be either print-based or ‘online’ but will use appropriate mixes of available media designed to meet different learner needs [1].

Teachers and students have reported that the use of AR in teaching and learning has led to high engagement and interactive, situated, collaborative problem solving [2, p.1].

In our example of the rendering of three dimensional ball bearing images, augmentation enables us to exploit the particular power of the computer (regardless of their networked status) as a symbiotic aid to cognition. Computer code is particularly adept at taking one kind of data input and expressing it in novel and revealing ways. So, for example, the viewer we developed allowed students to manipulate the three dimensions of the rendering independently. As Allen [3, p.7]puts it, augmentation involves learning where humans and machines can work together to solve problems and present the results of that investigation. If such augmentation is also networked, it enables further dissemination, analysis and consideration as well.

Many distance education institutions have invested heavily in print-based resources for students. More still are moving to a blended approach where resources are presented in print and some in a digital environment. Over 90% of the learners at the Open Polytechnic continue to express clear preferences for

either print-based distance learning (supported by freephone and email access to lecturer, library and other support services) or ‘mixed mode’ distance learning (the print-based service plus e-support services) [1].

Many disciplines have concepts which do not translate well into the print medium and this is one of the problems facing distance education providers. In practical or skills based courses such as science or engineering, learners find it difficult to get access to physical artifacts such as tools, chemicals and components. Some distance students will have difficulty sourcing many of the artifacts relevant to their subject. For example: plumbing valves, electrical components, chemicals, etc. The question of how this content could be best rendered and delivered to the students via the print medium was investigated by a team at the Open Polytechnic of New Zealand [4]. Aside from delivering the content the team also was challenged to produce an interface where there was minimal set up and low cognitive investment (learning curve) on behalf of the content developer and the user, and also to do it without reliance on networks or online technologies.

This paper describes the process without reliance on networks but the same process can be adapted to deliver the content via intranet, internet or cellular and mobile networks.

The adage goes that ‘A picture is worth a thousand words.’ A 3D image would exponentially increase that to many thousands, but how do you put a number on a 3D image whoseangle of “viewing” you can manipulate yourself?

True learning is experiential and the more senses that are involved, the more powerful the learning experience. Successful E-learning and distance educators are putting in the effort to engage their students in meaningful activities that involve interaction at multiple levels; through sound, sight, touch, emotions, etc. In real life humans are able to move and turn objects freely in space; this natural interaction allows the user to investigate the object from different perspectives. Traditional methods of learning spatially-related content involved viewing 3D objects on a computer screen: The objects are manipulated in space through keyboard commands or mouse clicks. One step to improving this interaction involves presenting learners with 3D visualizations of components with the ability to manipulate the ‘view’ from the users’ perspective. Some doubt has been put on the pedagogical value of Augmented Reality (AR) because of the predominant use of

978-1-4673-4925-3/12/$31.00 ©2012 IEEE November 6-8, 2012, Amman, Jordan2012 International Conference on Interactive Mobile and Computer Aided Learning (IMCL)

Page 58

the technology in entertainment and games. However, Shelton[5] argues that merit lies in the technologies ability to allow for viewing things in a natural environment that otherwise would be impossible to show, such as labels on parts of an engine or forces on the poles of a magnet.

“Augmented reality has the potential to engage and motivate learners to explore material from a variety of differing perspectives, and has been shown to be particularly useful for teaching subject matter that students could not possibly experience firsthand in the real world” [5, p. 1].

A. What is Augmented Reality?Augmented reality technology is a part of the emerging

field of tangible interfaces. These in turn, are part of a wider body of technologies called ubiquitous computing – computing technology that is so embedded in the world that it ‘disappears’.

Unlike virtual reality, where the user is immersed inside a virtual world and explores a simulated reality, AR lets the user see the real world and real objects and superimposes virtualobjects or data over the top. AR is a technology which results in a blending of the real world with virtual objects. To the user it seems that the virtual and real objects coexist in the virtual space.

Augmented reality systems have found application in several areas; surgery, where AR has been used to combine medical imagery scans, ultrasound imaging and equipment readouts to render real time data and images over a patient’s body. Production and maintenance, for example BMW has been using AR head mounted glasses and audio to train the mechanics and technicians in their workshops. In the military AR has been used for simulations and warfare training. AR simulations can allow soldiers to experience unusual or dangerous conditions in a controlled environment. Architecture and Archaeology have been using AR applications for collaborative urban planning [7] and visualising ancient ruins [8]. Augmented Reality is gaining in popularity: Many movies have AR supported websites [9][10] and it is used in advertising and promotion (e.g. GE Smart Grid, Earth Hour - World in your hands ), music videos [11], packaging [12] and in online catalogues and magazines [13] [14].

Augmented Reality is used under several setups including ‘see-through head mounted displays’; In the past viewing devices such as head mounted goggles and helmets were cumbersome and restrictive, both physically and economically. For application in the distance education realm any headsets or specialised equipment are not feasible, but this has been largely overcome with the use of monitor based systems, handheld devices and the ‘magic mirror’ visualisation technique. This technique involves a person holding a marker in view of a webcam – the video stream is captured and sent to the computer for processing through the AR software – the video feed is then sent to the screen showing the real and digital content. The equipment used is inexpensive, and in most cases, common computer equipment and a webcam is all that is needed to visualise 3D content.

For AR to be feasible in the distance education realm several criteria needs to be met for tutors, lecturers, course

development teams, instructional designers and the students. The software needs to be user friendly enough to create, implement and deliver the technology to the students, and the product should be as intuitive as possible for students to use. This was achieved with the use of BuildAR software from HitLabNZ [15], an off the shelf webcam and common low-specification computer hardware.

B. How AR worksThe result was the creation of course materials with

embedded ‘tags’ which were read by the webcam. Briefly, the ‘tags’ are ‘recognised’ by the AR software and matched to a ‘pattern’ which refers to a predetermined 3D file which is then augmented over the tag virtually. The video stream is then rendered on the computer monitor showing both ‘real’ and ‘virtual’ content simultaneously. The ‘tags’ can be just about anything and some recent AR developments are not even using tags. In this case the actual name of the object which will be rendered is used in the ‘tag’ with a large black border for distinguishing the ‘tag’ from the background (the development of ‘tags’ is described in detail later).

Referring to Fig 1: The camera captures video of the real world and sends it to the computer, (1) software on the computer searches through each video frame for any square shapes (the black border of the ‘tag’). If a square is found (2), the software uses some mathematics to calculate the position of the camera relative to the square. The software on the computer then searches through the pre-programmed and stored patterns for a match (3). Once a match is found the corresponding computer graphics model is drawn from that same position (4). This 3D model is drawn on top of the video of the real world and sent back to the display (5). When the user looks through the display they see an image overlaid on the real world (6).

The AR product discussed in this paper allowed the user toview 3D images in perspective and in context with the relevant text describing the object they were viewing. Making the interaction with the object and the course materials as effortless as possible, the traditional input mechanisms of keyboard and mouse were replaced with a web camera which picked up the ‘tags’ embedded in the course materials. Interactions were as naturalistic as possible with the moving of the ‘tag’ relative to the camera changing the orientation of the 3D object on the screen created a different perspective. So hence the user could see the 3D object from many different angles. The computer is able to do this calculation fast enough that the square marker can be moved and the computer graphics will move with it. The only requirement is that the camera has to be able to “see” the whole square marker in order for the tracking to work. This interaction creates a more actively engaged learner.

The three main components in the AR system are patterns, markers and 3D objects; the creation of each of these is discussed with reference to the BuildAR software.

C. Patterns and markers (tags)BuildAR software has a function for the creation of patterns

and markers. A pattern is the image stored in the AR software which corresponds to the 3D object. This is used to match the printed marker ‘tag’ in the course materials.

978-1-4673-4925-3/12/$31.00 ©2012 IEEE November 6-8, 2012, Amman, Jordan2012 International Conference on Interactive Mobile and Computer Aided Learning (IMCL)

Page 59

Figure 1. Overview of how the AR system works. [4, p. Appendix 1]

Markers (or tags) are images which can be printed (embedded in the print material) and are used by the software as a reference point. The image becomes a target that can be ‘read’ by a camera-like device (e.g. webcam). Refer to fig 2.

Figure 2. Example of a Marker

To be effective the marker must be designed in such a way that it is unique enough to be easily identified from others markers and easily distinguished from the other artifacts on the page, such as figures and images. The file format accepted by the software was a bitmap (BMP) format. The BMP file is loaded into the ‘generate markers’ function, the heavy black frame is automatically added and a matching pattern file is saved by the software. These pattern files can be exported as an image file and printed or saved for inserting into documents. So now we have the marker (printed or embedded target image with the thick black frame) and the matching patterns (internalto the software).

D. 3D objectsThe project described in [4] initially used object (OBJ) file

format, but these proved to be very large and took a long time to load in the software. The more compressed (IVE, Three-dimensional model file created by Open Scene Graph [OSG])or native Scalable Vector Graphic (SVG) formats are much smaller so they load very fast. The 3D content in project [4] was obtained readymade and fit for purpose from a suite of engineering objects created by SKF (S'venska Kullagerfabriken the Swedish ball bearing manufacturing company).

There are several software options for creating 3D content and it largely depends on the content you require and the skill set of the developer, but any software which exports to SVG or IVE is recommended for AR application.

Figure 3. Users reading the course text while holding the webcam in preparation of viewing the AR content [4]

Figure 4. An image of a roller bearing, viewed on a computer screen via a webcam focused on the augmented reality ‘tag’ embedded on a page of a

course manual [4].

Fig. 3 shows a user investigating the AR content in the Open Polytechnic project and Fig. 4 is a close up of the final AR rendered content presented to the user. [4]

While there appears to be great potential, relatively little is known about the capability of this technology to support teaching and learning, especially in the distance learning environment. Through the use of augmented reality technology the printed page can become a means to move students from a static, passive consumer to active observer of dynamic content; this means that textbooks and printed course materials no longer need to be static sources of information [7].

This paper has introduced an AR application where there was minimal set up and low learning curve on behalf of the user. Almost anyone with computer skills and basic knowledge of image formats can develop an AR application with BuildAR.

Although the processes described is using desktop computing hardware and no networks or online technologies; any computing technology with a camera capable of streaming video can utilize AR to render 3D content especially handheld and mobile technologies.

1

23

456

Match markers with patterns in

software

Identify 3D object associated with marker

RaceBearing

Video stream from camera

Markers identified by the black marker frame

Render 3D object in video

frame Position and orient object

Search for markers

Find marker position and orientation

978-1-4673-4925-3/12/$31.00 ©2012 IEEE November 6-8, 2012, Amman, Jordan2012 International Conference on Interactive Mobile and Computer Aided Learning (IMCL)

Page 60

REFERENCES

[1] The Open Polytechnic of New Zealand , “Investment Plan 2008-2010,” The Open Polytechnic of New Zealand, Lower Hutt, New Zealand, 2007.

[2] M. Dunleavy, C. Dede and R. Mitchell. Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning. (2009) Journal of Science and Education and Technology. 18:7–22. DOI 10.1007/s10956-008-9119-1

[3] M. Allen. 2011. Networked, Integrated, Augmented: towards a future when all learning is e-learning. Presentation for the Centre for Studies in Higher Education University of Melbourne, March 2011

[4] C. Brown, M. Glaser, S. E. Maathuis-Smith and G. Mersham, “Enhancing learning for engineering trade learners: Augmented paper-based materials in course design,” AKO Aotearoa, Wellington, 2010

[5] B. E. Shelton, “Augmented Reality and Education Current Projects and the Potential for Classroom Learning,” 2003. [Online]. Available: http://www.kabk.nl/lab/docs/education3.pdf.

[6] L. Kerawalla, R. Luckin, S. Seljeflot and A. Woolard, “Making it real”:exploring the potential of augmented reality for teaching primary school science,” Virtual Reality, vol. 10, pp. 163-174, 2006.

[7] UCL Bartlett School of Graduate Studies, “Augmented Round Table for Architecture and Urban Planning,” 2006. [Online]. Available: http://www.vr.ucl.ac.uk/projects/arthur/. [Accessed 27 November 2009].

[8] V. Vassilios, J. Karigiannis and N. Ioannidis, “Augmented Reality Touring of Archaeological Sites with the ARCHEOGUIDE System,” Cultivate Interactive, vol. 9, no. http://www.cultivate-

int.org/issue9/archeoguide/, 2003. [9] “Transformers,” [Online].

Available: http://www.weareautobots.com/nz/index.php. [10] Star Trek, “YouTube.com,” [Online].

Available: http://www.youtube.com/watch?v=v9Zl1e5pAHA.[11] Lost Valentinos music Video, “YouTube,” [Online].

Available: http://www.youtube.com/watch?v=T4Nuje_YCoM.

[12] Doritos, “YouTube,” [Online]. Available: http://www.youtube.com/watch?v=tJ_4tYUIQ8Y.

[13] Mini, “YouTube,” [Online]. Available: http://www.mini.de/de/de/webcam/index.jsp.

[14] “Esquires,” [Online]. Available: http://www.engadget.com/2009/11/10/esquires-augmented-reality-issue-goes-on-sale-and-we-have-vide/.

[15] Human Interaction Laborotory New Zealand (HITLabNZ). “BuildAR”. Available: http://www.hitlabnz.org.

978-1-4673-4925-3/12/$31.00 ©2012 IEEE November 6-8, 2012, Amman, Jordan2012 International Conference on Interactive Mobile and Computer Aided Learning (IMCL)

Page 61