University of South Australia

School of Computer and Information Science

Minor Thesis

Master of Computer and Information Science

Modifying 3D Objects inAugmented Reality

Environments

Student:Quang Le

Supervisor:Dr. Christian Sandor

July, 2011

Contents

1 INTRODUCTION 81.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.2 Research Question . . . . . . . . . . . . . . . . . . . . . . . . 91.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.4 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 REQUIREMENT ANALYSIS 112.1 System Interactivity Model and Scenario . . . . . . . . . . . . 112.2 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3 Technical Analysis . . . . . . . . . . . . . . . . . . . . . . . . 122.4 Summary of Reqirements . . . . . . . . . . . . . . . . . . . . . 14

3 BACKGROUND 153.1 Augmented Reality environments . . . . . . . . . . . . . . . . 153.2 MR Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.3 Haptic devices . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.4.1 Collision Detection . . . . . . . . . . . . . . . . . . . . 203.4.2 Deformation Methods . . . . . . . . . . . . . . . . . . 223.4.3 Other Object Manipulation Researches or Programs . . 24

4 SYSTEM DESIGN 254.1 System Configuration . . . . . . . . . . . . . . . . . . . . . . . 254.2 System Architecture . . . . . . . . . . . . . . . . . . . . . . . 264.3 System Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3.1 3D Model . . . . . . . . . . . . . . . . . . . . . . . . . 284.3.2 Painting . . . . . . . . . . . . . . . . . . . . . . . . . . 294.3.3 Deformation . . . . . . . . . . . . . . . . . . . . . . . . 304.3.4 Subdivision . . . . . . . . . . . . . . . . . . . . . . . . 314.3.5 OpenCL . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.3.5.1 Parallel Computing. . . . . . . . . . . . . . . . . . . . . . . . . 33

4.3.5.2 CPU vs. GPU. . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3.5.3 OpenCL. . . . . . . . . . . . . . . . . . . . . . . . . 36

1

5 IMPLEMENTATION 385.1 Manipulation Algorithm’s Flowchart . . . . . . . . . . . . . . 385.2 Moving Direction . . . . . . . . . . . . . . . . . . . . . . . . . 395.3 Intersection and Updating Vertex position . . . . . . . . . . . 41

5.3.1 Checking for Sphere . . . . . . . . . . . . . . . . . . . 425.3.2 Checking for Cylinder . . . . . . . . . . . . . . . . . . 43

6 CONCLUSIONS 456.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7 REFERENCES 48

2

List of Figures

1 Paint on a virtual shoe in AR, shown in TEDx Adelaide 2010 . . . . . 102 Interactivity Model (Courtesy of Magic Vision Lab) . . . . . . . . . . 113 Relationship between coordinates . . . . . . . . . . . . . . . . . . . 134 X3D in AR program . . . . . . . . . . . . . . . . . . . . . . . . . 145 An AR scene (Courtesy of ECRC) . . . . . . . . . . . . . . . . . . 156 Mechanism of a Video See-Through Head Mounted Display . . . . . . . 167 A CANON Head Mounted Display (Courtesy Magic Vision Lab) . . . . 168 AR Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 A marker in the program . . . . . . . . . . . . . . . . . . . . . . . 1710 Tracking by marker in MR Platform . . . . . . . . . . . . . . . . . 1811 User paints on virtual cup using haptic device . . . . . . . . . . . . . 1912 Phantom OMNI . . . . . . . . . . . . . . . . . . . . . . . . . . . 2013 Spidar - String based haptic . . . . . . . . . . . . . . . . . . . . . 2014 Building Bounding Volume Hierarchy . . . . . . . . . . . . . . . . . 2115 Building Bounding Volume Hierarchy . . . . . . . . . . . . . . . . . 2216 Bump shape modified by various tool (Courtesy of Noble and Clapworthy) 2317 Discretize in x in [0..1](3) . . . . . . . . . . . . . . . . . . . . . . . 2418 Polyhedral form(4) . . . . . . . . . . . . . . . . . . . . . . . . . . 2419 Table of 3D Object Manipulation Researches . . . . . . . . . . . . . 2520 AR Working Space . . . . . . . . . . . . . . . . . . . . . . . . . . 2621 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 2622 Component Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 2723 A decomposition of a 3D cube model . . . . . . . . . . . . . . . . . 2824 Class diagram of data structure for storing 3d model . . . . . . . . . . 2925 Texture mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 2926 Painting effect . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2927 The stylus deforms the objects surface . . . . . . . . . . . . . . . . . 3028 Update position for a vertex . . . . . . . . . . . . . . . . . . . . . 3029 Big polygons created when updating vertices positions . . . . . . . . . 3130 Big polygons created when updating vertices positions . . . . . . . . . 3231 People solve puzzle in sequence . . . . . . . . . . . . . . . . . . . . 3332 People solve puzzle in parallel . . . . . . . . . . . . . . . . . . . . . 3333 Class diagram of Model3D structure . . . . . . . . . . . . . . . . . 3434 Structure of CPU and GPU(5) . . . . . . . . . . . . . . . . . . . . 3535 Benchmarks of CPU and GPU(6) . . . . . . . . . . . . . . . . . . . 3636 Manipulation Algorithm’s Flowchart . . . . . . . . . . . . . . . . . 3837 Moving direction at different points on the stylus . . . . . . . . . . . 3938 Benchmark for Paint Function . . . . . . . . . . . . . . . . . . . . 4539 Benchmark for Deformation function . . . . . . . . . . . . . . . . . 46

3

40 Paint on a virtual shoe in AR, shown in TEDx Adelaide 2010 . . . . . 4741 Deform a 3D model of a bunny . . . . . . . . . . . . . . . . . . . . 47

4

ACKNOWLEDGEMENT

It is a pleasure to thank those who make this thesis possible.Firstly, I would like to thank my supervisor Dr. Christian Sandor for hishelp, support and patience during the last year. I would thank him for giv-ing chance to work in the Magic Vision Lab where I could have a great chanceto improve my knowledge.Secondly, I am very grateful for the help of members from the Magic VisionLab for their support especially Ulli, Arindam and Peter. Their advices arevery helpful for my thesis. I am greatly appreciated for the demonstrationfrom Donald in OpenCL. I am really proud to be a member of the MagicVision Lab. Thank you all for giving me chance to work with you.Above all, I would like to thank my family. They gave me the opportunityto come here and study in Australia. Especially to my wife Hong, she wasalways the great support at all times.Without your help, this thesis would not have been possible. Best wishes foryou all!

Quang Le10/07/2011

5

DECLARATION

I, Quang Nhat Le, certify that all the work presented in this thesis is my ownwork. It has not been submitted, either the whole or in part, in any collegesor universities before. Wherever contributions of others are involved, everyeffort is made to indicate this clearly, with due reference to the literature,and acknowledgement of collaborative research and discussions.

Quang Nhat Le10/07/2011

6

ABSTRACT

Model manipulation is always a major task in the field of industrial design.Traditional method, which physically manipulate model, has high risk ofmaking wrong or broken models. This wastes time and money of designer torepair and build a good model. There have been researches on manipulatingobjects virtually to overcome the limitations of traditional method. However,virtual manipulation is hard to use and requires long time to study.In this thesis, we introduced a new system for object manipulation on Aug-mented Reality Environment. User could benefit from the advantages ofvirtual manipulation such as safe manipulation and saving time and money.Also, with the enhancement of augmented reality environment, user couldinteract with the system more natural and thus, easier to use than on virtualenvironment.We proposed a simple mathematical approach to 3D object manipulation incomparison with other existed methods. The core manipulation algorithmsin this system were implemented on OpenCL to utilize the power of parallelcomputing. Also, the system was investigated to be an extension part ofX3D in order to help other developers to re-use it effectively.A demonstration of one major part of this system, which is the paintingfunction, was shown in TEDx Adelaide in 2010. The demo has received alot of attraction and good feedbacks from audiences.It can be seen that thestudy, which is manipulation object in augmented reality environment, is apotential trend and worth to be more investigated.

7

1 INTRODUCTION

1.1 Motivation

In the field of industrial design, bad or broken models are always a primaryconcern for designers. A model can get corrupted due to various reasonssuch as wrong color choice or bad manipulation. Once model is corrupted,a fresh new model is required to accomplish the task. This means an extraamount of money and time has to be spent to build a new model to a statejust before the previous model was corrupted.Current state of the art allows designers to design and manipulate threedimensional (3D) models through virtual manipulation. There are set ofmodeling softwares such as Blender, ZBrush, Maya which allow designers tocreate, modify, save and load models virtually. Using these tools, designerscan load a new model in a second instead of spending time and money tobuild a new one. Moreover, the ability to undo any erroneous manipulationon a virtual 3D object makes the designing experience hassle free and easierthan physical manipulation. These functions are the major advantages ofthe virtual manipulation helping designers to overcome almost the issues oftraditional method. However, the virtual manipulation still has some limita-tions.First, 3D modeling programs are hard to use and requires users to spendmonths to familiarize. Second, the input method for virtual manipulation,which is mostly based on mouse and keyboard, is in a two dimensional (2D)environment. Although, there are techniques for 3D input using mouse andkeyboard such as Arcball (Shoemake K., 1992) and shortcuts, these tech-niques are still inadequate to provide a comfortable environment to the users.The difficulty of virtual manipulation is one of the main reasons that inspireresearch on techniques to build simpler and more interactive programs. Itwill be more natural for users’ interaction to manipulate 3D objects if theyget a realistic sense of vision and tactile feedback than virtual manipulation(Sandor et. al., 2007). Recently, Visuo-Haptics Mixed Reality (VHMR) sys-tems that allow users to see and touch virtual objects are being investigated(Sandor et. al., 2007). Some of these studies proved the possibility of creat-ing 3D modeling program. First, the Visionary Painting Application (Sandoret.al. 2007) in VHMR allows user to use virtual brush and paint on a virtualteacup. Second, the Haptics-Based Deformable Solids (Kevin et. al. 2002)successfully emulated the deformable virtual clay material. Third, the HapticPottery Modeling (Lee et. al., 2008) enables users to deform the virtual clayand create pottery models using haptic tool. Especially, the Visuo-Hapticsin Augmented Reality (VHAR) framework developed at Magic Vision Lab

8

(Eck U., 2010) enables developers to create high fidelity VHAR programs inshorter time with less effort as it simplifies the programming phases such asinteraction with devices, calibration, and integration. This opens the possi-bilities to create a better 3D modeling program which allows users to directlyinteract with the model using a haptic device (for touching sense) in a real3D environment by wearing a head-mounted display (HMD).One of the main challenges when creating 3D modeling programs in VHARis real-time manipulation, as user manipulates the model the changes shouldrapidly be updated on the model. This requires an efficient method for colli-sion detection (Lee et. al. , 2008). In 2008, The Khronos Compute WorkingGroup released the framework for parallel computing OpenCL (Open Com-puting Language). OpenCL enables developers utilize the parallel computingfeature on either Graphic Processing Unit (GPU) or Central Processing Unit(CPU). This is also a potential solution for the challenge of real-time collisiondetection.According to some successful implementations and some potential solutionsfor creating 3D modeling program in AR such as VHAR framework andOpenCL, it is possible to conclude that there is a good potential of creatinga better 3D modeling program that will overcome the disadvantages of thephysical and the virtual manipulation methods.

1.2 Research Question

The motivation inspires this thesis to solve these following questions:- Is it possible to create a program that can paint on a 3D object in

augmented reality environment?- Is it possible to create a program that can deform a 3D object in aug-

mented reality environment?- Integrate the painting and deforming feature as a part of VHAR so that

developers can reuse them to create another AR program.

1.3 Contributions

This study contributes in these following fields of Computer Graphics andAugmented Reality:

- Since almost prior studies on deformation are based on Finite ElementsMethod, those studies require a high complexity of computation as well ascomplicated mathematical knowledge. This provides a simple mathematicalapproach for simulating deformation compared with other prior studies.

- It provides a method on simulating for the mesh-to-mesh deformation byusing parallel computing in OpenCL for Haptic device while the supported

9

deformation function for Haptic is using point-based deformation.- It provides some extended functionalities, such as painting and deform-

ing functions, for VHAR Framework of Magic Vision Lab(Eck U., 2010) thatallows user to create rapid applications for their study.

1.4 Acknowledgement

There is one big achievement of this project as the painting program has beenshown in the TEDx Adelaide in 2010 and we have got many good feedbacksfrom audiences.

Figure 1: Paint on a virtual shoe in AR, shown in TEDx Adelaide 2010

www.youtube.com/watch?v = U2Y E2lHULWA

This is also a strong motivation for me to continue to do the rest of thisresearch.

10

2 REQUIREMENT ANALYSIS

This chapter analyses the requirements for this project. First, this will pro-pose an interactivity model of the system and the scenarios of using it. Sec-ond, this will define users and categorizes users into specific group based onusers activities. Next, this will analyze the required techniques that used tofulfill the requirements. The final part is the summary of the requirements.

2.1 System Interactivity Model and Scenario

The model below is a proposal model for a complete 3D Modeling Programin AR. This project does not cover all features of the program but a part of it.

Figure 2: Interactivity Model (Courtesy of Magic Vision Lab)

The images above describe how the system works. Basically, the usage of thesystem will be in 3 steps. Step 1, a 3D object will be created by placing a realobject in between 2 kinects. Kinect is a Microsofts Gaming Device which wasfirstly announced in June 2009 in a project named Project Nathal(1). Kinectcan detect user motion as well as the depth information of the real scene byusing Infra-red structured light(2). Back to the system, the two kinects willobtain the depth-map information of the real object and then combine tobuild a 3D model of the real object. Step 2, recent created 3D model will beloaded into the AR workspace. The 3D model will be located in a specificposition. User uses the head mounted device to see the object and he usesthe Phantom Omni Haptic device to modify the 3D object. For example, hecan paint or deform the object. Step3, after the manipulation is finished, the3D object will be saved in to hard disk or be printed by 3D printer.This project just focuses on the second step which allows user to paint anddeform a 3D object using a Phantom Omni Haptic device.

1”E3 2010: Project Natal is ”Kinect””, June 13, 2010, IGN, Viewed August 20, 2011¡http://xbox360.ign.com/articles/109/1096876p1.html¿.

2”How Kinect depth sensor works stereo triangulation?”. Mirror2image.wordpress.com. Viewed 20

11

2.2 Users

There are 2 main types of users who will interact with the system.Designer or Application User Users of this type experiment treat thesystem as a black-box without knowledge about the low-level technique ofthe system. They use the provided features to create or manipulate 3D ob-jects.Developer Users of this type have a deeper knowledge about the sys-tem compared with users in Application User group. They can inherit theprovided abilities of the system such as painting and deforming in the imple-mentation level to create their own programs or to experiment another fieldof AR.

2.3 Technical Analysis

According to the proposed system, user will interact with AR by using a headmounted display for graphical presentation and a haptic device for manipu-lating virtual objects. The head mounted display is responsible for showingthe real world merged with virtual objects (Azuma R., 1997). However, thereis a challenge that is how to render a virtual object in a specific location inthe AR scene. This requires a method that strongly registers the relationshipbetween virtual object and real scene (Vallino J. and Brown C., 1999). Toovercome this challenge, we use MR Platform to match the virtual world andthe real world by using markers (Uchijama et al., 2002).As user moves the stylus of the haptic devices, it collides with the virtualobject in the scene. When a collision of the stylus and the object occurs,the object will be updated in some properties such as shape or color at theintersection point. However, haptic device and head mounted display areseparated devices and working independently. Each device works on its localcoordinate without any information about others coordinate. Therefore, col-laboration between coordinates is required to do precise collision detection(Vallino J. and Brown C., 1999). To solve this, we establish a relationship oftwo devices coordinates via markers coordinate.

August, 2011, ¡http://mirror2image.wordpress.com/2010/11/30/how-kinect-works-stereo-triangulation/¿.

12

Figure 3: Relationship between coordinates

Figure above shows the relationship between coordinates. First, we placea marker in a specific location so that we can identify the transformationfrom markers coordinate to the haptics coordinate. Therefore, haptic devicecan get information about the marker coordinate. Second, using MR plat-form get the markers coordinate. Routing between coordinates can be donevia the markers coordinate after some Affines transformations.The next requirement of this program is about real-time manipulation. Theprogram should be able to response immediately if there is any modificationmade on the model. For example if user uses the haptic stylus and pusheson the model, in intersection area should be instantly deformed. To achievethis, we need a good method to detect the collision between the stylus andthe virtual model. In the program, the stylus will be overlaid by a 3D modelof a modification tool which is a set of polygons. Therefore, mesh-to-meshcollision detection is required. However, the complexity of mesh-to-mesh de-tection naively is O(n2) (Backman N. , 2010). This requires a huge checkingtimes even for just a simple model. We will use Open Computing Language(OpenCL) that utilizes parallel computing ability, in order to reduce thechecking time (Khronos Group, 2008).The last requirement is to make this project to be reusable. This featureallows other programmers to reuse the existing functions to create anotherAR programs. To achieve this, the major functions such as painting anddeformation should be implemented as libraries so that other programs canimport them to use.

13

Figure 4: X3D in AR program

For this project, AR programs will be defined in X3D files which is a fileformat for presenting 3D computer graphics developed by X3D SpecificationWorking Group. Therefore, the painting and deformation functions will beimplemented as X3D nodes for the reusability. User will defines these nodesin X3D files and load them by H3D launcher provided by VHAR framework(Eck U., 2010).

2.4 Summary of Reqirements

According to the analysis, we can summarize the requirements for the systemas following:Functionality User can use the program to manipulate 3D objects in twoways: Painting and Deforming object. This is the major objective of thisproject.Precision The manipulation should be applied correctly. This means therouting between devices coordinates should be correct to get accurate ma-nipulations. This is the second major objective of this project.Real-time Operation The program should be able to work with normalobject (less than 10.000 polygons) in real-time responsibility, particularly, atleast 25 frames per second.Reusability The program should be implemented in a way that other pro-grammers can reuse as much as possible. The objective of this program, thepainting and deforming function should be implemented as x3D nodes.

14

3 BACKGROUND

This section aims to provide detailed information that has briefly introducedin the requirement analysis. There are two parts in this section. The first partwill introduce backgrounds of required techniques that have to be applied tothe system. After that, the next part will provide a survey of some relevantresearch.

3.1 Augmented Reality environments

Augmented Reality is an alteration of Virtual Reality (VR) (Azuma R.,1997). In VR, user is working in the virtual environment without percep-tion of the real world. On the contrary, AR is a synchronization of realworld and virtual objects as it shows both in the same workspace (AzumaR., 1997). In AR, users perception of the real world is modified with thevisual information which generated by a computer so that user can inter-act with that information (Eck U., 2010). This brings new approach to thehuman-computer interaction that applications such as medical training, en-tertainment, navigation could benefit from that (Azuma et al., 2001).

Figure 5: An AR scene (Courtesy of ECRC)

The picture above shows a real table and real phone. Also, there are virtualchairs and a virtual lamp around and on the table. User has to wear an opticalsee-through head mounted display to be able to see this. The head-mounteddisplay is an AR device that used to combine real and virtual (Azuma R.1997).

15

Figure 6: Mechanism of a Video See-Through Head Mounted Display

Figure 7: A CANON Head Mounted Display (Courtesy Magic Vision Lab)

First, camera records the video of the real world and transfers to the VideoCompositor. Second, the head tracker tracks the movement of the headto get the head location and send to the Scene Generator. Based on thehead location, the scene generator will generate the graphic images thatappropriate to the users view and send to the Video Composer. Third,Video Composer combines the video of real world and graphic images andsends to Monitors. Therefore, user can see a mixed of real and virtual on themonitors.Next section will discuss MR Platform which is a tracking system of theproject.

16

3.2 MR Platform

As described in AR environment, Head-mounted Device is responsible forrecording the real scene and then adds virtual object in displays. The sceneof real world will be recorded by camera of HMD and transfer to simulator.Then, simulator will add virtual elements to the recorded scene and rendersto the displays that showed to user.

Figure 8: AR Environment

noindent Virtual object has to be rendered at a specific point in the workspace.This means the simulator has to know the correct position in the recordedimages to add virtual object. However, the recorded data from HMD, whichis just images, does not contain the information about any specific locationof the real world. Moreover, users perspective of the real world recorded byHMD will be changed as user moves his head when working. This causesdifferences in frames that record from HMD. Therefore, it is difficult for sim-ulator to localize a position in the real world.In order to overcome this issue, a tracking method is required. In fact, therehave been existed many tracking methods available classified into 2 groups:marker and vision based (marker-less) tracking (Rolland et al., 2001). In thisproject, we used a marker based system, which is MR Platform, for trackingmethod.

Figure 9: A marker in the program

17

Markers are printed images that used to localize the coordinate system foran AR scene. A marker has special figures that can be recognized by thesimulator. Using markers, simulator can extract their features to get theexact pose of the camera relate to them (Kato et al., 2000).The above image is an example of a marker of MR Platform that had beenused in this system. In MR platform, there can be maximum of one hundredvarious markers used in a program. A number is assigned for a particularmarker to distinguish with another marker. User can render 2 different ob-jects by using 2 different markers.A marker based tracking system requires a camera to record the real worldscene, and a system of pre-defined markers to be recognized in the scene.Here is the mechanism of a marker-based tracking system.

Figure 10: Tracking by marker in MR Platform

We used markers in MR Platform in order to show virtual object at a correctlocation in the workspace. First, camera records the world scene includingmarkers and transfers these image data to computer. Second, MR Platformwill receive that data and extracts marker information from the scene. Oncemarker is extracted from the real scene, MR platform will analyze markerfeatures to identify the global coordinate. Third, based on the retrieved co-ordinate, we draw virtual objects to the scene and then display to user.

18

3.3 Haptic devices

The interaction between human and computer is mainly based on mouseand keyboard. However, keyboard is specifically designed for text input andmouse is originally designed to be a screen pointer (Subramanian et al., 2005).Although, there are research to enhance interactivity of mouse and keyboardin 3D environment, they are still inadequate to provide a realistic 3D feelingto users (Hamam et al., 2010). For example, user cannot feel the depth orthe force feedback because it is impossible to transfer these kinds of data tomouse and keyboard.

Figure 11: User paints on virtual cup using haptic device

Haptic, which is originally from a Greek word meaning to touch, is a tech-nique that provides high-fidelity human-computer interaction method. Usinghaptic devices, users can experiment the tactile feedback in a 3D environ-ment. For example, in the figure above, a user is painting on a virtual cupusing haptic device (Sandor et al., 2007). There are several types of hapticdevices. First, the PHANTOM device, which is built as a robot arm, enablesuser to interact with virtual object using the stylus. Second, the string-basedhaptic device SPIDAR-G that provides naturally manipulation with virtualobjects. SPIDAR-G not only allows users to touch but also the width of thevirtual object (Kim et al., 2002).

19

Figure 12: Phantom OMNI Figure 13: Spidar - String based haptic

CourtesyofMagicV isionLab

3.4 Related Work

This section discusses some prior work that relevant to this research. Thefirst part will look at some methods to do collision detection and how toreduce the computational complexity for that. The second part will discusssome methods that do deformation. After that, the last part of this sectionwill introduce some typical object manipulation research or programs thathave been done in both virtual and AR environments.

3.4.1 Collision Detection

In fact, virtual objects are images lacking physical constructed elements. Iftwo objects are overlapping each other, there will be no physical effect occurs.This causes problems in programs that simulating the high-fidelity physicssimulation such as surgery simulator or 3D modeling program. Therefore,collision detection is required to overcome this problem. Collision detectionis the core technique of various applications that try to simulate the physicalpresence of the real world such as haptic rendering or geometric modeling(Zhang X. and Kim J. Y., 2007).However, collision detection requires a strong computational ability for edge-face intersection check (Smith et. al, 1995). Naively, the complexity of meshto mesh collision detection is O (n2) which n is the number of polygons in-volved in the test (Backman N., 2010). According to this complexity, check-ing the collision for 2 normal models, each created by 1000 polygons, shouldrequire 1000 * 1000 = 1.000.000 checking time. Moreover, if the program isrequired for real-time running, suppose 25 frames per second, the checkingtime will be 25.000.000 times in a second. Therefore, there have been manyintensive researches, which try to reduce the complexity of the algorithm, on

20

collision detection problem (Zhang X. and Kim J. Y., 2007).There are three main approaches for collision detection: Bounding VolumeHierarchy (BVH), Distance Tracking and Spatial Division (Weipeng A. andZhibang C., 2010). Bounding Volume are regions, which are normally box orsphere areas that approximately cover the shape of an object (Smith et al,1995).

Figure 14: Building Bounding Volume Hierarchy

Figure above shows the algorithm of creating Bounding Volume Hierarchy.First, bounding volumes will be calculated for each object. Second, a hierar-chy structure will be set up based on bounding volumes. The collision detec-tion will be done between bounding volumes instead of checking edges-facesintersections. This method quickly localizes the collision area and avoidschecking on irrelevant objects (Smith et al, 1995).The second method, Distance Tracking is operated by monitoring distancesbetween each pair of objects. Test for collision detection will be done if thedistance between two objects is smaller than a limit value (Smith et al, 1995).

21

Figure 15: Building Bounding Volume Hierarchy

The third method for Collision Detection is Spatial Division method.The idea of this method is to recursively divide the space into partitions.The division method is defined by some initial definitions such as horizontalsplitting or vertical splitting. Then, a hierarchical structure will be built,based on some alignment methods such as axis-alignment, to operate thecollision detection for the whole spatial subdivisions (Jin et al, 2009).

3.4.2 Deformation Methods

This section will not go too deep into technical matter as a lot of mathemat-ical knowledge required. Therefore, the objective of this section is just tooutline some methods that deal with object deformation problem.In 1987, Terzopolous developed a pioneer study on simulating deformableobjects (Terzopoulos et al., 1987). His research was a foundation for manyposterior studies on the same field. Recently, researches on deformable ob-jects have been investigated for gaming industry, simulation systems (Mulleret al, 2005). There have been many approaches in simulating deformableobjects such as the Finite Difference Approaches (Terzopoulos et al., 1988),Finite Element Method (Muller et al., 2002), NURBS-based Free-Form De-formations (Lamousin and Waggenspack, 1994).In the Finite Difference Approach, Terzopoulos suggested a method to breaka deformable object into two parts, one is deformable and the other is rigidpart. The locations of mesh in rigid part were calculated by a displacementfunction while the deformable part was operated independently. However,the deformation didnt operate consistently on the whole body of the object asthey use one rotation matrix for the whole model (Terzopoulos and Witkin,1988).

22

In 1994, Free-Form Deformation (FFD) method based on Non-Uniform Ra-tional B-Splines (NURBS) was introduced by Lamousin and Waggenspack.This method provided a real-time deformation effect on simple lattice ob-jects. However, the ability in deformation with complex objects wasnt shownin the study (Henry and Waren, 1994). Based on the study of Lamousin andWaggenspack, Noble and Clapworthy proposed a new NURBS-based FFDmethod in 1999. In this study, the authors introduced three deforming toolsto modify the bumps shape by changing the weight of the control points ofthe NURBS.

Figure 16: Bump shape modified by various tool (Courtesy of Noble and Clapworthy)

This method requires a method to generate suitable NURBS FFD meshwhich still on investigated (Noble and Clapworthy, 1999).One famous method, which is used in many studies on object deformation,is Finite Element Method (FEM). FEM is a numerical approach for solvingthe partial differential equations with specific ranges of variable. The ideaof this method is to discretize the complex continuous of a function intoapproximate functions in smaller value ranges. For example, the figure belowshows a function that was partitioned into approximate functions, the redlines represent for the approximate functions.

23

Figure 17: Discretize in x in [0..1](3) Figure 18: Polyhedral form(4)

In Mullers study (2004), he used Finite Element Method to divide a structureinto a number of smaller elements that turn the object into polyhedral form.The global deformation will be fragmented to local deformation at verticeson polyhedral. Then, deformation vectors vertices of the polyhedral will becalculated by interpolation functions. This method provided a fast way toestimate the rotational field while the stiffness matrix is changed frequently.Also, it provided the ability to do real-time animation for deformation objects(Muller et al., 2004).

3.4.3 Other Object Manipulation Researches or Programs

This section shows a comparison table of some other relevant applications orresearches in object manipulation.The comparison is carried out base on four criteria: Paint, Deform, Haptic,and AR Environment. Value of criteria: Paint, Haptic, AR Environmenthas Yes/No values. Yes indicates the application supports the appropriateability and No indicates not support. Value of Deform column is denoted bynumeric value. These are 0, 1 and 2 which represent for Not Support, PointBased Deformation and Mesh to Mesh Deformation respectively.

3Alexandrov O., 2006, viewed 26 May 2011,¡http://en.wikipedia.org/wiki/File:Finite-element-method-1D-illustration1.png¿

4Alexandrov, O. 2007, viewed 26 May 2011,¡http://en.wikipedia.org/wiki/File:Finite-element-solution.svg¿.

24

Figure 19: Table of 3D Object Manipulation Researches

4 SYSTEM DESIGN

In this chapter, we provided information of how the system is designed inboth physical and logical detail. We also describe the methodologies of twomain functions of the system: Painting and Deforming 3D objects.

4.1 System Configuration

This section describes the physical structure of the system including hardwaredevices and how they are physically organized.As have been discussed in Requirement Analysis section, there are severaldevices take place in the system:

- For AR visualization, we used CANON Head-mounted Device.- For tracking, we used MR Platform and markers to set a common co-

ordinate to integrate other devices coordinates.- For tactile interaction with the system, we used the Omni Phantom

Haptic device. We placed the Omni Phantom device in the middle of thework space and surrounded by a number of markers. The reason why we usedmore than one marker is to avoid tracker lost. When user interacts with thesystem his hand or parts of the haptic device can hide the marker.

25

Figure 20: AR Working Space

The figure above shows how devices in working space are organized. Useruses Head-Mounted Display for seeing the scene and using phantom to ma-nipulate the virtual object. The Phantom haptic device is placed in betweenof markers. The markers are carefully measured to localize the global coor-dinate for devices.

4.2 System Architecture

In the system, there are several components and packages for working withdifferent devices and integration.

Figure 21: System Architecture

Tracking package is responsible for analyzing the recorded images from cam-

26

era to extract the information of the world coordinate. Base on this coordi-nate, Graphic engine can add virtual objects at the correct position in thereal world scene.Haptic is responsible for managing various types of Haptic devices such asNovint Falcon, Phantom Omni or SPIDAR. Haptic devices are the main in-put method of the system. Using haptic devices, user can get the tactilefeedback while interacting with the system. Calibration is a module that re-duces the inaccuracy in graphical coordinates between devices. Each devicehas its own inaccuracy and this become worse when we apply transforma-tions to get all devices working together.H3D is a module to load AR applications. It manages all devices in thesystem and loads AR scene from X3D files.This project is a part of the VHAR Framework of Magic Vision Lab. It con-tributes 2 modules to the VHAR framework: Paint and Deform. The mainfocus of these modules is to provide to user the capability of painting anddeforming a 3d object.

Figure 22: Component Diagram

According to the requirement that these modules could be reused as much aspossible, we have implemented these component as extension nodes of X3D.Other developers can reuse these modules by declaring these extension nodesin X3D file and then run the application.Therefore, Painting and deforming modules are parts of X3D package. Thesecomponents inherit from the base class X3DNode of X3D package. This couldhelp other developers can reuse these components naturally as X3D nodes.To reduce the bottleneck of calculating for collision detection, we used anexternal package, OpenCL, to utilze the parallel computing feature. TheOpenCL module will be executed on GPU instead of running on CPU.

27

4.3 System Mechanism

This section will explain the mechanism of the system. First, we introducethe data structure used for models. Second, we discuss the mechanism oftwo modules. Also, we will provide algorithms for each module.

4.3.1 3D Model

In computer graphics, a model is usually created by a set of polygons andeach polygon is determined by a number of vertices. In 3D environment, avertex is defined by a set of three values which are the coordinates in 3 axesX, Y and Z of that vertex. Graphical data of model is stored in a specificformat which depends on the creator program. There are several well-knownformats for storing 3D objects information such as Wavefront (*.obj), 3DStudio (*.3ds), X3D (*.x3d)... Based on a specific format, the program usesa suitable method as well as data structure to retrieve graphical data torender 3d objects in the file.In this project, we used IndexedTriangleSet data structure, which is definedin X3D, as the mechanism for storing and loading 3D objects. Basically,IndexedTriangleSet contains two arrays. The first array is a list of 3-tuple offloat values which stores coordinates of a vertex in X, Y, Z axes. The indexof each vertex is the position of it in the array starting from 1. The secondarray is a list of integer values which stores 3-tuple of indices of vertices.

Figure 23: A decomposition of a 3D cube model

As the above figure, a cube is created by 12 triangles and 8 vertices. TheIndexedTriangleSet structure will be defined by two arrays like following:

- Vertex Array = (-1 , 1 , -1, -1, 1, 1, 1, 1, 1, 1, 1, -1, -1 , -1 , -1, -1, -1,1, 1, -1, 1, 1, -1, -1)

- Vertex Index Array = (1, 5 ,6, 1, 2, 6, 1, 5, 4, 4, 5, 8, 1, 2, 4, 2, 3, 4, 3,4, 7, 4, 7, 8, 2, 3, 7, 2, 6, 7, 5, 6, 8, 6, 7, 8)

28

To store models information in the program, we defined a data structure with3 classes: Model3D, Triangle, and Point3D. Model has an array of Triangles,and a Triangle has 3 Point3Ds. The UML class diagram is as the belowmodel:

Figure 24: Class diagram of data structure for storing 3d model

4.3.2 Painting

In 1974, Catmull introduced texture mapping method for enhancing 3D ob-jects appearance such as adding color or adding texture on the surface ofobject (Catmull E., 1974). A polygon has to register on a specific region onthe texture in order to apply texture on its surface. Each vertex of a polygonis registered to a 2D coordinate on the texture. The other pixels are theninterpolated to get an appropriate 2D coordinate based on the registered co-ordinates of vertices. The color of a pixel is determined by the color of itslocation in the texture.

Figure 25: Texture mapping

According to this technique, we implemented the painting effect by changingthe color of the texture at the polygons that intersect with the stylus.

Figure 26: Painting effect

29

When using painting, user will choose a color to paint on the object. Oncethe stylus touches the surface of the object, we calculated the intersectionarea to determine which polygons are needed to have color changing. Then,we change the color of these polygons to the selected color.

4.3.3 Deformation

User uses stylus to push on the surface of the virtual object to deform it.The deformation shape is determined by the intersection area between thepen and the object. Not only the stylus tip can distort the object but alsothe body of the pen can make the distortion. If the body collides with theobject, the area of collision also be distorted. The figure below shows variousposes of the stylus interacts with the object.

Figure 27: The stylus deforms the objects surface

As described in the figure, the deformation effect can be achieved by updat-ing the position of vertices that collide with the stylus.

Figure 28: Update position for a vertex

30

The figure above describes how to update position of a vertex. The newposition of a vertex depends on the moving direction of the stylus. The al-gorithm for updating vertex position is as following:

Declaration: Let Pi is an element of the array of vertices P[];Vector = (A, d) indicates a vector starting from point A and having directiond;V is a vector represents for Moving direction of the stylus;

AlgorithmFor each Pi in P[]if (Pi collides with Stylus), thenCreate vector u = (Pi, V); with V is moving directionFind P’ = u ∩ Stylus;Set P = P’;endif

4.3.4 Subdivision

As the pen deforms the virtual object, positions of vertices are updated.While updating, some polygons become bigger as shown in the figure below

Figure 29: Big polygons created when updating vertices positions

This makes deformation become rough, not smooth. To overcome this, thebig polygons have to be broken into smaller polygons to make the deformationsmoother. We applied a subdivision method for the triangle by selectingmidpoints of every edge of that triangle. Then we connect these midpointsto break the triangle into 4 smaller triangles.A threshold will be used for determining if a triangle is required to be subdi-vided or not. First, we do the checking for subdivision for every polygon ofthe model. The model is stored using Indexed Triangle Set, so every polygonof the model is a triangle and the model is can be considered as an array of

31

Figure 30: Big polygons created when updating vertices positions

triangles. Second, we check the lengths of 3 edges of each triangle with athreshold which set to be half size of the stylus radius. Third, if there existone of the edge length exceed the threshold, we subdivide that triangle into4 smaller triangle. Last, we remove the current triangle from the array andadd 4 smaller triangles into the array. The pseudo code for subdivision is asfollowing:

Declaration: Let Ti is an element of the array T[];δ is a threshold;ei is the length of ith edge;vi is the position of ith vertex;Triangle = (A, B, C) is a triangle created by 3 vertices A, B, and C

AlgorithmFor every Ti in T[]:if (Ti .e1 > δ) OR (Ti .e2 > δ) OR (Ti .e3 > δ) then:m1 = (Ti.v1 + Ti.v2) / 2;m2 = (Ti.v1 + Ti.v3) / 2;m3 = (Ti.v2 + Ti.v3) / 2;Create new triangles:ST1 = (Ti.v1, m1, m2); ST2 = (m1, Ti.v1, m3);ST3 = (m1, m3, m2); ST4 = (m2, m3, Ti.v1);Remove Ti from T[];Add ST1, ST2, ST3, ST4 in T[];endif

Next section will discuss the OpenCL framework which is responsible forparallel computing.

4.3.5 OpenCL

The objective of this section is to provide a background of parallel comput-ing. Then we proved that this system can apply the enhancement of parallel

32

computing to do the collision detection checking. Next, we discuss the ca-pability of CPU and GPU in parallel computing. Last, we introduced theOpenCL which is a framework for parallel computing in various platforms.

4.3.5.1 Parallel ComputingTraditionally, applications in computer have been programmed in serial

computation. When executing, a program is a serial of instructions that pro-cessed one by one by the CPU. After an instruction is finished, the next willbe processed (Blaise and Livermore, 2007).In contrast, parallel computing enables to process multiple instructions at atime. By breaking a program into many independent parts and run thesepart simultaneously, the computing time is reduced significantly. Accordingto the Amdahls law, the maximum speed-up for using parallel programminginstead of serial is: S = 1/α, with α is a fraction of running time for non-parallelized parts (Amdahl G., 1967).This can be easily seen via an example of a puzzle game. A puzzle gamecould be solved 10 times faster if there are 10 people playing it simultane-ously instead of they solve it sequentially.

Figure 31: People solve puzzle in sequence

Figure 32: People solve puzzle in parallel

However, not all programs can be parallelized. The possibility of paralleldepends on the dependency of elements in the algorithm. Bernstein (1966)

33

gave the conditions to check the dependency of 2 processes Pi and Pj asfollowing:These processes are independent if:

Ii and Oi are the input and output of process Pi;Ij and Oj are the input and output of process Pj.

• Ii ∩ Oj = Ø

• Ij ∩ Oi = Ø

• Oi ∩ Oj = Ø (Ø stands for empty set)

Next, we assess the possibility of using parallel computing for collision detec-tion as collision detection is the main bottleneck of this program. Basically,manipulation happens when phantom stylus interacts with the model. Thismeans collision detection should be checked for every polygon of the styluspen and all polygons of the model.Apply the dependency conditions given, let Pi, Pj are the processes thatcheck collision between polygon I and j of the object with the stylus. Theinput is polygons of the model. As described in the 3D Model section, anobject is an array of independent polygons, and each polygon has its ownvertices which dependent to another polygons vertices. The output of eachprocess is a Boolean value that indicates collision happens or not for a par-ticular polygon of the model.

Figure 33: Class diagram of Model3D structure

4.3.5.2 CPU vs. GPUIn comparison with Central Processing Unit (CPU), Graphic Processing

Unit (GPU) is designed for highly intensive parallel computing (NVIDIA).The figure below shows the architectures of CPU and GPU. Then greenblocks are Arithmetic Logic Unit (ALU). All arithmetic and logical oper-ations are processed by ALU. The yellow blocks are Control or Controller

34

blocks. Controllers are responsible for data driven in between Cache andALU or between ALU and ALU. Lastly, the caches, which are the interme-diate memory for CPU, are represented in orange blocks.In the GPUs structure, there are more transistors than in CPU. This enablesGPU simultaneously executes more processes than CPU and thus, betterthan CPU in parallel computing.

Figure 34: Structure of CPU and GPU(5)

Here in the graphs below were some benchmarks that compared the capabil-ity of CPU and GPU. The first one was floating point operations per secondbenchmark and the second one was memory bandwidth that can be processedin a second. In the graphs, green lines describe the capability of GPU andblue lines are for CPUs. As we can see, the capability of GPU in both graphsis much higher than CPU.

5Khronos 2010, Open Programing for the CUDA Architecture, digital image, Khronos,viewed 4 April 2011, http://developer.download.nvidia.com/OpenCL/NVIDIA-OpenCL-JumpStart-Guide.pdf

35

Figure 35: Benchmarks of CPU and GPU(6)

However, GPU is not always better than CPU. GPU is not good for algo-rithms that require data sharing between threads as it does not have con-troller in between ALUs and caches. For example, there is no straight methodto do recursion in GPU.

4.3.5.3 OpenCLIn 2008, the Khronos Computing Group introduced a framework for paral-

lel computing called Open Computing Language (OpenCL). OpenCL enablesdevelopers to create hundreds of threads that work simultaneously to solve aproblem (NVIDIA). OpenCL provides a capability to do parallel computingon both CPU and GPU as user’s requirement. Basically, OpenCL syntaxand language are similar to C++ language. A program written in OpenCLis not a stand-alone program. It is a module that needed to be loaded toa main C++ program (host program) at runtime called kernel. After finishrunning, kernel will return values to be read back at host program. Here isan example OpenCL code to calculate sum of two matrices:

6Khronos 2010, Open Programing for the CUDA Architecture, digital image, Khronos,viewed 4 April 2011, http://developer.download.nvidia.com/OpenCL/NVIDIA-OpenCL-JumpStart-Guide.pdf

36

OpenCL

When the function is executed, hundreds of threads will be created. Eachthread has a ThreadID that can be retrieved by the built-in command get global id(0).Each element of c matrix will be calculated in an individual thread. There-fore, if there are 10 threads running simultaneously then the running timecould be 10 times faster.

37

5 IMPLEMENTATION

This chapter goes in the detail of the implementation of the system: algo-rithms, mathematical explanations.

5.1 Manipulation Algorithm’s Flowchart

Figure 36: Manipulation Algorithm’s Flowchart

The algorithm runs on two computational platforms: CPU and GPU andcan be stated as following steps:

• Step 1: Load a 3D model to a data structure

• Step 2: Get the Pen’s position (or the Stylus)

• Step 3: Start doing Collision Detection by invoking OpenCL module.The input for OpenCL module is the pen’s position and polygons ofthe model.

• Step 4: OpenCL creates threads to check and update positions ofvertices

• Step 5: CPU process retrieves data from OpenCL including: Updatedvertices, Polygons that need to be subdivided.

• Step 6: CPU re-draw the model based on the retrieved data usingOpenGL.

• Step 7: Go to step 2.

38

5.2 Moving Direction

The updated positions of vertices when user deforms the object are deter-mined by moving direction of the stylus. Moving directions are different fromvarious positions on the stylus. Each position on the stylus is affected by amoving direction. To calculate moving direction for a specific point on thepen, we calculated the transition of the stylus poses in 2 adjacent frames.

Figure 37: Moving direction at different points on the stylus

The pose of the stylus is determined by the core of the pen. The coreis an abstract line that connects centers of Top and Bottom of the stylus.The body of the stylus is a cylinder. A cylinder is a set of circles havingsame radius and their centers are in the core line. To identify the movingdirection that affects a point, we have to identify the circle which containsthat point. The moving direction equals the shift of circles center positionin two adjacent frames.

As in the figure showed, the moving direction affects M is vector H1H2.Here the calculation for identifying H1H2, symbols used according to thefigure above:

Step 1 : Identify H.

Dot product of 2 vectors : −→a .−→b = |−→a ||

−→b | cos(−→a ,

−→b )

cos(−→a ,−→b ) is cosin of angle created by −→a and

−→b

39

⇒

cosα =

−−→BM .

−→BT

|−−→TM ||

−−→BT |

cos β =−−→TM .−→TB

|−−→TM ||

−→TB|

⇒cosαcos β =

−−→BM.

−→BT

|−−→BM ||

−→BT |

. |−−→TM ||

−→BT |

−−→TM.

−→TB

=−−→BM.

−→BT−−→

TM.−→TB× TM

BM (1)

According to the figure, we have{cosα = BH

BM

cos β = THTM

⇒ cosαcosβ = BH

BM×TMTH (2)

(1) and (2) give

BHTH=

−−→BM.

−→BT−−→

TM.−→TB

= k (3)

According to parameter function of BT , and H ∈ BT

⇒ H(xH , yH , zH) = B(xB, yB, zB)+BHBT .−→BT

However BT = BH +HT

⇒BHBT =

BHBH+HT =

1BH+THBH

=1

1+THBH

=1

1+1k

=kk+1

Step 2 : Calculate−−−−→H1H2 :

We have

{H1(xH1 , yH1 , zH1) = B1(xB1 , yB1 , zB1) + k

k+1.−−→B1T1

H2(xH2 , yH2 , zH2) = B2(xB2 , yB2 , zB2) + kk+1

.−−→B2T2

−−−→H1H2 = H2(xH2 , yH2 , zH2)−H1(xH1 , yH1 , zH1)

=−−−→B1B2 + k

k+1.(−−→B2T2 −

−−−→B1T1)

40

5.3 Intersection and Updating Vertex position

The stylus of haptic device is overlaid by a virtual pen. The shape of thevirtual pen is a combination of cylinder and half of a sphere. The body shapeof the pen is a cylinder and the tip is half-sphere.

Each time checking for collision of the pen with a vertex, it is required tocheck for both the body and the tip of the pen with the vertex. Therefore,we need 2 times checking for each vertex. We applied an angle pre-checkmethod to determine exactly which part of the pen could collide with thevertex. The pre-check method was based on the angle created by the corevector of the pen and the vector from tip’s center to the vertex. The corevector of the pen is an abstract vector from bottom of the pen to the tip ’scenter. If the created angle is greater than 90 degree then we apply checkingfor cylinder otherwise, we check for the sphere.

The mathematical explanation of the pre-check method is as following:

−→V1 ×

−→V2 = |

−→V1||−→V2| sin(

−→V1,−→V2), therefore :

⇒ sin(−→V1,−→V2) =

−→V1×−→V2

|−→V1||−→V2|

(−→V1,−→V2) = arcsin(

−→V1×−→V2

|−→V1||−→V2|

)

41

5.3.1 Checking for Sphere

We need to identify the intersection point I of the pen tip and the movingvector at vertex A.

I : intersection ⇒ |−→OI|2 = R2(1)

moving direction −→v (x, y, z)

I ∈−→AI

xI = xA + t.vx

yI = yA + t.vy

zI = ZA + t.vz

−→OI =

x = xo − xi = xo − xA − t.vxy = yo − yI = yo − yA − t.vyz = zo − zI = zo − zA − t.vz

⇒−→OI =

−→AO − t.−→v

⇒ |−→OI|2 = |

−→AO|2 − 2.t.−→v .

−→AO + t2|−→v |2 (2)

(1)&(2)⇒ |−→AO|2 − 2.t.−→v .

−→AO + t2.|−→v |2 −R2 = 0

Let

A = |−→v |2

B = −2.−→AO.−→v

C = |−→AO|2 −R2

and 4 = B2 − 4AC

if 4 < 0 : no intersection

if 4 = 0 : no intersection

if 4 > 0 : t =

{−b+√4

2A−b−√4

2A

As−→AI has same direction with −→v , we choose t > 0

42

After the intersection I was identified, we update A to the position of I.

5.3.2 Checking for Cylinder

We need to identify the intersection point I of the pen body and the movingvector at vertex M.

M(xM , yM , zM) Moving direction−→v (a, b, c)

⇒

xI = xM + t.a

yI = yM + t.b

zI = zM + t.c

We have I ∈ Cylinder ⇒ d(I,Bo) = R

⇒|−→BI×

−→OI|

|−−→BO|

= R

⇒ |−→BI ×

−→OI| = R.|

−−→BO|

⇒−→BI ×

−→OI =(

−−→MB − t.−→v )× (

−−→MI − t.−→v )

=−−→MB×

−−→MO− t.−→v ×

−−→MO+

−−→MB× (−t.−→v )+ t2.−→v ×−→v

However −→v ×−→v = 0 because −→v ×−→v = |−→v ||−→v | sin(−→v ,−→v )−→nwhere −→n is normal vector and sin(−→v ,−→v ) = sin(0) = 0

⇒−→BI ×

−→OI =

−−→MB ×

−−→MO − t.−→v × (

−−→MO −

−−→MB)

43

Moreover I ∈ Cylinder : d(I, OB) = R

⇒ |−→BI ×

−→OI|2 = R2.|

−−→BO|2

⇒ |−−→MB ×

−−→MO − t.−→v × (

−−→MO −

−−→MB)|2 −R2.—

−−→BO|2 = 0

Let

{−−→MB ×

−−→MO =

−→P

−→v × (−−→MO −

−−→MB) =

−→Q

⇒ |−→P − t.

−→Q |2 −R2.|

−−→BO|2 = 0

⇒ t2Q2 − 2.t.−→Q.−→P + |

−→P |2 −R2.|

−−→BO|2 = 0

Let

A = Q2

B = −2.t.−→Q.−→P

C = |P |2 −R2.|−−→BO|2

We have

4 = B2 − 4AC

if 4 < 0 : no intersection

if 4 = 0 : no intersection

if 4 > 0 : t =

{−b+√4

2A−b−√4

2A

Because−−→MI has the same direction with −→v so we choose t > 0

After the intersection I was identified, we update M to the position of I.

44

6 CONCLUSIONS

6.1 Results

This system was implemented on a desktop computer with the hardware con-figuration:

We tested the system for two functions, painting and deforming, with several3D models. In both functions, we used two criteria to evaluate the capacityof the system. The first criterion is the number of polygons of the testedmodel. The second is number of frames that system could render in a sec-ond, called Frame per Second (FPS).We rated the capability of the system in each test into three levels: Good,Average, and Bad. A good level will be acquired if FPS is bigger or equalto 24. If FPS is in the range of 12 to 23, than the result will be an Averagelevel. Bad level is for FPS less than 12.The models used for test in painting function were: Bunny, Cow, Car, and agenerated plane. The generated plane is generated by computer. The num-ber of polygons of the generated plane is set up by user for testing purpose.

Figure 38: Benchmark for Paint Function

45

According to the received results, the paint function was running quite goodwith almost models. There was only one Average level for the 50.000 polygonplane.For the deformation function, we also use the same models that tested forpainting function. However, the generated plane we used for deformationonly has 20 thousand polygons. Below is the benchmark of deformation test.

Figure 39: Benchmark for Deformation function

In this test, we have got 2 Good, 1 Average, and 1 Bad result. We found thatthere was a reason causing bottleneck. The problem was we had to calculatenormal vector for every polygon each time the model was updated. We didnot implement an effective method that just only updates normal vectorsfor updated polygons. We have tried to remove the calculation for normalvectors and the result was much better.

6.2 Summary

In summary, we have successfully solved almost the requirements of thisproject. That is we could prove the possibility of building an applicationthat can directly manipulate 3D object on Augmented Reality environment.Using the system, we can benefit from the advantage of AR application suchas ease of use, tactile feedback, and realistic 3D interactions. We can paintor deform a 3D model directly.

46

Figure 40: Paint on a virtual shoe in AR, shown in TEDx Adelaide 2010

www.youtube.com/watch?v = U2Y E2lHULWA

In 2010, we have successfully integrated the paint function to AR environ-ment. This system had shown in TEDx Adelaide and had received a lot ofgood feedback from audiences.

Figure 41: Deform a 3D model of a bunny

However, we did not have enough time to integrate the deformation part toX3D package as an X3D node. The deformation part was just finished atthe implementation for an OpenGL application to prove the theory of defor-mation in this project. We still continue to work on the integration of thisfunction to AR environment.

6.3 Future Work

There are some problems of the system that need to be solved. First, thedeformation function is not yet integrated to AR environment. Second, thesubdivision is not good enough as it still not good for some particular issuessuch. For example, the introduced subdivision method is not effective fornarrow triangles. The last thing we hope to upgrade is to apply some bettermethod for checking collision as the complexity is now still O(n2). Hopefully,if we apply some advance methods such as Nearest Neighborhood, or BSPTree, we can significantly reduce the complexity of the current algorithm.

47

7 REFERENCES

An, W & Cai, Z 2010, ’Collision Detection Technology Based on BoundingBox of Virtual Reality’, paper presented at the International Conference onE-Product E-Service and E-Entertainment (ICEEE 2010), 7-9 Nov. 2010.

Azuma, RT 1997, ’A survey of Augmented Reality’, Presence: Teleopera-tors and Virtual Environments, vol. 6, no. 4, pp. 355-385.

Backman, N 2010, ’Collision Dectection of Triangle Meshes using GPU’,Master Thesis in Computing Science, UMEA University.

Bernstein, AJ 1966, ’Analysis of Programs for Parallel Processing’, Elec-tronic Computers, IEEE Transactions on, vol. EC-15, no. 5, pp. 757-763.

Cutler, B, Dorsey, J, McMillan, L, Muller, M & Jagnow, R 2002, A pro-cedural approach to authoring solid models, ACM, San Antonio, Texas, pp.302-311.

Eck, U and Sandor, C, TINT: Towards a Pure Python Augmented Real-ity Framework, Proceedings of the Third Workshop on Software Engineeringand Architectures for Realtime Interactive Systems, pages 41–46, Waltham,MA, USA, March, 2010.

Gibson, SFF 1996, ’3D ChainMail: a Fast Algorithm for Deforming Vol-umetric Objects’, December 1996.

Hamam, A, Georganas, ND & El Saddik, A 2010, ’Effect of haptics on theQuality of Experience’, paper presented at the 2010 IEEE International Sym-posium on Haptic Audio-Visual Environments and Games (HAVE), 16-17Oct. 2010.

Hanjun, J, Zhiliang, L, Tianzhen, W & Yanxia, W 2009, ’The Researchof Collision Detection Algorithm Based on Spatial Subdivision’, paper pre-sented at the 2009 International Conference on E-Product E-Service andE-Entertainment (ICEEE), 22-24 Jan, 2009.

Horan, B, Najdovski, Z, Nahavandi, S & Tunstel, E 2008, ’3D Virtual HapticCone for Intuitive Vehicle Motion Control’, paper presented at the Sympo-sium on 3D User Interfaces (3DUI), 8-9 March 2008.

48

Kato, H, Billinghurst, M, Poupyrev, I, Imamoto, K & Tachibana, K 2000,’Virtual object manipulation on a table-top AR environment’, Proceedings ofthe IEEE and ACM on International Symposium Augmented Reality (ISAR2000), 2000.

Khronos 2008, OpenCL Programming Guide for the CUDA Architecture,NVIDIA.

Kim, S, Hasegawa, S, Koike, Y & Sato, M 2002, ’Tension based 7-DOFforce feedback device: SPIDAR-G’, Proceedings of IEEE on Virtual Reality,2002.

Lamousin, HJ & Waggenspack, NN, Jr. 1994, ’NURBS-based free-form de-formations’, Computer Graphics and Applications, IEEE, vol. 14, no. 6, pp.59-65.

Liu, P, Georganas, ND & Roth, G 2006, ’Handling Rapid Interference Detec-tion of Progressive Meshes Using Active Bounding Trees’, journal of graphics,gpu, and game tools, vol. 11, no. 4, pp. 17-37.

Muller, M, Dorsey, J, McMillan, L, Jagnow, R & Cutler, B 2002, ’StableReal-Time Deformations’, paper presented at the Proceedings of ACM SIG-GRAPH Symposium on Computer Animation.

McDonnell, KT & Qin, H 2008, ’PB-FFD: A Point-Based Technique forFree-Form Deformation ’, Journal of Graphics, GPU, & Game Tools, vol.12, no. 3, pp. 25-41.

Muller, M, Heidelberger, B, Teschner, M & Gross, M 2005, Meshless de-formations based on shape matching, ACM, Los Angeles, California, pp.471-478.

Noble, RA & Clapworthy, GJ 1999, ’Direct manipulation of surfaces usingNURBS-based free-form deformations’, paper presented at the Proceedingsof the IEEE International Conference on Information Visualization, 1999.

Page, F & Guibault, F 2003, ’Collision detection algorithm for NURBS sur-faces in interactive applications’, paper presented at the IEEE CanadianConference on Electrical and Computer Engineering (IEEE CCECE), 4-7May 2003.

49

Rolland, JP, Baillot, Y & Goon, AA 2001, ’A survey of tracking technol-ogy for Virtual Environments’.

Sandor, C & Klinker, G 2005, ’A rapid prototyping software infrastructure foruser interfaces in ubiquitous augmented reality’, Personal Ubiquitous Com-put., vol. 9, no. 3, pp. 169-185.

Sandor, C, Uchiyama, TKS & Yamamoto, H 2007, Exploring Visuo-HapticMixed Reality, Human Machine Perception Laboratory, Tokyo.

Smith, A, Kitamura, Y, Takemura, H & Kishino, F 1995, ’A simple andefficient method for accurate collision detection among deformable polyhe-dral objects in arbitrary motion’, paper presented at the Proceedings of theVirtual Reality Annual International Symposium, 11-15 Mar 1995.

Terzopoulos, D, Platt, J, Barr, A & Fleischer, K 1987, Elastically deformablemodels, ACM, pp. 205-214.

Uchiyama, S, Takemoto, K, Satoh, K, Yamamoto, H & Tamura, H 2002,’MR Platform: a basic body on which mixed reality applications are built’,paper presented at the Proceedings of the International Symposium on Mixedand Augmented Reality (ISMAR), 2002.

Vallino, J & Brown, C 1999, ’Haptics in augmented reality’, paper presentedat the IEEE International Conference on Multimedia Computing and Sys-tems, Jul 1999.

Wen, Q 2004, ’A prototype of video see-through mixed reality interactivesystem’, In Proceedings of the 2nd International Symposium on 3D DataProcessing, Visualization and Transmission (3DPVT 2004), 6-9 Sept, 2004.

Xinyu, Z & Kim, YJ 2007, ’Interactive Collision Detection for DeformableModels Using Streaming AABBs’, Visualization and Computer Graphics,IEEE Transactions on, vol. 13, no. 2, pp. 318-329.

Yanlin, L, Ping, G, Hasegawa, S & Sato, M 2004, ’An interactive molec-ular visualization system for education in immersive multi-projection virtualenvironment’, Proceedings of the Third International Conference on Imageand Graphics, 18-20 Dec. 2004.

50