7
IEEE Wireless Communications • June 2013 136 1536-1284/13/$25.00 © 2013 IEEE A CCEPTED FROM O PEN C ALL INTRODUCTION Nowadays, we are witnessing a great deal of attention in the use of VE based class of appli- cations, such as virtual walkthrough [1], multi- player online games [2] just to mention a few. Most of these applications are based on the client-server architecture where the entire VE should be stored in advance on the client. How- ever, given that the size of the VE application is growing exponentially, the pre-installation is no more a feasible approach. To overcome this issue, most of the applications opted for 3D streaming that consists of gradually downloading and rendering 3D content such as meshes and textures to permit the interaction of the user with its virtual world without a full download or a pre-installation [3]. Streaming 3D meshes is not easy to realize given that it is network band- width consuming. Therefore, providing an effi- cient streaming solution would open the door to many advanced applications. Depending on the nature of the application and its requirements, several 3D streaming techniques [4–6] have been designed and can be found in the literature. With the advances of wireless communication and mobile computing, several applications [1, 4, 7] started to take advantage of 3D streaming over thin mobile devices such as PDA, iPhones, and head mounted devices (HMD). However, this was not an easy task since thin mobile devices can not render large and complex 3D scenes, and have limited 3D resources and capa- bilities. Researchers worked extensively to solve this issue and several approaches [1, 4] have been presented. Most of the adopted approaches are based on the remote visualization [1] that is a client-server based architecture and that uses an image-based technique [8] instead of sending a 3D mesh. However, some obvious drawbacks for remote visualization include server bottle- neck, significant delay, lack of scalability, and communication overhead that occurs during the manipulation of the 3D object and where extra images need to be streamed. Obviously, all of the previously mentioned issues affect the streaming quality of the 3D data and conse- quently the user’s exeperience during the multi- media session. To overcome the beforehand mentioned issues, researchers opted for the use of Peer-to-peer (P2P) overlay network as an architecture for the 3D streaming based applica- tions. Although, the use of P2P-based architec- ture addressed the issues encountered by the client-server based architecure, it brought up additional challenges that need to be faced. In this article, we present a taxonomy of the 3D streaming techniques discussed in the litera- ture while focusing on how these techniques have been modified to facilitate 3D streaming over thin mobile devices, while taking into account the number of challenges that need to be faced. The remainder of this article is as fol- lows. The next section explains the impacts of the network impairments on the VE applica- tions, and the challenges and issues that arise in HAIFA RAJA MAAMAR, SMR RESEARCH LABORATORY AND DIVA RESEARCH CENTER AZZEDINE BOUKERCHE, PARADISE RESEARCH LABORATORY AND DIVA RESEARCH CENTER EMIL PETRIU, SMR RESEARCH LABORATORY AND DIVA RESEARCH CENTER ABSTRACT We are witnessing a significant growth in applications using thin mobile devices, such as social networks, virtual walkthrough, massively multiplayer online gaming (MMOG), and aug- mented reality (AR), just to mention a few. Vir- tual environments (VE) based class of applications have recently attracted a large num- ber of users. Applications that applied the con- ventional client-server architecture require the VE to be stored on the client, which is not very feasible due to the client’s memory constraints. To address this issue, 3D streaming techniques have been designed and are widely used nowa- days. However, several challenges exist and affect the user’s Quality of Experience (QoE). By all means, these challenges need to be resolved before the 3D streaming technology over thin mobile devices becomes a commodity. In this paper, we provide a survey on the existing 3D streaming techniques by classifying them based on the nature of the application, and we centralize our attention on methods applied to adapt 3D streaming techniques to the changes in the wireless network conditions. Therefore, we discuss the challenges that the 3D streaming techniques face from a network point of view, as well as the approaches and solutions proposed. S TREAMING 3D M ESHES OVER T HIN M OBILE D EVICES

Streaming 3D meshes over thin mobile devices

  • Upload
    e

  • View
    243

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013136 1536-1284/13/$25.00 © 2013 IEEE

AC C E P T E D F R O M OP E N CALL

INTRODUCTIONNowadays, we are witnessing a great deal ofattention in the use of VE based class of appli-cations, such as virtual walkthrough [1], multi-player online games [2] just to mention a few.Most of these applications are based on theclient-server architecture where the entire VEshould be stored in advance on the client. How-ever, given that the size of the VE application isgrowing exponentially, the pre-installation is nomore a feasible approach. To overcome thisissue, most of the applications opted for 3Dstreaming that consists of gradually downloadingand rendering 3D content such as meshes andtextures to permit the interaction of the userwith its virtual world without a full download ora pre-installation [3]. Streaming 3D meshes is

not easy to realize given that it is network band-width consuming. Therefore, providing an effi-cient streaming solution would open the door tomany advanced applications. Depending on thenature of the application and its requirements,several 3D streaming techniques [4–6] have beendesigned and can be found in the literature.

With the advances of wireless communicationand mobile computing, several applications [1, 4,7] started to take advantage of 3D streamingover thin mobile devices such as PDA, iPhones,and head mounted devices (HMD). However,this was not an easy task since thin mobiledevices can not render large and complex 3Dscenes, and have limited 3D resources and capa-bilities. Researchers worked extensively to solvethis issue and several approaches [1, 4] havebeen presented. Most of the adopted approachesare based on the remote visualization [1] that isa client-server based architecture and that usesan image-based technique [8] instead of sendinga 3D mesh. However, some obvious drawbacksfor remote visualization include server bottle-neck, significant delay, lack of scalability, andcommunication overhead that occurs during themanipulation of the 3D object and where extraimages need to be streamed. Obviously, all ofthe previously mentioned issues affect thestreaming quality of the 3D data and conse-quently the user’s exeperience during the multi-media session. To overcome the beforehandmentioned issues, researchers opted for the useof Peer-to-peer (P2P) overlay network as anarchitecture for the 3D streaming based applica-tions. Although, the use of P2P-based architec-ture addressed the issues encountered by theclient-server based architecure, it brought upadditional challenges that need to be faced.

In this article, we present a taxonomy of the3D streaming techniques discussed in the litera-ture while focusing on how these techniqueshave been modified to facilitate 3D streamingover thin mobile devices, while taking intoaccount the number of challenges that need tobe faced. The remainder of this article is as fol-lows. The next section explains the impacts ofthe network impairments on the VE applica-tions, and the challenges and issues that arise in

HAIFA RAJA MAAMAR, SMR RESEARCH LABORATORY AND DIVA RESEARCH CENTERAZZEDINE BOUKERCHE, PARADISE RESEARCH LABORATORY AND DIVA RESEARCH CENTER

EMIL PETRIU, SMR RESEARCH LABORATORY AND DIVA RESEARCH CENTER

ABSTRACTWe are witnessing a significant growth in

applications using thin mobile devices, such associal networks, virtual walkthrough, massivelymultiplayer online gaming (MMOG), and aug-mented reality (AR), just to mention a few. Vir-tual environments (VE) based class ofapplications have recently attracted a large num-ber of users. Applications that applied the con-ventional client-server architecture require theVE to be stored on the client, which is not veryfeasible due to the client’s memory constraints.To address this issue, 3D streaming techniqueshave been designed and are widely used nowa-days. However, several challenges exist andaffect the user’s Quality of Experience (QoE).By all means, these challenges need to beresolved before the 3D streaming technologyover thin mobile devices becomes a commodity.In this paper, we provide a survey on the existing3D streaming techniques by classifying thembased on the nature of the application, and wecentralize our attention on methods applied toadapt 3D streaming techniques to the changes inthe wireless network conditions. Therefore, wediscuss the challenges that the 3D streamingtechniques face from a network point of view, aswell as the approaches and solutions proposed.

STREAMING 3D MESHES OVERTHIN MOBILE DEVICES

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 136

Page 2: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013 137

supporting 3D streaming over thin mobiledevices. The section following that illustrates thefour categories of 3D streaming techniques:geometry replication, progressive meshes, impos-tors, and image based model. Then we discussthe vulnerabilities of 3D streaming over thinmobile devices and present techniques to elimi-nate these vulnerabilities. Finally, we present ourconclusion and give future directions for 3Dstreaming techniques over thin mobile devices.

PROBLEM STATEMENT AND CHALLENGES

Several 3D streaming techniques have been pro-posed in the literature [9, 10]. However, severalnetwork challenges need to be addressed before3D streaming technology becomes a commodity.Moreover, the use of 3D streaming across wirelessnetworks and/or mobile ad hoc networks(MANET) induces extra challenges that need alsoto be taken into consideration. Therefore, a given3D streaming solution is considered efficient if ittakes into account the following challenges:

Mobile device limitations: due to low pro-cessing power, limited storage capacity, limitedgraphics’ hardware and graphics’ accelerator,just to mention a few. These limitations make itvery diificult for thin mobile devices to renderand process large and complex 3D scenes.

Wireless network bandwidth limitation:where the wireless medium access is constantlyexposed to background noise, multipath fading,shadowing and interferences, which makes thebandwidth variant over the time, and leads tolink disruptions and thereby resulting in higherror rates and packet loss.

Density: has a significant impact on the quali-ty of the streaming where a high node densitymay lead to a significant communication over-head, resulting into packet collision, while a lownode density induces a decreased signal strengthresulting into a high packet drop.

Nodes mobility: in dynamic networks createsa new challenge for 3D streaming based systems.This is mainly due to the dynamic route changes.

Streaming performance: such as latency, net-work congestion, long data acquisition times,and invalid requests just to mention a few, mayhave a devastating effect on 3D streaming overmobile networks.

Scalability: considered as one of the mostimportant requirements in networked virtualenvironments (NVE) applications, may be hard

to obtain on client-server based models asopposed to P2P networks, and therefore need tobe carefully investigated.

Given all of the beforehand mentioned issues,streaming a 3D mesh over a thin mobile deviceis considered extremely challenging. In the fol-lowing, we shall present the existing 3D stream-ing techniques and show how these techniquescan be used to avoid and address the challenges.

3D STREAMING TECHNIQUES

So far a significant body of work has been dedi-cated to the implementation of an effective 3Dstreaming technique. Based on the nature of theapplication, several solutions have been designedand implemented. All of the proposed tech-niques can be grouped into four categories:Geometry replication, progressive meshes,impostors, and Image Based model. Table 1illustrates a comparison of these techniques stat-ing their advantages and disadvantages.

GEOMETRY REPLICATIONThe geometry replication technique [10] consistsof having a copy of the 3D models geometrystored on the client and rendered by its localhardware. The copy of the 3D model can eitherbe acquired from the client’s hard disks or opti-cal drivers as done by computer games, or down-loaded from the server. However, this techniquepresents major disadvantages when dealing withwireless networks using thin mobile devices.Firstly, the size of the VE is extremely large andtakes a long time to download. Secondly, due tothe limited mobile resources and capabilities,i.e., low processing power, limited storage capac-ity, limited graphics hardware and graphicsaccelerator, mobile clients are unable to renderthe received 3D models in the same quality asrendered by the server, and consequently unableto process large and complex 3D scenes. There-fore, this technique does not suit well applica-tions running on thin mobile devices.

PROGRESSIVE MESHESProgressive meshes (PM) [9] technique consistsof having virtual objects rendered and transmit-ted progressively. With the PM technique, theclient is able to visualize the 3D model in lowerquality and then progressively refine it until itobtains its original quality. To do so, first, PMgenerates and streams a basic shape low polygon

Table 1. Comparison of 3D streaming techniques.

Approach Technique Advantages Limitations

Geometry replication 3D model stored on the client Scalability Not applied on thin mobile devices

Progressive mesh 3D objects rendered andtransmitted progressively

3D model visualized at lowerquality and refined Long times for mesh generation

IBR Reference images, panoramas Light, applied on thin mobiledevices Display problems; pixel loss, distortion

Impostors 3D model rendered to an imageand texture mapped to a shape

Faster rendering times andlower bandwidth usage

Inadequate for objects close to usersNew impostor for ever user rotation

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 137

Page 3: Streaming 3D meshes over thin mobile devices

version of the original 3D model. The simplestrepresentation of the 3D model is called basemesh. Then details to generate more complexrepresentations, called refinement layers, are gen-erated and progressively streamed [5, 11]. ThePM technique is based on the edge collapse andedge split mechanisms. The former aims atreducing the resolution of the object, whereasthe latter aims at applying the inverse process,i.e. increasing the resolution of the object [11].Figure 1 illustrates the different representationsof a teapot.

Several derivatives [9, 11–13] of the PM tech-nique exist. Isenburg and Lindstrom [9] proposedthe streaming meshes technique where a mesh isstored into a fixed size buffer, and triangles andvertices are either added or removed from themesh in order to reconstruct it. Kircher and Gar-land [11] proposed a multi-resolution representa-tion for deforming objects with a high qualityapproximation. The multilevel mesh proposed bythe authors aims at having iterative edge contrac-tion, and uses less space since it stores progres-sive representation (mesh connectivity at eachlevel) instead of the entire hierarchy. However,edge contraction makes the mapping false sincevertices forming the children can move. Fang andTian [14] implemented a mesh simplificationbased on the triangle contraction simplification.Pajarola and Rossignac [12] proposed the com-pressed progressive meshes (CPM) approachaiming at improving the PM technique by focus-ing on removing the overhead and latency engen-dered by PM. For this matter, CPM uses theimplant sprays technique to refine the mesh byassembling the vertex splits into batches. In con-sequence, CPM occupies 50 percent less storagethan PM model. Modified Compressed Progres-sive Meshes (MCPM) technique [13] improvesCPM by including a decision module that selectsthe most suitable transport protocol for eachgeometric sub-layer taking into account the net-work bandwidth and the loss ratio.

PM techniques have several advantages, how-ever, on the downside, it is considered complexand not efficient in terms of compression ratioand times for mesh generation. Moreover, pro-gressive refinement induces a considerable over-head specially when the entire mesh has to bedownloaded. And finally, PM techniques is not

the best solution when dealing with thin mobiledevices, given that large and complex 3D modelsare difficult to stream due to the mobile device’slimitation in terms of memory.

IMPOSTORSIn the impostor technique [10], the server, basedon the client’s orientation, is responsible for ren-dering the complex 3D model, transforming itinto an image and sending it to the client; whilethe client is responsible for texture mapping theobtained image to a simple shape such as a planeor a box [10]. Figure 2 illustrates an impostor.This technique has faster rendering times andlower bandwidth usage, since impostors are sim-ple shapes with smaller sizes compared to theactual 3D model, which makes them easy to ren-der and transmit. However, this technique is notconsidered a good choice for objects close to theuser, since the latter can notice the lack of imagedepth. Moreover, this technique induces signifi-cant overhead given that new impostors have tobe streamed every time the user moves andchanges its orientation.

IMAGE BASED RENDERINGRendering 3D models [1] requires powerfuldevices with high resources and capabilitiesgiven the large size of the 3D models. This sizeissue makes the 3D models ineligible for lowbandwidth networks, and impossible to use inlight-weight mobile devices. The image-basedmodel, also called image based rendering (IBR),overcomes the beforehand mentioned difficultiesby representing the VE using images instead ofgeometry. In IBR, the server is responsible forthe rendering tasks since it is equipped withpowerful rendering hardware, while the clientsare simply responsible for displaying the receivedimages. This concept is called Remote Render-ing and is considered the best choice when deal-ing with thin mobile devices [1]. IBR techniquescan be applied using two methods: Referenceimages [8] and Panoramas techniques [1].

Reference Images — In order to avoid displaydelays and long waiting times, several IBR tech-niques are applied on the clients by using a setof reference images [8] to construct intermediateviews, new images or portion of them. Thesetechniques are based on the plenoptic functionthat states that the world can be perceived as aset of light rays filling the space that can be seenby eyes or cameras. The plenotpic function rep-resents real world views, where the user can beat any position, look anywhere, at any time.However, this is not an easy task when dealingwith VEs. As a matter of fact, the user experi-ence is more restrained, depending on the appli-cation and the available hardware. Therefore,subfunctions of the 7D plenoptic function can beused to restrict the viewing space.

Panoramas — Nowadays, most of the 3D IBR rep-resentations use panoramas that give to the userthe illusion of seeing new views. Panoramas con-sist of a series of panoramic images obtained bycapturing different images of the visible views inall directions and projecting them on a 3D shapesuch as spheres, cylinders, cubes just to mention

IEEE Wireless Communications • June 2013138

Figure 2. Impostors technique.

Figure 1. Level of details for a teapot.

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 138

Page 4: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013 139

a few. Figure 3 illustrates a panaorama projectedon a cylinder. Spherical panorama allows theuser to look anywhere from his position, since itcovers 360 degrees horizontally vertically. How-ever, this type of panoramas produce image dis-tortions and involve multiple rendering passeswhich makes them inefficient. Cubic panorama,also covers 360 degrees horizontally and vertical-ly, and projects the result on a cube’s six facesthat has each a 90*90 field of view. The cubicalprojected images represent usually the imagesource that several spherical panorama viewersare based on. Cubic panoramas generation ismore efficient compared to spherical panoramasand is supported by most of today’s standardgraphics cards [1]. Google Street View andQuickTime VR (QTVR) are two applicationsbased on panoramas.

CHALLENGES OF 3D STREAMING OVERMOBILE DEVICES

Geometry based-3D streaming requires a largeamount of data and high-resolution textures tobe transmitted in real time over the network.This obviously creates network bottlenecks,overuses the network resources and affects thestreaming performance. When applied in wire-less networks using thin mobile devices, 3Dstreaming becomes more difficult to achieve dueto the devices’ limited capabilities. Therefore,3D streaming over thin mobile devices is not aneasy task due to the number of challenges thatneed to be faced. These challenges can be relat-ed to the mobile devices limited capabilities, thenetwork impairments such as network bandwidthover-utilization, latency, network congestion,scalability, just to mention a few, or related tothe architecture of the application. In the lastsection, we reviewed the main existing 3Dstreaming techniques. In this section, we shallpresent the vulnerabilities to which 3D stream-ing applications are exposed to and explain howthe 3D streaming techniques eliminated thesevulnerabilities. Table 2 presents an overview ofthe challenges that the 3D streaming face andthe possible solutions that can be applied toaddress these issues. In what follows, we will dis-cuss some of the challenges in more details.

DEVICE LIMITATIONOne of the most important challenges of stream-ing 3D data over thin mobile devices consists inthe device limitations in terms of resources andcapabilities, i.e., low processing power, limitedstorage capacity, limited graphics’ hardware andgraphics’ accelerator that make it very difficultfor mobile devices to render and process largeand complex 3D scenes. The device limitationcan limit the user’s capabilities to produce orrender large 3D scenes. In order to overcomethe mobile device limitations, several researchworks opted for using remote visualization,where the server is responsible for rendering therelevant scene, while the mobile client is onlyresponsible for displaying the received images.There are two types of remote visualization:• The server side where a remote server hosts the

3D environment, and is responsible for ren-

dering the complex 3D scenes and sendingonly images to the mobile user according toits view

• The hybrid side where the server is responsiblefor rendering some parts of the 3D environ-ment such as high resolution of the 3D envi-ronment, while the client is responsible forrendering the remaining parts such as theresidual error images and low resolutiongeometryA second technique, Remote Line Rendering

[4], also have been implemented to overcomethe mobile device limitation. The authors intend-ed to balance the load between the server andthe mobile clients. For this reason, they pro-posed to execute most of the tasks on the serverand to render only the 2D line primitives sent bythe server on the client. The client reducestherefore the server’s rendering load. As for theserver, it will be responsible only for maintaininga global view of the VE and converting the 3Dmodels into 2D line primitives using 2D windowcoordinates [4]. This remote line rendering tech-nique has several advantages, however, it experi-ences limitations due to using 2D lines, speciallywhen dealing with fully shaded and colored

Figure 3. Panoramic images.

Table 2. 3D streaming challenges and solutions.

Challenges Solution

Device limitations 1. Remote visualization2. Remote line rendering

Bandwidth overutilization 1. IM techniques (AOI)2. Zoning techniques

Network congestion 1. Rate control mechanisms2. P2P-based streaming

Latency Dead reckoning techniques

Scalability P2P-based 3D streaming

Wireless bandwidth limitation 1. View-dependent simplification2. LOD technique

Node mobility Efficient routing protocols

3D data retrieval Supplying partner strategies

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 139

Page 5: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013140

objects.

BANDWIDTH OVERUTILIZATIONSeveral papers [1, 2] have pointed out the chal-lenge of 3D streaming characterized by band-width over-utilization. Most of the proposedsolutions, suggested to use Interest Mamange-ment (IM) [2], or zoning techniques that aim atfiltering the data to be processed in order toreduce the network load and avoid the band-width over-utilization. In the following we shalldescribe the different techniques that have beenimplemented to reduce the network load or tocope with the bandwidth limitation. Table 3illustrates a comparison of these techniques stat-ing their advantages and limitations.

Interest Management: One of the most usedtechniques to address the VE network load issueis the IM that aims at minimizing the transmis-sion of unnecessary messages. Indeed, broadcast-ing updates to all users within the VE, knowingthat each user has a limited visibility, is not effi-cient. When 3D applications are used on mobiledevices, this issue, i.e., the transmission of unnec-essary messages, has a higher impact and may bea major cause of streaming performance degra-dation. In fact, all of the mobile nodes will bebroadcasting every update message received,which leads to the flooding problem, networkcongestion and to the bandwidth over-utilization.To overcome this problem, several approachesagreed upon designing an IM technique to filterthe messages and to send only updates that areof interest to each user [2]. Most of the IM tech-niques [2] are based on the user’s perception(i.e., what the user can see) that illustrates itsArea of interest (AOI) also known as region ofinterest (ROI). Most of the time, the user’s AOIis represented by a fixed size circle having theuser at the center. Only objects that are withinthe user’s AOI are processed and streamed tothe client, reducing, therefore, a significantamount of processing time and network load.

Zoning Techniques: Another set of filteringmechanisms that have been discussed in the lit-erature [3] make use of the zoning techniques.When dealing with 3D environments and inorder to restrict the interest of a user, proximityis manifestly the criteria to take into account,since entities close to the clients will have nearly

the same view of the VE, and will likely sharethe same interest. Therefore, filtering is accom-plished by dividing the VE into regions. Eachuser will only receive data and updates relatedto objects within its region, or adjacent regions.Different shapes were adopted during the zon-ing, such as honeycomb regions [2], or voronoidiagram [3]. However, one of the significantdrawbacks of the zoning-based techniques is thatthe amount of data filtered is highly dependenton the size of the zone.

LATENCYLatency may be the result of different networkissues such as a network congestion, invalidrequests, and long times to acquire data. All ofthese issues greatly affect the user’s QoE giventhat the quality of the streamed 3D data is con-siderably degraded. Network congestion forexample can take place if many requests have tobe served at the same time. This case happensspecially in a client-server architecture, wherethe server becomes a bottleneck since it is theonly supplier for clients’ requests. To addressthis issue, two sets of idea have been proposed: • A rate control mechanism [1] that allows the

sender to adapt its transfer rate based on theinformation returned from the requesters

• P2P streaming, where systems are designedbased on the P2P architecture instead of theclient server architecture [7] Long times to acquire data affects the stream-

ing performance of the system since the clientsexperience delay to display the received 3D data.The user in this case does not have a satisfactoryexperience during the multimedia session. Inorder to eliminate this delay, dead reckoning [1]is the technique that have been applied. Deadreckoning is based on prediction, where usingthe behavioral model of a given object, and itsprevious states, the user can predict the actualremote object’s state. Dead reckoning parame-ters include the object’s speed, its position anddirection. Several studies used the dead reckon-ing especially in predicting where the user isprobably going and which images need to bestreamed [1]. By determining the movementdirection of the user, the server can stream inadvance the images that have a high probabilityto be displayed in the close future decreasing

Table 3. Comparison of 3D streaming approaches for network load reduction.

Approach Technique Advantage Limitations

IM Only relevant is sent Minimization of the amount ofdata processed Highly dependent on the zone size

View-dependentsimplification

Adaptation of the 3D modelresolution to the factors’ change Network load reduction 3D model stored in memory Increase of

the 3D model size

LOD Different representation of eachobject

Taking into account the user’sbandwidth Data redundancy

Dead reckoning Object’s state prediction Latency and updates reduction Sudden changes in the user’s movement

Remote linerendering

2D line primitive rendered locallyon the client

Network load reduction Loadbalancing between C/S

Inefficient with fully shaded coloredobjects

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 140

Page 6: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013 141

therefore the time to acquire the 3D data andenhancing the user’s QoE.

Invalid requests take place when a userrequests data from another peer that does nothave the relevant data causing therefore aninvalid request and consequently a delay toacquire data. To address this issue, supplyingpartner schemes have been proposed and aimedat addressing the invalid requests and latencyissues [7].

WIRELESS MEDIUM LIMITATIONStreaming 3D meshes over wireless networks isvery challenging due to the wireless medium lim-itation that is constantly exposed to backgroundnoise, multipath fading, shadowing and interfer-ences. This makes the bandwidth variant overthe time, and leads to link disruptions resultingin high error rates and packet loss, which obvi-ously degrades the quality of the 3D datastreamed. In order to overcome this issue,researchers opted for increasing the wavelengthof the signal carrier. However, this solutionreduces the amount of available bandwidth forthe nodes and is considered inadequate giventhat it does not satisfy the stringent require-ments of the 3D streaming based applications.Other researchers opted to reduce the size ofthe 3D mesh by using techniques such as LOD[10], PM [5], and view dependent simplification[6], just to mention a few.

View Dependent Simplification: The AOI tech-niques are usually tightly used with another set oftechniques called multi-resolution or view depen-dent simplification [15]. The view dependent polyg-onal simplification methods [5] are based on a datastructure englobing different representations of agiven 3D object while changing its resolution. Theresolution of the 3D model and its level of detail,as illustrated in [15], are adapted to the change ofseveral factors such as the user’s location, and illu-mination, just to mention a few. The object’s reso-lution is determined based on the user’s depth ofsight and on the distance that separates the userfrom the object. An object is considered visible toa given user, if it is within the user’s AOI or, if theuser’s scope and the object’s scope overlap. Despitethe fact that this technique reduces the bandwidthover-utilization, the view dependent simplificationtechnique does increase the size of the 3D model.Moreover, it requires the entire 3D model to bestored in the main memory. To address the lastissue, especially when dealing with mobile clients,external memory, also called out-of-core, was pro-posed to store the VE on the disk. However, themobile device performance is seriously affectedand degraded since disk access speed is very slowwhen compared to CPU speed.

Levels of Detail: Most of the 3D streamingsolutions assign different importance levels tothe objects located within the user’s field of viewand their display resolutions are therefore adapt-ed to the amount of perceived details. LODtechniques [10] take advantage of this paradigm,and provides different representation for eachobject, by reducing or increasing the object’scomplexity and changing the number of polygonsand/or the texture resolution among otherparameters. Although, the LOD technique takesinto account the user’s perception and available

bandwidth, on the downside, it produces, foreach 3D object, multiple models at different res-olutions to be sent over the network. This unfor-tunately may lead to data redundancy andcomputation overhead.

Another significant challenge related to thewireless medium is the time varying link thatmay be instable due to the node mobility. Thismay greatly affect the 3D streaming quality dueto an increase in the packet drop which unfortu-nately increases the 3D data acquisition time.Efficient routing protocols are therefore neededin order to establish new routes and ensure areliable delivery of the 3D data.

SCALABILITYIn order to insure the scalability requirement,researchers agreed upon using P2P networks. Aconsiderable number of solutions [3] used P2Pstreaming over wired networks and severalframeworks such as FLoD [3], aiming at provid-ing scalability in NVE, have been implemented.In wireless networks, we notice that existingresearch works focused on video streaming andproposed P2P based protocols for video stream-ing over thin mobile devices [7]. However, onlyvery limited work [7] has applied the P2P based3D streaming over wireless networks.

3D DATA RETRIEVALThe 3D data retrieval consists of finding the rele-vant 3D data and the peer that holds it in aninstructured P2P networks. This issue is especial-ly important for P2P based 3D streaming applica-tions, where there is no entity that holds theentire VE and that is responsible for streamingthe relevant data to interested clients. The sup-plying partner selection strategy consists ofselecting the suitable source that has the relevantdata to stream it to the requesters. In P2P basedapplications, peers collaborate together and par-ticipate to provide the relevant 3D data. There-fore peers can be both requesters and suppliersfor other users. Usually, a suitable supplyingpartner to choose is an adjacent peer; however,this peer may not have enough bandwidth totransfer the relevant data. Moreover, streaming3D data using mobile devices, overuses themobile device’s resources in terms of energy andmemory. Therefore, extra factors such as themobile device’s residual energy, need to be takeninto consideration during the selection of thesupplying partner. MOSAIC [7] is among thesupplying partner protocols designed for P2P 3Dstreaming applications over thin mobile devices.

CONCLUSION

In this article, we studied the 3D streaming tech-niques designed up to today by classifying theminto four categories, namely: the geometry repli-cation, progressive mesh, impostors and imagebased rendering. We then oriented our focustoward the challenges encountered when stream-ing 3D meshes over thin mobile devices. To doso, we identified a number of vulnerabilities towhich 3D streaming based applications areexposed such as: the mobile device limitations,the wireless network impairements, and thearchitecture used, just to mention a few; and we

The 3D data retrievalconsists of finding therelevant 3D data and

the peer that holds it inan instructured P2P net-

works. This issue isespecially important for

P2P based 3D streamingapplications, where

there is no entity thatholds the entire VE and

that is responsible forstreaming the relevant

data to interestedclients.

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 141

Page 7: Streaming 3D meshes over thin mobile devices

IEEE Wireless Communications • June 2013142

presented how techniques have been applied toeliminate the beforehand mentioned vulenrabili-ties. Streaming 3D meshes over thin mobiledevices is still considered very challenging andextra efforts are still needed in order to makethis technology a commodity.

ACKNOWLEDGMENTThis work is partially supported by the CanadaResearch Chairs program, NSERC, the EARResearch Award, ORF/MRI, and theOIT/Ontario Distinguished Research Award.

REFERENCES[1] A. Boukerche and R. W. Pazzi, “Remote Rendering and

Streaming of Progressive Panoramas for MobileDevices,” Proc. 14th Annual ACM Int’l. Conf. Multime-dia, 2006, pp. 691–94.

[2] A. Yu and S. T. Vuong, “MOPAR: A Mobile P2P OverlayArchitecture for Interest Management of MMOG,” Proc.Int’l. Wksp. Network and Operating Systems Supportfor Digital Audio and Video Pages, 2005, pp. 99–104.

[3] S.Y. Hu et al., “FLOD: A Framework for Peer to Peer 3DStreaming,” Proc. IEEE INFOCOM, 2008, pp. 1373–81.

[4] J. Diepstraten and T. Ertl, “Remote Line Rendering forMobile Devices,” Proc. Int’l. Computer Graphics, 2004,pp. 454–61.

[5] H. Hoppe, “Progressive Meshes,” Proc. 23rd AnnualConf. Computer Graphics and Interactive Techniques,1996, pp. 99–108.

[6] J. El-Sana and A. Varshney, “Generalized View-Depen-dent Simplification,” Proc. EURO GRAPHICS ’99, 1999,pp. 83–94.

[7] H. R. Maamar, A. Boukerche, and E. Petriu, “MOSAIC —A Mobile Peer-to-Peer Networks-Based 3D StreamingSupplying Partner Protocol,” IEEE/ACM 14th DS-RT ’10,2010, pp. 61–68.

[8] S. Chen and L. Williams, “View Interpolation for ImageSynthesis,” Proc. 20th Annual Conf. Computer Graphicsand Interactive Techniques, 1993, pp. 279–88.

[9] M. Isenburg and P. Lindstrom, “Streaming Meshes,”IEEE Visualization, Nov. 2005, pp. 231–38.

[10] R. Jarrar, Image-Based Rendering Protocols for RemoteInteractive Walkthroughs on Mobile Devices, Master’sThesis, 2009.

[11] S. Kircher and M. Garland, “Progressive Multiresolu-tion Meshes for Deforming Surfaces,” Proc. 2005 ACMSiggraph/Eurographics Symp. Computer Animation,2005, pp. 191–200.

[12] R. Pajarola and J. Rossignac, “Compressed ProgressiveMeshes,” IEEE Trans. Visualization and ComputerGraphics, vol. 6, no. 1, 2000, pp. 79–93.

[13] H. Li and B. Prabhakaran, “Smart Decision Module forStreaming 3D Meshes over Lossy Networks,” Proc. Int’l.Conf. Distributed Multimedia Systems and Int’l. Wksp.Visual Languages and Computing, 2004, pp. 275–78.

[14] T. Fang and Z. Tian, “Efficient Storage and ProgressiveRendering of Multi-Resolution Mesh,” Advances in Mul-timedia Info. Processing (PCM), vol. 48, 2007, pp.814–21.

[15] W. Cheng, D. Liu, and W. T Ooi, “Peer Assisted ViewDependent Progressive Mesh Streaming,” Proc. 17thACM Int’l. Conf. Multimedia, 2009, pp. 441–50.

BIOGRAPHIESHAIFA RAJA MAAMAR ([email protected]) is aComputer and Electrical Engineering Ph.D. candidate at thePARADISE Research Laboratory and SMR Research Labora-tory at the University of Ottawa, Ontario, Canada. Shereceived her B.Sc. and M.Sc. in Computer Engineering fromthe University of Ottawa in 2006 and 2008, respectively.Her research interests include wireless multimedia, 3Dstreaming, wireless mobile networks, mobile augmentedreality, and distributed simulations.

AZZEDINE BOUKERCHE is a full professor and holds a CanadaResearch Chair position at the University of Ottawa (uOt-tawa). He is a fellow of the Canadian Academy of Engineer-ing and the founding director of the PARADISE ResearchLaboratory, School of Information Technology and Engineer-ing (SITE), Ottawa. Prior to this, he held a faculty position atthe University of North Texas, and he was a senior scientist atthe Simulation Sciences Division, Metron Corp., San Diego. He

was also employed as a faculty member in the School ofComputer Science, McGill University, and taught at the Poly-technic of Montreal. He spent a year at the JPL/NASA-Califor-nia Institute of Technology, where he contributed to a projectcentered about the specification and verification of the soft-ware used to control interplanetary spacecraft operated byJPL/NASA Laboratory. His current research interests includewireless ad hoc and sensor networks, wireless networks,mobile and pervasive computing, wireless multimedia, QoSservice provisioning, performance evaluation and modeling oflarge-scale distributed systems, distributed computing, large-scale distributed interactive simulation, and parallel discrete-event simulation. He has published several research papers inthese areas. He served as a guest editor for the Journal ofParallel and Distributed Computing (special issue for routingfor mobile ad hoc, special issue for wireless communicationand mobile computing, and special issue for mobile ad hocnetworking and computing), ACM/Kluwer Wireless Networks,ACM/Kluwer Mobile Networks Applications, and Journal ofWireless Communication and Mobile Computing. He serves asan Associate Editor of IEEE Transactions on Parallel and Dis-tributed systems, IEEE Transactions on Vehicular Technology,Elsevier Ad Hoc Networks, Wiley International Journal of Wire-less Communication and Mobile Computing, Wiley’s Securityand Communication Network Journal, Elsevier Pervasive andMobile Computing Journal, IEEE Wireless CommunicationMagazine, Elsevier’s Journal of Parallel and Distributed Com-puting, and SCS Transactions on Simulation. He was therecipient of the Best Research Paper Award at IEEE/ACM PADS1997, ACM MobiWac 2006, ICC 2008, ICC 2009 and IWCMC2009, and the recipient of the Third National Award forTelecommunication Software in 1999 for his work on a dis-tributed security systems on mobile phone operations. He hasbeen nominated for the Best Paper Award at the IEEE/ACMPADS 1999 and ACM MSWiM 2001. He is a recipient of anOntario Early Research Excellence Award (previously known asPremier of Ontario Research Excellence Award), Ontario Dis-tinguished Researcher Award, and Glinski Research ExcellenceAward. He is a cofounder of the QShine International Confer-ence on Quality of Service for Wireless/Wired HeterogeneousNetworks (QShine 2004). He served as the general chair forthe Eighth ACM/IEEE Symposium on Modeling, Analysis andSimulation of Wireless and Mobile Systems, and the NinthACM/IEEE Symposium on Distributed Simulation and Real-Time Application (DS-RT), the program chair for the ACMWorkshop on QoS and Security for Wireless and Mobile Net-works, ACM/IFIPS Europar 2002 Conference, IEEE/SCS AnnualSimulation Symposium (ANNS 2002), ACM WWW 2002, IEEEMWCN 2002, IEEE/ACM MASCOTS 2002, IEEE Wireless LocalNetworks WLN ’03–’04; IEEE WMAN ’04–’05, and ACMMSWiM ’98–’99, and a TPC member of numerous IEEE andACM sponsored conferences. He served as the vice generalchair for the Third IEEE Distributed Computing for SensorNetworks (DCOSS) Conference in 2007, as the programcochair for GLOBECOM 2007–2008 Symposium on WirelessAd Hoc and Sensor Networks, and for the 14th IEEE ISCC2009 Symposium on Computer and Commmunication Sym-posium, and as the finance chair for ACM Multimedia 2008.He also serves as a Steering Committee chair for the ACMModeling, Analysis and Simulation for Wireless and MobileSystems Conference, the ACM Symposium on PerformanceEvaluation of Wireless Ad Hoc, Sensor, and Ubiquitous Net-works, and IEEE/ACM DS-RT.

EMIL M. PETRIU [M’86‚ SM’88‚ F’01] received the Dipl.Eng.and Dr. Eng. degrees from the Polytechnic Institute ofTimisoara, Timisoara, Romania, in 1969 and 1978, respec-tively. He is currently a Professor and the UniversityResearch Chair with the School of Information Technologyand Engineering, University of Ottawa, Ottawa, ON, Cana-da. During his career, he has published more than 200technical papers, authored two books, and edited twoother books. He is the holder of two patents. His researchinterests include robot sensing and perception, intelligentsensors, interactive virtual environments, soft computing,and digital integrated circuit testing. He is a Fellow of theCanadian Academy of Engineering and of the EngineeringInstitute of Canada. He is an Associate Editor of the IEEETransactions on Instrumentation and Measurement and amember of the Editorial Board of the IEEE I&M Magazine.He is currently the Chair of TC-15 Virtual Systems and theCochair of TC-28 Instrumentation and Measurement forRobotics and Automation and TC-30 Security and Contra-band Detection of the IEEE Instrumentation and Measure-ment Society. He was a corecipient of the IEEE’s Donald G.Fink Prize Paper Award and the recipient of the IEEE Instru-mentation and Measurement Society Award in 2003.

Extra factors such as themobile device’s residualenergy, need to betaken into considerationduring the selection ofthe supplying partner.MOSAIC is among thesupplying partner proto-cols designed for P2P3D streaming applica-tions over thin mobiledevices.

MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 142