Streaming 3D meshes over thin mobile devices

  • Published on

  • View

  • Download

Embed Size (px)


  • IEEE Wireless Communications June 2013136 1536-1284/13/$25.00 2013 IEEE


    INTRODUCTIONNowadays, we are witnessing a great deal ofattention in the use of VE based class of appli-cations, such as virtual walkthrough [1], multi-player online games [2] just to mention a few.Most of these applications are based on theclient-server architecture where the entire VEshould be stored in advance on the client. How-ever, given that the size of the VE application isgrowing exponentially, the pre-installation is nomore a feasible approach. To overcome thisissue, most of the applications opted for 3Dstreaming that consists of gradually downloadingand rendering 3D content such as meshes andtextures to permit the interaction of the userwith its virtual world without a full download ora pre-installation [3]. Streaming 3D meshes is

    not easy to realize given that it is network band-width consuming. Therefore, providing an effi-cient streaming solution would open the door tomany advanced applications. Depending on thenature of the application and its requirements,several 3D streaming techniques [46] have beendesigned and can be found in the literature.

    With the advances of wireless communicationand mobile computing, several applications [1, 4,7] started to take advantage of 3D streamingover thin mobile devices such as PDA, iPhones,and head mounted devices (HMD). However,this was not an easy task since thin mobiledevices can not render large and complex 3Dscenes, and have limited 3D resources and capa-bilities. Researchers worked extensively to solvethis issue and several approaches [1, 4] havebeen presented. Most of the adopted approachesare based on the remote visualization [1] that isa client-server based architecture and that usesan image-based technique [8] instead of sendinga 3D mesh. However, some obvious drawbacksfor remote visualization include server bottle-neck, significant delay, lack of scalability, andcommunication overhead that occurs during themanipulation of the 3D object and where extraimages need to be streamed. Obviously, all ofthe previously mentioned issues affect thestreaming quality of the 3D data and conse-quently the users exeperience during the multi-media session. To overcome the beforehandmentioned issues, researchers opted for the useof Peer-to-peer (P2P) overlay network as anarchitecture for the 3D streaming based applica-tions. Although, the use of P2P-based architec-ture addressed the issues encountered by theclient-server based architecure, it brought upadditional challenges that need to be faced.

    In this article, we present a taxonomy of the3D streaming techniques discussed in the litera-ture while focusing on how these techniqueshave been modified to facilitate 3D streamingover thin mobile devices, while taking intoaccount the number of challenges that need tobe faced. The remainder of this article is as fol-lows. The next section explains the impacts ofthe network impairments on the VE applica-tions, and the challenges and issues that arise in



    ABSTRACTWe are witnessing a significant growth in

    applications using thin mobile devices, such associal networks, virtual walkthrough, massivelymultiplayer online gaming (MMOG), and aug-mented reality (AR), just to mention a few. Vir-tual environments (VE) based class ofapplications have recently attracted a large num-ber of users. Applications that applied the con-ventional client-server architecture require theVE to be stored on the client, which is not veryfeasible due to the clients memory constraints.To address this issue, 3D streaming techniqueshave been designed and are widely used nowa-days. However, several challenges exist andaffect the users Quality of Experience (QoE).By all means, these challenges need to beresolved before the 3D streaming technologyover thin mobile devices becomes a commodity.In this paper, we provide a survey on the existing3D streaming techniques by classifying thembased on the nature of the application, and wecentralize our attention on methods applied toadapt 3D streaming techniques to the changes inthe wireless network conditions. Therefore, wediscuss the challenges that the 3D streamingtechniques face from a network point of view, aswell as the approaches and solutions proposed.


    MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 136

  • IEEE Wireless Communications June 2013 137

    supporting 3D streaming over thin mobiledevices. The section following that illustrates thefour categories of 3D streaming techniques:geometry replication, progressive meshes, impos-tors, and image based model. Then we discussthe vulnerabilities of 3D streaming over thinmobile devices and present techniques to elimi-nate these vulnerabilities. Finally, we present ourconclusion and give future directions for 3Dstreaming techniques over thin mobile devices.

    PROBLEM STATEMENT AND CHALLENGESSeveral 3D streaming techniques have been pro-posed in the literature [9, 10]. However, severalnetwork challenges need to be addressed before3D streaming technology becomes a commodity.Moreover, the use of 3D streaming across wirelessnetworks and/or mobile ad hoc networks(MANET) induces extra challenges that need alsoto be taken into consideration. Therefore, a given3D streaming solution is considered efficient if ittakes into account the following challenges:

    Mobile device limitations: due to low pro-cessing power, limited storage capacity, limitedgraphics hardware and graphics accelerator,just to mention a few. These limitations make itvery diificult for thin mobile devices to renderand process large and complex 3D scenes.

    Wireless network bandwidth limitation:where the wireless medium access is constantlyexposed to background noise, multipath fading,shadowing and interferences, which makes thebandwidth variant over the time, and leads tolink disruptions and thereby resulting in higherror rates and packet loss.

    Density: has a significant impact on the quali-ty of the streaming where a high node densitymay lead to a significant communication over-head, resulting into packet collision, while a lownode density induces a decreased signal strengthresulting into a high packet drop.

    Nodes mobility: in dynamic networks createsa new challenge for 3D streaming based systems.This is mainly due to the dynamic route changes.

    Streaming performance: such as latency, net-work congestion, long data acquisition times,and invalid requests just to mention a few, mayhave a devastating effect on 3D streaming overmobile networks.

    Scalability: considered as one of the mostimportant requirements in networked virtualenvironments (NVE) applications, may be hard

    to obtain on client-server based models asopposed to P2P networks, and therefore need tobe carefully investigated.

    Given all of the beforehand mentioned issues,streaming a 3D mesh over a thin mobile deviceis considered extremely challenging. In the fol-lowing, we shall present the existing 3D stream-ing techniques and show how these techniquescan be used to avoid and address the challenges.

    3D STREAMING TECHNIQUESSo far a significant body of work has been dedi-cated to the implementation of an effective 3Dstreaming technique. Based on the nature of theapplication, several solutions have been designedand implemented. All of the proposed tech-niques can be grouped into four categories:Geometry replication, progressive meshes,impostors, and Image Based model. Table 1illustrates a comparison of these techniques stat-ing their advantages and disadvantages.

    GEOMETRY REPLICATIONThe geometry replication technique [10] consistsof having a copy of the 3D models geometrystored on the client and rendered by its localhardware. The copy of the 3D model can eitherbe acquired from the clients hard disks or opti-cal drivers as done by computer games, or down-loaded from the server. However, this techniquepresents major disadvantages when dealing withwireless networks using thin mobile devices.Firstly, the size of the VE is extremely large andtakes a long time to download. Secondly, due tothe limited mobile resources and capabilities,i.e., low processing power, limited storage capac-ity, limited graphics hardware and graphicsaccelerator, mobile clients are unable to renderthe received 3D models in the same quality asrendered by the server, and consequently unableto process large and complex 3D scenes. There-fore, this technique does not suit well applica-tions running on thin mobile devices.

    PROGRESSIVE MESHESProgressive meshes (PM) [9] technique consistsof having virtual objects rendered and transmit-ted progressively. With the PM technique, theclient is able to visualize the 3D model in lowerquality and then progressively refine it until itobtains its original quality. To do so, first, PMgenerates and streams a basic shape low polygon

    Table 1. Comparison of 3D streaming techniques.

    Approach Technique Advantages Limitations

    Geometry replication 3D model stored on the client Scalability Not applied on thin mobile devices

    Progressive mesh 3D objects rendered andtransmitted progressively3D model visualized at lowerquality and refined Long times for mesh generation

    IBR Reference images, panoramas Light, applied on thin mobiledevices Display problems; pixel loss, distortion

    Impostors 3D model rendered to an imageand texture mapped to a shapeFaster rendering times andlower bandwidth usage

    Inadequate for objects close to usersNew impostor for ever user rotation

    MAAMAR LAYOUT_Layout 1 6/21/13 11:58 AM Page 137

  • version of the original 3D model. The simplestrepresentation of the 3D model is called basemesh. Then details to generate more complexrepresentations, called refinement layers, are gen-erated and progressively streamed [5, 11]. ThePM technique is based on the edge collapse andedge split mechanisms. The former aims atreducing the resolution of the object, whereasthe latter aims at applying the inverse process,i.e. increasing the resolution of the object [11].Figure 1 illustrates the different representationsof a teapot.

    Several derivatives [9, 1113] of the PM tech-nique exist. Isenburg and Lindstrom [9] proposedthe streaming meshes technique where a mesh isstored into a fixed size buffer, and triangles andvertices are either added or removed from themesh in order to reconstruct it. Kircher and Gar-land [11] proposed a multi-resolution representa-tion for deforming objects with a high qualityapproximation. The multilevel mesh proposed bythe authors aims at having iterative edge contrac-tion, and uses less space since it stores progres-sive representation (mesh connectivity at eachlevel) instead of the entire hierarchy. However,edge contraction makes the mapping false sincevertices forming the children can move. Fang andTian [14] implemented a mesh simplificationbased on the triangle contraction simplification.Pajarola and Rossignac [12] proposed the com-pressed progressive meshes (CPM) approachaiming at improving the PM technique by focus-ing on removing the overhead and latency engen-dered by PM. For this matter, CPM uses theimplant sprays technique to refine the mesh byassembling the vertex splits into batches. In con-sequence, CPM occupies 50 percent less storagethan PM model. Modified Compressed Progres-sive Meshes (MCPM) technique [13] improvesCPM by including a decision module that selectsthe most suitable transport protocol for eachgeometric sub-layer taking into account the net-work bandwidth and the loss ratio.

    PM techniques have several advantages, how-ever, on the downside, it is considered complexand not efficient in terms of compression ratioand times for mesh generation. Moreover, pro-gressive refinement induces a considerable over-head specially when the entire mesh has to bedownloaded. And finally, PM techniques is not

    the best solution when dealing with thin mobiledevices, given that large and complex 3D modelsare difficult to stream due to the mobile deviceslimitation in terms of memory.

    IMPOSTORSIn the impostor technique [10], the server, basedon the clients orientation, is responsible for ren-dering the complex 3D model, transforming itinto an image and sending it to the client; whilethe client is responsible for texture mapping theobtained image to a simple shape such as a planeor a box [10]. Figure 2 illustrates an impostor.This technique has faster rendering times andlower bandwidth usage, since impostors are sim-ple shapes with smaller sizes compared to theactual 3D model, which makes them easy to ren-der and transmit. However, this technique is notconsidered a good choice for objects close to theuser, since the latter can notice the lack of imagedepth. Moreover, this technique induces signifi-cant overhead given that new impostors have tobe streamed every time the user moves andchanges its orientation.

    IMAGE BASED RENDERINGRendering 3D models [1] requires powerfuldevices with high resources and capabilitiesgiven the large size of the 3D models. This sizeissue makes the 3D models ineligible for lowbandwidth networks, and impossible to use inlight-weight mobile devices. The image-basedmodel, also called image based rendering (IBR),overcomes the beforehand mentioned difficultiesby representing the VE using images instead ofgeometry. In IBR, the server is responsible forthe rendering tasks since it is equipped withpowerful rendering hardware, while the clientsare simply responsible for displaying the receivedimages. This concept is called Remote Render-ing and is considered the best choice when deal-ing with thin mobile devices [1]. IBR techniquescan be applied using two methods: Referenceimages [8] and Panoramas techniques [1].

    Reference Images In order to avoid displaydelays and long waiting times, several IBR tech-niques are applied on the clients by using a setof reference images [8] to construct intermediateviews, new images or portion of them. Thesetechniques are based on the plenoptic functionthat states that the world can be perceived as aset of light rays filling the space that can be seenby eyes or cameras. The plenotpic function rep-resents real world views, where the user can beat any position, look anywhere, at any time.However, this is not an easy task when dealingwith VEs. As a matter of fact, the user experi-ence is more restrained, depending on the appli-cation and the available hardware. Therefore,subfunctions of the 7D plenoptic function can beused to restrict the viewing space.

    Panoramas Nowadays, most of the 3D IBR rep-resentations use panoramas that give to the userthe illusion of seeing new views. Panoramas con-sist of a series of panoramic images obtained bycapturing different images of the visible views inall directions and projecting them on a 3D shapesuch as spheres, cylinders, cubes just to mention

    IEEE Wireless Communica...


View more >