Cube-map Rendering and Streaming of 360o Videos

Research Track Exploration Program, 2016

Team:Vikulp Bansal, Dept. of Electrical Engineering, IIT KanpurGaurav Verma, Dept. of Electrical Engineering, IIT Kanpur

Mentors:Professor Yao Wang, Department of Electrical and Computer Engineering, NYU

Professor Gaurav Sharma, Department of Computer Science and Engineering, IIT KanpurFanyi Duanmu, PhD Candidate, Video Lab, New York University

What is a 360o Video A 360o video is created with a camera system that simultaneously records all 360 degrees of a scene. Viewers can pan and rotate a 360 video's perspective to watch it from different angles.

Equirectangular Representation Cubemap Representation

Conventional Method to Render a 360o Video

● The equirectangular layout of a 360o video is used to render it in a browser● Rendering primarily involves wrapping a 2D texture onto a 3D Sphere

● For any point P on the sphere, let d is the unit vector from P to sphere’s origin. Assuming that the sphere’s poles are aligned with the Y axis, the normalized coordinates on the 2D texture are given as:

Drawback of Conventional Method:● Equirectangular layout inherently involves redundancy● It uses an entire line in the layout to represent a single point on the globe

Advantages of using Cube-map layout:● Cube-map layout drops the redundant information to a significant extent● As a result of which, file size reduces by 25%● Can be used to develop an efficient streaming method for DASH of 360o

videos: View-based adaptive streaming

Equirectangular to Cubemap Conversion

Use Ffmpeg tool for conversions

Apply transform filter of Facebook API

Input Equirectangular Video

Cube-map layout video

This custom video filter uses point projection to transform equirectangular layout to cubemap layout.Simply putting, for each frame - 1. Put a sphere inside a cube.2. Wrap an equirectangular texture of this frame on the

sphere.3. Expand the sphere in every direction until it fits the

cube.Output layout corresponds to the six faces of a cube with the position of every face fixed.

Morphing cube-map texture over sphere

1. UV coordinate mapping is done to convert 2d cube-map texture into a 3d cube geometry.

3d cube is morphed into a sphere using vertex normalization techniques.

Dynamic Adaptive Streaming Over HTTP (DASH)❖ Adaptive Streaming of media content over

internet from HTTP servers to the client. ❖ Media file is converted to one or more segments.❖ DASH contains Media Presentation Description

(MPD) file which describes segment information (timing,URL, media characteristics like video resolution and bit rates).

❖ DASH client parse MPD file to get the relevant information.

❖ Client fetches segments of different alternatives(Bit rate, resolution) according to the available bandwidth.

DASH Content Generation● X264 encodes the input video into H.264 format.● Mp4Box converts raw x264 file into mp4 format and also generates content for DASH.● Mp4Box generates one MPD file which contains all the representations of the video file.● Open Source dash.js streams the multimedia file using DASH.

On Demand View-based Streaming

● α is the horizontal view angle, spanning from 0o to 360o

● is the vertical view angle, spanning from 0o to 180o

● The horizontal plane is quantized into 12 bins (30o/bin)● The vertical plane is quantized into 6 bins (30o/bin)● Hence giving a total of 72 view-ports

● The view-port requests from client, are dynamically monitored and appropriate view-ports are streamed using DASH from the server end

● Efficient, as the portion that is not being demanded by the client, is not streamed

Horizontal Plane Quantization Vertical Plane Quantization

On Demand View-based Streaming● The MPD (Media Presentation Description) is to be updated with viewport specification under

Representation tag:

<Representation id="1" mimeType="video/mp4" codecs="avc1.4d401e" width="640" height="360" frameRate="30" sar="1:1" startWithSAP="1" bandwidth="610368" viewport="1">

● The player at the client end must be updated to support the parsing of viewport field● The view-port in focus is streamed at a higher resolution, compared to the ones around it. Streaming

other viewports (surrounding the one that is being demanded by the client) is important to ensure that the user has a buffer-free experience.

● The possibility of encoding the video so as to add tracking information to detect temporal movement in the video, is being explored. This will help in predicting the viewport that the user is going to demand.

Thank you!