Development of immersive real- time applications Miroslav Andel [email protected]

Development of immersive real-time applications

Miroslav [email protected]

Miroslav Andel - Development of immersive real-time applications

2

• Development of immersive real-time applications– Online/direct rendering– Hardware– Programming

• Video production– Offline rendering– Hybrid solutions

Lectures

2014-09-03


3

• Hardware– Understanding the technology– Clusters and GPUs– Warping, color matching & blending

• Immersion in a virtual environment (VE)• Software design

– Desktop application vs. immersive cluster application– Scalability & configuration of VEs– Frameworks– Scene graph

Outline

2014-09-03


4

Hardware

2014-09-03


5

• 6x Digital Projection Titan• 1920 x 1200 @ 120 Hz• Frame doubling

Projectors

2014-09-03


6

Image Generators (IGs)

2014-09-03

– Workstation high-end graphics cards• Frame sync• External house sync• Genlock (video signal sync)• Swaplock (swap buffer sync)• 3D sync (for 3D glasses)

– Supports active stereoscopy using OpenGL– Nvidia Quadro or AMD/ATI FirePro– No gaming cards


7

• GPU is pretty much the same but different driver software and firmware

• Gaming cards are optimized for speed and DirectX– Hi frame-rate in games, highly optimized

• Workstation cards are optimized for precision, OpenGL & OpenCL– Designed for 3D modeling, CAD, video editing and computation– Higher floating point and sub-pixel precision– Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD

FirePro 9100)• Ray tracing on the GPU

• Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance)

Gaming vs. Workstation GPU

2014-09-03

Cluster of IGs

• Often consists of workstations with the same specification

• Master – Slave• In our case the master has an

additional surround sound card (live encoding to Dolby Digital or DTS)


8

Clusters(at the visualization center)

• Dome– Alpha (primary real-time cluster used for shows)– Beta (secondary development cluster)– Delta (movie and presentations playback cluster)– Gamma (single node eyefinity machine)

• VR-Arena– Theta– Videowall (just a single node with 6 outputs)

used for movie and presentations playback


9


11

• What is white?

Color/gammut matching

2014-09-03


15

• Digital blending – good bright levels– Not a linear gradient (gamma)– Done after color and brightness matching

Blending

2014-09-03


16

• Optical blending for better dark levels– Projected black is not black, still projected light

Blending

2014-09-03


17

• Optical blending for better dark levels– Projected black is not black, still projected light

Blending

2014-09-03


18

• Blending in the dome– Blue = Projection borders– Red = Blend zones

Blending

2014-09-03


19

• Correct the rendered frame to match the geometry of the screen, lens and projector’s position

• Correct overlaps between adjacent channels• Spatial 2D transform

Warping/geometric correction

2014-09-03


20

• Processing unit with input & outputs cards (FPGA) with minimal latency

Warping and blending hardware

2014-09-03


21

• No latency but uses a bit of the GPU

Warping and blending using OpenGL

2014-09-03


22

• Manual warping and blending calibration takes many hours– Few reference points in the dome– Moving mesh points by hand causes linearity problems

• Semi-automated calibration• Automated camera-based systems

– Current systems don’t handle optical blends

Calibration process

2014-09-03


23

Hardware (end)

2014-09-03


24

• To give the audience the impression that they have stepped into a synthetic world

Immersion

2014-09-03


25

• To give the audience the impression that they have stepped into a synthetic world

• Enhance immersion by:– Large field of view– High resolution– Stereoscopy (can also reduce immersion)– Surround sound– Smooth navigation (based on a physical model)– Realistic rendering (shadows etc., to improve depth cue)

Immersion

2014-09-03


26

• Stereoscopy in the dome?– The paradox– Field of view wearing 3D glasses– Use stereoscopy wisely

Immersion

2014-09-03


28

Software design

2014-09-03


29

• Several projectors/screens– A lot of pixels– Alignment, blending & warping– Color & Brightness matching

• Several computers (cluster)• Synchronization

– Frame– Buffers swap– Data– Video signal (genlock)

• Multi-threading

The difference(from normal desktop application development)

2014-09-03


30

• How does it work?

Stereoscopy

2014-09-03


31

Stereoscopy

2014-09-03


32

Stereoscopy

2014-09-03


33

Stereoscopy

2014-09-03


34

• Co-location– Camera position = eye position– Projection plane = real physical screen

Stereoscopy

2014-09-03


35

• Don’t change eye separation, change the scale of your scene

• ~15 meters limit

Stereoscopy

2014-09-03


36

2014-09-03


37

2014-09-03


38

2014-09-03


39

• Object in focus in negative parallax

Stereoscopy

2014-09-03


40

Stereoscopy

2014-09-03

• Object in focus in positive parallax


41

• Problems with a fixed head setup (no head tracking)

Stereoscopy

2014-09-03


42

• One application can run in several virtual environments without the need of changing the code– In the dome– In the VR-Arena– In the VR-Lab– At home using your computer– At the bus using a laptop

• If the platform is the same then there is no need for recompiling the application (just copy & paste)

Scalability

2014-09-03


43

The application uses a configuration file that describes the hardware in terms like:• Number of IGs• Number of displays• Number of viewports• Physical location of displays/projection planes• Physical location of the user• Eye separation of the user• Type if Stereoscopy• Etc…

Scalability

2014-09-03

44

• “3d drawing”

Configuration

2014-09-03

45

Configuration

2014-09-03

Immersive applicationsOften uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV)

Monoscopic projection

46

Configuration

2014-09-03

Stereoscopic projection

Configuration47

Configuration48

All users except the target user will perceive the rendered object incorrectly.

Configuration49

All users except the target user will perceive the rendered object incorrectly.

One solution would be using HMDs or a multi-view screen.


50

• In a desktop application a user navigates by moving the camera

Navigation

2014-09-03


51


• In a scalable application the number of cameras/frustums is not known

Navigation

2014-09-03


52


• In a scalable application the number of cameras/frustums is not known

• Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction

Navigation

2014-09-03


53

• Will help you with– Cluster setup & synchronization– Frustums & viewports– Input– Tracking

Frameworks(the good stuff)

2014-09-03


54

Frameworks

2014-09-03

• VR Juggler

• CAVElib (commercial)• Equalizer


55

• Massive and have a lot of dependencies• Big initial effort to get started• Often in legacy OpenGL• Takes usually several days to get started

– Getting all dependencies– Compile all dependencies– Compile the whole framework– Read the documentation

Frameworks(the bad stuff)

2014-09-03


56

SGCTSimple Graphics Cluster Toolkit

2014-09-03


57

• Static library based on– GLFW– GLEW– GLM– Freetype– TinyThread++– TinyXML– PNG– Z– VRPN– TurboJPEG


2014-09-03


58

• Developed here and is free for everybody to use• Designed to get started immediately

(rapid development)• Cross-platform C++• Modern and Legacy OpenGL support (OGL 2.2 – 4+)• Single static library (no dll-hell)• Using callbacks rather than

inheritance/polymorphism– Easy porting application from GLUT, GLFW or similar


2014-09-03


59

• Wiki at: c-student.itn.liu.se– Documentation– Tutorials– Configuration files– Downloads (installers)– Support

• Forum (on it’s way)


2014-09-03


60

• Has an engine which handles– Initiation of OpenGL– Rendering– Synchronization– Network communication– Configuration handling– Input

• The user can bind callbacks (function pointers) to the engine to customize the functionality

SGCT – How it works

2014-09-03

61

2014-09-03


62

• Set only the callbacks you need• A minimal cluster application contains:

– Draw callback: Render your stuff– Pre Sync callback: generate the data to be shared– Encode callback: Send the data to the slave(s)– Decode callback: Receive the data from the master

• Other callbacks to handle input devices and external GUIs

SGCT – How it works

2014-09-03

63

• Frame-, swapbuffer- and application data-synchronization

• 3D/Stereoscopic rendering– Active/Quadbuffer stereo– Side-by-side stereo– Top-bottom stereo– DLP/Checkerboard stereo– Line interleaved stereo– Anaglyph stereo

SGCT - Features

2014-09-03 Miroslav Andel - Development of immersive real-time applications


64

• Fisheye rendering• PNG texture management• GLSL shader management• Freetype font management• Error messaging across the cluster• External TCP control interface• Tracking support• Joystick, gamepad, Kinect, steering wheel etc. input• Antialiasing• 3D screenshots in maximum texture resolution• Overlays• Statistics and performance graphs

SGCT - Features

2014-09-03


65

• XML configuration• All nodes are using the same configuration and

can automatically determine if they are a slave or a master

SGCT - Configuration

2014-09-03

66<?xml version="1.0" ?><Cluster masterAddress="127.0.0.1">

<Node ip="127.0.0.1" port="20401"><Window fullscreen="false">

<Size x="640" y="360" /><Viewport>

<Pos x="0.0" y="0.0" /><Size x="1.0" y="1.0" /><Viewplane>

<Pos x="-1.778" y="-1.0" z="0.0" /><Pos x="-1.778" y="1.0" z="0.0" /><Pos x="1.778" y="1.0" z="0.0" />

</Viewplane></Viewport>

</Window></Node><User eyeSeparation="0.065">

<Pos x="0.0" y="0.0" z="4.0" /></User>

</Cluster>


67

• Normal use: app.exe –config “config.xml”

• Running cluster application locally as master:app.exe –config “cluster_config.xml” –local 0

• Running cluster application locally as slave:app.exe –config “cluster_config.xml” –local 0 --slave

SGCT arguments

2014-09-03


68

• Command prompt/terminal• Create a Batch file (.bat)• Create a Shortcut• From IDE

Running SGCT

2014-09-03


69

• The application folder needs to be accessible from all IGs/nodes1. Copy the whole folder

• Use Microsoft’s robocopy tool for mirroring folders across a cluster

2. Use a network share

• DemoManager tool for launching (and killing) processes across a cluster

• Or manually start the application on all nodes

Running cluster applications

2014-09-03


70

SGCT – Example

2014-09-03

• Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces)


71

• What to share?– Time or dt– Run simulations only on master or on whole

cluster?– Random numbers?– Minimize traffic– Compression– Bit-masking

Data synchronization

2014-09-03


72

• OpenGL Mathematics (GLM) is included in SGCT• Computer graphics is linear algebra!• GLM makes life easy

– Vectors– Matrices– Quaternions

• Similar syntax to glsl

Math

2014-09-03


73

• Local cluster on a single computer• View

Testing anywhere

2014-09-03


74

Testing anywhere

2014-09-03


75

Open Scene Graph (OSG)• www.openscenegraph.org• Feature rich• SGCT manages

– Cluster sharing & synchronization– Window, frustum and viewport creation

• OSG manages– Content

• OSGMaxExporter (3Ds Max plugin)

SGCT + OSG

2014-09-03


76

Scene graph

2014-09-03

• Tree structure of nodes• Intelligent structure

– Visibility & occlusion culling


77

OSG example

2014-09-03


78

• Positional 3D audio– OpenAL– Fmod Ex

Audio

2014-09-03


79

• Video pipeline• Offline rendering for the dome

Next time

2014-09-03

Documents

Development of immersive real- time applications Miroslav Andel [email protected]