1. Making 3D Games on the PlayStation 3 by Ian Bickerstaff & Simon Benson,Sony Computer Entertainment Stereoscopic 3D Team

2. Making 3D Games on the PlayStation 3 What is stereoscopic 3D and why have 3D games? Stereoscopic theory and how to make high quality 3D images Rendering 3D images on the PlayStation 3 Case studies Future work 3. What is stereoscopic 3D? Images from slightly different viewpoints are presented to each eye The difference between the two images is called parallax The brain combines the images to reconstruct the 3D world 4. No th ing ne w Le petit journal pour rire 1859 5. No th ing ne w 1 8 3 2 firs t 3 D v ie w ing de v ic eSir Charles Wheatstone reflecting stereoscope 6. No th ing ne w Brewster1 8 3 2 firs t 3 D v ie w ing de v ic e stereoscope &stereocards1 8 4 4 3 D ph o to v ie w e r Col Ian BickerstaffRobert Howletts 3D imageof Great Eastern CaptainWilliam Harrison, 1857 7. No th ing ne w Jules DuboscqBioscope disc1 8 3 2 firs t 3 D v ie w ing de v ic e1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ksCol Joseph Palteau, University of Ghent For Cross-eyed viewing 8. No th ing ne w 1 8 3 2 firs t 3 D v ie w ing de v ic e1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)Look through here!DAlmeida Eclipsestereoscope 9. No th ing ne w 1 8 3 2 firs t 3 D v ie w ing de v ic e1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)1 8 9 3 3 D po la ris e d gla s s e s John Andersons Stereoscope for polarised light1930s polarised 3D glasses 10. No th ing ne w Paul Mortier1 8 3 2 firs t 3 D v ie w ing de v ic eelectrically1 8 4 4 3 D ph o to v ie w e rsynchronised shutterglasses1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)1 8 9 3 3 D po la ris e d gla s s e s1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s 1930s 3D shutter glasses 11. No th ing ne w Lumiere brothers1 8 3 2 firs t 3 D v ie w ing de v ic e Autochromeprocess1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)1 8 9 3 3 D po la ris e d gla s s e s1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s1 9 0 3 c o lo ur 3 D ph o to gra ph sCol Ian Bickerstaff 12. No th ing ne w 1 8 3 2 firs t 3 D v ie w ing de v ic e1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)1 8 9 3 3 D po la ris e d gla s s e s1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e sRadio News, November 19281 9 0 3 c o lo ur 3 D ph o to gra ph sBairdStereoscopic television1 9 2 8 3 D TV de m o ns tra te dModification by Erik Kurland 13. No th ing ne w Ivan SutherlandsHead Mounted1 8 3 2 firs t 3 D v ie w ing de v ic e Display1 8 4 4 3 D ph o to v ie w e r1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)1 8 9 3 3 D po la ris e d gla s s e s1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s1 9 0 3 c o lo ur 3 D ph o to gra ph s1 9 2 8 3 D TV de m o ns tra te d1 9 6 5 inte ra c tiv e 3 D CGI 14. Why has 3D returned? New LCD TVs make high quality 3D affordable H I s ta nda rd no w inc lude s 3 DDM The PlayStation 3 is an ideal 3D image generator 3D gaming will be widely available soon 15. Why have 3D games? Increases the immersion and realism 16. Why have 3D games? Improves the clarity of visual information 17. Why have 3D games? Improves the clarity of visual information 18. Why have 3D games? Improves depth perception 19. How stereoscopic 3D worksThe difference in the position of our eyesgenerates parallaxThe closer the object, the more parallax isobservedDistant objects produce virtually noparallaxReplace our eyes with cameras to capturethe 3D effectThe distance between the cameras iscalled the InteraxialIncreasing the interaxial increases theparallax 20. How stereoscopic 3D works imagine a 3D TV in the room 21. How stereoscopic 3D works the perfect 3D TV 22. How stereoscopic 3D works the parallax matches perfectly 23. How stereoscopic 3D works- render the images from our cameras Will this recreate the effect? 24. How stereoscopic 3D works the parallax doesnt match.compensate for the sideways shift 25. How stereoscopic 3D works our current image again 26. How stereoscopic 3D works what is required convergence: controls the depth placement of the image 27. How stereoscopic 3D works back in the room it now looks correctortho-stereoscopic viewing 28. How to implement interaxial and convergence shifts y z Interaxial: a local world space translation in the x axis x y Convergence: a screen space translation in the x axis x 29. How to implement interaxial and convergence shifts y z Interaxial: a local world space translation in the x axis x y Convergence: a screen space translation in the x axis x asymmetrical viewing frusta point of convergence ...determines on which side of the screen objects lie 30. Interaxial and convergence in real cameras Stereoscopic camera with built in convergence Stereo Realist 35mm camera 1947 Non-stereoscopic cameras toed in to achieve convergence 31. The problem with toeing in Keystone distortion! Always avoid toeing in your cameras! 32. Story so farAdjusted the interaxialAdjusted the convergenceWe have replicated reality! 33. The limitations of parallax positive and negative parallax 34. The limitations of parallax viewing difficulty always focus here 35. The limitations of parallax the acceptable limitsC O MF O R T ZON E 36. The limitations of parallax -divergence 37. The limitations of parallax: cinemas the screen is far awayClose objects require too much negative parallax 38. The limitations of parallax: televisions the screen is close to the viewerDistant objects require too much positive parallax 39. Exactly how much positive parallax?We need to know...The size of the screen maybe but not guaranteedThe viewing distance need head trackingThe amount of light entering theviewers eyes not practical to measureThe viewers stereoscopic ability not practical to measure 40. Parallax Management use an arbitrary valueA de fa ult m a xim um +v e pa ra lla x o f1 / 3 0 th s c re e n w idthA o ids div e rge nc e fo r s c re e ns up to 9 0vinc h e s1/30th screen width 41. Parallax Management use an arbitrary valueA de fa ult m a xim um +v e pa ra lla x o f1 / 3 0 th s c re e n w idthA o ids div e rge nc e fo r s c re e ns up to 9 0vinc h e s 1/30th screen widthOn a typic a l s c re e n th e m a xim um de pthis a ppro xim a te ly th e s a m e a s th ev ie w ing dis ta nc eA o inc lude a 3 D s tre ngth s lide r ls 42. Limiting the positive parallax: solution 1as it originally was... Restrict the depth in your world to be within the required parallax range 43. Limiting the positive parallax: solution 1with a wall added... Restrict the depth in your world to be within the required parallax range 44. Limiting the positive parallax: solution 1 how the depth is perceivedmax parallaxreal world viewed worldAccurate depth but large restrictions on the game design 45. Limiting the positive parallax: solution 2 as it originally was...Reduce the amount of convergence 46. Limiting the positive parallax: solution 2 convergence reduced ...but the objects are now in front of the screenReduce the amount of convergence 47. An aside: placement and scaling in stereoscopic 3D Decrease the convergenceEverything moves closery x Everything decreases in size 48. An aside: placement and scaling in stereoscopic 3D Decrease the convergenceEverything moves closeryxEverything decreases in size Decrease the interaxial Objects in the foreground move furthery awayz Objects in the foreground increase inx size Objects in the distance are unchanged 49. An aside: placement and scaling in stereoscopic 3D First adjust your convergence to position y your distant objects x y z Then adjust the interaxial until your foreground objects are correct x ...see how this affects the room scene 50. Limiting the positive parallax: solution 2 with reduced interaxial ...foreground objects now the correct side of the screen 51. Limiting the positive parallax: solution 2max parallax display infinity compressed expandedreal worldviewed world ...acceptable parallax, but the world appears distorted 52. What happens if you alter your viewing distance?real worldviewed worldThe amount of perceived depth increases with viewing distance 53. Choice of lenses real worldviewed worldWhat happens if the viewer is stationary but the camera and field of view are altered? 54. Choice of lenses wide angle lensesreal world viewed world A wide angle lens exaggerates depth 55. Choice of lenses telephoto style lenses real world viewed world A telephoto-style lens will compress depth not good for 3D 56. Lens choice to improve the solution 2 distortion: beforemax parallax display infinity real worldviewed world Can the distortion from before be improved?.... 57. Lens choice to improve the solution 2 distortion: after max parallax display infinity real worldviewed world The foreground objects are now closer to their correct positions 58. Summary a compromiseTelevisions limit the amount of positive parallax we can useEither limit the depth of our world game design issuesOrAlter the 3D settings - distortionAdjusting the 3D settings can control the distortionThis is the challenge of stereography! 59. Negative parallax limitations? objects coming out of the screen What we would like to see... 60. Negative parallax limitations? objects coming out of the screen What we actually see window violations 61. Other parallax limitations: 2D overlays on the screenWhat we would like to see 62. Other parallax limitations: depth conflictsWhat we actually see 63. Summary about negative parallax Keep most objects behind the screen Best for fast moving objects and special FX Occasional negative parallax is entertaining! 64. Other issues: Fusion regions of similar parallaxThe brain can fuse together regions ofsimilar parallax to form a single imageLarge variations in parallax cant befusedHandy for extracting objects frombackgroundsRepeatedly switching between fusionzones is tiring 65. Scaling the world to maximise the parallax Our stereoscopic vision works best for objects close to us What if our game world operates outside of that range? 66. Scaling the world to maximise the parallax Here the cameras are metres apart It is as if we are a giant looking down on the world The apparent size of our 3D world is a creative decision 67. Summary of other issues Avoid the requirement to switch from fusion zone to zone The world can be scaled to bring it into our 3D perception range 68. Implementation on the PlayStation 3 SDK facilities available supports 3D output tells you if the TV supports 3D gives you the screen size The choice to switch to 3D is up to the user from within your game 69. Implementation on the PlayStation 3 a 3 step process1280 pixelsStep 1: Generate two images Left 720 pixels Render into a 1280x1470 buffer Images automatically converted into HDMI 3D output at 59.94Hz1470 pixels 30 pixel gap filled with any uniform colour 30 pixels Any frame rate up to 59.94 can be used Avoid frame tearing Right720 pixels 70. Implementation on the PlayStation 3 a 3 step process 1280 pixelsStep 1: Generate two imagesH rdw a re ups c a ling is a v a ila b le aLeft 720 pixelsU c a le d 3 D im a ge s te nd to lo o k m uc hpsb e tte r th a n 2 D im a ge sU e go o d a nti- a lia s ing s 1470 pixels 30 pixel gap filled with any uniform colour 30 pixelsLo w - re s a nti- a lia s e d im a ge s lo o k m uc hb e tte r th a n h igh re s im a ge s w ith no a nti-a lia s ing Right 720 pixels 71. Implementation on the PlayStation 3 a 3 step process Step 2: Apply convergence to define the maximum positive parallax A 2D x axis translation in screen space Shift the left image to the left and the right image to the right Normally use 1/30th screen width as the defaultalternating left/right frames 72. Implementation on the PlayStation 3 a 3 step process Step 2: Apply convergence todefine the maximum positiveparallaxTake care the convergence is appliedto all your rendering pipeline alternating left/right frames 73. Implementation on the PlayStation 3 a 3 step process Step 3: Apply the interaxial by horizontally offsetting the cameras Keep the cameras parallel Maximise the depth by increasing the interaxial until the closest objects are just behind the screenalternating left/right frames Surprisingly, dynamically altering the interaxial is not that noticeable to the viewer interaxial = 2 X tan (horizontal field of view/2) x closest distance30 74. Implementation on the PlayStation 3 a 3 step process Step 3: Apply the interaxial by horizontally offsetting the cameras Be careful with large interaxials miniaturisation? The field of view and camera distance can be adjusted tooalternating left/right frames Try to avoid window violations and depth conflicts interaxial = 2 X tan (horizontal field of view/2) x closest distance30 75. Implementation on the PlayStation 3 a 3 step process Step 3: Apply the interaxial by horizontally offsetting the cameras Take care with any 2D rendering tricks- are they OK in 3D? alternating left/right frames Reflections will need to be calculated for each eye 76. PlayStation 3 Case studies 77. WipEoutHD Max resolution : 1080P Max refresh rate : 60Hz 78. WipEoutHD Advantages Immersion Understanding of track flow 79. WipEoutHD Performance 80. WipEoutHD 3D User interface 81. WipEoutHD Missile marker 82. MotorStorm: Pacific Rift Max resolution : 720P Max refresh rate : 30Hz 83. MotorStorm: Pacific Rift Advantages Immersion Accessibility Scale 84. MotorStorm: Pacific Rift Performance Frame tearing 85. MotorStorm: Pacific Rift Field of view 86. MotorStorm: Pacific Rift User interface opacity 87. MotorStorm: Pacific Rift Split screen 88. SuperStardustHD Max resolution : 1080P Max refresh rate : 60Hz 89. SuperStardustHDAdvantages Clarity Entertainment 90. SuperStardustHD Performance 60Hz 91. SuperStardustHD Negative parallax (out of screen) 92. Future work- a rendering technique used by Sony Pictures Entertainment Pa ra lla x fro m de pth m a p R nde r aes ingle im a ge a nd c re a te th e s e c o nd im a ge us ing th e de pthm ap 93. Future work- a rendering technique used by Sony Pictures Entertainment Pa ra lla x fro m de pth m a p La rge pa ra lla x c re a te s ga ps w h ic h ne e d to b e b a c kfille d Tra ns pa re nc y a nd re fle c tio ns w o n t w o rk w ith th is a ppro a c h Ne e d to c o m b ine it w ith full re nde ring fo r diffic ult o b je c ts Ta ke s a b o ut 3 % o f th e to ta l SPUtim e pe r 6 0 H fra m ez 94. AcknowledgementsF u rt h e r re a din g :Introduction to stereoscopic 3D:Mendiburu, 3D Movie Making; Stereoscopic Digital Cinema from Script to Screen,2009History References:Reynaud, Tambrun & Timby Paris in 3D; from Stereoscope to Virtual Reality 1850-2000, 2000F Drouin, The Stereoscope and Stereoscopic Photography 1894 95. Questions?