Virtual Worlds Technology Roadmap
December 2008
Seung Wook Kim, Francisco Imai, Stefan MartiHCI Research Team, Computer Science Lab
Samsung R&D Center, San Jose
SISA CSL “Innovate by doing”
VW Technology Trends Analysis
1. State of The Art1.1 Virtual World Clients for Fixed Platform1.2 Virtual World Clients for Mobile Platform1.3 Virtual World Server Technologies
2. Catalyst Elements for VW2.1 3D content2.2 Natural interaction for 3D2.3 Artificial Intelligence in VW
3. Mixed Reality as a Killer Application
Backup Slides Mixed Reality Workgroup Mission and Goals Issues in 3D
- 2 -
SISA CSL “Innovate by doing” - 3 -
Trends: 1. Variety of VW clients: - standalone, Web-based (ActiveX or Flash), CE-based (game console), etc 2. Multimodal in-world communication (e.g. textual + bubble + voice chat) 3. Support of user created 3D content via standard format (e.g. COLLADA) Challenges:•Insufficient client performance: 3D rendering quality, physics, animation•Voice chat is becoming standard, but it still doesn’t benefit from non-verbal communication (i.e., non-intuitive avatar gesture/action control)•Complicated 3D content creation pipeline: CAD/CAM skills are required
Trends: 1. Variety of VW clients: - standalone, Web-based (ActiveX or Flash), CE-based (game console), etc 2. Multimodal in-world communication (e.g. textual + bubble + voice chat) 3. Support of user created 3D content via standard format (e.g. COLLADA) Challenges:•Insufficient client performance: 3D rendering quality, physics, animation•Voice chat is becoming standard, but it still doesn’t benefit from non-verbal communication (i.e., non-intuitive avatar gesture/action control)•Complicated 3D content creation pipeline: CAD/CAM skills are required
1.1 State of The Art: VW Clients for Fixed Platforms
VW Web Plug-In
VW on CE Devices
Sony’s Playstation Home
VIVATY Scenefor AIM
Opportunities:•Actively apply multicore/parallel computing technologies (for both CPU and GPU) on VW client development •Sensor technologies that will accurately translate physical motions into virtual actions•Design intuitive and intelligent UI for easier creation and/or manipulation of virtual objects
Opportunities:•Actively apply multicore/parallel computing technologies (for both CPU and GPU) on VW client development •Sensor technologies that will accurately translate physical motions into virtual actions•Design intuitive and intelligent UI for easier creation and/or manipulation of virtual objects
Flash-based VW
WebFlock SmallWorlds
Standards for 3D content
Forterra’s Olive VW Platform supporting COLLADA*
Current status:• The Electric Sheep
Company: Providing Flash-based & private-labeled VW service, WebFlock
• SMALLWORLDS: Flash-based VW
• VIVATY: Combining Web plug-in Virtual Community (VIVATY Scene) with social networking services (AIM and Facebook)
• SONY: Providing VW community service (HOME) based on the game console Playstation3
• FORTERRA: Support for COLLADA* into its open and standards-based VW platform, OLIVE
Current status:• The Electric Sheep
Company: Providing Flash-based & private-labeled VW service, WebFlock
• SMALLWORLDS: Flash-based VW
• VIVATY: Combining Web plug-in Virtual Community (VIVATY Scene) with social networking services (AIM and Facebook)
• SONY: Providing VW community service (HOME) based on the game console Playstation3
• FORTERRA: Support for COLLADA* into its open and standards-based VW platform, OLIVE
SISA CSL “Innovate by doing”
Trends: 1. Mobile 3D clients start providing users with permanent access to VWs 2. Connecting VW users in real-time from instant messaging client 3. Emergence of motion sensors and haptic feedback on mobile devices
Challenges:• Limited performance of remote rendering & streaming technologies• No voice/text enabled thin client connecting VW on mobile platform• Lack of new use models for sensor-mounted mobile devices
Trends: 1. Mobile 3D clients start providing users with permanent access to VWs 2. Connecting VW users in real-time from instant messaging client 3. Emergence of motion sensors and haptic feedback on mobile devices
Challenges:• Limited performance of remote rendering & streaming technologies• No voice/text enabled thin client connecting VW on mobile platform• Lack of new use models for sensor-mounted mobile devices
1.2 State of The Art: VW Clients for Mobile Platforms
State-of-Art Mobile Device for 3D Graphics
Mobile VW client
Vollee’s SL client
Vivox’s SLim client
Opportunities:•Enhance the performance of 3D graphic accelerating processor for mobile devices
•Integrate VoIP/messaging system with mobile immersive 3D client
•Develop highly-efficient mobile VW client and its UI mechanism
Opportunities:•Enhance the performance of 3D graphic accelerating processor for mobile devices
•Integrate VoIP/messaging system with mobile immersive 3D client
•Develop highly-efficient mobile VW client and its UI mechanism
3D Motion Tracking
Apple’s iPhone with ARM1176JZF & Samsung S3C6400
Thin VW client
- 4 -
Current status:• VOLLEE: Leader in mobile VW
service with remote rendering & streaming technologies
• VIVOX (w/ Linden Lab): First commercial voice-driven IM client (SLim) for Second Life
• APPLE: iPhone is equipped with a top-level chip (ARM1176JZF) with 3D accelerating coprocessor (Samsung S3C6400)
Current status:• VOLLEE: Leader in mobile VW
service with remote rendering & streaming technologies
• VIVOX (w/ Linden Lab): First commercial voice-driven IM client (SLim) for Second Life
• APPLE: iPhone is equipped with a top-level chip (ARM1176JZF) with 3D accelerating coprocessor (Samsung S3C6400)
SISA CSL “Innovate by doing”
Trends: 1. Open-source approach to interoperable server platform 3. Efforts for building MMOG environments in VW 2. Both open and private-labeled virtual world (VW) services are increasing
Challenges:• Client-neutral, flexible and highly-trusted server platform is required• Difficult to attract and hold a large volume of concurrent users• Server maintenance cost is still expensive (per user)
Trends: 1. Open-source approach to interoperable server platform 3. Efforts for building MMOG environments in VW 2. Both open and private-labeled virtual world (VW) services are increasing
Challenges:• Client-neutral, flexible and highly-trusted server platform is required• Difficult to attract and hold a large volume of concurrent users• Server maintenance cost is still expensive (per user)
1.3 State of The Art: VW Server Technologies
MMOG* in VW
Multiverse
Opportunities:•Develop software standards for interoperable platforms for online 3D applications and virtual worlds
•Connect different hardware platforms (e.g. mobile devices and fixed systems) via VW applications
•Rethink VW platform architecture (e.g. cloud computing)
Opportunities:•Develop software standards for interoperable platforms for online 3D applications and virtual worlds
•Connect different hardware platforms (e.g. mobile devices and fixed systems) via VW applications
•Rethink VW platform architecture (e.g. cloud computing)
Interoperable VW
Open-Source Community
Open Grid by IBM+Linden Lab
OpenSimulator
- 5 -
Current status:• OpenSIM: Open-sourced and
flexible 3D application server supporting various VW clients
• IBM + Linden Lab: Releasing the first interoperable VW service platform (Open Grid Beta)
• Multiverse: Providing open development toolkit for MMOG-type VW, including authoring tools and source code
Current status:• OpenSIM: Open-sourced and
flexible 3D application server supporting various VW clients
• IBM + Linden Lab: Releasing the first interoperable VW service platform (Open Grid Beta)
• Multiverse: Providing open development toolkit for MMOG-type VW, including authoring tools and source code
SISA CSL “Innovate by doing”
Trends: 1. Renaissance of 3D movies - 11 movie titles released (from 2005 to 2008) and many more in 2009 2. Consistent popularity of games with 3D content 3. Explosive increase in medical imaging with 3D content ($1.4B by 2013) 4. Web-based VW and Mirror World (SL, Google Earth) 5. Stereo Synthesis - 2D to 3D conversion, depth capturing, multi-aperture imaging
Challenges:•Displays: auto-stereoscopic displays are expensive and cause user fatigue and most of 3D viewing solutions require wearing awkward glasses•UX: there is no good solution in the market for natural and intuitive user interface for 3D content
Trends: 1. Renaissance of 3D movies - 11 movie titles released (from 2005 to 2008) and many more in 2009 2. Consistent popularity of games with 3D content 3. Explosive increase in medical imaging with 3D content ($1.4B by 2013) 4. Web-based VW and Mirror World (SL, Google Earth) 5. Stereo Synthesis - 2D to 3D conversion, depth capturing, multi-aperture imaging
Challenges:•Displays: auto-stereoscopic displays are expensive and cause user fatigue and most of 3D viewing solutions require wearing awkward glasses•UX: there is no good solution in the market for natural and intuitive user interface for 3D content
2.1 Catalyst Elements: 3D Content
Opportunities:•Create innovative and intuitive user interaction model for immersive 3D VW client on consumer electronic devices
•Provide mobile solutions for 3D content navigation
•3D interactive movies for VW
•Remote medical service and training in VW
Opportunities:•Create innovative and intuitive user interaction model for immersive 3D VW client on consumer electronic devices
•Provide mobile solutions for 3D content navigation
•3D interactive movies for VW
•Remote medical service and training in VW
- 6 -
Current status:• Philips: Developed auto-
stereoscopic displays. Technical issues such as narrow angle of view and user fatigue. No compelling UX
• 3DV Systems: Detection of body gestures using depth cameras for avatar control. Issues of user fatigue and lack of resolution
• GestureTek: Control of digital content by tracking gestures. Could lead to user fatigue
Current status:• Philips: Developed auto-
stereoscopic displays. Technical issues such as narrow angle of view and user fatigue. No compelling UX
• 3DV Systems: Detection of body gestures using depth cameras for avatar control. Issues of user fatigue and lack of resolution
• GestureTek: Control of digital content by tracking gestures. Could lead to user fatigue
Source: Insight Media
Source: 3D SMPTE standard activities
Wendy Aylsworth (Warner Bros.)
SISA CSL “Innovate by doing”
Trends: 1. Beyond keyboard and mouse: evolution towards alternative HCI tools 2. Multi-touch: Evolution towards gesture interfaces on surface 3. Multitude of sensors: Progressive adoption of orientation, position and motion sensing (optical sensors, inclinometers, accelerometers, compass)
Challenges:•User fatigue: solutions that rely on mapping human body movements often lead to user fatigue•Lack of feedback: it is difficult to give appropriate feedback for 3D interaction
Trends: 1. Beyond keyboard and mouse: evolution towards alternative HCI tools 2. Multi-touch: Evolution towards gesture interfaces on surface 3. Multitude of sensors: Progressive adoption of orientation, position and motion sensing (optical sensors, inclinometers, accelerometers, compass)
Challenges:•User fatigue: solutions that rely on mapping human body movements often lead to user fatigue•Lack of feedback: it is difficult to give appropriate feedback for 3D interaction
2.2 Catalyst Elements: Natural UI for 3D
Opportunities:•Combine different technologies to provide a more immersive and engaging UX for 3D interaction
•Create closed-loop systems for user experience and 3D object rendering by dynamically displaying 3D object according to user position and interaction with virtual content
Opportunities:•Combine different technologies to provide a more immersive and engaging UX for 3D interaction
•Create closed-loop systems for user experience and 3D object rendering by dynamically displaying 3D object according to user position and interaction with virtual content
OrangeSeeingmachines Softkinetic
Mgestyk
- 7 -
Current status:• Seeingmachines: Developer
of face API that tracks human face with normal webcam and displays images based on view-dependent-rendering
• Softkinetic: Provides 3D real-time gesture recognition middleware
• Mgestyk: Combines 3DV systems depth camera with hand-gesture processing.
Current status:• Seeingmachines: Developer
of face API that tracks human face with normal webcam and displays images based on view-dependent-rendering
• Softkinetic: Provides 3D real-time gesture recognition middleware
• Mgestyk: Combines 3DV systems depth camera with hand-gesture processing.
SISA CSL “Innovate by doing”
Trends: 1. A.I. engines programmed to learn by interacting with humans in VW - VW gives advantage of embodiment to A.I. engine without hassle or costs that comes with physical robotics - VW allows exponential increase in interaction for AI engines to learn 2. There are efforts to make VW presence and interaction more engaging
Challenges:•Recreate self-mind in VW: there is need to create a self-awareness as present in human interaction by appropriately planning•Pattern mining: finding complex patterns requires prohibitively inefficient searches•Self-modeling: creating learning algorithms capable to embody intelligence•Language and gesture processing: interpretation of language, gestures
Trends: 1. A.I. engines programmed to learn by interacting with humans in VW - VW gives advantage of embodiment to A.I. engine without hassle or costs that comes with physical robotics - VW allows exponential increase in interaction for AI engines to learn 2. There are efforts to make VW presence and interaction more engaging
Challenges:•Recreate self-mind in VW: there is need to create a self-awareness as present in human interaction by appropriately planning•Pattern mining: finding complex patterns requires prohibitively inefficient searches•Self-modeling: creating learning algorithms capable to embody intelligence•Language and gesture processing: interpretation of language, gestures
2.3 Catalyst Elements: Artificial Intelligence in VW
Opportunities:•Virtual pets can be a killer application for A.I. in VWs
•Adding ability to have pets genuinely learn and respond to the environment will make them more real to the user and increase the user/virtual pet bond
•Autonomous avatars for creating rich virtual environments
Opportunities:•Virtual pets can be a killer application for A.I. in VWs
•Adding ability to have pets genuinely learn and respond to the environment will make them more real to the user and increase the user/virtual pet bond
•Autonomous avatars for creating rich virtual environments
- 8 -
Current status:• Novamente IRC: methodology
for teaching:
• Imitative learning: instructor shows by example
• Reinforcement learning:– student repeats movements with feedback
• Corrective learning:– instructor actively corrects student’s movements
Current status:• Novamente IRC: methodology
for teaching:
• Imitative learning: instructor shows by example
• Reinforcement learning:– student repeats movements with feedback
• Corrective learning:– instructor actively corrects student’s movements
SISA CSL “Innovate by doing”
Trends: 1. Increase in information overlay (Googlemaps, VWs)2. See-through devices as a platform for mixing VW and RW (iPhone App)
Challenges:•Design of wearable displays: current solutions are bulky and too geeky•Need for an intelligent agent: user should not be overwhelmed by profusion of virtual data but system should have an embedded agent that filters relevant or preferred information•Lack of precision of location sensors: current location sensors need to improve precision in order to identify location for data augmentation
Trends: 1. Increase in information overlay (Googlemaps, VWs)2. See-through devices as a platform for mixing VW and RW (iPhone App)
Challenges:•Design of wearable displays: current solutions are bulky and too geeky•Need for an intelligent agent: user should not be overwhelmed by profusion of virtual data but system should have an embedded agent that filters relevant or preferred information•Lack of precision of location sensors: current location sensors need to improve precision in order to identify location for data augmentation
3. Mixed Reality as a Killer Application
Opportunities:•Develop technologies for an unobtrusive system with natural UX for mixed reality
•Possible applications: - Travel planning with shared
experience and enhanced participation
- Training of first responders- Inter-reality physics: virtual reality
system coupled with real-world one
Opportunities:•Develop technologies for an unobtrusive system with natural UX for mixed reality
•Possible applications: - Travel planning with shared
experience and enhanced participation
- Training of first responders- Inter-reality physics: virtual reality
system coupled with real-world one
Virtual Continuum (by Paul Milgram, 1994)
- 9 -
Current status:• BBC R&D: Have explored the
impact of mixed reality in media: http://www.bbc.co.uk/rd/projects/virtual/mixtv/index.shtml
• Nokia Research: Research on mobile mixed reality system with augmentation of real world using mobile phones.
• GaTech: Augmented Reality in SL
Current status:• BBC R&D: Have explored the
impact of mixed reality in media: http://www.bbc.co.uk/rd/projects/virtual/mixtv/index.shtml
• Nokia Research: Research on mobile mixed reality system with augmentation of real world using mobile phones.
• GaTech: Augmented Reality in SL
Nokia Research BBC R&D
SISA CSL “Innovate by doing”
Backup slides
SISA CSL “Innovate by doing”
Mixed Reality Workgroup: Mission
We believe that Mixed Realities have the potential to change the way we interact with the world in a radical way.
Under Mixed Reality, we refer to the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.
But radical change can only happen if the technology is available on a broad basis.
The more people who can interact with Mixed Realities, the clearer the advantages of such a technology will become.
Therefore, our mission is to improve human interaction with real and virtual worlds by encouraging the mass adoption of Mixed Reality technologies, and the creation of an industry with very low barriers to innovation and business success.
SISA CSL “Innovate by doing”
Mixed Reality Workgroup: Goals
We propose to provide information aimed at technology providers, investors, and business decision makers:
• Casebook that documents past efforts, lessons learned, and best practices
• Roadmap of key technical requirements for "pacing mass usages“
• “Grand Challenges” collection to inspire innovation in the industry
We are an Active Workgroup that develops documents which frame the usages, as well as key technical and business model barriers, to the success of the Mixed Reality vision.
SISA CSL “Innovate by doing”
Issues in 3D
3D content creation - Quality x Quantity - Mastering 3D content for home x cinema - 3D creation and workflow training and education
3D content storage, transmission and distribution - File formats, compression, transmission - Coordinate work in standard bodies (DCI, MPEG, IEEE 3D Consortium, 3D4YOU, etc…)
3D Promotion - Common naming conventions and vocabulary - Develop content for web distribution - End-user and retailer/reseller education and in-store demos
3D Displays - 3D display quality metrics – ghosting, brightness, etc… - 3D human factors – eye fatigue, viewing time, viewing angle dependency, etc
SISA CSL “Innovate by doing”
3D Display Technologies
SISA CSL “Innovate by doing”
3DTV vs. 3D-ready TV
SISA CSL “Innovate by doing”
Cost
SISA CSL “Innovate by doing”
PDP display market forecast
Samsung manufactures 3D-ready PDP-TV. This is the expected PDP forecast in the USA with 3D TVs dominating the market in the next years.
SISA CSL “Innovate by doing”
3D LCD display market forecast
It is expected that 3D LCD is going to emerge in the market and grow exponentially
SISA CSL “Innovate by doing”
3D and 3D-ready display market
SISA CSL “Innovate by doing”
3D content – movie as a driving force
SISA CSL “Innovate by doing”
TV – How do you know it is 3D-ready?
SISA CSL “Innovate by doing”
3D-ready TV – 3D glasses and PC requirements
SISA CSL “Innovate by doing”
3D-ready TV – 3D glasses and PC requirements
SISA CSL “Innovate by doing”
3D-ready TV – PC requirements