Immersive Connected Experiences How connected visual computing will change the
virtual world as well as the actual world
Jim Held, Sean Koehl, John Hengeveld, Mic Bowman and Inga Vailionis
Intel Corporation
Revision: 11/20/2008
Executive Summary
Immersive Connected Experiences (ICE) are a new category of connected visual computing
usage models in which people will share experiences and information online within highly
intuitive, interactive visual interfaces. Initial examples of ICE include two main categories:
Simulated Environments such as virtual worlds, online multiplayer games and 3-D movies,
and Augmented Reality where images from the real world are combined with digital
information to provide an enhanced view of the globe around us. This paper describes the
key technical challenges in bringing these usages to the mainstream, including new
client/server platform innovations, more robust distributed computing techniques, tools to
facilitate user-generated 3-D content, and techniques to improve experiences on mobile
devices. It also describes Intel’s research agenda aimed at removing key technical barriers to
the widespread adoption of ICE.
White Paper Research @ Intel
White Paper: Immersive Connected Experiences
2
Introduction
A frequent driver of innovation is convergence. The convergence of radio and computing, for instance, brought us
wireless computing and opened up a world of connected mobility as wireless Internet access grew. Today the
convergence of several Internet trends involving “connected” visual computing technologies is enabling a new set of
usage models.
Visual computing combines real world and digital world experiences in fascinating ways. The potential of connected
visual computing as a transformative technology is just beginning to be explored, but when combined with
broadband internet connectivity, it holds great promise for connecting people within highly intuitive, interactive
visual interfaces that will enhance the ability to share experiences and information. Intel calls such connected visual
computing usages Immersive Connected Experiences (ICE). Intel is taking a leading role in driving adoption of ICE and removing the technical barriers to its widespread usage.
This paper will describe ICE and its potential, plus discuss the technology and ecosystem required to enable easy
creation of rich, immersive and scalable experiences.
The Convergence of the Visual
and the Connected
More and more people today use computers and many mobile devices
(handhelds, cellular phones, etc.) that have integrated cameras that allow
them to capture the world around them. At the same time, broadband
connectivity is making it easier to share visual experiences and giving rise
to social networks where people congregate for exchanging personal
content and other aspects of their lives. With the rapid growth of
Facebook, YouTube, Picasso, and similar personal content-oriented sites,
the Internet is becoming as much or more about user-generated content
as it is about communications. What’s more, the growth and popularity of
virtual worlds, such as Habbo and Second Life, is beginning to
demonstrate how similar these worlds can be to the real world and how
these sites (and the people participating in them) take on lives of their
own.
Looking at these Internet trends (social networking, user-generated
content, broadband connectivity, mobile computing, and visual, immersive
worlds), Intel sees an ever quickening convergence of visual and
connected computing that is blurring the traditional divisions between
real world activities and the digital world. This convergence is the basis
for ICE, an emerging usage model that offers great opportunities for
expanding ways we interact and experience the world and each other.
To better understand ICE and its potential, let’s look deeper at its
components.
Connected Computing
Connected computing, as a usage model, is easy to comprehend. We’ve all
grown accustomed to, even dependent on, being able to connect to each
other and exchange data, voice and video through the Internet. But
connectivity as a computing trend has really just begun. The increase and
diversity of mobile computing devices, faster broadband speeds, plus the
growth in input devices such as cameras and various types of sensors,
are not just making it easier to stay connected all the time – always on,
always connected – but also spawning new uses and making it more
important to have a continuous connection.
Visual Computing
Most of us think of computing as visual. After all, we look at a screen. But
what we mean here by “visual computing” is immersive, real-time, life-like
computing experiences. We’re at the dawning of an age of such visual
computing experiences. Examples of visual computing can already be
seen in the 3D visual models used in financial services, interactive medical
imaging, and the virtual worlds of Internet gaming.
Visual computing applications are evolving to deliver ever more
immersive visual experiences that look real, act real, and feel real. Many of
the elements necessary to bring such holistic visual experiences into
mainstream computing are in development and, with improvement, will
make visual computing an integral part of daily life.
Some of these elements include:
• Greater photorealism
• Improved modeling (better physics and intelligence)
• Better multimedia (improved video/audio editing and
processing)
White Paper: Immersive Connected Experiences
3
• More visual and tactile interactivity (advanced user
interfaces – including new touchscreen and haptic
technologies [1] – and other devices)
Immersive Connected Experiences
Technologies that combine connected and visual computing enable
people to share experiences and information within a highly intuitive,
interactive visual interface. Through connected visual computing
technologies, we’ll be able to use computing devices to enhance and
extend our experiences of the real world, create new virtual worlds, and
bridge the digital and real world with increasingly rich, Immersive
Connected Experiences.
ICE usages will include enhanced social networking, workplace
collaboration, retail sales, virtual tourism, marketing, distance learning,
training, gaming, and the arts, to name a few.
Current examples of ICE fall into two main categories.
Simulating Environments – These are multi-user graphical
environments where large numbers of people interact for the
purposes of gaming, socialization, learning, or content creation.
Examples include virtual worlds, massively multiplayer online
games (MMORGs), and 3-D cinema.
Augmented Reality – There are environments where
information from the actual world is combined with digital
information to provide an enhanced view of the real world.
Examples include Earth mapping applications such as Google*
Earth.
The rapid growth of virtual worlds is particularly indicative of the
potential for ICE. In June 2008, the research firm Strategy Analytics
predicted that the percentage of registered users that will go on to
become active users in virtual worlds will increase from 10 percent now
to 27 percent by 2017, creating a market base of almost 1 billion users
[2]. This is up from today’s 137 million active users. Pre-teens make up
more than half the users, spending 16 hours per week (more than on TV
or magazines) and purchasing items in the virtual worlds [3]. Many
convert their parents into virtual world users. Even more interesting will
be what happens if many of these pre-teens carry their virtual world
habits into adulthood. Strategy Analytics puts the 2008 total market for
user services at $310 million. The research group predicts that it will
grow to $8 billion in 2017 [4].
Augmented reality is also growing and evolving. Many people are familiar
with how online maps have evolved to include the overlay of satellite
images and recently street-level surround images. Combining this mixture
of real and virtual will be particularly compelling with the connectivity and
sensors of a mobile Internet device (MID). Images, video, audio, GPS,
acceleration sensors could all provide rich input that could be used to
create and then present to the user an augmented form of reality.
Consider being able to take a photo of a monument and then doing a
visual search using the image for real-time recognition of the monument
and information about its significance. With the processing power,
database and communication capability of a continual Internet
connection, there are many exciting opportunities that can be enabled.
Meeting the Challenges of ICE
Removing key technical barriers to the adoption of Immersive Connected
Experiences will require research into hardware and software innovations
that improve end user experience, as well as the development of
standards that improve interoperability. “We want cinematic quality in the
graphics,” said Intel CTO Justin Rattner in his Fall 2007 IDF R&D keynote.
“We want to make user content creation much easier. Of course we want
to retain the persistence, but we want that persistence to apply across
virtual worlds, not to be contained in a single virtual world. And higher
levels of social interaction, better human-computer interfaces to enable
that, and we'd like the behavior -- the way things act in the virtual world -
- to be much more natural.”
The desired qualities for future ICE environments include the following:
Immersive visuals. Because they’re based on connected visual
computing technologies, the user experience should be greatly
enriched by immersive, highly realistic visuals.
User-generated 3D content. Many ICE applications will be
forms of social media. Thus, people will need and want to
create or modify visual content within ICE environments. This
will require new tools such as 3D character designers that are
as easy to use as the blogging, photo, and video editing tools
of today.
Interoperability. Like the Internet, ICE applications based on
broadly accepted standards will enable data and personas to
be shared between various ICE environments in creative ways.
ICE Example 1: Simulated Environments
Imagine if instead of having to waste a day traveling to a
conference and another day traveling back, you could attend a
virtual conference in a simulated environment. Take that a step
further and imagine, between presentations, being able to have
conversations with other attendees’ avatars where facial
expressions and body language actually help you and others make
your points with the correct emphasis.
Example: Qwaq* Forums
White Paper: Immersive Connected Experiences
4
In fact, according to Strategic Analysts vice president and
research director Barry Gilbert, "What's holding back the
numbers [participants] is the reliance on any single virtual
world because they're not yet portable."
Robust distributing computing. Because ICE applications are
distributed, good performance requires high quality, well-
managed interactions among a wide variety of clients and
servers. ICE applications should adapt to a diverse array of
clients so that the user experience scales smoothly from
performance-optimized home or office systems to compact
mobile devices optimized for energy-efficiency.
Scalability. ICE Environments should be able to scale over time
to incorporate new technologies which improve their quality as
well as to add more users as they grow in popularity.
Intel is driving ICE development through its continuing research into
many of the technologies and advancements that will be required to
bring ICE into the mainstream. This includes research in redefining
mobility, workload analysis, simulation, model-based computing, and tera-
scale computing.
Intel divides the challenges into four areas: platform optimization, visual
content, distributed computation, and mobility.
Platform Optimization
ICE applications are very compute-intensive. They demand a lot of
servers, clients and the network. Consider the complexity of the objects
and the need for accurate simulations, particularly realistic object
behavior. Then multiply that by the number of users and, the scene
complexity as more users participate and interact, and it’s easy to see
how ICE can be a tremendous challenge to system and network scaling.
What’s needed are new architectures that scale well and deal with a
range of client performance and capabilities, dynamically adapting to
changing demands and virtual world circumstances. Intel believes that ICE
applications will follow the same evolution the rest of social networking
and Internet applications have, evolving from proprietary solutions to
standard protocols and building blocks that can be used to build more
sophisticated services. Such modular design will also allow for scaling
through distribution and replication of components, as well as
repartitioning functionality among types of servers and the diversity of
clients.
Intel has a number of internal teams, projects and software-related
efforts underway to speed the transition to connected visual computing
technologies and future usage models like ICE. One extremely relevant
example here is Intel’s Tera-scale Computing Research Program. This is a
worldwide effort to advance computing technology by scaling multi-core
architectures to 10s to 100s of cores and embracing a shift to parallel
programming. Through this effort, Intel aims to enable applications and
capabilities like ICE (see Figure 1). Other research includes redefining
mobility, workload analysis, simulation and model-based computing –
most of which will benefit from tera-scale computing.
Figure 1: Tera-scale microprocessors based on many cores will be
essential to computing the elements of ICE applications in parallel.
Visual Content
One of the distinguishing features of ICE applications, such as Second
Life, is the existence of player-generated content. Enabling easier end-
user creation and use of rich visual content is essential to ICE. In ICE
worlds based on the premise of user-customized 3D interactive content,
such data-rich content won’t be conducive to distribution at startup or
even on a DVD ahead of time. It must be distributed real-time as users
explore and enter newly created regions. This will place severe demands
on the network requiring innovations in how content can be concisely
represented as well as cached and pre-fetched. This will also require a
new generation of technologies and tools designed to enable
interoperability.
To simplify content creation, Intel is researching a variety of areas. One is
parameterized content creation. This is the development of sets of
parameters for configuring and controlling an object, such as an avatar.
End-user creation is simplified to selecting and adjusting parameters,
such as the fullness of a face or the sharpness of a chin. This reduces
ICE Example 2: Augmented Reality
You’re walking through a town in a foreign country and see an
interesting building and want to know more about it. You look at
your Mobile Internet Device’s screen and, from communicating with
nearby sensors and using GPS, it knows precisely where you are
and provides an description of it, links to relevant websites
providing more information about the building and who used it, and
even provides a 3-D video tour of the interior.
White Paper: Immersive Connected Experiences
5
real-time distribution by enabling a very compressed format of a few
parameters (see Figure 2). Behavioral scripts can animate those few
parameters for example to make an avatar frown, smile, laugh, or grimace.
Delivering immersive visual experiences that look real, act real, and feel
real will require a programmable, ubiquitous and unified architecture.
Programmability using industry standard architecture and tools will be
necessary to proliferate the algorithms that go into visual computing.
Intel’s strong track record in processors, graphics and media architecture,
platform architecture and software developer tools will be important in
providing the foundation and leadership for delivering visual computing.
In fact, at a SIGGRAPH conference in August 2008, Intel presented a
paper disclosing details of the Larrabee architecture, the codename for a
many-core architecture that Intel is developing separately from its
current line of integrated graphics accelerators. The Larrabee
architecture is expected to kick start an industry-wide effort to create
and optimize software for the multitudes of cores expected to power
future computers.
Figure 2: Parameterized faces would make it easier to develop a
personalized, expressive avatar.
Over time, the consistency of Intel architecture and developer freedom
afforded by the Larrabee architecture will bring about massive innovation
in many areas and market segments. For example, while current games
keep getting more and more realistic, they do so within a rigid and limited
framework. Working directly with some of the world's top 3-D graphics
experts, Larrabee will give developers of games and application
programming interfaces (APIs) a blank canvas onto which they can
innovate like never before.
Distributed Computation
What makes ICE applications even more demanding is that they’re
distributed systems using shared and partitioned server resources and
connected to a range of desktop and mobile clients. In the future, we will
want to enable seamless connections to other environments and allow
movement between them. Imagine thousands of users connected
through clients with simulations connected to other users and global
resources, each wanting to transition from one environment to another.
All the connections are potential chokepoints and the distributed
architecture must balance the location and partitioning to best match
available compute resources to demand.
In Intel’s research, Intel is determining the impact and requirements of
connected visual computing technologies and usage models like ICE for
future Intel platforms, engaging the community to address the
distributed system architectural issues that they involve. One obvious
area to address is enhancing user experiences by expanding the
capabilities of mobile platforms and enabling execution on diverse clients.
Another is expanding the use of more parallelized code. This will improve
performance by taking better advantage of the growing number of cores
in upcoming computing platforms.
Improving Mobile Experiences
Enabling truly immersive computing experiences within the power,
bandwidth and intermittent connectivity constraints of a mobile internet
device (MID) is a real challenge. Intel’s Carry Small Live Large research
initiative includes efforts to enable Composability, in which wireless
devices discover, identify and connect with other nearby computing
resources such as sensors, displays, storage devices or other computers.
In doing so, they will be able to compose ad-hoc computing systems that
are capable of sharing functionality and computing power. Compression
and power efficient communication techniques will further enable
maximum use of available wireless bandwidth.
Augmented reality is one area of research in improving mobile
experiences. This is the use of MIDs and sensors to give users a more
immersive experience of the real world and provide additional information
regarding what’s around them. A good example would be using a MID
programmed to automatically identify significant buildings nearby for an
architecture buff and provide both information about the building and its
architect. Another example would be virtual instruction. Your power
windows freeze in your car and, through communication between your
car and MID, instructions on how to find and change the blown fuse
causing the problem would be automatically displayed.
Another related and promising Intel research project is synopsis-based
reasoning. This is the use of sensor input in combination with
connections to remote server resources and other users to enable mobile
devices to learn and understand the everyday activities and environment
of an end user. Devices could then infer everything from when you might
like to be notified that a colleague is nearby or whether a restaurant
you’ve liked before is in the area.
Accelerating ICE Adoption Intel sees connected visual computing technologies as the basis for an
exciting set of Immersive Connected Experiences that can provide clear
end-user benefit based on the performance and capabilities of future
platforms. While there is much that we can do with our products, we are
applying our research to accelerating adoption by engaging the ICE
community. Open source efforts such as OpenSim (a Virtual Worlds
Server which can be used for deploying 3D virtual environments) provide
excellent settings to develop the innovations necessary to address the
scaling and interoperability challenges. Through OpenSim, Intel is working
White Paper: Immersive Connected Experiences
6
with industry players (virtual world service providers, IBM, Microsoft, and
many others) to explore the necessary innovations in distributed system
scaling and interoperability for ICE applications.
ICE presents a number of technical challenges that correspond well with
a wide variety of areas Intel has been actively researching for a number
of years. Intel researchers are today driving new innovations through
optimizing platforms, improving visual content processing, developing
better distributed computing architectures, inventing new technology for
user-generated content, and improving the mobile experience. This
research, together with industry and academic collaborations, will help
enable a rich new world of experiences and interactions the significance
of which we can only begin to comprehend.
References [1] Haptic technology refers to technologies that interface the user
through the sense of touch by enabling forces, vibrations and/or motions
to be perceived by the user. Such mechanical stimulation can assist in the
creation, control and experience of virtual objects.
[2] “Interview: Strategy Analytics' Barry Gilbert: 137M Virtual Worlds
Users Now; 1B by 2017,” Virtual Worlds News, June 3, 2008.
[3] Ibid.
[4] Ibid.
Information in this document is provided in connection with Intel® products. No
license, express or implied, by estoppel or otherwise, to any intellectual property
rights is granted by this document. Except as provided in intel's terms and
conditions of sale for such products, intel assumes no liability whatsoever, and
intel disclaims any express or implied warranty, relating to sale and/or use of intel
products including liability or warranties relating to fitness for a particular purpose,
merchantability, or infringement of any patent, copyright or other intellectual
property right. Intel products are not intended for use in medical, life saving, life
sustaining applications.
Intel may make changes to specifications and product descriptions at any time,
without notice.
Copyright © Intel Corporation 2008
* Other names and brands may be claimed as the property of others.
Recommended
