Augmented Reality and Sixth Human Sense

Augmented Reality and Sixth Human Sense

Abstract: Augmented reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality. Sixth Sense is based on the concept of anywhere computing. It essentially allows you to access information from the Internet and project that information on any surface, including your hand. It reads human gestures and turns any surface into an interactive screen. It senses objects around you, reads your hand gestures and immediately displays relevant information. This paper is a primary research on the understanding of the technology, finding its current applications in all possible domains and then suggesting future of these technologies. The Paper will conclude by focusing on the human and ethical aspect for the selected topic, before giving a future scope of work that can be carried out using this primary research work.Introduction to Sixth Sense and Augmented Reality

Sixth Sense [1]

Sixth Sense: 'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information.We've evolved over millions of years to sense the world around us. When we encounter something, someone or some place, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world.

Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. ‘SixthSense’ frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. 

Augmented Reality [2]

Augmented Reality: Video games have been entertaining us for nearly 30 years, ever since Pong was introduced to arcades in the early 1970s. Computer graphics have become much more sophisticated since then, and game graphics are pushing the barriers of photorealism. Now, researchers and engineers are pulling graphics out of your television screen or computer display and integrating them into real-world environments. This new technology, called augmented reality, blurs the line between what's real and what's computer-

2

generated by enhancing what we see, hear, feel and smell.

Everyone from tourists, to soldiers, to someone looking for the closest subway stop can now benefit from the ability to place computer-generated graphics in their field of vision. audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. Similar devices and applications already exist, particularly on Smartphone like the iPhone.

Augmented Reality Technology [3]

The objects of the real world observed by the user (world reference frame) are sent to the computer, which generates the virtual objects (in virtual reference frame). The real and the virtual objects (scenes) are mixed and rendered in such a way to give the user the perception of observing one integrated scene. Depending on what kind of equipment is used, different approaches can be followed, which result in diverse type of systems architectures. However, four very general components (and corresponding technologies) are always present in an AR system, i.e. displays, tracking systems, devices for interaction and graphics systems (computer).

How it Works?1. The scene is viewed by an imaging

device, which in this case is depicted as a video camera.

2. The camera performs a perspective projection of the 3D world onto a 2D image plane.

3.  The generation of the virtual image is done with a standard computer graphics system.

4. The virtual objects are modeled in an object reference frame.

5. The graphics system requires information about the imaging of the real scene so that it can correctly render these objects.

6. This image is then merged with the image of the real scene to form the augmented reality image.

Figure 1: Technology of Augmented Reality

Technology of Sixth Sense [4]

Figure 2: How it works

The sixth sense technology uses different technologies like gesture recognition, image processing, etc. At present the commercial product isn’t launched but the prototype is prepared. The sixth sense prototype is made using very common and easily available equipments like pocket projector, a mirror, mobile components, color markers and a camera.

The projector projects visual images on a surface. This surface can be wall, table, book or even your hand. Thus, the entire world is available on your screen now. When user moves their hands to form different movements with colored markers on the finger

3

tips, the camera captures these movements. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. Recognition is made using computer vision technique. These markers act as visual tracking fiducials. The software program processes this video stream data and interprets the movements into gestures. The gestures are different from one another and are assigned some commands. These gestures can act as input to application which is projected by the projector. Since, the projector is aligned downwards for compactness; therefore images would be formed at the user’s feet if mirror wasn’t used. The mirror reflects the image formed by the projector to front. The entire hardware is fabricated in the form of a pendent.

Augmented Reality

Applications as of 2011 [5]

Advertising: Usage of AR to promote products via interactive AR applications is becoming common now. For example Nissan (2008 LA Auto Show), Best Buy (2009), Volvo (2011, S60 Campaign) and others used webcam based AR to connect 3D models with printed materials. There are numerous examples of connecting mobile AR to outdoor advertising.

Task support: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. AR can include images of hidden objects, which can be particularly effective for medical diagnosis or surgery.

Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound and confocal microscopy probes or open NMR devices. AR can enhance viewing a fetus inside a mother's womb.

Navigation: AR can augment the effectiveness of navigation devices. For example, building navigation can be enhanced to aid in maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can provide navigation and traffic information. Head-up displays are currently used in fighter jets. These systems include full interactivity, including gaze tracking. They are also currently used by car manufacturers as BMW, Corvette and GM in car windshields, mostly to display meter information, traffic information.

Industrial: AR can be used to compare digital mock-ups with physical mock-ups for efficiently finding discrepancies between them. It can safeguard digital data together with existing real prototypes, and thus reduce the number of real prototypes and improve the quality of the final product.

Military and emergency services: Wearable AR can provide information such as instructions, maps, enemy locations, and fire cells.

Art: AR can help create art in real time integrating reality such as painting, drawing and modelling. AR art technology has helped disabled individuals to continue pursuing their passion.

Architecture: AR can simulate planned construction projects.

Sightseeing: Guides can include labels or text related to the objects/places visited. With AR, users can rebuild ruins, buildings, or even landscapes as they formerly existed.

Time: Historical events such as battle re-enactments can be augmented onto current landscapes.

Collaboration: AR can help facilitate collaboration among distributed team members via conferences with real and virtual participants.

4

Entertainment and education: AR can create virtual objects in museums and exhibitions, theme park attractions, games and books. AR can also be used to make games that can be played outdoors. 

Commerce: AR can be used to display certain products at another way. For example, the lego-boxes in the lego-store at Schaumburg are fitted with a QR code which show the product when it's assembled. AR can also be used as an aid in picking clothing.

Performance: AR can enhance concert and theatre performances. For example, artists can allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.

Translation: AR systems can provide dynamic subtitles in the user's language. [8, 9, 10, 11, 12]

AR Tools

Open source software [13, 14, 15]

Argon, augmented reality browser by Georgia Tech's GVU Centre that uses a mix of KML and HTML/JavaScript/CSS to allow developing AR applications; any web content (with appropriate meta-data and properly formatted) can be converted into AR content; as of November 2011, available for iPhone only

ARToolKit , an open source (dual-license: GPL, commercial) C-library to create augmented reality applications; was ported to many different languages and platforms like Android, Flash or Silverlight; very widely used in augmented reality related projects

ArUco, a minimal library for augmented reality applications based on OpenCv; licenses: BSD, Linux, Windows

Goblin XNA, a platform for researching 3D user interfaces, including mobile augmented reality and virtual reality, emphasizing games; written in C#, based

on Microsoft XNA Game Studio 4.0, BSD license

GRATF, open-source (GPLv3) project, which includes C# library for detection, recognition and 3D pose estimation of optical glyphs. The project includes application, which does 2D and 3D augmented reality

mix Augmented Reality Engine (mixare), open-source (GPLv3) augmented reality engine for Android and iPhone; works as an autonomous application and for developing other implementations

Open Mobile Augmented Reality (OpenMAR), component framework for the Symbian platform.

PTAM, non-commercial use only

Free closed source

Qualcomm AR SDK (QCAR) is a free AR solution for detection and tracking of reference images and markers using natural feature detection that is available for both Android and iOS.

AR Content Management Systems

buildAR.com, a content platform for geolocation and natural feature tracking based augmented reality applications in a web browser.

Hoppala Augmentation, a content platform for geolocation based mobile augmented reality applications.

Sixth Sense

Applications as of 2011 [6]

Make a call: You can use the Sixth Sense to project a keypad onto your hand, and then use that virtual keypad to make a call.

Call up a map: With the map application we can call up the map of our choice and then use thumbs and index fingers to navigate the map

5

Check the time: Draw a circle on your wrist to get a virtual watch that gives you the correct time

Create multimedia reading experiences: Sixth Sense can be programmed to project related videos onto newspaper articles you are reading.

Drawing applications: The drawing application lets the user draw on any surface by tracking the finger tip movements of the user’s index finger.

Zooming Features: The user can zoom in or zoom out using intuitive hand movements.

Weather information: We can get weather information directly from the newspaper.

Fig 3: Sixth Sense Applications [7]

Technologies that uses Sixth sense as Platform [7]

Radio Frequency IdentificationSixthSense is a platform for Radio Frequency Identification based enterprise intelligence that combines Radio Frequency Identification events with information from other enterprise systems and sensors to automatically make inferences about people, objects, workspaces, and their interactions.Radio Frequency Identification is basically an electronic tagging technology that allows the detection and tracking of tags and consequently the objects that they are affixed to. This ability to do remote detection and tracking coupled with the low cost of passive tags has led to the widespread adoption of RFID in supply chains worldwide.

Advantages of Augmented Reality [16]

The New Sphere: As a result of creating Augmented Reality a new sphere has formed known as ‘The Virtual Sphere’. New campaigns are beginning to include Augmented Reality as part of their communications strategies. A recent campaign which has incorporated this is The Gorillaz for their new album, “Plastic Beach”. They have promoted it in the latest edition of NME Magazine which comes complete with an A5 booklet filled with Gorillaz information and inside is an Augmented Reality marker, which when held up to a webcam the user is presented with a 3D “Plastic Beach” which may be navigated around. It could be argued that interactivity of this nature works well in PR campaigns and Augmented Reality is the next generation of interactivity with your consumers. According to Howard Rheingold, a critic & writer in the communications industry, interactivity can create a false sense of power which consumers strive for and this makes it

6

worthwhile to incorporate it in PR strategies.

Personal Experience: Once mixed with your social media applications, it can become a personal experience with offers/coupons from shops you like and places you enjoy going to – hence a marketers’ dream. This could prove to be highly useful to PROs when trying to reach their target market as the displays are only perceived by a single user, raising the possibility of creating rich, personal experiences for all occupants of a shared space. 

Making Mobile Search Invisible: Augmented Reality is set to revolutionize the mobile user experience as did gesture and touch (multi-modal interaction) in mobile phones. This will redefine the mobile user experience for the next generation making mobile search invisible and reduce search effort for users.

Improves Mobility: Augmented Reality improves mobile usability by acting as the interface itself, requiring little interaction (this Interaction Design technique is known as Direct Manipulation). Imagine turning on your phone or pressing a button where the space, people, objects around you are “sensed” by your mobile device- giving you location based or context sensitive information on the fly.

Disadvantages of Augmented Reality

Technology Performance Constraints: Current performance levels (speed) on today’s [2009] iPhone or similar touch devices like the Google G1 will take a few generations to make Augmented Reality feasible as a general interface technique accessible to the general public.

Narrow or Obscure Content: Content may obscure or narrow a user’s interests or tastes. For example, knowing where McDonald’s or Starbucks is in Paris or

Rome might not interest users as much as “off the beaten track information” that you might seek out in travel experiences.

Privacy Control: Privacy control will become a bigger issue than with today’s information saturation levels. Walking up to a stranger or a group of people might reveal status, thoughts (Tweets), or other information that usually comes with an introduction, might cause unwarranted breaches of privacy.

Will it ever take off? It is all very well and good if PROs start to include this media platform in their campaign tactics however it is not definite that it will take off in the UK like it has in Japan.

Advantages of Sixth Sense [17]

SixthSense has a user guide that can be used very simply. Some of important gestures that people can learn easily are like taking a picture or making a phone call or trying to know the time. In order to snap a picture, the user has to make a rectangle in the air with their fingers as if they are snapping a picture. The picture will be taken and it will be placed in the memory card of the device. Whenever the user wants to edit the picture, they can always access the picture. A person can resize and edit it by making few finger motions on the projected screen on any hard surface.

A phone call, using the SixthSense, can be made so easily. All a person needs to do is to touch digital keys on one’s palm or whichever surface he decides to project. If the user wants to check the time, all they have to do is gesture a circle on the user’s wrist. The SixthSense will recognize the movement and project an analog clock on to the wrist

SixthSense can scan a book you want to read and it can present you detailed reviews, ratings and even summaries of that same book. It can recognize a newspaper, a magazine. While Pranav

7

Mistry presented his SixthSense, he did an experiment using a newspaper. While he scanned a picture from the news paper, the SixthSense pulled up a video online about the same news that is published in the newspaper. This device comes in handy and can used very easily by all the people because it uses hand gestures which everyone actually uses in everyday life.

SixthSense can also be a great substitute for a laptop or a computer. Checking email, web browsing, other programs like Microsoft word, adobe are also included in this device. For instance, when a user wants to check his email, he needs switch on the projector and write “@” with his fingers. The camera quickly recognizes and presents the user with different email options he can use.

Disadvantages of Sixth Sense [18]

Security & Privacy Concerns: For example when a person is taking a picture using the SixthSense, he just snaps with his fingers which have little marker caps on them. People don’t want to be captured on a camera of a random person. One can never tell when SixthSense is taking a picture, because it is not a big object and nothing is being help in your hand in order to take a picture like a camera or a phone. This is invading other’s privacy which can be a huge problem.

Health Issues: Some of the health issues are regarding SixthSense’s projection technology. When the device is projecting on a hard surface people around him can see the projection since it is very detailed. Projection is better in the night time and dark areas rather than mornings and bright areas. This is an issue because the vision of the user can be damaged when using this instrument..

Pricing: Concerns about the pricing of this device are also rising among the people. Mistry announced that the present device is costing about $350 for each, on his

website. This information was updated in 2009, but ever since then there is not news about this device. Mistry is working on many other technologies and inventions, but the world doesn’t know whether the work on this device is stopped (Doherty, 2009).

Augmented Reality-Ways in which it will change your world:

This new technology will have a profound impact on businesses. Like social media, augmented reality is a fairly new technology that is still being developed into tools that add some use or productivity to our lives. At this early stage, it is often found in games, but businesses are slowly adopting the technology and experimenting with it. The most basic definition is that it adds something to improve or heightens reality.

Augmented Reality in 2015: Augmented Reality Urban Information System:[19] When managing a city there are a lot of invisible or hidden facilities (such as gas pipes, water pipes, sewer pipes, and communication cables) that need to be taken into consideration. In order to manage these facilities, Urban Information Systems (UIS) have been widely used to date. Augmented reality and mobile communication technology is a perfect match for bringing UIS into the field. Severe gas pipe explosions and the cutting of communication cables during excavation works are still far too frequent, even in major world cities. These accidents are caused by a lack of information, as well as the invisibility of the underground facilities. This problem could be solved by a fusion of UIS data and augmented reality. While a worker is excavating a road, 3-D UIS data could be projected through an augmented reality system that would enable a worker see the location of such underground facilities.

8

In short, sub-metre accuracy is required for augmented reality applications to become ‘useful’ in a professional stance. This accuracy will be reached via image recognition technologies in the next five years which combined with a lightweight data transfer standards will enable a fully fledged augmented.

Augmented Reality in 2020: The Stark HUD 2020 Augmented Reality goggles: [20] The goggles are visually attractive and appear to be 'socially acceptable' to wear. Information is context sensitive. The idea of context-aware computing has been around for quite some time, and there are patent filings on it dating at least 10 years ago. However, the availability of location-based services, and development in applications that know more and more about the user (Apple's Siri is a great new example), make the goggles useful much more than just being used as a media viewer.The field of view appears wide, and resolution is good enough to read.There is plenty of natural interaction with the device.

Sixth Sense in 2025: [21] This invention will become a revolution when we realize that Sixth Sense Technology will affect the way we sense the real world around us. Between 2011 and 2018, Microsoft, Sony, and Nintendo research labs will work on the large diffusion of this technology in industrial societies. In 2025, people will live with a sixth sense providing a bridge between the physical world and the digital world. We will design what we call mixworlds by adding objects and/or information of the digital world into the real world. Each mixworld is personal and depend on the objects and/or information we add in a specific real world situation. Parents design mixworlds for their children depending on the abilities/capacities they expect them to

develop. They also share these mixworlds with other parents in the community. In 2025, children are raised through these mixworlds.

In education, the first issues surrounding this new technology will arise when students will bring their Sixth Sense in to the classroom. They will add digital objects into the real world to play during the class. They answered teachers' knowledge and procedural questions by browsing their digital world.

The governmental response is estimated to be global because Sixth Sense will promote mutual understanding among people by solving language problems. Translation will no longer be necessary because as you speak in your mother language, Sixth Sense translates into the language you choose in real time. People who were deaf and hard of hearing quickly embraced Sixth Sense, speaking by tapping on their virtual keyboard. The government's global response initiated in 2015 may be called, "Life with Sixth Sense Technology." This initiative consisted of two parts: to increase the diffusion of Sixth Sense Technology around the world and to examine how Sixth Sense Technology affected human beings.

Ethics in Augmented Reality and Sixth Sense World:[22]

While most improvements in the field of augmented reality can be seen as positive additions to our resources, some developments may arouse the curiosity of critics and leave us with a few questions.

An important question regarding all technological advancements geared at aiding human beings is how dependent people become of these. While an AR system might provide ground breaking new possibilities, the more it simplifies

9

certain tasks, the less a person needs to know. Pushed to the extreme, this consideration can translate into loss of knowledge and understanding. The phenomenon is nothing new and has been observed in other fields. Arguably, the gain may outbalance the loss, yet it is still important to remain cautious. Being overly dependent on a technology also leads us to question its reliability.

Another concern that critics point out is a decrease in personal privacy and an undermining of civility. Imagine a widespread wearable system with advanced face recognition software, pulling identity information on every passer-by from distant stored databases. This could contain delicate information on a person’s personal life and commentaries from other users.

Conclusion: Our Findings

Both the technology Sixth Sense and AR are still in their nascent phase. Applications are still very basic and constrained to domains like entertainment and fun. The real importance can be realized only when the technologies are taken seriously and big players show an interest in investing into these domains. A real breakthrough will come only when such technologies are used more and more for health, education and defence purpose. The biggest challenge is: will this be another technology amongst the already existing lot or it will be the next generation technology- The technology with a difference. This only time can tell. We as technology and management students can take a follow up and do further research to add to this domain. Meanwhile our key learning has been “knowing not only the current trends but also comment upon the future trends in the technology and business domain”. This research work can be a precursor report for many Indian

companies like Infosys, TCS, HCL to strategize their future work upon these technologies.

References

[1] Pranav Mistry. ABOUT. [Online]. http://www.pranavmistry.com/projects/sixthsense/, [19 Jan 2012].

[2] HowStuffWorks. HowStuffWorks. [Online]. http://www.howstuffworks.com/augmented-reality.htm, [19 Jan 2012].

[3] Gdmc Group. “Technology of Augmented Reality”. Internet: http://www.gdmc.nl/publications/reports/GISt17.pdf. [19 Jan,2012]

[4] Pranav Mistry. “Technology of Sixth Sense” Internet: http://www.pranavmistry.com/projects/sixthsense/. [16 Jan,2012]

[5]Wikipedia. “Augmented Reality” Internet: http://en.wikipedia.org/wiki/Augmented_reality#Applications_as_of_2011, 17 Jan 2012 [19 Jan 2012].

[6] Michelle Robart. “Gadgets -MIT Researchers Unveil Computing Device that Turns Any Surface into a Touchscreen” Internet: http://gadgets.tmcnet.com/topics/gadgets/articles/50131-mit-researchers-unveil-computing-device-that-turns-any.htm, 05 Feb 2009 [19 Jan 2012].

[7] Venkat Padmanabhan. “Sixth Sense” Internet: http://research.microsoft.com/en-us/projects/sixthsense/, June 2008 [19 Jan 2012].

Print References

Books

[8] Woodrow Barfield, and Thomas Caudell, eds. Fundamentals of Wearable Computers

10

and Augmented Reality. Mahwah, NJ: Lawrence Erlbaum, 2001. ISBN 0-8058-2901-6.

[9] Oliver Bimber and Ramesh Raskar. Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters, 2005. ISBN 1-56881-230-2.

[10] Michael Haller, Mark Billinghurst and Bruce H. Thomas. Emerging Technologies of Augmented Reality: Interfaces and Design. Idea Group Publishing, 2006. ISBN 1-59904-066-2, publisher listing

[11] Rolf R. Hainich. "The end of Hardware: A Novel Approach to Augmented Reality" 2nd ed.: Booksurge, 2006. ISBN 1-4196-5218-4. 3rd ed. ("Augmented Reality and Beyond"): Booksurge, 2009, ISBN 1-4392-3602-X.

[12] Stephen Cawood and Mark Fiala. Augmented Reality: A Practical Guide, 2008, ISBN 1-934356-03-4

Conferences

[13] 2nd International Workshop on Augmented Reality (IWAR'99), San Francisco, United States, October 1999.

[14] 2nd International Symposium on Mixed Reality (ISMR'01), Yokohama, Japan, March 2001.

[15] 10th International Symposium on Mixed and Augmented Reality (ISMAR 2011), Basel, Switzerland October 2011

Online

[16] ProgrammableWeb. “Augmented Reality News & Treands” Internet: http://www.scoop.it/t/augmented-reality-news-and-trends/p/108302695/advantages-and-

disadvantages-of-augmented-reality-behind-the-spin, Jan. 18, 2012 [Jan. 19, 2012].

[17] Sneha Allu Blog. “Course: The Sixth Sense”. Internet: http://sallu-sixthsense.blogspot.com/2011/10/sixth-sense-advantages-and.html, Oct. 11, 2011 [Jan. 15, 2012].

[18]eLearning Blog. “Advantages & Drawbacks of Using Augmented Reality”. Internet: http://www.dontwasteyourtime.co.uk/technology/advantages-and-drawbacks-of-using-augmented-reality-augmentedreality/, Mar. 30, 2010 [Jan. 12, 2012].

[19]Business from technology. “Mobile Augmented Reality”. Internet. http://mediafactory.aalto.fi/wp-content/uploads/2011/09/Mobile_AR_State-of-the-Art.pdf, [Jan 20,2012]

[20]Discover Magazine. “Your Augmneted Reality Real Life”. Internet. http://blogs.discovermagazine.com/discoblog/2010/04/17/your-augmented-reality-life-coming-soon-in-2020/, [Jan 20,2012]

[21] Pranav Mistry. “ABOUT”. [Online]. http://www.pranavmistry.com/projects/sixthsense/, [19 Jan 2012].

[22] Future Conscience. “Augmented Reality, the good, the bad and the ugly”. Internet. http://www.futureconscience.com/augmented-reality-good-bad-ugly-part-two-the-bad/ . [19 jan,2012]

11