Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 1

CONSUMER AR APP TRENDS [SYNOPSIS]

By Floris van Eck and Hans Hartman

Contributor: Vincent Leeuw

Updated March 2018

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 2

ABOUT THE AUTHORS

About Floris van Eck

Floris van Eck is the founder of FutureOutpost, a firm that drives innovation by providing strategic analysis & insights around emerging and future technologies, with a focus on virtual reality, augmented reality and artificial intelligence.

About Hans Hartman

Hans Hartman is chair of Visual 1st (formerly Mobile Photo Connect), a global conference focused on promoting innovation and partnerships in the photo and video ecosystem, and founder of Suite 48 Analytics, a market research firm for the photo and video industry.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 3

TABLE OF CONTENTS OF PLATINUM VERSION

ConsumerARAppTrends........................................................................................................................................1

AbouttheAuthors........................................................................................................................................................2

TableofContents..........................................................................................................................................................3

ExecutiveSummary.....................................................................................................................................................6

1.AR–TheConcept.....................................................................................................................................................9

2.AR–TheFeatures................................................................................................................................................10

1.MotionTracking...........................................................................................................................................10

2.Recognition....................................................................................................................................................10

3.ContentAnchoring......................................................................................................................................11

4.Lightestimation...........................................................................................................................................11

5.Userinput........................................................................................................................................................12

3.AR–TheHardware.............................................................................................................................................13

FormFactors...........................................................................................................................................................13

1.HandheldAR......................................................................................................................................................14

2.SpatialAR.............................................................................................................................................................14

3.WearableAR..................................................................................................................................................15

4.SyntheticAR.......................................................................................................................................................18

FormFactorTimeline.........................................................................................................................................19

4.AR–TheExperienceTypes.............................................................................................................................20

FilterExperience..............................................................................................................................................20

OverlayExperience.........................................................................................................................................21

AssistantOverlayExperience.....................................................................................................................22

LensExperience................................................................................................................................................23

SimulationorGameExperience................................................................................................................24

ARExperiences–InSum..............................................................................................................................26

5.AR–TheApplications........................................................................................................................................27

Filterapps............................................................................................................................................................27

DashboardOverlayApps..............................................................................................................................29

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 4

LocationOverlayapps...................................................................................................................................32

IntelligentOverlayapps................................................................................................................................34

FaceMarkupapps............................................................................................................................................39

WorldMarkupapps........................................................................................................................................41

VirtualTourapps.............................................................................................................................................45

VirtualTutorapps............................................................................................................................................46

VirtualShowroomapp...................................................................................................................................48

MagicMirrorapps............................................................................................................................................50

Bring-to-Lifeapps............................................................................................................................................54

AugmentedVirtualityapps..........................................................................................................................61

HoloObjectapps...............................................................................................................................................63

HoloControllerapps.......................................................................................................................................69

HoloSpaceapps................................................................................................................................................72

VirtualWorldsandGamingapps..............................................................................................................76

Holocamapps.....................................................................................................................................................77

Location-basedARGameapps..................................................................................................................80

VirtualPetsapps...............................................................................................................................................83

TabletopARGameapps................................................................................................................................85

ARApplications–InSum.............................................................................................................................88

6.AR–TheDevelopmentToolPlatforms......................................................................................................91

OS-basedARToolPlatforms...........................................................................................................................92

ARKit1.5(Apple).............................................................................................................................................92

ARCore1.0(Google).......................................................................................................................................95

MixedReality(Microsoft)............................................................................................................................97

OtherARToolPlatforms...................................................................................................................................99

CameraEffectsPlatform(Facebook)......................................................................................................99

Sumerian(Amazon)......................................................................................................................................100

LensStudio(SnapInc.)................................................................................................................................101

ARToolKit6(DAQRI)....................................................................................................................................102

Vuforia7(Vuforia)........................................................................................................................................102

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 5

Blippbuilder(Blippar).................................................................................................................................102

GameEngines.......................................................................................................................................................102

Unity(Unity).....................................................................................................................................................103

UnrealEngine(EpicGames).....................................................................................................................105

DevelopmentToolPlatforms–InSum.....................................................................................................107

7.AR–RecommendationsforDevelopingARSolutions.....................................................................110

ChoosebetweenARorVR..............................................................................................................................110

ChoosetherightARexperiencetype........................................................................................................110

ChoosetherightARdevelopmentplatform..........................................................................................111

Think3D..................................................................................................................................................................111

DesignforARhandheld;planforARwearable....................................................................................112

Makecontentvisible.........................................................................................................................................112

StandardizeARaccess......................................................................................................................................113

TakescrapbookingtoAR................................................................................................................................113

Thinklayers...........................................................................................................................................................114

Turncustomersintoplayers.........................................................................................................................114

Convertcustomersonthefly........................................................................................................................115

Don’tforgettheaudio.......................................................................................................................................116

8.AR–RecommendationsforImagingEcosystemVendors..............................................................117

CloudStorageProviders..................................................................................................................................117

PhotoandVideoSharingPlatforms...........................................................................................................117

CameraVendors..................................................................................................................................................118

PhotoandVideoAppDevelopers...............................................................................................................118

PhotoPrintProductVendors........................................................................................................................119

9.Index.........................................................................................................................................................................120

AppendixA.ARApps.............................................................................................................................................122

AppendixB.OtherARSolutions.......................................................................................................................125

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 6

EXECUTIVE SUMMARY

Remember when “reading” only meant reading in print? We all still do this at times and speaking personally, we often enjoy—as many people do—taking a break from staring at my screen. But in many cases, reading a digital version of an article, along with embedded links to additional materials, other perspectives, live commenting chats, extra photos, or instructional videos that clarify the main points, can be much more informative, easier to absorb, and more enjoyable. For digital natives, reading articles on screen is the default.

AR will bring that same quantum change to how we’re absorbing the world around us. With AR, we will continue to perceive the world as we always have, but we will also experience useful or entertaining augmented content. AR is a live experience that incorporates our physical, real-world surroundings, as well as computer-generated sensory input (such as video, sound, graphics or haptic feedback) to augment how we perceive our physical world.

To provide that augmented experience of the world around us, high-fidelity AR provides ways to track motion, understand our environment, estimate light, anchor the content that we’re seeing, and provide ways for us to interact with our augmented world.

While most consumer AR applications are currently offered on smartphones that superimpose augmented content to what the smartphone cameras show us of the world around us (Handheld AR), other AR hardware solutions are being developed that offer us hands-free and more persistent augmented reality experiences, such as Spatial AR (projecting images onto the user’s physical environment), Wearable AR (glasses, noise-canceling headphones, visor-equipped helmets, contact lenses, to name a few), and even Synthetic AR solutions that will offer a brain/computer interface, allowing the brain to control external devices.

AR applications provide five different types of AR experiences that augment how we perceive the real world: they can filter what we see or hear (think live Instagram filters or noise-canceling headphones); overlay contextually relevant information (show the altitude of the mountain ranges we are viewing), overlay assisting information (show pricing and ordering options for the shoes we’re checking out), add a Lens that provides a viewpoint into another world by blending virtual content with our real-world surroundings (show virtual rabbits on the lawn in front of us), and provide ways for us to play Games or simulate different outcomes (try catching virtual monsters at different locations).

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 7

While consumer AR hardware is still in its early stages, AR applications are already blossoming and being developed for many different use cases and user types. In all, we cover 89 apps, divided into the following 20 categories and generating these types of AR experiences, which we’ve divided into the following 20 categories taking into consideration the types of AR experiences they enable:

AR Apps Categories AR Experience Types

Augmented Virtuality Simulation

Bring-to-Life Overlay, Lens, Simulation/Games

Dashboard Overlay Overlay

Face Markup Overlay

Filter Filter

Holo Controller Lens, Simulation/Games

Holo Object Lens, Simulation/Games

Holo Space Lens, Simulation/Games

Holocam Lens

Intelligent Overlay Overlay

Location Overlay Overlay

Location-based AR Game Simulation/Games

Magic Mirror Overlay

Social Space Overlay, Lens

Tabletop AR Game Simulation/Games

Virtual Tour Overlay, Lens

Virtual Tutor Overlay, Lens, Simulation/Games

Virtual Pet Simulation/Games

Virtual Showroom Lens, Simulation/Games

World Markup Overlay, Lens

AR developers don’t have to do it all by themselves—there already exists a fledgling but rapidly growing ecosystem of AR API and SDK development toolkit providers, with new entrants, including the Big 5 Tech companies (Amazon, Apple, Facebook, Google, Microsoft) putting their considerable weight behind further advancing AR toolkits.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 8

Some AR development platforms are specific to their vendors’ OS or social media platforms, which enables deeper integration with these platforms, but could also lead to balkanization by forcing AR developers to use multiple development environments.

However, we are pleased to see a trend towards these environments starting to leverage each other. For instance, Apple’s ARKit and Google’s ARCore are supported by multiple other AR development platforms; and gaming engines Unity and Unreal are supported by many other development platforms as well, including ARKit and ARCore.

With easier as well as more versatile AR toolkits coming to market, we expect to see an explosion of AR apps being developed with different levels of technical sophistication. The 89 apps covered in this study are just the beginning.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 9

1. AR – THE CONCEPT

AR augments how we perceive the world around us. It filters our physical world or adds virtual content to it. Revolutionary as it may sound, the concept is one we’re already familiar with through many analog predecessors: take sunglasses, for instance, which block certain light wavelengths to improve how we view the world in sunny conditions – not fundamentally different from AR Filter apps, which we’ll describe in chapter 5.

So what is AR? We see AR as a live experience that incorporates our physical, real-world surroundings, as well computer-generated sensory input (such as video, sound, graphics or haptic feedback) that augments how we perceive our physical world.

Let’s dive deeper into the main ingredients of AR1:

Live: Looking through a smartphone camera and seeing a virtual pet projected on the living room floor is AR; looking at a photo of the living room floor with a pet Photoshopped in it, is not AR.

Experience: AR is how we perceive the world with any of our 5 human senses. While the majority of our examples involve visuals, they could also involve augmenting other senses, such as hearing when we use noise-canceling headphones that affect how we hear music.

Our physical, real-world environment: Unlike VR, AR ultimately augments the physical world around us. VR headsets shield us from the world around us so that we can fully immerse in a virtual world; AR see-through headsets let us still perceive see the world around us, while this is augmented in some fashion.

Computer-generated sensory input: We exclude the world of analog devices that serve to augment how we experience our world. In chapter 3, we’ll describe the computer-generated hardware (from handheld smartphones to smart lenses) through which the AR apps described in chapter 5 augment our reality.

Responsive to real-world inputs: Computer-generated virtual content is not just static content placed into the user’s point of view or hearing range; it is responsive content that adapts to how the user interacts with the physical world. You turn on the kitchen light, and your virtual pet will be lit up; you walk into a noisier environment, and your noise-canceling headset will filter out more noise.

1 Microsoft uses the term Mixed Reality (MR), which many observers find hard to distinguish from AR, as MR in essence also refers to mixing the physical with virtual worlds.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 10

2. AR – THE FEATURES

The five core elements present in every high-fidelity AR experience are:

Motion Tracking

Recognition (Environmental Understanding)

Content Anchoring (Positioning)

Light Estimation

User Input

1. Motion Tracking

Motion Tracking in action: Identifying feature points in the scene. (Image Source: Google)

AR systems know where the sensor device is in relation to the physical world. If the sensor device moves, it recalibrates its new position relative to the objects it has identified. So if you walk farther away from an object, then it will look smaller or will sound less loud.

2. Recognition Key to AR systems is understanding that they recognize our environmental context—our surroundings. An AR system is constantly searching for feature points and planes in the scene. When the device camera detects a flat surface—such as a table, wall, or floor—it is converted into a plane that is made available for the AR experience. This way, a chair or hologram is set on top of the floor instead of having it float around in space. The better an

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 11

AR system understands its environment, the more realistic the experience it can deliver.

3. Content Anchoring Almost all AR content will use some form of content anchoring. Content anchoring ensures that the virtual content is “fixed” to a real-world object or scene, usually a surface or plane, allowing it to maintain its position over time even if the user is moving around. The user can walk around virtual objects and inspect them from all angles, the way they would with physical objects. They can even come back weeks later and the virtual objects are still in place.

For example:

When previewing an Ikea dining room table in my living room, it is anchored to my floor

When previewing my new virtual makeup, I can see the effect from different camera angles

Content Anchoring ensures content remains stable, so the crocodile sits on the table and doesn’t float in mid-air. (Image Source: Google)

4. Light estimation The sensors in an AR device (cameras, ambient light sensors) enable the AR to record the light intensity and light quality in its surroundings. This information is then used to render virtual objects under the same simulated lightning conditions as their real-world surroundings. Note that real-time, realistic rendering is a computing intensive process that can now be done on the smartphone instead of requiring pre-rendering through cloud-based services.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 12

5. User input AR systems enable users to interact with their augmented world, such as by touching the screen, gesturing or making eye movements.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 13

3. AR – THE HARDWARE

Model wearing Google Glass in public. (Image Source: Google)

Ultimately, AR applications require hardware to let us augment our perception of the world. AR hardware solutions come in many shapes and forms. As this report is primarily about AR applications, we will only provide a high-level overview here to get a sense of what types of AR hardware products are on the market and what we could expect to come to market in the coming years.

Form Factors AR hardware could be Handheld, Wearable, Spatial or Synthetic. Within these categories, a number of interfaces can be distinguished through which consumers can access AR content.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 14

Category Characteristic Persistence Device-Interface

Handheld AR Handheld Low Smartphone, Tablet, Portable Game Console

Spatial AR Hands-free Medium Projector, Screen

Wearable AR Hands-free High Smart Glasses, Visors (Helmet) or Contact Lenses

Synthetic AR Hands-free High Brain-Computer Interface

1. Handheld AR Handheld mobile AR is how most people currently enjoy AR. You aim your phone’s front camera at yourself, and then you can apply virtual makeup to envision how you would look if you’d apply the real-world equivalent of the same makeup. The good news? You don’t need to purchase an AR device—you already own it. The bad news? Your AR experience is not hands-free and takes a toll on your phone’s limited battery life, limiting the potential for a persistent, always-on enhancement of reality.

Handheld AR is not only offed on smartphones. For instance, Portable Game Consoles, such as the PlayStation Vita and Nintendo 3DS, also provide AR solutions. In the future, we expect to see other types of photo or video cameras as well, and even monitors with see-through screens.

2. Spatial AR Spatial AR projects images onto the user’s physical environment using video projectors, holograms, or other optical elements, coupled with interactive rendering algorithms and calibration techniques. How is this different from a slide projector displaying images onto a wall? With Spatial AR the projected virtual objects are automatically anchored to real-world objects through recognition and therefore responsive to real-world (user) inputs. In other words, the observer might walk around the projected images and see it from all angles; they might also give voice or other commands to interact with these virtual objects.

A key benefit of Spatial AR is the fact that Spatial AR technology is part of the environment and the user does not have to wear a device to enjoy or participate in the experience. On the downside, this means that a space or room needs to be prepared beforehand, which is not always feasible and could be cost-prohibitive.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 15

A “shopping window” with spatially augmented products via a context-aware computer system (click here to see animation). AR without the need for a headset is the key unique selling factor of Spatial AR. (Image Source: Lightform)

3. Wearable AR

Steve Mann’s evolution over time. (Image Source: Otherzine)

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 16

Bose QC35 II noise-canceling headphones now come with Google Assistant [Assistant Overlay] integration. (Image Source: Bose)

The DAQRI Smart Helmet is fully self-contained, includes an Intel RealSense camera and is intended for work sites. Image Source: DAQRI)

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 17

University of Washington professor Babak Amir Parviz and his students are working on solar-powered augmented lenses. The third (freaky) image shows a rabbit testing a lens to determine the impact on the eye. (Image Source: University of Washington)

Wearable AR, along the lines of Google Glass, is what many of us think of when discussing AR. Wearable offers many of the smartphone-based AR benefits while fixing its most serious shortcomings:

Wearable AR is hands-free. This is a major benefit, especially for enterprise, educational, industrial or medical applications.

Wearable AR is persistent. Although our phones are always with us, they are not persistent. With Wearable AR, augmented content will always be in our field of vision or line of hearing.

Still, the current iterations of Wearable AR have faced serious consumer objections—in their current iterations, they are considered too intrusive for daily use.

But these adoption hurdles could very well be lowered when Wearable AR lenses come to the consumer market. Scientists at the University of Washington have already developed a contact lens containing one built-in LED, powered wirelessly by radio

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 18

frequency waves, and companies like Samsung have already filed patents for their envisioned smart contact lens. It’s not a matter of if AR contacts will come to market; it is a matter of when.

4. Synthetic AR If contact lenses don’t sound futuristic enough, let’s review Synthetic Reality, i.e., a discipline aimed at making technology and biology indistinguishable from each other by, for instance, developing a brain-computer interface that allows the brain to control external devices. For instance, the winners of the 2017 annual BCI Research Award include an online adaptive brain-computer interface with attention variations, individual word classifications during imagined speech, and BCI prosthetic hand to control phantom limb pain.

An AR brain-computer interface will control part of our brainwaves to make us believe that we are seeing things that are not actually there, or perhaps the reverse: not seeing things that are actually there. This type of interface taps into the visual signals sent to our brain and alters them so that we end up with a fusion of real and fake imagery that is indistinguishable from one another.

We can also imagine an evolved form of spatial augmented reality to be developed through which matter can be (re)programmed in real-time. Think the Holodeck2 or Replicator3 devices featured in Star Trek. These two technologies might be the ultimate “immersive reality.”

2 The Holodeck is a large room with objects that appear to be there but physical reside somewhere else. The objects and people are simulated by “holomatter”: a combination of matter, beams and fields onto which holographic images are projected. Source: Memory Alpha Holodeck. 3 The Replicator is a machine capable of creating objects out of pure energy. Source: Memory Alpha Replicator.

Courtesy of The Imaging Alliance Copyright Suite 48 Analytics

Use code TIA85 to get 25% discount on either the Executive or Platinum version of the report 19

Form Factor Timeline The following table gives an overview of the expected timeline for AR form-factors:

Consumer AR Phase4

Handheld 1.0; Spatial 1.0

Handheld 2.0; Spatial 2.0

Wearable 2.0 Wearable 3.0, Synthetic

Timeline 2003 - 2013 2013 - 2020 2020 - 2025 > 2025

Type of AR Location-based Recognition-based

Recognition-based

Recognition-based

Processing On smartphone On smartphone Tethered Self-contained

Connectivity 2G/3G 3G/4G 4G/5G 5G/6G?

Additional processing In cloud

Power Using your phone battery

Using your phone battery

Using your phone battery, battery pack

Wireless, Solar

Input Device Touch Touch, Voice Voice, Gestures, Eye-tracking

Voice, Gestures, Eye-Tracking, Body Movement, Brain-Computer Interface

4 These classifications are based on when we expect the technology to be widely available and affordable for purchase by consumers as a final retail product, not a developer preview.