Mar general pan hui ppt

Mobile Augmented

Reality augmented reality on the go

Pan Hui

symlab.ust.hk HKUST

Mobile Augmented Reality

Augmented Reality

Mobile Computing

§ computing devices, linked by wireless networks, allow us to use computing facilities while roaming the real world

§ Today we also in WAR

Combines real and virtual world

Interactive in real time

Registered in 3D

Data Visualization

 With Augmented reality, we can:   Display data directly on data source (thermometer, gauges, …) without display

device.   Connect data with physical context.   Do not require display simplification (no context interpretation).

http://emcl.iwr.uniheidelberg.de/research sv.html)

http://www.maptek.com/

User Interaction

 By augmented reality, we are able to interact with data:  Without sucking yourself away from real world.  Without physical dimension restriction.  In a collaborative way.

http://vimeo.com/46304267 http://www.avatarmovie.com/index.html

Retail

 In retail, augmented reality is able to:  supply vertical retail to individual consumers (price, date, …).  Display accurate product recommendation and personalized

advertisement.  Enrich shopping experience.

http://www.asleyetracking.com/ https://www.youtube.com/watch?v=XM9ZOWPeiAk

Tourism

 With big data, augmented reality enables to:  Aggregate and compile the redundant fragmented data to build a detailed and

complete environmental model. (e.g. Google Earth)  Provide personalized travel guide information. (intelligent recommendation is

regarded as the most attractive expectation of tourism. [Business Insider, 2012])  Gamification of travel (e.g. Ingress)

http://www.ingress.com/http://www.theguardian.com

Healthcare

 AR enables to:  Quickly access valuable information of patients. (with face recognition)  Provide In-suit visualization of historical illnesses or tissue damage.  Give health suggestion based on health stats and diet.  Enable remote diagnosis in a collaborative way.

http://www.iqiyi.com/v_19rrh3pj3k.html https://www.youtube.com/watch?v=jZkHpNnXLB0&feature=kp

Mobile Augmented Reality

Develop the underlying technology

Develop prototype projects

Produce high standard AR applications

A bottom-up approach

�

Computation offloading�

Object tracking� Localization� Privacy� Hand gesture

recognition�Face recognition�

Mobile Augmented

Reality�

Driving assistant

Natural user interface Privacy

protection

Reading assistant

“ 27 published and filed patents in the field of

Mobile Augmented Reality (including Mobile Computing,

Networking, Privacy, Localization)

Our latest research was featured in China Daily

Total number of document(s): 15

Experts hand it to an augmented future

South China Morning Post | 2016-09-21Newspaper | CITY3 | CITY | technology | By Kinling Lo

Word Count: 248words | Image No: 1/1 | Image Size: 131cm-sq(14.4cm x 9.1cm)

科大打造AR開發平台慳時間省成本有望「一統江湖」

Ta Kung Pao | 2016-09-21Newspaper | A18 | 教育


科大研AR新平台 App一周寫好

Wen Wei Po | 2016-09-21Newspaper | A21 | 新聞透視眼

Word Count: 794words | Image No: 1/1 | Image Size: 319cm-sq(22cm x 14.5cm)

科大研發平台拒AR記錄臉孔

Hong Kong Economic Times | 2016-09-21Newspaper | A18 | 互聯網＋ | By 黃蘊華

Word Count: 1,053words | Image No: 1/1 | Image Size: 408cm-sq(32.4cm x 12.6cm)

科大實驗室申AR 領域18項專利

Hong Kong Commercial Daily | 2016-09-21Newspaper | A15 | 香港新聞


雲機實鏡技術加快程式運作

Ta Kung Pao | 2016-09-21Newspaper | A18 | 教育


科大AR研究申18項專利

Sing Tao Daily | 2016-09-21Newspaper | F02 | 教育


Ubiquitous interface and interaction Towards Seamless Interaction between Digital and Physical Worlds

BACKGROUND

Traditional GUI

´  Display digital information

Augmented Reality

´  Unobtrusive

´  Sensing in environment

´  Blend physical and digital worlds

Ubii - Overview An Intergrated interface which

allows users to interact with objects in the environment with hand gestures

Take advantage of hand-

gesture recognition and object tracking

Connect and communicate

with smart devices in the environment

RELATED WORK

Interaction at a distance

´  Manipulate content that are unreachable

´  Manipulate devices that are incapable of touch interaction

Freehand interaction

´  Natural, intuitive, effective

´  Mid-air interaction

´  Recognize hand gestures/ body movements

S. Boring, D. Baur, A. Butz, S. Gustafson, and P. Baudisch.Touch projector: mobile interaction through video.

Microsoft Kinect

SYSTEM DESIGN — physical affordance

computer – file transfer printer - printing projector screen - projecting physical surface – file manipulation

all made possible with simple hand gestures

System flow of Ubii

SYSTEM DESIGN — Menu Design

Placement •  Object-referenced placement •  Attach menu to physical object •  Tag visual marker on physical

object for alignment


Placement Orientation

•  Align the menu with physical object surface

•  Improve readability •  Better 3D spatial presence


Placement Orientation Trigger Mechanism

•  Menu are activated and deactivated based on the hand gestures

pinch normal


Placement Orientation Trigger Mechanism Ring Menu

•  Menu items are distributed on an ring around the object

•  A few menu items are active and the unused ones are folded

•  Ring menu can be rotated to change the active items

The rotational Ubii menu

SYSTEM DESIGN — Interaction Design

a) pick b) drop c) drag d) rotate ring menu e) zoom

The 5 supported hand gestures of Ubii

IMPLEMENTATION —Object Tracking

IMPLEMENTATION — Hand Gesture Detection

Recognize user hand gestures


´  Sample skin color in HSV space

´  Extract hand contours

´  Wrap hand contours with polygonal hulls

´  Calculate convexity defects to identify pinch gestures

Identifying pinch, drop and no-pinch gestures

Sampling skin colors from sample points


´  Algorithm can distinguish hand gestures from the background

´  Can distinguish between a pinch gesture and non-pinch holes

The pinch gesture detection. a) a typical pinch gesture; b) outer contour of hand is extracted; c) and d)red

closed regions are recognized as pinch holes; e) and f) green closed regions are eliminated as non-pinch holes.

IMPLEMENTATION —System Implementation

How the Ubii system interact with other devices

EVALUATION — How effective Ubii is

1.  Copying documents between computers

2.  Printing documents 3.  Displaying documents

on projector screens 4.  Sharing documents

Comparison of task competition time of four experiments by using Ubii and traditional

methods.

On average, Ubii can reduce the operation time by at least half

Most participants are satisfied with their experience

EVALUATION — How effective Ubii is

Cardea: Context-Aware Visual Privacy Protection from Pervasive Cameras

Background

´  Technologies benefit lives, but also raise privacy concerns!

´  Easier for people to take photos without obvious signals

´  Popularity of online photo sharing platforms

´  Advanced recognition techniques

´  Google Glass is spotted as an example of rising visual privacy concerns from the public

Motivation

´  User studies

´  Consent mechanism is welcomed when being recorded

´  Life loggers care about the privacy of bystanders

´  Privacy concerns depend on the context: who, what, when, where, why, and how

´  Limitations of previous solutions

´  Static policies

´  Aesthetically awkward

´  Extra sensors (e.g., infrared imager)

Design Principles

´  Cardea design

´  Problem setting: mobile/wearable devices with built-in camera

´  Technical solution: computer vision techniques

´  Protection enforcement time/level: in situ/application level

´  Protection object: bystanders’ visual privacy

´  Opt-in vs. opt-out: opt-in

´  Objectives

´  Context dependent

´  Individualized

´  Dynamic

Design Overview

´  Allowing individuals to proactively convey their

context-dependent privacy preferences bound with

face features.

´  Location

´  Scene

´  People in the image

´  Hand gestures can be used to interact with cameras

to temporarily update current preferences

´  Bystanders can also use static tags as privacy

indicators.

System Architecture

§  Bystander app: registration; setting privacy preference profile §  Recorder app: taking images §  Cloud: storing users’ profiles and training face recognition model;

responding to clients; requests

Scene Classification

´  Places2 dataset: 401 scene categories, 10 million training images.

´  Training a classifier on 9 general scene categories which are common or sensitive.

Classification examples

Face Recognition & matching

´  Face detection and alignment

´  Feature extraction using lighted CNN

Gesture Recognition

´  Training a gesture detector on a combined dataset

Recognition examples

Workflow of Cardea

User Interface - Registration

Privacy Protection Example

Evaluation – scene classification

´  8 volunteers take 759 images “in the wild”, with 638 images selected, covering 9 general scene groups

(a) recall (b) Confusion matrix

Evaluation – face recognition and matching

´  Selecting 50 subjects from LFW dataset, 5042 features for training and validation, 511 features as user test set, 166 features from 100 other subjects as non-user test set.

Evaluation - face recognition

´  Selecting 50 subjects from LFW dataset, 5042 features for training and validation, 511 features as user test set, 166 features from 100 other subjects as non-user test set.

(a) Training accuracy (b) Testing accuracy with probability threshold

Evaluation - face matching

´  Selecting 23 subjects from user test set who has more than 10 features. Using these 230 features as database features, and the other 281 features as query features.

(a) Cosine distance (b) Euclidean distance

Evaluation – gesture recognition

(a) Recall for different scenes

(b) Precision for different scenes

´  338 hand gesture images with 208 “Yes” gesture, 211 “No” gestures, and 363 natural hands.

Evaluation – scene classification

´  8 volunteers take 759 images “in the wild”, with 638 images selected, covering 9 general scene groups

(a) recall (b) Confusion matrix

Evaluation – Runtime

´  Client: Samsung Galaxy Note 4

´  Server: Intel i7-5820K CPU, 16GB RAM, GeForce 980Ti Graphic Card

´  Network: eduroam

(a) runtime

Evaluation – overall performance

´  5 volunteers register as Cardea users, set their privacy profiles. In total, we take 224 images for evaluation.

Demo video

Cardea summary

´  Design

´  Context awareness

´  Interactive control

´  Implementation

´  Deep neural networks

´  Deployment on android

´  Feasibility

´  Micro vision benchmarks

´  Overall performance

´  Runtime evaluation

The future

Now we have the technology, we are ready to put ideas into business

The future

Using AR technology, users can view information of the devices around them. They can also visualize data and communicate with the Internet of Things

The future

With augmented reality and a natural user interface, users can interact with any smart objects in the environment In a factory, users can control and issue commands to robots with hand gestures

Place your screenshot here

Thanks! Any questions?

Prof. Pan Hui [email protected] symlab.ust.hk

Art & Photos

Mar general pan hui ppt