Wearable technologies for human empowerment

0 Copyright 2014 FUJITSU

Human Centric Innovation

Fujitsu Forum 2014

19th – 20th November


Wearable Technologies for Human Empowerment

Naoyuki Sawasaki Director, Human Interaction Laboratory, Human Centric Computing Laboratories


Agenda

Introduction

Overview of Wearable Computing

Wearables for Industrial Solutions

Wearable Service Platform

Summary


Introduction


Vision: Human-centric, Intelligent Society

Environment Energy Food Business

Efficiencies Health Safety Security

Real

world

Various Kinds of Information from

Human Awareness and Sensing Devices New

Services

&

Solutions

New

Services

&

Solutions Big data

Extend the application field of ICT to real human life


Paradigm Shift in ICT

Transform business processes

Network-centric

Create knowledge,

support human activities

Intelligent Society

Human-centric

Create new value 1990 2010 2020 2000

Sco

pe

of IC

T U

sa

ge

Improve productivity

Computer-centric

Shift from “Technology-centric” to “Human-centric”:

Support of activities that are more people-focused


Relationship between ICT and Human

From ICT to face the person to ICT that wraps around the person

Now (Machine centric) Future (Human centric)

• ICT is being synchronized to the real world

• Required services are pushed to people

(Human empowerment)

• ICT is being separated from the real world

• People must have access to the required

services by terminal operations


Wearable Technologies as Means for Human Empowerment

Enable interaction between ICT and human activities in the real world ICT can process not only the behavior that the user is doing but also information that the

user perceives.

Enable to support the user’s work by information services even when the user is acting.

Real world

Wearable devices

Visual, auditory and

motion information

ICT Support information


Impact on Solution Business

Wearable technologies enable ICT to support various on-site work. Enable providing information services in a timely fashion to make work more efficient,

with fewer errors and oversights, even when the operator is not an expert

Enable improvement of the business by visualizing the situation of the site work

Hospital

Shop

Inspection

Maintenance

Distribution Office


Overview of Wearable Technologies


History of Wearable Computing

1990s 1980s 1970s 1960s

The first HMD (MIT)

1968

S. Mann (MIT)

designed a wearable computer.

1981

CMU developed a portable

wearable computer for

industrial applications.

1991

(http://www.cs.cmu.edu/~wearable/vuman.html) (http://www.flickr.com/photos/pargon/2444155973/)

(http://www.bluebugle.org/2013/02/

google-glass-wearable-computer.html)

Concept and applications have been explored since the 1980s. The Head mounted display was invented in the 1960s.

An actual wearable computer was designed in 1981.

Research on industrial applications started in the 1990s.


Concept of Wearable Computing

Steve Mann proposed the definitions of a wearable computer(*). Constancy: Always running and is ready to interact with the user

Augmentation: Enhances the user's ability by supporting the user's work on site

Mediation: Encapsulates the user as an information filter

(http://wearcam.org/pictures.html) （*）“Wearable Computing as Means for Personal Empowerment”, ICWC-98, May 12-13, Fairfax, VA

Idea of human centric but it was limited to individuals

Not only research on equipment but also research on real world computing are required.


Research Issues of Wearable Computing

Key technologies to realize wearable computers have been developed in three research areas:

Mobile computing

Small, lightweight and low power consumption devices with network connectivity.

User interface

Devices and techniques for an easy to use interface without interrupting user’s work.

Head mounted display, Augmented reality, and other multi-modal (audio-visual) interfaces.

Context-aware computing

Devices and techniques for sensing user’s condition and situation.


Advancement of Mobile Computing

Google

Glass

2010s 2000s 1990s 1980s

The first PDA

“Newton” (Apple)

The first

Smartphone

“iphone” (Apple) Windows phone

4G 3G 2G 1G

Cellular phones

1989 1992

The first book computer

“Dynabook” (Toshiba)

2007 2010 2013

Android phone

By the rapid progress of semiconductor technology and the wireless telecommunications technology, a smart phone was developed in 2007 and it has spread rapidly.

GPS, camera and other sensors enable the development of applications to support the user’s daily life. (navigation, activity monitoring and life-logs, etc.)

Development of technologies for user interface and context-awareness are boosted.


Wearables Emerging as Consumer Products

NIKE+ FUELBAND SE

Eyewear Watch

Sensors Accessories

Wearable Devices

CSR

BT Smart

Jewelry

Smarty Ring

O.R.B. BT Headset

Sony Smart Watch Recon JET

Fitbit Flex

Vuzix Video eyewear Apple Watch Galaxy Gear

Encouraged by the Google Glass, eyewear, watches and other devices are now heating up in the consumer mobile market.

http://www.google.co.jp/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=c__n_4qTo0S81M&tbnid=OOWUxN9YKoJY-M:&ved=0CAcQjRw&url=http://www.phonebunch.com/news/samsung-galaxy-gear-update-full-notifications_455.html&ei=S7UuVM70IITl8AXnpID4CQ&bvm=bv.76802529,d.dGc&psig=AFQjCNE0yQvmjdXM7b65p27hmpycq0oPEA&ust=1412433561435589


Functionalities of Consumer Wearables

Eyewear Incorporated with HMD, camera, GPS and inertial sensors

Enable utilization of information services while performing activities

Enable the recording of the information that the user sees and hears in a timely fashion

Watch Incorporated with small touch panel display

Extends smartphone user interface, making it possible to utilize information services at hand

Sensors & Accessories

Incorporated with motion and vital sensors

Enable the recording of the action and the vital information of daily life for management of health

http://www.google.co.jp/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=c__n_4qTo0S81M&tbnid=OOWUxN9YKoJY-M:&ved=0CAcQjRw&url=http://www.phonebunch.com/news/samsung-galaxy-gear-update-full-notifications_455.html&ei=S7UuVM70IITl8AXnpID4CQ&bvm=bv.76802529,d.dGc&psig=AFQjCNE0yQvmjdXM7b65p27hmpycq0oPEA&ust=1412433561435589


Market Trend

Killer applications in the consumer area are still unclear, but the industrial application area is growing rapidly.

13%

29%

5% 5%

39%

8% 1% Medical

FitnessGaming

IndustrialLifestyle

Entertainment

Pets AnimalsAreas of Market Focus

13%

28%

5% 1%

43%

10% MedicalFitnessGamingIndustrialLifestyleEntertainment

2014.02

2014.10 (http://vandrico.com/database)

Areas of Market Focus




Market Potential

Vast number of industrial workers are still outside the ICT solutions. They are most in need of support by ICT for making work more efficient and safe.

However, there is a big hurdle to interface technology that can utilize the ICT support while working.

Construction work Maintenance of old facilities and infrastructure


Smart Device Solution

Interstage AR Processing Server Offer augmented reality interface by smart devices

• The system detects the working context by recognizing visual marks and provides services to support subsequent tasks.

Enable to reduce the error and oversight, but needs extra operations for smart device

Paper and pen Smart device


Input

devices

Development of wearables for a easy to use interface while working Easy-to-see HMD with embedded camera and voice communication

Input devices for various work scenarios to minimize user’s operations


HMD

wearable

keypad


Glove-Style Wearable Wrist Device

NFC tag reader

Contact-detection

sensor

Gyro sensor

Accelerometer

NFC tag reading unit

Soft shell cover

NFC: Near Field Communication

Input device utilizes user's touch operations and hand gestures.

Enables a touch-based tangible user interface. Users can interact with a digital system by simple physical motion.

Enables gesture input by motion sensors.


Supported Functionalities

NFC tag

① Detects

touch

② NFC tag reader

goes active

③ Reads tag data

④ Transmits tag

data

⑤ NFC tag reader

goes idle

up

down

left

right

rotation left

rotation right

NFC tag reader on the finger tip offers “wake on touch” capability.

Reliable gesture-recognition technology works with a variety of tasks and postures.

Hand gesture input

NFC tag reading Gesture motion was detected with the

condition of wrist joint (backward bending).

Extension

Common

range

Gap

Maximum


Demonstration of Gesture input


Typical Use Cases

By taking the natural action of touching an object, the system can detect the working context and offer work instructions and other information to the user.

Task results can also be input using simple gestures so that operations can be carried out smoothly.

Bluetooth

Gesture

command

Information

WiFi

Remote

support

Center

NFCtag

Glove

device

HMD & Headphone

Contact event and tag data

Gesture input

Work instructions

HMD, Headset

Server

Management/

recording of operations

Mobile application


Use Case Example: Wiring Cables

Where to insert？

Operator ICT

Takes hold of a cable Indicates the target socket

Inserts the cable

Confirms the job status using a gesture

Records the result

Visual Support

HMD

(1) Detect the touch event and identify the cable

(2) Indicate the target socket visually by head mounted display

(3) Gesture based confirmation input and record to DB


Demonstration of Supporting Wiring Cables Task


Use Case Example: Inspection of Facilities

Operator ICT

Touches the label Offers judgment

criterion aid Inspects the equipment

Inputs the result using gestures Records the result

NFC Tags Deployed Voice Instruction

Headset (Earphone)

(1) Detect the touch event and identify the instrument to inspect

(2) Offer sound instruction through the headset’s earphone

(3) Gesture based confirmation input and record to DB


Demonstration of Supporting Inspection of Facilities


Wearable Ring Device

Wearable device for gesture command and text input in the midair Enables handwriting from anywhere at anytime

Enables to write down notes on the scene that the user sees in HMD


Wearable Service Platform


Front-end Integration by Mobile Applications

Mobile application enables front-end integration of cloud services without modifying the back-end system.

Also enables to utilize a wide variety of wearables to offer optimized interface to on-site workers

High performance and low power

mobile processor

Cloud Wearable Devices

Wireless personal area network

・Bluetooth, etc.

Web-API

Mobile Computing Platform

4G Application

Services

Data

Front-end Back-end


Issues of Enterprise Solutions

Gadgets Gadgets

app1

Gadgets

app2 app3

Backend system

The competitive edge of enterprise solutions is now in the development of mobile applications. Efficient development of mobile applications to suit various work scenarios on site

• Utilize wearable devices to offer suitable interfaces to various on-site works

Management of on-demand delivery of applications to workers


Required Technologies

Service platform including,

Application development

Independent from mobile operating systems, programming languages and wearable devices

• Web application (HTML5 + JavaScript)

• Runtime API extension for wearable device usage

Application management

Context-driven scheme of application delivery

• Required application is automatically downloaded and executed depending on the work context on site


Service Platform Architecture

Enables to provide services selected from a variety of backend services depending on the user's work context

Enables to run on smart devices with a variety of wearable devices

Service platform (Terminal middleware)

Mobile Application (HTML5+Javascript)

Service platform (Application server)

Backend services

Application

distribution

Work

context

Cloud

Enterprise app store

Wearable devices Smart device


Service Platform under Development

Service Platform enables “context switch” of web application Application server manages the sets of applications for each work context to push on

event notification.

Terminal middleware executes web application with plug-in API for wearable devices.

Enterprise

application store Terminal middleware

Context

switch

Application

Push

Application server

Event

notification

Work

context

Wearable

devices Smart device

Web apps

(HTML5+javascript)

Watch

HMD

Headset

Gadget drivers

(plug-in)


Summary


Summary

Wearable technologies Enables interaction between ICT and human behavior in the real world

Extend the solution business to various on-site work

Wearables for industrial solutions Key technologies to open potential market

Wearable service platform Efficient development and management of mobile applications is a competitive

edge in the enterprise solution business.


Devices & Hardware

Wearable technologies for human empowerment