Download pptx - Designing for energy-efficient vision-based interactivity on mobile devices

Designing for energy-efficient vision-based interactivity

on mobile devices

Miguel Bordallo Center for Machine Vision Research

Objective of the research

• To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements

• To provide insight into the computing needs and characteristics of the future camera-based applications

Smartphones are not smart

• Current mobile devices lack interactivity– Unable to detect if you hold them– Unable to detect if you are looking– Unable to predict your intentions

• Mobile devices don’t ”watch you” (or listen)

– You need to actively indicate what you want– Application launch has VERY high latency

Typical UIs and interaction methods

• Buttons– Reduced functionality

• Touch screens– Needs (often) two hands operations

• Motion sensors (+ proximity, light, etc)

– Mostly used when the user is ”active”

Vision-based interactivity

• Using cameras as an Input modality

• Enables recognizing the context in real time (and to see the user and environment)

• Current mobile devices integrate touch screen, sensors and several cameras• But UI s don’t use them together !!

• The small size of handheld devices and their multiple cameras and sensors are under-exploited assets

Vision-based UI

Vision-based Interaction methods

Interactive image capture

Head movement triggers

Automatic start of applications

Why don’t* we have these kind of methods on our mobile devices?

*(some of them are coming)

Challenges/needs of vision-based interactivity


• Very low latency (below 100 ms.)



• Computationally costly algorithms




• Sensors (cameras) ”always” on





• Energy-efficient solutions





• Energy-efficient solutions

Are mobile platforms energy-efficient?

Energy-efficiency on mobile devices

• Battery life is a critical mobile device feature– App. performance is constrained by battery life

• Energy efficiency is managed by switching off complete subsystems– Cameras, motion sensors, GPS, CPU cores, ...

• Only ”important” subsystems are always on and responsive (standby mode)

– GSM/3G modem, buttons

Battery capacity

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

500.00

1,000.00

1,500.00

2,000.00

2,500.00

10001100

12001320

1450

1830

2000

mah

Battery vs. CPU frequency

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

500.00

1,000.00

1,500.00

2,000.00

2,500.00

0

200

400

600

800

1000

1200

1400

1600

10001100

12001320

1450

1830

2000

mahmhZ

Battery vs. CPU power

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

500.00

1,000.00

1,500.00

2,000.00

2,500.00

0

200

400

600

800

1000

1200

1400

1600

10001100

12001320

1450

1830

2000

mahmhZmW

Battery vs. talk time

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

500.00

1,000.00

1,500.00

2,000.00

2,500.00

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

10001100

12001320

1450

1830

2000

3 3.5 4 57

8.512

mahtalk h

Battery vs. ”active use”* time

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

500.00

1,000.00

1,500.00

2,000.00

2,500.00

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

10001100

12001320

1450

1830

2000

3 3.5 4 57

8.512

60

47

30

97 6.5

5

mahtalk hhours

* Don’t trust these numbers

”Active use”* time

* Don’t trust these numbers

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

talk hhours

Active use vs processor power

6630 n70 n95 N900 N9/Lumia 800

lumia 900 lumia 925

2004 2005 2006 2007 2008 2009 2011 2012 2013

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

0

0.0025

0.005

0.0075

0.01

0.0125

0.015

0.0175

talk hhours1/mW

Current platforms

How can we improve the energy efficiency of Vision-based

interactive applications and UI s?

Offering Computer Vision algorithms and apps as a part of a Multimedia/CV Framework - Filtering, feature detection, robust estimators, classifiers, block matching, - Face detection, motion estimation, blending

Avoid the use of the application processor for ”sensing” tasks

Asymmetric multiprocessing(Heterogenous computing)

• Concurrently use different processors on a mobile device to perform suitable tasks

• Processors identical (multicore) or heterogenous (CPU+GPU+DSP+CoDec)

GP-GPU-based interaction acceleration• GPUs are present in most modern mobile devices• GP-GPU exploits GPUs for general purpose algorithms• Mobile GPUs have architectural advantages• Computer Vision on GPUs very popular field

but....• Cameras and sensors lack fast data transfer• Image formats not always compatible• IDE and interfaces not mature (OpenCL, OpenGL ES)

Sensor processor assisted context recognition

• Dedicated chips for sensor/camera processing– IVA2+, ISP

• Based on DSP processors + HW codecs• Good interconnections with sensors/cameras• Reasonably good performance/efficiency

but...• Complicated and obscure interfaces– Access not always allowed to regular developer

• Limited flexibility

Dedicated computing for vision-based User Interfaces

• Dedicated (programmable) architectures offer:– Incredibly high performance (Hybrid SIMD/MIMD)

or..– Extremely good energy efficiency (TTA)

but...• Not incorporated into current devices– Not likely to be anytime soon

Performance of different processors

CPU CPU+NEON DSP mGPU ISP/Codec

550

670

248

93 85

Platform:OMAP3530(Nokia N900)


CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA

550

670

248

93 85

720

1.5

Power (mW)Platform:OMAP3530(Nokia N900)



550

670

248

93 85

720

1.5

Power (mW)


113

76

12

22

60.31

20

Performance (Cycles per pixel)




550

670

248

93 85

720

1.5

Power (mW)


113

76

12

22

60.31

20

Performance (Cycles per pixel)


104

86

7

19

1.75 1.5 0.2

Energy efficiency (pJ per pixel)


Battery discharge time (constant load)

0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

1320mAh

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

2100 mAh

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

7000 mAh !!

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

Battery time (h)

Pow

er c

onsu

med

(mW

)

”Knee” region Standby zone

Activ

e-us

e zo

ne


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

RegularUI (standby)

RegularUI (active state)

Application processor (100%)

GSM (standby)

VGA camera (15 fps)

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

VB UI (Standby)RegularUI (standby)


Application processor (100%)

GSM (standby)

VGA camera (15 fps)

VB UI (active state)

Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

VB UI (Standby)RegularUI (standby)


VGA camera (15 fps)


Battery time (h)

Pow

er c

onsu

med

(mW

)


0 20 40 60 80 100 120 140 160 180 2000

100

200

300

400

500

600

700

800

900

1000

VB UI (Standby)

RegularUI (standby)


QVGA Camera(1 fps)

QVGA Camera

(10 fps)


Battery time (h)

Pow

er c

onsu

med

(mW

)

Designing for interactivity

• Mobile devices need architectural changes to incorporate Vision-Based Uis

• Small footprint processors close to the sensors

• Sensors ”always” ON at a small framerate

• Only processed data arrives to the application processor

Current platforms

Current platforms

IRcam

QVGA

IRcam

VGA

IRcam

QVGA

IRcam

VGA

Thanks!? ? ? ? ?

? ? ? ?? ?Any question?

? ? ? ? ? ? ? ? ? ? ??