Embed Size (px)
Citation preview
Designing for energy-efficient vision-based interactivity
on mobile devices
Miguel Bordallo Center for Machine Vision Research
Objective of the research
• To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements
• To provide insight into the computing needs and characteristics of the future camera-based applications
Smartphones are not smart
• Current mobile devices lack interactivity– Unable to detect if you hold them– Unable to detect if you are looking– Unable to predict your intentions
• Mobile devices don’t ”watch you” (or listen)
– You need to actively indicate what you want– Application launch has VERY high latency
Typical UIs and interaction methods
• Buttons– Reduced functionality
• Touch screens– Needs (often) two hands operations
• Motion sensors (+ proximity, light, etc)
– Mostly used when the user is ”active”
Vision-based interactivity
• Using cameras as an Input modality
• Enables recognizing the context in real time (and to see the user and environment)
• Current mobile devices integrate touch screen, sensors and several cameras• But UI s don’t use them together !!
• The small size of handheld devices and their multiple cameras and sensors are under-exploited assets
Vision-based UI
Vision-based Interaction methods
Interactive image capture
Head movement triggers
Automatic start of applications
Why don’t* we have these kind of methods on our mobile devices?
*(some of them are coming)
Challenges/needs of vision-based interactivity
Challenges/needs of vision-based interactivity
• Very low latency (below 100 ms.)
Challenges/needs of vision-based interactivity
• Very low latency (below 100 ms.)
• Computationally costly algorithms
Challenges/needs of vision-based interactivity
• Very low latency (below 100 ms.)
• Computationally costly algorithms
• Sensors (cameras) ”always” on
Challenges/needs of vision-based interactivity
• Very low latency (below 100 ms.)
• Computationally costly algorithms
• Sensors (cameras) ”always” on
• Energy-efficient solutions
Challenges/needs of vision-based interactivity
• Very low latency (below 100 ms.)
• Computationally costly algorithms
• Sensors (cameras) ”always” on
• Energy-efficient solutions
Are mobile platforms energy-efficient?
Energy-efficiency on mobile devices
• Battery life is a critical mobile device feature– App. performance is constrained by battery life
• Energy efficiency is managed by switching off complete subsystems– Cameras, motion sensors, GPS, CPU cores, ...
• Only ”important” subsystems are always on and responsive (standby mode)
– GSM/3G modem, buttons
Battery capacity
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
10001100
12001320
1450
1830
2000
mah
Battery vs. CPU frequency
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
0
200
400
600
800
1000
1200
1400
1600
10001100
12001320
1450
1830
2000
mahmhZ
Battery vs. CPU power
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
0
200
400
600
800
1000
1200
1400
1600
10001100
12001320
1450
1830
2000
mahmhZmW
Battery vs. talk time
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
10001100
12001320
1450
1830
2000
3 3.5 4 57
8.512
mahtalk h
Battery vs. ”active use”* time
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
10001100
12001320
1450
1830
2000
3 3.5 4 57
8.512
60
47
30
97 6.5
5
mahtalk hhours
* Don’t trust these numbers
”Active use”* time
* Don’t trust these numbers
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
talk hhours
Active use vs processor power
6630 n70 n95 N900 N9/Lumia 800
lumia 900 lumia 925
2004 2005 2006 2007 2008 2009 2011 2012 2013
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
0
0.0025
0.005
0.0075
0.01
0.0125
0.015
0.0175
talk hhours1/mW
Current platforms
How can we improve the energy efficiency of Vision-based
interactive applications and UI s?
Offering Computer Vision algorithms and apps as a part of a Multimedia/CV Framework - Filtering, feature detection, robust estimators, classifiers, block matching,
- Face detection, motion estimation, blending
Avoid the use of the application processor for ”sensing” tasks
Asymmetric multiprocessing(Heterogenous computing)
• Concurrently use different processors on a mobile device to perform suitable tasks
• Processors identical (multicore) or heterogenous (CPU+GPU+DSP+CoDec)
GP-GPU-based interaction acceleration• GPUs are present in most modern mobile devices• GP-GPU exploits GPUs for general purpose algorithms• Mobile GPUs have architectural advantages• Computer Vision on GPUs very popular field
but....
• Cameras and sensors lack fast data transfer• Image formats not always compatible• IDE and interfaces not mature (OpenCL, OpenGL ES)
Sensor processor assisted context recognition
• Dedicated chips for sensor/camera processing– IVA2+, ISP
• Based on DSP processors + HW codecs• Good interconnections with sensors/cameras• Reasonably good performance/efficiency
but...
• Complicated and obscure interfaces– Access not always allowed to regular developer
• Limited flexibility
Dedicated computing for vision-based User Interfaces
• Dedicated (programmable) architectures offer:– Incredibly high performance (Hybrid SIMD/MIMD)
or..– Extremely good energy efficiency (TTA)
but...
• Not incorporated into current devices– Not likely to be anytime soon
Performance of different processors
CPU CPU+NEON DSP mGPU ISP/Codec
550
670
248
93 85
Platform:OMAP3530(Nokia N900)
Performance of different processors
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
550
670
248
93 85
720
1.5
Power (mW)Platform:OMAP3530(Nokia N900)
Performance of different processors
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
550
670
248
93 85
720
1.5
Power (mW)
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
113
76
12
22
60.31
20
Performance (Cycles per pixel)
Platform:OMAP3530(Nokia N900)
Performance of different processors
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
550
670
248
93 85
720
1.5
Power (mW)
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
113
76
12
22
60.31
20
Performance (Cycles per pixel)
CPU CPU+NEON DSP mGPU ISP/Codec Hybrid TTA
104
86
7
19
1.75 1.5 0.2
Energy efficiency (pJ per pixel)
Platform:OMAP3530(Nokia N900)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
1320mAh
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
2100 mAh
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
7000 mAh !!
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
Battery time (h)
Pow
er c
onsu
med
(mW
)
”Knee” region Standby zone
Activ
e-us
e zo
ne
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
RegularUI (standby)
RegularUI (active state)
Application processor (100%)
GSM (standby)
VGA camera (15 fps)
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
VB UI (Standby)RegularUI (standby)
RegularUI (active state)
Application processor (100%)
GSM (standby)
VGA camera (15 fps)
VB UI (active state)
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
VB UI (Standby)RegularUI (standby)
RegularUI (active state)
VGA camera (15 fps)
VB UI (active state)
Battery time (h)
Pow
er c
onsu
med
(mW
)
Battery discharge time (constant load)
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
900
1000
VB UI (Standby)
RegularUI (standby)
RegularUI (active state)
QVGA Camera(1 fps)
QVGA Camera
(10 fps)
VB UI (active state)
Battery time (h)
Pow
er c
onsu
med
(mW
)
Designing for interactivity
• Mobile devices need architectural changes to incorporate Vision-Based Uis
• Small footprint processors close to the sensors
• Sensors ”always” ON at a small framerate
• Only processed data arrives to the application processor
Current platforms
Current platforms
IRcam
QVGA
IRcam
VGA
IRcam
QVGA
IRcam
VGA
Thanks!? ? ? ??
? ? ? ?? ?
Any question? ? ? ? ? ? ? ?
? ? ?? ?
