Designing for energy-efficient vision-based interactivity on mobile devices

  • Published on
    23-Feb-2016

  • View
    41

  • Download
    0

Embed Size (px)

DESCRIPTION

Designing for energy-efficient vision-based interactivity on mobile devices. Miguel Bordallo Center for Machine Vision Research. Objective of the research. To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements - PowerPoint PPT Presentation

Transcript

Designing for energy-efficient vision-based interactivity on mobile devices

Designing for energy-efficient vision-based interactivity on mobile devicesMiguel Bordallo Center for Machine Vision ResearchObjective of the researchTo gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements

To provide insight into the computing needs and characteristics of the future camera-based applicationsSmartphones are not smartCurrent mobile devices lack interactivityUnable to detect if you hold themUnable to detect if you are lookingUnable to predict your intentions

Mobile devices dont watch you (or listen)You need to actively indicate what you wantApplication launch has VERY high latency

Typical UIs and interaction methodsButtonsReduced functionality

Touch screensNeeds (often) two hands operations

Motion sensors (+ proximity, light, etc)Mostly used when the user is activeVision-based interactivityUsing cameras as an Input modality

Enables recognizing the context in real time (and to see the user and environment)

Current mobile devices integrate touch screen, sensors and several camerasBut UI s dont use them together !!

The small size of handheld devices and their multiple cameras and sensors are under-exploited assets

Vision-based UI

Vision-based Interaction methods

Interactive image captureHead movement triggersAutomatic start of applicationsWhy dont* we have these kind of methods on our mobile devices?

*(some of them are coming)Challenges/needs of vision-based interactivityChallenges/needs of vision-based interactivityVery low latency (below 100 ms.)

Challenges/needs of vision-based interactivityVery low latency (below 100 ms.)

Computationally costly algorithms

Challenges/needs of vision-based interactivityVery low latency (below 100 ms.)

Computationally costly algorithms

Sensors (cameras) always on

Challenges/needs of vision-based interactivityVery low latency (below 100 ms.)

Computationally costly algorithms

Sensors (cameras) always on

Energy-efficient solutions

Challenges/needs of vision-based interactivityVery low latency (below 100 ms.)

Computationally costly algorithms

Sensors (cameras) always on

Energy-efficient solutions

Are mobile platforms energy-efficient?Energy-efficiency on mobile devicesBattery life is a critical mobile device featureApp. performance is constrained by battery life

Energy efficiency is managed by switching off complete subsystemsCameras, motion sensors, GPS, CPU cores, ...

Only important subsystems are always on and responsive (standby mode)GSM/3G modem, buttonsBattery capacityBattery vs. CPU frequencyBattery vs. CPU power Battery vs. talk timeBattery vs. active use* time* Dont trust these numbers Active use* time* Dont trust these numbers Active use vs processor power

Current platforms

How can we improve the energy efficiency of Vision-based interactive applications and UI s?Offering Computer Vision algorithms and apps as a part of a Multimedia/CV Framework - Filtering, feature detection, robust estimators, classifiers, block matching, - Face detection, motion estimation, blending

Avoid the use of the application processor for sensing tasksAsymmetric multiprocessing(Heterogenous computing)Concurrently use different processors on a mobile device to perform suitable tasks

Processors identical (multicore) or heterogenous (CPU+GPU+DSP+CoDec)GP-GPU-based interaction accelerationGPUs are present in most modern mobile devicesGP-GPU exploits GPUs for general purpose algorithmsMobile GPUs have architectural advantagesComputer Vision on GPUs very popular field

but....Cameras and sensors lack fast data transferImage formats not always compatibleIDE and interfaces not mature (OpenCL, OpenGL ES)Sensor processor assisted context recognitionDedicated chips for sensor/camera processingIVA2+, ISPBased on DSP processors + HW codecsGood interconnections with sensors/camerasReasonably good performance/efficiency

but...Complicated and obscure interfacesAccess not always allowed to regular developerLimited flexibilityDedicated computing for vision-based User InterfacesDedicated (programmable) architectures offer:Incredibly high performance (Hybrid SIMD/MIMD) or..Extremely good energy efficiency (TTA)

but...Not incorporated into current devicesNot likely to be anytime soon Performance of different processorsPlatform:OMAP3530(Nokia N900)Performance of different processorsPlatform:OMAP3530(Nokia N900)Performance of different processorsPlatform:OMAP3530(Nokia N900)Performance of different processorsPlatform:OMAP3530(Nokia N900)Battery discharge time (constant load)

1320mAhBattery time (h)Power consumed (mW)Battery discharge time (constant load)

Battery time (h)Power consumed (mW)Battery discharge time (constant load)

2100 mAhBattery time (h)Power consumed (mW)Battery discharge time (constant load)

Battery time (h)Power consumed (mW)Battery discharge time (constant load)

7000 mAh !!Battery time (h)Power consumed (mW)Battery discharge time (constant load)

Battery time (h)Power consumed (mW)Battery discharge time (constant load)

Battery time (h)Power consumed (mW)Battery discharge time (constant load)

Battery time (h)Power consumed (mW)Knee regionStandby zoneActive-use zoneBattery discharge time (constant load)

Battery time (h)Power consumed (mW)Battery discharge time (constant load)

VB UI (active state)Battery time (h)Power consumed (mW)Battery discharge time (constant load)

VB UI (active state)Battery time (h)Power consumed (mW)Battery discharge time (constant load)

VB UI (active state)Battery time (h)Power consumed (mW)Designing for interactivityMobile devices need architectural changes to incorporate Vision-Based Uis

Small footprint processors close to the sensors

Sensors always ON at a small framerate

Only processed data arrives to the application processor

Current platforms

Current platforms

IRcam

QVGAIRcamVGAIRcam

QVGAIRcamVGAThanks!??????? ? ?? ?Any question? ? ? ? ? ? ? ? ?????

Recommended

View more >