PowerPoint-esitys

Designing for energy-efficient vision-based interactivity on mobile devicesLectio Praecursoria15.12.2014 Miguel Bordallo Lpezmiguelbl@ee.oulu.fiUniversity of Oulu Graduate SchoolFaculty of Information Technology and Electrical EngineeringDepartment of Computer Science and EngineeringInfotech OuluMiguel Bordallo Lpez, 2014Honorable Custos, Honorable Opponents, Ladies and Gentlemen:

First of all, thank you all for coming to my doctoral defence.

I will now present the introduction to my doctoral dissertation:Designing for energy-efficient vision-based interactivity on mobile devices

1Mobile device as a multimedia platform

Miguel Bordallo Lpez, 2014Mobile devices have become attractive platforms for multimedia Applications.

Most of the recent devices have increasingly been equipped with several built-in cameras, a large set of sensors,and high resolution touch-screens.

Future devices are expected to include a wider range of subsystems, improving in their multimedia capabilities.2Typical interaction method

Buttons

Touch ScreensMiguel Bordallo Lpez, 2014However, and despite of the progress in mobile technology, the typical ways of interacting with our devices have stayed relatively simple.

Nowadays, most of the devices include either physical or logical buttons. Pushing this buttons, turns on the device in the interaction sense, illuminating the displays and providing for an input. Once the device is in this active state, the devices can be used operating on its touch screen.

Touch screens allow the creation of detailed and powerful User Interfaces. 3

PointingInteracting with devicesClicking

Miguel Bordallo Lpez, 2014It is possible to argue that human-computer interaction has not changed fundamentally for nearly two decades.

If we have to define the most typical ways of interacting with our devices, even nowadays, we could summarize the most of our interactions in two actions to pointing and clicking actions

4

Pointing and ClickingMiguel Bordallo Lpez, 2014Several modalities for pointing and clicking have been emerging.

For example, older devices allowed the selection of User Interface elements with keypads and pointers.5Pointing and Clicking

Miguel Bordallo Lpez, 2014The introduction of touch screens allowed the elimination of the screen pointer and put the pointer in our hands as an stylus device

6

Pointing and ClickingMiguel Bordallo Lpez, 2014The evolution of touchsreens eliminated the stylus to allow the utilization of our own fingers as clicking pointers

7

LimitationsTwo-hands operationObstructing the viewMiguel Bordallo Lpez, 2014However, the latest evolution into touch-screen devices, has originated some limitations and potential problems.

The user, most of the times, needs to use both hands to operate the device.

Also, and especially in smaller screens, the users hand or fingers are partially obstructing the view of the device when the user is interacting with it, compromising the eventual perception of the displayed augmented information.

8Novel interaction methods

Motion sensors

Vision-based interactionsVoice commands

Miguel Bordallo Lpez, 2014To complement current tactile user interfaces, researchers and developers have been devising new methods to overcome the limitations of the ones currently used.

Motion sensor interaction is emerging as a modality to actively interact with the device with simple gestures such as shaking the device or turning it in a fast movement. However, they are mostly useful as complementary features when the user is already actively interacting with the device.

Voice commands are rapidly gaining traction, but utilizing them in public migh compromise the privacy of the user that might be easlily overheard.

Last, camera or vision-based user interaction is slowly being integrated in some of the newer platforms.This doctoral work focus mostly in these kind of methods.9Vision-based interactivity Using cameras as an input modality

Enables recognizing the context in real time

Utilizes already existing cameras

The small size of handheld devices and their multiple cameras and sensors are under-exploited assets

Miguel Bordallo Lpez, 2014In vision-based interactivity, the camera of a device is used essentially for sensing purposes, and utilized to control some aspects of the device to interact with it. The applications of vision-based interactivity range from gaming and augmented reality, to vision assisted general user interfaces.

One advantage of vision-based interactivity is that the rich information provided by the camera enables the recognition of the context in real time, in a way of seeing the user and the environment.

In addition, all mobile devices already integrate several cameras for different purposes so, utilizing them for interactive purposes is relatively straightforward.

However, mobile devices are still mostly replicating the same functionality as digital cameras were already some time ago. The small size of handheld devices and their multiple cameras are still under-exploited assets.

10History of vision-based interactivity

Mozzies (Siemens, 2003)Miguel Bordallo Lpez, 2014Vision-based interactivity (even in mobile devices) is not as new as we might think. Already in 2003, Siemens presented the first application (in this case a game) that was able to make use of the built-in camera for controlling purposes. The game, called mozzies (and shown in this video) utilized the buil-in camera, to make the user search for artificially created mosquitos, to track them, and catch them.11Vision-based interaction methods

Marker-based augmented realityMiguel Bordallo Lpez, 2014Since then, a lot of vision-based interaction techniques have been demostrated on embedded systems.

For example, marker-based augmented reality has been demostrated on mobile devices as a way of showing enhanced information ovelaying real-world images.12Vision-based interaction methodsMotion estimation-based browsing

Miguel Bordallo Lpez, 2014The use of motion estimation techniques has been utilized in several applications such as browsing big documents or galleries. 13Vision-based interaction methodsFinger and hand gesture recognition

Miguel Bordallo Lpez, 2014Finger tracking and hand gesture recognition has been demonstrated in several controlling applications such as different games or map browsing14Vision-based interaction methodsHead-movements triggers (looking at something to interact)

Dialogue BookmarksPage shown in full screenDialogue of bookmarksjumps floating on screenTrigger eveals bookmarks panel when head moves Users head at the middleUser moves head leftHead kept in left positionMiguel Bordallo Lpez, 2014... And even head and face movement tracking has been proposed asa way to control certain parts of the user interface. 15Why dont we usethese kind of methods on our mobile devices?

Miguel Bordallo Lpez, 2014So,

If all these methods have been demonstrated, (some of them even in embedded devices),

Why is it that we dont use them (at least consistently) on our mobile device.

What is it that makes them unpractical?16Challenges of vision-based interactivityVery low latency (below 100 ms.)

Computationally costly algorithms

Sensors (cameras) always on

Energy-efficient solutions

Miguel Bordallo Lpez, 2014The reason is that together with the difficulties posed by the creation of these interaction methods, there are some other challenges that are inherently tied to the platform, in this case the mobile device.

Vision-based interactivity requires very low latency and crips response. Even response times as low as 100ms. can be considered disturbing for many user interface functionalities.

The camera-based algorithms are esentially working with very large amounts of pixels. This usually makes them computationally very costly.

Interacting with the camera, requires that the camera is (or at least appears to be) always on. If we need to push a button to start the camera, the chances of a user interacting using it are smaller.

All these challenges esentially trace back to one characteristic of the mobile devices. They are essentially battery powered devices. Any interaction method that is included on a future mobile platform requires to be energy-efficient.17Objectives of the research Study the relationship between interactivity and energy efficiency

To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements

To provide insight into the computing needs and characteristics of the future camera-based applicationsMiguel Bordallo Lpez, 2014In this context, the main objective of this research was to establish a relationship beween vision-based interactivity and energy efficient. How they relate with each other and what are the implications of considering both at the same time, and not as separate concepts.

The expected implications of the results of this analysis are two fold:

First, to gain understanding on how to build the future mobile platforms to satisfy their interactivity needs

Second, to provide insight into the computing needs and characteristics of the future camera-based applications18

Miguel Bordallo Lpez, 2014To define the process for designing future highly interactive mobile platforms, the doctoral work presented today analyzes the implications of vision-based interactivity at several levels.

In this context, the creation of interactive capture methods and the offer of sufficient user feedback, can engage the user in the collaboration for image acquisition, mitigating the limitations of current applications.

19

Miguel Bordallo Lpez, 2014These concepts have already have an impact in current mobile applications, such as panorama stitiching.

This picture, taken from a Nokia N9 panorama application, shows how the image capturing stage asks for the collaboration of the user and guides him at the same time.

20

Miguel Bordallo Lpez, 2014Applications are usually built on top of complete user interfaces.

The implementation of a vision-based user interface and its analysis allows the understanding of the needs of vision-based interactivity.

Interacting with the device intuitively, using just one hand, can be done by tracking the users position and reacting to the user movements accordingly.

21

Miguel Bordallo Lpez, 2014These concept has already found its way to commercial devices such as the one shown in this short video, the Amazon Fire Phone.

In the video it is possible to perceive the effect of the virtual 3D environment, that practically works as a window to the virtual world.

Just tilting the device or changing where we look at, it is possible to reveal hidden information, not needing any interaction with the touch screen or any button.

22

Miguel Bordallo Lpez, 2014The integration of data from different sensors together with the camera images, and the development of an specific sensing subsystems could enable reducing the latencies in the startup of the camera and the impact in the battery life.

In the picture, we can see that how raising the device to the sight level, starts turning the camera application already ON, giving the impression of an always on camera system, always ready for interaction

23We are stuck with technology when what we really want is just stuff that works

- Douglas AdamsMiguel Bordallo Lpez, 2014The work presented in this thesis, aims at making it possible to change the way that we interact with our devices in the future.

Because, as it was said by Douglas Adams, the author of the Hitchhiker Guide to the Galaxy,

We are stuck with technology when what we really want is just stuff that works

24