MACHINE VISION GROUP

Multimodal sensing-based camera applications

Miguel Bordallo1, Jari Hannuksela1, Olli Silvén1 and Markku Vehviläinen2

1 University of Oulu, Finland2 Nokia Research Center, Tampere, Finland

Jari Hannuksela, Olli SilvénMachine Vision Group, Infotech Oulu

Department of Electrical and Information EngineeeringUniversity of Oulu, Finland

MACHINE VISION GROUP

Outline

Introduction• Modern movile device with multiple

sensorsVision-based User InterfacesSensor data fusion systemApplication case implementations

• Motion-based image browser• Motion sensor assisted panorama

imagingConclusions/Summary

MACHINE VISION GROUP

Introduction

• More and more applications and features are being crammed into handhelds

• Causes usability complications given the constraints of current mobile UIs

• Increased computing power not harnessed for UIs

• Keypad and pointer based UIs and/or touchscreens in current devices– User’s hand obstructs the view– Require two handed operation

MACHINE VISION GROUP

Modern mobile device with multiple sensors• The phone includes touch screen, GPS,

accelerometers, light sensor, proximity sensor • Two cameras: low resolution for video calls and high

resolution for photography and video capture• Newer phones will include magnetometers,

gyroscopes

MACHINE VISION GROUP

Motivation for vision based user interfaces

Allow recognition of the context- Detect user’s actions- Recognize environment

Allow 3D informationProvide interactivity

- Real-time feedback- Single hand use

MACHINE VISION GROUP

Limitations of vision based UIs

Fast movements

Low light

Difficult conditions

+

MACHINE VISION GROUP

The solution: sensor data fusion

Fusing the data obtained by several sensors

• Ambience light sensor determines illumination conditions

• Video analysis detects ego-movements and analyzes the context

• Accelerometers provide complementary motion data

MACHINE VISION GROUP

Video analysis

- Every frame divided into regions- Selection of feature blocks- Estimation of block displacements- Analysis of uncertainty

- Results: 4-paramenter model- X, Y, Z, r

MACHINE VISION GROUP

Sensor data fusion: Accelerometers

MACHINE VISION GROUP

Sensor data fusion

Model the device movement with the folowing

Define a state vector: position, speed, acceleration

Define a measurement model

Apply Kalman filtering adding accelerometer values: State prediction + state correction

MACHINE VISION GROUP

Application cases

• Sensor data fusion method applied in two applications– Implemented on a Nokia N900 mobile phone

• Motion based image browser– Allows browsing large images and maps with one hand operation– Works under different light conditions

• Sensor assisted panorama imaging– Stitches panorama images in real time from video frames– Increased robustness against fast movements and no-texture

frames

MACHINE VISION GROUP

Motion based image browser

Uses fusion model from accelerometers + video analysis to generate commands

• Scroll up/down/left/right• Zoom in/out

Light sensor decides:• if camera should be turned on • weighting factors and uncertainties• 3 modes defined:

• Good image quality (video analysis + accelerometer correction)• Bad image (accelerometers have increased contribution)• No image (only accelerometers are used)

MACHINE VISION GROUP

Motion based image browser II

MACHINE VISION GROUP

Sensor assisted panorama Imaging

•Based on video analysis

•Guides the user with instructions

•>360 degrees panoramas •Real-time registration•Real-time frame evaluation and selection•Real-time frame correction

•Increased robustness via sensor-data integration

MACHINE VISION GROUP

Panorama imaging: Sensor uses

•Uses sensor fusion model to compute camera motion•Increased robustness against fast movements and frames with low/smooth texture

Registration

MACHINE VISION GROUP

Panorama: Sensor uses II

•Uses accelerometer data to detect blur•Detects unwanted shake/tilt•Integrated in scoring system

Selection

MACHINE VISION GROUP

Summary

• Vision based interfaces offer high interactivity with one hand operation

• They present several limitations• Sensor fusion improves motion estimation

adding robusness against fast movements and dark conditions

• The framework can be included in several applications (e.g. as a part of Motion Estimation API)

MACHINE VISION GROUP

Conclusions

• We have presented a sensor fusion framework that fuses vide analysis with motion sensors (acelerometers+magnetometers+gyroscopes)

• We have presented two applications cases that make use of sensor data fusion and integration

• The applications presented are by no means the only ways to apply vision or multiple sensors, and one may find new interesting possibilities in further research

MACHINE VISION GROUP

Thank you!

Any question???