Rock steady video stabilization on your on-the-move video devices

Copyright © 2016 FotoNation 1

DIGITAL GIMBALRock-Steady Video Stabilization Without Extra Weight!

Dr. Petronel BigioiMay 3, 2016


IMAGING

Wide ExpertiseStrong Innovation

Top Quality

SOFTWARE

Always Real TimePlatform Agnostic

Full Flexibility

HARDWARE

ProvenLow Power

High Performance

QA

Advanced Data AcquisitionMulti-million Image Database

Testing Automation Tools

FotoNation in a Nutshell


FotoNation in Numbers

550+ 1000+ 60% 2.5+ 450+imaging patents man years of experience high end smartphone penetration billion units shipped million devices / year

understand ● enhance ● accelerate


DIGITAL GIMBALA state-of-the-art, energy-efficient image stabilization solution for high frequency vibration by FotoNation


Problem Statement

High quality videos 4K @ 60 FPS are accepted as norm in cameras on the fly and move

Low frequency motion is not the only problem anymore

HIGH and VERY HIGH frequency motion due to motors and activities need to be addressed including motions with horizon lock


Mechanical Gimbal Solution — Current State- of-the-Art to Attenuate Motors Vibration Effects

ADVANTAGES

DISADVANTAGES

WHAT IS IT?• Mechanical assembly built to counteract vibration effects• Uses gyros in close loop to control motors

• Have large range of movement

• Cost!!!• Large & inconvenient• Heavy• Consume power• Reduce flight time due to extra weight• Slow reaction time


Digital Gimbal Solution — FotoNation’s technology to deal with vibrations (low and high frequency)

ADVANTAGES

DISADVANTAGES

WHAT IS IT?• Algorithms removing the need for mechanical assembly; uses gyro to measure high frequency

vibration and digital image processing for frame to frame registration to reverse the effects in the image/frames

• Low cost• No extra weight• Low power• Ultra fast reaction time• Advanced image stabilisation• Rolling shutter correction

• Limited range of movement


Image Stabilization Requirements

Image stabilization has always been a subject of great interest especially when associated with camera modules on the move; today’s solutions are influenced by system requirements (cost) as well as jitter specific problems (usage).

PerformanceSystem requirements

good sampling qualitystable frame rate

small error estimationslow power consumption

future proof (2K, 4K, 8K)

roll (x-y rotation)panning

out of plane rotationsrolling shutter distortions

large moving objects


The rolling Shutter Problem

Hand jitter vibration patterns Drone/motorbike jitter vibration patterns

Rolling shutter effects — caused by slow camera motion during line by line exposure and frame readout

Rolling shutter effects — caused by rapid camera motion during line by line exposure and frame readout

Have low frequency (up to 15Hz) Have high frequency (hundreds of Hz)


ISP

Inertial Sensor

Motion Filtering

Camera projection

model

Motion Correction

Image

Corrected Image

Correction Grid

Motion Estimation

Image Sensor

Legend:

Memory Buffer

Digital Gimbal Components

HW ModuleApp Processor & DDR

256 independent motion vectors;

IMU samplesacquisition and sync;

sensor fusion(IMU + frame to frame)

Corrects lens distortion, roll, pan & complex

rolling shutter in 1 pass

Adaptive motion filtering & rolling

shutter effects estimation

System Architecture and Workflow


Motion Estimation — Synchronization

• Accurate synchronization of inertial and video data is essential

• Hardware core used to read inertial data (IMU) and assign precise timestamps, with same time source for frames and IMU samples

• Two possible implementations• Pure HW implementation — ensures minimum latency and highest timing

accuracy, low power consumption• Hybrid implementation — more flexible, but higher power. Requires high

sampling frequency to minimize motion measurement errors


SOFInterrupt

SOFInterruptSample with

FSYNC markerSample with

FSYNC marker

unknown timeperiods

Maximum unknown period is equal to IMU sample interval. Very inaccurate for low sampling frequencies

Hybrid Synchronization Timing Accuracy Limit


Bias — a constant value added to the measured signal

Motion Filtering — IMU calibration


Sampled IMU data needs to be filtered (4x rule) in order to avoid aliasing

Motion Filtering — Data Filtering


Use sensor fusion to track camera orientation over time

Using gyroscope alone to track camera orientation leads to error accumulation

Acceleration and magnetic field lack short-term accuracy but provide good long-term reference frame

Gyroscope provides short-term accurate changes in the camera orientation

Motion Filtering — Sensor Fusion


• The real camera orientation is tracked using quaternion arithmetic

• The required correction is the difference between the real and the estimated trajectory

• Our motion filters adapts to the camera motion in order to produce naturally smooth experience and react fast to the intentional motion

Motion Filtering


• Stabilisation margin is limited by the sensor size. Sudden camera movements can cause optimal correction to exceed available correction margin. Excessive motion needs to be limited to the available correction margin

• Limits imposed on 2D projection must control 3D rotation represented as 4D quaternion — difficult task. Problem gets even more complicated for highly distorted rectilinear lenses (left) or fisheye lenses (right)

Motion Filtering — Motion Limitation


time

reference line

• Orientation of the camera in the first line is the reference point

• Each consecutive line is transformed to counteract camera movement up to the given line

• Inertial samples and the video frames must be preciselysynchronized

• Camera projection must be accurately modeled

Motion Correction — Rolling Shutter Correction


Motion Correction — Rolling Shutter Correction


Motion Correction – Rolling Shutter Correction


Camera framereference vector

Gravity vectorfrom

accelerometer

Correction amount

After correction vectors are aligned

The horizon locking feature keeps the horizon line in the middle of theframe and horizontal regardless of the drone orientation.

Motion Correction — Horizon Locking Option


Motion Correction With “Follow Me” Feature / Owner Tracking


FUTUREPROOF

FAST TODEPLOY

EFFICIENTLYDESIGNED

In a seamless way that is simultaneously

A DEDICATEDHARDWARE — IPU

Low gate count implementationto offload heavy image and video computation. Most difficult tasks

completed locally and then passing processed data to host CPU

How Do We Deliver This?


OBJECT DETECTION ENGINE~ 1M gates, 240 kB Sram, 54 mW

Multi-Core CPU

GPU DSP ISP

3G/4G Baseband

Memory & I/OsIPU

Video Encoder/Decoder

MOTION PROCESSING ENGINE~ 125K gates, 9.8 kB Sram, 2 mW

DISTORTION CORRECTION ENGINE~ 390K gates, 48 kB Sram, 18 mW4K@60FPS, 28nm technology Typical AP

IPU components overview


• Precise lens modeling allows for lens distortion correction and re-projection.

• Any type of correction is possible (rectilinear to perspective, fisheye to perspective, distorted fisheye to perfect fisheye, fisheye to cylindrical, arbitrary input projection to arbitrary output projection and freeform image warping).

• All distortion corrections are done at the same time with stabilisation and high frequency rolling shutter correction (single pass).

• Adding distortion correction has no impact on system performance.

• Frames are resampled only once to ensure maximum image quality.

• Lowest possible power consumption (18mW for 4k@60fps correction in 28n technology).

Motion Correction — Uses FotoNation’s DCE(Distortion Correction Engine)


DCE GPU

Designed to work withrectangular texture mapping for 2D image transformations

Native support forbicubic resampling

Scalable with increasing input size and frame rata

Small gate count

Low power consumption

Low bandwidth thanks to specific cache design

Designed to support generic triangular texture mapping for

3D scenes rendering

Bi-cubic resampling requires extra processing power & time

Designed to work at display resolution

Large gate count

High power consumption

High bandwith

Motion Correction — Done DCE (Distortion Correction Engine) Part of FotoNation’s IPU (Image Processing Unit)


INPUT OUTPUT

Correction — DCE example

Corrected by FotoNation DCE


Correction — DCE example

INPUT OUTPUT

Corrected by FotoNation DCE


FotoNation EIS / High Frequency Stabilization


• Digital Gimbal components (synchronization, motion filtering and correction) are part of FotoNation’s IPU (Image Processing Unit) for optimum low power high performance implementation.

• IPU incorporates additional units to enable more features such as owner recognition and tracking combined with image stabilization (e.g. high performance object detection and tracking, high performance face detection and tracking and face recognition).

Conclusion


Thank You


FotoNation EIS / a no trade-off user experience


FotoNation EIS / a no trade-off user experience

Devices & Hardware

Rock steady video stabilization on your on-the-move video devices